I was recently in a conversation with testers about how acceptance tests should be structured. As we revisit how Blackboard conducts test automation, we were looking at patterns and practices to employ. I shared my following perspective on how to think about the architecture of your acceptance test infrastructure, the lifecycle of a specification, as well as who should conduct which step. This post is really a teaser post, as I am merely smearing pretty pictures on the web, without much background. Isn't that what follow-up blogs are for?
Acceptance Test Composition
A well formed acceptance test architecture is composed of five core components:
- Specification - This is the document that describes what the system should be doing. The requirement that will be exercised by the application. In an automated testing framework, this is usually implemented in Gherkin (Cucumber) or a wiki (Fitnesse). It should be easily read and understood by all members of the organization at any level and written in the business language. I recommend collaboratively white boarding your specs (a la Three Amigos meeting) and have somebody write it in the technology later.
- Fixture - This is the glue code. This code understands the format that the specification is written in, and knows how to talk to either the application driver or the Automation Layer to implement the features of a test. This is generally code. Ideally it is written in a language similar to the SUT so everybody can update it. In Cucumber, these are the Step Definitions and in Fitnesse these Fixtures.
- Automation (DSL) - Don't start building this from day one. The automation layer is the Domain Specific Language (DSL) that will evolve as you build out your fixtures. It is an abstraction layer over the application driver. If you design this correctly to be a fluent API or even a full on DSL, then you will find that non-technical members of the team can look at this an possibly even contribute here. But don't set out to build a DSL...cause that ain't easy.
- Application Driver - This is the code that knows how to talk to the system under test. For a web application, this is typically a framework like Selenium. If you are writing tests for a web service, then you will need to pick a tool that can send/receive SOAP or REST messages. If your application is a batch processing system that takes a package of zipped XML files and puts them in the database, then you need to build a custom framework that knows how to build the zip file, and verify the database got updated properly.
- System Under Test (SUT) - This is the product you are building. This is what you are testing.
Acceptance Test Lifecycle
An acceptancetest should evolve collaboratively via a cross functional team. I think about it in three phases:
- Collaborate and Define - This is typically where you have a 'Three Amigos Meeting' where members of the business/analysts, testers, and engineers (and others too) will get together and write the requirement and the acceptance criteria. This can happen on a white board or on paper.
- Automation Phase - Once a specification has been authored, then the specification will be captured in the automated testing tool and checked into version control. Then the test fixture and DSL will be authored to automate the failing test.
- Verification Phase - This is the last phase. The code to satisfy the test is written now. The details of the DSL might also be filled in here (for instance, elements on the page to be called). The test is then run and re-run until it passes. When the test passes, the feature is "DONE". The verification will continue via continuous integration.
You may note that you this may smell like Test Driven Development. It should, cause it is...or Acceptance Test Driven Development.
I would also like to comment on who should do what.
- Business and/or Analyst - should focus on the user story and the acceptance criteria.
- QA - should focus on the user story, acceptance criteria, the test fixture and the DSL.
- Engineer - An engineer should be involved in all aspects of the acceptance test. They are ultimately responsible for satisfying the requirement.
I firmly believe that an engineer (or an Engineer in Test) needs to be focused on the DSL layer and the SUT. This is heavy coding and design and should be not be where "testers learning to code" should focus their time.