How we (re)defined our E2E testing with BDD
End to end (E2E) tests are the type of software testing which is used to test application flow according to critical path use cases based on the business requirements. This type of software testing is introduced when the software has developed past early stages of rapid iteration and wholesale change and therefore provides expectations of a predictable level of stable functionality for end users. .
According to Scrum values, testing is the responsibility of everyone in the organization. However, the main responsibility for the tests’ maintenance and development could shift, depending on the size and maturity of the software development team. In this scenario E2E tests would be handled by a dedicated QA team.
Continuous clarification between requirements and tests is essential
Fig. 1 Example feature understanding in standard product development model
Under these circumstances a problem occurs when the capacity dedicated to maintenance of the test suite is not sufficient to keep up with the product development happening within the development teams. Then automated tests could lose their significance and value to the product development process.
In another scenario, E2E test could be maintained partially by the development team, which brings up possible issues with the synchronization of the test suite between teams.
Proposed solution
To improve collaboration and clarity between business, testers, and developers we have introduced Cucumber as a Behaviour Driven Development (BDD) framework.
In order to better understand these terms as well as the relationship between them, refer to their source docs for detailed info:
Cucumber considers product features and scenarios as shareable artifacts that could be understood by everyone on different technical expertise levels. BDD artifacts are written in domain specific language, Gherkin. Scenarios could be used as the single source of truth in manual tests and tests automation. BDD may leverage team responsibility among QA. Scenarios could also be used as a documentation.
Implementation
Steps are written in Gherkin, a domain specific language designed for BDD, and saved in a product tracking tool like JIRA. At Brainly we use Teamcity as our CI/CD platform to run E2E tests on a staging environment after each merge to the master branch.
The core libraries we use in test automation are:
- Jest – A NodeJS framework for writing various test types, mainly unit tests. We use Jest for basically running tests suites written in JS.
- Puppeteer – A Chromium based engine used to run tasks in the browser. We use Puppeteer to open a page on the Brainly website. We simulate user actions, like clicking and typing using the Puppeteer API.
- Jest-Cucumber – A Gherkin file loader, to unwrap/enable-unwrapping particular steps and scenarios using regular expressions.
Regular tests suite written in Jest contains steps and expected outputs defined by a development team. As they are maintained only by technical people, they may be too imperative – they describe more how to perform scenarios from requirements instead of just what to perform. Therefore, they might be not understandable for business representatives.
Cucumber promotes the separation between business logic and technical implementation. Firstly, Gherkin considers features as basic units representing business requirements. . Example of the feature could be an answer to the question. Secondly, the feature itself could contain several scenarios containing specified steps. Each step of the scenario could have a different prefix:
- prerequisite (given)
- action (when)
- output (then)
Implementation of the feature from Fig 4 may be done using the Cucumber library, available in the most popular programming languages, like Java, JS, Python, C#, etc. and integrated with the Gherkin language. A solution worth considering for testing stack (which is based on Jest) is the jest-cucumber library. Jest-cucumber is used for loading feature files written in Gherkin and executing them in Jest. Each step is being extracted from the feature files using regex. In case some step or scenario is not covered, jest-cucumber will throw an error. his helps to keep scripts maintained.
Thanks to jest-cucumber, migration to BDD is not time-consuming or an all-or-nothing process. Test suites can be replaced one file after another. Once the entire codebase is migrated to BDD, an integration between JIRA and the Git repository containing the source code would be a step forward in order to synchronize automated scenarios.
Conclusion
Updating an existing codebase to use BDD is only the first step. Application requirements may change day-by-day and test coverage should strive to be ahead of new app versions. That is why collaboration between a business, QA and development teams is so significant.
To sum up, Behaviour Driven Development can bring impressive benefits. Some of the most crucial of which are:
- Tests are defined in the same way for every member of the organization.
- Tests are meaningful as every use case is visible for the business (and thereafter might be externally supervised).
- Requirements are clarified and the capacity previously used for communication between teams can now be used for improving descriptions of the features.
