There has been a lot of hype around Behaviour-Driven Development (BDD) in the testing community over the last couple of years. Certain tools started gaining popularity, people started discussing this ‘new approach’ to testing, and others even adopted and implemented it in their solutions. Everything moved very quickly to the point that if you search for any test automation framework today, chances are it supports BDD.

Although BDD has gained in popularity, when I hear people from the testing world talking about it, most of the time they are not truly aware of what it is. Instead, they often talk about BDD in a very limited scope and are simply referring to tests written in a business readable syntax or to BDD tools such as Cucumber.

With this being the case, I decided to write this post to provide a better understanding of what BDD is, the ways in which Cucumber supports BDD processes, and if Cucumber is the right testing tool for you to use.

WHAT IS BDD REALLY ABOUT? 

In 2003, a small, closed group of people from the XP community got together to explore better ways to do Test Driven Development (TDD). During that time, Dan North, an experienced technology consultant, introduced the concept of BDD as a means of clearing confusion between testers, developers, and businesspeople.

BDD’s introduction was important because clear communication and understanding between engineers and businesspeople is key for a project to succeed. Miscommunication in between teams and in requirements complexity can often result in a technically functional end product which fails to meet the exact requirements of the business.

"BDD is a process designed to aid the management and the delivery of software development projects by improving communication between engineers and business professionals. In so doing, BDD ensures all development projects remain focused on delivering what the business actually needs while meeting all requirements of the user." – Konstantin Kudryashov, Alistair Stead, Dan North from the blog post ‘The Beginner's Guide to BDD’

• Encouraging collaboration across roles to build shared understanding of the problem to be solved
• Working in rapid, small iterations to increase feedback and the flow of value
• Producing system documentation that is automatically checked against the system’s behaviour

The idea behind BDD is to combine automated acceptance tests, functional requirements, and software documentation into one format that could not only be understandable by engineering teams, but also by non-technical people. The influential software delivery consultant Gojko Adzic gave this process a different name, calling it ‘Specification by Example’. I find this phrase to be more descriptive, but it might not be as widely used.

A very simplified version of the process consists of two activities: Specification workshops and outside-in development. For the first activity, the ‘three amigos‘ – that is, a business analyst, a developer, and a quality assurance team member–gather to discuss how features should be implemented and elaborate on a set of examples regarding how the software should behave in a ubiquitous language. The second activity is where the developers start to build the application. It is called ‘outside-in’, because they typically start with the functionality that is closest to the user and work their way into the core as they discover more of what needs to be implemented.

BDD AND TEST AUTOMATION 

Now that you know what BDD is really about, let’s discuss the role Test Automation plays in it.

Previously, I mentioned one of the main activities of BDD called ‘Specification by Example’, where the three amigos deliver the examples of the system’s behaviour. These examples are written in Gherkin, a domain-specific language originally created and maintained by Cucumber, one of the most popular BDD frameworks. This semi-formal language is the key ingredient which makes specifications naturally easy to automate and understand by anyone on the team.

The scenarios obtained from ‘Specification by Example’ are grouped into a corresponding feature file in which we define the expected behaviour of our application. Scenarios are meant to be short and to sound like plain English. Each scenario has the following structure:

1. Given some initial state
2. When an action is taken
3. Then verify an outcome

This is a perfectly valid behaviour example which could be used to begin building an application and communicate to the team what is expected from it. Although this is a great starting point, we will need a tool for this to translate to Test Automation.

Remember the group of people from 2003 exploring the concepts of BDD that I mentioned earlier? Well, one of those people was Aslak Hellesøy, Cucumber’s mastermind. He went on to found Cucumber in 2008 with a vision to create a language, a process, and a tool that would provide ‘a single source of truth of software behaviour for both non-technical and technical project members’.

This is where the aforementioned misconceptions surrounding BDD originates.

Image from Cucumber’s blog

If you consider Cucumber as a software tool, you could say it is basically a ‘Test Runner’ such as JUnit or TestNG, and if you ask me, it is a fair statement. The thing is, though, you cannot get the value of tools built to support BDD if you take out of the intended context of collaboration.

When Cucumber is adopted solely as a tool to write automated tests without any input from business analysts, they tend to become imperious and lose their documentation value. Why? Because Cucumber features should drive your implementation, not reflect it.

"If all you need is a testing tool for driving a mouse and a keyboard, don’t use Cucumber. There are other tools that are designed to do this with far less abstraction and typing overhead than Cucumber." – Aslak Hellesøy, Cucumber creator

The added complexity through Gherkin and Step Definitions can be perfectly justifiable if it helps improve collaboration and reduce misunderstandings. However, if you use the tool simply as a test runner and you write your scenarios after the application is built, then it just makes no sense.

CHOOSING THE RIGHT TOOLS 

A defining factor for successfully applying test automation in software projects is choosing and using the right set of tools.

Nowadays there are a wide variety of tools you can pick for your automation project, but it all depends on which programming language you plan on using to implement your test automation solution. Picking the right tool will depend on the scope of your project, the types of testing you will need to perform, libraries integration, technology limitations, etc.

There is no such thing as a ‘standard’ solution that will work perfectly in every case. However, if you make sure the tool includes some of the following features you will be off to a good start:

• Supports parallel executions
• Integrates seamlessly with CI tools 
• Open source (if possible)
• The repository is actively maintained 
• Has great documentation
• Fair support online on forums
• Good reviews
• Major companies use the tool

There is one other decision you have to make, one that is key, and that is if you plan on going down the BDD lane or not. Ask your team, and if they’re planning to work with a true BDD approach then you should definitely consider using a tool such as Cucumber.

If that is not the case, then I would suggest you go with a more ‘traditional’ tool. Remember, using the wrong tool for a job doesn’t lead to success.

The problem with applying BDD to test automation when there’s not a true BDD process in place is that we will be dealing with the complexity of a BDD tool and end up undermining what we’re trying to accomplish in the first place. Let me tell you why.

Cumbersome syntax

Automation Engineers certainly play a vital role in the delivery process, but unfortunately, we are not the most qualified to write the requirements of a system in Gherkin syntax. Ignoring rules such as writing all steps in a third-person point of view and writing steps as a subject-predicate action phrase results in failed attempts to write a scenario which is clear, simple and atomic, and instead we end up writing one that contains way too many steps, which can lead to overcomplication.

An extra layer of complexity

Every step in our feature files requires a matching logic implementation. This is called a ‘step definition’ or ‘step implementation’. Writing reusable steps is hard and finding a particular step you have already implemented once your project has grown in size is even harder. Not only does this add an extra layer of complexity to our Test Automation Solution, but also if we consider my previous point I can assure you there will be plenty of ‘garbage’ or ‘one-time-use’ steps which is definitely not a good practice, no matter what.

More dependencies 

Another issue with BDD tools is that they add another layer of dependencies. Limiting the number of dependencies your solution has is key. The more dependencies you have, the higher chance there is for something to go wrong. And trust me, you do not want to deal with dependency conflicts.

Performance issues 

This can be divided into two main areas: Human error and tool limitations.

A very common mistake when we think of test automation is to think of it as end-to-end, GUI-based examples. This type of thinking tends to create slow, brittle tests, while tests at the service layer are more stable and run faster since they are not so prone to changes and do not rely on so many components to fulfill a request. This is mainly influenced with the BDD approach because BDD scenarios are written from a user’s point of view, but that doesn’t mean the functionality needs to be tested directly from the GUI.

Tools certainly play a vital role in the performance of our solution, especially nowadays that we need to test our suites across a vast number of OS/browser combinations and even mobile devices. This is an area where some tools unfortunately still struggle either due to language limitations or simply because of lack of support.

Limited parallelisation

No matter if we are running tests at the service or GUI layer, we can always benefit from a bit more performance, and in the case of Test Automation, performance cannot be achieved without parallelisation. Even if you run a couple tests in parallel, it will still be better than running the whole suite sequentially. However, that should be limited by the infrastructure, not your choice of Testing Framework.

Parallelisation can be achieved on two different levels:

1. By feature
2. By scenario / outline

Parallelisation at feature level means every scenario in a feature will be executed in the same thread, therefore your features will be triggered in parallel, but every test within that feature will run sequentially. Parallel executions by scenario, on the other hand, allow scenarios and rows in a scenario outline to be executed in multiple threads.

Therefore, if you have a couple of suites with five scenarios each and every test takes a minute to run, your execution would take five minutes to finish with the first approach, while with the second one it would take only one minute (assuming your infrastructure or cloud service can handle any number of tests in parallel, of course).

I call this ‘100% effective parallelism’, which means your suite should be as fast as your slowest test. That is what you want to achieve.

Prior to Cucumber 4’s release you could only run features in parallel. This was not optimal but there was nothing you could do, since that was directly tied to Cucumber’s underlying test runner. Of course you can always think of different workarounds, but that will cost you time and money, and even if you get to improve the performance, you’re still working around an issue that could have been avoided by using the right tool for the job.

As of version 4 you now have the option to use TestNG as the base runner, enabling parallel executions at the scenario level by simply setting a few parameters and extending a base class. Cucumber-JUnit, on the other hand, does not support parallel executions at the scenario level, yet.

You can find more details on this subject in Cucumber’s parallel execution docs.

CLOSING REMARKS

Success heavily relies on collaboration, no matter if you are doing BDD or not. Proper communication and teamwork are a must to achieve your goals. Once you understand these goals you can then discuss the potential approaches with your team and decide onwhat the best fit for you is.

If Cucumber is the right tool for the job, then make sure you use as it was intended by their creators and follow their best practises and guidelines. Remember, most of the Test Automation best practises are also tool-agnostic, like writing small, atomic and autonomous tests to reduce the risk of unreliable results, but Cucumber also has its own set of best practises. Focus on what your user wants to do – not on the implementation details. Write non-conjunctive, reusable steps with a robust structure and clear meaning, and make sure you take a good look at Cucumber’s Gherkin Reference documentation.

I hope this article helped you better understand what BDD is and why tools such as Cucumber should not be used solely as test automation tools.