We often hear teams complaining about test automation. They usually want more of it, but don’t have time to do this. Everyone struggling to do enough testing thinks automation will be the solution. However, if you speak to teams with large numbers of automated tests and they will tell you the other side of the story. Their tests are often slow and take too long to run, so they don’t run them as often. Or the results are unpredictable or flaky so they end up ignoring the results much of the time. So what’s the verdict, do we want more test automation or not?
What do we mean by test automation?
Of course the devil is in the details. When we dig deeper, everyone is talking about test automation as if it is one thing but they all mean something slightly different. Developers are mostly referring to unit testing. Testers are mostly referring to GUI automation. And managers, bless them, are asking for 100% coverage without fully understanding what that means.
A good starting point is introducing everyone to the Test automation pyramid. You can read about it in detail here: https://martinfowler.com/articles/practical-test-pyramid.html
Do you have a volcano or an ice cream cone?
Once everyone understands the different levels of test automation in the pyramid, we ask teams to draw how their “pyramid” looks today. Where the size of the layer in the pyramid is representative of the relative number of tests in that layer. The ‘ideal’ picture looks like a volcano (see the image below), but mostly we find teams who draw an ice cream cone. A small number of unit tests, a larger number of API and GUI tests, and a big scoop of manual tests to top it all off!
How much do you duplicate tests?
Even worse than the ice cream cone, is the realisation that test cases are being duplicated at every level.
Consider an example of a login screen.
Since it’s often the first thing you use in a website or application, most people want to automate this first. And, because they are excited about their new GUI test automation, they automate all the test cases they can think of. Wrong user names, wrong passwords, passwords too short, passwords too long. If you can think of it, someone has automated it. But guess what, the developer also added some API tests for it. And actually you are using a shared login component. So of course the team who wrote the shared component tested all the same cases in the unit tests.
Do you even need to automate it?
Of all the parts of your application or website, what do you use daily? The login, right? If the login breaks, how long do you think it would take you to know there was a problem? Even if you have zero test automation for it? This is probably the one thing you never need to test, as its something everyone (including devs and testers) use daily to use the system. If something is wrong you will know VERY quickly!
What can we do instead?
So what is the solution? We suggest you use your refinement sessions and planning sessions to talk about test automation in detail as a whole team.
Once you understand the solution together, you can discuss the risk of problems and decide what needs to be automated, and which level is most appropriate. Discuss which test cases you will automate at which level. Remember, if you are testing all the different password options at the unit test level then don’t repeat those tests manually! That is a waste of time and effort.
Push test automation down the pyramid
You can have tests at every level of the pyramid, but it’s a good idea to figure out which tests would be most effective at each level, and in all cases aim to push tests down the pyramid to where they are faster to run, and more robust. You might do a single UI level test, a few API level tests, and use unit tests for all the possible permutations of error conditions. It is much cheaper to write one unit test, than to manually test something every single time you want to release.
Include test automation in your definition of done
When you create the tasks for a feature or user story, include creating the automated tests as part of the story. That way they won’t fall behind. Tests are just as important as the code you write, and your automated tests should be version controlled and labelled along side your code. Add test automation to your definition of done to make sure test automation is keeping pace with development. Remember the whole team is responsible for quality, so anyone on the team can pick up automation tasks.
Your turn
What’s the biggest mistakes you’ve seen with test automation? Let us know in the comments!
How agile is your testing?
Take our quiz to find out…