What is Your Software Testing Strategy?

When you lack a robust software testing strategy, you run the risk of not meeting stakeholder needs, and ultimately delivering a bad end-client experience. Simply put, bugs need to be caught, and not by clients.

There are different efforts that can improve testing such as automating tests, automating the hand-offs between tools and improving collaboration between dev and testers. Before these different areas are addressed, having an overall strategy for the volume and frequency of different types of tests will help focus energy on how to optimize the process.

For example if there’s no strategy in place and a large number of end-to-end tests discover a large number of issues, but the underlying reasons are largely unknown…this volume of tests simply isn’t productive.

Testing Pyramid Explained

A lot has been written on the Test Pyramid and anti-patterns to the pyramid. The popular concept, developed by Mike Cohn, remains the best chance at building up a QA team that should jumpstart testing. However, given the makeup of most software development teams there are significant challenges to implementing the Test Pyramid. Let’s take a closer look at how a Test Pyramid provides high value.

At its tenet, the Test Pyramid answers the question of when and where to test your software product. Its framework for “testing as needed” powers both developer teams and QA teams because it eliminates unnecessary testing, freeing teams to focus on high-quality software products. Implemented correctly, the Testing Pyramid reduces the time required for developers to identify and resolve any bugs in their software. Altogether, integrating the Testing Pyramid into your product development will help build a more reliable test suite.

The Testing Pyramid specifies the different types of tests to be included in the automated test suite. It systematically outlines the test sequence and expected frequency of them. The key to the Test Automation pyramid is the need to have quick feedback that ensures any code changes do not cause significant disruptions within the software product.

The Test Automation Pyramid operates at three levels:

Unit Tests

Unit tests are the first level in the test automation pyramid. Testing the individual functionalities and components within this basic level should be done to validate that each works seamlessly, even in isolated conditions. It would be prudent to carry out several scenarios within this level, including happy path and error handling, among others.

Note that since this is the most significant subset within the pyramid, writing and running code needs to be prompt. The number of unit tests you conduct is mainly dependent on the number of features you add. You need to run this test suite every time you add a new feature so your developer team can receive immediate feedback from each individual component.

You’ll also want to come up with a fast-running unit test suite. This will encourage your developer team to run the test as often as possible. Consider using test-driven development (TDD) to build a robust unit test, which should leave you with a simpler, clearer, bug-free code.

Integration Tests 

Level 2 of the Test Pyramid is integration tests. It takes over from the unit tests that verify smaller pieces of the codebase. Integration tests take on the task of checking how each of these code subsets interacts with the rest of the code. Specifically, this level within your Testing Pyramid should help validate the interaction of one piece of code with external components such as databases and external services.

As the second level of your pyramid, it should not be run as frequently as you would the unit tests. Also note that since integration tests involve interaction with external services, they will likely run slower than your unit tests. Further, you will need a pre-production environment to be able to run the tests efficiently. Altogether, integration tests check how a feature added by your developer team communicates with external dependencies.

As integration tests are complex and difficult to automate, it’s not a surprise that they can become a bottleneck in the overall process and a key contributor to a long testing cycle.

End-to-End Tests 

At the top of the Testing Pyramid is the End-to-End tests. This sequence of tests is expected to ensure that the different elements within the code function as required. As alluded to in the name, this Test Suite is expected to check if the software works seamlessly from start to end.

At this juncture, you want your dev and QA teams to imagine the user’s perspective. They should consider how the user will interact with the software product. After that, these teams should come up with the test codes that replicate these interactions.

The End-to-End tests take the longest to run. These are fragile because the test has to make room for a variety of user scenarios. In some instances, the test suite might require that the different features communicate with external dependencies, which affects the turnaround time.

Not officially part of the testing pyramid strategy is the unknown-sized cloud above the pyramid, which consists of additional manual tests. These are either performed by the team, or simply represent the bugs caught by clients (hopefully a small amount).

Anti-Patterns and Their Drawbacks 

As you embark on product testing, you want to have a good understanding of what is considered the right way to test your code and what is not. As such, there are three anti-patterns of software development that should guide your teams.

Ice Cream Cone 

Alister Scott first introduced the Ice Cream Cone in a bid to explain dev teams who believe they can take on UI independently. This strategy (usually inadvertently selected) has a large sized cloud of manual testing at the top, followed by an inverted pyramid (extra e2e, less integration, even less unit).

This pattern is exceptionally detrimental as it eats into the budget and takes a toll on delivery time. The overall opportunity to automate is minimal, due to the reliance on manual testing. If you don’t continuously test, you won’t be able to continuously deploy. Yikes! The goal was hitting the market as expected.

Cupcake 

The Cupcake anti-pattern was first introduced in the ThoughtWorks blog. This anti-pattern occurs when there is insufficient low-level testing, including those within the unit, integration, and component levels. Instead, there will often be too many tests running through the Graphic User Interface (GUI) in conjunction with myriad other manual tests. In retrospect, the developers conducting the automated integration tests and unit tests will be at the very bottom, followed by the Automated GUI testers and a larger group of manual testers at the very top.

A common trap of the cupcake anti-pattern is that most of the team will be repeating tests which are largely ineffective. Your teams are then unable to focus on more complex test issues, which takes a toll on the deployment schedule.

Hourglass 

The hourglass anti-pattern is evident where you have your dev team writing unit tests and an additional QA team writing UI tests. The result is that there is no one within your organization writing any of the middle-layer tests. Particularly, there will be engineers working on the unit tests and those working on the end-to-end tests, but no one working on integration tests.

Anti-pattern summary

Altogether, these anti-patterns create a test architecture that is unreliable and prone to missteps. They create a development environment in which the different teams are unable to move with the confidence and speed needed for mature Agile development

Being stuck within the ice cream cone, cupcake, or hourglass means that the current dev team puts most of its QA efforts and labor on manual tests and executions. The number of required tests grows linearly and exponentially. They can lead to the enterprise to restaff to make room for testers who are supposed to check for the bugs and avoid outages.

Overall, these patterns are largely unsustainable for enterprise as it results in a growing amount of testers and costs while deployment remains delayed. Testing becomes a bottleneck in the overall software delivery lifecycle, impeding DevOps collaboration and stifling innovation. The lack of effective testing also creates a situation where it is impossible to test out new additions to the code base, increasing the risk of poor-quality code being released to clients.

Using the Testing Pyramid

These software testing anti-patterns no longer cut it in the current marketplace. Fortunately, the Testing Pyramid remains an effective strategy to increase software quality, and when used properly, will accelerate software delivery.

The ideal scenario would move away from separate dev and QA teams, instead having a quality focused team member that shifts testing priorities into the development team. The move to the Testing Pyramid opens the way to automate not just individual tests, but the testing process itself, including the auto-generation of ephemeral environments with infrastructure as code (IaC).

With support from the Guide Rails platform, you can quickly automate your overall testing process. Capabilities like shared pipelines, tests, and test data coupled with auto creation and tear down of environments that match production all come standard. Dev and test teams can come together to focus on writing better tests, reducing the cost of the entire product and the time between conception and market.