The focus is often on the wrong problems
Test automation is often presented as a natural step in the evolution of software development. Organizations want to release faster, improve quality, and reduce their dependence on manual testing. The benefits seem obvious. Yet many initiatives still fall short of their objectives after a promising start.
The first reaction is often to look at the technology. Was the right framework selected? Are the tests well designed? Is the CI/CD pipeline optimally configured?
Technology is frequently blamed. In practice, however, technology is rarely the limiting factor.
While these are relevant questions, after many years in the field I continue to see the same pattern:
Most test automation projects fail not because of technology, but because of culture.
By culture, I do not mean corporate values displayed on office walls. I mean the day-to-day choices teams make regarding ownership, collaboration, and responsibility for quality.
The focus is often on the wrong problems
When an organization decides to invest in test automation, much of the attention goes to tooling and architecture. Frameworks are selected, pipelines are configured, and dashboards are built. Discussions revolve around programming languages, test strategies, and reporting.
These are important topics. Without a solid technical foundation, sustainable test automation is impossible. However, a fundamental misunderstanding often emerges at this stage. Test automation is assumed to be primarily a technical challenge, while the greatest success factor is often organizational.
In organizations where test automation eventually stalls, the same patterns frequently appear:
- Developers see quality primarily as the responsibility of testers.
- Test automation is assigned to a specialized team.
- Management focuses mainly on cost savings and delivery speed.
- Success is measured by the number of automated tests.
- Quality only becomes a topic of discussion after functionality has been developed.
Under these circumstances, even the best technical solution cannot meet expectations.
When test automation tries to solve the wrong problems
In the early stages, a test automation initiative often appears successful. More tests are added, the first results become visible, and stakeholders see dashboards filled with green checkmarks.
Over time, however, cracks begin to appear. Maintenance efforts grow faster than expected. Tests become unstable. Execution times increase. Teams lose confidence in the results. Eventually, test automation is perceived as something that consumes time rather than adds value.
This is usually not a tooling problem.
- A framework cannot correct unclear requirements.
- A pipeline cannot solve a lack of ownership.
- A test suite cannot create a quality culture.
- What test automation can do is expose the strengths and weaknesses of an organization’s approach to quality.
In that sense, test automation acts as a magnifying glass. Organizations with a strong quality culture derive greater value from test automation. Organizations with structural quality issues simply expose those issues more quickly.
The misconception that quality is created during testing
One of the most persistent misconceptions in software development is the idea that quality is created during the testing phase.
This is understandable. Testing is, after all, the moment when defects become visible. However, discovering a problem is not the same as creating quality.
Quality is established much earlier:
When requirements are discussed and refined.
When risks are identified and made explicit.
When design decisions are made.
When teams consider testability, maintainability, and user experience.
In other words, quality is created through decision-making
Testing and test automation play a crucial role in this process, but they do not add quality that was never present in the first place. Instead, they provide insight into the quality that has—or has not—been built into the product.
What successful organizations do differently
When we look at organizations that succeed with test automation, we repeatedly see similar characteristics.
First, they do not treat quality as a separate activity. Instead, quality is viewed as a shared responsibility within the team. Developers, testers, product owners, and other stakeholders collectively own the outcome.
Second, quality is discussed early in the process—not only during acceptance testing, but also during refinement sessions, design discussions, and technical decision-making.
Third, test automation is viewed as a means rather than an end. The goal is not to automate as many tests as possible. The goal is to obtain faster feedback, manage risks more effectively, and deliver software with greater confidence.
This requires a different mindset.
The question shifts from: “How can we automate more?” to: “How can we make quality a structural part of software development?”
This shift aligns with the broader movement from testing toward quality engineering. The focus moves away from finding defects and toward creating the conditions in which quality can emerge from the very beginning.
Culture determines the return on technology
Technology remains important. Modern tooling enables teams to develop faster, receive feedback earlier, and manage risks more effectively.
However, technology is rarely the limiting factor.
The real question is not whether an organization has chosen the right framework. The real question is whether the organization’s approach to quality gives that framework any chance of succeeding.
As long as quality is viewed as something that must be safeguarded by a small group of specialists, the results of test automation will continue to fall short of expectations.
When quality becomes a responsibility shared by the entire team, an environment is created in which test automation can truly demonstrate its value.
Conclusion
Many organizations begin a test automation journey with the expectation that technology will solve their quality problems.
In practice, the opposite is often true: test automation reveals the quality problems that were already there.
This is not a weakness of test automation. It is precisely its strength.
Perhaps the most important lesson our industry has learned in recent years is this:
Quality is not created during testing.
Quality is created through decision-making.
Anyone who wants to achieve sustainable success with test automation must look beyond frameworks, pipelines, and dashboards.
Ultimately, technology determines how you automate. Culture determines whether it works.
Jeroen Mengerink
R&D Manager, Test Architect en Trainer bij Polteq
