Why Didn’t QA Catch This?

Quality Assurance (QA) can get a lot of flack throughout a release cycle, naturally. Isn’t QA the gatekeeper for an application? If a bug is found after a release by a customer then many look to that team first. If bugs continue to pop up then tests – from manual test cases to automation – can be placed under scrutiny and questions arise about the level of experience with the team.

How often these types of situations pop up and how they are handled can reveal a lot about a team’s processes. If bugs are consistently making their way past testing then it could be time to take a step back and dig into potential root causes.

So why isn’t QA catching all the bugs? Here are a few things that could be happening:

Missing Phases or Environments in Testing Workflows

One of the first things to look at is the testing process itself. Maybe there is a critical step in the workflow that is missing and needs to be incorporated. For example, is QA only testing individual feature branches? A crucial aspect of testing involves seeing how well these individual features will interact with one another once released. This is also where QA can flush out regressions in the system before customers get access to it.

So is an integration environment or testing phase missing in the workflow? If so, then it may be time to talk about spinning up a new environment to deploy the set of features that will become candidates for a release.

Incomplete Acceptance Criteria for New Features

Stories help define development and what should be tested. If the set of acceptance criteria for a feature is incomplete then things will be missed in testing. Anything from permissions levels to expected notification messages can be overlooked if they are not defined or if the team isn’t well acquainted with the application.

Work with the whole team to determine what makes a complete story and include QA in that discussion. Having well-defined acceptance criteria will help them create the best plan of action for thorough testing before a release.

Lack of Business Knowledge

Sometimes, a bug makes it past testing simply because the team isn’t as well-versed in the application they are working in. That knowledge grows with time and experience, but there are ways that can help support everyone, not just QA, in the meantime.

For new features, demos or training can be incredibly helpful for the whole team. This gives everyone from Project Managers to Customer Support the opportunity to learn about what is planned for a release. Knowledge sharing keeps the lines of communication open across each team and helps ensure that silos are not being created.

When a bug is found post-release due to that lack of knowledge, document it in a place that is easily accessible and share that with the team. Turn it into a learning opportunity.

Too Many Shifts in Priorities

It’s to be expected that priorities shift, especially in an Agile environment. Shifts in priorities are not necessarily an indicator of an unhealthy process. If they happen too often then they really are more of a distraction and can throw the entire rhythm of a team off, including testing.

What do these unhealthy shifts in priorities look like? Last minute changes to a feature, new features that need to be incorporated in the next release (that happens to be a day away), and scope creep are a few common ones. QA can’t thoroughly test if priorities are constantly shifting without adequate communication or if they are not given enough time before a release.

It’s important to protect everyone involved with the development of new features and candidates for an upcoming release from these distractions and changes as much as possible.
There is a reason why something was missed in testing. The health and stability of an application doesn’t fall on any one team, so if there are a lot of bugs popping up post-release then look at current processes and work towards finding and fixing any gaps found.

What are some gaps you’ve found that contributed to missing bugs and how did you fix it? Let us know in the comments!