10 Signs Testing is Not Enough (Part 2)

What Are the Signs That Strike Us Most That There Is Something Not “Quite Enough"?

Sign 2: The Glitches And The Excessive Amount of Production Bugs

The bugs in production are usually the reason to have the culture of “us and them” between the developers and the testers, because when bugs in production are found the first question usually asked is “was the testing effort sufficient?” That may sound a bit crazy, but still happens in some organizations.

Bugs in production are an obvious sign that you are supposed to invest more in the testing phase.

An even larger issue is that bugs found in the production phase are the bugs that have been in the software from an early stage of the development, have survived and were not found in multiple test cycles. If bugs are not discovered over several test cycles, this clearly implies that the testing you have undertaken to perform is quite insufficient. It is a proven fact the earlier the bugs are found, the cheaper it is to fix them.

Why are bugs so expensive to be fixed? Two main reasons:

If customers find them and the bugs really annoy them, this can lead to loss of clients. And if the bugs are quite serious and hinder the normal work with the product, then you are definitely in a big trouble.
To fix a bug in production takes time. Sometimes a lot of time, because you usually want to recreate it on your internal environment, then fix it, then test the whole software again to see that nothing else is being broken and finally, upgrade the production again. This time is resource – consuming – development, testing and management.

When you have bugs in production that has not been discovered over several test cycles...

You should not only invest more in testing, but also consider changing the chosen and applied so far testing methodology. Because, the worst scenario is that those bugs will be found by your customers resulting in bad PR for you and your company may lead to inevitable effects for your business and it will be definitely more expensive to fix them now when they are “live”.

Sign 2: Bugs in User Acceptance Testing Phase

This is a problem as the User Acceptance Testing Phase is done to build confidence in the product (that the product meets the business requirements). The UAT is also one of the final stages before releasing the product and there should be no serious deviations from the intended design. The goal is, to be sure you have a qualitative product, to find only few bugs – minor bugs. No critical bugs are allowed during that stage and if you have other than just design flaws, then it is for sure that you should invest more in testing.

Why do we have the case of finding bugs, even major ones in the UAT?

Some professionals say it is so as the QAs/ test engineers/ testers influence the UAT too much and think of it as part of the system testing. Well, it is not and the goal is to assure that the actual user finds the software fit for use. That means:

That we have a software that is working and that has been proven by the previous test stages – not the mandate of the UAT
The coverage usually is mainly on the UI – it is what the user usually sees
The corner cases are covered by the System Testing and has been proven correct
No major bugs should be found in UAT and anything that does not work for the user should be a change request, not bug, i.e. the primary requirement was at fault and not the software
Testers should not be involved a big deal in the UAT phase

The UAT proves Fit for Purpose (FfP) and that answers the question “is the software usable from a day -to-day business perspective.

If that FfP is influenced by open bugs that will be catastrophic for the business as fixing the defects will be quite expensive, just like fixing defects found in production. That also means that you are experiencing serious issued with the SDLC you chose to use, the management and the governance of the project. Things found in UAT are supposed to be functional or systematic changes that need to be made in future releases.

Look forward to our next article getting into details about the next three signs: “Bug counts grow”, “Not investing enough” and “No clear criteria”.

Quality House Team