Making Good Software

What follows is a summary of my own high level approach for testing software, this strategy is based in facts and principles.

Facts

1. It is impossible to detect all the bugs from an application.

The closer you get to 100% coverage, the harder it is to find the remaining bugs.

2. The more important bugs are in the core layer and in the integration layer (backend).

That’s where the testing needs to be focus on. Core layer bugs and integration layer bugs are the most important because they create a cascade effect causing several parts of the application to fail.

3. Using UI automated tests makes harder the detection of bugs.

Even though they are still very popular, UI automated tests are not very effective finding bugs because they test the core layer and the integration layer indirectly. Testing indirectly the backend makes difficult to exercise it, and makes hard to tell where an error is coming from. It is also important to notice that UI automated tests are also slow and expensive to maintain.

4. Manual testing is still necessary.

There are some important bugs that can only be detected through manual testing, that’s the case of the bugs that can be found doing usability testing and exploratory testing.

5. Testing is worthless if is not executed in a continuous basis.

What’s been proven correct through testing now is going to change very soon so it will have to be proven right again. If this feedback is not fast enough, new changes won’t get proven and eventually new bugs will be entered into the system.

Principles

1. Prioritize what’s going to be tested.

Never have a test strategy that expects to cover 100% of the application.

2. Have as much automated tests as possible.

From your previous prioritization, automate as much as you can.

3. Schedule time for the necessary manual testing to be performed on the project.

Two of the main manual testing activities that are necessary to perform are usability testing and exploratory testing.

4. Use preferably backend automated tests instead of UI automated tests.