The best bug tracking system. Don’t raise bugs, write an automated test!
Bug tracking systems are bottlenecks, they perpetuate the continuous handovers between testers and developers and they require extensive documentation to be maintained.
The standard life cycle of a bug is usually as follows: A QA tester finds an error in the application, creates a new bug in their bug tracking system and sends it to the developer, which in turn, as soon as he fixes the bug, notifies back to the QA tester, which then, notifies the product owner which then, closes the bug.
In paper, the previous process sounds wonderful, but there are quite a few drawbacks.
- Communication problems. Sometimes the description of the bug is poor, or the developer misunderstands the problem.
- Regression testing. This process requires the handover of code back and forward between QA and development, this causes versioning problems and duplicated testing effort.
- Low robustness. The process doesn’t guarantee that the error will appear again in the future.
- Bureaucracy.Traditional bug tracking systems switches the team focus to bureaucracy from quality.
The solution for this problems is simple. When anyone finds a bug, he/she immediately writes an automated test that fails based on the bug just discovered. This mechanism have the following advantages.
- Gets rid of versioning problems, everyone works on the head.
- Guarantees that the bug is never gonna reappear undetected.
- Eliminates uncertainty on what the bug exactly is.
- Eliminates bureaucracy
- Consolidates the development activities in one place, the code.
The challenges to switch to this process, where a bug tracking system is not necessary are:
- Requires an automated build. This build have to be in place and running all the automated tests, so when a new automated test is entered because of a bug, the build will break and the team will have to take a decision whether they will fix the fix now, or whether they will ignore it for the time being (That way you will know that all the ignored tests are non-fixed bugs)
- Requires cooperation. Developers and testers must work together so they can create new tests as soon as they find new bugs. You are going to require testers with programming skills.
- Requires discipline. The team must understand that without exception, for every bug found, a test is created, and for every time the build is broken, the priority is to fix the build.
- Requires organization. Tests must be created with criteria and using a common naming convention.
Summarizing, in my opinion, the future of bug management is to get rid of all the traditional bug tracking systems and all the bureaucracy, I also think, as explained just above, that there are quite a few challenges, but if you have the skills and tools in your team to overcome them, I would seriously think about getting rid already of your old bug tracking system.
Related posts:
- How to write a good test case: 5 tips to write better test cases
- Book review: xUnit Test Patterns
- How to write good tests. Top 5 considerations to build software without defects.
- Stop waiting to the end of the project to start QA!!! (And other QA principles)
- Continuous integration, going beyond continuous compilation.
![[Google]]( http://www.makinggoodsoftware.com/wp-content/plugins/easy-adsenser/google-light.gif)
Writing test is not a substitute for a bug-tracking system. It’s just a good practice to cut down the number of bugs. You will still have bugs due to human mistakes.
Paco
23 Feb 10 at 6:54 am
How do you know the code you create for testing the bug, does not have errors itself?
CJ
23 Feb 10 at 7:16 am
All bugs are not created equal. For critical bugs that should be fixed ASAP this advice could work.
For bugs of a lesser severity this will lead to a build that is broken 24/7. Since every spelling error or misaligned button will break the build.
Also, prioritizing the bugs would be a major pain.
surolja
23 Feb 10 at 7:19 am
I think you’re forgetting a few things about bug tracking systems. They provide historical information (who discovered it, how they did, when they did, etc etc). They also provide a way to collect metrics (bug discovery/generation rate, impact/severity, time spent on bugs, etc etc).
On another note, with your idea you’ve now injected a failing test into your system. I can see this working if the very next step is to go fix the bug, but how often does that happen? Many times the bug you find isn’t important enough to justify fixing right that moment. The longer you go without fixing the bug, the longer that failing test stays there. As you point out, you need an automated build process for this, but now you have an eternally failing build. It doesn’t take long for people to simply ignore the build status that always fails, which means it takes longer to notice actual problems.
What we do is, when a bug is discovered, write an automated test with an Ignore tag. File a bug in the bug tracking system and note the Ignored test. I think this process captures all of the advantages you listed, without the drawbacks (except Bureaucracy, but that feels like a problem with the team’s maturity, not a problem inherent in bug tracking systems).
Robert Herbig
23 Feb 10 at 7:54 am
Well…. This may sound radical, but in my opinion: If you are not sure if the test code you just have written have a bug, then probably you haven’t wrote the right test case, or your source code is not very testable.
There are also cases where you cannot simply write an straight forward test case, for those scenarios write a few more asserts that checks for positive conditions, boundary conditions and negative conditions, if all them pass, then you are likely to have a free of bugs test case.
Alberto Gutierrez
23 Feb 10 at 8:10 am
Hi surolja!
I think you are completely right, your point is actually something I was trying to remark here:
[...] the team will have to take a decision whether they will fix the fix now, or whether they will ignore it for the time being (That way you will know that all the ignored tests are non-fixed bugs) [...]
By ignoring tests, they get flagged in every build, so you know how many bugs are pending and the build doesn’t break
Alberto Gutierrez
23 Feb 10 at 8:14 am
Hi Robert!
As with surolja, I completely agree with you… I didn’t mean to say that you should keep the build broken..
[...] the team will have to take a decision whether they will fix the fix now, or whether they will ignore it for the time being (That way you will know that all the ignored tests are non-fixed bugs) [...]
My only point is that if you guys are currently doing this, does it add any value your bug tracking system?
Alberto Gutierrez
23 Feb 10 at 8:15 am
Sorry about that, I must not have seen your remark.
I also need to clarify my earlier comment: we write a unit test to fail whenever possible, but that doesn’t mean it always happens. Sometimes the test team is not familiar enough with the code/mock-interface to write a unit test. Some bugs do not lend themselves to having a unit test (eg: UI bugs, timing issues).
However, when a developer begins work on the bug, the first step is to write a unit test (if possible) if one does not already exist.
I guess what I’m trying to get to is that there is no reason to go with one approach and exclude the other. Using a bug tracking system has its own set of advantages (see my previous post), as does a unit-test-failure way of illustrating bugs.
Robert Herbig
23 Feb 10 at 8:29 am
Long ago, in the 80′s (yeah I’m old
), I worked on a piece of software whose bug-tracking system was … a whiteboard in the project lead’s office. The whiteboard had the currently open bugs on it, and who they were assigned to. It was backed up by a filing cabinet with hardcopy information of the closed bugs, for history.
In spite of (or because of?!) our primitive bug-tracking system, this was one of the highest-quality pieces of software I’ve ever worked on. We never had more open bugs than we could fit on a whiteboard. Frequently, the whiteboard was empty!
Since then I’ve worked on software projects that have database-backed bug tracking systems. Often with 100′s or even 1000′s of open bugs. IMHO, as soon as you need a database to track your open bugs, that in and of itself is indicative of a quality problem.
On the question of automated regression tests, yes I believe they are a good idea. We had one way back on that high-quality software I worked on back in the 80′s, and it was instrumental in ensuring good quality. I’m not sure a test system is good substitute for some kind of bug-tracking system. I mean, I see how you could use it that way: run the tests and look for failures. But a good bug-tracking system (e.g. a whiteboard
) lets you add notes, assign bugs to people, keep history, do queries, etc, all things a test system isn’t really designed for. But a test system is a good idea regardless.
Just my 2 cents.
rtitle
23 Feb 10 at 8:54 am
I’d like to see the unit test that checks things look ok Internet Explorer, or that the system generally does what the client pays for…
It’s all well and good trying to change the world so that a ‘perfect’ system can be implemented – but it’s far more practical to implement a system that functions in a non-perfect world.
Tom Carnell
23 Feb 10 at 8:58 am
[...] full post on Hacker News If you enjoyed this article, please consider sharing it! Tagged with: automated [...]
The best bug tracking system. Don’t raise bugs, write an automated test
23 Feb 10 at 9:16 am
@rtitle
Wow! that board thing sounds awesome, I really thing that is more helpful than any bug tracking system, I actually believe is the ideal complement for the bug tracking process I am suggesting… Thanks for that great input rtitle!!!
Alberto Gutierrez
23 Feb 10 at 9:18 am
Hi Tom!
Yeah, that’s a flaw I realized the process had while I was writing the article, but let me ask this, for this kind of errors which you can’t provide unit tests (which in my opinion is only 5% of them), what is the value of having them in a bug tracking system, also, just because this 5% of bugs can’t be easily represented with automated tests… are you gonna have the other 95% of the bugs handled in a bug tracking system?
Using rtitle idea, what about using the build and automated tests for the tests you can automate, and a board for the other bugs?
Alberto Gutierrez
23 Feb 10 at 9:22 am
for someone a bug tracking would be a advance … others dream in tdd without having a bug tracking system or even a QA dep
jneira
23 Feb 10 at 12:12 pm
Does it have to be either/or? Why is anyone recommending jettisoning all our understanding of bugtracker best practices, when all we need to do is add a field/upload/attachment to the bugtracker for the auto-test? That would reduce the need for programmer-testers and make it possible for beta testers and clients to file their own bugs (as now), the early response to which would be a developer writing an appropriate test case for it….
blahedo
23 Feb 10 at 1:01 pm
[...] The best bug tracking system. Don’t raise bugs, write an automated test! | Making Good Software (tags: management productivity testing automation) [...]
links for 2010-02-23 « Blarney Fellow
23 Feb 10 at 6:33 pm
[...] Social Media Internet Explorer gains modicum of HTML5 More Details About Spotify’s New Money The best bug tracking system. Don’t raise bugs, write an automated test YouTube Puts Another Nail in the IE6 Coffin Are Aggregation Services Security Risks? Twitter Sees [...]
Google faces EU investigation – Web Review 24/02/10 « Vexed Digital Blog
24 Feb 10 at 3:50 am
Hi,
Very intresting post but in my opinion there are not huge diffrence between writing possitive test case to negative one. Good practice is only the thing that you should always cover error with test case.
Test Case Tutorial : http://www.testandtry.com/2010/02/24/8-tips-to-create-complete-test-cases/
best regards
Marcin
Marcin Zręda
25 Feb 10 at 11:04 am
Great idea but effective only in team of developers. Imagine environment of more than 1000 people working on a product, new version released twice a year.
Lots of errors are cannot be easily located by testers, many of people involved in error reporting do not even have access to code or have any idea where to locate it. That’s only inside company – what about customers?
And versions – every customer may have other version of a component – it’s rarely head, more probable is that version at customer is over year old…
I believe that fault reporting system is bottle neck and it could be somehow optimized but the idea of tests instead – is just too optimistic.
Lukasz K
25 Feb 10 at 3:50 pm
[...] The best bug tracking system. Don’t raise bugs, write an automated test! (tags: programming qa bugs automatedtesting software code tips) [...]
links for 2010-02-26 | sashidhar.com
26 Feb 10 at 2:02 am
That’s an interesting post. I don’t think that automated tests would be a replacement for the whole bureaucratic procedure though – rather an useful addition making the bug reproduction easier. Don’t forget that it requires the QA people to posses extra skills, too.
Michal Huniewicz
26 Feb 10 at 3:07 am
Yes, when a bug is discovered one should immidiately write a test case to fail it. But IMHO Bug tracking helps in maintaining history, output, (When started when ended.) And all other info at one place. (svn commits for bug, codereviews etc..) Its history which can be referred again.
In Ideal and small (developer) world, Yes, Only good test should do the job. It save time.
Jigar shah
26 Feb 10 at 6:31 am
Why can’t we do both? We use the already well understood bug tracking system to record the bug. And also create a test case to actually reproduce the bug.
Infact, isn’t that what is supposed to be done? Only the test case is written by the developer and not the tester. Ideally, if you are following TDD then you know you missed a test case and just add it to your tests.
Kevin
26 Feb 10 at 8:38 pm
[...] read another interesting article about bug tracking and system testing. Think about this… rather than creating a problem ticket for issues found in system test, [...]
Who should write unit tests? | Soapbox Rants and Raves
31 Mar 10 at 10:14 pm
You could make a screencast demonstrating the bug.
Dathan
28 May 10 at 1:36 pm
That’s an interesting post. I don’t think that automated tests would be a replacement for the whole bureaucratic procedure though – rather an useful addition making the bug reproduction easier. Don’t forget that it requires the QA people to posses extra skills, too.
Emily
31 May 10 at 10:18 pm
[...] taught you to write unit test cases for all your code? She probably insisted that you should write a new test case for every reported bug? You probably wondered why, but still followed her advice like any dutiful [...]
Auto-fix your bugs with UT and SBSE « Agile Project Manager
3 Jun 10 at 2:28 pm
[...] The best bug tracking system. Don’t raise bugs, write an automated test! [...]
Are you using too many technologies in your projects? The 7 anti-patterns for technologies, frameworks and other technicalities in software development. | Making Good Software | the leading lady
17 Jun 10 at 5:39 am
How to raise a new bug http://bit.ly/cTGxdY
Eusebiu Blindu
19 Aug 10 at 7:21 am