Accounting for Bugs in an Agile World - How we do it

Today I'd like to share with you a really important aspect of Agile software development - how to account for bugs in your daily plans. I am frequently asked by teams I consult with how to deal with bugs if you're Agile and using Scrum. Questions like the following come up time and time again: "Should I track hours burned fixing bugs?" "Should the hours burned on fixing bugs count toward my Story Point velocity?" "When do you schedule bugs and should this be done during Sprint planning?"

You Need to Account For Bugs

My company is often working on many simultaneous projects and having to juggle resources. To get everything done on time and bug free requires careful planning. As a result we have to ensure that we're considering all aspects of software development including bug fixing. Bottom line is Bugs take time to fix and so you have to account for them. If you don't, you will have an inaccurate measure of your teams productivity - something that will definitely lead to poor planning and late deliveries. (....read more) Time to fix bugs therefore, should come out of your teams available capacity. So when one schedules bugs to be fixed, bugs should be estimated just like you do with tasks on the sprint backlog. Developers must therefore determine for each bug scheduled, the hours remaining to complete the bug daily. And this WILL be reflected in the daily burn-down along with tasks.

Calculating the Teams Capacity

Essentially teams have a certain capacity i.e. available time during the week to do meaningful work. This can be calculated as follows: (# of team members) * (# days in the week) * (hours in the day) = capacity You should however modify the capacity to reflect reality (time spent in meetings, doing email, vacation etc) so this can get quite complicated. The best thing to do is to figure out what percentage of time per day you get real work out of the team on AVERAGE. At our company we work on a 4 or 5 hour day therefore: Actual Capacity = Capacity * 0.5 Once again, don't get hung up on being extremely accurate as this all averages out in the end. Just be reasonable in your approach. If you want to take vacation into account etc one can get quite fancy about this, but usually a rough estimate works fine. It does for my team.

Resolving Bugs Shouldn't Count Toward Your Story Point Velocity

The time spent resolving bugs however should NOT be counted towards the Teams overall Story point velocity. i.e. I'm not going to give my team credit for buggy code! Bugs should be scheduled into a sprint or iteration along with user stories during the Sprint planning meeting. So the total time to implement tasks and fix bugs should not be more than the available capacity you determined for the Team (See above capacity calculation). Note that this total time in hours is the intersection with the Y-axis on your burndown. I know many coaches will argue that bugs should be tracked seperately and that bugs should be dealt with in the sprint that they're found. However in practice this is not always practical. Try it, and you'll improve your teams predictability significantly. Next week I will give a detailed account of how we do Planning. If you have any questions about Planning or Bug Tracking in general or if you have strong opinions on this topic, please don't hesitate to pitch in!


Written by Jack Milunksy - COO at Brightspark, certified ScrumMaster and Co-founder of Agilebuddy (Agile project management software that lets you easily Create, Estimate, Plan and Track your software development projects). For great Agile tips follow Jack at: www.twitter.com/agilebuddy. To get more info on Agilebuddy please visit: www.agilebuddy.com

Comments

[j pimmel]

Another thing to bear in mind is ensuring - where possible - not to open bugs against stories currently in progress.

At EnergizedWork the preference is to iterate over story acceptance criteria and deliver slices of completed functionality to QA. Having such short cycles enables QA to find bugs that developers didn't know existed prior to story completion.

Resisting the temptation as developers to raise it as a bug and instead recognising that certain issues are directly coupled with attaining a certain quality of delivered product is a useful and efficient way to reduce the number of bugs which need flagging to the broader team.

There are, obviously, always exceptions, but placing an emphasis on doing the right thing today, rather than deferring to the next, goes a long way toward lubricating a high performing agile process.

[Anna Forss]

I've changes my mind concerning this a couple of times, but what I've landed in right now is to include bugs in velocity BUT I link it to the previous story. Som when I count the size of the original story, I add all the bugs too.

But the main reason for including bugs is that I really don't care if something is a bug or a change request. We have software which we're not satisfies with. If developers are punished for bugs, they can work hard to get stuff classified as something else. So, then you have to go over the history and how productive is that?

If we have to change software because the developers missed a comma, is that really so much worse than the business people missing that comma in the spec? The comma is still missing

[Jack Milunsky]

I think your approach is valid and practical. I think however you do it, as long as the process works for you then that is good.

Thanks for your contribution

Jack

[Brendan]

Nice read... I disagree with one point. You state, "Bottom line is Bugs take time to fix and so you have to account for them." So why not account for this in your team's velocity? Not having any defects is a tangible goal that every team should strive for. However, defect debt is a reality that when planning, I need to account for how many stories I can take on in a given sprint in a addition to how many defects can be worked on. While the team should not be patted on the back for fixing defects, context plays a big part here.

[WSimacek]

I agree with Brendan's comment, basically that you take bug time out of your team's velocity. As your team get's more bug-free, you have a nice graphic showing your velocity is improving.

[Amit Goyal]

I am completely agree with the fact that "bugs should be estimated just like you do with tasks on the sprint backlog". My only concern is how do you estimate bugs? Believe me if you are working on a large legacy project its really very much unpredictable that how long will it take to identify the cause of the particular bug/issue. For an example, suddenly your application server started loosing data from session intermittently. You don't know how long will it take to identify this particular issue.

I would really appreciate if you can put some light to estimate such kind of issue/bug.

[Jack Milunsky]

Hi Amit,

Thanks for your post.

Estimating bugs can be very difficult. There's no magic bullet. But what you do is as you learn more from debugging you update your estimate to reflect reality. That's the best you can do

Jack

[Jason Dean (ScrumMaster)]

It is very clear to me that you should account for defect work done in the sprint. There is more than one kind of defect however and they don't all need to be treated the same.

Defects found during the sprint that are a result of coding done during the (sprint defects) should be part of your normal code/fix work. We don't track hours or points for these because they are part of the original story point/task hour plan.

Defects found that are found that are not a result of the sprint or that are found during the sprint and the PO chooses not to fix them, are put into the backlog for future sprints. These need to be prioritized by the PO and estimated just like any other backlog item.

If an defect is planned to be fixed during the sprint, we consider it part of the sprint backlog and it gets counted in the velocity. We typically add a "bug" story that gets x number of story point assigned to it, then we add defect tasks that approximate the story point estimation. This allows us to keep working on defects that are found at other times or by other stakeholders/customers.

I don't think that punishing the team by not including their defect work in the sprint is productive. We have a lot of legacy code that the teams are the originators on and to make them accountable for previous years mistakes (before Agile) doesn't make sense. I do however believe it's important to keep the team accountable to their commitment of story points and to completing the sprint defect free.

[Jason Dean (ScrumMaster)]

Sorry, in my post, in the last paragraph, I said that "we have a lot of legacy code that the teams are th originators on" - I meant to say that they are "NOT the originators on".

So the basic point is that the team working on an issue didn't necessarily create it. Even if they did, not getting credit for past work doesn't seem right. I understand the idea of not rewarding a team for producing lower quality (defects), but I think there are other ways to do this. I'm not saying that it can't work, but I haven't seen it work well. I've seen it operating, but it's questionable to me that it makes a difference. I'm open to discussion about it though. :)

[Jack Milunsky]

Interesting points you raise Jason.

So there's a couple things I'd like to respond on.

1. Assuming that the team wern't the original coders, I think in that particular case it's fine for the work to count towards their velocity. In fact even if they were it's not a big deal. As long as you're consistent and it's working for you then that is good.

2. In regards, "Defects found during the sprint that are a result of coding done during the (sprint defects) should be part of your normal code/fix work." Here's my issues with not tracking the hours associated with these bugs. I really like the burndown to reflect reality. i.e. the burndown should reflect the total hours remaining to get all the user stories completed i.e. "done" So if you find a bug and you don't track the hours it's going to take for the team to fix, then you're not going to know whether you can finish or not. The developers during the sprint can add tasks to the sprint as required even after the sprint planning meeting. When they add tasks they have to add the hours required to complete that task. Similarly if the task hours go up instead of down, that should reflect reality. So then why not also track the time required to fix bugs found during the sprint. Remember the team has x Capacity. They can't magically make up the time. I much prefer my burndows to reflect exactly where the teams are at.

3. If you are however not using hours for the burndown but just using story points for the burndown (I prefer story points for planning, and hours for the burndown) then I think it would be ok to treat bugs as you suggest.

Hope this makes sense to you.

Thanks a ton for your comments.
Jack

[Rolf Goetz]

Jack,
thank you for your thought provoking post. We're on the same page.

I'd like to add another flavor to the idea of accounting for defect work.
What exactly is your team's defect removal efficiency (that is: number of defects found after release divided by number defects found before release)? World-class is >95%, U.S. average 2007 about 85% (<- Capers Jones)

Testing, if it is the only defect removal activity (or one of very few), is not very effective in finding defects. Effective defect removal activities include all kinds of inspections, for instance.

Your description of the problem sounds like your team could do better QA upstream. This option, being wise from an economic POV, would bring the number of defects in the code down, effectively decreasing your needed testing efforts. I don't know the numbers you produce, but that might bring about an important change:

Proper (upstream) QA certainly counts as value work, hence should be included in velocity calculations. AND you can pat your team on the back for eliminating defects within the timebox or the release, respectively. AND you may have to release more often, due to increased overall quality.

It is important to note that you will NOT slow down, at least not after a few iterations in which you need to adapt. Finding and fixing defects upstream is cheaper and faster than downstream.

Thanks again,

Rolf

[Jack milunsky]

Thanks Rolf,

Appreciate your kind words and for your insights. Actually, I can't say that we track defects before and after. We probably should try that.

Based on my experience though, my gut is telling me that we don't have a very high bug rate for the amount of code we generate. Our unit tests are >95% coverage. But we fix most bugs we find as soon as we find them. As a result we don't have a final QA closure phase at all in fact.

Cheers
Jack

[Generic Viagra]

The original structure of arthropod appendages was probably biramous, with the upper branch acting as a gill while the lower branch was used for walking. In some segments of all known arthropods the appendages have been modified,

[NFL Jerseys]

It is very clear to me that you should account for defect work done in the sprint. There is more than one kind of defect however and they don't all need to be treated the same.

Defects found during the sprint that are a result of coding done during the (sprint defects) should be part of your normal code/fix work. We don't track hours or points for these because they are part of the original story point/task hour plan.

Defects found that are found that are not a result of the sprint or that are found during the sprint and the PO chooses not to fix them, are put into the backlog for future sprints. These need to be prioritized by the PO and estimated just like any other backlog item.

If an defect is planned to be fixed during the sprint, we consider it part of the sprint backlog and it gets counted in the velocity. We typically add a "bug" story that gets x number of story point assigned to it, then we add defect tasks that approximate the story point estimation. This allows us to keep working on defects that are found at other times or by other stakeholders/customers.

I don't think that punishing the team by not including their defect work in the sprint is productive. We have a lot of legacy code that the teams are the originators on and to make them accountable for previous years mistakes (before Agile) doesn't make sense. I do however believe it's important to keep the team accountable to their commitment of story points and to completing the sprint defect free.

[Wholesale NFL Jerseys]

Unless or Until these issues are not fixed and cleared Electric Cars will not ba as popular as other Cars.

[accountant]

Hi Amit, thanks for your impressive blog. It's really difficult to estimate bugs. Im still on the process on learning how to debug. Thanks!

[lacoste mens shoes]

Defects found that are found that are not a result of the sprint or that are found during the sprint and the PO chooses not to fix them, are put into the backlog for future sprints. These need to be prioritized by the PO and estimated just like any other backlog item.