Measuring velocity is not enough to determine team productivity

Introduction

Another interesting question was brought up in a LinkedIn discussion thread: "Is velocity the right approach to measure productivity of team members working in Scrum. If not, then how can productivity of team members be measured in Scrum?" Here are my thought on this topic.

The purpose of velocity

Like burndown charts, velocity is just another metric which the team can use to reach what I believe is the ultimate goal – sustainable throughput. Velocity in my opinion, is not a metric for determining productivity. Productivity or team efficiency is difficult to measure and probably best left for another blog post.

From my perspective, velocity should be used for planning purposes only i.e. how many user stories I can plan into a Release or Sprint based on my teams previous velocity (using story points of course).

Knowing your teams velocity is an important part of Scrum. Knowing your velocity is also central to lean thinking in that it defines the capacity for how much work your team can manage at any given point in time. It's important to note that velocity only remains static if the teams conditions remain the same. For example, if the team composition changes, velocity will change. If changes are made to the process (e.g. start pair programming or start writing unit tests), then you can expect velocity to change as well.

I think that Paul Hodgetts, Agile Coach and Certified Scrum Master, said it best when defining the purpose of measuring velocity:

“Velocity is more of a quantity of work completed metric. Useful and important, but not the sole measure of success. I think you would want a set of metrics that together help us understand our current capabilities to deliver as well as if those capabilities are changing from one point in time to another. I don't believe there is one magic "agility number" to measure [productivity].”

Another important aspect of velocity is that it's a measure of how much work is fully completed at the end of the Sprint i.e. represented by how many completed stories there are at the end of the iteration.

Relieving pressure

Moreover, why I am such a big fan of velocity is because I believe that once a teams velocity has been established, it takes a lot of unnecessary tension out of the development process. In my 20 year software development career, I have rarely had a situation where the business fully trusted the development team. It always seems to boil down to the development team not working hard enough, leading to many hours of overtime and ultimately burnout.

Velocity on the other hand is the neutralizer. Once this number is established, the business gets used to understanding how much work can be shoved into the development pipeline and have a good chance of being delivered on time Sprint after Sprint.

Why it can't be used to measure productivity

I think it's worthwhile explaining why I think velocity is not a great indicator of productivity.

Productivity in my book is how efficient my team is, while velocity can tell you how much you're getting done; it's not going to tell you if you should be getting this done faster or not. One would have to look to industry standards to try to measure productivity like function points.

Consider this...

Your team could have reached it's sustainable throughput, continuously delivering the same amount of software, and still be counter-productive. Why? Because you later find out that what you just delivered didn't result in any value! At the end of the day, it’s the customers and stakeholders that measure the success of the project. Velocity isn’t the end game, the most important thing is that you’re delivering features that the customers want and with quality that’s expected by them.

Conclusion

In summary, what's important is that the team is delivering value in a sustainable fashion (value as determined by the customer). So in order to enhance productivity, there are many things one can do:

• You have to get the requirements down pat. The best way to do this is to work with the customer to define the user stories and get their intent up-front.
• Ensure you are capturing the acceptance test criteria from the customer up-front. This helps solve the greatest risk of all – getting the wrong product.
• Define your definition of "done" up-front so that there's no argument at the end.
• Ensure you build quality in right from the start.

If you do this well, your teams will become highly efficient and the need to measure productivity in the first place disappears!

Written by Jack Milunksy - COO at Brightspark, certified ScrumMaster and Co-founder of Agilebuddy (Agile project management software that lets you easily Create, Estimate, Plan and Track your software development projects). For great Agile tips follow Jack at: www.twitter.com/agilebuddy. To get more info on Agilebuddy please visit: www.agilebuddy.com

Comments

[Tim Gregory]

Hi Jack
In principle I agree with your assertion that the most useful application of Velocity is in sprint planning.
However, Velocity (where I work) is also an important measure of the increasing/decreasing efficiency of the team, and a measure of how effective proposed changes in working habits are over time (back to your examples of pair-programming, unit tests etc).
The only area of productivity measure that I would avoid using Velocity for is to try compare the productivity of 2 teams based on their Velocity. It's clearly going to be quite different between 2 teams.

[Jack milunsky]

Hi Tim,

Thanks for your insight. My only issue with this is based on experience. It's really hard to get a steady state velocity, one that can be be that reliable, that you can compare productivity of the team with. We have some sprints where our velocity is more sometimes less, and it depends on many factors. So the sprints where the velocity is less does not necessarily mean they weren't productive or as productive. It's an indicator but no something you can hang your hat on I think.

Jack

[Asheesh Mehdiratta]

Hi Jack
We have experienced similar as you suggest but have a slight variation to retrospect on the velocity trendline, and making sure that it 'trends' up and not reading more than that into it. For more comments you can see what we are trying on this link http://bit.ly/Vpohj

[Dave Rooney]

Velocity wasn't intended to measure producitivty - it was intended to help the team figure out at what rate they can deliver value.

Once velocity is used to measure the team, they will game it. You'll find that the estimates for user stories will become larger, therefore a higher number of points are completed in each sprint.

In the end, Jack, if the business for whom the team is building software is happy with the results and the rate at which they are being delivered, who gives a crap about the team's productivity? Seriously.

[Oliver Oberdorf]

I'm with Dave, once the team is judged by their velocity they will feel pressure to inflate the numbers. Velocity will just creep up over time while the time to complete estimates also get inflated.

You can try to work around the gaming, such as having people low-bid against eachother to take on areas of work, but I suspect it's more trouble than it's worth.

[Jack Milunsky]

If the team feels pressure to inflate numbers then (and I really hate to say this) they're not doing scrum and they have missed the spirit of what its all about.

The burndown is there for the team to measure how well they are doing relative to the plan. It's not a tool for management (at least it wasn't desinged that way) It's for the team to reflect and adjust accordingly

Thanks for you comments

Jack

[Mark Levison]

Jack - I see some other problems with this approach.

If you tell the team you're measuring their productivity this way then they have an incentive to jam more stories out the door faster with less regard to quality.

Remember any measure that you choose will cause the system to behave accordingly you have to consider whether you want to behave that way. In addition eventually most measures will be gamed whether consciously or not.

If you must measure something I would focus on business value - i.e. value accrued to a customer. Anything else just measures a subset of the system and will likely produce only local optimizations.

Finally, why is you want to measure productivity in the first place? What will you do with this information?

Cheers
Mark

[Dan Pickett]

Velocity is not a performance metric. While it's helpful in projecting capacity and project planning, story estimates are just what they are: estimates.

If you're going to evaluate developers on velocity, you'll have to measure the impact/quality of their estimates.

Also, since the developers are the people making the estimates, they can easily inflate their metrics. If they aren't making the estimates, how can you make them accountable for something they didn't estimate?

Technical managers need to set goals for developers, not story point quotas. IE - increase test coverage by 5%, reduce story rejection by 10%, develop project management skills.

Velocity is great to see how your engineering department is operating as a whole, but I don't think it's right to hold individuals accountable for regular, accurate, and rhythmic velocity. Usually there's too many unknowns and risks with a story to do that.

[MArtin Sudmann]

Hi all,

I guess that one of the most important things about velocity is: It cannot be used for outside communication, but it is for team use only.
It's only by using your teams velocity in external communication that it becomes a measure of your teams "worth". And its completely useless as such - velocity for team A is not the same scale as velocity of team B, and btw.: what do we gain in comparing teams anyway.

[Kevin Thompson, CSP]

Hmm. I think there is some confusion of terminology here, with respect to "velocity" and "productivity."

My American Heritage dictionary defines productivity as, "The rate at which goods or services are produced especially output per unit of labor."

Jeff Sutherland defines velocity as, "the number of story points they can turn into working code at the end of a sprint."

I would claim that velocity and productivity are the same thing: Both describe the rate at which (usable) work is completed.

Nowhere in either definition does the concept of the value of the work enter in. Nor should it: The team cannot estimate the value of the work. That is the responsibility of the business, and specifically, the Product Owner.

The issue of inflating story points to make the team look better is an artifact of how story points are defined. If they represent work hours (e.g., 1 story point = 8 work hours), then inflation isn't possible, because the definition is tied to the clock. If they are purely relative measures of difficulty, then yes, they can be inflated, but they are also meaningless outside the narrow context of the team's estimation process, so why bother?

[Jack Milunsky]

Nice comments Kevin.

I don't thing tying story points to hours or not will make a difference to inflating numbers. I think mature teams will realize that there is no point in tricking themselves. My teams try real hard to get to sustainable throughput. Obviously it's never perfect so the velocity does deviate from sprint to sprint even on my teams that have tons of experience and many sprints under their belt.

But there are so many variables. So you have to be mindful of that. Teams change, complexity changes. But if you try and use historical data and have a cross functional team do the relative estimating. You can get pretty good at this. You also need a manager that understands the process and will work with the team and not use the burndown as a police cop.

With Agilebuddy in the beginning we were finding that we 1. underestimated and we scheduled to much so we just started scheduling less until we got it bang on. But I had the patience and understanding to work with the team to reach this point. Once you get there, it's like clockwork, sprint after sprint after sprint.

Appreaciate all your comments.

[Jack Milunsky]

@ Martin

Again, I think it one needs to be careful about how you use velocity but I am not sure that it can't be used to communicate externally

My point is that if you know that your team on average can churn through 30 points of stories. Then that's a good number to share with your PO (PO is internal) and Client

It really helps them to prioritize accordingly.

It's really a cool feeling when you get to a point where you tell the client you have 30 points of work for a sprint and you nail it (or come close or heaven forbid exceed this) sprint after sprint after sprint. The problem starts when you tell the client you can do 30 but you deliver 5 .. this just sets the team up for failure. That's why I like to spend quite a bit of time especially in the early stages getting this right. So I typically err slightly on spending more time estimating, and talking about stories and then breaking them down into tasks especially in the beginning of the project.

[Martin Sudmann]

Jack,

why do you want to tell the client that you burnt 30 SP? That doesn't mean anything to him, except if he is knee-deep into agile methodology. Why not just telling him that your team had committed to finish the stories X, Y and Z and that you have succeeded at 100%, just like the last few sprints (if the team is that constant)?

I don't see how the knowledge of your teams burn down can help your client to prioritize his stories, normally he fixes his priorities before the stories are estimated, no?
As far as the PO is concerned, he can work with story points and velocity, but like you said he is part of the team.

[Jack Milunsky]

well the po/client decides what get's done and when. i once read some place that a story without an estimate is like a product in the store without a price. PO's/clients need to know this in order to effectively prioritize. So you tell them how big things are and you tell them how much you can fit in per sprint, the rest is pretty straight forward ... at tleast in my opinion.

[Anil Apte]

I feel, like any other metric, Velocity can only be compared with past value of Velocity. The conditions of measurement have to be the same to make them comparable.

If a team X reaches velocity V1 per full-time-equivalent member(FTE) in sprint 3 and reaches a velocity of V1+20%/FTE in sprint 6 - then they have improved their productivity.

I also feel that it can be applied to a single member also (measure "his" historical velocity with "his" current velocity).