Clojure
Rich Hickey discusses Clojure features and syntax, example code, functional programming, concurrency semantics, transactions, software transactional memory, agents, implementation and pain points.
- Java,
News
Opinions: Measuring Programmers' Productivity
Posted by Sadek Drobi on Oct 05, 2008 01:31 PM
In the field of software development, just as in any other area, managers need to appreciate performance of their programmers and the progress of their projects. However, defining appropriate metrics to this end appears to be a tricky task.
Measuring source lines of code (SLOC) is one of commonly used approaches that presents however a number of important limitations highlighted recently by Shahar Yair and Steve McConnell. First of all, the amount of lines of code does not allow effectively measuring the progress of a project because it focuses on activity rather than results. LOC do not have any value as such: the value of the final product depends on its performance and quality, not on the amount of code. Hence, focusing on the latter is actually a very limited approach to productivity.
SLOC doesn’t tell anything about the complexity of the issue at hand or about the quality of the final product in terms of maintainability, flexibility, extensibility, etc. With regard to quality, it can actually be negatively correlated. Refactoring as well as some design patterns result in reducing LOC while improving code’s quality. Larger code base may mean more noise, more unnecessary complexity, less readability and flexibility.
What is particularly risky about adopting such a reductive view of programmers’ performance is that it creates wrong incentives. Rather than optimizing their work in terms of final product, developers may be encouraged to favor the quantity of code to the detriment of quality and even intentionally write more verbose code. “What gets measured gets done”, recalls Steve McConnell.
He points out that some of these issues can be solved by using function points as measuring metric. The program size is then determined by the number of inputs, outputs, queries and files. Nevertheless, this approach also has its downsides. McConnell mentions some practical issues like the necessity of having a certified function point counter and the difficulties with mapping each function point back to individual programmers. A certified function point specialist Daniel Yokomizo highlights in his comment other limitations of such approach: lack of tools to measure function points’ complexity and to take into consideration things such as code sharing, frameworks, libraries, etc., that affect the time needed to create a feature.
Even though many commentators involved in the discussion about measuring approaches agree on their limitations, they do not necessarily dismiss the need for measuring programmers’ performance. Many insist on the fact that SLOC, for instance, can be used as a baseline for a more complex analysis combining different factors. This goes along the lines of four principles outlined by McConnell that should guide any analysis of programmers’ productivity:
1. Don’t expect any single dimensional measure of productivity to give you a very good picture of individual productivity […]
2. Don’t expect any measures […] to support fine-grain discrimination in productivity among individuals. [They] give you questions to ask but they don’t give you the answers […]
3. Remember that trends are usually more important than single-point measures […]
4. Ask why you need to measure individual productivity at all […]
In which context measuring programmers’ productivity is actually meaningful? What criteria can be used for it? How can they be combined? Many questions are still open for discussion and if your experience has brought you some answers, don’t hesitate to share.
RelatedVendorContent
Smart agile development within dispersed teams
Lean Software Development Governance, a whitepaper by Per Kroll and Scott Ambler
IBM software architect eKit: Grady Booch podcast, whitepapers, articles
Gnip Case Study: Reliable and Scalable Access to Massive Data Streams from Multiple Sources
The Agile Business Analyst: Skills and Techniques needed for Agile
A minor nitpick
by
Daniel Yokomizo
Posted
Oct 5, 2008 2:12 PM
Why do I care?
by
Bruce Rennie
Posted
Oct 5, 2008 7:28 PM
Re: Why do I care?
by
Sadek Drobi
Posted
Oct 6, 2008 4:50 AM
Re: Why do I care?
by
Thomas Cagley
Posted
Oct 6, 2008 5:25 AM
Re: Why do I care?
by
Bruce Rennie
Posted
Oct 6, 2008 7:23 AM
Re: Why do I care?
by
John Reynolds
Posted
Oct 6, 2008 8:20 AM
Re: Why do I care?
by
John Jimenez
Posted
Oct 6, 2008 4:02 PM
LOC
by
Stefan Wagner
Posted
Oct 5, 2008 10:36 PM
Danger - will destroy teams
by
Mark Levison
Posted
Oct 6, 2008 9:00 AM
Re: Danger - will destroy teams
by
Zachary Shaw
Posted
Oct 6, 2008 10:36 AM
Re: Danger - will destroy teams
by
Mark Levison
Posted
Oct 6, 2008 10:52 AM
Re: Danger - will destroy teams
by
John Jimenez
Posted
Oct 6, 2008 4:24 PM
Re: Danger - will destroy teams
by
Mark Levison
Posted
Oct 6, 2008 7:40 PM
Hudson Continuous Integration Game
by
Zachary Shaw
Posted
Oct 6, 2008 9:26 AM
Know why you're measuring, and keep it squeaky clean
by
Robert Merrill
Posted
Oct 7, 2008 4:32 PM
Quantitative performance measure will be the disaster to team. Be careful!
by
James Wu
Posted
Oct 17, 2008 6:32 AM
-
Nice article, but I have a minor nitpick. I was a certified function point specialist by IFPUG, I'm not anymore since 2005 (IIRC, I don't remember the exact date), as I said in my comment on Steve McConnell's blog. OTOH I was still working regularly as a specialist after that. IFPUG has a time limited certification, when came the time to certify again I didn't felt the money was worth it (as I didn't want to work primarily with function points anymore).
-
Why do I care about programmer productivity? I'm not selling lines of code, am I? What, exactly, is one programmer producing?
<P></P>
Where does the desire to measure developer productivity come from? Do they measure the productivity of doctors? Engineers? Plumbers? And those don't tend to work in teams. -
If I would have the choice to choose from 2 projects, which do the same thing, one with x lines of code, and one with 2*x lines of code, I would - of course - choose the smaller one (assuming no other information). Code-reuse means less code! More code encourages copy and paste.
If you bundle some subprime measurements together, do you get a triple-A measurement, or do you get junk?
Maybe I would start writing code-generators, if LOCs would be measured, but most probably I would leave. -
@Bruce: It can be quite interesting to trace where this productivity measuring culture comes from...
-
Off course they measure productivity of doctors, engineers, plumbers and even football players (sacks per game is a productivity measure). The goal is improve efficiency. When money is loose it is less important however when the economy slows or contracts it becomes critical to try to squeeze every cents worth of value out of projects if for no other reason than to protect the jobs in organization.
Tom Cagley
www.spamcast.net -
Heh, good comment about football players.
<p></p>
Maybe doctors do have productivity measurements (patients seen per hour), but they also have a strong professional code that trumps any productivity measurement. Doctors talk about lives saved, not surgeries per day. Ditto for engineers: only the bridges that are still standing count.
<p></p>
As I also pointed out, things get even more fuzzy when a team is involved. In a team environment, measuring and optimizing at the individual level feels like I'm ignoring Theory of Constraints. I should be measuring and optimizing for overall throughput.
<p></p>
As a final question: does measuring productivity on a failed project make any sense? -
To Bruce's last point "As a final question: does measuring productivity on a failed project make any sense?", more meaningful to a development team would be:
- actual development time compared to estimated time
- re-work / bug fixing time vs. actual development time (this can be useful if you think about it)
- customer satisfaction / feedback survey
John -
When you measure the performance of individuals you destroy teams. Once I know I'm being measured for producing code, here are somethings I will never do again:
1. Spend time in planning/retrospective meetings
2. Mentor jr team members
3. Pair Programming
4. Code Reviews
5. Simplify code, remove duplication, anything that might reduce my function point count
6. Anything else that would take time away from creating function points
Further more I will learn what function points are and learn to game them by writing code that maximizes the number of function points.
Find a measure of the individual and I will show you how it destroy's teams. Net result I recommend against measuring individuals or even teams. Instead I focus on outcomes. -
We've started "playing" the Hudson Continuous Integration Game I think it's a great tool to start talking about productivity / performance. I don't know if I'd love to be judged on it by management, but I do think it's a good tool for intra-team measurement. I talked about it a little on my blog recently.
The basic gist is that you get / loose points based on your checkin's. The more you checkin, the more tests you write, the more warnings / TODO's you clean up, the more points you get. It's true so far here "What get's measured get's done", warnings have gotten cleaned up, more tests have gotten written, and people checkin more frequently. It doesn't necessarily measure who's really productive, but it measures who is creating future work, so maybe in a way that is a measurement of productivity. -
So I agree that performance measuring can hurt, but if you measure on things you care about, code coverage, tests written, warnings removed, builds not failing etc. your measurement might start having a positive effect. (Just a side note, the Hudson Game would encourage #5)
I think it's important to measure things that you care about your team doing. I also think it's important for the team to agree on what they're being measured on. If you get team buy-in then it can lead to healthy competition.
I do not think that SLOC is a good measurement. It doesn't speak to quality. But there are a bunch of static analysis tools that do speak towards quality, that hook into a build. I think the idea of the Hudson game is to use that information to measure developers. Honestly, we've just started using the Hudson game, and it seems good, so I may be a little blinded by a new shiny toy, but can anyone see a downside to the Hudson game? -
My objections were to the measures as described. My real concern is there is any linkage between pay, bonus etc and any individual measure. See: gojko.net/2008/08/07/paying-programmers-are-bon... for a description of pay and performance.
I'm familiar with Hudson and think it an excellent way of highlighting quality issues as long as the information is kept private and within the team. I would be concerned if the team started to take it too seriously or management got involved. -
To follow your example: If you have two engineers, both create bridges that "still stand" but one designs two bridges for every one the other designs. Wouldn't you want to know that? Wouldn't the more productive engineer deserve more $$ or a promotion? Maybe there is something hindering the other engineer's performance. Isn't there value in knowing that?
Thanks. -
I believe you're right in the case you are working on a high-performing team. But, what if you have a team which isn't reaching deadlines or under performing? I believe you need empirical data (or as close to it as possible) to address members who are under-performing. Without it management could be accused of favoritism or worse.
Ideally one would prefer a culture to perform come from peers directly, through healthy competition, peer evaluations, etc. However, if this doesn't exist from within then it's management's job to make is so and productivity metrics are a vital tool. -
A couple of problems is this "lack of productivity" real or a sign of something else? Measure my productivity on a team and you will find its low. I'm the guy who acts as a mentor to everyone, I help review alot of code and I'm always refactoring and improving the infrastructure. When I'm part of team quality slow improves. Am I productive or not? By your measure not all.
What really matters here is the output of the team and are the team happy working together. If the team is producing high quality well tested, simple code - why does it matter how productive they are? If they're happy then who cares that one team member is really just acting as the glue.
When I coach I encourage management to set goals for the team and measure whether they meet those goals. Anything done on the individual level will destroy the hard won gains at the team level. -
<p>I ran the metrics program for a small (15-20 developers) project shop for 5+ years. We measured productivity for only one reason--project estimation for client proposals. We estimated two ways, using an agile-style planning game, and also using function point analysis. (I was a CFPS from 2002-2005). I counted each project as built, and used the time records to determine h/FP.</p>
<p>When it was time to estimate, I would count the requirements and consult the project history to predict h/FP. Having an independent estimation method was very helpful, and it also held up better under client pressure than the fully subjective methods. (Function Point Analysis is also an excellent way to spot requirements gaps).</p>
<p>To keep the whole thing squeaky clean (even though it was a high-trust culture already), I was the only one with access to the project history. Management never saw it, except in summary form.</p>
<p>Metrics are an amplifier. They will make a high-trust organization even more effective, and will make a low-trust organization even more toxic.</p>
<p>www.ufunctional.com</p> -
I worked in a company and it ever pushed a productivity improvement program, "2X", which means double your teams' productivity. It was not at individual developer level. We measured the productivity in code lines. You can guess what we got eventually, tons of redundant code in system.
Deming said that you never measure your people in digits. That is not only true in manufactory, but also in SW industry. SW development is more complicated than assembly line.
When you link personal performance with individual's productivity, you definitely can get the number increased. However, it is not your revenue or quality. Your people is smarter than you. They can figure out everything to make your happy to increase their salary and bonus.
As Bruce said, "Why do I care about programmer productivity?". Exactly right. We are selling quality products not the code lines or even funtional point. Sofware is not built by blocks.
If you DO want to measure sth of performance, following DON'T/DOs maybe useful:
1. DON'T use the digits that the team can control. Use inputs or outputs from/to customer. Do link it with your revenue.
2. DON'T compare data across teams. Do compare it with your past to see the trend.
3. DON'T use complicate/fancy methods. That will definitely will increase overhead, or effort waste, to your team. DO use some indicators which are really simple and easy to collect.
If you can use your leadership, forget the digits.
16 comments
Reply