Developer productivity. It's a very difficult thing to measure objectively and it only gets more difficult with size. Finding an accurate gauge of developer performance, one that is quantifiable, is the holy grail of IT management in larger organizations.
Most measurement of developer productivity today is subjective. Managers have a vague sense of a person’s ability to produce and they measure the developer largely based on their ability to meet deadlines. This type of non quantifiable measurement has, among other negatives, the side effect of masking severe development team problems. There is little understanding by the management chain of how things are being accomplished, who the most valuable team members are, and where the bottlenecks stem from. Things only get worse when a mandate is sent down saying that productivity needs to increase by X% this year. If you’ve got no metric to measure and you’ve got no baseline to begin with, how are you expected to measure success or failure?
I don’t know what the answer is to create a purely objective system of measurement for developers that works. If I did I’d likely be a millionaire. There have been a myriad of tactics attempting to add hard data to development efforts such as lines of code written (*shudder*), version control commit data, automated code analysis, and so forth. These methods tend to suffer from the same problem, they can be easily gamed. If someone’s job or a promotion depends on a metric of this type, you can bet they’ll write code geared toward boosting that metric whether it makes sense or not. Until someone figures out a measurable system that isn’t easily gamed and that produces useful and accurate metrics, why not start by improving subjective measurement?
If you’re an attentive manager, you’ll see the level of code quality of your developers by observing the number of problems resulting from their code and the effort needed to correct the bugs. If a bug is caused by a simple typo or minor logic oversight, it’s much better than an entire section of code that needs to be re-written due to a massive structural issue for example. Building on that, if the developer usually delivers what is asked by the deadline or earlier, you can combine that knowledge with the error rate and severity of the error that you see over time. Likewise, if the developer is consistently late with their deliverables, you combine that with the number of bugs and their severity to form a quasi-consistent form of measurement, albeit a subjective one. Using these two measurements together is obviously better data than using one alone. If someone is consistently late with their deliverables but has much fewer bugs, it can be said that they are equally as or more productive than another developer who delivers on time but with a greater number of bugs etc.
The way I measure slightly more objectively, and the way I've seen it done in larger groups at previous jobs, is to break things down into units of work with defined time constraints. A unit of work can be an entire project, or it can be a feature or component of a larger project, or even a simple task or bug fix. You assign a worker to a unit of work and enforce a time period for the work. When the time period expires, you evaluate the result. That's where it get's subjective but you can at least gain metrics on the worker relative to a unit of work and time interval.
After a unit of work is complete, that unit is attached to the worker who completed it. If new units of work result from that work (bugs), you can track that back to the worker and prior unit of work to add to the metrics. Eventually this should show you the short and long term efficiency of the worker. To gain additional insight, you can employ Planning Poker or simply make your own estimate of a unit of work’s level of difficulty for use in your metrics on that unit of work / developer. By tracking these estimates, the resulting delta from the estimate, and the additional days or story points resulting from bugs, you’ll build a measurable metric to make your subjective measurement more qualified.
In bigger groups, you also need to track the ability of development cells and their managers. The leaders should be defining the right goals and directing the right technique. They should also be properly sizing the right units of work and time to complete, as well as properly estimating the days or story points. The superior would have their own metrics based on a larger unit of work and larger time period. And so forth up the chain to a single point of accountability so that overall development effort can be measured in some meaningful way. It’s by no means perfect but it’s more than a little better than nothing.
Having some form of unbiased measurement is important. It will identify problem points in the organization and drive improvement. Teams will get better at estimating efforts when the performance can be gauges. When times get tough it will help with the tough decisions. When things are good, it will justify raises and promotions. After all, you don’t want these types of things relying on the opinion, bias, or preference of an individual.
What are some other ways to add measurable data to coding performance? What have you seen that has worked well or failed miserably?