Why Goals and Measures Matter
Given the power of goals to direct and focus our attention and motivation, it’s probably a good thing to invest at least some effort into being deliberate about how we choose them, be they personal or business, micro or macro.
In enterprise settings where incentives are involved, people will bring their own perspectives on what matters most, consciously or otherwise. The challenge in that context, beyond inspiring action, is to avoid conflicting priorities, and that is easier said than done.
Goal-setting frameworks like OKR are useful in helping to structure, communicate and track goals, but they don’t on their own solve the problem of choosing the right objectives and the right measures. Thoughtfulness and care are necessary if you want to maximise the chances of driving the right behaviours and avoid some of the pitfalls around measurement. Frameworks help, but they can’t do the thinking for you.
Used carelessly, goals and incentives have strong potential for unintended consequences, which can be very expensive and even take you completely in the wrong direction.
When Goals Go Wrong
Take the story of the Great Hanoi Rat Massacre. In 1902, French colonial administrators in Vietnam had built sewers under the part of Hanoi where they lived, and unfortunately these sewers became a perfect breeding ground for rats. This concerned them enormously because there was an outbreak of bubonic plague in China. This was at a time when scientists had recently linked the spread of plagues to rodents, so they feared it being carried by rats on ships and trains that came from there.
Their solution was to enlist the locals to help them kill rats. They offered a bounty of 1 cent per rat tail.
- Goal: Prevent the spread of bubonic plague by reducing the rat population of Hanoi.
- Success measure: Number of rat tails.
- Outcome: Oh. More rats. Damn!
So what went wrong? It seemed to go well at first; plenty of rat tails and bounties. But gradually, officials started noticing more and more live rats without tails. Worse than that, rat farms sprang up around Hanoi. The enterprising locals had gamed the system. The system, in fact, had asked for it by creating a rat economy.
The well-intentioned incentive had put the organisational goal at odds with those of the people doing the work. Success meant fewer future income opportunities, making rat tails a very bad proxy for reduction in the rat population.
The administrators understandably cancelled the rat tail bounty, a move unpopular with the locals, as I’m sure you can imagine. Which, incidentally, brings us to one reason to be careful working with incentives. Giving something and then taking it away is worse than not having given it in the first place. People have memories, so once you introduce an incentive, you change the system. Not getting it right first time can make it harder to get it right next time, because now you may have introduced resentment and resistance. For this reason, it is better to start small with a trial if you can, to limit the blast radius if things go wrong.
Fixing Metrics vs Fixing Intent
Anyway, it’s easy with hindsight to see that rat tails were a bad proxy measure, but that’s another feature of complex systems – causes often seem obvious in retrospect, but can be difficult to predict looking forward.
What could our unfortunate French administrators have done instead? What lessons can be learned from stories like these to help us avoid similar mistakes?
How about a bounty of whole rat bodies instead of rat tails? Well, that might remove the incentive to free rats to breed after chopping off their tails, but it wouldn’t disincentivise the rat farming operations. A partial solution at best, but still open to abuse. Adjusting the metric without revisiting the intent simply moves the problem elsewhere.
So, let’s take a step back and consider the ultimate intent behind the initiative – to prevent or minimise bubonic plague infections. The main success measure, the lagging indicator they really care about, is the infection rate. Framing success in those terms doesn’t pre-suppose a solution. It leaves open the possibility for initiatives other than killing rats in the sewers of Hanoi. It opens up the solution space to allow us to consider other dimensions like prevention. Infection-carrying rats arriving on ships were known to be a key vector of infections, so they could look at measures to control the entry points, stricter port controls, mandatory periods at anchor before docking, inspections before unloading, and all sorts of other initiatives.
The OKR framework wasn’t a thing in those days, but if it had been, the administrators could have used it to approach the problem with a balanced set of leading and lagging indicators, working in short cycles, and closely monitoring for impact to catch unintended consequences early.
Let’s try to write an OKR for our French colonial administrators (in reality, an endeavour like this would use a set of related OKRs focused on different aspects of the problem, but I want to keep this simple just to illustrate the point).
An OKR could have looked something like this (don’t get hung up on the numbers. I’m just guessing here. Really they would be whatever was ambitious but doable in the time period covered by the OKR):
Objective: Prevent the introduction and spread of bubonic plague in the city
KR 1: Increase the percentage of arriving ships held at anchor at least 500 feet offshore and inspected before docking from 0 to 80%
KR 2: Increase the percentage of arriving ships to have implemented rat prevention measures before docking from 10% to 90%
KR 3: Increase the percentage of sewer access points in port adjacent districts that are sealed or trapped from 0% to 75%
KR 4: Increase percentage of new infections reported within 7 days from 20% to 80%
KR 5: Decrease 4-week rolling window infection rates near ports from 60 to 20
Hopefully, you can see from this quick example that being clear about the core intent in the objective statement instead of jumping straight to a solution opens things out. I’ve been a bit lazy here because ideally an objective should be focused on one thing, whereas this one chases two. We could consider splitting this into two prevention focused OKRs, one each for introduction and spread. Still, even with this purist-offending, slightly lazy OKR, notice how we have some key results that are leading indicators that readily suggest initiatives, one that improves data-gathering and another lagging indicator that measures for overall impact.
Success for the objective is defined as meaningful progress in all these dimensions, not just one single metric. This is a core strength of the OKR framework – it encourages a holistic approach to defining what must be true for an objective to be achieved. The requirement to come up with multiple key results and balance them out makes you think harder.
It’s not a silver bullet and you can still get things wrong, but the rigour involved improves your chances. Importantly, defining success at the objective level rather than focusing on a single metric reduces the chances of driving unintended behaviours.
The Corrupting Impact of Measures
The potential for unintended consequences driven by measures has been observed by many and well studied. Most often cited are Goodhart’s law and Campbell’s law.
Goodhart’s law states:
When a measure becomes a target, it ceases to be a good measure
Campbell’s law states:
The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
Both apply to our rat-tail example, but Campbell’s is probably the best fit because it captures the corrupting effect. Of course, Goodhart’s is true too – number of rat tails did indeed cease to be a good measure, if it ever was one, but in this case the attachment of an incentive led to deliberate corrupt behaviour and an unintended outcome.
We see this effect in education when a focus on test results leads to the behaviour of teaching to the test. We see it in software engineering when a focus on lines of code leads to a bloated and unmaintainable code-base. Once measures are introduced, behaviour adapts (not always deliberately or consciously) and this effect is magnified when you throw incentives or penalties into the mix. It’s human nature.
Measurement is more than just observation, it’s intervention too. Once you introduce a measure, you change the system, especially if you attach incentives. If you’re not careful, you can invite gaming and leave yourself wide open to unintended outcomes. Balance your metrics, monitor closely and favour short cycles. And remember, goal-setting frameworks are more forcing function than silver bullet, serving to help you maintain discipline around how you do goal setting. If anything, they should make you think more, not less.