This is a continuation of Part 1 where I began looking at Ron Baker's article on his Seven Moral Hazards of Measurements. I am converting Ron's hazards into some Dangers of Measuring Reality that can trip you up if you're not careful. We pick up with Ron's Hazard #4.
~~~~~~~~~~~~~~~~~~~~
Ron's Hazard #4: Measures Are Unreliable
This is Ron's hazard that I disagree with the most.
Ron's example here is that GDP increases when a couple get divorced but decreases when a child is born. Because the measure increases when something "bad" happens (divorce) and decreases when something "good" happens (birth), the measure therefore is unreliable. Ron's reasoning here gets a bit ... well, ridiculous.
Measurement unreliability has nothing to do with moral projections that one forces onto it. Unreliability has to do with the consistency and accuracy of data collection, summarization, and presentation of a measurement. It has nothing to do with a person's presumption that all "good" things should make measures go up and all "bad" things should make measures go down.
Using the earlier example of measuring the room temperature, is an increase in temperature good or bad? Depending on the starting temperature and the number of degrees it has increased, it's hard to say. In fact, half the people in the room might think it's good and half might think it's bad. Who is right? It's a completely subjective evaluation. When the outside temperature drops in January so the rain turns to snow, the snow boarders are happy and the driveway shovellers are unhappy. It all depends on one's perspective.
GDP is not defined to measure the sum of all "good" and "bad" things that happen in a nation. It measures the total economic output, and when calculated per capita, it will go up or down whenever the population or the economic output changes. If it does that accurately, then it is a reliable measurement.
Darren's Danger #4: Do Not Impose Meaning On A Measurement That Does Not Exist
With the exception of a few measurements (such as crime rates), most metrics are amoral. In other words, changes in the metrics may be good or bad depending on the situation or one's perspective. Imposing morality on measures will result in them being used inappropriately, or as Ron seems to suggest, being thrown out altogether.
~~~~~~~~~~~~~~~~~~~~
Ron's Hazard #5: The More We Measure The Less We Can Compare
Surgeon's death rates are the example for this hazard. Because patient death rates vary by the complexity of the patient's condition, simple death rates by surgeon can be misleading. Therefore, death rates are ALWAYS adjusted for risk so they are comparable across different patient populations.
Ron's example contradicts his point though. The more we measure, the more easily we can compare measures. Without measuring both surgeons' death rates AND patient complexity, we cannot reliably compare surgeons. The first measure is interesting, but not useful by itself.
Darren's Danger #5: Do Not Compare Apples and Oranges In A Single Measure
If you absolutely must compare apples and oranges, make sure you convert them into an equivalent measurement before you draw up your comparison. Surgeons' raw death rates are like comparing apples and oranges, but once you adjust for different levels of patient risk, those risk-adjusted death rates by surgeon become comparable -- apples to apples.
~~~~~~~~~~~~~~~~~~~~
Ron's Hazard #6: The More Intellectual the Capital, the Less You Can Measure It
I happen to agree completely with Ron on this one. There are some things that are just impossible to measure, but are very important nonetheless. As I quoted at the beginning of Part 1, not everything that counts can be counted. The knowledge and abilities of the people in an organization are incredibly important to the future value of the company, but it is very difficult to quantify those assets hidden inside an individual's brain.
And so I will state my danger as an alternate way of saying what Ron has said:
Darren's Danger #6: Do Not Forget The Important Things That Cannot Be Measured
The list of measurements published every month in your management report do not comprise all of the important things you must manage. It is important to remember the un-measurable aspects of your business too.
~~~~~~~~~~~~~~~~~~~~
Ron's Hazard #7: Measures Are Lagging
Ron is correct that most measures are snapshots of history, like driving by looking only in your rear-view mirror. He is correct, but that reality is quickly changing.
Most website hosts now provide real-time tracking of your website visitors, and large data warehouses are beginning to provide near-real-time reporting on some high-priority areas. For instance, I worked for a banking client that was implementing near-real-time reporting of credit card transaction data to identify fraudulent activity within minutes. It is not easy to do, but it is being done. Just because traditionally reporting has been slow and retrospective does not mean all reporting is that way.
On the other hand, a lot of processes do not require real-time reporting because they simply do not move very fast. Knowing how your customer service process performed last month is probably still useful to direct your efforts to improve your process this month. Just because the data is not immediately current does not mean is it without value. Timeliness is always dependent on the purpose of the measurement.
Darren's Danger #7: Data Without a Timestamp Is Probably Older Than You Think
Always, always, always include a date and time when a data record was captured. Never presume it's current unless it states it's current.
No comments:
New comments are not allowed.