Obtaining data is the first step towards improving any process. Our English word "data" comes from the Latin datum meaning "something given." All data must connect back to something in the real world, which gives that data some meaning or usefulness.
However, measuring things in the real world is not always a simple exercise.
Jason Goto, a former colleague of mine, points this out in his blog titled "Are you reporting what you can? ... Or reporting what you should?" Many companies collect lots of data, but it's not always the data they should be collecting. The most important measures are often the most difficult to collect. As Einstein famously said (or maybe it was William Bruce Cameron), "Not everything that counts can be counted, and not everything that can be counted counts."
I recently found Ron Baker's article on his Seven Moral Hazards of Measurements on LinkedIn. I believe his main point is that we need to recognize the limits of measurements, which is true. While his article is thought provoking, I beg to differ with most of his hazards. I'd like to convert them into some Dangers of Measuring Reality that can trip you up if you're not careful.
~~~~~~~~~~~~~~~~~~~~
Ron's Hazard #1: We Can Count Consumers, But Not Individuals
Aggregate data is not the same as individual data records, but that's OK. It's not supposed to be the same. Aggregate measures answer questions about groups or populations.
Ron's example of room temperature not equating to how many individuals feel hot or cold is an example of misusing a measurement for an unrelated purpose. The room temperature is measured using a thermometer on the wall and it answers the question, "How warm is the room?" To answer Ron's question, you need to take a very different form of measurement, probably a survey question that says, "Do you feel warm or cold right now?" Counting the responses to that question will permit you to know whether to raise or lower the thermostat in that room. We required a different measurement to answer a different question.
Darren's Danger #1: Do Not Force Measurements To Answer Unrelated Questions
All measures and indicators are created to answer a specific question. Know what question a measure is intended to answer, and don't force it to answer other questions.
~~~~~~~~~~~~~~~~~~~~
Ron's Hazard #2: You Change What You Measure
Heisenberg's Uncertainty Principle applies to the realm of the exceedingly small, such as shining a light on an individual atom. When a photon hits the atom, the atom is moved as a result of the collision and you no longer know that atom's location. In that case, the act of measuring truly does change reality.
However, in the big world that we live in, the act of measuring rarely changes the object that is measured. When I step up on a bathroom scale to measure my weight, the act of stepping on the scale might burn a small fraction of a calorie and thus reduce my weight by a minuscule amount. However, the accuracy of the scale cannot detect that change. Unfortunately I'm just as heavy now as I was before I stepped on the scale.
In fact, when we want to intentionally change a process for the better, simply measuring it will not produce any change. From my experience, just measuring and reporting on the process rarely improves it. It is not until some kind of incentive is linked to the measure (like the manager's bonus) that the process will start to change for the better.
Ron's point with this hazard is specifically related to performance targets, and how people will try to manipulate them to their advantage. That is a real problem, but it is not solved by giving up on measuring. It is solved by setting targets and incentives that are not easily gamed.
Darren's Danger #2: Do Not Set Targets That Are Easily Manipulated
Setting targets is fine, but not if the measurement can be manipulated. Incentives must be aligned with measures that are clearly defined and objectively determined.
When I worked in healthcare, a lot of measures were considered to measure hospital efficiency. Many indicators were ultimately rejected because it was possible to show improvement in the indicator by a means other than improving the hospital's efficiency, such as simply moving budget from one hospital department to another. Measures that are vaguely defined or simply "the best data we've got" will seldom make good candidates as target indicators.
When I worked in retail, there was one director who spent a lot of his time arguing that the definition of an efficiency metric should be changed because occasional random events reflected poorly on his performance (and thus his annual bonus). He conveniently ignored the other random events that showed an improvement on his performance! Whenever a target is set that affects one's pay, some people will spend more time trying to manipulate the metric than actually improving the business process itself. Performance metrics must be completely objective and airtight.
~~~~~~~~~~~~~~~~~~~~
Ron's Hazard #3: Measures Crowd Out Intuition and Insight
I think Ron's point on this hazard is fair, in that entrenched metrics can lull managers into a thinking there are no other important indicators or other areas to look for improvement. The traditional measures can act to stifle creativity and problem-solving skills that are still desperately needed in the company. This problem tends to occur when a reporting system reaches a level of maturity and acceptance in an organization.
However, the opposite is also true. Bringing new measures to a problem can often generate new perspectives and focus on problem solving, especially for large and complex processes. When I led a process improvement effort on a blood laboratory, the problems and process was just too large to try and fix everything at once. Introducing new measures quickly showed where the process bottlenecks were and allowed problem-solving to focus on those areas that would show the most improvement with the least effort. In that case, measures stimulated insight and creative problem-solving.
Darren's Danger #3: Do Not Stop Improving Your Metrics
Once you have implemented some metrics in your organization, do not get lulled into the deception that you are done. Keep improving, keep adding metrics, and have the courage to remove metrics when their usefulness has passed.
~~~~~~~~~~~~~~~~~~~~
To be continued in Part 2.
No comments:
Post a Comment