JPMorgan Learns About Exponential Distributions The Hard Way

Most people are familiar with the normal distribution, more commonly known as the bell curve and technically known as the Gaussian distribution.  Anyone who has had a teacher who marks on the bell curve is familiar with the concept.  It is a histogram in the shape of a bell, with most of the values near the mean and with equal portions on either side of the it, diminishing as you get further from the mean.

However, it seems the smartest minds on Wall Street assumed EVERYTHING in the world of finance follows the bell curve.  BIG mistake!  As it turns out, for JPMorgan it was a $6.2 billion big mistake.  Ooops.  You can read about their mistake here on Slate.com.  First, let me explain a few of the concepts involved and then we'll come back to J. P. Morgan's incredibly wrong assumption.

What is a Bell Curve?

The bell curve below is from mathisfun.com.  It shows that within 3 standard deviations (+/- 3 sigma) of the mean of a normal distribution, you will find 99.7% of the observations.  In other words, there just aren't a lot of events that occur very far from the mean.  The process improvement method Six Sigma takes its name from this concept that if you go out +/- six standard deviations from the mean, you should effectively never get any events occurring outside of this range.  That is only true of course if the variations in the manufacturing process follow a normal distribution, which fortunately is usually true.

The bell curve accurately describes variations in student marks and student heights, many manufacturing processes, and even the daily movements in the stock market, but it doesn't apply to everything we find in the real world.  For example, simple everyday events such as wait times at your grocery store checkout or bank teller, hospital inpatient length of stay, and the duration of telephone calls do not follow a bell curve.  These events follow an exponential distribution.

What Is An Exponential Distribution?

Exponential distributions are asymmetrical (i.e. skewed to one side of the mean), limited on one side by a minimum (usually zero), and have long tails.  In other words, events far from the mean can and do happen with much more frequency than in normal distributions.

My graph on the left shows one type of exponential family of curves called the Erlang Distributions, named after a Danish telephone engineer A. K. Erlang who began using them in the early 1900's.  These in turn are part of a larger family of exponential functions that engineers call Gamma Functions.

What Erlang discovered is that duration of telephone calls did not follow a normal distribution.  You could not have a call of zero duration or less (i.e. a minimum), but you could have an occasional telephone call that lasted hours (i.e. no practical maximum).  Erlang's job was to accurately predict how much switchboard capacity was required, which he ultimately succeeded in doing.

Because of these occasional large-value events, the rules of normal distributions do not apply.  For instance, you cannot presume that 99.7% of your events will fall within 3 sigma of the mean.  Depending on the specific shape of the exponential distribution, a measurable and significant portion of the events will fall far past 3 standard deviations from the mean, and Erlang had to take those events into account when planning his capacity.

JPMorgan's Mistake

As stated in the article, JPMorgan got into the business of complex credit swaps and assumed they would behave according to a normal distribution.

"Credit default swaps simply don’t behave in line with the normal, or Gaussian, distribution typically assumed. The so-called tail risks, or the chances of extreme events, are bigger than that theory predicts. Ina Drew, who ran the CIO, referred to one day’s mark-to-market losses as an eight standard deviation event, according to the report. That translates mathematically into something that should only happen once every several trillion years. It makes no more sense than Goldman Sachs finance chief David Viniar’s famous remark as the crisis unfolded in 2007 about seeing 25 standard deviation events, several days in a row."

Of course, the mathematics DOES predict this if you use the correct exponential distribution.  The credit swaps had very large potentials for one-day losses, which veer far away from the daily means, but those potentials were either ignored or just presumed to never exist.  JPMorgan failed to recognize the correct distribution for their credit default products and when losses mounted they just blamed mathematics for their wrong assumption.

One cannot simply ignore the rare-but-large-value events.  JPMorgan learned the hard way that with exponential distributions, the tail wags the dog.

And so JPMorgan shareholders are out $6.2 billion for that little mathematics oversight.  Oops.

The Dangers of Measuring Reality - Part 2

This is a continuation of Part 1 where I began looking at Ron Baker's article on his Seven Moral Hazards of Measurements.  I am converting Ron's hazards into some Dangers of Measuring Reality that can trip you up if you're not careful.  We pick up with Ron's Hazard #4.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #4: Measures Are Unreliable
This is Ron's hazard that I disagree with the most.

Ron's example here is that GDP increases when a couple get divorced but decreases when a child is born.  Because the measure increases when something "bad" happens (divorce) and decreases when something "good" happens (birth), the measure therefore is unreliable.  Ron's reasoning here gets a bit ... well, ridiculous.

Measurement unreliability has nothing to do with moral projections that one forces onto it.  Unreliability has to do with the consistency and accuracy of data collection, summarization, and presentation of a measurement.  It has nothing to do with a person's presumption that all "good" things should make measures go up and all "bad" things should make measures go down.

Using the earlier example of measuring the room temperature, is an increase in temperature good or bad?  Depending on the starting temperature and the number of degrees it has increased, it's hard to say. In fact, half the people in the room might think it's good and half might think it's bad.  Who is right?  It's a completely subjective evaluation.  When the outside temperature drops in January so the rain turns to snow, the snow boarders are happy and the driveway shovellers are unhappy.  It all depends on one's perspective.

GDP is not defined to measure the sum of all "good" and "bad" things that happen in a nation.  It measures the total economic output, and when calculated per capita, it will go up or down whenever the population or the economic output changes.  If it does that accurately, then it is a reliable measurement.

Darren's Danger #4:  Do Not Impose Meaning On A Measurement That Does Not Exist

With the exception of a few measurements (such as crime rates), most metrics are amoral.  In other words, changes in the metrics may be good or bad depending on the situation or one's perspective.  Imposing morality on measures will result in them being used inappropriately, or as Ron seems to suggest, being thrown out altogether.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #5: The More We Measure The Less We Can Compare

Surgeon's death rates are the example for this hazard.  Because patient death rates vary by the complexity of the patient's condition, simple death rates by surgeon can be misleading.  Therefore, death rates are ALWAYS adjusted for risk so they are comparable across different patient populations.

Ron's example contradicts his point though.  The more we measure, the more easily we can compare measures.  Without measuring both surgeons' death rates AND patient complexity, we cannot reliably compare surgeons.  The first measure is interesting, but not useful by itself.

Darren's Danger #5:  Do Not Compare Apples and Oranges In A Single Measure
If you absolutely must compare apples and oranges, make sure you convert them into an equivalent measurement before you draw up your comparison.  Surgeons' raw death rates are like comparing apples and oranges, but once you adjust for different levels of patient risk, those risk-adjusted death rates by surgeon become comparable -- apples to apples.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #6:  The More Intellectual the Capital, the Less You Can Measure It
I happen to agree completely with Ron on this one.  There are some things that are just impossible to measure, but are very important nonetheless.  As I quoted at the beginning of Part 1, not everything that counts can be counted.  The knowledge and abilities of the people in an organization are incredibly important to the future value of the company, but it is very difficult to quantify those assets hidden inside an individual's brain.

And so I will state my danger as an alternate way of saying what Ron has said:

Darren's Danger #6:  Do Not Forget The Important Things That Cannot Be Measured
The list of measurements published every month in your management report do not comprise all of the important things you must manage.  It is important to remember the un-measurable aspects of your business too.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #7:  Measures Are Lagging
Ron is correct that most measures are snapshots of history, like driving by looking only in your rear-view mirror.  He is correct, but that reality is quickly changing.

Most website hosts now provide real-time tracking of your website visitors, and large data warehouses are beginning to provide near-real-time reporting on some high-priority areas.  For instance, I worked for a banking client that was implementing near-real-time reporting of credit card transaction data to identify fraudulent activity within minutes.  It is not easy to do, but it is being done.  Just because traditionally reporting has been slow and retrospective does not mean all reporting is that way.

On the other hand, a lot of processes do not require real-time reporting because they simply do not move very fast.  Knowing how your customer service process performed last month is probably still useful to direct your efforts to improve your process this month.  Just because the data is not immediately current does not mean is it without value.  Timeliness is always dependent on the purpose of the measurement.

Darren's Danger #7:  Data Without a Timestamp Is Probably Older Than You Think
Always, always, always include a date and time when a data record was captured.  Never presume it's current unless it states it's current.

The Dangers of Measuring Reality - Part 1

Obtaining data is the first step towards improving any process.  Our English word "data" comes from the Latin datum meaning "something given."  All data must connect back to something in the real world, which gives that data some meaning or usefulness.

However, measuring things in the real world is not always a simple exercise.

Jason Goto, a former colleague of mine, points this out in his blog titled "Are you reporting what you can? ... Or reporting what you should?"  Many companies collect lots of data, but it's not always the data they should be collecting.  The most important measures are often the most difficult to collect.  As Einstein famously said (or maybe it was William Bruce Cameron), "Not everything that counts can be counted, and not everything that can be counted counts."

I recently found Ron Baker's article on his Seven Moral Hazards of Measurements on LinkedIn.  I believe his main point is that we need to recognize the limits of measurements, which is true.  While his article is thought provoking, I beg to differ with most of his hazards.  I'd like to convert them into some Dangers of Measuring Reality that can trip you up if you're not careful.


~~~~~~~~~~~~~~~~~~~~


Ron's Hazard #1: We Can Count Consumers, But Not Individuals
Aggregate data is not the same as individual data records, but that's OK.  It's not supposed to be the same. Aggregate measures answer questions about groups or populations.

Ron's example of room temperature not equating to how many individuals feel hot or cold is an example of misusing a measurement for an unrelated purpose.  The room temperature is measured using a thermometer on the wall and it answers the question, "How warm is the room?"  To answer Ron's question, you need to take a very different form of measurement, probably a survey question that says, "Do you feel warm or cold right now?"  Counting the responses to that question will permit you to know whether to raise or lower the thermostat in that room.  We required a different measurement to answer a different question.

Darren's Danger #1:  Do Not Force Measurements To Answer Unrelated Questions
All measures and indicators are created to answer a specific question.  Know what question a measure is intended to answer, and don't force it to answer other questions.

~~~~~~~~~~~~~~~~~~~~


Ron's Hazard #2:  You Change What You Measure
Heisenberg's Uncertainty Principle applies to the realm of the exceedingly small, such as shining a light on an individual atom.  When a photon hits the atom, the atom is moved as a result of the collision and you no longer know that atom's location.  In that case, the act of measuring truly does change reality.

However, in the big world that we live in, the act of measuring rarely changes the object that is measured.  When I step up on a bathroom scale to measure my weight, the act of stepping on the scale might burn a small fraction of a calorie and thus reduce my weight by a minuscule amount.  However, the accuracy of the scale cannot detect that change.  Unfortunately I'm just as heavy now as I was before I stepped on the scale.

In fact, when we want to intentionally change a process for the better, simply measuring it will not produce any change.  From my experience, just measuring and reporting on the process rarely improves it.  It is not until some kind of incentive is linked to the measure (like the manager's bonus) that the process will start to  change for the better.

Ron's point with this hazard is specifically related to performance targets, and how people will try to manipulate them to their advantage.  That is a real problem, but it is not solved by giving up on measuring.  It is solved by setting targets and incentives that are not easily gamed.

Darren's Danger #2:  Do Not Set Targets That Are Easily Manipulated
Setting targets is fine, but not if the measurement can be manipulated.  Incentives must be aligned with measures that are clearly defined and objectively determined.

When I worked in healthcare, a lot of measures were considered to measure hospital efficiency.  Many indicators were ultimately rejected because it was possible to show improvement in the indicator by a means other than improving the hospital's efficiency, such as simply moving budget from one hospital department to another.  Measures that are vaguely defined or simply "the best data we've got" will seldom make good candidates as target indicators.

When I worked in retail, there was one director who spent a lot of his time arguing that the definition of an efficiency metric should be changed because occasional random events reflected poorly on his performance (and thus his annual bonus).  He conveniently ignored the other random events that showed an improvement on his performance!  Whenever a target is set that affects one's pay, some people will spend more time trying to manipulate the metric than actually improving the business process itself.  Performance metrics must be completely objective and airtight.


~~~~~~~~~~~~~~~~~~~~



Ron's Hazard #3:  Measures Crowd Out Intuition and Insight
I think Ron's point on this hazard is fair, in that entrenched metrics can lull managers into a thinking there are no other important indicators or other areas to look for improvement.  The traditional measures can act to stifle creativity and problem-solving skills that are still desperately needed in the company.  This problem tends to occur when a reporting system reaches a level of maturity and acceptance in an organization.

However, the opposite is also true.  Bringing new measures to a problem can often generate new perspectives and focus on problem solving, especially for large and complex processes.  When I led a process improvement effort on a blood laboratory, the problems and process was just too large to try and fix everything at once.  Introducing new measures quickly showed where the process bottlenecks were and allowed problem-solving to focus on those areas that would show the most improvement with the least effort.  In that case, measures stimulated insight and creative problem-solving.

Darren's Danger #3:  Do Not Stop Improving Your Metrics
Once you have implemented some metrics in your organization, do not get lulled into the deception that you are done.  Keep improving, keep adding metrics, and have the courage to remove metrics when their usefulness has passed.


~~~~~~~~~~~~~~~~~~~~



To be continued in Part 2.