Nuclear Launch Codes and an Idiot's Luggage: The Danger of False Assumptions

I saw a fascinating article this week by Karl Smallwood on the website www.TodayIFoundOut.com. Apparently for 20 years, the Minuteman nuclear silos all had the same launch code of 00000000. Truth really is stranger than fiction.

It reminds me of the Spaceballs scene where the king is blackmailed into revealing his secret combination of 12345.  Rick Moranis' character shouts back, "That's the stupidest combination I've ever heard in my life! That's the kind of thing an idiot would have on his luggage!" Apparently the luggage of idiots was about as secure as the American nuclear arsenal for many years.

The launch code was revealed in 2004 by a former Minuteman launch officer. Apparently this passcode security was initiated by JFK to prevent a rogue general from starting a nuclear war on his own. The passcode was accompanied by a strong security system that could not be hot-wired to get around the passcode -- the missile simply could not launch without it. However, that entire safety system was undermined by generals who ignored the president's order and required the passcodes to be reset to zeroes after each inspection to ensure the missiles could be launched quickly if the Russians ever fired first.

I experienced a similar situation at one large organization that had very tight security on their enterprise data warehouse, although unlike the generals in charge of missiles this was an unintentional breach of security policy. Because a portion of the data warehouse contained sensitive client information, separate passwords were issued only to those employees that had a justifiable business need to access the data.  The complex password could not be changed by the user and it expired after a few months so the user had to repeat the application process to prove that they still had a business need to access the data. It was a fairly complex security system that was completely undermined by a single failed assumption. A new security analyst was transferred from the helpdesk to the data warehouse team to issue passwords, and he thought it was the same process as at the helpdesk, namely issuing temporary passwords to users who had forgotten theirs. No manager got around to telling him what his new role actually entailed. When I got my data warehouse password, I was shocked to see it was abc123. I soon found out that everyone in the department had the same password and that this had been happening for a few weeks already. Because no user liked the old complex passwords that they couldn't remember, everyone was thrilled with this new simple password and no one had any incentive to notify the data security manager of the obvious flaw in the process. Despite very strict data security processes and policies, sensitive client data had been made as secure as an idiot's luggage.

What's the lesson? 

It is so easy to make assumptions that will undermine the greatest of plans. Everyone assumed that generals would always obey an order from the president. That turned out to be a false assumption. Everyone assumed that each new data security analyst would be trained in the process of issuing new data warehouse passwords. On one occasion that turned out to be a false assumption. Simple human failures can undermine even the most complex security processes when assumptions are not identified and tested.

I've designed data processes for many years. It's a helpful discipline to get in the habit of asking myself:

1. What assumptions have I made this time?  
2. If any of those assumptions turn out to be false, will that have a significant effect on my process?
3. If the effect is significant, can I build control processes to identify when those assumptions are false?

For example, in the data warehouse password situation above, the assumption was that the person issuing passwords will be trained on the procedures for issuing passwords. If that assumption is false, the effect is very significant. Therefore a simple control process would be to generate a report each day showing all the new passwords that have been issued. One glance at such a report would've shown that all passwords were suddenly identical and that an urgent problem existed.

The challenge is that answering question #1 is incredibly difficult. We simply aren't used to identifying our assumptions -- they're automatic. One way to get around that is to skip question #1 entirely and rephrase question #2 to be:

    2. If my process fails, will that have a significant effect on the business?

If yes, then build some control processes to identify process failures as early as possible. The failed assumptions will become clear later once problems are encountered.

The 75 KPIs Every Manager Should Know? Seriously?

I like LinkedIn for a number of reasons, but one reason is that articles my connections read show up on my update stream too.  One article that caught my eye recently was from Bernard Marr entitled, "The 75 KPIs Every Manager Needs To Know" which is an excerpt from his book of the same name.

When I saw the title, I actually laughed out loud at my desk.  Probably not the best thing to do in an open concept office.  My desk neighbours must have thought I was watching a YouTube video of a cat playing piano.

Do managers really need to know 75 Key Performance Indicators (KPIs)? The short answer is no.

Firstly, there is simply no need for any manager to know 75 KPIs.  That's like asking people to memorize the reference books in the library in case they ever need to use any of that information.  You just need to know where to go to look things up when you need them.

Secondly, no business can have 75 Key Performance Indicators.  They may very well have 75 Performance Indicators, but they cannot all by Key.  In Bernard's defense, he does say in his article that you shouldn't use all 75 of his KPIs, but then why would he say that every manager should know all 75 of them if they're not supposed to use them all?

Can you imagine having 75 keys on your key chain?  How would you keep track of what each key opened?  How would you carry them around?  Having that many keys is just not workable, and it's the same with Key Performance Indicators.  You just need a handful of the important ones.

Bernard's list of KPIs is interesting but certainly not exhaustive.  There are other potential indicators to use, especially industry-specific ones.  Learning a list of possible indicators is just not an important step on the way to developing KPIs for your business.  In fact, it's not even a necessary step.

The process of KPI development begins with your organization's purpose, not with a list of indicators.  Once you've stated a purpose, you can then start creating (or looking up) KPIs that answer the question, "Are we achieving our purpose?"

Save your brain cells -- don't bother memorizing 75 KPIs.

Visualizing Your Network via Gmail

An interesting visualization tool has been created by a team at MIT called Immersion.  It maps your network of contacts by analyzing the metadata in your Gmail account.  It doesn't read your emails per se, but just some of the metadata.  (Coincidentally, that's what the NSA is reading from your emails as well!)

See the sample image below from their demo.

I have about 7 years worth of emails in my Gmail account, so it provides an interesting picture of my contacts.  I don't use Gmail for all of my business communications -- once I'm on contract with a client, they usually issue me a company email account -- however, it still gives a clear picture of what parts of my network are connected and which clusters are distinct.

Not sure I can do anything useful with this knowledge, but it was an interesting visualization to see.  At least it made it worth keeping all those old emails that I felt I should've cleaned up long ago.

Procrastination has its benefits!

Data Presentation Is Always Subjective

One of the endearing qualities of data is that it is factual.  Facts are a piece of truth and are not simply a matter of one's opinion.  Data can be misrepresented, incomplete, or erroneous, but accurate data is always objective.

At least that's what I believed in my early days of working with data.  The reality is that any presentation of data is subjective, and therefore the objectiveness of the underlying raw data becomes less significant.  One can avoid subjectivity by never presenting the data, but that also assures the data remains useless.

Jake Porway has an interesting blog article at Harvard Business Review entitled The Chart Wars Have Begun.  He refers to an interactive chart in The New York Times that shows how job data and the unemployment rate can be presented three ways:  the "normal" way, through a Democrat lens and through a Republican lens.  None of the views are dishonest or misleading, they are just choices that must be made for any presentation of data.  It's a simple but excellent demonstration of the power of data presentation.

Screen capture of the NY Times chart mentioned above.  Click here for the interactive version.

Every data chart or table requires the creator to make decisions that include:

• what minimum and maximum values should be shown on the y-axis.
• how much historical data should be displayed.
• what scale should be used.  (i.e. linear or logarithmic)
• whether multiple data sets should be displayed together.
• the type of chart (e.g. pie, bar, line, scatter) or the organization of the table (e.g. which fields are displayed in which order).
• whether certain information should be highlighted, such as an arrow showing a trend or an arbitrary baseline.

Each of these decisions will lead the reader to realize or overlook certain conclusions from the data.  The data itself may be truth, but the presentation will always be dependent on the subjective choices of the presenter.

Exponential Distributions, Part 4 - Manage to the Mean, Not the Mode

In Part 3 we looked at the forgetfulness of exponential queues, namely the memoryless property.  In this last article, we look at how to manage queues and systems that behave according to an exponential distribution.

Exponential Attribute #4 - The Mean is More Important Than the Mode

Or, Your Gut Feeling Will Fail You

In a normal distribution (bell curve), the peak value coincides with the mean value.  The most common occurrence (the mode) also happens to be the average.

Not so with the exponential distribution.  The asymmetrical tail skews the average away from the peak.  The mean will always be larger than the mode.

So what?  It turns out this subtle mathematical difference can play tricks on your mind if you're not careful.  If you're a manager, that translates into consistently being over-budget and consistently failing to meet your operational performance targets.

Managing to the Mode or to the Mean?

As a manager, you gradually get a feel for how your department operates.  You have a sense of the typical system behaviour, whether it's how long a call on your help desk lasts or how much it will cost to reprogram 100 lines of code.  There is randomness involved, but your experience tells you the randomness averages out over time.  If your system behaves according to a normal distribution, then it's hard to know if you're managing to the mode or to the mean.  In fact, it doesn't matter because the two values are the same.

However, if your system behaves according to an exponential distribution, it suddenly matters which value you are managing to.  When you want to manage to achieve the budget targets, you must use the mean.  The mode is irrelevant.

I have found that people tend to manage to the most common events.  It is easy to get a "gut feel" for the things that happen the most frequently, but it is not as easy to get a "gut feel" for the average.  That means your "gut" manages to the mode.

Therefore if you manage according to your gut, an exponential system will cause you to fail every time.  If you budget to the mode, you will always end the year over budget.  If your target time for callers kept on hold is set by the mode, you will never achieve your target.  You must budget and manage to the mean.

Beware of Management CFIT

In aviation, investigators created a term called CFIT, which means Controlled Flight Into Terrain.  It can be the cause of a crash when the aircraft is working fine, including the instruments, but the pilot ignores the signs and flies by "gut feel" instead.  When visibility is poor, that means the pilot can fly the plane directly into the ground, sea, or mountain without realizing it.

When you are managing a system that behaves in an exponential way, you must fly by instruments and not by your gut.  Your instruments tell the nurse manager that the average stay of her inpatients is 5 days, but her gut feel is that 3 days is more common.  Her gut is right (i.e. the mode length of stay is 3 days) but she must manage to the mean of 5 days to stay on budget.

That is why reliable and frequent reporting is key to managing exponential systems, because without reliable instruments you cannot pilot the system properly.

Manage to the Mean AND Build Flexibility

Managing to the mean is still not enough to be a successful manager however.  As discussed in Part 2, with exponential distributions the tail wags the dog.  You must be prepared for the large but rare outlier events that will have a huge impact on your system, such as the spinal injury patient whose length of stay is greater than 1 year.  Over the long term the average value will become correct, but during a single budget year there may not be enough time to absorb the effect of such a large-valued event.  To handle those events, you need to build in some flexibility into the system.

Flexibility can take different forms.  It can involve cross-training of staff to permit additional capacity when a long-duration event occurs.  For instance, asking a mortgage specialist to cover as a bank teller when one customer has an hour's worth of coins to deposit or having an extra doctor on call when the Emergency waiting time exceeds a few hours.  It can also take the form of budget reserves.  For instance, the cost of that rare spinal injury patient cannot be predicted by simply budgeting to that nursing unit's averages, but across a hospital with a few dozen nursing units, those rare outlier events may happen with much more predictability.  (This is the Central Limit Theorem in action.)  A once-in-a-half-century event for a nursing unit may become a bi-annual event across the entire hospital, which becomes much easier to budget for.  When an unusual event occurs, transfer that reserve to the affected nursing unit and the departmental budgets stay on track.  You have managed to predict an unpredictable event and reduced it's impact on your system.

Summary

What I hope to have shown in these few articles is that exponential distributions are real, they have some counter-intuitive effects, but they are actually quite predictable and manageable if you understand their behaviour.

Exponential Distributions, Part 3 - Queues Just Don't Remember

In Part 1 and Part 2, we have looked at some characteristics of exponential distributions, particularly the Erlang distributions.  In this article we tackle another attribute of an exponential queue.

Exponential Attribute #3 - The Memoryless Property

Or, Always Press the Reset Button When You Arrive

Queues are just line-ups, so how can a queue possess a memory?  
As it turns out, some queues and systems do have a memory, but not all.  It is actually quite intuitive that all queues should have a memory.  We expect that what happened before you got in line should affect what happens after you arrive.  However, queues that are memoryless are not affected by what happened before you arrive.  That behaviour is counter-intuitive, at least at first glance.

What is a "Memoryless" Queue?

A memoryless queue refers to the property that what happened before you arrived in the queue will have no affect on the timing of the next event after you arrive.  Why does this matter?

Let's create a fictional example of waiting for a taxi cab to pass by on a street corner to take you to your next meeting.  You arrive at a random time and taxis are passing that corner at random intervals.  How long will you expect to wait for the next taxi on average?

If some data is collected at that corner over a period of days, let's assume the data shows the taxi intervals follow a normal distribution (bell curve) with an average interval of 5 minutes.  When you arrive on the corner, sometimes you will have just missed a taxi and would expect to wait a full 5 minutes for the next one.  At other times, you might arrive 5 minutes after the last taxi and would expect to wait a very short time for the next one.  On average therefore, you would expect to wait 1/2 of the average interval time, or 2.5 minutes.  Further, if there was a hotdog vendor on that corner who paid more attention to taxis than to his customers, you could ask him how long ago the last taxi passed.  If it was 5 minutes ago, then you would expect to wait only a short time, whereas if the last taxi passed seconds ago, you would expect to wait about 5 minutes.  In other words, the memory of the previous event in the system affects your expected wait time.  That is intuitive.

However, now assume the data instead showed that taxi intervals on that corner follow an Erlang distribution with an average interval of 5 minutes.  How long would you expect to wait for a taxi now?  It turns out the correct answer is 5 minutes -- not half the average interval, but the average interval itself.  Further, it doesn't matter what the hotdog vendor tells you about the last taxi to pass by.  Whether it was 5 seconds ago or 10 minutes ago, the expected average wait time for you will be 5 minutes from when you arrive.  It is like the queue has no memory -- it doesn't matter what happened before you arrived, the expected wait time clock resets itself.  How could this be?  Surely it cannot be true.  It appears counter-intuitive.

The easiest way to demonstrate this is with a non-queue example.  Imagine a factory that manufactures pipe. One machine extrudes pipe continuously and an automated saw immediately cuts the pipe into either 1 meter or 9 meter lengths, depending on the next customer order in the computer system.  Effectively the order of pipe lengths is random, but on average the machine produces 50% short pieces and 50% long pieces each shift.  The pipe continues moving along the assembly line at a constant speed where a robotic arm randomly selects pieces of pipe for quality control testing.  An arm shoots out and knocks the pipe that happens to be passing by at that moment into a bin.  The arm is programmed to randomly activate a certain number of times per shift.

At the end of each shift, what proportion of long and short pieces of pipe would you expect to find in the bin?  Even though that machine produces half short and half long pipes, that will not be the composition in the bin.  Because the long pieces of pipe take 9 times longer to pass the robotic arm, the odds are 9 times greater that a long piece will be selected.  Therefore you would expect 90% of the pipes in the bin to be long ones.

It is the same principle with memoryless queues.  Even though a long interval between taxi cabs is rare, the odds are greater that you will happen to arrive while one of those long intervals is occurring.  That increases the expected wait time, and it turns out to increase it exactly to the mean value.  Even if that last taxi came 8 minutes ago, you can still expect to wait 5 more minutes.  You could just be in one of those rare but long duration intervals.

And that is what it means for a queue to be memoryless.  Whatever happened before you arrived is irrelevant -- the clock resets when you get there.

Exponential Distributions, Part 2 - The Tail Wags the Dog

In Part 1, we looked at the asymmetry of exponential distributions, particularly Erlang Distributions.  We concluded that in terms of randomness, bad luck will come more frequently than good luck.  Now in Part 2, we look at the magnitude of that bad luck.  It's not just about the number of bad luck events, but how large those individual events can be and how much effect they can have on the behaviour of a system.

Exponential Attribute #2:  Outliers Cannot Be Ignored
Or, The Tail Wags the Dog

What is an outlier?

An outlier is a statistical term for an observation that is markedly different from the other observations in the dataset.  It is ultimately a subjective definition even though they are often determined using statistical formulas.  Common practice uses +/- 3 standard deviations from the mean as the boundary for outliers, which is reasonable for normal distributions.  However, there is no mathematical rule that says this boundary definition is better than any other.

So what is the purpose of identifying outliers?

The primary purpose is to exclude measurement errors or other unusual occurrences that would bias the dataset and lead one to make a wrong conclusion.  That's a worthwhile goal.  However, the opposite danger is to exclude valid measurements as outliers and that too can lead one to make a wrong conclusion.  Excluding outliers is a double-edged sword.

The chart on the right is a box plot of the Michelson-Morley Experiment results from 1887 where the speed of light was measured as the earth moved through the supposed "aether wind" of space.  There were 5 experiments with 20 observations each.  The top line of each box shows the 75th percentile value, the bottom line is the 25th percentile, the middle bold line is the median, and the T's are the maximum and minimum.  The small circles represent outliers as per the boundary definition above.  For experiment #3, four of the observations are deemed to be outliers -- two large-value outliers and two small-value outliers.  As you can see, while statistically they may be considered distant from the rest of the values in their group, three of them are within the normal "inlier" ranges of the other four experiments.

This is a good example of the difficulty and arbitrary nature of defining an outlier.  When you realize that ALL of the variation in this experiment is solely due to measurement error -- the speed of light is not changing -- then it raises the question as to why some measurement errors would be accepted as inliers and other errors would be deemed outliers.  How does one know if the two low-value outliers in experiment #3 are closer to the true value than the other 18 higher values?  As it turns out, the true speed of light is at value 792 in this chart (792 + 299,000 km/s).  That means one of the low-value outliers in Experiment #3 is just as close to being correct as the 25th percentile value.  It was a valid measurement and should not be classified as an outlier.

So even though this dataset was roughly normally distributed, it was still difficult to find the true outliers.  With exponential distributions, it gets even more difficult because of the long tail.  Simply going out 3 standard deviations from the mean does not give you a reasonable boundary for identifying outliers.

For example, using an outlier boundary of +/- 3 standard deviations on a normal distribution would define 0.27% of the events to be outliers, or 1 out of every 370 events.  That means if you were measuring the outdoor temperature once per day, you would expect to see either a high or low outlier about once per year.  If however the temperature happens to follow an Erlang distribution (with k=3, lambda=6), you would define 1.2% of the events to be outliers using the 3 standard deviation rule.  That means you would expect to see an outlier temperature once per quarter.  That's not particularly infrequent.  (Yes I know temperature doesn't follow an Erlang distribution, but humour me for a minute!)  Erlang distributions do not have the heaviest tails either -- other distributions such as the Log-normal or Weibull can have much heavier tails and therefore much greater proportions of their events beyond the 3-sigma boundary.

It's not just the number of outliers that's important to understand, but the size of each outlier as well.  The heavier the tail, the higher the outlier value can be.  In a normal distribution, the rare outlier events will still have values fairly close to the outlier boundary.  In fact, an outlier greater than +/- 6 standard deviations from the mean essentially never occurs in a normal distribution.  Not so with exponential functions.  The tails go on and on, and some outliers can have absolutely huge values.  Using our Erlang function parameters above, we would expect 0.016% of outliers to fall beyond 6 standard deviations above the mean, or 16 of every 100,000 events.  That's not frequent, but it's a far cry from never!  To use our weather analogy one last time (I promise!), that's an incredibly extreme temperature once every 17 years.  Or it's like a "storm of the century" that happens 5 or 6 times a century.

What's important to understand is that these very large and rare events are not outliers.  They are valid, real events that are an inherent characteristic of these exponential distributions.  They are not measurement errors or aberrations from what is expected.  They should not be dismissed but rather expected and planned for.

An Economist Discovers the Exponential World of Healthcare

I saw this play out in stark reality on a project a number of years ago.  I was part of a team doing data projections showing how the retiring baby boomers would affect the provincial healthcare system over 25 years.  One analysis involved predicting the demand for inpatient services.  A semi-retired and respected economics statistician on our team built a spreadsheet model for inpatient demand, crunched the numbers, and declared that inpatient days would go down over the coming decades as the baby boomers retired, and that the total inpatient costs would also drop.  The team leader was thrilled with the good news, but I was skeptical.  I had read a couple studies in peer reviewed journals that made the exact opposite conclusion, namely inpatient days would rise, average length of stay would rise, and costs would increase materially.

I asked the statistician for his model and data to review his analysis.  The data was fine and his model was sound, except for one minor step -- he ignored all of the data points above 3 standard deviations from the mean.  I asked him his reason for this omission of a significant portion of the dataset and his reply was, "Well, that's what I always do."  Obviously his entire career was spent using macro-economic data that was normally distributed, and he got into the habit of eliminating the outliers for every analysis without even thinking about it.  The problem was that one cannot do that with inpatient length-of-stay data, which follows an Erlang distribution.  This economist had used his "outlier magic wand" and made the sickest patients in the province instantly disappear!  And what do you know?  Hospital costs go down when you make the sickest patients disappear!  Unfortunately doctors and hospital administrators don't have that magic wand in their pockets and they know the sickest patients have to be treated, often at significant cost to the system.  Eliminating them from the projection made no sense.

I redid the analysis using all of the inpatient data and it agreed with the published studies, namely that inpatient days, average length-of-stay, and costs would all rise as baby boomers retired.  It took a lot of convincing of the team leader that my analysis was identical to the economist's analysis, except that I included the sick people!


When dealing with exponential distributions in the real world, remember that the tail end of the distribution drives the behaviour of the system.  Do not ignore the outliers.  They are likely real events, they will happen more frequently than you would like, they will be larger than you like, and if you ignore them you will get the wrong answer.

With exponential distributions, the tail wags the dog.

Postscript

After publishing my article this morning, I found an article by Carl Richards at Motley Fool who argues that outliers in normal distributions shouldn't be ignored either.  Seems to be the theme for the week!

Exponential Distributions, Part 1 - Expect More Bad Luck Than Good

In a recent blog post, I discussed how JPMorgan got burned by assuming derivatives behaved according to a normal distribution when in fact they followed an exponential distribution.  I want to delve a bit further into some of the counter-intuitive aspects of exponential distributions because I have found most people are not familiar with them.

Randomness

First it is important to understand that all of these distributions I am referring to deal with randomness.  When the next event cannot be predicted precisely, such as tomorrow's stock market close or the roll of the dice, it is an example of randomness.  However, if a population of related random events is studied as a group, the randomness will take on a particular form.  The histogram for rolling a single die will be flat, with each value from 1 through 6 having an equal 1/6th chance of occurring.  Sometimes those histograms take the shape of a bell curve (normal distribution) or an exponential distribution or something completely different.  It just happens that normal and exponential curves appear a lot in the real world.

Erlang Distributions, shown to the right, are just one family out of many exponential curves.  The mathematics of Erlang functions can be viewed here, but it's not necessary to know the math in order to understand their behaviour.  Most queues in the real world follow an Erlang curve (think the line-up at the bank or how long you will be kept on hold by your company's help desk).  Exponential distributions come in many shapes and sizes, but they share some common attributes that differ significantly from the Normal Distribution.  These mathematical differences create real-world consequences that must be understood in order to manage them appropriately.

Exponential Attribute #1:  Assemetry
Or Expect More Bad Luck Than Good

Normal distributions are symmetrical about their mean, which implies random variation will occur equally above and below the average value.  If you equate randomness with luck, and one side of the mean as good and the other as bad, then you will get approximately equal amounts of good luck (above the mean) and bad luck (below the mean).

Exponential distributions however are lop-sided.  This is because there is a minimum value but no practical maximum, and because the mean is relatively close to the minimum.  For example, inpatient length of stay in a hospital cannot be less than 1 day but it can extend out to hundreds of days on rare occasions.  The duration of a telephone call cannot be less than zero seconds but it can extend into hours or even days if it is a computer modem that's making the call.

How is this important?  

It means unusual random events will almost ALWAYS be on the long side of the distribution.  You will not get enough randomly small events to balance out the randomly large events as you would expect in a normal distribution.  For instance, if shoe size in a population happens to follow a normal distribution, we would expect to find an unusually large set of feet (i.e. more than 3 sigma above the mean) on a rare occasion (about 1 out of every 370 people).  Similarly we would also expect to find an unusually small set of feet (i.e. less than 3 sigma below the mean) on an equally rare occasion.  If our sample size of the population is sufficient, those large and small outliers would balance each other out and our mean value would be unaffected.  Not so with exponential outliers.  When you get an inpatient who stays in a hospital ICU bed for 2 years (such as a spinal injury from a diving accident) you will never get an inpatient who stays -2 years to balance things out.    That one outlier event will significantly impact the average length of stay for that ICU and probably for the entire hospital.

Therefore, exponential distributions bring unbalanced randomness.  Warren Buffet found this to be true of the insurance industry, as he states in his 2005 Letter to Shareholders:
"One thing, though, we have learned – the hard way – after many years in the business: Surprises in insurance are far from symmetrical. You are lucky if you get one that is pleasant for every ten that go the other way."

JP Morgan learned this as well.  Days where derivative contracts lost big money were not balanced out by days where those same contracts earned big money.  The random behaviour of that system was severely unbalanced.  The bad luck exceeded the good luck.

Except it wasn't luck.  It was predictable randomness.

If your system's randomness follows an exponential curve, then you must plan for more bad luck than good.


In Part 2 we will look at another implication of this asymmetry, namely the large tail of an exponential distribution.

Warren Buffet's Investing Wisdom, Part 2

Today Warren Buffet issued his latest annual report and letter to shareholders. Continuing from Part 1, here are some more of my favourite nuggets of Warren Buffet's wisdom from his previous annual letters to shareholders.

• John Stumpf, CEO of Wells Fargo, aptly dissected the recent behavior of many lenders: “It is interesting that the industry has invented new ways to lose money when the old ways seemed to work just fine.” (2007)
• I’ve reluctantly discarded the notion of my continuing to manage the portfolio after my death – abandoning my hope to give new meaning to the term “thinking outside the box.” (2007)
• For me, Ronald Reagan had it right: “It’s probably true that hard work never killed anyone – but why take the chance?” (2006)
• Warning: It’s time to eat your broccoli – I am now going to talk about accounting matters. I owe this to those Berkshire shareholders who love reading about debits and credits. I hope both of you find this discussion helpful. All others can skip this section; there will be no quiz. (2006)
• As a wise friend told me long ago, “If you want to get a reputation as a good businessman, be sure to get into a good business.” (2006)
• Long ago, Mark Twain said: “A man who tries to carry a cat home by its tail will learn a lesson that can be learned in no other way.” If Twain were around now, he might try winding up a derivatives business. After a few days, he would opt for cats. (2005)
• When we finally wind up Gen Re Securities, my feelings about its departure will be akin to those expressed in a country song, “My wife ran away with my best friend, and I sure miss him a lot.” (2005)
• Comp committees should adopt the attitude of Hank Greenberg, the Detroit slugger and a boyhood hero of mine. Hank’s son, Steve, at one time was a player’s agent. Representing an outfielder in negotiations with a major league club, Steve sounded out his dad about the size of the signing bonus he should ask for. Hank, a true pay-for-performance guy, got straight to the point, “What did he hit last year?” 
• When Steve answered “.246,” Hank’s comeback was immediate: “Ask for a uniform.” (2005)
• Long ago, Sir Isaac Newton gave us three laws of motion, which were the work of genius. But Sir Isaac’s talents didn’t extend to investing: He lost a bundle in the South Sea Bubble, explaining later, “I can calculate the movement of the stars, but not the madness of men.” If he had not been traumatized by this loss, Sir Isaac might well have gone on to discover the Fourth Law of Motion: For investors as a whole, returns decrease as motion increases. (2005)
• R. C. Willey will soon open in Reno. Before making this commitment, Bill and Scott again asked for my advice. Initially, I was pretty puffed up about the fact that they were consulting me. But then it dawned on me that the opinion of someone who is always wrong has its own special utility to decision-makers. (2004)
• John Maynard Keynes said in his masterful The General Theory: “Worldly wisdom teaches that it is better for reputation to fail conventionally than to succeed unconventionally.” (Or, to put it in less elegant terms, lemmings as a class may be derided but never does an individual lemming get criticized.) (2004)
• Charlie and I detest taking even small risks unless we feel we are being adequately compensated for doing so. About as far as we will go down that path is to occasionally eat cottage cheese a day after the expiration date on the carton. (2003)
• [Regarding the annual meeting:] Charlie and I will answer questions until 3:30. We will tell you everything we know . . . and, at least in my case, more. (2003)
• Borsheim’s [Jewellery Store] operates on a gross margin that is fully twenty percentage points below that of its major rivals, so the more you buy, the more you save – at least that’s what my wife and daughter tell me. (Both were impressed early in life by the story of the boy who, after missing a street car, walked home and proudly announced that he had saved 5¢ by doing so. His father was irate: “Why didn’t you miss a cab and save 85¢?”) (2003)
• When I review the reserving errors that have been uncovered at General Re, a line from a country song seems apt: “I wish I didn’t know now what I didn’t know then.” (2002)
• We cherish cost-consciousness at Berkshire. Our model is the widow who went to the local newspaper to place an obituary notice. Told there was a 25-cents-a-word charge, she requested “Fred Brown died.” She was then informed there was a seven-word minimum. “Okay” the bereaved woman replied, “make it ‘Fred Brown died, golf clubs for sale’.” (2002)
• Bad terminology is the enemy of good thinking. When companies or investment professionals use terms such as "EBITDA" and "pro forma," they want you to unthinkingly accept concepts that are dangerously flawed. (In golf, my score is frequently below par on a pro forma basis: I have firm plans to "restructure" my putting stroke and therefore only count the swings I take before reaching the green.) (2001)

Warren Buffet's Investing Wisdom, Part 1

Around this time of year, Warren Buffet releases his shareholder letter in the annual report for his company Berkshire Hathaway.  For those not acquainted with Buffet, his writing style is much closer to the Farmer's Almanac than the Wall Street Journal.  It is far from the typical public relations hype that most corporations pile into their annual reports.

I remember the first time I discovered a Warren Buffet annual report.  I was rather surprised at the folksy anecdotes mixed in with razor-sharp astute financial insights.  More surprising to me was his brutal honesty about his failures and his modesty about his successes.  Humble billionaires are hard to come by.

His ability to understand and explain financial concepts is remarkable.  The best place to understand how insurance companies work is by reading his shareholder letters.  Because he built his empire on generating and investing "float" from his insurance ventures, he takes pains to have his shareholders understand what that concept means.

In his 2011 letter, his humility and insight is present as usual.  He admits his error on predicting a housing recovery in the U.S.

Last year, I told you that “a housing recovery will probably begin within a year or so.” I was dead wrong.

He then goes on to explain the two reasons why the housing market is guaranteed to recover:  hormones and in-laws.

Every day we are creating more households than housing units. People may postpone hitching up during uncertain times, but eventually hormones take over. And while “doubling-up” may be the initial reaction of some during a recession, living with
in-laws can quickly lose its allure.

Every other financial expert I have read makes economics sound complicated.  Warren makes economics sound profoundly simple.

Here is a collection of some of my favourite nuggets of Warren Buffet's wisdom from his annual letters to shareholders.

• I have made more than my share of mistakes buying small companies. Charlie long ago told me, “If something’s not worth doing at all, it’s not worth doing well,” and I should have listened harder. (2011) 
• Today, a wry comment that Wall Streeter Shelby Cullom Davis made long ago seems apt: “Bonds promoted as offering risk-free returns are now priced to deliver return-free risk.” (2011) 
• Cultures self-propagate. Winston Churchill once said, “You shape your houses and then they shape you.” That wisdom applies to businesses as well. Bureaucratic procedures beget more bureaucracy, and imperial corporate palaces induce imperious behavior. (As one wag put it, “You know you’re no longer CEO when you get in the back seat of your car and it doesn’t move.”) (2010) 
• Home ownership makes sense for most Americans, particularly at today’s lower prices and bargain interest rates. All things considered, the third best investment I ever made was the purchase of my home, though I would have made far more money had I instead rented and used the purchase money to buy stocks. (The two best investments were wedding rings.) For the $31,500 I paid for our house, my family and I gained 52 years of terrific memories with more to come. (2010) 
• As one investor said in 2009: “This is worse than divorce. I’ve lost half my net worth – and I still have my wife.” (2010) 
• Part of the appeal of Black-Scholes to auditors and regulators is that it produces a precise number. Charlie and I can’t supply one of those. Our inability to pinpoint a number doesn’t bother us: We would rather be approximately right than precisely wrong. (2010) 
• Unquestionably, some people have become very rich through the use of borrowed money. However, that’s also been a way to get very poor. When leverage works, it magnifies your gains. Your spouse thinks you’re clever, and your neighbors get envious. But leverage is addictive. Once having profited from its wonders, very few people retreat to more conservative practices. And as we all learned in third grade – and some relearned in 2008 – any series of positive numbers, however impressive the numbers may be, evaporates when multiplied by a single zero. (2010) 
• Long ago, Charlie laid out his strongest ambition: “All I want to know is where I’m going to die, so I’ll never go there.” (2009) 
• Long ago, Ben Graham taught me that “Price is what you pay; value is what you get.” Whether we’re talking about socks or stocks, I like buying quality merchandise when it is marked down. (2008) 
• As we view GEICO’s current opportunities, Tony and I feel like two hungry mosquitoes in a nudist camp. Juicy targets are everywhere. (2008) 
• Investors should be skeptical of history-based models. Constructed by a nerdy-sounding priesthood using esoteric terms such as beta, gamma, sigma and the like, these models tend to look impressive. Too often, though, investors forget to examine the assumptions behind the symbols. Our advice: Beware of geeks bearing formulas. (2008) 

More to come in Part 2.

JPMorgan Learns About Exponential Distributions The Hard Way

Most people are familiar with the normal distribution, more commonly known as the bell curve and technically known as the Gaussian distribution.  Anyone who has had a teacher who marks on the bell curve is familiar with the concept.  It is a histogram in the shape of a bell, with most of the values near the mean and with equal portions on either side of the it, diminishing as you get further from the mean.

However, it seems the smartest minds on Wall Street assumed EVERYTHING in the world of finance follows the bell curve.  BIG mistake!  As it turns out, for JPMorgan it was a $6.2 billion big mistake.  Ooops.  You can read about their mistake here on Slate.com.  First, let me explain a few of the concepts involved and then we'll come back to J. P. Morgan's incredibly wrong assumption.

What is a Bell Curve?

The bell curve below is from mathisfun.com.  It shows that within 3 standard deviations (+/- 3 sigma) of the mean of a normal distribution, you will find 99.7% of the observations.  In other words, there just aren't a lot of events that occur very far from the mean.  The process improvement method Six Sigma takes its name from this concept that if you go out +/- six standard deviations from the mean, you should effectively never get any events occurring outside of this range.  That is only true of course if the variations in the manufacturing process follow a normal distribution, which fortunately is usually true.

The bell curve accurately describes variations in student marks and student heights, many manufacturing processes, and even the daily movements in the stock market, but it doesn't apply to everything we find in the real world.  For example, simple everyday events such as wait times at your grocery store checkout or bank teller, hospital inpatient length of stay, and the duration of telephone calls do not follow a bell curve.  These events follow an exponential distribution.

What Is An Exponential Distribution?

Exponential distributions are asymmetrical (i.e. skewed to one side of the mean), limited on one side by a minimum (usually zero), and have long tails.  In other words, events far from the mean can and do happen with much more frequency than in normal distributions.

My graph on the left shows one type of exponential family of curves called the Erlang Distributions, named after a Danish telephone engineer A. K. Erlang who began using them in the early 1900's.  These in turn are part of a larger family of exponential functions that engineers call Gamma Functions.

What Erlang discovered is that duration of telephone calls did not follow a normal distribution.  You could not have a call of zero duration or less (i.e. a minimum), but you could have an occasional telephone call that lasted hours (i.e. no practical maximum).  Erlang's job was to accurately predict how much switchboard capacity was required, which he ultimately succeeded in doing.

Because of these occasional large-value events, the rules of normal distributions do not apply.  For instance, you cannot presume that 99.7% of your events will fall within 3 sigma of the mean.  Depending on the specific shape of the exponential distribution, a measurable and significant portion of the events will fall far past 3 standard deviations from the mean, and Erlang had to take those events into account when planning his capacity.

JPMorgan's Mistake

As stated in the article, JPMorgan got into the business of complex credit swaps and assumed they would behave according to a normal distribution.

"Credit default swaps simply don’t behave in line with the normal, or Gaussian, distribution typically assumed. The so-called tail risks, or the chances of extreme events, are bigger than that theory predicts. Ina Drew, who ran the CIO, referred to one day’s mark-to-market losses as an eight standard deviation event, according to the report. That translates mathematically into something that should only happen once every several trillion years. It makes no more sense than Goldman Sachs finance chief David Viniar’s famous remark as the crisis unfolded in 2007 about seeing 25 standard deviation events, several days in a row."

Of course, the mathematics DOES predict this if you use the correct exponential distribution.  The credit swaps had very large potentials for one-day losses, which veer far away from the daily means, but those potentials were either ignored or just presumed to never exist.  JPMorgan failed to recognize the correct distribution for their credit default products and when losses mounted they just blamed mathematics for their wrong assumption.

One cannot simply ignore the rare-but-large-value events.  JPMorgan learned the hard way that with exponential distributions, the tail wags the dog.

And so JPMorgan shareholders are out $6.2 billion for that little mathematics oversight.  Oops.

The Dangers of Measuring Reality - Part 2

This is a continuation of Part 1 where I began looking at Ron Baker's article on his Seven Moral Hazards of Measurements.  I am converting Ron's hazards into some Dangers of Measuring Reality that can trip you up if you're not careful.  We pick up with Ron's Hazard #4.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #4: Measures Are Unreliable
This is Ron's hazard that I disagree with the most.

Ron's example here is that GDP increases when a couple get divorced but decreases when a child is born.  Because the measure increases when something "bad" happens (divorce) and decreases when something "good" happens (birth), the measure therefore is unreliable.  Ron's reasoning here gets a bit ... well, ridiculous.

Measurement unreliability has nothing to do with moral projections that one forces onto it.  Unreliability has to do with the consistency and accuracy of data collection, summarization, and presentation of a measurement.  It has nothing to do with a person's presumption that all "good" things should make measures go up and all "bad" things should make measures go down.

Using the earlier example of measuring the room temperature, is an increase in temperature good or bad?  Depending on the starting temperature and the number of degrees it has increased, it's hard to say. In fact, half the people in the room might think it's good and half might think it's bad.  Who is right?  It's a completely subjective evaluation.  When the outside temperature drops in January so the rain turns to snow, the snow boarders are happy and the driveway shovellers are unhappy.  It all depends on one's perspective.

GDP is not defined to measure the sum of all "good" and "bad" things that happen in a nation.  It measures the total economic output, and when calculated per capita, it will go up or down whenever the population or the economic output changes.  If it does that accurately, then it is a reliable measurement.

Darren's Danger #4:  Do Not Impose Meaning On A Measurement That Does Not Exist

With the exception of a few measurements (such as crime rates), most metrics are amoral.  In other words, changes in the metrics may be good or bad depending on the situation or one's perspective.  Imposing morality on measures will result in them being used inappropriately, or as Ron seems to suggest, being thrown out altogether.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #5: The More We Measure The Less We Can Compare

Surgeon's death rates are the example for this hazard.  Because patient death rates vary by the complexity of the patient's condition, simple death rates by surgeon can be misleading.  Therefore, death rates are ALWAYS adjusted for risk so they are comparable across different patient populations.

Ron's example contradicts his point though.  The more we measure, the more easily we can compare measures.  Without measuring both surgeons' death rates AND patient complexity, we cannot reliably compare surgeons.  The first measure is interesting, but not useful by itself.

Darren's Danger #5:  Do Not Compare Apples and Oranges In A Single Measure
If you absolutely must compare apples and oranges, make sure you convert them into an equivalent measurement before you draw up your comparison.  Surgeons' raw death rates are like comparing apples and oranges, but once you adjust for different levels of patient risk, those risk-adjusted death rates by surgeon become comparable -- apples to apples.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #6:  The More Intellectual the Capital, the Less You Can Measure It
I happen to agree completely with Ron on this one.  There are some things that are just impossible to measure, but are very important nonetheless.  As I quoted at the beginning of Part 1, not everything that counts can be counted.  The knowledge and abilities of the people in an organization are incredibly important to the future value of the company, but it is very difficult to quantify those assets hidden inside an individual's brain.

And so I will state my danger as an alternate way of saying what Ron has said:

Darren's Danger #6:  Do Not Forget The Important Things That Cannot Be Measured
The list of measurements published every month in your management report do not comprise all of the important things you must manage.  It is important to remember the un-measurable aspects of your business too.

~~~~~~~~~~~~~~~~~~~~

Ron's Hazard #7:  Measures Are Lagging
Ron is correct that most measures are snapshots of history, like driving by looking only in your rear-view mirror.  He is correct, but that reality is quickly changing.

Most website hosts now provide real-time tracking of your website visitors, and large data warehouses are beginning to provide near-real-time reporting on some high-priority areas.  For instance, I worked for a banking client that was implementing near-real-time reporting of credit card transaction data to identify fraudulent activity within minutes.  It is not easy to do, but it is being done.  Just because traditionally reporting has been slow and retrospective does not mean all reporting is that way.

On the other hand, a lot of processes do not require real-time reporting because they simply do not move very fast.  Knowing how your customer service process performed last month is probably still useful to direct your efforts to improve your process this month.  Just because the data is not immediately current does not mean is it without value.  Timeliness is always dependent on the purpose of the measurement.

Darren's Danger #7:  Data Without a Timestamp Is Probably Older Than You Think
Always, always, always include a date and time when a data record was captured.  Never presume it's current unless it states it's current.

The Dangers of Measuring Reality - Part 1

Obtaining data is the first step towards improving any process.  Our English word "data" comes from the Latin datum meaning "something given."  All data must connect back to something in the real world, which gives that data some meaning or usefulness.

However, measuring things in the real world is not always a simple exercise.

Jason Goto, a former colleague of mine, points this out in his blog titled "Are you reporting what you can? ... Or reporting what you should?"  Many companies collect lots of data, but it's not always the data they should be collecting.  The most important measures are often the most difficult to collect.  As Einstein famously said (or maybe it was William Bruce Cameron), "Not everything that counts can be counted, and not everything that can be counted counts."

I recently found Ron Baker's article on his Seven Moral Hazards of Measurements on LinkedIn.  I believe his main point is that we need to recognize the limits of measurements, which is true.  While his article is thought provoking, I beg to differ with most of his hazards.  I'd like to convert them into some Dangers of Measuring Reality that can trip you up if you're not careful.


~~~~~~~~~~~~~~~~~~~~


Ron's Hazard #1: We Can Count Consumers, But Not Individuals
Aggregate data is not the same as individual data records, but that's OK.  It's not supposed to be the same. Aggregate measures answer questions about groups or populations.

Ron's example of room temperature not equating to how many individuals feel hot or cold is an example of misusing a measurement for an unrelated purpose.  The room temperature is measured using a thermometer on the wall and it answers the question, "How warm is the room?"  To answer Ron's question, you need to take a very different form of measurement, probably a survey question that says, "Do you feel warm or cold right now?"  Counting the responses to that question will permit you to know whether to raise or lower the thermostat in that room.  We required a different measurement to answer a different question.

Darren's Danger #1:  Do Not Force Measurements To Answer Unrelated Questions
All measures and indicators are created to answer a specific question.  Know what question a measure is intended to answer, and don't force it to answer other questions.

~~~~~~~~~~~~~~~~~~~~


Ron's Hazard #2:  You Change What You Measure
Heisenberg's Uncertainty Principle applies to the realm of the exceedingly small, such as shining a light on an individual atom.  When a photon hits the atom, the atom is moved as a result of the collision and you no longer know that atom's location.  In that case, the act of measuring truly does change reality.

However, in the big world that we live in, the act of measuring rarely changes the object that is measured.  When I step up on a bathroom scale to measure my weight, the act of stepping on the scale might burn a small fraction of a calorie and thus reduce my weight by a minuscule amount.  However, the accuracy of the scale cannot detect that change.  Unfortunately I'm just as heavy now as I was before I stepped on the scale.

In fact, when we want to intentionally change a process for the better, simply measuring it will not produce any change.  From my experience, just measuring and reporting on the process rarely improves it.  It is not until some kind of incentive is linked to the measure (like the manager's bonus) that the process will start to  change for the better.

Ron's point with this hazard is specifically related to performance targets, and how people will try to manipulate them to their advantage.  That is a real problem, but it is not solved by giving up on measuring.  It is solved by setting targets and incentives that are not easily gamed.

Darren's Danger #2:  Do Not Set Targets That Are Easily Manipulated
Setting targets is fine, but not if the measurement can be manipulated.  Incentives must be aligned with measures that are clearly defined and objectively determined.

When I worked in healthcare, a lot of measures were considered to measure hospital efficiency.  Many indicators were ultimately rejected because it was possible to show improvement in the indicator by a means other than improving the hospital's efficiency, such as simply moving budget from one hospital department to another.  Measures that are vaguely defined or simply "the best data we've got" will seldom make good candidates as target indicators.

When I worked in retail, there was one director who spent a lot of his time arguing that the definition of an efficiency metric should be changed because occasional random events reflected poorly on his performance (and thus his annual bonus).  He conveniently ignored the other random events that showed an improvement on his performance!  Whenever a target is set that affects one's pay, some people will spend more time trying to manipulate the metric than actually improving the business process itself.  Performance metrics must be completely objective and airtight.


~~~~~~~~~~~~~~~~~~~~



Ron's Hazard #3:  Measures Crowd Out Intuition and Insight
I think Ron's point on this hazard is fair, in that entrenched metrics can lull managers into a thinking there are no other important indicators or other areas to look for improvement.  The traditional measures can act to stifle creativity and problem-solving skills that are still desperately needed in the company.  This problem tends to occur when a reporting system reaches a level of maturity and acceptance in an organization.

However, the opposite is also true.  Bringing new measures to a problem can often generate new perspectives and focus on problem solving, especially for large and complex processes.  When I led a process improvement effort on a blood laboratory, the problems and process was just too large to try and fix everything at once.  Introducing new measures quickly showed where the process bottlenecks were and allowed problem-solving to focus on those areas that would show the most improvement with the least effort.  In that case, measures stimulated insight and creative problem-solving.

Darren's Danger #3:  Do Not Stop Improving Your Metrics
Once you have implemented some metrics in your organization, do not get lulled into the deception that you are done.  Keep improving, keep adding metrics, and have the courage to remove metrics when their usefulness has passed.


~~~~~~~~~~~~~~~~~~~~



To be continued in Part 2.