Visualizing Your Network via Gmail

An interesting visualization tool has been created by a team at MIT called Immersion.  It maps your network of contacts by analyzing the metadata in your Gmail account.  It doesn't read your emails per se, but just some of the metadata.  (Coincidentally, that's what the NSA is reading from your emails as well!)

See the sample image below from their demo.

I have about 7 years worth of emails in my Gmail account, so it provides an interesting picture of my contacts.  I don't use Gmail for all of my business communications -- once I'm on contract with a client, they usually issue me a company email account -- however, it still gives a clear picture of what parts of my network are connected and which clusters are distinct.

Not sure I can do anything useful with this knowledge, but it was an interesting visualization to see.  At least it made it worth keeping all those old emails that I felt I should've cleaned up long ago.

Procrastination has its benefits!

Data Presentation Is Always Subjective

One of the endearing qualities of data is that it is factual.  Facts are a piece of truth and are not simply a matter of one's opinion.  Data can be misrepresented, incomplete, or erroneous, but accurate data is always objective.

At least that's what I believed in my early days of working with data.  The reality is that any presentation of data is subjective, and therefore the objectiveness of the underlying raw data becomes less significant.  One can avoid subjectivity by never presenting the data, but that also assures the data remains useless.

Jake Porway has an interesting blog article at Harvard Business Review entitled The Chart Wars Have Begun.  He refers to an interactive chart in The New York Times that shows how job data and the unemployment rate can be presented three ways:  the "normal" way, through a Democrat lens and through a Republican lens.  None of the views are dishonest or misleading, they are just choices that must be made for any presentation of data.  It's a simple but excellent demonstration of the power of data presentation.

Screen capture of the NY Times chart mentioned above.  Click here for the interactive version.

Every data chart or table requires the creator to make decisions that include:

• what minimum and maximum values should be shown on the y-axis.
• how much historical data should be displayed.
• what scale should be used.  (i.e. linear or logarithmic)
• whether multiple data sets should be displayed together.
• the type of chart (e.g. pie, bar, line, scatter) or the organization of the table (e.g. which fields are displayed in which order).
• whether certain information should be highlighted, such as an arrow showing a trend or an arbitrary baseline.

Each of these decisions will lead the reader to realize or overlook certain conclusions from the data.  The data itself may be truth, but the presentation will always be dependent on the subjective choices of the presenter.

Medians, Distributions, and Cancer Statistics

I found a fascinating essay archived  on cancerguide.org by evolutionary biologist Stephen Jay Gould entitled, "The Median Isn't The Message."  It's a story about how cold, stark cancer statistics became intensely personal after Gould was diagnosed with a rare form of cancer in 1982.

Having spent years myself as a data jockey in the healthcare system, including in a cancer agency, I am familiar with those cold statistics.  I had many of them memorized for easy retrieval.  Whenever I heard about an acquaintance who had been diagnosed with cancer, I would immediately determine in my head the likelihood of treatment success.  It was not the kind of information that was particularly helpful to the patient, and I quickly learned to keep that information inside my head!

Gould's cancer had a median survival of 8 months from diagnosis -- not very good odds at all.  Half of these cancer patients survived LESS than 8 months!  Being a biologist familiar with statistics, he forced himself away from the reflex conclusion that he had about 8 months to live.  The important information was the shape of the survival distribution curve, not in the median value.

As with most healthcare distributions, cancer survival is an exponential curve with a long tail to the right, meaning a few people survive much longer than the median.  The second consideration was to determine what factors classified those patients into subgroups within that distribution.  Age was one, and since Gould was relatively young, his odds at beating the median survival were better than average.

In Gould's case, a new experimental treatment turned out to separate him from the rest of the distribution and he survived for another 20 years.  When he died in 2002, it was from an apparently unrelated form of cancer.

The main point of the article is to always understand your data before you draw conclusions from it, and the best way to understand data is to look at it visually.  I always told my analysts to start with a graph of the dataset and then focus the analysis on the interesting parts.  Simply starting with averages, medians, minimums, maximums, and even fancy statistical measures like correlation coefficients will not tell you enough.

That's where Business Intelligence and Data Mining tools really make life easy.    Visualizing the data will convey much more useful information than a table of statistics.  In BI, a picture is definitely worth a thousand words!

Statistician Francis Anscombe demonstrated this principle elegantly in 1973 by creating four datasets with very different distribution shapes, but almost identical statistical properties.  They are known as Anscombe's Quartet. 

The datasets all have the same average values for x, same average values for y, and the same variances for each.  In a normal or bell curve distribution, you can think of the variance as a measure of the width of the bell.  In a cluster of data points, variance can be understood as average minimum distance from the best-fit line running through them.  The Anscombe datasets also have the same linear regression result (i.e. the best-fit line through the data points) and the same correlation values.  Correlation is a measure of how much the x- and y-values move together.  A high positive correlation means that when x increases, so does y.

In Anscombe's Quartet, the first dataset is a textbook example of how these statistical functions are supposed to work.  The variance, regression line, and correlation are all accurate descriptions of the behaviour of this group of data points.

However, the third dataset is skewed by a single outlier.  The slope of the regression line is off, and the correlation should be a perfect 1 if the outlier is excluded.   Graphing the data makes outliers very obvious.  Sometimes outliers are measurement errors and should be excluded, but if there are many outliers it suggests there are two different datasets being displayed together.

The second dataset shows the bias of incomplete measurement.  The data is clearly curving, so a linear regression is arbitrary in that it would differ greatly if we made one more measurement, and it would differ again if we made two more measurements.  The x- and y-values are related to each other, but in a completely non-linear way.  Regression should not be considered in this case as it is meaningless.

The fourth dataset shows a completely uncorrelated, non-linear dataset with a single outlier, yet the statistical measures still generate the same results as the first dataset suggesting a high degree of correlation and linearity.  This is a stellar example of how statistical measures can be completely erroneous if used blindly.

The Anscombe Quartet highlights the frailty of some statistical measures when you leave the realm of linear or normal distributions.  Bell curves are nice, but they don't exist everywhere in the real world.  If you assume bell curves or linear relationships without verifying them first, your analytical conclusions will be way off base.