Dashboards are fundamentally backward-looking tools
They help you identify patterns that have already occurred in the past, and are next to useless when it comes to novel patterns
For the uninitiated, I have the habit of putting throwaway lines on social media (LinkedIn / Facebook / Twitter) and then later expanding them into blogposts. This is one of them.
As the more regular of readers will know, I spent December and parts of January interviewing people on how they use dashboards in their organisations, and how they make decisions using data.
There is one aspect of dashboards that I’ve not discussed (directly) before - that they are essentially backward looking instruments. And this applies even to dashboards that are built off live data.
The problem is this - dashboards primarily identify patterns that have been seen earlier.
Let’s say some metric you are tracking in your dashboard has moved adversely. Maybe sales are down. As a good business manager, you immediately wonder why, and look through the rest of the dashboard to find the reasons.
Digging deeper
The dashboard has some neat “cuts” - sales by product, sales by region, etc. The reason these particular cuts exist is primarily because there was a time in the past when sales had moved adversely, and some bespoke analysis had shown these as possible reasons for sales being down.
Now, as long as these patterns repeat (and in a lot of cases, in business, they do), you are good. So if sales are down because sales of a particular product are down, your dashboard will immediately help you catch it.
However, what if the sales are down secularly across all the components of a cut? You look at sales by region, and see that all regions have been (approximately equally) adversely affected. You look at sales by product type, and again you see that all product types have been adversely affected. And so on.
Soon you will realise that the dashboard doesn’t tell you the reason, and you call your head of analytics and commission a bespoke analysis. This analysis returns something new - let’s say a particular segment of customers (let’s say “students”) have suddenly started buying less, and because this is spread across product types and regions, your dashboard couldn’t catch it.
What do you do now? You call back the analytics team and ask them to add “sales by customer segment” as a further “cut” in your dashboard. They are not particularly happy, because this freezes the way customers can be segmented, but they add it to the dashboard anyway.
Instrumental variables (or, correlation != causation)
Sometimes, missing a signal is better than reading the wrong signal. The problem with conventional dashboards is that because you are only tracking previously seen signals, you might make the wrong interpretation.
Let us take the above example yet again. You are tracking sales on your dashboard, primarily by region and product type. Sales go down. You look into the dashboard and find that a particular product type is responsible for 80% of the decrease in sales.
You think there is something wrong with this product type, and then do something to fix it - maybe you slap on a promotional offer for it, or increase its marketing, or explore if there are any quality issues.
However, unprompted by you, your analytics team has noticed the same thing and done some (laborious) analysis, to conclude that there is nothing fundamentally wrong with the product type, but because sales along a particular segment (let’s say “students” once again) went down, this product type (that is popular with that segment) has also gone down.
In other words, the right “cut” to look at would have been customer type, but that has so far not been there in the dashboard, and so there was a very big chance that you would have taken the wrong decision on this, had it not been for the keen eye of your analytics team.
Then again, when you have so many metrics to track, it is not humanly possible to do bespoke analysis on each one.
Boiling the ocean
One way in which you can avoid missing out on insights is by putting all possible cuts in the dashboard. This month, you found a significant variation by customer segment, and you added that to the dashboard. Next month, you found a significant variation by product price band, and you add that as well. Soon, “everything will be there on the dashboard”.
Or as Balaji says here, “dump the entire dataset into the dashboard”.
One obvious issue with this is that dashboards can become incredibly slow. During one of my interviews last month, someone clicked on a dashboard, and it took over a minute to load. And the issue with that is that by the time the dashboard loads, you lose patience, go over to do something else and miss the insight altogether.
While this can be solved with clever data engineering and caching, there is another (bigger) problem with dashboard bloat - when you are tracking too many things on your dashboard, there is an information overload for the viewers, and even if there is insight somewhere, there is a good chance that it might be missed.
So if you put too many “cuts” into the dashboard, you run into the information overload / slowness problem. You put too few, and you run the risk of drawing the wrong insights from it. Is there a solution to this?
Have you faced this problem in your job, where the dashboard either has too much or too little information? How have you proceeded to solve it? I’d love to know more! And - do you actually try to use dashboards to understand why something happened, or do you just default to using human analysts for that?