The "industrial revolution" of data insights
The data insights industry is going through its own industrial revolution. And Babbage Insight is part of that!
Panic Attack
Last Monday, I had a minor panic attack, the kind that most founders periodically have. I started having existential questions regarding Babbage Insight - that it’s been close to a year since we started, and all we have is a v0.1. That I’m not doing enough to acquire more customers. That I’m not meeting enough people. That what we are building may not be that spectacular at all. And all such.
Inevitably, I started imagining the scenario that in a few months’ time, we might make the decision to shut down Babbage Insight, having failed to get any traction. And I panicked even more.
Now, in case of all panic attacks, the best (and counterintuitive) thing to do is to continue imagining and take the story further forward. So I started thinking of what might happen if we had to shut down Babbage Insight.
Painkiller
What would I do? Take a job, I guess! What job would I take? Most likely, a Head of Analytics job somewhere. And then that job would involve, among other things, me supervising other human beings building dashboards. Actual human beings building dashboards frame by frame, using tools like Tableau or PowerBI, painstakingly dragging and dropping each component into place, getting the formatting just right so that the leadership can hopefully “self-serve”.
The job would also involve “leadership meetings” where my CXOs would ask me questions on certain business metrics, and why some things happened the way they did. They would all be entirely predictable questions, but I’d wait to react to these questions rather than preempting them (assuming I take over an already existing team, I’ll have to largely continue with their ways of working).
That’s not something that I look forward to. To be honest, this is a world that I’m actively working hard to avoid - if I make Babbage Insight work, it will mean that I, along with all other Heads of Analytics, would be spared of the task of building dashboards manually, and of dropping everything we are doing just to answer the question that just dropped from the CXO’s desk.
Babbage will anticipate what questions might be asked, and preemptively answer them. You only need to tell it the basic metrics you are tracking (and the queries needed for tracking them), and you’ll get back stories.
Some Heads Are Gonna Roll
Recently, someone sent me this blogpost on different use cases of Generative AI. The first chart, albeit not very well made (in my opinion - you don’t want to put two quantities with different units on the same bar chart (even if it’s occasionally okay to do it in a line chart)), is instructive:
Basically data analytics is a highly under-funded use case of generative AI, if you look at the total money raised and the “number of companies that have a project”. However, it is not that people are not trying:
There are plenty of people (apart from us) building in this space, trying to deliver “more automated actionable insights”, in whatever shape or form. Many are going the chatbot route (thanks to the interface that made generative AI big - and that also means that whoever we talk to asks us for a chatbot (!!!) ), but there are various solutions being built.
All Guns Blazing
As I continued to ruminate on this, I got reminded by a vacation I’d taken a decade ago to an arts festival in Kutch (Gujarat). This was a festival promoting local artisans, and they proudly showed us off how they are preserving handlooms, pottery and other traditional forms of craftwork.
Unfortunately, I only saw inefficiency. “This is so laborious and boring”, was my constant instinctive thought, and I started wondering how the looms and spindles and wheels might be automated. And then I realised this was precisely what the industrial revolution had achieved - using energy from steam to automate the laborious process.
When I see people building dashboards by hand, I feel the same way. It is an insanely laborious process (at some point during my last job, where my nominal title was “SVP of BI”, I decided it might be a good idea to make dashboards using QuickSight. It was so painful that we abandoned the effort in a couple of days).
It is the same with writing long and complicated SQL queries in response to specific business questions - SQL in my opinion is impossible to debug (I strongly prefer more structured formats like dbplyr or PySpark), and laborious to write (I sometimes liken writing SQL queries to playing blindfold chess).
One Shot at Glory
There is no reason that to deliver data insights to someone else without repeated manual effort, you need to build dashboards the way they are built nowadays. There is no reason that you laboriously plan out the precise metrics you want to show, what kind of graphs to show them in, what order to show them in, how much space to allocate to each, what to hide under filters and what to show, and all such.
Given that people hardly use dashboards anyway, it just seems like a lot of work for not much reward at all. And when the business changes, or priorities change, the effort needs to be repeated all over again (though I understand the likes of Microsoft are offering copilot tools so you can use their text-to-SQL engines to build the dashboards).
There is also no reason that you wait for the business users to come to you with specific requests before you do some investigative analysis into the data - if you look at the data carefully enough (or have an AI do that for you), you will know what questions will be asked, and a large part of answering those can be done by algorithms itself, given the tech available to us today.
Going back to the analogy, what is happening in AI-driven data analytics nowadays (largely in startups, and I’ll put ourselves as part of this) is nothing short of revolutionary. And if I can extrapolate only just a little bit, I can say that (assuming we execute well) Babbage Insight has an opportunity to be part of the industrial revolution in data insights. This is our one shot at glory!
Last Monday, as I thought through this, the panic attack disappeared. And I got back to work!
Beyond the Realms of Death
It’s been a few hundred years since the industrial revolution took place, and demand still exists for handwoven cloth. There are artisans who supply that. Similarly, I see a niche demand, driven by particularly detail-oriented managers, for manual dashboards and business insights. And you will have a small community of dashboard builders painstakingly crafting the dashboards by hand, using tools such as PowerBI or Tableau.
Like handloom cloth, this will be an indulgence, a premium experience. Most companies will realise they don’t really need this, and AI-driven data insights can give them a lot of what they want.
The way people get insights from data is going to change significantly, and very soon. I’m feeling happy that Babbage Insight is going to be part of this change! Now to get back to executing..
I recently met with several consulting firms offering Gen AI-driven BI solutions. Most of them were essentially GPT wrappers, where you ask questions and receive summaries. Babbage 0.1 from your blog seemed quite novel in comparison.
I wonder if the new post Revolution data analytics would be an integral part of what we today call as SaAS software. Because it seems like if we have to fundamentally change how data is looked at and how insights are used it is going to come packaged with Revolution and how data records are maintained stored processed and maybe even the tools which capture this data on an ongoing basis.