Context-sensitive "feelings"

The difference between a human and a computer evaluating a situation is that the human relies on prior intuition on how the situation has developed. Can we leverage this to do better data analysis?

Two kinds of data analysis

Based on my dozen plus years’ experience in data science, I think there are broadly two ways in which people approach data analysis.

The first is the “procedural school”, or what I’d uncharitably call the “boot camp school”. Here, data analysis is a series of steps to be performed.

To illustrate with an example, in my last job at Delhivery, hiring started with a take-home assignment process. The assignment would typically be of the sort of “take this data set and tell a story based on it in not more than six slides”. And in over 90% of the responses (none of whom really progressed in the application process), it was a rather procedural process.

The first four of the six slides would be dedicated to “data cleaning” (not the best prioritisation, IMHO), with a series of procedures - fill in missing values, remove outliers and the like. The fifth slide would be dedicated to “performing EDA” (I learnt through my evaluations that EDA = Exploratory Data Analysis), which again was a small set of procedures. And in the sixth slide, there would be an XGBoost model fit on something.

As you might have figured out in the language in the last three paragraphs, I’m not a fan of this kind of data analysis. What I’m partial to is what I would call as a more “feelings-based” approach. Where you look at the data and inspect it; get a “feel” for it; figure out what you want to do based on what the data looks like. Actually I’ve written about this earlier in this blog, in the post on “observing data”.

AlphaZero vs Stockfish

What we are looking to do at Babbage Insight is to build a system that can analyse data by observing it. An analogy that I like to use (and one which I’ve unleashed on a bunch of potential investors) is how the world of computer chess changed with the coming of AlphaZero in 2017.

Prior to that, chess programs were all about brute force and computing power. They would evaluate all possible lines (with reasonable modifications, such as “branch and bound”) for a very large number of moves, and then evaluate the position there (typically using simple heuristics such as pawn counts) in order to pick what is the best move now.

Of course, humans got influenced by the computers then and this led to some interesting “computer moves”, one of which was the comeback of the Berlin Defence (popularised by Kramnik in his match against Kasparov).

AlphaZero approached the game completely, in terms of the way it learnt. The model was based on deep reinforcement learning, where the system played millions of games against itself. And then it built itself a deep learning model correlating each intermediate position with the ultimate outcome of the game (I’m oversimplifying here), which it used to evaluate the intermediate positions.

So what AlphaZero had, through its deep reinforcement learning process, was a model that could get a “feel” for a position. It was amazing how with no prior knowledge (unlike StockFish, AlphaZero wasn’t initialised with knowledge of human games - all it knew were the rules), it rediscovered several well-known chess openings. One of the first reactions to AlphaZero was how “human” its game play was (apart from its penchant for playing rook pawn moves - something that’s now permeated to top level human chess nowadays).

Drawing the analogy, what we are looking to do here is - just like AlphaZero can get a “feeling” for a game position, we want Babbage to get a “feeling” for the insight in the data by just looking at the data.

For example, I put out this tweet yesterday:

The moment I saw these graphs, I got one big message from it - which is that Manchester United made no attempts on goal (hence zero xG) after scoring their third goal. The question, as I’ve put in the follow-up tweet, is how Babbage Insight can be trained to get similar insights by just observing the data.

(we’ll be starting to hire soon, so if this is the kind of problem that might excite you, hit me up)

Context-sensitive feelings

Now coming to the tricky part. This morning I was watching Daniel King’s analysis of the first round of the Candidates tournament (he had picked the game between Americans Fabiano Caruana and Hikaru Nakamura), and was thinking about AlphaZero’s “feelings”. I was trying to get my own feelings on the position and predict the moves, and I was happy to get them right a few times (I’d played competitive chess in 1994 and 1995).

After describing this entire game, King switched to a position from the Firouzja-Praggnanandhaa game, and suddenly I was lost. All through the game that I saw in full, it was easy for me to get intuition on the positions, and figure out the possible set of next moves. Now suddenly, faced with a mid-game position on a game that didn’t have context of, I was lost, and had no clue what was happening there.

That, I realised, is one big difference between how humans and machines (such as AlphaZero) evaluate. Human intuition is context-sensitive. As you play the game, you start developing an idea on all the possible position patterns that this game might throw up, and are able to evaluate all of them intuitively. When you are given a random pattern from another game, it is suddenly “out of syllabus” and it throws you off.

AlphaZero won’t have any problem with this, though - because all it is trained for is to take a position and evaluate its goodness. So whether the position is similar to what was seen before in the game, or is completely new, doesn’t matter to it, and it simply applies its model.

Training Babbage

As we build out Babbage, at this stage of the company, we won’t be able to afford the kind of resources that Deepmind had when they were training AlphaZero. One way in which we’re mitigating this is by using a pre-trained model (an LLM) and building on top of those. Like I keep telling my potential investors - an LLM is a “good bad data scientist”, and what we need to do is to provide it an executive function layer to know what kind of analysis to do.

As I was writing this post (one reason I write so much is because I think heavily while writing), I started wondering if another way to cut the search space is through data familiarity. The first use case we are planning for Babbage is to give “exceptional insights” to business decision-makers.

Now that I think of it, this basically involves looking at the same data set, but at different points in time (when new data would have got generated in the meantime). You can think of this as a chess game - where, as you play the game, the search space of positions that can occur narrows.

Similarly, in the business data context, if our system has been seeing a few particular sets of data in a customer’s business repeatedly, shouldn’t it be able to have better intuition on these particular data sets as new data points come in? This is yet another way in which we can make Babbage more relevant even if the resources were to be limited!

(we’ll be starting to hire soon, so if this is the kind of problem that might excite you, hit me up)