“Context layer” was a new term that I learnt yesterday (one of the big problems with LLMs is that jargon gets produced well-at-a-faster-rate, making it impossible to keep up; and you won’t even know which jargon will be relevant a few months down the line!).
And having come across the concept, the more I think about it, the more I think that it is actually the most critical component of building a data chatbot.
Funnily enough, the writer of the blogpost where I learnt about context layers himself admits that this is not a well-defined concept, and then goes about disambiguating stuff before explaining what the context layer is all about. You can find his whole post here, and it’s fairly insightful:
The title is misleading and possibly clickbait - context layers have absolutely NOTHING to do with semantic layers, which are artefacts of data warehouses that allow people unfamiliar with the data and its structure to query and make sense of it.
Basically, the context layer is a part of a chatbot or agentic workflow, and in response to a user question, surfaces all the context necessary to answer it.
In my last post on this topic, I had written:
Basically, given a query, the first part of the problem will be to figure out what metrics we need to get the data for, and at what granularity and period. This is going to be a search problem that is going to be led by the documentation (possibly driven by wikis built on top of the documents).
This is precisely where the context layer comes in. The responsibility of the context layer is — (yes this is a human-put em-dash) given a query and who you are, surface all the necessary information and load it into the chat context so that the LLM knows where to search and how to help you.
Going back to Yali Sassoon’s post on context layers:
The job of a context layer is to give an agent the most relevant, precise, succinct yet complete picture of the situation it is in, together with the knowledge it needs to act, so it has the best possible chance of success, however the agent defines success. (”Succinct yet complete” is deliberately a tension: enough to act on and no more, because context is not free.)
Taking the example of a data chatbot, the first port of call after you get a user query is to build the context. Given this user and question, what information does the LLM need to answer it? This needs to include things like:
What set of metrics are necessary to answer this query, and at what filter or granularity or grouping?
How do these metrics come together to actually answer the user’s question?
If multiple definitions of a metric exist, which version is relevant (based on who this user is) here?
A pointer to either already existing queries for each metric (this is likely to come from simple or vectorised search over the query repository, rather than having to load all queries into an LLM, which will kill context anyway), or closely related queries from which queries for this can be built (I still maintain that most queries will be simple changes - grouping, time periods, filtering - over already existing queries)
Documents (can include emails, transcripts, etc.) from that past that discuss how these metrics were investigated in the past, in case we need to do such investigations now. Actually we don’t need full documents - the context layer can simply extract the relevant sections and stack them up with referencesz
How the result needs to be communicated back to the user once all the analysis is done. Includes text / visualisation styling, relevant jargon, etc.
Once this context layer has been generated for the question at hand, the real analysis (no not THAT KIND of real analysis) happens. The LLM identifies / constructs the necessary queries (remember that in most practical cases, new queries don’t have to be written. only lightly modified from pre-existing ones), fetches the necessary data and does the requisite analysis on it.
If there is any investigation required, pre-existing investigation patterns will get prioritised and implemented. And then the final result is packaged as required and sent.
In fact, if you think about it, most of the work required in building analytics copilots / chatbots lies in constructing the context layer! Semantic layers, robust metric definitions, etc. are all good, but the context layer (as defined in the above linked post) is the main deal!
If the context layer hasn’t surfaced the right kind of information, in terms of queries, disambiguation, old documents, how to do the analysis, etc. there is no use of having the most precisely defined metrics. The LLM won’t be able to do anything anyway.
In my last post on this topic, I spoke about two kinds of analytics copilot companies - the data-first players (like Databricks Genie) who prioritise semantic layers and precise metric definitions, and code-first players like Hasura’s PromptQL who use wikis to build knowledge and answer questions.
Now thinking about it, both of them attack parts of the problem and neither is complete. The wiki approach is absolutely required - to construct the context layer! And then we need a precise definition / query-mapping of metrics, which as I’ve argued can be built from existing SQL queries (like the stuff developed by Snowflake or Datahub).
Unlike what these companies claim, the two are not substitutes for each other. And if you were to get me to build this for you, I’d probably use a context-first engine (like Wisdom or PromptQL) to build up the context layer (obviously an LLM needs to be involved in this process in production), or even just an enterprise search product such as Glean (since you’ll be hiring me as a consultant I’ll tailor my approach to the stack you already use!), and then use existing queries as a pseudo semantic layer (or even use Datahub’s tech for this) and tie everything together.
And of course, you’ll need to give me the existing queries for it to work. For that holds the most context!