A general-purpose data analysis platform

The big 4

A data analysis platform primarily does four individual tasks. You can think of it as four different layers of processing. The four steps are extraction, curation, analysis, and presentation.

Below is how I imagine a data analysis platform — as a stack of the four layers I mentioned.

Notice that inside each layer are three distinct sub-components. The lowermost component is the core. This is akin to a kernel in an OS. It contains the actual capabilities of that layer — be it scraping for data extraction, a PDF generator for the presentation layer, and so on. Essentially, all the code we write for software is full of abstractions on top of code already written; similarly, the core capabilities are the very core here.

The second layer is for tools built on the core capabilities. These tools — consider them APIs to the core — are much easier to use than the core, just like assembly is fast but we mostly write software in high-level languages like Java and Python because they are an order of magnitude easier to write code in.

The point to remember: generic core and specific top.

The third layer is where we write application-specific code. If this is a platform for financial data analysis, we'd have financial models built in this layer for analysis; if it's a logistics platform, the topmost layer in presentation is about daily transportation reports. These components differ from company to company, depending on what business they are in.

This is what companies do when they build such a platform: they don't consider each layer as separate units, but try to build the entire layer — say, extraction — as a whole. There is a lot of abstraction going on still; however, the goal is not to make configurable platforms that take time, but to build a platform they can ship by the weekend.

However, with just a change in philosophy, one can build a general-purpose data analysis platform that would turn out to be far superior.

01 — The extraction layer

This layer has exactly one job: get the data, from wherever it resides. Starting from any type of digital document — an image, receipts, cards, recorded audio and video, websites — to physical documents, a live feed from sensors, or a camera. This layer's core is built to handle all types of data. It may be noisy, with a lot of gibberish, but it ingests it all. It scrapes websites and platforms that allow it, it integrates APIs, or connects with MCPs. It also exposes APIs for data to be pushed in, and this is the most common way this platform would be used. All for just one purpose: collect data.

While building this, for example, we don't build a specific scraper for a website to begin with; what we build is a scraping core. The only job of this core is to scrape blindly. This, in conjunction with a sessions core and a login/auth core, would be able to log in to a page and scrape data from there. It's bare-bricks building, right from the very bottom.

The most important thing here — and I'll reiterate this philosophy several times later as well — is that all of this is built on extremely well-defined structures.

I am a sucker for structures.

In this case, when I refer to structures, I mean the output of this layer would be one from a defined set of structures. Say the output for a food bill receipt is a very well-defined JSON format, common for all the food bills that might be uploaded or fetched. It will have an exhaustive list of parameters, and in most cases a change in this structure would simply be an amendment, not an edit.

The reason for solid structures is that this makes the foundation of the system anti-fragile. Essentially, it fails to break, even on purpose.

Therefore these structures have to be thought through religiously from day one, because everything downstream is blindly built against them — and every version has to stay backwards compatible unless you've deliberately committed to a clean break, which would be the rarest of rare cases.

02 — The curation layer

Extraction is deliberately blind. It has no opinion about what matters, so it takes everything it can reach. This means it also took the navigation bar, the cookie banner, the ad slot, the tracking parameter, and the field that came back empty. Curation is where opinions, rules, and guides are applied for the first time with rigour (even extraction has rules, such as don't scrape a page when the response is an HTTP 500 — but those are more about common sense applied in code than about real analytics).

All sorts of filters are applied here to remove noise and unwanted data, in certain cases intelligent masking, and also certain conversions (like currencies being converted to USD so that there is a standard for the next layer).

Curation is, frankly, an undersell here — because not only are filters and sorts done here, even the addition of data from static or live datasets is done here. Take the previous example of currency conversion: we'd need the currency rate for this. So curation also plugs into static or live data to handle this conversion.

Now, here one must ask: why not have this fed in from the extraction layer itself? The reason for not taking that road is the additional, useless effort. If for every transaction (defined as a process flowing through all data layers to generate an output) I need to get those rates from extraction, the extraction layer is overwhelmed with similar and predictable tasks. So rather, have that task performed only when required — and that too by curation. Also, currency is not raw data on which any process happens; it is a tool, an operator, whereas what we get from the extractor is the operand.

Lastly, this is the last place in the system where raw data is updated or changed, which becomes the only source of truth for the next layer.

03 — The analysis layer

This is the important one — i.e. the brain of the platform. Everything before it was mostly logistics; this is where a transaction actually gets thought about. The unit of analysis here is called a parameter.

A parameter is a self-contained logical unit that accepts inputs (from the previous layer or other parameters), does logical processing, and outputs a response in a fixed structure (the structure is specific for each parameter, and not the same for all parameters). Each parameter also has all possible error scenarios, such as an error in input, a logical error, or anything else — whatever can go wrong is already defined in a parameter. The last thing it has is a version. Anything about a parameter can essentially be updated with a new version of the parameter. Also, multiple versions may live in the same parameter at the same time, each one invoked by a configuration. Below is a depiction of a single parameter.

A real transaction needs hundreds, if not thousands, of them. Also, as I mentioned earlier, a parameter may depend on other parameters. So we can visualise it like a node-based directional graph, where the direction is downwards and the root is the output of the curation layer. Because of dependencies, we can put parameters in layers, where a parameter in, say, layer 4 would depend on at least one parameter of layer 3 and no parameter of layer 4 or beyond. In simple words, all dependencies of a parameter are computed in the previous layer itself, and this is true for all parameters.

The way the analysis layer works is that it contains an engine which first identifies which parameters are to be computed (every transaction may require a different set of parameters). Once these parameters are identified, the engine then runs up the graph from these parameters, identifying dependencies and putting them in the same bag as the parameters that were requested. After this step, we know which parameters are to be computed. The engine then starts computing the parameters (in parallel) of each layer, one by one. Failures don't crash, because of the error conditions.

A parameter's logic can also call an API. Some computation is proprietary, or lives in a subsystem it'd be unnecessary to rebuild (the obvious being an LLM). It's all statistics underneath, but you don't assemble one inside your analysis layer when it's already sitting behind an API. The work happens outside, and the answer comes back into the unit to be used. In and out, the same discipline as everywhere else — standard structures both ways.

04 — The presentation layer

Each of the layers prior is operated on entirely by the system, where presentation does not matter — structure does. This is the last layer, where we bring presentation into the picture. Any analysis platform is meant to be used by a human in the end, and this is where we make output according to what makes sense to the human. It may be a PDF report, an HTML page, or a video — all are forms of output a human can make sense of, and therefore needed.

This layer contains three critical parts. The core is, of course, the fundamental builders, such as a PDF builder or an HTML renderer. The middle layer on top of this contains templates and data mappers: a template is essentially a prepared output structure, and a data mapper decides which data goes to which placeholder in the template. The topmost layer is the one that prepares the data for this process.

This layer never really finishes. Every new use case wants the same answer shaped slightly differently, so the template library keeps growing. This would be one of the most worked-on components, because it deals with design — and designs ALWAYS keep changing.

Bringing it all together

Four layers, one continuous pipeline. Each takes the layer above it as input, fans that input out across its own node types, and folds the results back into a known shape, which is the exact input the next layer is waiting for. Here's the whole thing at once, extraction through to a delivered artefact.

Why this wins

None of this is for its own sake. Because every node is an independent unit built on the foundation — and nothing else — the whole thing gets easy exactly where big systems get hard: building it, scaling it across teams, debugging it, and keeping it alive. Four big wins, and that too with simple, piecemeal components.

What adds weight to it is that each new component is an addition to the whole system, not an isolated piece. It's as close to open source as possible, and wonders can be created with this. I've got some ideas already. Would you like to chat about it?

Click here to contact me →

← All work