Working note No. 13 — on the distance between the menu and the meal.
Abstract. Surveys of the open-source data ecosystem — Mezhanskiy's (2026) is the best current one — catalog a landscape of remarkable depth: four credible options per layer, a dozen layers, comparison tables with commit velocities. This note is the other half of that conversation: what a one-person research lab actually runs. Our answer is five components, all open source, most of them boring on purpose — and the note is less about the five than about the selection principles that produced them, because the principles generalize where the tool names will not. Two of those principles came from unexpected places: one from an earlier note's argument about embedded engines, and one from reading software licenses the way our compliance training taught us to read contracts — for the counterparty risk.
1. Introduction
The modern data stack is usually drawn as a wall of boxes: ingestion, lake, warehouse, table format, catalog, orchestration, streaming, metadata, BI — each layer a market with contenders, each contender a repository with stars. The wall is genuinely impressive, and the survey literature that maps it is useful precisely because someone must track which boxes are thriving and which are quietly dying.
But the wall is a menu, not a meal. Nobody runs all of it, and the interesting question for a small research program is not which tool wins each layer but which layers deserve to exist at all at our scale. Fig. 1 shows both halves of the answer.
Fig. 1. This figure contrasts the surveyed landscape with a working configuration. Left: the modern data stack as a menu — roughly twenty components across six layers, each a dependency carrying its own maintenance and counterparty risk. Right: the five components this lab runs — scripts for ingestion, columnar files for storage, an embedded engine for compute (red: the only component we consider irreplaceable on less than a week's notice), a scheduler old enough to vote, and the research vault of note No. 11 serving as both metadata layer and "BI." The topology is, as always, schematic.
2. The five
Ingestion: small Python scripts. Not a platform, not a framework — scripts, versioned in the same repository as everything else, each one short enough to read at a sitting. Our ingestion problem is a handful of sources on predictable schedules; a platform would be a solution renting space from a problem we do not have.
Storage: plain columnar files (Parquet), immutable and content-addressed, as described in note No. 12. No lake, no table format, no catalog service — the "catalog" is a file small enough to read at breakfast. The format wars above this layer are real and consequential for organizations; at our scale, the winning move was declining to have the war.
Compute: DuckDB, embedded in whatever process is asking — the one choice we would defend at any scale that still fits on machines we control, for reasons note No. 12 lays out: no server, no contention, concurrency that scales with the number of questions.1
Orchestration: cron and a Makefile. We are aware of what this sentence does to our credibility with platform engineers, and we stand by it: our dependency graph has one digit of nodes, and the failure mode that matters — a job silently not running — is covered by the incident log, not by a scheduler with a control plane. The orchestrators in the survey are excellent software for graphs that deserve them.2
Metadata and "BI": the vault. Study pages, burial records, and the index — markdown traversed by humans and agents alike — do the work that metadata platforms and dashboard tools do for teams. Our dashboards are the figures in this series; our semantic layer is a protocol file. We do not say this is sufficient for an organization. We say it is sufficient for the truth-keeping of one lab, which is the job.
3. The selection principles
The five share properties that were criteria, not coincidences.
Embedded beats client-server, local beats hosted. Every component runs inside processes we own, on machines we own. This is the architecture argument of note No. 12 extended stack-wide: shared services are bets on consumer politeness, and our consumers are agents.
Boring beats shiny, and stars are not survival. Survey metrics — stars, commit velocity — measure attention, and attention is mean-reverting. The ecosystem's recent history makes the point better than we can: projects with real communities abandoned mid-flight, a prominent object store archived outright, a wave of consolidation folding independent tools into one vendor's roadmap.3 Our filter is a decade-shaped question: does this tool's continued existence depend on decisions made by people whose incentives I cannot see? Cron passes that filter magnificently.
Read the license like a contract, because it is one. Relicensing events — open tools narrowing to "test and academic" use, source-available conversions — are the dependency world's version of a counterparty quietly amending terms. Our compliance habit turns out to be the right tool: the license is the fine print of the dependency, the clause most engineers treat as noise and we treat as signal. A tool with a permissive license and a governance foundation behind it is a different asset than the same code under a single vendor's changeable terms, and we price them differently.4
Extensibility is newly cheap. The classic argument for paying vendors was the last twenty percent — the connectors, the edge cases, the polish that open tools lacked. Mezhanskiy's sharpest observation is that agentic development collapses the cost of exactly that twenty percent. Our experience agrees: where we once would have filed a feature request, the agent now writes the extension against the tool's own source, in an afternoon, in our repository. Open source plus a capable agent is a build option that did not exist three years ago, and it moves the buy-versus-build frontier further than any pricing page admits.
4. Conclusion
The stack is not the achievement; the shortness of the stack is. Every component we declined to adopt is maintenance we do not perform, a license we do not monitor, and a failure mode we do not carry — the drawer of note No. 9, applied to infrastructure, again. The five that remain earn their place the way everything in this program earns its place: demonstrated need, on realized load, revisited when the load changes.
Choose open source for the reasons that survive audit — transparency, extensibility, control — and then choose as little of it as the work allows.
Notes
- It is also the one component whose replacement we have actually rehearsed: the storage layer is plain files, so the blast radius of swapping the engine is one import and the query strings. Rehearsed replaceability is, in our view, the honest measure of how "locked in" a stack is — not the license alone.
- The one orchestration feature we did adopt from the platform world is the idea of a run manifest: every scheduled job writes what it read, what it produced, and the digests of both — into the vault, next to everything else. It is a hundred lines of Python and it is most of what we envied.
- We name no projects because the specifics age badly and the pattern does not: the mortality of well-loved infrastructure is structurally identical to the mortality of well-loved strategies — survivorship is visible, funerals are quiet, and the graveyard is undercounted. Every dependency decision is a bet on someone else's research program.
- The same reading explains a preference the survey shares: formats and engines stewarded by foundations (or small, focused teams with long records) over tools whose openness is a growth-stage marketing decision. In contract terms: we prefer counterparties whose incentives are legible in the document itself.
References
Helland, P., 2015. Immutability changes everything. In: Proceedings of the 7th Biennial Conference on Innovative Data Systems Research (CIDR).
Mezhanskiy, G., 2026. The modern data stack: open-source edition. Datafold blog, datafold.com/blog.
Raasveldt, M., Mühleisen, H., 2019. DuckDB: an embeddable analytical database. In: Proceedings of the 2019 International Conference on Management of Data (SIGMOD), 1981–1984.
Keywords: open source, data stack, DuckDB, licensing risk, minimalism, agentic development.