Give Every Agent Its Own Database

Working note No. 12 — on inverting the warehouse for agent workloads.

Abstract. Note No. 11 described the memory our agents share; this note describes the data they don't. When we let language-model agents and batch validation jobs loose on our research infrastructure, the first casualty was the central analytical store: agent workloads are concurrent, bursty, retry-happy, and indifferent to the queue etiquette that human analysts provide for free. Our fix inverted the architecture — every agent now carries its own embedded analytical engine and reads immutable, content-addressed snapshots instead of querying a shared warehouse. The inversion cost one afternoon of plumbing, eliminated the contention class entirely, and delivered an unplanned dividend that turned out to matter more: backtests that name their inputs by digest are reproducible in a way that "we queried the database on Tuesday" never was. Reis (2026) recently articulated the general architecture with a rigor we won't attempt to match; consider this the field report from a one-person lab that arrived at the same shape.

1. Introduction

A central analytical database is a reasonable design under an assumption so old it is invisible: that the things issuing queries are people. People query at human pace. They inspect results before acting. They do not retry a failed query four hundred times in ninety seconds, and they do not spawn six copies of themselves to check a hypothesis from six angles.

Agents do all of these things, and our infrastructure met them the way most does: with a queue. Validation folds waited on report generators; report generators waited on an agent's exploratory scan; everything waited on whoever asked the biggest question first. The warehouse had become what shared mutable state always becomes under concurrency — a lock with a data model attached.¹

2. The inversion

The fix we landed on has two components, both boring on purpose.

Embedded engines. Every agent and every batch job gets its own analytical database — an embedded, in-process engine (ours is DuckDB; the choice matters less than the property that it runs inside the consumer and shares nothing).² There is no server to contend for. Concurrency scales with processes, which for a research program means: with the number of questions being asked.

Immutable slices. Consumers never read "the data." They read a slice — a snapshot of a dataset, frozen at a point in time, named by what it contains:

slice://bars_1min/2026-07-31
  snapshot:  2026-07-31T22:00Z
  schema:    sha256:1f3a…
  content:   sha256:9c47…
  tier:      SOURCE          # derived slices may not claim SOURCE

A small catalog maps names to digests; the slices themselves live as plain columnar files that any engine can open. Producing a new snapshot never mutates an old one. An agent that needs fresher data takes a newer slice; an agent mid-study keeps the one it pinned. Fig. 1 contrasts the two worlds.

Fig. 1. This figure contrasts the two architectures. Left: a central warehouse in which every consumer — human or agent — queues for the same query engine, and one expensive question stalls the rest. Right: the inverted design, in which each agent (gray) carries its own embedded analytical engine (dark) and consumers exchange immutable snapshots addressed by content digest; the central component shrinks to a catalog of identifiers (dashed), which serves no queries. The topology is schematic.

3. The dividend: reproducibility by construction

We adopted the inversion to kill a queue. Its lasting value turned out to be epistemic, which is why it belongs in this series.

A backtest whose input is "the prices table" is not a fact; it is an anecdote about whatever that table contained at the moment of the run. Re-run it after any upstream revision — a vendor correction, a late fill, a schema tweak — and the result moves, and there is no way to say why. A backtest whose input is content: sha256:9c47… is a different object entirely: every study in our archive now names its inputs by digest, which means any result can be re-derived, byte-identical, from the vault's burial record — or shown to depend on a revision, which is itself a finding.³

The tier field does quiet governance work of the same flavor. Slices marked SOURCE can only be produced by the ingestion path; anything an agent derives is DERIVED, permanently, no matter how many transformations later. Reis (2026) calls the failure this prevents laundering provenance — derived data drifting upstream in authority until someone conditions a study on their own artifact. In a lab where agents produce most of the intermediate data, we regard this single bit as load-bearing: it is the data-layer version of the rule that the agent writes pages, not verdicts.

4. What we skipped

Honesty about scale: Reis's full design — peer-to-peer slice exchange, three-layer semantic contracts, signed lineage DAGs — is architecture for organizations. A one-person research program needs perhaps a fifth of it, and we built roughly that fifth: embedded engines, content-addressed snapshots, the tier bit, and a catalog small enough to read at breakfast. We record what we skipped as deliberately as what we kept, because infrastructure has the same failure mode as strategy design — components added for plausible reasons, never validated by load, carried forever.⁴ The mesh can apply for admission when the lab has peers.

5. Conclusion

The pattern generalizes past data: shared mutable anything is a bet that consumers will be polite, and agents are not polite — they are fast, parallel, and tireless, which is why we hired them. Giving each one its own engine and immutable inputs replaces etiquette with arithmetic. That it also made every backtest in the archive citable by digest was, for us, the difference between an optimization and a principle.

The warehouse is dead; long live the catalog.

Notes

1. The failure mode is not load in the benchmark sense — the volumes involved would amuse anyone running real infrastructure. It is interference: heterogeneous consumers with heterogeneous deadlines sharing one scheduler. The smallest agent retry storm degraded every study in flight, which for a validation protocol is not slowness but contamination of timing-sensitive runs.
2. Raasveldt and Mühleisen (2019) state the design point precisely: analytical workloads embedded in the consuming process, eliminating the client-server boundary. That the same property later turned out to be exactly what agent workloads want is one of those accidents that looks like foresight from a distance.
3. Helland (2015) is the standing reference for this worldview — append-only, immutable data as the foundation that makes distributed systems (and, we would add, research archives) reasonable about the past. Our slices are his "immutability changes everything" applied at the scale of one stubborn lab.
4. The parallel is exact enough to state as policy: infrastructure components face the same adoption bar as strategy components — demonstrated need on realized load, not anticipated need on imagined load. The drawer of note No. 9 has a shelf for architecture too.

References

Helland, P., 2015. Immutability changes everything. In: Proceedings of the 7th Biennial Conference on Innovative Data Systems Research (CIDR).

Raasveldt, M., Mühleisen, H., 2019. DuckDB: an embeddable analytical database. In: Proceedings of the 2019 International Conference on Management of Data (SIGMOD), 1981–1984.

Reis, J., 2026. To every agent its own database. Practical Data Modeling (Substack), joereis.substack.com.

Keywords: agents, embedded databases, DuckDB, immutability, provenance, reproducibility.