Context Engineering for a Quant Research Agent: Proof Levels and Data Stores

Deciding what an agent may trust

Picture an AI-run research setup built for the WorldQuant BRAIN quant competition. Before any specialist agents, lifecycle checks or autonomous loops exist, there's a more basic design problem: which statements may the system treat as true? The raw material arrives in four forms that have little in common.

The first is official documentation from the platform and the tournament, which spells out how the expression language works, which operators and datasets are available, and which simulation settings are legal. The second is methodology. Marcos López de Prado's three books give tools for reasoning about backtest statistics, strategy risk, multiple testing, portfolio structure and falsification, while Igor Tulchinsky's Finding Alphas covers idea construction, turnover, correlation, overfitting, bias and durability in a hands-on way. The third is a body of papers and deep-research notes on economic anomalies, market participants and information flow, which can be arranged as a graph linking axioms, theories, anomalies, mechanisms and implementation shapes. The fourth is whatever the project generates as it runs: simulation responses, daily profit-and-loss curves, portfolio correlations, resilience statistics, platform checks, experiment histories, operational events, failed builds and the lessons extracted from them.

Every one of these is context, yet they carry unequal truth value, and none of them fits cleanly into a single prompt, a single document format or a single database. So context engineering in this setting begins somewhere other than token budgets. It begins with designing levels of proof, plus the data systems that keep each level intact.

On the left sits the naive first design, in which rules, books, papers and the system's own results are squashed into one prompt and all carry equal weight. On the right each claim retains its level. Rules form the walls, beliefs propose tests, observations trim beliefs, and nothing inside the walls is able to edit them.

Three levels of evidence

"Proof" is meant practically. For any claim, ask what evidence is enough, who is responsible for it and what is permitted to overturn it. Three layers handle those questions.

L1 captures what the platform declares the game to be, using official platform and tournament material. When the agent needs to know which settings exist, what an operator does, which checks a candidate must clear or how the tournament computes its score, this is where it looks. A local copy may age, but no surprising experiment can alter a rule.

L2 captures what the system really saw. Simulation outputs belong here, along with daily PnL series, today's portfolio, platform checks, operational events and the newest leaderboard state. Each measurement is stamped with the time it was observed. A rerun doesn't overwrite anything. It produces a fresh observation, so history never gets tidied after the fact.

L3 captures what the system presently considers useful. Books, papers, economic theories, specifications, knowledge-graph readings, written-down lessons and scoped falsifiers sit in this layer. They steer the next action and remain open to being overturned.

The ordering isn't strict, with L1 always beating L2 and L2 always beating L3. Each level answers a different kind of question. For a factual dispute between sources, the strongest direct one prevails, so an official rule outranks a paraphrase inside an agent prompt, and a measured response outranks what a book led anyone to expect. For the question of what to try next, though, rules and measurements run out of answers. That's when L3 is needed to place a reasoned bet.

Three ideas are tracked in this chart. Post-earnings drift wasn't killed by its falsifier, which instead identified the construction that broke, so the idea was rebuilt around analyst revisions and two versions got submitted. A peer-basket idea started life as pure belief with zero evidence and still got through. A dividend-clock idea failed all ten constructions, and the lesson shrank the belief to dividend dates only. Green squares cleared platform checks, rose ones failed, grey ones have no checks on file, and ringed ones were submitted.

A single database may contain both a measurement (L2) and someone's reading of it (L3), which means the level attaches to the claim, not to its storage location. A useful context record is accordingly richer than plain text. It should record the claim itself, its epistemic level, source, owner, scope, the observation or review time, the uncertainty, a stable identifier and whatever evidence could revise it. For facts that change over time, a pair of timelines helps: when something held true, and when the system found out. APIs that answer late and backfilled historical data make that gap matter.

Context is really a small data platform

Each store is drawn according to its behavior. The ledger stacks evidence in layers and one lineage chain runs through them, old events fade out, the graph is a pair of tables that turns into a multigraph in memory, and files are grouped by lifecycle. Stable IDs hold the pieces together. Daily PnL still sits in a spot that the project's own spec says it has left.

Once proof levels exist, another point becomes obvious: context isn't a single corpus. It behaves like a modest data platform. The largest part is the research ledger, which on the inspected workstation is a transactional SQLite database. Calling it an experiment table undersells it, because it works as an evidence ledger holding several families of records. One family is research intent, meaning mechanisms, hypotheses, predictions and the lineage of implementations. Another is measurement, which stores platform responses, in-sample statistics, checks, daily PnL and stress-test metrics. A third is interpretation, with attributions, lessons and scoped falsifiers. A fourth is governance, tracking candidate decisions, approvals and what happened at submission. The last is materialized analysis: portfolio deltas, correlation screens, pattern caches and stress-test summaries, all of which can be regenerated from canonical inputs.

Auditing it turned up a mismatch. A design document states that daily PnL had been moved into Parquet files, but the database still stores every row. The document is a belief (L3), the database is an observation (L2), and in a conflict the observation prevails.

Next comes an operational event store, kept as its own SQLite database. Events there can carry cycle, batch, candidate, hypothesis, agent and process identifiers along with a versioned JSON payload. Folding them into the ledger would be tempting, but retention needs differ. Research evidence is canonical and has to last. Debug output, dispatch events, queue transitions and temporary interventions swell quickly and can be compacted or expired. With the two apart, a burst of observability data can't bloat or lock up the scientific record.

Then there's the knowledge graph, a transactional property graph with a deliberately plain physical shape: a node table, an edge table and JSON attributes for evidence specific to each type, all loaded into an in-memory graph for traversal. The transactional layer avoids the corruption and concurrency headaches of one giant YAML file, while the in-memory form keeps reasoning comfortable.

The filesystem counts as a data system too. A repository holds untouchable platform documents, books, papers and derived reports, and pulling all of that into memory would be a mistake because lifecycles differ. Platform documents never change. Specifications are design priors under version control. Reports are derivations that need recorded inputs and a way to regenerate them. Memory files are short, readable pointers to reusable lessons. Queue files, caches and logs are throwaway operational state, and model binaries, PnL exports and big snapshots are artifacts best kept in object storage instead of Git.

The failure stories are telling. A top-performing alpha nearly got thrown out because the ledger claimed all checks passed, while a script was reading a run file that reported Sharpe 0. One early winner had no decision record at all until a reconciliation pass supplied it six weeks afterward. A betting-against-beta verdict existed only inside a Markdown memory file, so the ledger itself would have done nothing to prevent a retry.

What emerges is a multi-type context system, even though nearly all of it still runs on a single laptop. SQLite handles transactions, the graph handles relationships, Parquet is the planned format for large time series, Markdown carries knowledge that people can review, and transient files coordinate local processes.

One real mechanism appears here with its names removed. The left shows its ancestry in the knowledge graph and the right shows each hypothesis, experiment and candidate that put it to the test. Some failures recurred across every construction, whereas others were specific to one implementation. The two halves share nothing except the ID along the seam, and that single key is enough to join them into a compact context packet.

Mechanisms act as the link between the two stores. In the graph one appears as an economic idea together with its sources, and in the ledger as a tag on every hypothesis and candidate that exercised it. Empirical summaries may be reconciled back onto graph nodes to ease navigation, but those are only mirrors, and the experiment database remains where the numbers truly live. A good context packet is therefore assembled by joining across systems. It combines a small graph neighborhood, provenance for sources, current empirical rollups, recent scoped failures and the platform constraints that apply. The model has no use for the full graph or every PnL row. What helps is the shortest evidence path to the decision at hand.

When graph and ledger share a mechanism ID, a complete branch can be seen together: theory, experiments, every candidate, and a summary on the graph node that mirrors the ledger. A buyback-authorization idea came directly from a 1995 paper, yet it was never attached to the share-repurchase branch, so none of its outcomes are reachable from the graph.

Several data contracts would be worth making stricter. Objects that cross stores should carry an explicit JSON schema version. Any derived metric ought to name the code version, source snapshot and method version that produced it. Each experiment deserves a content-addressed manifest for reproducibility, listing the code commit, environment digest, platform settings and the hypothesis and implementation IDs. And reconciliation should be tested like any other data-quality concern, not assumed to be harmless plumbing.

To sum up the design stance, the graph is context, the ledger is context, and the link between them is context as well. That link merits the same engineering care as model code.

Where this meets harness engineering

No request is handed whole stores. Every store contributes a thin slice to a typed packet, and each specialist receives just what it requires. Results flow back by type to the right homes, the rules are never written to, and two loops refill the next packet at different speeds.

This marks the line between context engineering and harness engineering. Context management is responsible for L1 through L3: it keeps sources, measurements, priors, provenance, timestamps, scope and stable identifiers. L4, the agentic harness, chooses which evidence gets retrieved, how much is allowed past a stage boundary, which specialist gets it and which conclusions are legitimate. Beneath that are source code (L5), local infrastructure (L6) and distributed infrastructure (L7). Those layers will carry out a bad call flawlessly, but they have no say in whether an input was a rule, an observation or a belief.

An early iteration blurred that split. The main instruction file grew into a 627-line wiki blending tournament rules, live status, methods and the prior day's lessons. After it was cut down to a small router, context reached the agent through four routes. A short layer stays loaded all the time. A startup packet carries fresh lessons and warnings. Area-specific rules join only when a task touches that area, and specialist roles with their memories are fetched when needed.

One request, traced through

Consider the question "What should be tried next?" The harness starts by composing a preflight packet. From L1 it pulls the relevant expression, setting and scoring constraints. From L2 come the current portfolio, recent measurements, how much of the search space is covered and outcomes still unresolved. From L3 it takes the most relevant graph branch, the theoretical provenance, scoped falsifiers, research specs and empirical lessons.

Subagents then step in. A hypothesis specialist works from that packet to propose a few economic angles, without disguising thin reasoning behind formulas. After one angle is chosen, a modeling specialist takes the registered hypothesis and builds implementations that really differ from each other. An experiment specialist receives the implementation tree and the platform limits, then records what gets measured.

Each stage has its own destination. The ledger's L2 receives raw platform results and normalized measurements. Operational events such as run starts, tool calls, retries and coordination traffic belong in the event store. When a result yields a lesson or a scoped falsifier, it's filed under L3. Graph rollups come from reconciling against the ledger and never get hand-edited, while large PnL series, plots and fitted models turn into artifacts that other records point to.

Both alphas that were submitted began from packets already holding evidence. One family had previously improved the portfolio, and the other idea reused a build that had already yielded uncorrelated alphas. A rating-change idea also got a packet, an untested cell on the opportunity map plus a past winner to imitate, and all 18 of its constructions failed anyway.

Phasing the infrastructure

Moving everything at once would be a mistake. Each lane shifts in its own phase, and a gate opens only when a measurable trigger justifies it. The right side illustrates the single write path worth trusting (commit, outbox, exporter, projection), as opposed to a dual write that produces two rival truths.

The current local stack can do more than it appears to. SQLite is a great default for a lone researcher, with genuine transactions, simple backups, quick local queries and nearly no operations overhead. Strains are visible, however. A database of several gigabytes carries tens of millions of time-series rows, multiple workers may compete for local writes, runtime events keep multiplying, artifacts and snapshots are pinned to one machine, and reproducing a result elsewhere takes more hidden state than anyone wants. A five-phase evolution makes sense.

Phase 0 is about making the design match reality: either complete the half-done PnL migration or consciously roll it back, then automate the audit that compares documentation with live storage. Phase 1 pushes heavy artifacts outward, so daily PnL, plots, model files, large reports and database snapshots go to S3-compatible object storage, with PnL split into compressed Parquet partitions backed by immutable manifests, checksums and lifecycle rules. Phase 2 moves metadata only once concurrency forces the issue. When remote workers, several machines or collaborators must write simultaneously, the ledger and event store go to managed PostgreSQL, as separate databases or at least separate schemas with their own retention policies. Phase 3 adds MLOps as a projection and never as the source of truth, so Weights & Biases or MLflow can help without replacing the ledger or graph. Phase 4 decouples orchestration from state: research workers run in containers with immutable job manifests, while a queue or workflow engine hands out idempotent jobs with leases, retries and timeouts. Phase 5 brings richer analytics only when the workload earns them. A lakehouse table format, a dedicated graph database or a streaming platform might eventually pay off, but only once specific queries start to hurt, such as scans across billions of observations, many simultaneous graph traversals, schema evolution over large Parquet collections or low-latency event consumers.

No exact performance gain can be credited to this architecture, as its effect was never isolated. Its real contribution is letting the system contradict itself honestly. A paper suggests, a simulation disagrees, a falsifier narrows the belief, and the tournament rule stays the rule.