Why Agent Logs Will Not Satisfy Article 12: Decision Records, Causality and Provenance

In August 2026 a rule came into full force that most engineering teams have yet to feel. Article 12 of the EU AI Act says high-risk AI systems must technically permit automatic recording of events across their whole lifetime, with traceability proportionate to the system's purpose. Article 26 gives the deployer a matching duty: keep those logs for at least six months, longer where other law demands it.

An engineer reading the text notices an odd imbalance. The goal is stated precisely, while the required format is barely mentioned. The goal is being able to reconstruct, after the fact, one individual decision. An APM product does not provide that, since it was built for something else. A vector store does not provide it either.

The timing is awkward, because agents have just stopped answering questions and begun acting. They approve spending, resize infrastructure, route insurance claims, turn down applicants and sign off on releases. Each of those is a decision someone may eventually ask a company to justify, and in most organizations today the truthful reply would be that it cannot.

A small claims case that previewed the problem

Before the regulation existed, and before agents could act, a tribunal in British Columbia tested the same weakness on a small scale. It remains the clearest example available.

Jake Moffatt's grandmother died in November 2022. That same day he visited Air Canada's website to book a flight to Toronto and, unsure how bereavement fares worked, asked the airline's chatbot. The bot said he could purchase a full-price ticket immediately and request the discounted bereavement fare retroactively, within 90 days. Before relying on this, Moffatt did something sensible and screenshotted the answer. He then booked, flew, attended the funeral and submitted his claim.

Air Canada turned him down. No such policy existed, and the bot had invented it.

The sequel is the instructive part. At the Civil Resolution Tribunal, the airline argued it could not be held responsible for what its chatbot said, describing the bot as a separate entity answerable for its own actions. The tribunal member, Christopher Rivers, called that argument remarkable and observed that the airline had never explained why anyone should accept it. A chatbot is part of a website, and a company answers for its website. Air Canada paid C$812.02.

That sum is trivial. The telling figure is zero, the number of records Air Canada could produce to show why its own system had said what it said. The only lasting evidence in the whole hearing was the customer's screenshot.

The same failure has since grown. In April 2025 a support bot at Cursor invented a policy limiting subscriptions to a single device. Customers began canceling, and the company learned of it only because a cofounder spotted the thread on Reddit. The pattern is identical, the blast radius larger and the pace faster, and no tribunal was needed.

Now imagine that same evidentiary weakness placed in front of a regulator, attached to an agent that moves money.

Debugging and accountability are different jobs

Most stacks never draw a distinction that matters here.

Observability is execution-centric. It captures what a process did during one run: spans, tool calls, token counts, retries and latency. It is sampled, expires after a retention window and is scoped to a request. That is exactly right for tracking down a bug on a Tuesday.

Accountability is domain-centric. It captures what the organization concluded, on what evidence, under which policy, and what happened afterward. It must survive framework migrations, model retirements, index rebuilds, a reorganization and the departure of two colleagues.

Elsewhere in a business this is routine. A production change is never rebuilt from application logs, since a change record carries an author, a ticket and an approval. A financial position is never rebuilt from HTTP traces, since journal entries exist. Orders, payments, approvals and deployments are durable objects, and each has a lifecycle, an owner and a retention rule.

Agent decisions are the one consequential kind of event still left as unstructured exhaust, with fingers crossed.

The 3am version

The compliance framing makes the failure sound exotic, when it is quite ordinary, so a concrete case helps.

Twenty minutes after a release, checkout begins returning 5xx errors. A triage agent lines the error rate up against the rollout window, sees no infrastructure alerts and labels the event a release regression. A remediation agent rolls the release back. Errors drop to normal. A release agent reopens the deploy queue. Someone drops a thumbs-up in the incident channel and the team goes to sleep. Eleven minutes end to end, and no incident review follows, because the problem seemed to resolve itself.

But the release was never at fault. A downstream payment provider had been degrading and happened to recover at about the moment of the rollback. The correlation was a coincidence, and the agents wrote a believable story on top of it.

The truth surfaces in October, when the same pattern recurs and somebody asks whether the April episode shared the same root cause. Time to go and check.

By now the traces have aged out, deleted in June. The prompt template has gone through two rewrites. When the embedding model changed, the retrieval index was rebuilt, so the runbook passages the agent read back then are not the ones it would pull today. The model version named in the request header has been retired. What survives is a chat thread ending in a thumbs-up emoji.

The question cannot be answered. Nobody was careless. Nothing in the stack was ever designed to hold the answer.

Why better memory will not fix it

The usual instinct is to grab the memory layer, which usually turns out to be a vector store. That tackles another problem entirely.

Embedding search handles one question well: what looks like this? It cannot say what the system believed at 14:02 on a given day, which earlier conclusion led to this one, where a particular fact originated, or whether the policy meant to gate an action actually fired.

A subtler flaw exists as well. Retrieval does not stay reproducible over time. Re-embed the corpus with a newer model and an identical query returns a different neighborhood, so the thing treated as a record has quietly changed beneath it. An audit trail that can be silently rewritten by upgrading a dependency is no audit trail.

Each layer answers its own question:

• Vector search: which items look like this one?
• Knowledge graph: what is linked to it?
• Provenance: where did it originate, and is it still valid?
• Decision records: what was concluded, and on what grounds?
• Causal links: which event shaped which?
• Rules: what policy dictates the outcome?
• Temporal state: what did the system hold true on 14 March?

A real "why did it do that" investigation needs most of these. Only one is a vector store.

Turning a decision into an object

This is where Semantica deserves a look. It is an MIT-licensed Python layer that sits beneath an agent framework and treats context as a graph instead of a blob of prompt text. Graph building, reasoning and provenance are deterministic, with no LLM involved, which matters when the output is supposed to serve as evidence.

The central idea is plain. A decision becomes something deliberately written, not something dumped into a log.

from semantica.context import ContextGraph

ctx = ContextGraph(advanced_analytics=True)
decision_id = ctx.record_decision(
    category="incident_triage",
    scenario="Checkout 5xx elevated 20 minutes after release 2026.31",
    reasoning="Error onset correlates with rollout window; no infra alerts; "
              "no dependency alarms visible at time of assessment",
    outcome="classified_as_release_regression",
    confidence=0.83,
)

The mechanics are simple. What changes is what the author is forced to spell out. Instead of free text in a log line, a category, a scenario, the grounds, an outcome and a confidence figure must be committed. All of that remains queryable eighteen months later, which prose does not.

Consider the reasoning string in the October scenario. It says there were no dependency alarms visible when the assessment was made. That single phrase reframes the story from an agent that blundered to an agent that drew a fair conclusion from a partial signal, which points to a monitoring gap worth closing. The two framings lead to very different meetings.

A caution about that field. It is not a place for the model's raw chain of thought, which is unstable, sometimes made up, and impossible to defend clause by clause before an outsider. Record the rationale the organization is willing to stand behind: the evidence weighed, the threshold crossed, the policy applied, the human who overrode it. That serves production better anyway.

A single record is a log line, a chain is infrastructure

Real agent workflows fan out, and the damage rarely sits in a single node. It sits in everything downstream that quietly absorbed a wrong conclusion.

The best-known pre-AI example is Knight Capital. A faulty deployment in 2012 left an outdated code path live, and within roughly 45 minutes the firm was down about $440 million. Nobody doubted what the software did, and each order had been logged. What nobody could do was follow the consequences by hand as quickly as they spread.

Semantica allows the links between decisions to be declared when they are written:

triage_id = ctx.record_decision(
    category="incident_triage",
    scenario="Checkout 5xx elevated after release 2026.31",
    reasoning="Error onset correlates with rollout window; no infra alerts",
    outcome="classified_as_release_regression",
    confidence=0.83,
)

rollback_id = ctx.record_decision(
    category="remediation",
    scenario="Contain checkout 5xx",
    reasoning="Regression attributed to release; rollback is lowest-risk action",
    outcome="rolled_back_to_2026.30",
    confidence=0.91,
)
unblock_id = ctx.record_decision(
    category="release_gate",
    scenario="Resume deploy queue after containment",
    reasoning="Error rate normal for 30 minutes post-rollback",
    outcome="deploy_queue_reopened",
    confidence=0.86,
)
ctx.add_causal_relationship(triage_id, rollback_id, relationship_type="CAUSED")
ctx.add_causal_relationship(rollback_id, unblock_id, relationship_type="INFLUENCED")
why = ctx.trace_decision_chain(unblock_id)
blast_radius = ctx.analyze_decision_impact(triage_id)

Three relationship types exist: CAUSED, INFLUENCED and PRECEDENT_FOR. The vocabulary is small, and that is acceptable. Nearly all the value comes from a single operation.

On discovering in October that the triage was wrong, the question changes. Asking which traces mention checkout returns forty thousand rows. Asking which decisions inherited that conclusion returns a short, bounded list: the rollback, the queue reopening, the customer messages sent, the postmortem nobody wrote. That is a blast radius for judgment calls, and it is the very operation required once agents stop suggesting and start doing.

find_similar_decisions() works in the opposite direction and returns precedent. Has this system met this kind of call before, and what did it do? That is how an agent that quietly contradicts itself every third Tuesday gets caught, a real failure that nobody watches for because no metric exists for it.

Causality is not provenance

The two are mixed up constantly, yet they answer separate questions. Knowing that decision C relied on decision A is of little use without knowing what evidence A rested on.

The provenance module in Semantica follows W3C PROV-O. It is unfashionable and correct.

from semantica.provenance import ProvenanceManager

prov = ProvenanceManager(storage_path="./provenance.db")
prov.track_entity(
    entity_id="release_2026_31",
    source="ci/pipeline_runs/2026-31/manifest.json",
    metadata={"stage": "canary", "extractor": "ManifestParser", "confidence": 0.99},
)
lineage = prov.get_lineage("release_2026_31")

Two design choices here are worth borrowing even by teams that never install the library.

First, provenance entries form a hash chain. Every record holds a sequence number plus a checksum tying it to the one before, and verify_chain() reveals whether history was edited or entries dropped. That is what separates a table that anyone can quietly UPDATE from something that works as evidence.

Second, facts are invalidated and not deleted. When something stops being true, it is superseded together with the actor and the reason:

prov.invalidate(
    "release_2026_31",
    agent_id="human_sre_lead",
    reason="Error attribution corrected; degradation traced to payments provider",
)

A mutable valid=false column reports the current state. A ledger reports that the state once differed, who changed it, when, and why. Return to the Air Canada hearing and ask which of the two the airline needed. The answer is obvious.

Moving policy out of the prompt

Another quiet advantage is that Semantica includes real reasoning engines: forward chaining, Rete, Datalog and SPARQL. The logic is straightforward. If an organization already knows a rule, there is no reason to ask a language model to improvise it in English on every request.

from semantica.reasoning import ReteEngine, Rule, Fact, RuleType

engine = ReteEngine()
engine.build_network([
    Rule(
        rule_id="dual_control_prod",
        name="Human approval required for high blast-radius prod changes",
        conditions=[
            {"field": "env", "operator": "==", "value": "production"},
            {"field": "blast_radius", "operator": "in", "value": ["high", "critical"]},
        ],
        conclusion="require_human_approval",
        rule_type=RuleType.IMPLICATION,
    )
])
engine.add_fact(Fact("chg_014", "change", [{"env": "production", "blast_radius": "high"}]))
matches = engine.match_patterns()

The model still handles the messy input and suggests the action. The change is in the check that approves or blocks that action: it is no longer a hopeful paragraph inside a system prompt, one an oddly worded edge case can argue its way past. Since the rule_id is storable on the decision record, asking which policy applied turns into a simple lookup, not a debate between two teams.

A caveat: the condition matcher is deliberately basic. Stress-test it against the actual rule set before it stands between an agent and any regulated action. A demonstration rule engine does not amount to a compliance control, and finding the gap mid-audit is a costly way to learn it.

Dull standards outlast frameworks

PROV-O handles provenance, RDF representation, SPARQL querying, SHACL constraints and OWL ontologies. None of it is thrilling, which is the whole point.

Agent frameworks have a half-life of roughly eighteen months. Audit requirements do not. A reviewer in 2030 will not care which orchestration library was in vogue when a decision was made, and nobody wants a permanent record stuck inside the proprietary export format of a startup that has since changed direction.

There is also an MCP server, launched through python -m semantica.mcp_server. Its tools cover extraction, decision recording, precedent lookup, causal chains and reasoning. With that in place the agent records its own decisions through an ordinary tool call, and no logging wrapper has to be bolted on that the agent might sidestep.

The real price

The costs deserve plain statement.

The first cost is a new stateful service, with its backups, migrations, schema and access control. The second is modeling effort, since no one else can settle what counts as a decision worth recording. Capturing every step yields another log, only dearer. Capturing too little leaves holes in the chain right where the question will fall.

A cultural cost exists as well, and it tends to be underestimated. A decision graph makes it painfully easy to see that an agent has been making one particular call badly for four months. That is the purpose, and the meeting will still be uncomfortable.

The project also moves quickly. Pin the version that was validated. Anyone needing unreleased behavior should install from a source checkout pinned to a specific commit. Treat it as infrastructure, never as a convenience import.

Begin with one workflow

Graphing everything is a mistake. Graph what would have to be defended.

Pick one workflow in which a bad call would cost real money or trust. Record a decision only at the points where authority is exercised, such as approving, rejecting, spending, writing to production or committing to a customer. Along with each, store the evidence it relied on and the rule ID when a deterministic policy was involved. Commit to the graph asynchronously so that a write never sits on a path a user is waiting for.

After thirty days, try answering a genuine "why did it do that" question using only the graph. Gaps will appear immediately, and they will not be the ones predicted. That exercise teaches more than any amount of schema design done up front.

What to remember

Two years went into chasing explainability, which mostly meant trying to look inside the model. That target is wrong and is not getting closer. The model remains probabilistic, and no clever prompting makes its internals admissible anywhere that counts.

Accountability is the achievable alternative, and every piece of it lives outside the model: the decision, its causal parents, the evidence, the provenance, the policy applied, the downstream effects and later corrections. All of it is ordinary structured data. Records like this have been kept since double-entry bookkeeping. The habit simply stopped when the decision-maker became a language model, as though that altered what a record is for.

Remove the legal language and the demand of Article 12 is modest. It does not ask for more logs. It asks for a record that keeps its meaning three years on, after the framework has been retired, the index rebuilt twice and the prompt's author has moved on.

A better test than the regulatory one is to revisit the 3am incident. Nobody there was at fault. The agents drew sensible conclusions from the information in front of them and the incident was closed in eleven minutes, yet the explanation had vanished within one quarter. If a routine Tuesday can wipe out the ability to explain a decision, an auditor will not be the first to spot it. The team will, in a meeting months later, as it gradually dawns on everyone that no one present knows what happened.

What really cost Air Canada was not C$812. It was appearing at a hearing with nothing to say about how its own system behaved and falling back on the idea that the software was somebody else's responsibility. The defense collapsed even though only one ticket and one bereaved passenger were involved, a passenger who thought to capture the chat.

Agents today approve, spend, deploy and reject faster than anyone checks, and nobody is capturing screenshots of that. Write the record first, before anyone requests it. The question always comes at the worst possible time, and by then the answer is either stored in a database or lost permanently.