You Can't Retrieve a Number Nobody Wrote Down: OKF vs a Vector Database RAG

Every project has a line it keeps coming back to, and here it is: compile the figures, retrieve the prose. The line came out of a chatbot that began the usual way. A dozen CDC documents, including the survey description, the codebook, the summary report and methodology PDFs, were loaded into a Bedrock Knowledge Base, with the hope that a well-stocked store would give good answers.

They didn't. The reason is the most common misunderstanding about retrieval-augmented generation: that more documents mean more accurate answers. A vector database can only hand back text that's already inside it. For any question whose answer was never written down, the size of the collection doesn't matter at all.

Those twelve documents described the diabetes variables in detail: what they mean, who was asked and how answers are coded. Not one contained the statistics a user would actually request. Those numbers are calculated from 29,522 rows of survey data and appear in no PDF, so no quantity of searching will turn them up.

This leads to a sharper error still. People assume that once a topic appears in the documents, RAG can handle any question about it. It handles only questions whose answers are written in a passage. Anything that has to be worked out is outside its reach.

So the chatbot was rebuilt. It started as a tutorial app combining Amazon Bedrock Flow, AWS Amplify Gen 2 and Next.js, and the rebuild swapped its search engine for a verified Open Knowledge Format (OKF) bundle. For a fair comparison, a genuine Bedrock Knowledge Base was also built over the same documents, and the two were tested head to head on the only questions that count: is the answer right, and how quickly does it arrive?

Asked the headline question, the knowledge base identified five relevant variables and then stated the limit accurately: the documents don't include the actual computed statistics. It even recommended the right analysis, a weighted proportion of people with one flag who also have another, which is precisely what OKF runs at build time, specified correctly by a system that can't run it. It reduces to one sentence: you can't retrieve a number nobody wrote down. The OKF side answered with a figure computed from the microdata under the right population definition and the right weights.

How the test was set up

The setup was deliberately symmetrical. Claude Sonnet 4.5 generated answers on both sides, in us-east-1, with one in-process SDK transport and the identical system prompt the deployed Lambda carries. The method was written down before any results existed, all seven questions and every trial are reported, and raw answers are kept. A caution about timing: the harness calls retrieval and Bedrock directly, while the live app routes through AppSync and Lambda first. The OKF code under test is the app's real code, the transport isn't, so the timings compare two designs measured the same way and aren't what a browser user would see.

The swap: one Lambda

The Amplify chatbot exposes tools as Amplify data tools, each a Lambda behind a GraphQL query, and the Bedrock Flow was one of them. So the change is a second tool, askOkf, which answers from a verified bundle instead of a Flow. Schema, frontend and deployment stay the same.

// amplify/functions/askOkf/handler.ts — the essence
export const handler: Schema["askOkf"]["functionHandler"] = async (event) => {
   // in-process TF-IDF over the bundle
  const hits = search(event.arguments.question, 3);
  const top = hits[0];

  // Grounded-or-refuse: below the score floor,
  // nothing verified matches → refuse
  // before ever calling the model.
  if (!top || top.score < MIN_SCORE) {
    return { answer: refusal().text, citation: "", grounded: false, mode };
  }

  // read the verified figure straight from frontmatter
  if (mode === "extractive") {
    // 0 model calls - sub-millisecond
    const a = extractiveAnswer(hits);
    return { answer: a.text, citation: a.citations[0] ?? "", grounded: true, mode };
  }

  // generative: ground Claude on the ONE retrieved concept,
  // safety-prompted grounded-or-refuse
  const concept = top.concept;
  const context = `Concept: ${concept.label}\nSource: ${conceptSource(concept.frontmatter)}\n\n${concept.text}`;
  const response = await bedrock.send(
    new ConverseCommand({
      modelId: env.MODEL_ID,
      // grounded-or-refuse; declines medical-advice intent
      system: [{ text: SYSTEM_PROMPT }],
      messages: [{ role: "user", content: [{ text: `Verified concept:\n${context}\n\nQuestion: ${question}` }] }],
      inferenceConfig: { maxTokens: 400, temperature: 0 },
    })
  );
  const text = response.output?.message?.content?.[0]?.text ?? "";
  return { answer: `${text}\n\nSource: ${citation}`, grounded: true, mode };
};

Inside the handler, search() scans four verified markdown files packaged with the Lambda. No vector database, no embedding service and no network round trip are involved. Only concepts that survived verification are included, which means someone ran the analysis and checked the result. The tempting wrong figure, the unweighted whole-sample insulin rate of 3.66%, appears in none of them, so it can never be served.

Two paths ship and the benchmark measures both. The extractive path reads the verified figure out of frontmatter with zero model calls. The generative path makes one grounded Converse call.

The baseline: a real vector RAG

The opponent needed to be strong. It's a managed Bedrock Knowledge Base on S3 Vectors holding the real CDC 2023 NHIS material: the survey description, the Sample Adult codebook, the summary and methodology reports, the imputation technical note and the pair-weights README. A capable engineer would build exactly this. Since the model is identical, the only variable left is where the answer originates, a vector store or a checked bundle inside the function.

Thirteen files were uploaded and ten were indexed. The three failures were all .zip archives, which Bedrock rejects, and they happened to be the raw microdata. This sounds like a loophole that saves the vector database, and it isn't one. Every document describing the diabetes variables indexed fine. What's missing is the raw rows, and retrieval over raw rows couldn't produce these figures anyway, because a survey-weighted percentage across 29,522 records with a skip-pattern universe is a computation and not a passage. You can't chunk your way to it.

One more candid observation: the vector side was the cheap half. A dozen documents and a Knowledge Base took minutes. The OKF side took actual work, modelling concepts and running the analysis on the microdata to check each number. Mistaking how fast retrieval can be stood up for having solved accuracy is its own misconception. Cheap plumbing and hard correctness are separate problems, and the ease is a reason to keep the vector database around, not to remove it.

Correctness

The question set was three figure questions with known verified answers, three refusal probes and one methodology question. Verdicts were scored against a rubric registered in advance, with 30 trials per figure question and two per probe, all reported. On all three figure questions, the vector RAG failed to produce the verified number in any trial.

Latency

For figure questions with 30 trials each (warm runs being trials 2 to 30), the generative OKF path was reported as 1.4 to 2.4 times faster, because it skips the retrieval hop over the network and works with a much smaller context. The extractive path is close to instantaneous, since no model is called at all. Read the spread and not an average: the gap is widest on insulin at about 2.4 times and narrowest on age at about 1.4 times, where the two are nearly tied, so the per-question rows are the result and not a single headline ratio. Speed isn't the argument anyway. The knowledge base's 3.3 to 3.7 seconds is respectable, and it loses on correctness, not on time.

The finding: retrieve versus compile

In every single trial, the vector RAG failed to deliver any of the verified figures, and poor engineering isn't the explanation. Those numbers simply aren't present in the documents. The survey-weighted 31.96% is derived from the microdata using the correct population and weights and exists in no PDF. A retriever can only hand back existing text. OKF can answer because the figure was computed and checked ahead of time, whereas the knowledge base has nothing to fetch.

Notice how the knowledge base fails: it says it can't find the percentage in the documents, and doesn't invent a number. That's honest, and it makes the point stronger. Even a well-behaved, properly built vector RAG is structurally unable to answer a computed-statistic question over documentation.

Two more misconceptions fall here.

• Grounding doesn't guarantee correctness. This knowledge base refused and didn't fabricate, but that's a matter of good defaults and not a property of RAG. A somewhat more eager model, or a summary document that happened to contain an unweighted frequency, and it would serve a confident wrong number.
• A citation proves little. A source link tells you a document was fetched, not that the answer follows from it. An OKF citation is a different kind of thing, since it names a figure that was executed against the data and verified at build time, not text found next to a similar sentence.

Better prompts and extra chunks won't repair this. The repair is to quit retrieving these numbers and compile them.

Limits and trade-offs of OKF

OKF is no universal answer engine, and the benchmark shows where it stops.

It answers "how much" and not "why". For the qualitative question of how survey weighting changes prevalence, the vector RAG gave a rich, grounded answer drawn straight from the survey description, while the four-concept OKF bundle could only offer a related number. A verified-figure bundle is exact about computed statistics and silent on narrative. That's a boundary and not a defect, and it points directly at the architecture below.

Fast means no judgment. The extractive path answers in under a millisecond, but asked for medical advice it just returns "31.96%", because it can't read what the user is after. The generative path declines the advice and gives the population figure with a clear warning that it says nothing about any individual's situation. That discernment costs a model call, around two seconds. Treat the instant floor and the intent-aware path as separate tools and report both without blending them.

The complement: a third tool

These limits lead to one conclusion. OKF supplies the verified layer, and the vector database does well on the qualitative, narrative layer that OKF can't reach. The sensible design uses both, as distinct tools the agent picks from.

The grounded agent already operates two tools in this spirit, both verified, in-process and limited to aggregates: a concept search (search_okf) and an allow-listed aggregate query (analyze_rows). Each returns only what it can vouch for. The Bedrock knowledge base becomes the third tool, marked plainly as unverified narrative. The agent then offers a verified figure for number questions, grounded prose for explanations and a refusal when neither applies, and it keeps those three apart.

This split is also the right use of effort. Minutes for the knowledge base, hours for the verified bundle. Fetch narrative cheaply and put the hard work only where a figure must be correct: don't compile what retrieval gives you for free, and don't retrieve what you have to verify.

A preview of a related result

Put four tools in one agent (verified, computed, retrieved and live) and a natural question follows: does a larger model keep the modes straight? A forged news headline with an invented statistic was shown to three models, and all three repeated the number in nine of nine trials, while labelling it an unverified third-party claim. An invented health figure still reached the reader each time. Only a provenance check enforced in code held it back, in all nine. The model chosen made no difference to safety.

That changes how this benchmark reads. A verified bundle supplies the right number, which is half the job. The other half is stopping the wrong number from travelling under the same label, and that depends on the code and not on the model.

Live and independently reviewed

None of this stayed on a laptop. askOkf went live in a real Amplify sandbox (Cognito, AppSync, Lambda) and answered with "31.96% currently take insulin" plus its citation, while turning down the medical-advice question, under the genuine IAM role. Ahead of that, an independent security review worked out the contract from the source and returned contract-satisfied: the query needs authentication, the bedrock:InvokeModel permission is least-privilege and restricted to a single model, nothing can be injected, and each request is limited to one capped model call.

A note on models: Bedrock had just flagged Claude 3 Sonnet as legacy, so the OKF Lambda and the benchmark both run Claude Sonnet 4.5. Keeping the model identical on both sides is what makes the comparison fair.

The upshot

A vector database is the reflexive choice and it deserves its place for open-ended, qualitative questions. Once the chatbot has to state a figure, such as a prevalence, KPI, revenue number or compliance threshold, retrieval is the wrong instrument, because a trustworthy version of that figure was computed and never written in prose. RAG will either miss it, which is honest, or invent it, which isn't. OKF computes the figure, checks it against the source when built and ships it: four markdown files in a Lambda, nothing to provision, right on every trial.

The dozen documents weren't the problem. The absent number was, and filling the store further was never going to supply it. Compile your numbers.