Practical notes: OKF + RAG: The Ultimate AI Agent

This walkthrough rebuilds the path from raw materials to a working system for: OKF + RAG: The Ultimate AI Agent Architecture. The focus is operable steps, explicit checks, and code that you can drop into a repo without guessing intent. For Overview, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

The Two Memory Systems

When working through The Two Memory Systems, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

What is OKF? (Open Knowledge Format)

When working through What is OKF? (Open Knowledge Format), write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

---
type: metric
title: "Monthly Churn Rate"
description: "Official formula for calculating monthly customer churn."
owner: "data-engineering"
tags: [revenue, kpi, board-report]
timestamp: 2026-06-20T10:00:00Z
---

# Monthly Churn Rate

The official churn rate formula used in all board reports and investor decks:

    Churn Rate = (Customers Lost During Month / Customers at Start of Month) × 100

### Rules
- **Do NOT** use trial accounts in the denominator.
- **Do NOT** count plan downgrades as churn.
- Source of truth: `analytics.monthly_churn_summary` table.

### Related
- [Monthly Active Users](mau.md)
- [Revenue Dashboard](revenue_dashboard.md)

Why This Matters

When working through Why This Matters, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through Why This Matters, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

What is RAG? (Retrieval-Augmented Generation)

What is RAG? (Retrieval-Augmented Generation) works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

The Hybrid Architecture: OKF + RAG

The Hybrid Architecture: OKF + RAG works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

How the Router Decides

How the Router Decides works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

Implementation Example

Implementation Example works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

from openai import OpenAI
import os
import glob
import yaml

client = OpenAI()

# 1. Load the OKF knowledge bundle from the local directory
def load_okf_bundle(bundle_path: str) -> dict:
    """Reads all Markdown files in the OKF directory into a searchable dict."""
    knowledge = {}
    for filepath in glob.glob(f"{bundle_path}/**/*.md", recursive=True):
        with open(filepath) as f:
            content = f.read()
            # Extract the title from YAML frontmatter
            if content.startswith("---"):
                _, frontmatter, body = content.split("---", 2)
                meta = yaml.safe_load(frontmatter)
                title = meta.get("title", os.path.basename(filepath))
                knowledge[title.lower()] = body.strip()
    return knowledge

# 2. Search OKF (deterministic, keyword-based)
def search_okf(query: str, okf_knowledge: dict) -> str | None:
    """Simple keyword match against OKF titles."""
    for title, content in okf_knowledge.items():
        if title in query.lower():
            return content
    return None

# 3. Search RAG (probabilistic, vector-based)
def search_rag(query: str) -> str:
    """Placeholder for your vector DB search (Pinecone, Weaviate, etc.)."""
    # results = vector_db.similarity_search(query, top_k=5)
    return "RAG context: [retrieved chunks would appear here]"

# 4. The Intelligent Router
def answer_query(query: str, okf_bundle_path: str) -> str:
    okf_knowledge = load_okf_bundle(okf_bundle_path)

    # Try OKF first (deterministic path)
    okf_result = search_okf(query, okf_knowledge)

    if okf_result:
        context = f"[SOURCE: Official Knowledge Base (OKF)]\n{okf_result}"
    else:
        # Fall back to RAG (probabilistic path)
        context = f"[SOURCE: Document Search (RAG)]\n{search_rag(query)}"

    # Send to LLM with the retrieved context
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer using ONLY the provided context."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}
        ]
    )
    return response.choices[0].message.content

Conclusion

Conclusion works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

Operational checklist

Operational checklist works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope.

Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline.

Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.

Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

Write a short runbook: how to rotate keys, how to drain the queue, how to roll back the last ingest.

Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for 26b9ceed44f1: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.