Validating LLM Output With an Executable Knowledge Graph and Two Gates

Fluent isn't the same as correct

Language models write smooth answers, but smooth doesn't guarantee right. Suppose a model says, "This customer qualifies for a full refund." It sounds fine. Yet the model may have made up a policy, misread a date, skipped an exception or guessed at missing facts. In customer-facing flows that's unacceptable, so generating language and checking facts need to be separate jobs.

There's a real precedent. Moffatt v. Air Canada, 2024 BCCRT 149, involved an airline chatbot that said a bereavement fare could be requested after the trip, when the airline's real policy didn't allow retroactive requests. The British Columbia Civil Resolution Tribunal held Air Canada liable for negligent misrepresentation and ordered damages, interest and fees. The lesson: once a customer-facing system quotes a policy, the company can be held to it.

So the goal is to keep the flexibility of an LLM while having something verify facts before an answer reaches the user. The approach here uses RelationalAI as an executable knowledge graph that validates the structured claims an LLM produces.

An agent loop with two gates

The pattern comes from a talk by Frank Coyle of UC Berkeley at the AI Engineer World's Fair, about combining LLMs with validation and verification. The core is an agent loop with a gate on each side of tool execution. It runs like this:

1. The user asks a question.
2. The LLM decides whether it needs to call a tool.
3. Before the tool runs, the system validates the tool input.
4. The tool runs against an authoritative system.
5. After the tool runs, the system validates the output.
6. The result is appended back into the conversation.
7. The loop repeats until the model produces a final answer.

As a picture:

LLM proposes an action
   |
Gate 1 validates the tool input
   |
Tool runs
   |
Gate 2 validates the tool output
   |
LLM continues or finishes

Gate 1 checks form. It asks whether the model produced a well-formed request, which is a job for schema validation: prior to running any tool, confirm the required fields are present and the values fit claim types the system recognizes.

Gate 2 checks substance, meaning whether the outcome agrees with the domain's facts and rules. An executable knowledge graph is useful at precisely this point.

What RelationalAI contributes

With RelationalAI you put an executable knowledge graph on top of your data. It stores facts, and you also write rules that produce further facts from them. That's what separates a graph you can look things up in from one you can reason with.

In this pattern it acts as the verification layer. The LLM interprets what the user wants and emits a structured claim. RelationalAI decides whether business facts and rules support that claim, and the application then accepts it, rejects it or marks it unsupported:

LLM output
   |
Pydantic validates the tool-call shape
   |
RelationalAI derives the authoritative value
   |
Claim is marked entailed, contradicted, or unsupported

A concrete example shows how it fits together.

Step 1: get structured claims from the model

The LLM can't answer with free-form prose alone. It must also output structured claims that can be checked. For instance:

{
  "claim_type": "refund_method",
  "subject_id": "A-91",
  "value": "ORIGINAL_PAYMENT",
  "evidence": "Refund policy for GOLD uses ORIGINAL_PAYMENT."
}

That gives the system something specific to test. The question changes from "does this paragraph sound right?" to "is this particular claim true for this particular order?" That's Gate 1 at work. An unrecognized claim type, or a missing mandatory field, stops the tool call before the graph is ever consulted.

Step 2: encode the refund policy in the graph

Next, RelationalAI models the authoritative business logic. The demo uses a small refund domain with three concepts: Customer, Order and RefundPolicy. Relationships and derived properties then describe how they connect:

- an order belongs to a customer
- a refund policy applies to a customer tier
- refund eligibility depends on final-sale status and return window
- refund amount depends on eligibility and restocking fee
- refund method comes from the applicable policy

From these facts and rules the graph derives the values that need checking:

- customer tier
- refund eligibility
- refund amount
- restocking fee
- refund method
- return window
- whether shipping is refundable
- whether the order was defective

The LLM shouldn't be trusted to invent or recall these values, and should query the graph for them. Say the known facts are:

Order A-91 belongs to Customer C-184
Customer C-184 has tier GOLD
GOLD customers have a 30-day return window

RelationalAI can then derive:

Order A-91 is refund eligible
Order A-91 refund amount is 129.99
Order A-91 refund method is ORIGINAL_PAYMENT

The graph becomes a source of executable truth. A shortened version of the model looks like this:

model = Model("refund_policy_kg")

Customer = model.Concept("Customer", identify_by={"customer_id": String})
Order = model.Concept("Order", identify_by={"order_id": String})
RefundPolicy = model.Concept("RefundPolicy", identify_by={"policy_id": String})

Customer.tier = ...

Order.customer = ...
Order.amount = ...
Order.final_sale = ...
Order.request_age_days = ...

RefundPolicy.tier = ...
RefundPolicy.return_window_days = ...
RefundPolicy.refund_method = ...

Then come the derived values that Gate 2 will query:

Order.expected_refund_method = model.Property(
    f"{Order} has {String:expected_refund_method}"
)
Order.expected_eligible_for_refund = model.Property(
    f"{Order} has {Boolean:expected_eligible_for_refund}"
)

model.where(
    Order.customer(Customer),
    Customer.tier == RefundPolicy.tier,
).define(
    Order.expected_refund_method(RefundPolicy.refund_method)
)

model.where(
    Order.customer(Customer),
    Customer.tier == RefundPolicy.tier,
    Order.final_sale == False,
    Order.request_age_days <= RefundPolicy.return_window_days,
).define(
    Order.expected_eligible_for_refund(True)
)

Neither eligibility nor refund method comes from the LLM. They're computed by a knowledge graph built on business rules, which domain experts and developers are able to review, version and govern.

Step 3: validate what the model said

With the graph in place, claims can be tested. Say the model replies: "Refund order A-91 as store credit, since the customer is Platinum." It sounds plausible, but the graph can show otherwise:

• the customer is GOLD and not PLATINUM
• the refund method is ORIGINAL_PAYMENT and not STORE_CREDIT
• the order isn't marked defective
• the policy doesn't make shipping refundable

The system rejects the answer. No second LLM made that call. Executable business logic over grounded facts did. The validation code is deliberately simple: for each claim, look up the matching expected property in RelationalAI, query it, and compare the result with the value the model proposed.

EXPECTED_PROPERTY_BY_CLAIM = {
    "refund_method": "expected_refund_method",
    "eligible_for_refund": "expected_eligible_for_refund",
    "refund_amount": "expected_refund_amount",
    "customer_tier": "expected_customer_tier",
}

def check_claims(model, concepts, claims):
    Order = concepts["Order"]
    results = []

    for claim in claims:
        expected_property = getattr(
            Order,
            EXPECTED_PROPERTY_BY_CLAIM[claim.claim_type],
        )

        df = model.where(
            Order.order_id == claim.subject_id
        ).select(
            Order.order_id.alias("subject_id"),
            expected_property.alias("expected"),
        ).to_df()

        if df.empty:
            status = "unsupported"
            expected = None
        else:
            expected = str(df.iloc[0]["expected"])
            status = "entailed" if claim.value == expected else "contradicted"

        results.append((claim, status, expected))

     return results

In production you'd normalize values carefully, handle unsupported claims explicitly and log the evidence behind each decision. The core flow stays the same, though: the LLM proposes and RelationalAI verifies.

Why split the work this way

LLMs are good at expression. They can understand intent, draft replies and explain results in natural language. They shouldn't be the last word on policy, pricing, eligibility, compliance or other operational facts. The better sequence:

1. Let the LLM interpret the request.
2. Make it emit structured claims.
3. Validate the generated object with Pydantic.
4. Derive the authoritative answer in RelationalAI.
5. Return only claims the knowledge graph supports.

The LLM handles language and interaction while RelationalAI handles verification.

The loop in code

At a high level the system looks like this:

context = [user_question]

while True:
    response = llm.generate(context)

    if response.stop_reason != "tool_use":
        return response.text

    tool_call = response.tool_call

    # Gate 1: validate the model's requested action.
    claims = validate_tool_input(tool_call)

    # Tool execution: query authoritative facts and rules.
    graph = relationalai_model()
    checked_claims = graph.validate_claims(claims)

    # Gate 2: only grounded claims move forward.
    if checked_claims.has_contradictions():
        context.append(rejection_message(checked_claims))
        continue

    context.append(tool_result(checked_claims.entailed()))

The loop is routine. Who owns which duty is what counts:

• the model suggests a statement or a lookup
• the first gate inspects the shape of the suggestion
• RelationalAI works out the facts that hold
• the second gate throws out anything contradicted or unsupported
• only claims that survived end up in the reply

Beyond refunds

The refund case is small on purpose, but the same structure works for contract terms, regulatory rules, discount and pricing logic, supply-chain limits, who is eligible for which product, and what each customer is entitled to. Any workflow where an LLM speaks for a business can use a verification layer, and the costlier the mistake, the more it matters to keep generated language apart from executable truth.

References:

• https://www.youtube.com/watch?v=Sir59K8ZDPU (the conference talk)
• https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html (Moffatt v. Air Canada)
• https://docs.relational.ai/ (RelationalAI documentation)