Why RAG Isn't Enough: A Semantic Layer for Graph, SQL and Vector Queries

Language models have made it easy to put enterprise questions in plain words. Getting an answer is a different thing from knowing what the question means across several data systems. Take this one:

Picture an analyst asking: which exchange-traded funds own firms that partner strategically with NVIDIA, manage over a billion dollars, and get favorable coverage in recent research notes?

It reads as a single question, but it actually contains three kinds of work:

• NVIDIA, strategic partner, ETF holding: walking relationships in a graph.
• AUM above $1B: exact numeric filtering.
• Positive latest research: understanding unstructured text.

Underneath sits a deeper issue. What does AUM mean in the underlying data model, and which source is the authority for it? Before a system can answer correctly, it has to grasp the meaning of the data and not only the data. Semantic engineering aims at that gap.

Where RAG stops

The typical RAG pipeline retrieves relevant passages and lets the model answer from them. That's excellent when the job is to find relevant text and respond from it. Enterprise questions often need more than similarity, and pure vector retrieval fails in four recognizable ways.

Fragmented retrieval

Picture a product manual cut into three chunks:

Chunk 1: Atlas Enterprise — vendor, category
Chunk 2: Pricing — base fee, usage fee
Chunk 3: Risks — lock-in, migration, data residency

Someone asks for the pricing and risks of Atlas Enterprise. The product name may appear only in the first chunk. The second and third hold the answer, but after chunking they no longer carry the entity context. Metadata enrichment, parent-child retrieval and reranking all help, yet fragmentation remains a basic weakness.

No relationship traversal

Now ask which ETFs hold companies that partner with NVIDIA. The reasoning path isn't one keyword match:

NVIDIA
   ↓ HAS_STRATEGIC_PARTNER
Company
   ↓ HELD_BY
ETF

Vector search will find documents that mention NVIDIA, its partners or ETFs. What it can't do reliably is conclude that those mentions form a particular multi-hop relationship. A graph can walk the path explicitly:

?nvidia corp:hasStrategicPartner ?company .
?etf etf:hasConstituent ?company .

Similar text is not the same as a connection between entities.

Entity ambiguity

A name like "NVIDIA" can point to several things:

NVIDIA     → Company
NVIDIA International  → Subsidiary
NVIDIA AI Enterprise    → Product

Their embeddings may sit close together, yet they're different entities of different types. An ontology supplies identity, type and relationships, which text similarity can't.

No exact numeric filtering

A request such as "customers with annual revenue above $1M and churn probability below 5%" is plain SQL:

SELECT customer_id
FROM customer_metrics
WHERE annual_revenue > 1000000
  AND churn_probability < 0.05;

Embeddings capture similarity. They don't dependably carry out greater-than, less-than, BETWEEN, aggregation or ranking.

One question, several engines

The opening question needs more than one retrieval strategy, and each engine suits a different type of reasoning:

Vector → meaning and unstructured text
BM25   → exact lexical matching
Graph  → relationships and multi-hop traversal
SQL    → filters, numbers and aggregation

The task has stopped being retrieval and become federated query execution. Federation raises three harder problems: decomposition, routing and mapping.

Decomposition

The system has to cut the natural-language question into units it can execute:

1. Resolve NVIDIA as a Company
2. Find its strategic partners
3. Find ETFs holding those companies
4. Filter AUM > $1B
5. Retrieve the latest research reports
6. Identify positive views

Routing

Each unit needs to go to the source that is authoritative for it:

relationships → Graph
AUM           → SQL
research view → Vector Search

Connecting databases isn't enough. The system has to know which source answers which part.

Mapping

This is often the toughest piece. A user says AUM, while the database may hold fields like these:

NET_ASSET_AMT
AUM_USD
FUND_NET_ASSET

Which one is the governed concept? Similarly, an ETF could be fund:ETF in the graph, dim_fund.fund_id in SQL and metadata.product_id in the vector index. The system needs a bridge from business vocabulary to physical schemas. This is schema grounding, and without it an LLM can write perfectly believable SQL against columns that don't exist.

The semantic layer

A semantic layer sits between what people mean and how data is stored. Rather than showing an LLM hundreds of raw tables and properties, you expose concepts:

Company
Strategic Partner
ETF
Constituent
Assets Under Management
Research Report

and attach meaning to each:

Assets Under Management
 ├─ business definition
 ├─ currency / unit
 ├─ effective date
 ├─ authoritative source
 └─ physical column

The layer can then take part directly in the query lifecycle. That changes what an ontology is for. It stops being static documentation and becomes executable context for the AI system.

How it might be organized

For a system that mixes a relational database, a knowledge graph and vector search, the layer could be laid out like this:

semantic_layer/
├── ontology/
│   ├── tbox.ttl              # classes and relationships
│   └── vocabulary.yaml       # business terms and synonyms
├── schemas/
│   ├── rdb.yaml              # tables, columns, types
│   ├── graph.yaml            # entities, predicates, graph paths
│   └── vector.yaml           # indexes and document metadata
├── semantics/
│   ├── metrics.yaml          # governed metrics such as AUM
│   ├── mappings.yaml         # concept → physical source mapping
│   └── relationships.yaml    # cross-domain relationships
├── query/
│   ├── routing.yaml          # Graph vs SQL vs Vector routing
│   └── examples.yaml         # representative query plans
└── validation/
    └── rules.yaml            # allowed fields and business rules

In that layout, tbox.ttl is where classes like Company and ETF are declared, along with the HAS_STRATEGIC_PARTNER link between them, while rdb.yaml documents the actual tables and columns. mappings.yaml links the Assets Under Management concept to the column fund_metrics.net_asset_amt. Finally routing.yaml lets the planner decide if a step goes to the graph, to SQL or to vector search.

The exact files depend on the implementation. What matters is writing down meaning, physical mappings and valid execution paths explicitly, instead of making the LLM rediscover them on every query. Four tasks benefit directly:

• Disambiguation: the company and similarly named products belong to different entity classes.
• Routing: partner and holding relationships come from the graph, while AUM comes from the warehouse.
• Schema grounding: Assets Under Management resolves to fund_metrics.net_asset_amt.
• Multi-hop reasoning: NVIDIA to strategic partner to ETF.

That is why semantic engineering is wider than drawing an ontology. The point is to make business meaning usable at runtime, through a shared vocabulary, an ontology, metric definitions, relationships, schema mappings, business rules, source metadata and validation checks. Seen this way, a semantic layer resembles a compiler stage for enterprise data far more than a glossary.

From layer to runtime

Plenty of agent designs today look like this:

User → LLM → Tools

The LLM receives tool descriptions and decides what to call. A semantic architecture can be more constrained:

User
 ↓
Semantic Resolution
 ↓
Logical Query Plan
 ↓
Validated Execution
 ↓
Evidence
 ↓
LLM

The model is spared from guessing its way through each table, metric, graph predicate and business rule. It works inside a bounded semantic environment. The target is a better interface between model and enterprise data, not simply a bigger model.

Why an open interchange format matters

Semantic definitions themselves tend to fragment. A company might define Net Revenue separately in its data platform, BI tool, catalog, analytics app and AI agent, and over time the definitions drift apart.

Open Semantic Interchange (OSI) set out to fix this. The initiative joined the Apache Incubator in June 2026 and continues as Apache Ossie (incubating). It aims to give systems a neutral way of exchanging semantic metadata. The core specification is written in YAML and covers datasets, fields, relationships, metrics and AI context. A cut-down model looks like this:

version: "0.2.0.dev0"
semantic_model:
  - name: investment_products
    datasets:
      - name: etf
        source: analytics.dim_etf
        fields:
          - name: aum
            datatype: Decimal
            ai_context:
              synonyms: ["assets under management", "net assets"]
    metrics:
      - name: total_aum
        expression:
          dialects:
            - dialect: ANSI_SQL
              expression: SUM(etf.aum)

The syntax matters little. What counts is that aum gets one definition, with its physical source, datatype, metric logic and AI-facing synonyms, and that definition travels as portable metadata instead of being rewritten inside each tool.

┌→ BI
                  ├→ Analytics
Semantic Model ───┼→ AI Agents
                  ├→ Data Catalogs
                  └→ Data Applications

A catalog could show the governed definition and lineage of AUM, and an AI agent could read the same metadata to resolve the phrase assets under management to the right field rather than guessing from raw schemas.

Beyond that, Ossie validates models against a JSON Schema, allows several expression dialects, and defines a converter model for moving between vendor formats. Planned work includes richer metrics, composability, catalog integration, finer relationship semantics and ontology representation.

The ontology direction is especially interesting for AI, because metrics and dimensions alone don't tell an agent facts like these:

Customer is an Organization
Company HAS_SUBSIDIARY Company
Company OWNS_PRODUCT Product
Document DESCRIBES Entity

If semantic models and richer ontologies converge, AI systems could receive portable business meaning and not just portable schemas.

Back to the NVIDIA question

A semantic query system can process the opening question step by step, with every decision grounded in explicit metadata. Without a semantic layer the LLM must infer meaning, source, schema and relationships at each step, and a single wrong assumption can break the whole chain.

The larger shift

Enterprise AI has moved through stages:

LLM + Prompt
     ↓
LLM + RAG
     ↓
LLM + Tools
     ↓
LLM + Federated Data
     ↓
Semantic Layer + Federated Execution + LLM

Retrieval says which documents look relevant. A graph says how entities are linked. SQL says which records satisfy exact conditions. The semantic layer handles the question that comes first: how does this question translate into the vocabulary of our data?

Systems will keep gaining databases, graphs, documents, APIs and tools, and wiring up more data alone won't cut it. The meaning has to be wired up too. Enterprise AI may rely less on handing a model extra data and more on giving it a firm understanding of what that data stands for.