Building a Slack RAG Assistant Without a Separate Vector Database

StarRocks is a fast open-source analytical database with a unified execution engine, flexible deployment and strong performance on real workloads. As its community has grown, so has the support load: maintainers keep fielding the same questions about the docs, GitHub issues, release notes and old Slack threads.

Rocky is the Slack assistant built to absorb that load. Its job is plain: take repetitive Q&A off the maintainers while keeping every answer grounded in the project's documentation and related sources.

The interesting part is the architecture. Rocky runs on StarRocks itself. Document chunks, keyword lookup, vector retrieval and similarity scoring all live in one OLAP table, so the assistant that answers questions about the database also runs on that same database. The whole thing amounts to about 600 lines of Python, one table and a Gemini API key.

The RAG foundation

The usual first move when building retrieval-augmented generation is to bring in a dedicated vector database. Every extra component brings a separate deployment, a backup plan and its own consistency model, and for a small community bot that overhead is hard to defend.

So everything stayed in StarRocks. The document chunks and their 768-dimensional Gemini embeddings sit in an ordinary OLAP table using the PRIMARY KEY model, and retrieval is a SQL query. The principle: if your analytical database already supports array columns and cosine similarity functions, you don't need a second data system for vector search.

The table looks like any other:

CREATE TABLE docs (
    id     BIGINT NOT NULL,
    path   VARCHAR(512),
    `index` INT,
    `text`  STRING,
    vector ARRAY<FLOAT>   -- 768-dim Gemini embedding
) ENGINE = OLAP
PRIMARY KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1;

And retrieval is one query using cosine similarity:

SELECT path, `index`, `text`,
       approx_cosine_similarity([0.012, -0.034, ...], vector) AS similarity
FROM docs
ORDER BY similarity DESC
LIMIT 8;

There's no SDK, no client library and no second system. From Rocky's side, the "vector database" is simply one more table it can already query.

From Slack event to grounded answer

The flow is deliberately minimal. An @Rocky mention in Slack triggers the bot, and the bot hands reasoning to Gemini 3 Flash using native function calling. The model decides whether to search the documentation or query Google, and may cycle through up to ten tool-call rounds in a turn.

1. Slack delivers the mention. The slack_bolt library for Python receives the event over Socket Mode and pulls out the user's query along with the thread context.
2. Gemini reasons and calls tools. The model gets a system prompt with strict honesty rules and two tools: search_starrocks_doc, a client-side vector retrieval, and google_search, server-side grounding through Gemini's built-in web search.
3. Retrieval runs inside StarRocks. The search tool embeds the query with gemini-embedding-001 using task_type=RETRIEVAL_QUERY and then runs the cosine-similarity SQL shown earlier.
4. The model writes the answer. Gemini assembles the retrieved chunks into a Markdown reply, which Rocky converts to Slack's mrkdwn format before posting.
5. Telemetry goes to observability. Every model call, tool use and token tally becomes an OpenTelemetry span, with thread_ts serving as the session ID.

The bot is roughly 600 lines of Python in one file. The surrounding tooling for chunking, embedding generation and index building adds only a few thousand more, and storage, retrieval and similarity scoring are all left to StarRocks.

Why this stack suits a light RAG app

Three choices keep the operational footprint small while still producing useful answers.

Primary key table plus Stream Load for swappable docs

The corpus isn't a streaming workload. When the docs change, the whole docs/en/ directory gets re-chunked, embeddings regenerated, and the entire table reloaded through Stream Load:

curl --location-trusted -u root: \
     -H "format: json" \
     -H "strip_outer_array: true" \
     -T docs_with_embedding.json \
     -XPUT http://host:8030/api/starsage/docs/_stream_load

One HTTP request loads around 5,000 rows, each with a 768-dimensional float array. At this scale Stream Load fits well: no broker to maintain, no Flink pipeline to run, no ongoing ingestion infrastructure. For thousands to tens of thousands of rows with embedded vectors, it's the simplest route.

The Primary Key model guarantees atomic replacement. Queries that arrive mid-reload see a consistent snapshot and never a half-updated corpus. That is the main reason for choosing a Primary Key table over a Duplicate Key one. It isn't about row-level upserts, since the workload is full rebuilds. It's that a PK table plus Stream Load makes each reload atomic, so Rocky serves either the old corpus or the complete new one and never a blend of the two.

Retrieval as a tool, not context stuffing

Documents aren't preloaded into the system prompt. search_starrocks_doc is exposed as a callable tool, and the model chooses when and how often to use it.

In practice a message like "hi Rocky" triggers no retrieval at all and costs next to nothing. A nuanced question, such as how GROUP_CONCAT behaviour differs between 3.5 and 4.0, triggers several calls: the model queries once per version and cross-checks the results. Retrieval is capped at eight calls per turn. Gemini's server-side google_search handles release notes and GitHub issues beyond the docs, and Rocky covers a wide area without a separate web-scraping pipeline.

An embedding cache for predictable cost

Community questions cluster. How to add a bitmap index and how to deploy on Kubernetes come up several times a week. A local SQLite cache keyed by query text and task_type (because RETRIEVAL_QUERY and RETRIEVAL_DOCUMENT yield different vectors) removes redundant embedding calls, so popular queries cost nothing.

On the LLM side, Gemini's cached input tokens bill at 25 percent of the standard rate. With the embedding cache, the per-answer cost is reported to sit around $0.001, which is sustainable for a community-funded project that runs indefinitely.

Six lessons from real traffic

These come from actual community use and not from benchmarks, and most carry over to any domain assistant on a chat platform.

1. Prompt changes beat retrieval tuning for hallucinations

Early versions invented Red Hat mirror names, made up release-note URLs and hallucinated version numbers, and community members clicked through to 404 pages. For a Q&A bot, a confident wrong answer is worse than none.

The fix wasn't better retrieval. It was a prompt change: an explicit honesty clause telling the model not to fabricate facts and, when it can't answer reliably, to say it isn't certain, point to authoritative sources and suggest asking the community. The effect was immediate. Retrieval matters, but prompt engineering is often the quicker and cheaper way to suppress hallucination.

2. Fix chunking before touching retrieval

StarRocks docs contain many MDX import lines, Markdown tables that straddle chunks and very long SQL DDL statements. Feeding that raw into embeddings is garbage in, garbage out. The chunking pipeline got heavy investment:

• Hierarchical splitting by header. Chunks break at Markdown headers, with the whole header path stored as metadata, giving the model a sense of where a chunk belongs in the document hierarchy.
• Noise filtering. MDX import lines are stripped, and code blocks over 4,000 characters are dropped because they dilute the semantic signal without helping answers.
• Simpler output tables. Example output shown as MySQL-style tables is condensed into structured descriptions in place of raw ASCII grids.

Chunking took roughly ten times the effort of integrating vector search. When recall disappoints, examine how the text was chunked before swapping embedding models or fiddling with index settings.

3. RAG is a tool, not a pipeline

When retrieval is a function call and not baked into every request, the model can skip it whenever it isn't needed. Greetings, follow-up clarifications and simple factual questions never touch the vector index. Combining a client-side tool (search_starrocks_doc) and a server-side one (google_search), something that the Gemini SDK has fully supported since version 1.70, also gives Rocky a fallback for content outside the corpus with no custom scraper.

4. Follow the platform's session model

Slack threads come with thread_ts identifiers. Rocky uses that directly as the session ID, saving history to a JSON file named after it and including the six most recent turns in each request:

contents = []
for turn in history[-MAX_CONTEXT_MESSAGES:]:
    contents.append({"role": turn["role"], "parts": [{"text": turn["text"]}]})
contents.append({"role": "user", "parts": [{"text": query}]})

That's about twenty lines of code, far simpler than running a session database.

One recurring piece of user feedback says "Rocky has no memory". That comes from the fact that in Slack, Rocky only replies when it's @-mentioned, which makes it look forgetful. It's a product interaction issue and not a technical one.

5. Instrument early

After launch, the team wanted answers to basic questions. How many questions did Rocky handle this week? Where did the tool calls concentrate by topic? What was a typical reply costing? Which replies got pushed back on, corrected or followed up? Without traces the system was a black box, so a full OpenTelemetry layer went in afterwards. The lesson is to add tracing before launch. Without it, token usage, cost and tool-call patterns are invisible, and cost optimisation, hallucination attribution and weak-question analysis all get much harder.

6. Cache aggressively and track every dollar

Embedding APIs look cheap per call, but frequent community questions add up fast. The SQLite embedding cache together with Gemini's cached-token pricing keeps unit costs near $0.001 per answer. For a community-maintained project, that's the difference between "sustainable indefinitely" and "we need to talk about the budget".

Honest tradeoffs

Rocky is a lightweight RAG assistant and not a production support system, and a few limits deserve to be stated:

• Hallucinations are reduced, not eliminated. Prompt guardrails help but don't guarantee correctness. Well-documented topics work best, while edge cases and version-specific quirks still trip it.
• Wordy answers. Like most RAG bots, it over-explains. A switch for short, summary-style answers is planned.
• No memory across threads. Context lives inside one Slack thread, so a user who asked about materialised views yesterday starts from zero today.
• No human escalation yet. After several failed attempts it should tag a maintainer automatically, but that workflow isn't shipped.
• Limited troubleshooting. Deeper diagnosis will need to be combined with an open-sourced debugging skill the project released recently.

All of these are active roadmap items informed by community feedback.

What Rocky shows

Rocky is a small project, 600 lines of Python plus one table and one API key, but it supports a larger point. For lightweight AI applications, an analytical database with array columns and similarity functions can remove the need for a separate vector database. One table handles document storage, keyword search and vector retrieval, so the operational surface stays small and there's one system to look after and not two.

The bot handles real community traffic every day, and users rate it 4.2 out of 5 on average, according to the project, with each answer costing less than a tenth of a cent. For anyone building a similar domain assistant who wants simple infrastructure, the single-table pattern is worth a look, especially if StarRocks is already in the stack.