Do Small-Scale RAG Apps Really Need a Vector Database?
LangChain4j MemFile keeps embeddings in JVM memory and chunk text on disk. When that's enough for small RAG apps and when to move to a real vector database.
This article is published in English.
When people start a Retrieval-Augmented Generation app, the first architecture question is often which vector database to pick. The shortlist usually includes PostgreSQL with pgvector, Milvus, Qdrant, Pinecone, Elasticsearch, OpenSearch, Weaviate and Redis. Those are strong tools, and for many production systems they are the right call. Before adding another database, though, ask a colder question: does this application actually need one?
For a platform with millions of embeddings and hundreds or thousands of concurrent users, the answer may well be yes. For a few tens of documents, a few thousand chunks, a small knowledge base, few users and modest retrieval traffic, running and maintaining a separate vector database can be unnecessary complexity. That is where lightweight embedding stores become interesting.
LangChain4j offers many implementations behind a common EmbeddingStore abstraction. The docs list more than thirty options, from simple in-memory storage to PGVector, Milvus, Qdrant, Elasticsearch and others. One community integration worth a closer look is langchain4j-community-memfile. Below: where MemFile fits, why it exists and when something this simple is enough.
RAG in LangChain4j
At a high level, a basic RAG app embeds documents, stores the vectors, embeds a question and searches for nearby vectors, then feeds the matching text to the model. The embedding store's job is focused: hold vectors and, usually, their associated text or metadata.
LangChain4j expresses that through EmbeddingStore. The interface supports adding embeddings, searching for similar ones, removing entries and working with associated content. Because of that abstraction, application code need not couple tightly to a particular database. Conceptually you declare an EmbeddingStore<TextSegment> and choose what sits behind it: in-memory, PGVector, Milvus, Qdrant, JVector, MemFile and so on. That flexibility is one of LangChain4j's strengths.
Start with the simplest option
LangChain4j's own RAG examples begin with InMemoryEmbeddingStore. For a small app, keeping a few thousand vectors inside the JVM is extremely convenient. There is no PostgreSQL, no Docker, no Milvus, no Redis, no network hop and no infrastructure to provision. The architecture collapses to the application process itself. For many workloads, that simplicity is valuable.
What MemFile actually does
The name sounds like an in-memory store that occasionally flushes to disk. The current MemFileEmbeddingStore is a hybrid:
- Embedding vectors live in JVM memory.
- Embedded content lives in local files.
- Frequently used content can sit in an optional memory cache.
It deliberately splits the two pieces that usually share one store. Embeddings stay in memory because similarity search needs them repeatedly. The corresponding TextSegment does not have to occupy heap continuously.
A typical store entry has an id, a vector and a TextSegment. The vector may hold hundreds or thousands of floats; the segment may hold paragraphs, metadata and custom attributes. If everything stays in memory, heap grows with both the vectors and the original text. MemFile keeps what search needs in RAM and moves the heavier text to disk, loading content only when a hit is returned. That helps when vector count is manageable but associated text and metadata are large. Example: 100,000 chunks with sizeable text. Vectors may still fit in memory, while every TextSegment as a Java object adds serious heap overhead. MemFile shrinks that overhead without removing the vectors required for search.
When MemFile is a good fit
It suits small to medium RAG apps where vectors fit comfortably in the JVM and TextSegment content is large enough that keeping everything on the heap hurts. Internal knowledge assistants, developer tools, documentation search, local AI apps, prototypes, desktop apps, CLI tools and other single-process systems are natural candidates.
It is especially attractive when the knowledge base is mostly static: index once, query many times. You keep embeddings hot for speed and leave chunk bodies on disk. You also avoid installing PostgreSQL, deploying Milvus, configuring Qdrant or running another container. The store stays inside the Java app; chunk content lives on the local filesystem.
Large text chunks make the case stronger. An optional LRU cache keeps hot chunks in memory while cold ones stay on disk.
When to avoid it
The biggest limit: embeddings themselves still live in memory. If the vector collection grows huge, moving only text to disk will not save you. Millions of high-dimensional embeddings consume serious heap before document text is even considered.
MemFile also currently uses brute-force similarity search. Every query is compared against all stored vectors rather than an approximate-nearest-neighbor index. That is fine for smaller collections and becomes costly as counts grow.
Horizontal scaling is another mismatch. If several service instances need to share and update the same collection, keeping local MemFile stores consistent is awkward. A centralized vector database fits that shape better.
High-churn workloads that add, update and delete documents continuously at volume are also a poor match.
A middle ground
Think of MemFile between a pure in-memory store and a full vector database. InMemoryEmbeddingStore works when vectors and content both fit in memory. MemFile becomes interesting when vectors still fit but keeping all original content on the heap is wasteful. Once vectors themselves are too large, query throughput is demanding, multiple instances need shared access or you need enterprise database features, move to an indexed or dedicated store such as JVector, PGVector, Qdrant, Milvus, Elasticsearch or OpenSearch.
Nobody is arguing that MemFile should displace dedicated vector databases. The point is to delay distributed infrastructure until the workload actually requires it. For the right workload, vectors in memory and content on disk is a remarkably simple and effective architecture.
A practical decision path
- Prototype with the in-memory store and a handful of documents. Confirm that retrieval quality is acceptable for your corpus.
- If heap pressure shows up from text segments while vector count is still modest, switch the store implementation to MemFile without changing the rest of the RAG code.
- Measure query latency and memory under realistic load. If brute-force search stays under your latency budget and the process stays healthy, stay put.
- When vector count, concurrency or multi-instance sharing starts to hurt, migrate to PGVector, Qdrant, Milvus or another indexed store. The EmbeddingStore abstraction makes that swap mostly a configuration change.
That path keeps complexity proportional to need. Many internal tools never leave step 2 or 3, and that is fine.
References
- LangChain4j embedding stores tutorial