Apache Doris 4.1: Vector Search, JSON Storage, OLAP and Lakehouse

Apache Doris 4.1 follows the 4.0 line, the release that brought vector search, hybrid search and the search() function to the project so it could serve AI agents, RAG, AI observability and real-time analytics. Version 4.1 pushes in the same direction and aims to make one engine handle both storage and retrieval for AI and search workloads. The release notes frame it around the problems teams hit when they move AI applications into production:

• keeping large volumes of AI data affordable,
• combining vector search, full-text search and structured filters in one system,
• fitting long documents and wide semi-structured records into a single table,
• answering real-time queries and observability questions for LLMs, agents and RAG,
• keeping the OLAP and lakehouse speed the project is known for, and
• merging large batch ETL with real-time pipelines.

The headline items are these:

• Vector search: new IVF and IVF_ON_DISK indexes aimed at billion-scale and larger collections.
• Full-text search: search() gains BM25 scoring and Elasticsearch-compatible syntax.
• Long-context storage: a single JSON document can now reach 100 MB, intended for agent memory and long-context work.
• OLAP speed: the project reports gains over 4.0 of 22.6% on TPC-H, 19.1% on TPC-DS and 14.3% on SSB, and first place for cold queries on ClickBench.
• Data lake: read and write for Apache Iceberg V2/V3, DDL for Apache Paimon through SQL, and a Parquet page cache worth roughly 20%.
• Compute-storage separation: a mode the project says runs at more than 2,000 companies, with better cache visibility, elasticity and cold-query behavior.

All performance numbers below come from the project's own announcement, so treat them as vendor-reported figures and verify them against your own data.

1. AI and search

Vector search

IVF joins HNSW. Version 4.0 offered HNSW as its vector index. Version 4.1 adds IVF (inverted file) for bigger datasets. The idea is to group vectors into buckets in advance and then look only inside the buckets most likely to hold neighbors, giving up a bit of precision for a big gain in speed. Compared with HNSW, IVF handles larger collections on less memory, at the price of somewhat lower recall. You enable it by setting "index_type"="ivf" in the index properties:

CREATE TABLE sift_1M (
  id int NOT NULL,
  embedding array<float>  NOT NULL  COMMENT "",
  INDEX ann_index (embedding) USING ANN PROPERTIES(
      "index_type"="ivf",
      "metric_type"="l2_distance",
      "dim"="128",
      "nlist"="1024"
  )
) ENGINE=OLAP
DUPLICATE KEY(id) COMMENT "OLAP"
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
  "replication_num" = "1"
);

IVF_ON_DISK. Libraries such as Faiss do well up to tens of millions of vectors because they keep the whole index in RAM. Reach tens of billions and that memory bill climbs steeply. Doris built IVF_ON_DISK on top of IVF, following the approach in Microsoft's SPANN research, and combines an in-memory cache with a local filesystem cache to prune vectors efficiently and cheaply.

• With compute-storage separation, it allows inexpensive, fast pruning and recall at very large scale.
• Next to DiskANN, the index build is said to cost far less, which offers a realistic route to trillion-scale search.
• Likely uses include AI knowledge bases, recommendation recall and large embedding stores.

Switching to it is almost the same as using IVF, with "index_type"="ivf_on_disk":

CREATE TABLE for_ivf_on_disk (
  id BIGINT NOT NULL,
  embedding ARRAY<FLOAT> NOT NULL,
  INDEX idx_emb (embedding) USING ANN PROPERTIES (
    "index_type"="ivf_on_disk",
      "metric_type"="l2_distance",
      "dim"="128",
      "nlist"="1024"
  )
) ENGINE=OLAP
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 8
PROPERTIES ("replication_num" = "1");

Quantization. Beyond the index algorithm, three quantization methods are supported: INT8 scalar, INT4 scalar and product quantization (PQ). For a small loss in recall they shrink the index to between a quarter and an eighth of its original memory, and combined with IVF_ON_DISK they cut hardware costs further for large retrieval systems. The DDL below shows PQ compression on 128-dimensional vectors:

CREATE TABLE product_quant (
  id BIGINT NOT NULL,
  embedding ARRAY<FLOAT> NOT NULL,
  INDEX idx_emb (embedding) USING ANN PROPERTIES (
    "index_type"="ivf_on_disk",
      "metric_type"="l2_distance",
      "dim"="128",
      "nlist"="1024",
      "quantizer"="pq",
      "pq_m"=64,
      "pq_nbits"=8
  )
) ENGINE=OLAP
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 8
PROPERTIES ("replication_num" = "1");

Faster queries. A new "Ann Index Only Scan" optimization lets a vector query skip disk IO on the raw columns, which trims overhead considerably. The project claims vector index queries run up to four times faster than in 4.0. In a test with one million vectors on a 16-core, 64 GB machine, Doris is said to reach about 900 QPS at 97% recall, enough for most production retrieval.

The announcement also cites public VectorDBBench numbers (as of January 2026) in which Doris builds vector indexes faster than Milvus, Qdrant and pgvector, crediting its data-layering architecture. In 4.1, vectors are batched in memory during construction and the index is built batch by batch, which raises build parallelism without hurting index quality.

The search() function

Where vector search handles semantic recall, full-text search deals with keyword lookup, troubleshooting, log search and text analytics. The new search() function brings it into SQL so a single query can filter and aggregate at once.

• It accepts Elasticsearch query_string syntax, which should ease migrations away from Elasticsearch.
• Operators include TERM, PHRASE, WILDCARD, REGEXP, PREFIX, NOT and NESTED, all of which can be nested and combined.
• BM25 relevance scoring is built in, with TopN optimization at the storage layer so entire result sets aren't shipped around.
• Paired with the VARIANT type, NESTED can look inside JSON arrays that are nested.
• Multi-field search supports best_fields (a precise match within one field) and cross_fields (a match spread over several fields).

A typical query looks like this:

-- Multi-condition: TERM + PHRASE + NOT evaluated in a single pass
SELECT request_id, error_msg, latency_ms
FROM inference_logs
WHERE search('
  level:ERROR
  AND error_msg:"CUDA out of memory"
  AND NOT module:healthcheck
  AND model_name:gpt*
')
  AND log_time > NOW() - INTERVAL 1 HOUR
ORDER BY latency_ms DESC LIMIT 100;
​
-- BM25 relevance scoring
SELECT request_id, error_msg, score() AS relevance
FROM inference_logs
WHERE search('error_msg:"memory allocation failed" OR error_msg:"CUDA error"')
ORDER BY relevance DESC LIMIT 20;
​
-- Nested search: query inside a VARIANT array
SELECT * FROM agent_logs
WHERE search('NESTED(steps, status:error AND tool:code_exec)');
​
-- search + aggregation: filter and analyze in one query
SELECT model_name, COUNT(*) AS error_count,
       PERCENTILE_APPROX(latency_ms, 0.99) AS p99_latency
FROM inference_logs
WHERE search('level:ERROR AND error_msg:"CUDA out of memory"')
  AND log_time > NOW() - INTERVAL 1 HOUR
GROUP BY model_name ORDER BY error_count DESC;

search() yields a boolean predicate, so it can take part in joins, window functions and subqueries, which makes full-text search an ordinary part of SQL.

JSON documents of up to 100 MB

AI agents, RAG and multi-turn, long-context applications increasingly want to keep long interaction context in one place. Doris 4.1 supports single rows holding JSON documents up to 100 MB, so full session data can live in the database: multi-turn conversations, long texts, transcripts of audio and video, traces of agent runs, logs of tool calls, and the context used for RAG. Nothing needs splitting, truncating or moving to external storage. Once written, these documents behave like other data, with filtering, conditional queries, aggregation and joins all working, so bulky AI context becomes a queryable, structured asset.

According to the project, storing large JSON this way lets teams:

• drop the dependency on a separate object store,
• avoid the logic that keeps metadata and raw content consistent,
• skip the work of chunked storage and reassembly, and
• get lower query latency, stronger transactional guarantees and simpler operations.

Segment V3 and wide tables

Search and AI workloads tend to generate wide, sparse, semi-structured data with many fields, shifting schemas and clear hot and cold access patterns. They need strong random reads, point lookups and heavy concurrent writes.

Before 4.1, Doris used the Segment V2 format, which keeps metadata in the file footer. That suits batch scans, but random reads and small-range queries must load the whole footer each time, adding disk IO and parsing cost.

Segment V3, influenced by newer formats like Lance and Vortex, separates metadata from the footer and loads it on demand. It targets metadata bloat, slow file opens and heavy random-read overhead. Candidates include wide tables, tables full of VARIANT sub-columns, jobs where cold-start latency on object storage hurts, and semi-structured AI or vehicle-telematics data that is read at random a lot. On a table with 7,000 columns and 10,000 segments, V3 reportedly opens up to 16 times faster and uses up to 60 times less memory. To enable it, set "storage_format" = "V3" in the table properties:

CREATE TABLE table_v3 (
    id BIGINT,
    data VARIANT
)
DISTRIBUTED BY HASH(id) BUCKETS 32
PROPERTIES (
    "storage_format" = "V3"
);

Sparse columns

Wide JSON usually has a long tail of rarely read paths and a few hot ones. Doris 4.1 reworks the sparse read path so the long tail doesn't bottleneck a single sparse column.

• Hot/cold separation: frequently used paths stay as columnar sub-columns, and long-tail paths move to sparse storage, which stops sub-columns from multiplying.
• Sparse sharding: variant_sparse_hash_shard_count spreads long-tail paths across several sparse columns, reducing read amplification on any one.
• Sparse cache: a read cache for sparse columns cuts repeated disk IO, decoding and deserialization.

This suits wide JSON from vehicle telematics, user profiles and event logs, anywhere a record has plenty of fields yet only dozens or a few hundred paths see frequent queries:

CREATE TABLE user_feature_wide (
    uid BIGINT,
    features VARIANT<
        'user_id' : BIGINT,
        'region' : STRING,
        properties(
            'variant_max_subcolumns_count' = '2048',
            'variant_sparse_hash_shard_count' = '32'
        )
    >
)
DUPLICATE KEY(uid)
DISTRIBUTED BY HASH(uid) BUCKETS 32
PROPERTIES (
    "storage_format" = "V3"
);

DOC mode

When the priority is write efficiency or retrieving whole JSON documents, DOC mode is the better choice. It keeps the raw JSON at write time and postpones sub-column extraction until compaction, lowering write cost and write amplification.

• Deferred materialization: raw JSON is retained on write, which reduces overhead for small batches.
• DOC sharding: variant_doc_hash_shard_count divides the Doc Store into several shards, which helps both whole-document reads and path extraction.
• Threshold: variant_doc_materialization_min_rows sets when materialization kicks in; batches smaller than that wait until compaction.

It targets LLM output, trace data, context snapshots and event replay, workloads that often return entire documents. To configure it, turn on variant_enable_doc_mode, then tune the threshold and shard count to your data volume. DOC mode and sparse columns exclude each other, so choose one, and pair it with storage_format = "V3":

CREATE TABLE trace_archive (
    ts DATETIME,
    trace_id STRING,
    span VARIANT<
        'service_name' : STRING,
        properties(
            'variant_enable_doc_mode' = 'true',
            'variant_doc_materialization_min_rows' = '100000',
            'variant_doc_hash_shard_count' = '32'
        )
    >
)
DUPLICATE KEY(ts, trace_id)
DISTRIBUTED BY HASH(trace_id) BUCKETS 32
PROPERTIES (
    "storage_format" = "V3"
);

2. Faster OLAP

Doris built its name on quick, real-time OLAP, and 4.1 keeps working on it by cutting wasted computation, moving less data and executing more efficiently, so that complex queries stay fast under realistic load.

Multi-table and wide-table benchmarks

On standard join and aggregation suites, the release reports these gains over 4.0: 22.6% on TPC-H, 19.1% on TPC-DS and 14.3% on SSB.

For wide tables, ClickBench uses 100 GB of data and 43 complex queries to stress columnar storage, vectorized execution and compression, which makes it one of the most demanding single-table benchmarks. On c7a.metal-48xl instances, the project says Doris 4.1 takes first place for cold queries and for storage space, and second place overall, behind only ClickHouse (web).

The optimizations behind those figures are described next.

Aggregation pushdown through joins

This optimization splits heavy aggregation and pushes it to both sides of a join. Each table is aggregated locally, the reduced results are joined, and the global aggregation finishes last. Compressing first and joining later cuts the data entering the join, with a clear effect on memory and latency. In the project's tests, overall performance rose by more than 200%, over half of the cases gained more than 50%, and nearly a third gained more than 100x.

Grouping sets

The engine now spots the finest-grained group and its aggregation ratio, and changes strategy from aggregating many groups in parallel to a two-step plan: finish the most detailed aggregation first to shrink the data, then derive the other groups from that result. Reported overall improvement is above 10%, with more than a fifth of the cases gaining over 20%, a best case of 160% and a worst regression under 5%.

Nested column pruning

The engine understands nested field structure more precisely. When a query touches only one subfield, only the relevant data is read and the rest is skipped, which saves disk IO. In 4.1 it works for internal tables and for external ORC and Parquet data. The reported overall gain is above 60%, and above 700% in extreme cases.

Condition cache

Large-scale analytics often reuse the same filters from one query to the next:

SELECT * FROM orders WHERE region = 'ASIA';
SELECT count(*) FROM orders WHERE region = 'ASIA';

Running an identical filter against the same segments again and again burns CPU and disk IO. The condition cache stores the per-segment filter result for later queries, trimming redundant scans and compute. For complex query workloads, the claimed improvement is above 10%.

Query cache

Analytical workloads also repeat the same aggregation over unchanged data:

SELECT region, SUM(revenue) FROM orders WHERE dt = '2024-01-01' GROUP BY region;

Each run scans the same tablets and recomputes the same aggregates. Query Cache keeps intermediate aggregation results and, when the query context matches, returns them directly, saving compute and disk IO.

CASE WHEN

CASE WHEN is a core construct for business classification, row-level logic and conditional aggregation. Version 4.1 adds branch merging, branch elimination, common sub-expression extraction, and extraction and pushdown of enum values. On the benchmark set the average gain is above 200%, with extremes beyond 50x.

3. Compute-storage separation

The project says the compute-storage separation architecture now runs for more than 2,000 production users, so stability and query performance have become the main focus areas.

File cache

File cache metadata is now persisted, which avoids the heavy disk IO that earlier versions needed to rebuild cache state on startup. A new system table, information_schema.file_cache_info, shows block-level cache details through SQL and can be queried by tablet_id, be_id, cache_path and type. That makes it easier to find hot data, diagnose imbalanced caches and notice abnormal growth.

One use is to inspect the cache composition of a single tablet:

mysql> select * from information_schema.file_cache_info where TABLET_ID = 1761571031445;
+----------------------------------+---------------+-------+--------+-------------+-----------------+---------------+
| HASH                             | TABLET_ID     | SIZE  | TYPE  | REMOTE_PATH | CACHE_PATH       | BE_ID         |
+----------------------------------+---------------+-------+--------+-------------+-----------------+---------------+
| 468448215c52334ae5bee147259b1027 | 1761571031445 | 15120 | index | | /mnt/disk1/project/filecache | 1761571031251 |
| 71bb73d34cd8ffe280b16dd329df5ba1 | 1761571031445 | 13117 | index | | /mnt/disk1/project/filecache | 1761571031251 |
| 77c6b69d1a7c4fe740a11bab5c1bbaa3 | 1761571031445 | 12249 | index | | /mnt/disk1/project/filecache | 1761571031251 |
+----------------------------------+---------------+-------+--------+-------------+-----------------------------------------------------------------------------+---------------+

Another is to aggregate the distribution to judge balance and resource usage across BE nodes:

SELECT be_id, tablet_id, type, SUM(size) AS cache_bytes
    FROM information_schema.file_cache_info
    WHERE tablet_id = 1761571031445
    GROUP BY be_id, tablet_id, type
    ORDER BY cache_bytes DESC;

Elasticity, cold queries and more

• Elastic scaling: in this mode, scaling operations across millions of tablets finish in minutes, because scheduling no longer depends on global tablet counts.
• Cold queries: prefetching based on Doris's page-scanning semantics speeds cold queries on internal tables, and parameter tuning lets Doris use more of the remote storage bandwidth.
• Large deployments: replica and tablet objects in the FE were slimmed down, cutting FE memory use by more than 30% at millions of tablets.
• Meta-service: a caching layer reduces repeated requests to the Meta Service and raises metadata throughput, and access paths for some system tables were optimized.
• Object storage cost: for high-frequency import, node-level request merging reduces the number of object storage requests and small files, and the project says overall cost can drop by up to 90%.
• Compression and encoding: a more efficient binary encoding with pre-decoding lowers runtime decode cost, and the default compression algorithm is gradually moving to ZSTD to balance ratio against decompression speed. For wide and detail tables this should mean smaller storage and better cold reads.

4. Data lakehouse

Version 4.1 moves lakehouse capability forward in format coverage, query speed and ecosystem compatibility. With Doris SQL you can read, write and manage the main open table formats such as Apache Iceberg and Apache Paimon using Doris SQL alone, without Spark or another external engine.

Lifecycle management

Creating tables, inserting, updating and deleting all run through Doris SQL, so one engine covers the whole lake workflow.

• Iceberg V2/V3 read and write: INSERT, UPDATE, DELETE and MERGE INTO are supported, along with V3 features such as Deletion Vector and Row Lineage. Ingestion, updates and deletes run end to end without outside engines.
• Paimon management: CREATE DATABASE, CREATE TABLE and other DDL run directly in Doris SQL. Writing to Paimon tables is planned for later releases.

Query performance on lake data

• Iceberg sorted writes: data can be written in a chosen sort order, producing sorted metadata (lower and upper bounds) that the engine uses to prune files. On TPC-DS this is said to improve queries by about 15%.
• Iceberg manifest caching: caching manifest-level metadata avoids repeated reads and parsing during planning, easing disk IO and CPU use. Under frequent queries, complex metadata resolution reportedly falls to the hundred-millisecond range.
• Parquet page cache: decompressed pages are cached in memory, so repeated decompression and disk reads drop, and latency under frequent queries falls noticeably. On ClickBench, overall performance rose by about 20%.

Usability of federated analytics

• Cache admission control: rules at the User, Catalog, Database and Table levels decide which queries may write to the cache, keeping cold data from evicting hot data and raising hit rates for hotspots. Rules are JSON files in a designated directory that hot-reload on change, with no FE restart, and EXPLAIN shows admission decisions so tuning can be observed.
• MaxCompute writes: CREATE TABLE and INSERT now work in the MaxCompute external catalog, completing the export path from Doris to MaxCompute. ARN cross-account access is supported, which eases integration with the Alibaba Cloud ecosystem.
• Parquet metadata inspection: a new table-valued function exposes partition, row-group and column statistics through SQL, handy for debugging file structure, checking partition pruning and tuning queries.

5. Batch processing

Doris keeps trying to unify interactive analytics and large batch processing in one engine, which reduces the architectural complexity of running several systems side by side. Version 4.1 strengthens native ETL and ELT further.

MERGE INTO

One standard SQL MERGE INTO statement handles INSERT, UPDATE and DELETE (an upsert), simplifying incremental merge jobs such as CDC and lowering development and maintenance effort:

MERGE INTO target t
USING source s
ON t.id = s.id
WHEN MATCHED THEN
    UPDATE SET t.value = s.value
WHEN NOT MATCHED THEN
    INSERT (id, value) VALUES (s.id, s.value);

Stronger spill-to-disk

Large joins, high-cardinality aggregations and global sorts are memory-hungry, and 4.1 extends spill-to-disk across those operators:

• recursive, multi-level spilling lets intermediate data move through several stages and avoids sudden memory pressure,
• it works across JOIN, aggregation and sort operators, and
• dynamic spill triggering balances memory use against disk disk IO to keep queries stable.

With these changes, the project says Doris can run all TPC-DS 10 TB queries on one BE node with 8 GB of memory, which it presents as stronger performance at lower resource cost than traditional Hadoop, Greenplum or Spark setups. As with every number here, that's a claim worth reproducing yourself.

6. Usability

Version 4.1 rounds out the analytical syntax, improves expressiveness in complex scenarios and works on compatibility with other systems.

Execution engine extensions

UNNEST. Native UNNEST simplifies analysis of arrays and nested structures in JSON and log data without awkward function rewrites:

SELECT user_id, tag
FROM user_profile,
UNNEST(tags) AS t(tag);

Recursive CTE. You can now write recursive queries to walk org charts, hierarchies and graph paths, or to analyse other relationships that need traversal:

WITH RECURSIVE org_tree AS (
    SELECT id, parent_id, name
    FROM org
    WHERE parent_id IS NULL
​
    UNION ALL
​
    SELECT o.id, o.parent_id, o.name
    FROM org o
    JOIN org_tree t ON o.parent_id = t.id
)
SELECT * FROM org_tree;

ASOF JOIN. This joins on the nearest match along a time column (<= or >=), useful for finance, IoT and monitoring workloads that need time-series alignment and nearest-value lookups:

SELECT t1.ts, t1.value, t2.price
FROM trades t1
ASOF JOIN prices t2
ON t1.symbol = t2.symbol
AND t1.ts >= t2.ts;

Ingestion and the write path

• Continuous S3 import: continuous jobs can be created from S3 sources, and the system detects new files and keeps importing them, suiting incremental ingestion from object storage.
• MySQL and PostgreSQL sync: Doris can now keep itself updated from MySQL and PostgreSQL in real time, covering full initialization and incremental sync. That simplifies building real-time pipelines from operational databases, for real-time warehousing, data consolidation and analytics acceleration.

Write and update models

• Adaptive write scheduling: the MemTable flush thread pool adjusts to cluster load and matches write concurrency to current conditions, improving throughput, resource use and stability under heavy writes.
• Multi-stream updates on the primary key model: sequence_mapping supports merging updates from several streams. Different streams can update different columns of one table and merge according to their own sequence fields, which suits concurrent real-time updates plus offline backfills.
• Routine Load: flexible partial updates on non-primary-key columns, dynamic parameter changes and adaptive batching improve stability at high throughput.
• Import auditing: Stream Load activity can be logged into the audit system table, so history, debugging and audits share one place to query.

TIMESTAMPTZ

Global workloads such as cross-region logs, user-behavior analytics and financial transactions carry explicit time zones. With a zone-less DATETIME, conversion happens in the application, which invites mistakes and extra development. Doris 4.1 introduces TIMESTAMPTZ (timestamp with time zone) for standard zone-aware handling:

• values are stored in UTC when written,
• reads are converted to whatever time zone the session uses,
• input with or without a zone is parsed and converted for you, and
• TIMESTAMPTZ and DATETIME can be converted into each other and work with existing functions.

This moves time-zone handling out of application code and into the database kernel, improving consistency across systems and lowering complexity for multi-zone data.

What it adds up to

Taken together, 4.1 is a considerable step toward positioning Doris as a core data layer for AI applications and real-time analytics: the IVF index extends vector search to far larger collections, 100 MB JSON documents make room for long-context conversations, and OLAP and lakehouse performance show measurable gains over 4.0. The pitch is one infrastructure for vector search, full-text search, real-time analytics and a lakehouse, so that fewer standalone vector databases, search engines and query engines are needed. Whether that consolidation pays off will depend on your scale, your latency targets and how the vendor-reported numbers hold up on your own workload.