MiniCPM5-2B vs 4B Rivals: Why an Agentic Harness Tells a Different Story

A 2.5-billion-parameter model that supposedly outscores 4B rivals on elite coding benchmarks sounds like a bargain. Wire the same model into a real multi-turn agentic harness, though, and its performance falls by as much as a factor of three. Single-shot evaluations say little about the messy reality of production software engineering. Looking past aggregate index scores and reproducing throughput locally yields a concrete framework for when this model is the right tool.

The claim

OpenBMB published MiniCPM5-2B on September 7, 2026. It is a dense causal language model with 2.516 billion parameters, an Apache 2.0 license and a native context of 131,072 tokens. According to the release, it is the best open model under 4B, averaging 53.9 across 34 benchmarks and ahead of several 4B-class competitors, Qwen3.5-4B included, with these scores: MATH-500 at 94.6, AIME 2026 at 86.5, LiveCodeBench v6 at 69.1 and SWE-bench Verified at 46.4.

Nobody is waving around a cherry-picked leaderboard screenshot here. The launch was an organized, serious one: eight artifacts landed together. The set covers checkpoints for the base stage, mid-training and SFT alone, then the final model tuned with RL and OPD, GGUF and MLX conversions, a 4-bit GPTQ build, and a speculative-decoding draft model named MiniCPM5-2B-DSpark. A lab that quietly games a benchmark does not ship like that. This one seems to expect scrutiny and arrives with files for each target runtime, from llama.cpp and Ollama through LM Studio, vLLM and SGLang to Transformers, MLX and FlagOS multi-chip.

At full strength, the claim is that a model about 44 percent smaller than Qwen3.5-4B beats it on math, code and software-engineering benchmarks. If that held without caveats, it would be a meaningful data point about scaling for reasoning-focused post-training. It does not hold without caveats, and the interesting part is exactly where it breaks down and why.

What is under the hood

Knowing what is actually being run helps, because the architecture accounts for both strengths and blind spots. MiniCPM5-2B is a decoder built much like LlamaForCausalLM: 42 transformer layers using grouped-query attention with 16 query heads sharing 2 key/value heads. A ratio of 8 to 1 is bold for a model of this size. It shrinks the KV cache per token considerably, which pays off when long tool-call histories must be kept or many agent sessions share one GPU. No exotic sparse or linear attention is used, even though relatives in the MiniCPM family, MiniCPM-SALA and MiniCPM4, experimented with hybrids. The team stayed with the familiar mechanism and invested elsewhere.

Where it went is context length and training-data curation. A native 131,072 tokens is unusual below 3B parameters, since most models of that size top out around 32K to 64K before quality slips. For agentic work, where the model must keep track of an ever-longer log of calls, diffs and shell output, that headroom counts for more than raw parameter count.

The model handles English and Chinese only, with no vision and no audio. That is a deliberate narrowing, not an oversight, because every parameter spent on multimodal alignment is one not spent on reasoning and tool use. OpenBMB also released the post-training data: UltraData-SFT-Agent-2609 with 500K agent-trajectory samples and UltraData-RL-2609 with more than 80K RL samples. That is real transparency, since the agentic behavior baked into the model can be inspected instead of guessed from benchmark deltas.

With the architecture understood, the benchmark claims turn from abstract numbers into testable hypotheses about one specific decoder and one training recipe.

The first crack: 23 versus 15

Trouble appeared within a day of launch. OpenBMB's announcement cited an Artificial Analysis Intelligence Index score of 23 for the model, while Artificial Analysis's independent tracker showed 15. A later revision of the index put it near 13. The predictable reaction was to ask whether OpenBMB had inflated its own numbers.

It had not, and the case deserves a careful look, because reflexive outrage is the wrong response to a benchmark discrepancy and also the most common one. The methodology note from Artificial Analysis explains the gap. In summary, the difference does not come from the model getting worse. The evaluation changed: private test sets now carry 40 percent of the total weight, up from earlier versions, and version 4.2 added two new components, the AA-Briefcase agentic evaluation and a long-context test.

OpenBMB's figure of 23 was measured on Intelligence Index v4.1.1. By the time people compared numbers, the public tracker had moved to v4.2, with different weighting and two components that did not exist when the launch figure was produced. Setting 23 on one version against 15 on another does not expose a lie. It compares two different measuring instruments and then expresses surprise that they disagree.

The real failure belongs to early coverage that printed both numbers side by side without saying they came from different index versions, which invited a gotcha conclusion the data never supported. A writeup citing an aggregate index score should name the version beside it. If it does not, 23 and 15 cannot be compared and are really two different questions sharing a label.

That crack can be resolved. A second one cannot be closed nearly as neatly.

The real gap: agentic harnesses

The number that ought to change how this model is perceived comes from Terminal-Bench v2.1. MiniCPM5-2B scores between 8.6 and 9 there, while Qwen3.5-4B scores 25.8. That is about a threefold gap, and it left MiniCPM5-2B in eighth place on that benchmark even though it tops the Intelligence Index among models its size. SWE-bench Pro shows the same picture, with 14.4 for MiniCPM5-2B against 28.2 for Qwen3.5-4B, roughly a twofold gap.

Set that beside the launch claim. OpenBMB's SWE-bench Verified result of 46.4, cited as beating Qwen3.5-4B, is one pillar of the beats-4B headline. But SWE-bench Pro is not the same test with a scarier name. It uses a broader and more adversarial set of real repository issues and runs through an actual agentic harness, where Verified is closer to single-shot patch generation. Terminal-Bench sits even further from single-shot evaluation. It is a live terminal environment in which the model must plan, run commands, read the output and recover from its own mistakes over many turns.

An independent evaluation by MindStudio put it bluntly: the model shows promise on some agentic benchmarks but falls well behind Qwen on the harder agentic tests, SWE-bench Pro and Terminal-Bench.

This result, far more than the index-version mismatch, ought to adjust expectations. Verified rewards writing a correct patch for a tidy, curated task, which is single-shot generation presented as engineering. Terminal-Bench and SWE-bench Pro reward staying on track across many turns of tool use, recovering from a shell command that failed, and avoiding a collapse three steps into a plan. The model is evidently good at the first kind of job and evidently weaker at the second. Anyone using it as an autonomous coding sub-agent, in place of a one-turn patch writer, will live in the second kind.

This gap is real and unresolved, and no methodology artifact explains it, so it deserves checking by hand instead of trusting either side's aggregate.

Reproducing it locally

Nobody seems to have published a systematic sweep of tokens per second by quantization level on real hardware. Coverage mostly repeats the benchmark table from the release or one anecdotal speed figure. What follows are single-run numbers, not a controlled sweep over seeds and batch sizes, so they serve as a prompt for measuring locally and not as a conclusion.

Memory footprint by quantization at 8K context:

• Q4_K_M: about 2.78 GB
• Q8_0: about 3.7 GB
• BF16/FP16: about 6.3 GB

Reported throughput from independent sources: about 144.23 tokens per second on an RTX 3070 Ti with 8 GB of VRAM, using the Q4 quant in LM Studio with a single stream, which is a modest consumer card. At the top end, a Hacker News discussion of the release mentioned 32 sub-agents running concurrently on one DGX Spark, averaging 931 tokens per second combined and peaking at 1,364. That second figure fits the intended use. Winning a single-stream latency race is not the aim. The GQA ratio and small footprint are designed for serving many sessions at once, and 32 agents sharing a node suits them well.

The quickest way to reproduce anything is Ollama:

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Recommended flags for flash attention + quantized KV cache
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve &

# Run directly from Ollama's registry
ollama run openbmb/minicpm5-2b

For a specific quantization pulled from Hugging Face into llama.cpp:

huggingface-cli download openbmb/MiniCPM5-2B-GGUF \
  MiniCPM5-2B-Q4_K_M.gguf \
  --local-dir ./minicpm5-2b-gguf

And for OpenAI-compatible serving at scale, vLLM:

pip install vllm

vllm serve openbmb/MiniCPM5-2B \
  --max-model-len 131072 \
  --dtype bfloat16 \
  --port 8000

# Query it like any OpenAI chat completion endpoint
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "openbmb/MiniCPM5-2B", "messages": [{"role": "user", "content": "Write a quicksort in Rust"}]}'

Quality varies by task in ways that aggregate benchmarks do not show. MindStudio's hands-on GGUF testing found that the Q8_0 model handled self-contained coding and verifiable math and physics reasoning well, but failed badly at a schema-specific SQL debugging task and at multilingual translation, giving confident but factually wrong answers.

The confidently wrong behavior counts for more than any accuracy percentage. Loud failures are easy to catch in review, but a failure delivered with certainty is how a flawed SQL migration reaches production. Reproducibility has another trap: llama.cpp behaves quite differently depending on its build version and on how the frontend applies the chat template. When local numbers differ from someone else's, check the build before deciding the model changed.

Put together, local evidence agrees with the gap on Terminal-Bench and SWE-bench Pro. Bounded, checkable tasks go well, and things get shakier wherever precise knowledge of an external schema or open-ended multi-step judgment is required. Independent sources agree on this pattern consistently enough to rely on it, which leads to the decision actually at stake.

When this model is the right pick

Once the benchmark dispute is stripped away, a useful decision framework remains, and it does not amount to a verdict of good or bad.

Strong fits:

• Swarms of many sub-agents running at once. Memory efficiency from GQA and the 931 tokens per second combined across 32 agents on a DGX Spark are the figures that count, far more than the index score.
• On-device and edge deployment. A Q4 footprint of 2.78 GB runs comfortably on consumer hardware, phones and mini-PCs through GGUF or MLX.
• Workflows that hold long tool logs or documents and actually use the 131K window, without a separate retrieval layer.
• Local inference where latency matters, using MiniCPM5-2B-DSpark as the draft model for speculative decoding.

Weak fits:

• Free-form or creative code generation. Outside testing saw weaker results than dedicated coding models on tasks such as building HTML pages or SVG art, since the model's strength lies in reasoning, math, tool use and function calling.
• Schema-specific SQL debugging and multilingual translation, where it has been seen failing with confidence and not with grace.
• Anything routed through a harder agentic harness. A production coding agent facing multi-turn terminal sessions or Pro-tier SWE-bench tasks will meet the two-to-threefold gap against Qwen3.5-4B, and that is no rounding error.

A word of fairness, since hot takes skip it: the objection that comparisons use stale baselines like Qwen2.5-2B or Gemma-2B fits secondary commentary and not the official model card. The card compares against up-to-date models, Qwen3.5-2B, Gemma-4-E2B-it, granite-4.2-3B and LFM2.5-2.6B. Criticism of this release should aim at its truly weak point, the agentic-harness gap, and avoid a baseline complaint the primary source refutes.

Benchmark claims are benchmark-specific

Looking past this one model, the broader lesson is the part worth keeping. The beats-4B claim is true but narrow. It stands on OpenBMB's math and single-shot coding suite, AIME, LiveCodeBench and SWE-bench Verified, and also on the Artificial Analysis aggregate once the version is fixed. It collapses on Terminal-Bench and SWE-bench Pro, where Qwen3.5-4B leads by two to three times. All of that is simultaneously correct.

The routine for any future sub-4B claim: run it through at least one adversarial benchmark shaped like real work, Terminal-Bench, SWE-bench Pro or something equivalent, before letting a single-shot suite predict how an agent will do. Producing one patch and carrying out a multi-turn task are separate skills, and a model can shine at the first while being average at the second. The tweeted aggregate never says which one is on offer.

For any version-pinned aggregate index, whether from Artificial Analysis or anyone else, the version number is not optional metadata. It separates a genuine discrepancy from a manufactured one. The 23-versus-15 story looked like a scandal for about a day, until the methodology notes showed a routine version bump. The next benchmark controversy may resolve the same way or may not, and the only way to know is to check.

What to take away

For its size, MiniCPM5-2B is genuinely well engineered, with 42 layers, an 8-to-1 GQA ratio, 131K of native context, open training data and eight coordinated artifacts spanning targets from a DGX Spark to a laptop. The headline claim holding on some benchmarks and failing on others does not erase any of that. Benchmarks are narrow instruments, each measuring something narrow, and a total score never tells the full story.

Before adopting the next small model that sounds too good to be true, reproduce at least one adversarial agentic benchmark and confirm the version behind any aggregate index score. A single headline number should not decide an architecture.