Floating Ground Truth: Dataset Version Pinning in LLM Evaluation Suites

When an LLM benchmark score moves between two runs, you want to know whether the model changed or the data it was graded against did. In most open evaluation setups, nothing records the second possibility. A recent audit of the task catalogue in lm-evaluation-harness, one of the most widely used harnesses behind open-model scores, found exactly one config out of 841 that sets a dataset revision field, and that single value turns out not to be a version pin at all. This article walks through how that number was produced, why the denominator matters as much as the numerator, how a different suite made the opposite trade-off, and how you can run the same check against your own evaluations.

The headline number, and the one that was discarded

The measurement is worth studying partly because of how it went wrong first. An early version of the count reported that one config in 13,986 pinned its data, a far more dramatic figure. It was thrown out within hours. Of those 13,986 files, 10,391 do not name a dataset themselves but inherit one from a parent through include:, and 2,966 are group files that name no dataset at all. Neither kind has anything to pin, so counting them inflates the denominator and makes the finding look far stronger than it is. The analysts noted that this was the second time in a single week that an impressive figure had come from failing to inspect what the denominator contained, which is a useful warning for anyone producing metrics of their own.

After the correction, the finding is this: in lm-evaluation-harness, 841 task configs name a dataset directly, and exactly one populates a revision field. That field holds refs/convert/parquet, a ref that selects a storage format rather than a specific version of the data. In practical terms, no config is pinned. Every run grades against whatever state the upstream dataset happens to be in on the day it executes.

Key findings at a glance

• Of 13,986 task configs, 841 name a dataset directly. The other 13,145 split into 10,391 files that inherit a dataset through include: and 2,966 grouping files with no dataset of their own.
• A single one of those 841 sets a revision or sha key, and what it holds, refs/convert/parquet, chooses a file format instead of pinning a version.
• Among the 673 Python task files, only 15 invoke load_dataset, and just 2 of those supply revision=.
• The configs point at 273 distinct datasets. For the median dataset, no config referencing it had been modified in 547.9 days. 196 of the 273 (71.8%) were past a year and 245 (89.7%) past six months.
• openai/evals takes the opposite approach: 455 of its 463 evals read an in-repository samples_jsonl file, backed by 722 data files committed to the repo. Its ground truth cannot drift, but it can go stale.
• Whether any upstream dataset actually changed was not checked, because huggingface.co was unreachable from the audit environment. The finding is that drift would go unnoticed, not that drift occurred.

In short

An evaluation suite is really a dependency graph, and software teams pin dependencies for good reasons. Walking every task config in lm-evaluation-harness, extracting the dataset each one grades against and looking for a pinned revision produced one candidate, which was not really a pin. Measuring how long those configs had gone untouched showed that the median dataset had not had a referencing config edited in roughly eighteen months. None of this shows that any published benchmark number is wrong. It shows that if one became wrong because its data shifted, the suite itself would give no signal.

Why a benchmark needs a pinned dataset version

The model under test is not the only thing that can change. Each task config refers to a dataset, for example alexandrainst/m_truthfulqa, OALL/ACVA or CogComp/mc_taco, and a dataset hosted on a public platform is a living artifact. It has maintainers, and it receives corrections, licence changes, re-splits, new configurations and now and then a silent correction to a label that was known to be mistaken. That activity is normal and mostly beneficial.

The trouble is what it does to comparisons. A score only means something relative to a stable reference. When a new model version produces a different score, you have learned something about the model. When the data shifted instead, you are looking at a measurement artifact that resembles a finding. Without a recorded revision, there is no way to tell the two apart, neither in hindsight nor at the moment you run the evaluation.

An unpinned score is a measurement whose ruler was never written into the record.

This is the same reasoning behind lockfiles in JavaScript projects: a package.json range such as ^4.2.0 lets builds silently pick up new code, and a lockfile makes the exact resolved version part of the record. Evaluation data deserves the same treatment.

How the count was produced

The methodology is where most of the interesting decisions live, especially around the denominator.

• Repository: EleutherAI/lm-evaluation-harness, cloned with full history using --filter=blob:none --unshallow. That covered 4,115 commits from 2020-08-27 up to a HEAD dated 2026-09-10, with the measurement taken on 12 September 2026. The catalogue changes constantly, so later runs will give different numbers.
• Config walk: every .yaml and .yml file under lm_eval/tasks. For each file, the script extracted dataset_path or hf_path, searched for any of dataset_revision, revision, dataset_sha or sha, and recorded whether the file used include:.
• Eligible denominator: only files that name a dataset themselves. Inheriting configs and group files that merely bundle other tasks were left out, since they have nothing of their own to pin.
• Staleness: a per-file git log gave the date each file was added and last modified, measured against the HEAD commit date rather than the current date so the numbers stay fixed as time passes.

That filtering left 841 eligible configs pointing at 273 distinct datasets.

How stale are the configs?

Staleness has to be reported two ways, because the obvious calculation turned out to be misleading, and that only became clear through manual inspection.

Counted per config, the median time since last modification is 729.8 days. 691 of the 841 configs (82.2%) had not been touched in over a year, and 551 (65.5%) had never been edited at all after being added.

Those figures exaggerate the picture. Just five commits added 50.9% of the 841 configs, and one commit alone, decc533d, introduced 272 of them on a single day. The per-config distribution therefore does not reflect 841 independent choices ageing on their own schedules; it reflects a handful of bulk contributions plus a long tail.

Grouping by dataset instead of by file cancels out that clustering:

• For the median dataset, 547.9 days had passed since any referencing config was edited.
• 196 of the 273 datasets (71.8%) were beyond one year.
• 245 of the 273 (89.7%) were beyond 180 days.
• The most neglected had gone 994.1 days.

The per-dataset figure is the one worth quoting, because it survives the clustering objection. The median is still about eighteen months, and nine in ten datasets are beyond six months.

Pinning is supported, just rarely used

It would be unfair to criticize the suite if it made pinning impossible, but it does not. The loader underneath is the standard Hugging Face datasets library, and load_dataset accepts a revision argument. In the Python portion of the task catalogue, 2 of the 15 files that call load_dataset pass revision=, out of 673 Python files in that directory, and one of those two pins to a pull-request ref.

So pinning is available, almost nobody uses it, and nothing in the workflow nudges contributors toward it. When the default is to float, a catalogue of 841 configs floats, because defaults are what large catalogues actually run on.

If you maintain evaluations, the cheapest follow-up is to search your own task definitions for a revision field today and see how many hits come back.

The opposite trade-off: vendored data in openai/evals

openai/evals answers the same question in the other direction, and its approach is not clearly inferior. Among 463 eval configs, 455 read a samples_jsonl file stored in the repo itself, which holds 722 data files to support them. The ground truth is vendored, which means it is pinned by construction: the data is versioned by Git along with everything else.

The benefit is perfect reproducibility: an evaluation first scored in 2024 can be repeated on byte-identical inputs. The price is currency. A vendored copy never receives upstream corrections, so while the suite cannot drift, it can slowly turn into a museum, grading new models against a snapshot whose mistakes were fixed elsewhere years ago.

Neither suite takes the third path: pinning a specific version and deliberately advancing it. One floats with no record; the other freezes with no refresh. In both cases the choice is made by default rather than by decision.

What the measurement does not show

The limits of the analysis matter as much as its findings.

• No dataset change was observed. The audit environment's network policy refused connections to huggingface.co, with the proxy returning 403 on CONNECT from both machines tried, so resolution and last-modified information could not be queried. Everything here concerns whether a change would be detected, not whether one happened.
• Unpinned does not mean wrong. Many of these datasets have probably never changed. The claim concerns a missing control, not an existing error.
• Stale does not mean neglected. A config nobody has edited in 700 days could be complete and accurate. Age signals that no one has looked again, which is different from a defect.
• One harness is not the whole field. One catalogue was measured in depth and one other used as a contrast. Tools such as promptfoo, deepeval and ragas are libraries rather than registries and have no comparable YAML task catalogue, so the result says nothing about them.
• Excluding include: files is a judgement call. A stricter reading might ask whether parent configs pin on behalf of their children. They were checked, and they do not.

Common questions

Are published benchmark scores unreliable, then?

No, and that interpretation should be resisted. A specific control is missing. A score is only compromised when the underlying data shifted; the audit shows that, should that happen, the suite keeps no record that would let you detect it afterwards.

Why not simply check whether the datasets changed?

That requires reaching huggingface.co to resolve dataset revisions, which the audit's network policy blocked with a 403 at CONNECT on both a cloud and a local machine. Instead of inferring drift from weaker signals, the claim was narrowed to what could be verified from the repository alone, which is why the finding concerns pinning rather than change.

Is pinning always the right answer?

Not automatically. A pinned eval never picks up genuine corrections to bad labels, which is how openai/evals can be perfectly reproducible and gradually inaccurate at once. A more defensible policy is pin-and-bump: lock a revision, move it forward intentionally, and log each move, which is precisely what almost no one does.

How can you check your own suite?

Walk your task definitions, extract whichever field names the dataset, and look in those files for any revision or sha key. The ratio of the second count to the first is the metric discussed here. The audit reported roughly thirty seconds of compute per thousand files, so this is an inexpensive check to add to CI.

Wrapping up

• Treat evaluation datasets as dependencies: record an exact revision alongside every score you intend to compare.
• Be suspicious of dramatic ratios until you have inspected what the denominator contains; inherited and aggregate configs nearly turned a modest finding into a misleading one.
• Floating data and frozen data fail in opposite directions: one drifts without a trace, the other ages without correction. Pin-and-bump, with a changelog, avoids both.
• Staleness and missing pins are signals of absent controls, not proof of broken results.

The open question for any team running evals in CI is concrete: when a score shifts between runs, what in your setup tells you whether the model or the data moved? If the answer is nothing, a revision field in your task definitions is the cheapest place to start. You can inspect the lm-evaluation-harness task catalogue and the openai/evals registry directly to compare the two approaches.