Public URLs Cited by the IANA Time Zone Database: A Link-Rot Census

The IANA Time Zone Database encodes how local clocks change around the world. Less obviously, its comment blocks also say where those rules came from. The upstream tree lives at https://github.com/eggert/tz.

A census of those citations asked a practical question: of the public URLs written into the comments, how many still respond today? Every such URL was collected, probed, and—when necessary—recovered, first through an unlocking proxy that bypasses common bot blocks, then through the Wayback API when the live host was gone. Code, census tables, and the retrieval pipeline for 1,327 URLs are published at https://github.com/sixthextinction/tzdata-citation-census.

The rabbit hole opened on a 2008 note by volunteer Patrice Scattolin. He was reconstructing the exact start of Morocco’s 2008 daylight-saving change (background: https://en.wikipedia.org/wiki/Daylight_saving_time_in_Morocco), reading a contemporary report at https://www.avmaroc.com/actualite/heure-dete-comment-a127896.html and wrestling with a French-to-English ambiguity. The note he left is typical of the corpus: it names decree 2–08–224 and admits the official text could not be located online at the time.

Historical databases are full of notes like that—decree numbers, gazettes, newspaper URLs, dates, and explanations of why one source won over another. The larger question is how many of those public URLs still answer after years of drift.

What Is tzdata, Actually?

The IANA Time Zone Database is a set of text files maintained by contributors. OSes and language runtimes consult it when they need wall-clock time for a civil zone. End users rarely open the files, yet nearly every Linux distribution and programming runtime ships them. Whenever a jurisdiction moves its clocks, the dataset must ship an update or every dependent app prints the wrong local hour.

Contributors can follow the correction guide hosted at https://data.iana.org/time-zones/tz-how-to.html when proposing fixes.

The rules themselves can be terse. The 2008 Moroccan daylight-saving change is two plaintext lines in the africa file:

# Rule NAME FROM TO - IN ON AT SAVE LETTER/S
Rule Morocco 2008 only - Jun 1 0:00 1:00 -
Rule Morocco 2008 only - Sep 1 0:00 0 -

Those lines define the June 1 and September 1 transitions that machines worldwide must honor.

The surrounding comments are where construction history lives. Maintainers record evidence: decree numbers, government gazettes, ministry releases, newspaper articles, or the person who supplied a tip, usually with the date the note was added.

Sources sometimes disagree. Maintainers occasionally document the conflict and state which side they trust. In 2006, Paul Eggert discarded atlas dates in favor of Austria’s national metrology office (https://www.bev.gv.at/):

# From Paul Eggert (2006-03-22): Shanks & Pottenger give 1918-06-16 and
# 1945-11-18, but the Austrian Federal Office of Metrology and
# Surveying (BEV) gives 1918-09-16 and for Vienna gives the "alleged"
# date of 1945-04-12 with no time. For the 1980-04-06 transition
# Shanks & Pottenger give 02:00, the BEV 00:00. Go with the BEV,
# and guess 02:00 for 1945-04-12.

Whether that sourcing culture was designed on day one or grew organically is unclear. Either way it leaves a usable audit trail for decisions made across more than thirty years of shared maintenance, especially when evidence is thin or conflicting.

So What Happens When You Actually Go Check the Citations?

Every URL was extracted from comment blocks across the nine tzdata source files. That sweep found sixteen hundred nine comment blocks containing thirteen hundred fifty-two URL mentions, collapsing to thirteen hundred twenty-seven distinct addresses across six hundred twenty-three hosts. Blocks that cite only books, decrees, or mail with no URL were out of scope.

Each address was probed with an ordinary HTTP GET, a conventional browser Identity header (as described in MDN’s User-Agent documentation), and a hard timeout so hung hosts could not stall the census.

That opening sweep returned usable bodies for 663 URLs. The other 664 failed for different reasons, so they were classified carefully: a 404 is not the same as a living page that rejects automation.

Of the failures:

• 433 could not be fetched directly but worked through a proxy. 178 refused the direct request (mostly HTTP 403, some 429). Another 255 failed from a home network with DNS, timeout, or TLS errors rather than an explicit bot block. Both groups were retried through Bright Data’s Web Unlocker and recovered.
• 123 were genuinely gone. They returned HTTP 404 or 410. Checked against the Internet Archive’s Wayback Machine, all but two produced a full archived copy—121 recoveries.
• 108 stayed unresolved (57 with no archive hit, 39 still failing through the unlocker, and 12 returning HTTP 500). Eighteen of the unresolved set, all on Mexico’s dof.gob.mx gazette, surfaced certificate errors—likely a host misconfiguration rather than a deleted page, but outside the census’s reach.

A vanished site is a preservation failure. A live site that rejects automated GETs is an access failure. Both block a naive crawl; they imply different things about whether the evidence still exists.

Numbers alone do not tell the operational story. The 663 direct successes include government portals, newspaper archives, and personal pages that still answer today. The 433 proxy recoveries show how often “broken” citations are really anti-automation policies. The 121 Wayback fills show how often the Internet Archive already held a usable snapshot—except for two 404/410 cases where even the archive was empty. Keeping those buckets separate is what makes the census interpretable for anyone repeating the measurement later.

Why Would a Citation Be Blocked in the First Place?

Most sites now treat scripted clients differently from ordinary browsers. Requests that skip JavaScript or fail browser fingerprint checks may be rate-limited, rejected, or challenged.

That complicates bulk verification of old citations. A URL that worked when a maintainer pasted it may still be live, yet refuse a simple automated request years later.

Official gazettes and legal databases appear often in the corpus—Portugal’s dre.pt, Turkey’s resmigazete.gov.tr, Israel’s nevo.co.il legal reporter, and similar hosts. Maintainers reused them heavily, and some of those pages still could not be fetched directly.

An unsuccessful request therefore does not prove disappearance. It may mean removal, or it may mean the retrieval method is no longer welcome. The recovery pipeline treated those cases as separate categories.

Is This Actually a tzdata Problem?

Not primarily. The experiment measures how well external sources recorded by a long-lived historical dataset have survived.

tzdata cannot preserve the material it cites. Rules live in a version-controlled tree and ship worldwide inside operating systems, but cited pages remain on independent websites, government portals, and newspaper stacks.

Access trouble is not new. A 2014 Egyptian daylight-saving note already recorded that the cabinet announcement page (http://www.cabinet.gov.eg/Media/CabinetMeetingsDetails.aspx?id=347) could not be loaded from outside the country. Geo-fenced evidence was a maintainer problem long before this census.

What Did the Recovery Pipeline Actually Look Like?

Recovery ran in separated passes so failure modes stayed distinct.

Trying every method on every URL—direct, then proxy, then archive—would inflate a single “recovered” count while blurring categories. A 404 differs from a temporary block. An archive snapshot also is not identical to the page a maintainer read; it is a capture from some later or earlier moment.

Pass one used only direct requests and recorded exact HTTP status. URLs that returned 404 or 410 went to a separate, rate-limited pass against the Internet Archive availability API. Timed-out or blocked URLs were not sent to the archive in that stage.

Conceptually:

async function checkCitation(url) {
  const direct = await get(url); // one plain GET, no tricks
  if (isOk(direct)) return { status: "live", via: "direct" };

  if (direct.status === 404 || direct.status === 410) {
    const archived = await wayback(url); // only for confirmed-dead
    if (archived.hit) return { status: "recovered", via: "wayback" };
  }

  return { status: "unresolved" };
}

The proxy pass was another stage. It targeted URLs the direct pass could not retrieve because of blocks or reachability failures, not URLs that had already declared 404 or 410. Blocked URLs were retried as ordinary GETs through Bright Data’s Web Unlocker acting as a native HTTPS proxy:

import { ProxyAgent, fetch as proxyFetch } from "undici";

const AUTH = process.env.BRIGHT_DATA_UNLOCKER_AUTH; // format like USER:PASS

const dispatcher = new ProxyAgent({
  uri: `@brd.superproxy.io:44445`">http://${AUTH}@brd.superproxy.io:44445`,
  requestTls: { rejectUnauthorized: false },
  proxyTls: { rejectUnauthorized: false },
});

const res = await proxyFetch(url, {
  dispatcher,
  headers: { "User-Agent": UA, Accept: "*/*" },
  redirect: "follow",
});

That path is not the browser-rendering product—just an HTTP fetch routed through the unlocker. Credentials belong in .env:

BRIGHT_DATA_UNLOCKER_AUTH=brd-customer-XXXXX-zone-web_unlocker:PASSWORD
BRIGHT_DATA_UNLOCKER_PROXY_HOST=brd.superproxy.io
BRIGHT_DATA_UNLOCKER_PROXY_PORT=44445

Separated passes let the final table distinguish direct hits, proxy recoveries, archive recoveries, and unresolved citations.

Methodologically, the unlocker pass matters because it separates “the page still exists behind a challenge” from “the page is gone.” Without that distinction, a naive crawler would over-count link rot. Likewise, sending only 404/410 responses to Wayback avoids flooding the archive API with hosts that were merely blocking bots, which would have muddied the recovery statistics and burned rate limits on low-value requests.

What Could You Do With the Sources Once You Have Them?

Availability was the question here: can the cited URL still be fetched, and if not, can the document be recovered elsewhere?

For a historical dataset, a further step is to preserve content from the recovered page itself. For a gazette or legal database that might mean extracting decree number, date, title, and relevant text and storing those fields beside the original citation. Evidence would then survive even if the URL later dies, without needing another crawl solely to rediscover links.

Repeated hosts are especially suited to automation. After unlocking fetches, a structured collector can define fields and run on the same infrastructure. Ordinary HTTP pages fit a code-oriented worker; JavaScript-heavy pages need a browser worker.

So What’s Actually Left When the Archive Doesn’t Have It?

One hundred eight citation URLs remained unresolved—about 8 percent of 1,327 unique URLs—small as a share, large as a pile of unrecoverable provenance. The share is modest, yet each URL once supported a concrete time-zone rule.

The rules remain in tzdata. What is missing is some or all of the external page at that address.

Of the 108, 57 returned 404 or 410 with no Wayback availability record. Thirty-nine still failed through the proxy (eighteen of those are the Mexico certificate errors). Twelve produced other HTTP errors that were neither retried nor sent to Wayback.

More exotic recovery was possible, but the census needed a fixed stopping rule: measure the citation landscape with one consistent procedure rather than chase a shrinking remainder forever.

The Limits of Citation as Preservation

Historical datasets and the pages they cite have asymmetric preservation properties.

A time-zone rule sits in version control and fans out through countless software distributions. Once incorporated, the rule can outlive the evidence that justified it.

Cited material has no such guarantee. It may live at one URL under one organization. If that organization moves, deletes, re-gates, or abandons the service, the citation becomes hard or impossible to retrieve. That failure mode is commonly called link rot.

tzdata maintainers have recorded sources for decades and often noted uncertainty. That makes later investigation far easier than unsupported assertions would allow. Careful citation still does not archive the cited artifact, and maintainers are not obligated to mirror everything they reference.

The census is therefore less a critique of tzdata than a demonstration of how far external URLs can be trusted as historical references. Most citations were recoverable in some form. A smaller but meaningful set were not. The database still holds the rules those sources established; in 108 cases it no longer recovers the URL.

Repeating the census a year later would likely move some URLs between buckets as sites change TLS, CDN rules, or robots policies. The interesting signal is not a single percentage frozen in time; it is how quickly citations drift from “direct OK” into “needs a proxy” or “archive only,” and how many fall into the unresolved remainder that no automated pass can fill.

For practitioners maintaining other long-lived datasets—CLDR locale data, geoname dumps, legislative scrapers—the same measurement recipe applies. Inventory every public URL in comments or provenance fields, classify failures by HTTP semantics, recover through an unlocker only when the failure looks like access policy, and consult an archive only when the origin declares the resource gone. Publish the bucket counts beside the raw URL list so later readers can re-run the same procedure instead of trusting a single headline percentage.