When the engineering is right and the measurement is wrong

Why a data source was ruled out · Observed State Index

Summary

This is not a study about an event in the world. It is a study about how a data source — the one with the best code in the system — turned out to be measuring something that did not serve its purpose, and how that conclusion was reached by comparing it against another source during a real event.

The lesson is not technical in the sense of syntax. It is methodological: a well-documented API, with well-formed data and a robust pipeline, can still be the wrong source. And that only comes to light when there is something to check it against.

The context

The system measures two domains as direct effects: air traffic (ADS-B) and network reachability (IODA). A third source, Cloudflare Radar, was integrated to cover internet traffic demand — a different signal from the infrastructure IODA measures.

Of the three, the Cloudflare pipeline was the best built. It had retries with exponential backoff and jitter, error classification by HTTP status code, an adaptive pause mechanism that tuned itself according to the API's response, metrics sent to CloudWatch, and a guard that prevented triggering the processing stage if fewer than 10% of countries responded. Neither of the other two pipelines had any of that at the time.

By design, it captured normalized hourly series of HTTP traffic per country, with a "degradation percentage" computed inside the ingest process itself, using a 20% threshold hard-coded in the source.

The test that gave it away

On 28 February 2026, a real event — the coordinated closure of Gulf airspace and the connectivity blackout in Iran — was already documented independently by two sources: ADS-B showed the collapse in air traffic, and IODA showed the sustained fall of ping-slash24 in Iran, at −96.7% change and lasting at least five days.

That produced something uncommon: a case with known ground truth. If Cloudflare was measuring the same thing as IODA, it had to show Iran collapsed and staying collapsed across the same days.

The same period was run over the Cloudflare data, time band by time band:

Cloudflare flagged the fall, and then reported an almost complete recovery within 24 hours. IODA, on independent data, said Iran was still dark five days later. Both sources could not be right at once.

Why it happened

The construction of the Cloudflare index was reviewed, and the exact mechanism of the error appeared.

The endpoint returns a 24-hour series normalized to its own maximum within that window, not an absolute volume. The value does not say how much traffic there was: it says what proportion of that 24-hour peak each point represents.

While the 24-hour window still mixed normal hours with blackout hours, the contrast between them exaggerated the apparent fall — hence the 0.105 on 1 March. But as soon as the full 24-hour window fell entirely inside the blackout period, the system recomputed its own maximum over already depressed values, and the index climbed back towards 1.0 — not because traffic had recovered, but because the reference point had moved down with it.

An indicator built this way is structurally incapable of measuring a sustained level. It can only detect abrupt transitions, and only for as long as the comparison window still contains them.

It was also found that the URL in use included a traffic filter (botClass=LIKELY_AUTOMATED) restricting the measurement to automated traffic — bots, crawlers, scanners — rather than human traffic. The source did not merely have a structural normalization problem: it was measuring the wrong variable from the query onward.

Why a code review would not have caught this

The Cloudflare pipeline had no programming errors. The retries worked, the error handling was correct, the data arrived well-formed and was stored without loss. Any code review would have approved it.

The defect was conceptual, not syntactic, and it only became visible when the system was asked for a claim that could be checked against a different source measuring the same thing by another route. Without that cross-check, the Cloudflare index would have gone on publishing an incorrect "Iran recovered" indefinitely, with a pipeline that would have looked healthy the entire time.

The decision taken

Cloudflare Radar was withdrawn from the main index. The capture pipeline stays running, but outside the calculation stage: it serves as a historical record and as the reference implementation for future pipelines, not as a measurement source.

A different endpoint of the same API was identified — incident annotations verified by hand at Cloudflare, with cause and scope declared — which does add value as a context layer: not as an index signal, but as confirmation of whether a detected event has already been reported by an independent third party.

What this case shows

That engineering quality and measurement validity are separate assessments. A pipeline can have the best error handling in the system and still be capturing the wrong metric.

That a real event with known ground truth is the strongest validation tool available. It was not a code audit that found the defect. It was putting two sources against the same verifiable fact and noticing that one contradicted the other.

That ruling a source out with evidence is worth as much as confirming one that works. The system measures two domains today instead of three, and that reduction is not a limitation: it is the result of applying the same standard of verification to the system's own instruments as to the phenomena the system sets out to measure.

Methodological note

This document does not invalidate the finding on the closure of Gulf airspace, published separately. The ADS-B and IODA results for 28 February 2026 do not depend on Cloudflare Radar at any point in their calculation; in fact it was precisely that finding, already validated by two other sources, that served as the reference for exposing the error in the third.