Diagnosis and rebuild of 242 days of data · Observed State Index
Summary
This document describes a defect that produced no exception, no error log and no visible symptom across months of operation. The system ran, published data, and went unnoticed — until an out-of-schedule run produced a result impossible to ignore: thirty air corridors flagged as anomalous on the same day, all at once.
The cause was not a system failure. It was that the system never knew, for certain, what hour it had measured what it measured.
The symptom
The system compares each air traffic capture against the history of that same time band. The identity of the band — which of the four daily captures each one is — was derived from the sequence number assigned to that run's folder in storage, not from any datum inside the capture itself.
That sequence number was computed by counting how many folders already existed for the current day. It is fragile logic by construction: if a run fails halfway, if a retry occurs, or if someone fires a manual capture outside the scheduled hours, the count goes out of alignment, and every later run that day inherits a number that no longer corresponds to its real hour.
For months, the system operated on the assumption that this number was stable. It was not.
The symptom became visible on running a test capture outside schedule. That run was compared against the baseline of a band that was not its own, nearly an hour away from its real hour. The result: all thirty monitored corridors came out classified as out of range in the same execution, including airports that were in fact operating normally.
An Atlanta airport with more than double its usual traffic was not a real anomaly: it was a comparison against the wrong hour of the day.
Why this kind of defect is dangerous
An error that throws an exception is noticed immediately. This one threw nothing. The pipeline ran without failures, wrote results that looked normal, and the problem only surfaced when the hours drifted far enough apart for the effect to be visible to the naked eye.
That means that, for months, some comparisons were probably made against the wrong band with no indicator flagging it. How many was measured later: in January and February 2026 the run number got the band right between 92% and 100% of the time, but in June and July — with retries multiplied by the timeout defect — accuracy fell to 23%. The system was not broken: it was quietly imprecise, which is a harder failure mode to detect than one that announces itself.
The diagnosis
The fix required dropping the dependence on the sequence number and anchoring the comparison to something that could not drift: the real hour at which each capture was taken, as reported by the data source at query time.
That hour had existed in the raw data from the start, but had never been extracted during processing into the final storage format. It was available; it simply was not being used.
The solution had two parts: capture that real hour in future runs, and rebuild the entire history so that the 242 days already accumulated would carry it too. Without the second part, the system would have been split between a past with no reliable hour and a future with one — an inconsistency that would have complicated any analysis crossing both periods.
The rebuild
Reprocessing 242 days of raw data meant reading several thousand small individual files and regenerating the whole storage layer with the new information built in.
The process did not come out clean on the first attempt, and deserves documenting exactly as it happened:
A code change never got saved. A helper function in the editing environment failed silently when persisting a modification, because of a version control configuration field left half-filled in an earlier session. The full rebuild ran with the previous version of the code, without the fix, and had to be repeated in its entirety once the problem was detected — this time verifying, before launching the process, that the deployed file did contain the expected change.
The partitions were registered against a location that was later deleted. During the investigation a test copy had been written to a temporary location, and the data catalogue was left pointing there. When that location was cleaned up, the queries were still valid in appearance, threw no error, but returned zero results, because they pointed at a folder that no longer existed. The solution was to recreate the table definition entirely, pointing it at the definitive location before registering the partitions again.
The compute configuration did not do what its name suggested. The transformation process was configured to use two worker units, on the expectation that both would process data in parallel. In practice one of those two units is always reserved for coordination and takes no part in the actual processing, so the job was running on a single effective unit. This produced no error, only performance markedly below expectation, which went unnoticed until it was compared against progress measured in real time.
None of these three stumbles was caused by the original design of the fix. They were frictions in the process of applying it, and each was resolved by verifying the system's real state directly rather than assuming a previous step had completed as expected.
The result
The completed rebuild left the 242 days of history with the real capture hour available on every record, allowing the corresponding time band to be derived reliably in any future query, without depending on any externally assigned sequence number.
As an unintended side effect, the process also revealed that roughly 800,000 records covering 43 additional days existed in raw storage but had never been incorporated into the queryable dataset, probably because of isolated failures in intermediate steps of earlier runs that were never corrected retroactively. The rebuild brought them in along with the rest.
What this case shows
That a system without visible errors is not necessarily a correct system. The absence of exceptions is not the absence of defects; some of the most persistent problems are the ones that never fail loudly.
That verifying the real state after each step is not optional when operating data infrastructure. Two of the three stumbles in the rebuild — the code that did not save and the partitions pointing at a deleted location — would have been avoided, or at least caught earlier, by a direct check instead of assuming the previous action had produced the expected effect.
That fixing a defect at the root tends to generate value beyond the defect itself. The original motivation was to solve a problem of temporal precision; the result also included recovering data that had simply been believed absent.
Methodological note
Until September 2026, this document claimed that the rebuild altered no conclusion of the two published analyses, because both fell on periods where the sequence-number assignment was correct. That was measured afterwards, by crossing the run number against the real hour carried in the raw data, and it turned out to be true for one of the two and false for the other.
The Gulf airspace closure is not affected. In its period (1 January – 5 March 2026) the run number did correspond to the real band: run 1 was right 64 times out of 64, and runs 2, 3 and 4 were right 59 out of 64 each. Retries existed, but they were occasional — 28 extra runs in 64 days.
The 2026 World Cup is affected, and badly. Its period coincides with the capture process's timeout degradation, when there were as many as twelve daily runs in triples per band. Under the assumption "run 3 = band 3", accuracy in that window was 23%, against 45% in its comparison window: two badly labelled periods, with different error patterns, compared against each other. The published finding did not survive review — the perfect separation between hosts and controls disappears on recalculating with the real hour. The detail is in the World Cup audit.
The lesson of that correction is the same as this document's, applied to the document itself: "independently verified" is a claim that also has to be verified. Here it was taken on trust with no table behind it, and the table said otherwise.
← Back to the index