A clock that loses one second a day is not "almost right".
A clock that loses one second a day is always "within one second" from one day to the next. After a year, it is six minutes slow. Each day's error is small, but it always goes the same way, and error that goes the same way adds up.
Ordinary compression does exactly that: it holds a constant value while the signal drifts slowly away, and only writes again when it touches the margin. Every returned reading is within the margin. But each one's error tends to lean the same way for long stretches. Noise cancels when you add it up; bias accumulates.
To measure how much of the problem is bias rather than size, we randomised the side of each error while keeping its size exactly the same. If size were the problem, nothing would change. It did: the real error came out 8.4 to 17.7 times worse for anyone who adds it up than noise of the same size.
The practical consequence: "I compressed within the sensor noise, so it is harmless" is a false statement for any client who adds things up. And almost every client does: fuel, energy, emissions, distance, dose, volume.
Plan ahead, and leave a receipt.
We rewrote the compressor to also promise something about what is reconstructed from the readings (position and velocity), not just about each reading on its own. There were two changes, and the second is what solved it.
Plan instead of react. The compressor used to decide on the spot when to write. It now decides ahead of time, choosing the anchor point in the middle of the motion instead of chasing it. Each write lasts much longer: 4 times less space.
The periodic receipt. Every so often, the compressor measures the deviation it has itself accumulated and writes that number down, with its sign (over or under). It takes a few bytes. The reader applies the receipts and the deviation stops growing. The more frequent the receipt, the tighter the guarantee: it is a dial, not a re-engineering.
The difference from what was already published is just the sign. Storing the size of the deviation lets you bound the error, and the errors keep adding up. Storing it with its sign lets you correct it, and they cancel.
The same aggregate, four ways to guarantee it
On a real set of 48 stations (166,000 readings), we measured how much the answer to the same aggregate can vary under each approach. The narrower the range, the sharper the guarantee.
It depends on how long you add up.
Before building anything, we wrote down the condition under which the idea would be dead: if our error contribution were far smaller than the instrument's own drift, there would be no product. The test puts both in the same unit, the one in the sensor's datasheet.
On the drone, compression dominated the error at every duration. On the satellite, the verdict changes with time: from 15 times the sensor's drift at 1 minute to below it from about 6 hours on. For long-horizon products, compression error disappears into the instrument's error. For short horizons it dominates, and that is where the receipt matters.
From an abstract number to arcseconds.
Nobody buys "0.001 on the fourth quaternion component" (the mathematical way of storing where the spacecraft points). They buy pointing accuracy in arcseconds, 1/3600 of a degree. The bridge between the two is an exact formula, not an estimate: if each quaternion component is off by at most ε, pointing is off by at most 2·arctan(2ε).
On GRACE-FO, the contract came out at 0.362 arcseconds, equal to the error the spacecraft's own star camera declares. The worst real error measured was 0.347": the bound is only 3.5% loose, which is why it can be signed. The file came out 14 times smaller than the same data stored losslessly.
One finding changed the recommendation: the mission's gyroscope product delivers the angle already summed, and archiving that angle beats archiving the quaternion: 3 times fewer bytes, with a contract 2 times tighter. They do not replace each other (the gyroscope angle drifts on its own; the quaternion comes corrected), so the right design is to keep the cheap angle and anchor it periodically with the star camera.
| value | in practice | |
|---|---|---|
| Pointing contract | 0.362" | equal to the star camera's declared error |
| Worst real error | 0.347" | the bound is only 3.5% loose |
| Space | 14× | smaller than the same data lossless |
Tighter or more compact.
A whole day of the GRACE-FO accelerometer: 86,400 readings. Each point on the curve is a choice: how much position error you accept over an hour of summing, and how many times smaller the file gets compared with the same data compressed losslessly.
Even at the most conservative point, with a guaranteed position error of 6 micrometres over one hour, the file is 41.6 times smaller. And the guarantee is not our word: each result is recomputed in exact arithmetic and checked against all three bounds (reading, velocity, position) before it is published.
Error that leans the same way. Small on each reading, large once added up.
A few-byte number, written every so often, with the accumulated deviation and its sign. The reader uses it to correct.
1/3600 of a degree. The unit in which a satellite's pointing accuracy is contracted.
Where this could be wrong.
The data does not say what the tolerance is
On GRACE-FO, two ways of estimating the margin from the data itself err in opposite directions: one measures the filter, the other measures what the filter removed. The tolerance has to come from a document, not from the data.
Filtered questions barely improve
Out of 10,000 readings, the 61 touching the bound account for 78% of the uncertainty. The uncertainty lies in which readings enter the count, and a value receipt does not touch that.
A requirement of the wrong kind
The pointing requirement we found is a control requirement (how well the spacecraft points), not a knowledge or archive-fidelity one. Using it as the tolerance would give a pretty and invalid number.
Two of our ideas failed
A detailed receipt per value band (3 to 15 times more space, almost no gain) and embedding the correction in the readings themselves (it works, but gives no guarantee). We publish both with the same prominence.
← Certified telemetry · stickybit.com.br
- GRACE-FO (NASA/GFZ), public level-1B products: 1 Hz accelerometer (satellite C, 10 Jun 2024), star camera and gyroscope.
- Every experiment on this page has its code versioned next to the result and runs on public data, including the ones that refuted us.
- Related demos: flight recorder (FOQA) and GPS trajectory with circular heading.