Like the photo that weighs less, but with a guarantee.
When your phone saves a photo, it throws away detail the eye does not notice, and the file becomes ten times smaller. Great for photos. Useless for measurements: nobody can tell you how much each point changed. On average the photo looks fine, but a single pixel may have moved a lot.
TUBE makes the same kind of saving, with one difference: before compressing, you agree on how much each reading may vary. A thermometer with half a degree of margin, a seismometer with one count. While storing, every reading that would fall outside that margin is stored exactly. Then the whole file is reopened and checked point by point.
The result is a smaller file that opens slightly different from the original, but never more different than agreed. And anyone can redo that check without having to trust us.
The margin is already written down somewhere.
TUBE does not invent the tolerance: it comes from whoever uses the data. In most cases there is already a document that says how much a reading may vary without changing any decision. It is the instrument's precision, the resolution of a mandatory report or the rule of a market.
When the margin comes from the standard, TUBE's guarantee becomes a sentence an auditor understands: "the stored file is within the precision the standard itself requires". What was a specification on paper becomes a checked property of the file.
If the margin is badly chosen, the guarantee is still true and stops being useful. That is why the choice belongs to whoever knows the use, never to us.
Predict, check, store only the surprise.
1. Predict. For each reading, TUBE makes a simple prediction of the next value from what it has already stored, for example "it will keep the same trend".
2. Check. If the real reading fits in the band around the prediction, there is nothing to store: whoever opens the file makes the same prediction and lands within tolerance.
3. Store the difference. If it does not fit, TUBE stores the difference between predicted and measured, rounded so that the guarantee still holds. That is the only data that takes space.
4. Reopen and seal. At the end, the file is reopened and compared with the original, point by point. Only then does it get a fingerprint and enter the signed record (GIRDER). The Go version and the in-browser version produce the same file, byte for byte.
Wins on stepped data, loses on smooth data.
We compared TUBE with SZ3, the reference scientific compressor for this kind of guarantee, on the same data and with the same tolerance. Real, public data in every case.
Where the data moves in steps, with integers, zeros and spikes (seismometers, smokestack emissions), TUBE makes the smaller file. Where the data is smooth and the tolerance wide (the undersea optical fiber), SZ3 wins on bytes. In both cases TUBE's guarantee was never violated.
In practice, on smooth data the reason to use TUBE is not the bytes. It is what comes with it: asking questions without opening the file, detecting events and sealing the proof.
| Real data | Tolerance | Measured result | In practice |
|---|---|---|---|
| IRIS seismometers, 17 stations, 40 Hz | ± 1 count | TUBE 1.22 MB vs SZ3 1.43 MB; 4.0× smaller than raw | 15% smaller than the competitor, nearly lossless |
| Undersea fiber, Svalbard, 50 Hz | ± 38 counts | TUBE 2.64 MB vs SZ3 2.03 MB | SZ3 30% smaller; near zero tolerance the two almost tie |
| EPA emissions, Barry plant (AL), 2023 | report resolution | TUBE 19.9 KB vs SZ3 42.2 KB, smaller even than lossless gzip | 2.1× smaller: emissions move in steps and have many zeros |
| Bearing vibration (CWRU, 12 kHz) | ± 0.1 g | 23.6× smaller than raw | corrected in Sep 2026: the old figure (36.8×) mixed 48 kHz and 12 kHz files |
| On-land optical fiber (PoroTomo), no event | per channel | ≈ 17× smaller; 5.8% of readings stored | during the earthquake it drops to 3.8×: the file grows when something happens |
| Processing cost (Go) | — | ≈ 53 ns per reading in the encoder | ≈ 19 million readings per second on one core |
Ask without opening, and notice the event for free.
Ask without opening the file. Because TUBE keeps a small index per block of readings, you can answer "how many readings crossed the limit?" by touching only the blocks that may hold the answer. On the undersea fiber, the question touched 0.04% of the file, answered 881 times faster than opening everything and returned the range [7, 15], which contains the right value (15). That is CLAMP.
Notice the event. When something happens, readings leave the band and the file has to store more. This "surprise" is a detector that costs nothing extra. On the French Broad river, during Hurricane Helene, it fired hours before the level crossed the official flood stage. See certified surprise.
Count with proof. At the EPA plant, "how many hours did NOx exceed 0.15?" came out as 162 hours, exact, straight from the compressed file, with the report sealed.
When the file has to prove what it says.
Good fit
- Data with steps, integers and spikes: seismometers, emissions, counters, prices. Here TUBE wins on bytes and on proof.
- Keeping months of readings for audit without paying for raw data, while keeping the spikes an average-based summary would erase.
- Mandatory reports where the margin is in the standard and someone has to check later.
- Fast, strong events (earthquake, flood, machine fault), noticed by the surprise.
Not the best choice
- Just saving space on smooth data with a wide tolerance: SZ3 makes smaller files.
- Detecting slow drift: the surprise is weak when the signal changes slowly.
- A signal hidden in noise (raw gravitational waves, for example): it needs a dedicated filter first.
- Compressing across neighboring channels of an optical fiber: we measured it, and with the guarantee enforced it came out worse than raw (0.66 to 0.99×).
Four rules TUBE does not break.
Tolerance comes first
Whoever uses the data chooses how much each reading may vary, before compressing. The margin is never adjusted after seeing the result.
Every reading, one by one
No "on average". The file is accepted only if the largest deviation, at any point, is at most 1× the tolerance.
The proof travels with it
Fingerprint, tolerance and parameters stay with the file. Anyone can reopen and check it without trusting us.
Declare how the data was measured
Sampling rate and unit of every file, in writing. Missing that inflated one of our vibration numbers, since corrected.
How much each reading may vary, chosen by whoever uses the data. Usually it comes from the instrument's precision or from a standard.
The "tube" around the original series, as wide as the tolerance. What the file returns always stays inside it.
The share of readings that escaped the prediction and had to be stored. It rises when something happens, which is why it works as a detector.
Where this could be wrong.
Bytes are not the promise
On smooth data SZ3 makes files up to 30% smaller. If space is the only goal, TUBE may not be the right choice.
The guarantee is for the chosen margin
TUBE proves it respected the tolerance. If that tolerance is too wide for the use, the file is correct and still not useful.
The surprise does not see everything
It catches strong, fast events. Slow drift barely registers, and a signal below the noise is invisible without its own filter.
Numbers already corrected
In the vibration demo, the old comparison mixed sampling rates. Redone, the separation between healthy and faulty machines exists, but with a margin of about 1 percentage point, not 76.
← Certified telemetry · stickybit.com.br
- Live demos, with real data and the guarantee checked: TUBE vs SZ3 on seismometers and undersea fiber (IRIS FDSN, SUBMERSE/EIDA) · EPA emissions (EPA CAMPD) · French Broad flood (USGS 03451500) · ISO 20816 vibration (CWRU Bearing Data Center) · undersea cable and earthquake.
- On-land optical fiber: PoroTomo/Brady (8,721 channels); per-channel compression and the cross-channel compression test, measured in Jul 2026.
- Research notebook (Portuguese): "Mais rápido, outros bits", on why sums done in parallel break "the same file on any machine".
- Full demo gallery.