Like searching the card catalog without skipping a drawer.
Picture a library whose card catalog is organized by subject, one drawer per topic. You ask for "everything similar to this book". A hurried clerk opens only the two most likely drawers. Most of the time that works. Sometimes a book that fit was in the next drawer, and nobody ever finds out.
SIEVE knows two things about each drawer: where its center is and which book lies farthest from that center. With that, it can say without opening it: "nothing in this drawer can possibly fit". It rules those out and opens the rest. Sometimes it opens more drawers than the hurried clerk, but at the end it can state that none was left out.
That is what market search indexes do not give you: they return results without saying how many true ones were left behind.
A silent "not found" is a risk.
The similarity search indexes on the market (the ones behind Qdrant, Weaviate, pgvector, Milvus and Pinecone) are approximate and silent. We measured the most widely used one with 40,000 items: depending on the speed setting, it finds 56% to 97% of the 10 true neighbors, and does not warn you when it misses.
For recommending a movie, that is fine. For checking whether a customer is on a sanctions list, finding frauds similar to a known one, deduplicating records, searching prior patents or auditing what an AI consulted, a forgotten result is a loss nobody can sign off on.
The new methods of 2026 already promise an accuracy level, but by estimate. One of them (DARTH, 2026) reports that about 13% of searches fall below the promised level, and the index does not know which ones.
Three sieves, none that cheats.
1. Drawers. Items are grouped into regions, and SIEVE keeps the center and size of each. A whole region leaves the count if even its closest point is outside the radius. A whole region enters without item-by-item checking if even its farthest point is inside.
2. Shadow. For the items that remain, SIEVE first looks at a "shadow" of them, a version with far fewer numbers. The distance in the shadow is never larger than the real one. So if even the shadow is far, the item is far, and the full computation is skipped.
3. Full computation. Only what passed both sieves gets its full distance computed. That is why SIEVE's worst case is the full search, never an incomplete answer.
None of this uses estimates, samples or confidence levels. It is geometry, and it is checked in tests against the full search in every configuration.
Faster than looking at everything, but not free.
On the public SIFT1M set (1 million image descriptors, 128 numbers each), the full search takes 109 ms per query. SIEVE in its fastest configuration takes 16.4 ms, 6.3× less, without leaving any result out. Reading from disk instead of memory, the advantage was 7.2×.
We also publish what did not work. The first version, with 50,000 items in memory, only tied with the full search: the initial promise of "4 to 8× faster" did not hold at that size.
And we measured how cost grows with the size of the collection. From 100,000 to 1 million items, full computations grow slowly (proportional to N0.41, similar to approximate indexes), but the cheap work of the sieves grows almost in proportion to the collection (N0.94). In practice: the advantage shrinks on very large collections, and today SIEVE does not have the same cost as approximate indexes at scale. The next step is a layered sieve.
| Measurement | Result | In practice |
|---|---|---|
| SIFT1M in memory, current configuration | 16.4 ms vs 109 ms | 6.3× faster than looking at everything, no result left out |
| SIFT1M reading from disk | 7.2× · 1,215× less reading | laying out data by drawer on disk is what pays |
| 50,000 items in memory, first version | ≈ tie | the 4–8× promise did not hold; published as such |
| Growth, 100 thousand → 1 million | full computations ∝ N0.41 · total work ∝ N0.94 | at hundreds of millions of items, cost grows almost in proportion to size |
| Hard data (GIST, 960 numbers) | ≈ 75% reach the full computation | it becomes a full search with proof: the guarantee stays, the speed does not |
| 32-bit math (SIMD) | recheck in 0.008% of cases | near the edge of the radius, it rechecks in double precision; see the notebook (Portuguese) |
Finding them all changes what you can state.
State absence. "No record crossed the limit" becomes a proven answer, not a "didn't find any". It is the sentence compliance and forensics need.
Catch AI agents going in circles. Reasoning models sometimes spiral, repeating ideas until they run out of memory. On 26 real labeled traces, SIEVE's search caught all three kinds of repetition (literal, alternating and reworded) with no false alarm on the 8 normal traces. The cheap detectors each missed one kind. See the full analysis.
Summarize the neighborhood with proof. Combined with CLAMP, you can ask "what is the average temperature of the cases similar to this one?" and get a guaranteed range: on real data, 327 of 327 answers contained the right value.
| Kind of repetition | Word repetition | Surprise | SIEVE |
|---|---|---|---|
| Literal ("Wait. Wait.") | 6/6 | 6/6 | 6/6 |
| Alternating (A·B·A·B) | 6/6 | 0/6 | 6/6 |
| Reworded | 0/6 | 6/6 | 6/6 |
| Normal traces with a false alarm | 0/8 | 2/8 | 0/8 |
When forgetting a result is expensive.
Good fit
- Compliance and sanctions: checking names and companies against official lists, with proof that nothing slipped.
- Forensics and fraud: finding every case similar to a known pattern.
- Deduplication and patents: where a forgotten look-alike becomes rework or litigation.
- AI audit: proving what an assistant consulted, and what there was not to consult.
Not the best choice
- Recommendations and product search, where finding 95% is enough: an approximate index is cheaper.
- Collections of hundreds of millions of items, until the layered sieve is ready: cost grows almost in proportion to size.
- Data with many independent dimensions (like GIST): the sieves rule out little and what remains is a full search with proof.
What SIEVE promises and what it does not.
No result left out
Every item within the radius shows up, or the exact 10 nearest. Checked in tests against the full search; one error fails.
It rules out only the impossible
Every sieve uses math that never overstates the real distance. Whatever it cannot decide, it passes on.
Speed is measured, not promised
We publish the gain (6.3× at 1 million) and the tie of the first version with the same weight.
Cost grows, and we say so
On larger collections the work grows almost in proportion to size. We do not sell the same cost as approximate indexes.
Finding the items most like an example. Each item becomes a list of numbers, and "similar" means "close" in that list.
How much an item may differ from the example and still count as similar. Whoever searches chooses it.
SIEVE's guarantee: every item within the radius appears in the answer. Not "almost all", not "in most searches".
Where this could be wrong.
Scale
Measured up to 1 million items. Beyond that the projection shows cost growing almost in proportion to size; at 100 million, a query would cost about 70× one at 1 million in the current configuration.
Hard data
On data with many truly independent dimensions, the guarantee holds, but the speed becomes that of the full search.
Precision of the math
Faster 32-bit math has rounding. SIEVE uses it only with a double-precision recheck band near the edge of the radius.
The radius is your choice
SIEVE guarantees it found everything within the requested radius. If that radius does not capture "similar" for your use, the answer is complete and still does not help.
← Certified telemetry · stickybit.com.br
- Our own measurements (Jul–Sep 2026): SIFT1M and GIST1M (ann-benchmarks), hnswlib with 40,000 vectors of 128 dimensions, growth curve from 100,000 to 1 million; library
github.com/andradeandrey/sieve, with the contract checked in tests against the full search. - Certified search, the technical paper (OFAC, UCR, AusTraits, SIFT1M and the sanctions screening case) · Doom loops · optical fiber demo.
- DARTH (SIGMOD 2026), early termination by accuracy target; DCO survey (EDBT 2026); "HNSW with Accuracy Guarantees Using Graph Spanners" (arXiv:2607.02338).
- Research notebook (Portuguese): "Mais rápido, outros bits", on the 32-bit recheck band.