SIEVE · similarity search with a guarantee

The search that leaves no one out.

Finding "what is similar to this" among millions of items is fast with market indexes, but they can forget results without telling you. SIEVE finds every item within the requested radius, proves none was left out and shows how much work that took.

Specimen · 420 items on a map. Click to move the search
foundinside the radius, left outoutside the radiusopened region
–found
–left out
–regions opened
–distances computed

Illustrative: synthetic points (fixed seed) in 2 dimensions; the real SIEVE works with vectors of 128 to 1,024 numbers. The "usual fast search" is a simplified imitation of an approximate index that visits only the most likely regions. SIEVE's rule is the real one: a region is ruled out only if even its closest possible point is outside the radius.

In everyday terms

Like searching the card catalog without skipping a drawer.

Picture a library whose card catalog is organized by subject, one drawer per topic. You ask for "everything similar to this book". A hurried clerk opens only the two most likely drawers. Most of the time that works. Sometimes a book that fit was in the next drawer, and nobody ever finds out.

SIEVE knows two things about each drawer: where its center is and which book lies farthest from that center. With that, it can say without opening it: "nothing in this drawer can possibly fit". It rules those out and opens the rest. Sometimes it opens more drawers than the hurried clerk, but at the end it can state that none was left out.

That is what market search indexes do not give you: they return results without saying how many true ones were left behind.

Hurried clerkopens the 2 most likely found 2 · 1 left out, silently SIEVErules out only the impossible found 3 · none left out, proven How SIEVE knows it can rule out a drawer your search distance to center if distance − drawersize > radius,nothing is there
The rule for ruling out is ruler geometry, with no estimate and no probability. That is why it holds for every search, not "most of the time".
Why it matters

A silent "not found" is a risk.

The similarity search indexes on the market (the ones behind Qdrant, Weaviate, pgvector, Milvus and Pinecone) are approximate and silent. We measured the most widely used one with 40,000 items: depending on the speed setting, it finds 56% to 97% of the 10 true neighbors, and does not warn you when it misses.

For recommending a movie, that is fine. For checking whether a customer is on a sanctions list, finding frauds similar to a known one, deduplicating records, searching prior patents or auditing what an AI consulted, a forgotten result is a loss nobody can sign off on.

The new methods of 2026 already promise an accuracy level, but by estimate. One of them (DARTH, 2026) reports that about 13% of searches fall below the promised level, and the index does not know which ones.

How many of the 10 true neighbors each search finds 50%62.5%75%87.5%100% usual index 56%97% SIEVE 100%, always the range moves with the speed setting; the index does not say where you are
Our measurement on hnswlib, the reference implementation of the most widely used approximate index (40,000 items, 128 numbers each). SIEVE is checked in tests against the full search: a single forgotten item fails the test.
How it works

Three sieves, none that cheats.

1. Drawers. Items are grouped into regions, and SIEVE keeps the center and size of each. A whole region leaves the count if even its closest point is outside the radius. A whole region enters without item-by-item checking if even its farthest point is inside.

2. Shadow. For the items that remain, SIEVE first looks at a "shadow" of them, a version with far fewer numbers. The distance in the shadow is never larger than the real one. So if even the shadow is far, the item is far, and the full computation is skipped.

3. Full computation. Only what passed both sieves gets its full distance computed. That is why SIEVE's worst case is the full search, never an incomplete answer.

None of this uses estimates, samples or confidence levels. It is geometry, and it is checked in tests against the full search in every configuration.

1 million items drawerswhole regions ruled out by the ruler shadowcheap math, safe discard full computation on 1 million vectors, only 0.71% of items get here
Each sieve rules out only what is impossible. Whatever it cannot decide is passed on. Percentage measured on the fastest configuration (SIFT1M, below).
What we measured

Faster than looking at everything, but not free.

On the public SIFT1M set (1 million image descriptors, 128 numbers each), the full search takes 109 ms per query. SIEVE in its fastest configuration takes 16.4 ms, 6.3× less, without leaving any result out. Reading from disk instead of memory, the advantage was 7.2×.

We also publish what did not work. The first version, with 50,000 items in memory, only tied with the full search: the initial promise of "4 to 8× faster" did not hold at that size.

And we measured how cost grows with the size of the collection. From 100,000 to 1 million items, full computations grow slowly (proportional to N0.41, similar to approximate indexes), but the cheap work of the sieves grows almost in proportion to the collection (N0.94). In practice: the advantage shrinks on very large collections, and today SIEVE does not have the same cost as approximate indexes at scale. The next step is a layered sieve.

Full searchitem by item
109 ms
Compaction onlyPQ
54.9 ms
Drawers + compaction
37.7 ms
Shadow only
33.0 ms
Drawers + shadowcurrent configuration
16.4 ms · 6.3×
SIFT1M, 40 queries, radius equal to the distance of the 10th neighbor, pure Go. Every row finds 100% of results; only the work changes.
MeasurementResultIn practice
SIFT1M in memory, current configuration16.4 ms vs 109 ms6.3× faster than looking at everything, no result left out
SIFT1M reading from disk7.2× · 1,215× less readinglaying out data by drawer on disk is what pays
50,000 items in memory, first version≈ tiethe 4–8× promise did not hold; published as such
Growth, 100 thousand → 1 millionfull computations ∝ N0.41 · total work ∝ N0.94at hundreds of millions of items, cost grows almost in proportion to size
Hard data (GIST, 960 numbers)≈ 75% reach the full computationit becomes a full search with proof: the guarantee stays, the speed does not
32-bit math (SIMD)recheck in 0.008% of casesnear the edge of the radius, it rechecks in double precision; see the notebook (Portuguese)
What the guarantee enables

Finding them all changes what you can state.

State absence. "No record crossed the limit" becomes a proven answer, not a "didn't find any". It is the sentence compliance and forensics need.

Catch AI agents going in circles. Reasoning models sometimes spiral, repeating ideas until they run out of memory. On 26 real labeled traces, SIEVE's search caught all three kinds of repetition (literal, alternating and reworded) with no false alarm on the 8 normal traces. The cheap detectors each missed one kind. See the full analysis.

Summarize the neighborhood with proof. Combined with CLAMP, you can ask "what is the average temperature of the cases similar to this one?" and get a guaranteed range: on real data, 327 of 327 answers contained the right value.

Kind of repetitionWord repetitionSurpriseSIEVE
Literal ("Wait. Wait.")6/66/66/6
Alternating (A·B·A·B)6/60/66/6
Reworded0/66/66/6
Normal traces with a false alarm0/82/80/8
26 reasoning traces with real repetitions, each step represented by 1,024 numbers (bge-m3 model). The "similar" threshold was calibrated on the normal traces only.
Where to use it

When forgetting a result is expensive.

Good fit

  • Compliance and sanctions: checking names and companies against official lists, with proof that nothing slipped.
  • Forensics and fraud: finding every case similar to a known pattern.
  • Deduplication and patents: where a forgotten look-alike becomes rework or litigation.
  • AI audit: proving what an assistant consulted, and what there was not to consult.

Not the best choice

  • Recommendations and product search, where finding 95% is enough: an approximate index is cheaper.
  • Collections of hundreds of millions of items, until the layered sieve is ready: cost grows almost in proportion to size.
  • Data with many independent dimensions (like GIST): the sieves rule out little and what remains is a full search with proof.
The rules

What SIEVE promises and what it does not.

  1. No result left out

    Every item within the radius shows up, or the exact 10 nearest. Checked in tests against the full search; one error fails.

  2. It rules out only the impossible

    Every sieve uses math that never overstates the real distance. Whatever it cannot decide, it passes on.

  3. Speed is measured, not promised

    We publish the gain (6.3× at 1 million) and the tie of the first version with the same weight.

  4. Cost grows, and we say so

    On larger collections the work grows almost in proportion to size. We do not sell the same cost as approximate indexes.

Three words on this page
Similarity search

Finding the items most like an example. Each item becomes a list of numbers, and "similar" means "close" in that list.

Radius

How much an item may differ from the example and still count as similar. Whoever searches chooses it.

None left out

SIEVE's guarantee: every item within the radius appears in the answer. Not "almost all", not "in most searches".

Limits

Where this could be wrong.

Scale

Measured up to 1 million items. Beyond that the projection shows cost growing almost in proportion to size; at 100 million, a query would cost about 70× one at 1 million in the current configuration.

Hard data

On data with many truly independent dimensions, the guarantee holds, but the speed becomes that of the full search.

Precision of the math

Faster 32-bit math has rounding. SIEVE uses it only with a double-precision recheck band near the edge of the radius.

The radius is your choice

SIEVE guarantees it found everything within the requested radius. If that radius does not capture "similar" for your use, the answer is complete and still does not help.

See also

← Certified telemetry · stickybit.com.br

Sources