Verification glossary — the words of contestable verdicts

The yardstick

also: oracle · acceptance criterion · test suite · benchmark

In everyday life: The tailor's tape measure. If it shrank in the wash, every suit comes out wrong, and nobody suspects the tape.
What it means here: Whatever decides whether something passed: the test suite, the benchmark, the acceptance report, the panel of AIs giving scores, the checking spreadsheet. Almost nobody checks the yardstick; that is where we start.
Where it appears: The section Can the yardstick fail?Notebook: green is silence

Three verdicts

also: three-valued verdict · meets / fails / undecidable

In everyday life: The football referee who, when in doubt, asks for the video instead of guessing.
What it means here: A report that answers meets, fails or undecidable. The first two only come out when the whole error range sits on one side of the line; the third says, out loud, where the measurement stopped deciding.
Where it appears: The section Three verdicts, in writing Notebook: the draw made the call

Action limit

also: threshold · SLA

In everyday life: The 60 km/h sign: below it nobody gets fined, above it someone acts.
What it means here: The size of difference from which someone changes what they do: ship, pay, fine, reject. It has to be written down before measuring. In our performance comparisons, we used 5%.
Where it appears: Three verdicts, in writing Notebook: the draw made the call

Error margin

also: confidence interval · uncertainty · ±

In everyday life: Weighing the suitcase on the bathroom scale: "about 23 kg, give or take two".
What it means here: The range around the measured number where the true value probably lies. An honest report compares the range with the limit, not just the middle number.
Where it appears: The section Three verdicts, in writing

Margin

also: safety margin · guard band · k

In everyday life: Getting to the airport an hour early, not a minute early: same flight, different odds of missing it.
What it means here: How far past the line the result must be to decide. Declared before measuring, it protects against chance. Picked after seeing the result, it is just another way of reaching the answer one wanted. The right value is still an open question: three ways of deriving it gave 1.4 and 2.0, and the third pointed to no value at all.
Where it appears: Three verdicts, in writing Notebook: the draw made the call

Undecidable

also: inconclusive · zone of doubt

In everyday life: The lab test that comes back with "repeat the sample": neither good nor bad, just honest.
What it means here: The third verdict: the error range crosses the line and the measurement lacks the precision to decide. It can be large: in one of our studies, 81.6% of the differences a statistical test called real did not support a decision against the limit.
Where it appears: The section Three verdicts, in writing Notebook: the draw made the call

No alarm

also: passed · green · approved (the wrong name)

In everyday life: The quiet smoke detector: maybe there is no fire, maybe the battery is dead.
What it means here: The verdict of someone who looked for a problem and did not find one. It is never written as "approved": it is worth what the detector is worth. That is why the question that comes first is whether the detector can go off.
Where it appears: The section Notebook: green is silence Notebook: judge what leaks

Control against itself

also: A/A test · null control

In everyday life: Putting the same bag of rice on the scale twice. If it shows different weights, the problem is the scale.
What it means here: Comparing a system with itself, where the right answer is "nothing changed". The rule most used in CI pipelines declared a winner in 33 of 198 such comparisons, 16.7%. The rule that looks at the range, in zero.
Where it appears: Can the yardstick fail?Notebook: the draw made the call

Planted defect

also: mutant · mutation testing

In everyday life: The inspector who slips a fake bill into the till to see whether the cashier notices.
What it means here: A copy of the code with a small deliberate mistake. If the tests stay green, they cannot see that kind of mistake. The share of defects caught is the grade of the suite, not of the code.
Where it appears: Can the yardstick fail?Notebook: green is silence

Seed

also: random seed

In everyday life: The raffle number that starts the draw: same number, same result, every time.
What it means here: The number that starts a computer-made draw, such as the one that computes the error margin. Fixing the seed makes the result repeat; it does not make the result stop depending on it.
Where it appears: Notebook: the draw made the call Can the yardstick fail?

Coin share

also: verdict instability

In everyday life: A penalty shoot-out: neither team got better or worse, but someone wins.
What it means here: The share of verdicts that changes when only non-data changes, such as the seed of the draw. Swapping 200 seeds in one of our studies, 3.6% of the decided verdicts flipped. It grows exactly where cases live close to the limit.
Where it appears: Notebook: the draw made the call Can the yardstick fail?

External judge

also: independent oracle · checker

In everyday life: The referee who plays for neither team.
What it means here: A yardstick that sits outside whoever produced the result, rests on something the producer does not control, is expensive to build and cheap to run, and has an owner who pays if it is wrong. The four conditions are on the section front page.
Where it appears: The section AI checking AI

Judges who fail together

also: error correlation · correlated failures

In everyday life: Three witnesses who read the same news story: they look like three, they count as one.
What it means here: When several judges get the same cases wrong, adding opinions yields little. In a 2026 study with nine AIs from seven families, the average pairwise error correlation was 0.391. The same lesson showed up in 1986, with programs written by different teams.
Where it appears: AI checking AI Notebook: the cheap one acquits

Effective votes

also: effective number of judges · effective n

In everyday life: A board of nine where seven always vote with the chair: in practice, about three votes.
What it means here: How many independent judges a correlated panel is actually worth. Nine AIs judging together were worth 2.18 votes. Stacking more similar judges barely moves that number.
Where it appears: AI checking AI The section

Blind spot

also: domain limit · instrument blind spot

In everyday life: The side mirror that does not show the motorbike next to you: not a mirror defect, just where it cannot reach.
What it means here: Where the yardstick cannot decide, by construction. A coarse sensor at rest always returns the same number, just like a frozen one; long averages swallow the signal. Declaring the blind spot is half the report.
Where it appears: Notebook: judge what leaks Checking without opening Can the yardstick fail?

Checking without opening

also: confidential audit · refutation under confidentiality

In everyday life: The inspector who checks the weight of the sealed truck without opening the load.
What it means here: Reaching a verdict without seeing the audited party's confidential data. Four routes, from simplest to strongest: a timestamped receipt, involuntary leaks, encrypted computation and zero-knowledge proof.
Where it appears: Checking without opening 0k-proof FHE

Involuntary leak

also: side channel · involuntary emission

In everyday life: A liar's sweat: nobody chooses to emit it, and it says something.
What it means here: What a system emits without choosing, alongside what it declares: the size of each packet, the time it arrived, the pitch of a radio. Someone who sees only sizes and times can refute "the sensor is alive" without reading a value. Absence convicts; presence does not acquit.
Where it appears: Notebook: judge what leaks Checking without opening

Receipt

also: cryptographic digest · hash · verifiable receipt

In everyday life: A document's fingerprint: it does not show the document, but it gives away any changed comma.
What it means here: A short, unique digest of a file, kept at delivery time. Any third party recomputes it and checks whether the file changed, without having to trust whoever kept it.
Where it appears: Checking without opening Telemetry The section

Timestamp

also: certified dating

In everyday life: The postmark on an envelope: it proves the letter existed on that day.
What it means here: A signed record that a receipt existed at a given moment. Together with the receipt, it proves that today's file is the one delivered on that date. To last decades, the signature has to resist quantum computers.
Where it appears: Checking without opening Telemetry

Registered prediction

also: pre-registration

In everyday life: Calling the pocket before taking the shot.
What it means here: Writing down what one expects to measure, and how, before measuring. Without it, any result can be told afterwards as the expected one. Every measurement in this section had its prediction registered beforehand.
Where it appears: The section Notebook: the draw made the call

Losing criterion

also: kill criterion · refutation condition

In everyday life: Agreeing before the game on the score that makes you admit you lost the bet.
What it means here: The condition, written beforehand, under which the idea is declared wrong. It has already killed bets of ours: packet size as a dead-sensor detector, and AI-generated tests as defect hunters (0 of 19).
Where it appears: Notebook: green is silence Notebook: judge what leaks The section

The section

To use in practice.

Front pageVerdicts that hold up when contestedThe whole section on one page, with the two- versus three-valued report. AuditCan the yardstick fail?Five tests for any suite, benchmark or panel. ConfidentialityChecking without opening the dataFour ways to reach a verdict without seeing what is confidential. Another glossaryThe words of 0k-proofProving without showing, term by term.

Talk to us

Does this apply to your case?

Tell us in two lines what you need to decide or measure. The first conversation is to see whether measurement solves your case, and if it does not, we say so.

Talk on WhatsApp algorithms@stickybit.com.br Stickybit · Porto Alegre, Brazil, since 2004

← Verification