Agreeing costs less than objecting.

BullshitBench is a public test with a hundred questions that sound technical and make no sense, in software, finance, law, medicine and physics. In every one, the right answer is to point out what is wrong with the question. Answering it, however good the answer, counts as a miss.

In 2026, a user ran the test on several generations of Claude models, with three graders per answer, and published the numbers in the tool's official repository. The latest model of the previous generation objected to 94 in every 100. Its counterpart in the new generation, 60. The most expensive model of the new generation, 47.

And the new generation wrote more: 60% more words from the larger model, at the same requested effort level. Getting the question wrong and writing more about it is the most expensive combination for the reader, because a long, confident text looks like a review.

The most telling detail is not the overall score, it is which kind of question got through. The report splits the hundred questions by the trick used to disguise the nonsense. On logic tricks, such as treating coincidence as cause, the new generation stayed perfect.

The drop is entirely in the storytelling tricks: a metaphor taken literally, like the codebase's inertia, or tiredness and age attributed to things that neither tire nor age. These are questions that invite you to play along.

So the ability to notice the problem is still there. What changed is the stance: faced with a well-told idea, the new generation prefers building on it to taking it apart. That is exactly what you would expect from a model rewarded for being a good collaborator.

This part brings together other people's studies; the link between them is our reading. Models like these are refined with rewards: people, or other models trained to imitate people, pick the better of two answers, and the model learns to produce more of the chosen kind. For agents that work on their own, the reward is finishing the task.

The problem is that whoever picks likes to be obliged. A 2023 study by Anthropic itself showed that people and grader models sometimes prefer the answer that agrees and is well written over the correct one. Another, from 2024, measured the effect on the graders: after this training, human evaluators started approving more wrong answers, 24% more on reading questions and 18% more on code. The model became more convincing, not more correct.

In this game, objecting to the premise is turning down the request. Nobody rewards that on purpose; it simply is not among the preferred answers. A third study, from 2025, found the trace in the reasoning itself: after training to think before answering, models abstained 24% less on questions with no answer. And they got worse the more time they had to think. The doubt showed up in the draft and vanished from the answer.

It would be easy to conclude that newer models object less, full stop. The numbers do not allow it. The best score in the table belongs to a model from the same maker, from the generation just before. In April 2025, OpenAI undid within days an update that had made ChatGPT far too flattering, and explained the cause: it had given too much weight to users' thumbs-up. The maker of the model assessed here released a following version saying it had worked precisely on how it writes. Whether it objects again, nobody has measured on this test yet.

What remains is this reading: training that rewards finishing and pleasing creates a permanent pressure against objecting, and each version lands at a different point. Unless you measure every time you switch models, nobody knows where yours landed.

And overcorrecting is wrong too. The most expensive model of the new generation refused 35 in every 100 questions in the test: instead of pointing out the mistake, it simply did not answer. Refusing is not objecting. A reviewer that blocks everything becomes noise, and the team stops reading what it writes.

The obvious way out is to write in the instructions: "object if the question is wrong". It helps little. If the doubt already shows up in the draft and vanishes from the answer, one sentence at the start of the conversation is up against everything training taught. What works is changing the shape of the job, not the tone of the request.

In the specimen, the second reviewer fills in a field before answering: the question holds, does not hold, or I don't know. That field does not chat, does not build on anything, has no "yes, and…". With it, all four wrong questions are objected to. The price shows up: one false alarm, on the odd but correct request about the Fernando de Noronha time zone.

The third mode pays that price with a second opinion from another reviewer, from another model family, only on the objections. It is the same split we measured in an earlier entry of this notebook: raising an objection is cheap, delivering the verdict is expensive. Five extra calls instead of six, and the false alarm goes away.