The same model gets opposite verdicts in the same week

Opposite takes in the same week are the normal case#

Claude Opus 5 launched around 24 July 2026. Within a week, r/ClaudeAI and r/ClaudeCode carried both raves and eulogies for the same SKU. One post called it a genie on a Blender one-shot at medium effort. Another called it extremely RL-fried on a large reverse-engineering job. A third redid four days of monorepo work and went back to Fable.

The load-bearing case sits in the middle. One personal eval suite called Opus 5 "an incredible coder and really painful to work with," scoring 100% as a worker and failing as a pilot. Same model. Same week. Opposite success criteria.

A megathread asked whether Opus 5 was bad or just Reddit hype, pulled hundreds of comments, and the auto summary landed on a blunt line. "The consensus is... there is no consensus," said the bot summary of that thread. Even the bot refused to pick a side. That should have been the end of treating the thread as a quality meter.

Zvi Mowshowitz recorded the split without treating it as noise. Capability can sit near Fable while many people hate talking to the model. His writeup says Opus 5 follows instructions well when kept on track, "which explains different harnesses causing different takes." Max Leander flipped from step-change praise on demos to rabbit-hole complaints days later. One person. Opposite poles. The feed still filed both under model weather.

Stop reading that split as a morality play. Opposite same-week verdicts are not proof half the community is lying. They are proof a model name is not a single controlled experiment. Not even close.

On long-horizon tasks, the harness can beat the SKU swap#

Zhang and co-authors make the Binding Constraint Thesis explicit in Stop Comparing LLM Agents Without Disclosing the Harness. On long-horizon agent tasks among frontier models, the execution harness often moves outcomes more than swapping the model. Holding the model fixed and changing only the harness raised Terminal-Bench 2 pass@1 from 69.7% to 77.0%.

The same paper's citations are not gentle. Claude Opus 4.5 scored 45.9% under a standardized SEAL scaffold on SWE-bench Pro and 55.4% under Claude Code. Independent monitoring reported up to 15 percentage points of scaffold-only swing on SWE-bench Verified. Controlled factorial work put harness variance several times model variance, with ranking reversals on the grid.

• Harness lift on Terminal-Bench 2 was 7.3 points with the model held fixed.
• Scaffold gap for Opus 4.5 hit 9.5 points between SEAL and Claude Code.
• Scaffold-only max on SWE-bench Verified reached 15 points in third-party monitoring.

Those gaps dwarf the 2 to 4 point moves papers still treat as meaningful model advances. When one coworker ships with a thin CLAUDE.md and high effort, and another ships with a fat instruction file, hooks, and different tools, they are not running the same trial. Calling both results "Opus 5 quality" is like timing two cars on different tracks and blaming the badge on the hood. Category error. A fat instruction file can also hurt frontier coding agents on its own, which is harness content, not a model personality shift.

A single run is a sample, not a personality#

Even when the knobs say deterministic, the draw is still a sample. Atil and co-authors measured five LLMs at temperature zero across ten identical runs and eight tasks. Their paper reports accuracy variations up to 15% across runs. A separate oracle Best-to-Worst gap can look larger if you cherry-pick the luckiest and unluckiest answers per question. That oracle number is not a single full-run score.

Coding is not exempt. Ouyang and co-authors found that temperature zero still left 18.29% of HumanEval problems with zero equal test outputs across five requests, and higher shares on harder sets. Thinking Machines later argued the user-facing cause is often batch non-invariance under load. Their Qwen3-235B temp-0 run produced 80 unique completions out of 1000, collapsing to one completion with batch-invariant kernels.

One green session and one broken session can both be honest. They still do not license a global verdict, and they sure do not justify you rewriting the default model after a single afternoon. Temperature zero is not a personality certificate.

The nerf panic is half true and still the wrong meter#

Take the strongest form of the panic seriously. Vendors do change product behavior under a stable brand. Effort defaults move. Thinking UI and cache logic changes. Prompt lines tax intelligence. Session limits tighten during peak hours. Anthropic's April 2026 Claude Code postmortem traced quality complaints to three product changes, not API weight downgrades. Medium effort defaults, a thinking-clear bug that fired every turn after idle, and a verbosity length limit that cost about 3% on an ablation.

Their summary is the line that should sit next to every "it got dumb overnight" post. "Because each change affected a different slice of traffic on a different schedule, the aggregate effect looked like broad, inconsistent degradation."

An earlier August 2025 infrastructure postmortem is harsher for the pure-noise story. Sticky misrouting of Sonnet 4 peaked at 16% of requests in a peak hour, hitting a large share of Claude Code users at least once. Two careful users could report opposite quality in the same week and both be right about their traffic slice. Anthropic still denied intentional demand-based quality cuts.

What failed as a quality meter was the viral shortcut, the BridgeBench-style screenshot that mixes different task counts so the board looks like a collapse when the shared tasks barely moved and the feed still treats the image as proof. Classic bad meter. VentureBeat's report captured the trust gap and the weak public proofs. Real product change does not turn a split megathread into a measurement system. Every model release also ages your skills, which is another reason the same SKU can feel worse after prompts stay frozen.

Pin the job, pin the harness, ignore the megathread#

Replace the thrash loop with a boring protocol. Boring on purpose. Name the job class before the model name. Greenfield demos, multi-hour agent loops with checkable goals, tight interactive PRs, monorepo reverse-engineering, and review-only passes are different products. Opus-style models can look elite on instrumented long runs and painful on chatty micromanagement in the same week.

1. Name the job before the SKU. Do not mix one-shot PR results with all-day agent loop results.
2. Pin the harness when you compare. Effort default, CLAUDE.md weight, hooks, tools, and subagent policy are part of the score.
3. Re-run prompts. Treat one session as a sample. Temperature zero still moves.
4. Keep six jobs. Needle fix, cross-file feature, instruction obedience, failing-test archaeology, surgical refactor, greenfield slice.
5. Switch on suite. Promote only when a new model wins at least four of six tasks with non-overlapping multi-run ranges.
6. Failover on regression. If the baseline fails two tasks it previously passed under the same locked harness, pause the default, check product version and effort, read vendor incident notes, and use a temporary fallback until the regression is explained.
7. Megathreads are texture. Useful for pain points. Useless as a ranked quality meter.

Run each suite task at least three times under a locked harness. Report ranges, not hero runs. Low weekly quota burn is not an IQ score. If you post publicly, disclose model ID, product version, effort, task class, pass criteria, and n.

The cost of holding this position is slower FOMO, and you will feel it every time a megathread hits while your suite still says hold. Defaults rotate less often. Hot takes wait for a re-run. That is the point. A model choice that survives a pinned suite is worth more than a model choice that survives a screenshot.

What would change the take is simple. Publish harness-locked, multi-run evals for the job class you care about, with the same tool surface, and show a real collapse that survives ranges. The launch-week job map is observational. Commenters rarely disclose a complete harness or traffic path. Until someone measures cause shares cleanly, opposite same-week verdicts remain what you should expect when composite scores get treated as pure model scores.