- Problem
- Almost every LLM evaluation grades itself. Either a human judges the output, or a second model does — so the benchmark inherits whatever bias the judge has, and nobody can say whether a change actually helped. Resolver systems (given a question and evidence, decide the outcome) are especially exposed: the interesting failures are confident wrong answers, which is exactly what a model-judge is worst at catching.
- Approach
- Score resolver output against settled Polymarket/UMA markets, so ground truth is how the market actually paid out — externally settled and financially incentivised rather than a judgement call. 50 questions drawn from 4,947 harvested candidates through a published funnel, every rejection recorded with its reason, and the domain’s 72%-NO base rate measured and then forced to 50/50 so the naive always_no baseline scores 0.500 and sits in the same table as every model. Every resolver call must return a fixed schema — outcome, confidence, reasoning, sources, conflicts detected — validated before it is scored; a malformed response gets one repair retry and then a recorded hard_fail rather than a crash.
- Result
- The headline result is that the evidence barely helps: against a forced closed-book control that must answer, 4,800 tokens of dated documents move accuracy from 0.520 to 0.533 — a lift of +0.013 with the two intervals ([0.393, 0.667] and [0.38, 0.66]) lying on top of each other. Two explanations survive and the benchmark does not yet separate them. Calibration is the clearer failure: 36% of the answers the model rates at least 90% confident are wrong, and an explicit UNRESOLVABLE option moved ECE from 0.192 to 0.114 at the cost of abstaining on 37% of questions instead of 25% (McNemar p=0.63, not significant). Three statistical bugs were found and fixed in review, all of which flattered the result. Citation grounding 1.000; $0.002 and p95 2.6 s per resolution; CI gate replays 182 cassettes with no API key and no network.
