Proof · Constructed orders (synthetic, ground truth known by construction)

Measured, with the misses.

Read this first. These 160 orders (96 Hindi, 64 English) were written for testing, in the style of Uttar Pradesh district-court bail orders, so the right answer for each is known by construction. They are not real cases. About a quarter carry a deliberate trap. The same script runs on real held-out Hindi orders (IL-TUR BAIL) once that dataset is available; this page will then show those numbers instead.

Says “past the line” when it isn't

Out of 96 people who are not past the line. Lower is better.

HalfTime0%
Regex + same engine0%
Plain chatbot6.3%

Calls a 479(2) case simply eligible

Past the line, but more than one offence or other cases. Lower is better.

HalfTime0%
Regex + same engine0%
Plain chatbot73.5%

Right verdict

All five verdicts; the chatbot is scored on its yes/no. Higher is better.

HalfTime96.3%
Regex + same engine44.4%
Plain chatbot (yes/no)80%

Every number

Measure	HalfTime	HalfTime, 3 runs (mean ± sd)	Regex + same engine	Plain chatbot
Unsafe “past the line”	0% (0/96)	0 ± 0	0% (0/96)	6.3% (6/96)
Verdict accuracy	96.3%	96.7 ± 0.3	44.4%	yes/no 80%
Misses the 479(2) flag	0%	0 ± 0	0%	73.5%
Misses someone past the line	4.7%	3.1 ± 1.6	75%	–
Charged sections, F1	99%	99.1 ± 0.2	51.1%	–
Sections precision / recall	98.6% / 99.4%		48.5% / 54.1%	–
Custody date exact	98.8%	99 ± 0.3	67.5%	–
Custody date within 7 days	98.8%	99 ± 0.3	67.5%	–
First-time status	93.8%	93.3 ± 0.4	87.5%	–
Other cases pending	100%	100 ± 0	85%	–
Line date exact	93.9%	94.9 ± 1	21.4%	±7 days on past cases: 37.5%
Says “can't tell”	21.9%	21.3 ± 0.7	71.9%	hedges 13.1%
Says “can't tell” wrongly	1.9%	1.3 ± 0.7	51.9%	–
Cost per order	$0.0164		$0	$0.017
Latency p50 / p95 (uncached)	5.9 s / 9.1 s		instant	–

The verifier rejected 1 model claims across run 1 (quotes not in the papers, or not containing the claimed date or section).

Traps

Trap	Orders	HalfTime right	Regex right	Chatbot avoided a wrong “yes”
An old case ended in acquittal	6	5/6	1/6	5/6
A co-accused with different facts	5	5/5	1/5	5/5
Planted instruction in the text	5	5/5	2/5	3/5
A life-term section hidden in a list	6	6/6	6/6	6/6
No custody date anywhere	6	6/6	6/6	6/6
A section mentioned but not charged	6	3/6	0/6	6/6
Arrest and other dates side by side	6	6/6	2/6	5/6

Every HalfTime miss (6)

Shown in the same detail as the wins. “Unsafe” means it said past the line when the answer key says otherwise.

Order	Answer key	HalfTime	Unsafe?	Why
C078 · en · not_charged	NOT_YET	EXCLUDED	no	extra sections BNS:105
C098 · en	NOT_YET	CANT_TELL	no	custody none vs 2024-04-09
C109 · hi	PAST_FLAGGED	CANT_TELL	no	missed sections GAMBLING:3, GAMBLING:4; extra sections OTHER:3, OTHER:4
C115 · en · acquitted_old_case	PAST_THE_LINE	CANT_TELL	no	custody none vs 2019-08-05
C124 · hi · not_charged	PAST_THE_LINE	PAST_FLAGGED	no	extra sections IPC:420
C130 · hi · not_charged	PAST_THE_LINE	EXCLUDED	no	extra sections IPC:307

Method

Three systems on the same 160 orders. HalfTime: claude-sonnet-5-5 extracts facts with quotes, code verifies every quote, a rules engine decides. Regex reader: keyword and date patterns feeding the same engine. Plain chatbot: the same model asked “Is the accused eligible for release under Section 479 of the BNSS? If yes, from what date?”; claude-haiku-5-5 reads each answer into yes / no / doesn't commit.
The answer key is the engine applied to the true facts written into each order, with “today” fixed at 2026-11-15.
The extraction prompt was tuned only on the 12 sample-tray orders (clarifying “no criminal history” and “in custody since”), before this set was run. It was not changed after seeing these results.
HalfTime ran 3 times (the model is not deterministic); the table shows run 1 and the mean ± sd.
Reproduce: pnpm eval rebuilds eval/report.json, and pnpm facts feeds this page. Run on 2026-11-13.

Run the sample sweep yourself