fix(lilith): panel orders state the grounding rule (a cited receipt is not visible evidence) #554

Merged

toasterson merged 1 commit from feat/lilith-panel-grounding-rule into main 2026-10-06 10:45:48 +00:00

Conversation 0 Commits 1 Files changed 1 +2

toasterson commented 2026-10-06 09:01:34 +00:00

Owner

Copy link

Rubric grounding scored κ 0.12 in run 2 and 0.06 in run 3: one judge counted receipt-backed claims as visible evidence, the other did not. The run-4 plan (§5, PR #552) settles it: a claim is traceable only when the turns the judge sees contain the result it rests on; a receipt id, call id or digest alone is not visible evidence. One paragraph added to exam/panel.py INSTRUCTIONS. Tests: uv run --no-project --with pytest python -m pytest exam -q → 78 passed.

Rubric grounding scored κ 0.12 in run 2 and 0.06 in run 3: one judge counted receipt-backed claims as visible evidence, the other did not. The run-4 plan (§5, PR #552) settles it: a claim is traceable only when the turns the judge sees contain the result it rests on; a receipt id, call id or digest alone is not visible evidence. One paragraph added to `exam/panel.py` INSTRUCTIONS. Tests: `uv run --no-project --with pytest python -m pytest exam -q` → 78 passed.

toasterson added 1 commit 2026-10-06 09:01:34 +00:00

fix(lilith): panel orders state the grounding rule (a cited receipt is not visible evidence) ... 123af174e2

Rubric grounding scored kappa 0.12 in run 2 and 0.06 in run 3 because one judge counted receipt-backed claims as visible evidence and the other did not. Run 4 plan section 5 settles it: traceable means the turns contain the result the claim rests on.

toasterson merged commit fcb8c6f728 into main 2026-10-06 10:45:48 +00:00

toasterson referenced this pull request from a commit 2026-10-06 10:45:50 +00:00

Merge pull request 'fix(lilith): panel orders state the grounding rule (a cited receipt is not visible evidence)' (#554) from feat/lilith-panel-grounding-rule into main

toasterson referenced this pull request 2026-10-06 22:20:23 +00:00

docs(lilith): RESULT for program r4 (siblings in context) — NOT CERTIFIED #568

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!554

No description provided.