feat(lilith): first-divergence preference pairs at export #553

Merged

toasterson merged 1 commit from feat/lilith-first-divergence-pairs into main 2026-10-06 10:44:47 +00:00

Conversation 0 Commits 1 Files changed 3 +169 -7

toasterson commented 2026-10-06 08:55:33 +00:00

Owner

Copy link

Run 3 (PR #550, hypothesis 3): every sibling-contrast preference pair was refused by the trainer (pair_prefix), because the pairs carried whole multi-turn trajectories and the trainer accepts one prompt and one response per side with a pure-suffix mask.

program.encode.encode_preference now renders first-divergence pairs: the prompt is the shared prefix up to the first assistant turn where the sibling and the raw episode differ; each side's response is that single turn. A turn that is a tool call is rendered as the calls plus the tool-stop token (_encode_block(open_calls=True)), so "read before you claim" is the contrast the pair carries. Records keep the full message lists for provenance; the manifest and the on-policy sidecar record pair_render: first-divergence. The trainer and encode_sft are unchanged.

Spot check with the real Gemma tokenizer on run 3's ten refused pairs (8 Green, 2 Brown): 10 of 10 accepted by train_program._build_preference; prompts 2.5k–27.5k tokens, responses 14–687 tokens; six fit 16k, all fit 32k.

Tests: uv run --no-project --with pytest python -m pytest program/tests -q → 145 passed, 4 skipped (new test_first_divergence.py).

Run 4 plan: docs/ai/plans/2026-10-06-lilith-program-r4-siblings-in-context.md (PR #552).

Run 3 (PR #550, hypothesis 3): every sibling-contrast preference pair was refused by the trainer (`pair_prefix`), because the pairs carried whole multi-turn trajectories and the trainer accepts one prompt and one response per side with a pure-suffix mask. `program.encode.encode_preference` now renders **first-divergence pairs**: the prompt is the shared prefix up to the first assistant turn where the sibling and the raw episode differ; each side's response is that single turn. A turn that is a tool call is rendered as the calls plus the tool-stop token (`_encode_block(open_calls=True)`), so "read before you claim" is the contrast the pair carries. Records keep the full message lists for provenance; the manifest and the on-policy sidecar record `pair_render: first-divergence`. The trainer and `encode_sft` are unchanged. Spot check with the real Gemma tokenizer on run 3's ten refused pairs (8 Green, 2 Brown): 10 of 10 accepted by `train_program._build_preference`; prompts 2.5k–27.5k tokens, responses 14–687 tokens; six fit 16k, all fit 32k. Tests: `uv run --no-project --with pytest python -m pytest program/tests -q` → 145 passed, 4 skipped (new `test_first_divergence.py`). Run 4 plan: `docs/ai/plans/2026-10-06-lilith-program-r4-siblings-in-context.md` (PR #552).

toasterson added 1 commit 2026-10-06 08:55:33 +00:00

feat(lilith): first-divergence preference pairs at export ... 469c1f7a34

Run 3's sibling-contrast pairs held whole multi-turn trajectories, so the token masks had holes and the trainer refused every pair (pair_prefix). Export now renders each pair as the shared prefix up to the first differing assistant turn plus that single turn per side, the shape the trainer accepts, while the records keep the full message lists.

toasterson merged commit 629049cddc into main 2026-10-06 10:44:47 +00:00

toasterson referenced this pull request from a commit 2026-10-06 10:44:49 +00:00

Merge pull request 'feat(lilith): first-divergence preference pairs at export' (#553) from feat/lilith-first-divergence-pairs into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!553

No description provided.