SCIP precision tier and measured resolution regression baseline #361

Merged

toasterson merged 6 commits from claude/wi-019fa8ea-wi-2-scip-precision-tier-the-resolution into main 2026-10-06 19:26:15 +00:00

Conversation 1 Commits 6 Files changed 26 +3437 -58

toasterson commented 2026-09-10 20:52:31 +00:00

Owner

Copy link

Implements a reproducible compiler comparison for Rust resolution on Akh-Medu. The ingester preserves SCIP and stack-graph provenance together; scoring compares exact reference/definition spans, retains parser gaps as false negatives, and enumerates ambiguous or otherwise unjudgeable sites. The recorded full-corpus baseline is replayed through the current rules and through deliberately broken rules that must fail with source-located disagreements.

Work item: 019fa8ea-daf9-7432-b322-86845cbc445e (WI-263).

Measured all 1,020 documents emitted by rust-analyzer at d4d46b598f: 996 indexed and 24 parser gaps. Resolution precision/recall is 92.954743% / 2.730278%; file-dependency precision/recall is 62.407132% / 6.517691%. Coverage, source/index/rule hashes, resource measurements and complete disagreements are committed under crates/shend/fixtures/akh-medu. The parent plan records the decision to require fresh unambiguous SCIP evidence or explicit degraded behavior; these low recall figures do not authorize downstream stack-graph consumers.

Validation: 12 unit tests, 3 focused parser/import tests, the actual full current/broken-rules replay, and strict all-target Clippy passed. Compiler capture took 293.965s; scoring 349.315s; full replay 525.141s. These are local host/default-feature measurements with explicit compiler limitations, not all-feature coverage or Solstice capacity proof. The workflow replays the recorded corpus on harness changes; expensive compiler regeneration is manual/on-demand. Current-head canonical review and native CI remain required.

Implements a reproducible compiler comparison for Rust resolution on Akh-Medu. The ingester preserves SCIP and stack-graph provenance together; scoring compares exact reference/definition spans, retains parser gaps as false negatives, and enumerates ambiguous or otherwise unjudgeable sites. The recorded full-corpus baseline is replayed through the current rules and through deliberately broken rules that must fail with source-located disagreements. Work item: 019fa8ea-daf9-7432-b322-86845cbc445e (WI-263). Measured all 1,020 documents emitted by rust-analyzer at d4d46b598f8c9b7597b4222f2ed2903a20ccabfa: 996 indexed and 24 parser gaps. Resolution precision/recall is 92.954743% / 2.730278%; file-dependency precision/recall is 62.407132% / 6.517691%. Coverage, source/index/rule hashes, resource measurements and complete disagreements are committed under crates/shend/fixtures/akh-medu. The parent plan records the decision to require fresh unambiguous SCIP evidence or explicit degraded behavior; these low recall figures do not authorize downstream stack-graph consumers. Validation: 12 unit tests, 3 focused parser/import tests, the actual full current/broken-rules replay, and strict all-target Clippy passed. Compiler capture took 293.965s; scoring 349.315s; full replay 525.141s. These are local host/default-feature measurements with explicit compiler limitations, not all-feature coverage or Solstice capacity proof. The workflow replays the recorded corpus on harness changes; expensive compiler regeneration is manual/on-demand. Current-head canonical review and native CI remain required.

toasterson added 1 commit 2026-09-10 20:52:31 +00:00

WI 2 — SCIP precision tier + the resolution accuracy harness that gates everything downstream ... c88dd9b526

Anima implementation session.

toasterson added 2 commits 2026-09-22 10:53:36 +00:00

Merge remote-tracking branch 'origin/main' into recover/wi-263-scip a4efc84f6d

feat(shend): add SCIP precision harness tier 6430b6c944

toasterson added 1 commit 2026-09-22 12:11:11 +00:00

fix(shend): build lexical scopes once per syntax node 7ecad5b406

toasterson commented 2026-10-01 16:23:51 +00:00

Author

Owner

Copy link

Full-lane babysit 2026-10-01 — WI-263.

Exact head 7ecad5b406a39f008eeb3c1b5abddeb21aaf7cdf, current main ad441fe9e2359a1bf315eda28a4b072cc56cee2d. Real merge-tree clean; branch is not already on main.

SCIP ingestion/harness implementation exists and hosted CI green with clean merge-tree. Acceptance remains incomplete: committed harness_baseline.json has watermark:null, files_indexed:0, empty predicates and no disagreements. It is an empty fixture, not measured Akh-Medu precision/recall. Produce actual same-watermark repo measurements and reviewed baseline before unlocking consumers; canonical gates also uncleared.

Individual disposition also recorded in the Anima WI thread. No approval or CI gate was overridden.

Full-lane babysit 2026-10-01 — WI-263. Exact head `7ecad5b406a39f008eeb3c1b5abddeb21aaf7cdf`, current main `ad441fe9e2359a1bf315eda28a4b072cc56cee2d`. Real merge-tree clean; branch is not already on main. SCIP ingestion/harness implementation exists and hosted CI green with clean merge-tree. Acceptance remains incomplete: committed harness_baseline.json has watermark:null, files_indexed:0, empty predicates and no disagreements. It is an empty fixture, not measured Akh-Medu precision/recall. Produce actual same-watermark repo measurements and reviewed baseline before unlocking consumers; canonical gates also uncleared. Individual disposition also recorded in the Anima WI thread. No approval or CI gate was overridden.

toasterson added 2 commits 2026-10-01 21:51:31 +00:00

Merge remote-tracking branch 'origin/main' into codex/wi263-actual-baseline-20261001 d4d46b598f

Measure real Akh-Medu SCIP accuracy and preserve truthful coverage 1a1adda337

toasterson changed title from WIP: WI 2 — SCIP precision tier + the resolution accuracy harness that gates everything downstream to SCIP precision tier and measured resolution regression baseline 2026-10-01 21:53:45 +00:00

toasterson referenced this pull request 2026-10-02 19:43:38 +00:00

WIP: Add session repository orientation with validated SCIP fallback (depends on #361) #511

toasterson merged commit 34a8198a64 into main 2026-10-06 19:26:02 +00:00

toasterson referenced this pull request from a commit 2026-10-06 19:26:05 +00:00

Merge pull request 'SCIP precision tier and measured resolution regression baseline' (#361) from claude/wi-019fa8ea-wi-2-scip-precision-tier-the-resolution into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!361

No description provided.