Bound Ptah durable cache and equivalence copies; sample cycle memory peaks #673

Merged

toasterson merged 2 commits from fix/ptah-memory-round16 into main 2026-10-19 12:07:09 +00:00

Conversation 0 Commits 2 Files changed 9 +485 -8

toasterson commented 2026-10-19 07:52:50 +00:00

Owner

Copy link

Ptah on ffdf032a reached its 3 GiB cgroup limit repeatedly after the #670 rule-join fix. The first two OOM boundaries follow rule/equivalence completion and persistence start; a third at 08:24:33Z follows 12,886 learned equivalences and consolidation start. Metrics show memory growing across repeated cycles. The exact fatal allocation remains unproven.

Bound redb's rebuildable cache to 64 MiB instead of its 1 GiB default, and borrow equivalence fingerprints instead of copying every predicate/object string a second time. Preserve ACID data, matching behavior, the existing rule candidate budget, and all enabled functionality. Add constant-space 100 ms memory sampling around every daemon task and relational snapshot, including periodic high-water records for cycles killed before completion. Sampled cycle peaks and the process-lifetime high-water mark are named separately.

Validation: small-cache eviction/reopen regression over 6 MiB of data; shared-fingerprint equivalence regression; sampler parser/lifecycle regressions; existing durable/equivalence/rule tests. Local validation: 75 executed, zero failed/ignored (2 sampler, 13 durable, 16 equivalence, 44 rule-engine). Disk guard minimum 32.315 GiB. Formatting and diff checks pass. Exact-head Solstice CI remains pending. This is bounded mitigation plus instrumentation, not a proven sole-cause diagnosis. Main image build and a Ptah-only Talos pin PR precede a fresh zero-restart 60-minute acceptance window. No resource bump or MausBot activation is included.

Private log/metric hashes and the bounded conclusions are recorded in docs/ops/ptah-memory-round16/.

Initial head 0ceef3e5 had two Solstice failures before checkout due TCP refusal, not a product test. Talos #234 is now merged at bcf011be and accepted with two Ready external endpoints on separate nodes, zero restarts and complete fresh 20/36 registry verification. Documentation follow-up f5852146 records that repair and starts fresh CI; all six implementation files are byte-identical to 0ceef3e5. No blind rerun or status rewrite was used.

Live follow-up: the third baseline restart filled the inherited 512 MiB tmpfs with quarantined local redb images and partial hydration retries. Each complete local image is about 193 MiB plus 64 MiB symbols. The later Talos pin will keep one forensic predecessor and size the RAM-only cache for the derived ~644 MiB hydration transient, with margin; container memory remains 3 GiB. Three closed old images are privately preserved. This storage correction and main release are separate from source CI acceptance.

Shared CI investigation found healthy Forgejo Actions polling and a saturated five-VM executor, not a stuck webhook receiver. Fresh exact-head Akh and Anima jobs now run after ingress HA without rerun, dispatcher restart or status rewrite. All required checks still must pass before merge.

Ptah on ffdf032a reached its 3 GiB cgroup limit repeatedly after the #670 rule-join fix. The first two OOM boundaries follow rule/equivalence completion and persistence start; a third at 08:24:33Z follows 12,886 learned equivalences and consolidation start. Metrics show memory growing across repeated cycles. The exact fatal allocation remains unproven. Bound redb's rebuildable cache to 64 MiB instead of its 1 GiB default, and borrow equivalence fingerprints instead of copying every predicate/object string a second time. Preserve ACID data, matching behavior, the existing rule candidate budget, and all enabled functionality. Add constant-space 100 ms memory sampling around every daemon task and relational snapshot, including periodic high-water records for cycles killed before completion. Sampled cycle peaks and the process-lifetime high-water mark are named separately. Validation: small-cache eviction/reopen regression over 6 MiB of data; shared-fingerprint equivalence regression; sampler parser/lifecycle regressions; existing durable/equivalence/rule tests. Local validation: 75 executed, zero failed/ignored (2 sampler, 13 durable, 16 equivalence, 44 rule-engine). Disk guard minimum 32.315 GiB. Formatting and diff checks pass. Exact-head Solstice CI remains pending. This is bounded mitigation plus instrumentation, not a proven sole-cause diagnosis. Main image build and a Ptah-only Talos pin PR precede a fresh zero-restart 60-minute acceptance window. No resource bump or MausBot activation is included. Private log/metric hashes and the bounded conclusions are recorded in `docs/ops/ptah-memory-round16/`. Initial head 0ceef3e5 had two Solstice failures before checkout due TCP refusal, not a product test. Talos #234 is now merged at bcf011be and accepted with two Ready external endpoints on separate nodes, zero restarts and complete fresh 20/36 registry verification. Documentation follow-up f5852146 records that repair and starts fresh CI; all six implementation files are byte-identical to 0ceef3e5. No blind rerun or status rewrite was used. Live follow-up: the third baseline restart filled the inherited 512 MiB tmpfs with quarantined local redb images and partial hydration retries. Each complete local image is about 193 MiB plus 64 MiB symbols. The later Talos pin will keep one forensic predecessor and size the RAM-only cache for the derived ~644 MiB hydration transient, with margin; container memory remains 3 GiB. Three closed old images are privately preserved. This storage correction and main release are separate from source CI acceptance. Shared CI investigation found healthy Forgejo Actions polling and a saturated five-VM executor, not a stuck webhook receiver. Fresh exact-head Akh and Anima jobs now run after ingress HA without rerun, dispatcher restart or status rewrite. All required checks still must pass before merge.

toasterson added 1 commit 2026-10-19 07:52:50 +00:00

Bound Ptah persistence cache and equivalence copies; sample cycle memory 0ceef3e5a3

toasterson referenced this pull request 2026-10-19 08:03:00 +00:00

Preserve typed provider failures for central admission and provenance #674

toasterson referenced this pull request 2026-10-19 08:12:26 +00:00

Require exact workspace release proofs and durable cleanup backoff #675

toasterson added 1 commit 2026-10-19 08:16:10 +00:00

Record verified ingress repair before fresh Ptah CI f585214628

toasterson referenced this pull request 2026-10-19 08:46:04 +00:00

Emit per-turn usage and explicit context diagnostics #678

toasterson referenced this pull request 2026-10-19 09:40:33 +00:00

Bound model tool envelopes with ranged artifacts and process cursors #676

toasterson referenced this pull request 2026-10-19 09:40:34 +00:00

Persist typed model usage identity and context provenance #677

toasterson merged commit 67786deb93 into main 2026-10-19 12:07:09 +00:00

toasterson referenced this pull request from a commit 2026-10-19 12:07:11 +00:00

Bound Ptah cache and equivalence copies; sample cycle memory peaks (#673)

toasterson referenced this pull request 2026-10-19 12:44:43 +00:00

Require exact workspace release proofs and durable cleanup backoff #675

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!673

No description provided.