Add checked private corpus ingestion and fleet image reads

toasterson commented

2026-10-05 07:05:25 +00:00

Owner

Private corpus ingestion needs source-version provenance, valid PDF page citations, and resumable insert-only reconciliation. Add local PDF/EPUB preparation and OCR, content/hash/page validation, immutable artifact publishing, and a Seshat importer that verifies stored chunks before skipping occupied slugs or retrying timeouts.

Add a disabled-by-default, collection-scoped bearer endpoint for content-addressed PNGs, with a bounded streamed read and hash verification. Backend scans can exclude staging and duplicate prefixes before reading bytes. Local visual descriptions carry image/model provenance and remain labelled separately from source text.

Validation: Rust workspace fmt, clippy and tests passed (container-dependent integration tests remain opt-in); the final API change passed 38 unit tests and clippy. Ten Python tests cover hashes, page offsets/coverage, actual stored-content reconciliation apparatus links and Unicode JSONL separators. Helm lint passed. A private live pilot verified stored chunks and metadata through Seshat and its authoritative archive; corpus bytes, selection inventory and credentials remain outside Git.

The production Dockerfile also built and published successfully on native Linux amd64 at this exact source revision. The digest-pinned GitOps rollout and independent cluster search-to-image check passed. Shared Forgejo/Solstice source CI remains queued; no CI pass is claimed.

Private corpus ingestion needs source-version provenance, valid PDF page citations, and resumable insert-only reconciliation. Add local PDF/EPUB preparation and OCR, content/hash/page validation, immutable artifact publishing, and a Seshat importer that verifies stored chunks before skipping occupied slugs or retrying timeouts. Add a disabled-by-default, collection-scoped bearer endpoint for content-addressed PNGs, with a bounded streamed read and hash verification. Backend scans can exclude staging and duplicate prefixes before reading bytes. Local visual descriptions carry image/model provenance and remain labelled separately from source text. Validation: Rust workspace fmt, clippy and tests passed (container-dependent integration tests remain opt-in); the final API change passed 38 unit tests and clippy. Ten Python tests cover hashes, page offsets/coverage, actual stored-content reconciliation apparatus links and Unicode JSONL separators. Helm lint passed. A private live pilot verified stored chunks and metadata through Seshat and its authoritative archive; corpus bytes, selection inventory and credentials remain outside Git. The production Dockerfile also built and published successfully on native Linux amd64 at this exact source revision. The digest-pinned GitOps rollout and independent cluster search-to-image check passed. Shared Forgejo/Solstice source CI remains queued; no CI pass is claimed.

toasterson added 1 commit

2026-10-05 07:05:25 +00:00