Hash and resume compressed research snapshots (P2-08-B) #643

Merged
erikinkinen merged 5 commits from phase2/p2-08-b-state-hash-resume into main 2026-10-20 22:38:18 +02:00
Owner

Task

Closes #98. Plan ID: P2-08-B.

Summary

Hash complete research execution state and replay compressed v4 logs from trusted acceleration checkpoints. Full execution, full replay and checkpoint resume produce identical final canonical execution and analytical-archive bytes across all seven mechanisms.

Scope

Add incremental SHA-256, separate canonical execution/archive codecs, complete hashed-event finalization and verification, validated physical graph/runtime restoration, a closed checkpoint codec/schema, and single-frame streamed zstd compression/replay. Capture completed initialization, epochs 128/256/... immediately before EpochEnd, and run end; deduplicate identical event positions. Add libzstd and the independent zstd CLI test dependency to CI.

Design intent

Execution hashes include operational state, retained observations, policy, handles, clocks and cursors. D has a separate archive hash and does not enter the execution digest. Restoration rebuilds actual graph indexes without rewriting stale epoch metadata. Every suffix event executes the actual operation and compares its complete recorded evidence; saved deltas never drive execution.

Resume requires an independently trusted checkpoint artifact hash from an accepted prior replay. The whole prefix is still read, canonical-validated and hashed; its clocks, cursors, terminal execution hash and historical query identities must agree. The returned count verifies that only suffix operations were re-executed. The low-level codec/restorer does not independently establish historical reachability.

Compression fixes level 3, zero worker threads, checksum enabled, and omitted content-size/dictionary ID. The decoder accepts exactly one standard frame, rejects trailing/concatenated/skippable/truncated input, and caps its window at 128 MiB. Stream buffers are bounded; state, query history, one record and returned checkpoints retain state-dependent storage. Both compressed and canonical-uncompressed identities and the zstd version are recorded.

Phase discipline

These checkpoints accelerate analysis; they are not modeled crash/recovery events. The required Phase 5 csaI recovery supplement remains outstanding. The preserved tamper corpus belongs to P2-08-C. The separately tracked oracle-fixture correction #641 / PR #642 is already merged and integrated. No scientific campaign, proof, scholarly review or manuscript acceptance is claimed.

Verification

All 76 CTests pass at signed source 85f8b2235af2b1d1ad5360cc6c4fccb0660e952f in Debug, Release and Clang ASan/UBSan. All eight local hooks and frozen protocol/source checks pass.

Each of seven mechanisms compares live/full replay with resumes at initialization, epochs 128 and 256, and run end, including all later checkpoint bytes. Boundary tests have 14 accepted stream cases and 41 rejected evidence/stream cases. The external checker uses CLI zstd, Python JSON/hashlib and JSON Schema to verify seven streams, 28 checkpoints and 322 schema rejection cases. It produces the same artifact fingerprint in all three local builds using zstd 1.5.7. Compressed-byte equality across library versions is not assumed.

The state-codec reference independently checks 290 canonical execution/archive documents. Legal histories with identical final operational state but different D histories have equal execution hashes and different archival hashes. Restoration covers stale/missing parents, movement, pending observations, view corruption, reserved IDs and retired handles.

Exact-source JUnit, build logs, toolchains and cross-build artifact evidence. Acceptance and input/output hashes are recorded in artifacts/backlog/2.2.1/P2-08-B/.

All eight final-head Forgejo checks are required before the single issue step is checked once and the PR is merged.

## Task Closes #98. Plan ID: P2-08-B. ## Summary Hash complete research execution state and replay compressed v4 logs from trusted acceleration checkpoints. Full execution, full replay and checkpoint resume produce identical final canonical execution and analytical-archive bytes across all seven mechanisms. ## Scope Add incremental SHA-256, separate canonical execution/archive codecs, complete hashed-event finalization and verification, validated physical graph/runtime restoration, a closed checkpoint codec/schema, and single-frame streamed zstd compression/replay. Capture completed initialization, epochs 128/256/... immediately before EpochEnd, and run end; deduplicate identical event positions. Add libzstd and the independent zstd CLI test dependency to CI. ## Design intent Execution hashes include operational state, retained observations, policy, handles, clocks and cursors. D has a separate archive hash and does not enter the execution digest. Restoration rebuilds actual graph indexes without rewriting stale epoch metadata. Every suffix event executes the actual operation and compares its complete recorded evidence; saved deltas never drive execution. Resume requires an independently trusted checkpoint artifact hash from an accepted prior replay. The whole prefix is still read, canonical-validated and hashed; its clocks, cursors, terminal execution hash and historical query identities must agree. The returned count verifies that only suffix operations were re-executed. The low-level codec/restorer does not independently establish historical reachability. Compression fixes level 3, zero worker threads, checksum enabled, and omitted content-size/dictionary ID. The decoder accepts exactly one standard frame, rejects trailing/concatenated/skippable/truncated input, and caps its window at 128 MiB. Stream buffers are bounded; state, query history, one record and returned checkpoints retain state-dependent storage. Both compressed and canonical-uncompressed identities and the zstd version are recorded. ## Phase discipline These checkpoints accelerate analysis; they are not modeled crash/recovery events. The required Phase 5 csaI recovery supplement remains outstanding. The preserved tamper corpus belongs to P2-08-C. The separately tracked oracle-fixture correction #641 / PR #642 is already merged and integrated. No scientific campaign, proof, scholarly review or manuscript acceptance is claimed. ## Verification All 76 CTests pass at signed source `85f8b2235af2b1d1ad5360cc6c4fccb0660e952f` in Debug, Release and Clang ASan/UBSan. All eight local hooks and frozen protocol/source checks pass. Each of seven mechanisms compares live/full replay with resumes at initialization, epochs 128 and 256, and run end, including all later checkpoint bytes. Boundary tests have 14 accepted stream cases and 41 rejected evidence/stream cases. The external checker uses CLI zstd, Python JSON/hashlib and JSON Schema to verify seven streams, 28 checkpoints and 322 schema rejection cases. It produces the same artifact fingerprint in all three local builds using zstd 1.5.7. Compressed-byte equality across library versions is not assumed. The state-codec reference independently checks 290 canonical execution/archive documents. Legal histories with identical final operational state but different D histories have equal execution hashes and different archival hashes. Restoration covers stale/missing parents, movement, pending observations, view corruption, reserved IDs and retired handles. [Exact-source JUnit, build logs, toolchains and cross-build artifact evidence](https://git.erikinkinen.fi/attachments/3b01e737-68bd-45a8-a73e-8fce67ee6c6d). Acceptance and input/output hashes are recorded in `artifacts/backlog/2.2.1/P2-08-B/`. All eight final-head Forgejo checks are required before the single issue step is checked once and the PR is merged.
erikinkinen added this to the Phase 2 milestone 2026-10-20 22:28:24 +02:00
Phase 2: record validated compressed snapshot acceptance (#98)
All checks were successful
markdownlint / markdown-lint (pull_request) Successful in 22s
clang-format / check-format (pull_request) Successful in 32s
ci / smoke (pull_request) Successful in 7m50s
ci / sanitizers (pull_request) Successful in 9m38s
markdownlint / markdown-lint (push) Successful in 15s
clang-format / check-format (push) Successful in 27s
ci / smoke (push) Successful in 4m50s
ci / sanitizers (push) Successful in 6m32s
4c260ccf9a
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erikinkinen/AES!643
No description provided.