Hash and resume compressed research snapshots (P2-08-B) #643
No reviewers
Labels
No labels
bug
cli
core
docs
event
experiment
figure
invariant
metrics
oracle
phase-0
phase-1
phase-2
phase-3
phase-4
phase-5
phase-6
provenance
revocation
tests
workload
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
erikinkinen/AES!643
Loading…
Reference in a new issue
No description provided.
Delete branch "phase2/p2-08-b-state-hash-resume"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task
Closes #98. Plan ID: P2-08-B.
Summary
Hash complete research execution state and replay compressed v4 logs from trusted acceleration checkpoints. Full execution, full replay and checkpoint resume produce identical final canonical execution and analytical-archive bytes across all seven mechanisms.
Scope
Add incremental SHA-256, separate canonical execution/archive codecs, complete hashed-event finalization and verification, validated physical graph/runtime restoration, a closed checkpoint codec/schema, and single-frame streamed zstd compression/replay. Capture completed initialization, epochs 128/256/... immediately before EpochEnd, and run end; deduplicate identical event positions. Add libzstd and the independent zstd CLI test dependency to CI.
Design intent
Execution hashes include operational state, retained observations, policy, handles, clocks and cursors. D has a separate archive hash and does not enter the execution digest. Restoration rebuilds actual graph indexes without rewriting stale epoch metadata. Every suffix event executes the actual operation and compares its complete recorded evidence; saved deltas never drive execution.
Resume requires an independently trusted checkpoint artifact hash from an accepted prior replay. The whole prefix is still read, canonical-validated and hashed; its clocks, cursors, terminal execution hash and historical query identities must agree. The returned count verifies that only suffix operations were re-executed. The low-level codec/restorer does not independently establish historical reachability.
Compression fixes level 3, zero worker threads, checksum enabled, and omitted content-size/dictionary ID. The decoder accepts exactly one standard frame, rejects trailing/concatenated/skippable/truncated input, and caps its window at 128 MiB. Stream buffers are bounded; state, query history, one record and returned checkpoints retain state-dependent storage. Both compressed and canonical-uncompressed identities and the zstd version are recorded.
Phase discipline
These checkpoints accelerate analysis; they are not modeled crash/recovery events. The required Phase 5 csaI recovery supplement remains outstanding. The preserved tamper corpus belongs to P2-08-C. The separately tracked oracle-fixture correction #641 / PR #642 is already merged and integrated. No scientific campaign, proof, scholarly review or manuscript acceptance is claimed.
Verification
All 76 CTests pass at signed source
85f8b2235af2b1d1ad5360cc6c4fccb0660e952fin Debug, Release and Clang ASan/UBSan. All eight local hooks and frozen protocol/source checks pass.Each of seven mechanisms compares live/full replay with resumes at initialization, epochs 128 and 256, and run end, including all later checkpoint bytes. Boundary tests have 14 accepted stream cases and 41 rejected evidence/stream cases. The external checker uses CLI zstd, Python JSON/hashlib and JSON Schema to verify seven streams, 28 checkpoints and 322 schema rejection cases. It produces the same artifact fingerprint in all three local builds using zstd 1.5.7. Compressed-byte equality across library versions is not assumed.
The state-codec reference independently checks 290 canonical execution/archive documents. Legal histories with identical final operational state but different D histories have equal execution hashes and different archival hashes. Restoration covers stale/missing parents, movement, pending observations, view corruption, reserved IDs and retired handles.
Exact-source JUnit, build logs, toolchains and cross-build artifact evidence. Acceptance and input/output hashes are recorded in
artifacts/backlog/2.2.1/P2-08-B/.All eight final-head Forgejo checks are required before the single issue step is checked once and the PR is merged.