[BUG] Terminal retirement loses notification or mediator CPU evidence #5
Labels
No labels
alpha-1
bug
ci
docs
duplicate
enhancement
help wanted
invalid
performance
phase-6
question
refactor
security
wontfix
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
erix/procd#5
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Terminal retirement can publish an incomplete local result: ordinary processes are marked terminal before a fallible visible-event queue write, and mediator retirement discards its managed record without retaining the final CPU counters. These paths prevent reliable final status/accounting delivery when native observations become explicitly acknowledged.
Reproduction
Source-audited baseline: Procd
ac8a12993bc8cbf134a11e141e459cb63df71123,src/processes_lifecycle.rs,retire_process_event. For an ordinary managed process, fill its visible-event queue and deliver an exact terminal event. For a mediator, retire the exact managed generation and then query its terminal CPU counters. These are deterministic producer-state inputs; a baseline VM reproduction has not been performed.Expected and actual behavior
Proposed final mediator disposition — not an implemented fix:
Use two capability-free owned operations on the existing private Launchd–Procd route:
READ_REALM_FINAL_ACCOUNTING(0x437) andACK_REALM_FINAL_ACCOUNTING(0x438). Selector0x436belongs to the separately proposed Provision successor. No additional endpoint, identity query, capability transfer or authority is introduced.All fields are little-endian, with exact framing:
Every request correlation and identity word is nonzero; only the two assigned operations are admitted in their corresponding message types. Dispositions are
Unconfirmed=0,Measured=1, andNeverExecuted=2. Successful responses and ACK requests require Measured or NeverExecuted and a complete nonzero supervisor pair; NeverExecuted requires both counters zero. Measured permits zero counters: only an actual terminal measurement establishes this disposition. Failure responses preserve the valid request/op/child/realm selector, require Unconfirmed, and zero the supervisor pair and counters. They never invent caller identity. No all-zero digest sentinel is introduced: accounting binds the original native child, original realm and actual caller, and does not restate startup authentication.READ obtains the supervisor identity from actual native CLAIM, which Procd checks against the retained original supervisor. This also works before ContextProbe has supplied Launchd's descriptive identity. A successful READ returns that authenticated pair. Launchd additionally compares it to its retained bootstrap context when one exists. ACK must match actual CLAIM and every retained snapshot field. Parsing bytes proves no native effect or authority. Codecs reject short, surplus, malformed, reserved or mismatched fields; encoding preflights the complete value and output length before any write.
The final READ does not consume a row. Running terminal observation retains its existing measured user/system counters, original supervisor and exact child/realm after native destruction and capability cleanup. Explicit RETIRE retains its separate exact terminal-event disposition. Ordinary QUERY_PROCESS_TIMES must become non-consuming for mediator rows so it cannot bypass the new ACK; ordinary-process behavior is a separate tracked concern.
NeverExecuted must be an explicit retained outcome of successful exact AbortStaged before any native START attempt. A private Provision or Retire that owns rollback binds its realm correlation and engages this scalar obligation before its first possible cleanup effect. Preserve an active scalar-only row after its successful unstarted abort, using an explicit disposed-unstarted state rather than fabricating a terminal event, EXITED state or CPU measurement. Bind an otherwise unbound realm only from that authenticated exact owner's Retire request. The ordinary guarded PREPARE/ABORT path, which never enters running-bootstrap ownership, keeps its existing disposition. An uncertain START can never become NeverExecuted. An absent row or kernel NOT_FOUND is not an accounting outcome.
The managed-row allocator and exact cleanup predicate must recognize both measured-terminal and proved-unstarted dispositions, require empty native/capability custody, no pending scalar obligation and no retained original invocation reservation, and preserve every other existing gate. Repeated cleanup of an already disposed scalar row must not issue AbortStaged or native destruction again. Original supervisor death disables the accounting obligation before its cleanup loop, so successful unstarted cleanup cannot recreate a pending consumer after that death.
Launchd retains one final snapshot in the original RetainedRealmStart. Its retirement phases become: exact child/local/caller/event cleanup, READ and retirement of that original caller, atomic local profiler commit, ACK and retirement of that original caller, fully Retired. Independent local and serving cleanup continues during these phases. Malformed results, uncertainty and unknown owners retain the original record and cannot report successful ABORT or permit release_retired.
Use a production scalar sink in RealmTable, initialized with its pristine storage and lasting for the current Launchd generation. Keep separate measured-retirement and never-executed counts plus checked cumulative mediator user/system nanoseconds, separate from POSIX self/waited-child/job totals. Preflight every aggregate addition, then commit all totals and the original owner's recorded flag together without fallible I/O between them. Arithmetic refusal retains the unchanged totals and the owned snapshot; repeated progress never charges twice. The bounded aggregate does not retain an unbounded history; the original exact snapshot remains in its realm record until ACK and retirement. A Rust-only read accessor supports the existing diagnostic output route; no public cross-client metrics endpoint is added.
ACK carries that already committed snapshot. Procd reserves its exact row, checks its immutable pending snapshot and original native caller, encodes the correlated reply, and clears cpu_time_observation_pending only after canonical native COMPLETE success, before releasing that same reservation. Failed or noncanonical COMPLETE leaves the obligation pending; no incoming accounting request can trigger cleanup of an unrelated or unvalidated child. Native COMPLETE is not caller RELEASE. It copies the result into the original invocation's owned result storage, independently of the managed row. Launchd already owns the snapshot before sending ACK, and retains the original ACK invocation through COLLECT and acknowledged RELEASE. Thus managed-row reuse after successful ACK COMPLETE cannot erase the original result or local accounting evidence. Launchd's realm record still cannot be reused until its own ACK caller and all other owners retire. Route failure may prevent successful retirement, but cannot justify replay, inferred acknowledgement or a second charge.
Client death still follows this accounting path while Launchd lives. Exact Launchd-generation death discharges only its own remaining scalar obligation, as the existing process-lifetime sink ends with that generation. No durable cross-restart profiling history or inherited supervisor authority is claimed.
Expected: capacity refusal precedes cleanup effects, a terminal marker denotes a complete notification/accounting commit, and mediator scalar counters survive capability cleanup until their authorized observation is discharged. Native acknowledgement must follow the complete local commit.
Actual in the published source: the ordinary path marks the row terminal before
push_visible_process_eventcan returnBUSY; a repeated event then takes the already-terminal return. The mediator path callsretire_realm_custody, which empties the row, then returns an unpersisted terminal copy without querying CPU counters. Neither path supports the required reliable acknowledged observation.Environment and identity
ac8a12993bc8cbf134a11e141e459cb63df71123; x86_64 host producer controls, Rust 1.97.1.Evidence and prior checks
Exact Launchd 2-realm/4-client/576-byte, Procd 38-record and Pagerd 102-binding budgets require reclamation. All six actual successful client exits, realm/client/plan refusals and original capability routes remain intact.
The client VM and older preparation/startup VMs pass every original marker, exit and 120/45-second gate. Retained client evidence binds 109 artifacts, 57,045 serial bytes, empty QEMU stderr and zero warnings. This accepts the native disposal/reuse sub-item. Separate positive NeverExecuted CPU accounting remains required: the first capture was rejected before boot because the host GDB package changed; after RPM identity/API verification and fresh two-positive/twenty-negative verifier controls, a separate capture reached the original 120-second limit after five Measured commits, before either NeverExecuted snapshot. Its failed result and cleanup are retained. No deadline, guest byte, authority or oracle was weakened; no numeric completion or speedup is claimed.
All fourteen Exsh development/release selections pass 988 tests each, strict host/native builds, Clippy, rustdoc and 355 checker controls pass without warnings, while native frame proof issue 4 stays open. Docs CI1099/1100 passes 45 document controls and the complete 2,657-page manual with zero final warnings. Integration passes its 187 host commands. Original CI1824 passes all 492 scenarios and subsequent image/input gates. CI1823 instead stops in the known physical utility host fixture, before catalog VMs. The companion pass does not resolve that intermittent failure.
All 22 original Integration runs 1803–1824 now have complete reviewed logs: 66 logs, 264,989,379 bytes, zero warning candidates. The smoke-policy defect is closed on its source correction and matching full-CI evidence. Remaining observations are retained under pipeline, ext4 quota, physical consent, interrupt reuse, canonical release, and the new canonical-template reports for release COM1 editor and canonical development. Their causes are not inferred from wrapper failures, and no unchanged CI job was rerun.
The phase checklist now records 419/879 completed sub-items, 17/460 accepted parents, and 15.45% weighted progress. Numerical NeverExecuted accounting, remaining authority/lifecycle cleanup, complete CI/frame acceptance, external Rust/LLVM/toolchain rebuilding and both complete EriX builds inside EriX remain required and unproved.
Native final CPU accounting — 25 September 2026: the original signed Integration 3bfa40a6 client image now has a complete host-debugger observation of all eleven final Measured snapshot additions.
The observation binds all executable bytes before and after, actual caller PCs, the original shared root and 137 active backing pages. At final exit, inactive virtual addresses deny user access; their restored mappings are not mistaken for the captured process storage. Hardware breakpoints and physical reads add no guest inspection endpoint, capability, memory/register mutation or success marker. All original required/forbidden/ordered markers, retirement/reuse checks, QEMU exit 33 and 120/45-second gates pass, with empty debugger/QEMU stderr and zero warning candidates. Two positive host transcripts and fifteen negative cases exercise the observer. These debugger-perturbed numbers are neither CPU utilization nor a speedup measurement.
Four rejected observer configurations remain preserved: two reached the original deadline; two completed the native scenario but failed host checks based on incorrect assumptions about inactive mappings. None earns acceptance. The final observer records each checked commit immediately, installs return/exit breakpoints only when needed, and separately checks captured storage and inactive protection.
Signed Integration 125bcbb5 records this status. Only ROADMAP changes; all maintained runtime, helper, profiler, catalog and CI bytes retain their validated identity. Fresh formatting, Markdown, dependency policy and diff checks pass without warnings. The phase tracker exposes this as one completed sub-item: 15.43%, 415/875 sub-items, 17/460 whole parents. Procd issue 5 remains open for its broader requirements. NeverExecuted native observation, remaining supervisor loss/exhaustion/cleanup, existing Exsh compiler/frame gates, full original CI review, external toolchain rebuilding and both complete EriX builds inside EriX remain required.
The corrected five-test control rejects the old configuration with
39 != 38; all 174 helper and six profiler commands, formatting, source policy and Markdown pass without warnings.The actual configured-client VM passes every original marker, exit and cleanup check within the unchanged 120/45-second guest bounds. It retains 109 hashed artifacts, 57,045 serial bytes and empty QEMU stderr. Independently decoded packaged tables contain exactly 38 Procd records and 102 Pagerd bindings; runtime slices use the authenticated count, so page padding does not add records. Six clients execute across three realm instances, including replacement while the peer stays live and rejection of stale selectors. This resolves the RTC inventory defect.
Capture review corrected: issue 95 was an invalid report caused by reading the combined display of two separate stream-tail commands as stdout alone. All allegedly missing build progress is present in the original 35,691-byte stderr, where the build scripts explicitly emit it. The 106-byte stdout correctly contains the three final validators. Both original streams have zero warning candidates and intact start/end records; no capture or source fix is needed. Two separate build-only diagnostics ran no VM; the broader trace exceeded its declared output budget and is retained as failed diagnostic evidence. Neither diagnostic substitutes for the original native run.
Docs ef2f8ae0 passes 45 controls, the complete 2,655-page manual, 491,436 word bounds and three visual reviews. Original CI1089/1090 also passes; four complete logs total 824,668 bytes, with early reference warnings converging to zero on the final pass. Integration CI1807/1808 remains queued, and older Integration runs remain under observation.
The demonstrated native record-reuse slice is now exposed as one additional completed sub-item: 414/874 sub-items, with 17/460 whole parents. Fixed parent/section weights are unchanged; rounded completion remains 15.42%. Final numeric CPU/profiler observation, supervisor loss, remaining exhaustion/cleanup coverage, existing Exsh compiler/frame gates, external Rust/LLVM/toolchain rebuilding and both complete EriX guest builds remain required.
The current source review confirms the independent scalar obligation after successful native retirement. It also confirms that native COMPLETE retains its response in the original invocation independently of the managed row. No new compiler, fault-injection or native accounting evidence is claimed by this proposal.
Published lifecycle source and published realm cleanup establish the two ordering/lifetime defects. The implementation now has producer controls for full queues, lost acknowledgements, failed counter queries, mediator cleanup, original supervisor death and native ID reuse. The first updated default host suite passes 299 tests. Full strict validation, original CI and coherent service VM acceptance remain pending; this report is not a claim of complete runtime acceptance.
Tracking
Client and mediator lifetime totals share one checked implementation in separate sinks; no public inspection authority or wire change is added.
Procd's explicit mediator Retire consumes that exact queued terminal event. Launchd neither demands a duplicate WAIT nor invents one from CPU or an action result. A spontaneous authenticated WAIT retains its original identity, kind and full status before joining retirement. Known Start/Ready or terminal evidence contradicts NeverExecuted; only positive native pre-start disposal establishes it.
The new regression fails on the preceding production source because retirement issued neither final Read nor ACK. The corrected component passes all 44 final checks: 335 library plus 161 runtime tests, 496 in each of six maintained configurations, strict host/native Clippy, private rustdoc, formatting, Markdown, dependency checks and warning-denied native builds. All six ELF files pass entry/permissions and complete version-7 layout checks (2,640-byte realm and 2,184-byte client rows). Nine added controls cover exact identities, recording before ACK, full-width refusal, altered ACK, native loss/unknown custody, overflow, spontaneous/duplicate events, public reply loss and local reuse. Eight older completed-retirement operation sequences now require final Read/ACK. Original CI177 and CI178 pass; all four complete logs are independently verified (179,140 bytes, zero warnings). No unchanged workflow was rerun.
README, ARCHITECTURE, ROADMAP and the mediator accounting contract are updated. Existing technical manual contracts already require these ownership and recording guarantees. The fresh structural audit covers 76 repositories, 3,189 authored code files below 1,000 lines, 264 full Git pins and 98 original crate documentation gates. This does not establish whole-fleet semantic authority acceptance.
Completion is 15.41% weighted checklist progress, 412/873 sub-items and 17/460 whole items. Launchd PR2 remains unfinished. Procd issue 5 stays open until a coherent native witness proves repeated managed-row reuse and Integration profiler observation. The full configure/seal/four-client I/O/failure/isolation witness, the complete EriX build inside EriX and external Rust/LLVM/toolchain rebuild inside EriX remain required. Original Integration CI1791/1792 remains green on its older source graph; that does not validate this new component graph.
Coordinate Procd realm work, Launchd realm ownership, and Phase 6 completion. Keep the bug open until exact final disposition, lost-result controls and actual repeated row reuse pass.
Related: Kernel terminal-accounting design, Procd WIP PR, and Phase 6 completion.
Reserve local publication storage before effects, preserve final scalar evidence after mediator authority disposal, and acknowledge only a fully committed exact-generation observation. Keep regressions for lost replies, independent queue saturation, original consumer loss and reused identities. Retain the open runtime and CI acceptance gates.
Committed terminal-accounting consumers — 21 September 2026:
Procd 66934642c4464fc738152a9e60790914ba27dd1c in WIP PR 2 reserves notification/crash/cleanup storage before effects, obtains exact final CPU evidence, commits local status and cleanup obligations, then acknowledges on every actual event polling path. Lost acknowledgement replies preserve the local result without duplicate counters or notifications. Mediators retain only scalar counters and the original authenticated supervisor identity after disposing all capability columns; supervisor death discharges the pending scalar observation and a replacement cannot inherit it. Four strict 299/305-test configurations, native builds, Clippy and private rustdoc pass. Original CI 298/299 passes from four complete hashed logs, 347,245 bytes, zero warnings. Bug 5 remains open for actual service acceptance.
Rootd 64c97b13c450003d9c2b6bd9ed2a627088684b46 in WIP PR 2 acknowledges bootstrap evidence only after exact native destruction and local endpoint absence; both operations remain unavailable after temporary Process custody transfers to Procd. Four strict 430/429-test configurations, native builds, Clippy and private rustdoc pass. Original 1039/1040 exposed bug 7: a stale source-call inventory and its matching semantic operation declarations. The correction explicitly inventories acknowledgement consumers and preserves the same temporary route and eventual Procd owner. All 64 Python controls, production-boundary, semantic baseline, threat model, phase contract and operation-ownership gates pass. Corrected original CI 1041/1042 passes from four complete verified logs, 222,969 bytes, zero warnings. Bug 7 is corrected; failed original runs remain retained without reruns or weaker gates.
Docs e4525848ad4462901c9a6794ef1794cf85ea9e6b updates the native contract, Procd/Rootd consumer custody and original signed IPC API references. Selector 55 is retired in both the detailed contract and summary; 58/59/60 are cross-checked against the shared registry. All 45 documentation tests and generated-reference checks pass. The complete 2,431-page manual builds without warnings; changed prose, selector and API pages pass visual review. Original documentation CI 995/996 and corrected-table 997/998 passes from eight complete logs, 1,549,628 bytes, with zero warnings in the final LaTeX passes. The 37 earlier convergence candidates per manual log are retained and resolved.
The separate Integration orchestration library checkpoint b06dfad00202765491a64552dde29eaca1c24838 passes four strict 320/321-test host/native configurations. Full service catalogs remain on their prior coherent graph while 35 remaining application/service repositories adopt the original shared revisions. Integration 1697/1698 remains running, and 1699/1700 plus 1701/1702 waits at the latest bounded observations. These are pending full regressions, not successful runtime acceptance.
No new canonical acceptance leaf is closed: 15/460 and 3.48% weighted. Ordinary Launchd metric-consumer restart/disposal semantics, coherent service CPU/profiler VMs, complete realm/provider authority and I/O, source/effect/frame proof, Pagerd, native external Rust/LLVM/runtime rebuilding and both full EriX-in-EriX build generations remain required. The static audit finds 3,162 authored code files below 1,000 lines, 74 manifests, 259 original Git pins, 173 direct missing-docs gates and 92 conventional crate roots; this does not establish semantic authority or complete private-documentation closure.
Terminal selector documentation reconciliation — 21 September 2026:
Signed Procd 68748bc65ea3fb798e810d624845dcb298ae1346 removes stale operation 55 prose and documents observation 59 plus exact acknowledgement 60. All runtime inputs are byte-identical to the validated consumer implementation; its four strict 299/305-test configurations remain applicable. Original documentation-checkpoint CI 300/301 passes from four complete hashed logs (347,266 bytes, zero warnings).
Ordinary consumer loss/restart and final reply disposition remain audited acceptance gaps. Failed ordinary metrics transport currently terminates Procd; it is not evidence of continued execution and row reuse. Full service CPU/profiler VM acceptance, complete realm lifecycle and both guest build generations remain open.
Actual service CPU scenarios and startup admission — 21 September 2026:
The published original Integration catalog
dff878dd3545c4751b3c05d37b2bdd5e21cce548passes the maintained shell times, two-CPU Extop and out-of-session denial VM scenarios under their unchanged 120-second guest limits. Each retains 106 hashed evidence files, warning-free image builds, empty QEMU stderr and every required, forbidden, ordered and unique marker check. Build-plus-scenario wall times are 119.876978, 34.802620 and 36.173027 seconds respectively; these are not guest-only or startup measurements.Times reports nonzero self and waited-child CPU. Extop observes both CPUs, memory and increasing job CPU nanoseconds with
CTRL no; both per-CPU percentages remain--.--%in the two samples, so numeric utilization is unproven. The denial scenario confirms the existing out-of-session boundary. Complete mediator/lifecycle acceptance in Procd 5 remains open.A separate ordinary development package also builds without warnings, but its required startup-contract preflight exits 1 before any VM or observer starts. Integration 69 records the missing full-runtime-transition contract selections and Kernel effective-feature/original-source evidence. Preserve the refused package and all admission controls; the 120/15/10 capture limits and performance thresholds are unchanged. No startup-profile acceptance is awarded. Complete source/effect/frame and Pagerd proof, native external Rust/LLVM/runtime rebuilding and both full EriX guest build generations remain required by Phase 6 completion.
Five actual service VMs pass: shell CPU accounting, exec successor replacement, two-CPU read-only inspection, out-of-session denial and guarded realm preparation.
Procd's four strict 308/314-test configurations and original CI 302/303 pass. The manual update passes 45 tests, all 2,431 pages, 450,185 word bounds and changed-page visual review. Original Docs CI 1003/1004 passes from four complete logs (774,710 bytes); retained reference-convergence warnings resolve to zero on final passes. All 3,166 authored code files remain below 1,000 lines.
Original Integration 1701/1702 remains running; 1703–1710 remains queued. Logd 240 remains failed with terminal logs unavailable through HTTP 500; no cause is inferred. No original job was restarted or cancelled. Remaining lifecycle control/event ownership, complete native fault/cleanup acceptance and the measured startup timing failures remain open. No whole acceptance leaf closes: 15/460, weighted 3.48%. Rebuilding the external Rust/LLVM toolchain inside EriX and using it in the required full EriX guest-build generations remain mandatory and unproven.
Distinct running-mediator scalar-retirement audit — 24 September 2026:
The current producer/consumer path leaves an additional final-accounting obligation after actual running mediator retirement. At Procd
d04311abff67e7355bd92bb2e48fb6161e0ec674, native terminal handling reads final CPU counters and setscpu_time_observation_pendingwhile the original authenticated Launchd consumer remains live. The RealmMediator branch disposes native custody, then restores the active scalar row with empty capability custody and its terminal counters. Managed-row reuse explicitly requires that pending flag to be false.Explicit private RETIRE does not observe or discharge those final counters. The separate exact terminal-event handoff correction owns notification disposal; consuming that event does not discharge this independent scalar obligation. At Launchd
cf6d4a70d5423dca65bbab5e8092fd374184c4ed, the realm event branch cancels the matched realm and returns before ordinary job accounting. Explicit realm retirement also performs no final process-times query. Existing session/job accounting enumerates ordinary jobs rather than mediator records.Consequently each fully terminated running mediator can retain a managed-table row until an authenticated exact final-counter query or termination of the original Launchd generation discharges it. Ordinary realm-client exit does not terminate that surviving consumer. Repeated create/retire cycles can therefore exhaust configured process bookkeeping even after the child's native and capability cleanup has succeeded. This is a concrete source-level resource-reuse gap; no guest exhaustion reproduction is claimed.
The existing
mediator_counters_survive_cleanup_and_lost_acknowledgementcontrol retains the final flag and clears it only after an explicit authenticated query. Direct custody-cleanup fixtures which empty a row do not model its subsequent restoration by the full native terminal consumer. This audit ran no compiler or VM and changes no current runtime acceptance. It requires an actual final-counter consumer or explicitly justified disposition, not removal of the pending flag merely to make storage reusable.The current two-mediator startup scenario creates both mediators before retiring either, then only reads and retires. It allocates no additional mediator after these final flags are set. This finding therefore does not require delaying that bounded startup/readiness witness, but a pass cannot establish complete retirement, repeated record reuse or final CPU/profiler accounting. Those remain mandatory in the realm lifecycle work, Procd PR 2, Launchd PR 2 and Phase 6 completion.
erikinkinen referenced this issue2026-09-21 19:16:59 +02:00
erikinkinen referenced this issue from erix/rootd2026-09-21 19:17:00 +02:00
erikinkinen referenced this issue from erix/docs2026-09-21 19:17:02 +02:00
erikinkinen referenced this issue from erix/integration2026-09-21 19:17:04 +02:00
erikinkinen referenced this issue from erix/launchd2026-09-25 08:05:24 +02:00
erikinkinen referenced this issue from erix/integration2026-09-25 13:15:44 +02:00
erikinkinen referenced this issue2026-10-30 12:04:20 +01:00