[BUG] Configured realm-client VM stalls after a private process exit #92

Closed
opened 2026-09-25 11:34:21 +02:00 by erikinkinen · 0 comments
Owner

Summary

The configured realm-client VM stalls after its three inherited realm markers and a process exit with status 5. The configured-client success marker and ordinary shell exit never appear. The generic native exit evidence does not identify the component or failing operation; ownership and root cause remain under investigation.

Reproduction

Select signed Integration 04bff04f and its complete original source catalog, then run:

bash scripts/run-scenario.sh tests/scenarios/appliance-disk-image-realm-clients-positive.toml components.toml

The first fresh local run fails under the original 120-second guest deadline and 45-second serial-progress deadline. No unchanged retry or relaxed bound was used.

Expected and actual behavior

Both configured realms must seal, execute their reciprocal byte clients with actual successful terminal outcomes, retire independently and support the replacement pair. All inherited markers and ordinary shell exit remain required.

Actual output reaches REALM_ADMISSION, REALM_PREPARATION and REALM_STARTUP VERIFIED, followed by:

ERIX_KERNEL:PROCESS_EXIT_TCB=0000000000000023
ERIX_KERNEL:PROCESS_EXIT_EP=0000000000000004
ERIX_KERNEL:PROCESS_EXIT_CODE=0000000000000005

The runner ends with status 1 after serial progress stops. These TCB/endpoint numbers alone do not identify the executable, establish client startup or explain the failure. The previous first-client preparation refusal does not recur in the visible serial output; successful complete client preparation is not inferred from its absence.

Environment and identity

  • Integration: 04bff04f4328676ba8205f7ac2b0df64213698a0; original signed graph of 76 repositories.
  • Loaderd: 5bcf522cfec873fa29ffe1b91bde1bf67e9f5ba2; Exsh: f36b1f51f138f4c5c56fcac45a881731d0471c81.
  • Launchd: 0b0788b5530ce7607097eee97babe22e71d8c1b9; Posixd: 3541ce5d621bfbf32e89df1882c3b15cb5ecd707.
  • Release x86_64 appliance, one CPU, two realm records/four client rows and 576 plan bytes; existing caller-owned shell routes only.
  • Packaged disk SHA256: 61d30816055d8e38555f1b11f1c015fb9af8655dc79e0faf9be6d0e0e8632085; retained post-VM disk bytes have the same digest.

Evidence and prior checks

Every one of the 119 failures in each run is the known policy-format issue: the older producer emits 14 fields while the consumer requires 16.

Admission, guarded preparation, private startup, configured clients, the pipeline descriptor script and ext4 quota all pass in both original runs. The selected Launchd 372ab70ea2ea1133a9bc3edb2cb621f9679b075d includes the configuration-carrier GRANT correction; the earlier client stall is absent. This corroborates the existing scoped closure of Launchd issue 10 and Integration issue 92. The unexplained older same-source pipeline pass/failure in issue 91 remains open; two later passes do not establish its cause. Original markers and deadlines are unchanged, and no unchanged job was rerun.

Both configured-client scenarios pass the admission, preparation and startup markers, then the original mediator process 35 exits with code 5 and the VM reaches its 45-second serial-progress deadline. The selected Launchd 0b0788b5530ce7607097eee97babe22e71d8c1b9 predates the configuration-carrier GRANT correction and is the exact graph already reported in Launchd issue 10 and Integration issue 92. Loaderd already contains its separate counted grant-destination fix. This historical evidence does not reopen the later corrected source; its component and native acceptance remain tied to their signed identities.

Native configured-client acceptance — 25 September 2026: the exact signed Integration 7a42fa26 graph passes the configured-client, preparation and startup VMs with all original markers, deadlines and cleanup gates. The configured witness seals two isolated realms, observes six successful byte-client exits across three realm instances, checks independent retirement, rejects stale client selectors and reuses a Launchd realm record with a new generation. This proves the native configure/seal/publish/mediated-I/O criterion; full Procd managed-row/final CPU reclamation, supervisor loss and resource ceilings remain separate requirements.

Launchd 372ab70e retains only the required result-carrier GRANT and preserves RECV-only client copies plus complete setup retirement. All six strict 497-test configurations and original CI179/180 pass. Loaderd 5bcf522c retains the counted grant destination; four strict 145-test configurations and original CI107/108 pass. The original red tests, failed native runs and admitted host TCG diagnostic remain preserved. Docs 6a1a9686 passes the full 2,655-page manual and 45 tests locally and in original CI1085/1086; four complete CI logs total 824,660 bytes and final warnings are zero.

The configuration carrier defect, counted grant defect and configured-client stall are resolved by this component and native evidence. The Exsh fixture report retains its pending release compiler/frame validation despite the three native passes. Complete Integration CI1801/1802 is still queued. Whole-fleet authority/source/effect/frame acceptance, final accounting/profiler observation, rebuilding the external Rust/LLVM/runtime toolchain inside EriX and both full EriX guest-build generations remain open.

Original prerequisite Integration CI1793/1794 each passes 490/491 scenarios; Rust and Markdown pass, and six complete logs total 27,432,406 bytes with zero warnings. CI1794 retains the ext4 quota timeout. CI1793 instead exposes a distinct one-shot replay QMP connection loss, recorded using the canonical bug template and phase label. No original workflow was rerun, and no deadline or authority check was relaxed.

Only the existing native-witness sub-item gains credit: 15.42% weighted progress, 413/873 sub-items. Whole acceptance remains 3.86%, 17/460 parents.

Signed configuration carrier correction — 25 September 2026: Launchd 372ab70e retains exactly RECV | GRANT only for counted-client result delivery. Readiness-only RECV, mediator SEND-only routes, RECV-only client copies, original authentication and complete setup disposal before seal/retirement remain required.

Docs 6a1a9686 documents the carrier/source distinction and its temporary setup lifetime. All 45 document tests, Markdown, template, link and unchanged-reference checks pass. The complete 2,655-page manual builds without final warnings, all 491,315 word bounds pass and three affected/adjacent pages are visually reviewed. Original CI1085/1086 is under observation.

Integration 7a42fa26 selects the two signed image sources. Both catalog policies, all 76 original signatures and complete dynamic closure pass. Helper, Rust, profiler and CI inputs remain byte-identical to their recorded validation; fresh formatting, source policy and Markdown pass. The corrected client VM is running from a fresh source-bound runner, followed by the older preparation/startup regressions. Original CI1801/1802 is queued. Launchd issue 10 and Integration issue 92 remain open pending native closure. Earlier failures and host counter evidence are preserved. No deadline or authority gate was relaxed, and these repairs add no canonical checklist credit.

The configured-client VM stalls after the same three inherited markers and a private process exit with status 5. Its original failure retains 109 artifacts and 56,130 serial bytes, zero image-warning candidates and no process-owner cleanup failure. Integration issue 92 preserves that observation. An admitted host TCG counter run against the identical guest disk reproduces the stall under the original 120/45-second bounds. Its complete capture contains 100,639 translations and 74,009 code groups. The top 100 groups cover 85.32% of the 65,912,963,324 translated-instruction upper bound; uniquely matching Kernel mapping-batch code accounts for 38.67% of that total. This is a performance investigation candidate, without a measured speedup, timing attribution or accepted native result. Shared code, addresses and process identity remain separate evidence questions.

Exact retained executable-region comparison with original symbol-bearing artifacts narrows the failure to Posixd configuration progress and its native exit; the exit block uniquely matches Posixd across all 68 packaged executables/shared objects. The source mismatch is confirmed independently: Launchd retains only RECV on its configuration receiver, while native COMPLETE requires carrier GRANT for capability-bearing client replies. Enforcing that actual gate in the maintained complete-record pull fixture fails the original implementation. Launchd issue 10 tracks the correction and original red control. The local correction retains exactly RECV | GRANT only for counted-client replies, preserves readiness-only RECV, mediator SEND-only routes and RECV-only client copies, and retains complete receiver disposal before seal/cancellation. Full component validation is running; corrected native client execution remains required.

Both original Docs CI1083/1084 pass, with 45 document tests and the 2,653-page manual. All four complete logs total 824,211 bytes; normal early-pass references converge to zero final warnings. Original Integration runs remain running or queued, without unchanged reruns. No canonical checklist credit is added: configured native clients, managed-row reuse, complete authority/frame closure, profiling improvements and both full guest-build requirements remain open.

The failed execution retains 109 artifacts and 56,130 serial bytes, zero image-warning candidates and no process-owner cleanup failure. Exact source, packaged artifacts, bounded runner status, serial output and writable disk are retained. Both preceding prerequisite VMs passed on the previous corrected-fixture graph; their regressions on this graph remain separate checks.

Loaderd's counted destination correction passes four configurations of 145 tests, strict host/native checks and original CI107/108. That component evidence cannot establish this full scenario. The next diagnostic will preserve the original guest inputs and bounds and use explicitly selected host execution counters to narrow attribution; generic status 5 is insufficient for a source-level root-cause claim.

Tracking

Counted destination defect, mediator fixture defect, Integration PR12, and phase completion.

  • Preserve the original failed execution and exact source/artifact identity.
  • Identify the original failing component and operation without adding guest diagnostic authority or weakening gates.
  • Reproduce and correct the responsible behavior with actual-code controls.
  • Pass the configured-client scenario and older native regressions, strict affected-component validation and original CI observation.
## Summary The configured realm-client VM stalls after its three inherited realm markers and a process exit with status 5. The configured-client success marker and ordinary shell exit never appear. The generic native exit evidence does not identify the component or failing operation; ownership and root cause remain under investigation. ## Reproduction Select signed [Integration 04bff04f](https://git.erikinkinen.fi/erix/integration/commit/04bff04f4328676ba8205f7ac2b0df64213698a0) and its complete original source catalog, then run: ```sh bash scripts/run-scenario.sh tests/scenarios/appliance-disk-image-realm-clients-positive.toml components.toml ``` The first fresh local run fails under the original 120-second guest deadline and 45-second serial-progress deadline. No unchanged retry or relaxed bound was used. ## Expected and actual behavior Both configured realms must seal, execute their reciprocal byte clients with actual successful terminal outcomes, retire independently and support the replacement pair. All inherited markers and ordinary shell exit remain required. Actual output reaches REALM_ADMISSION, REALM_PREPARATION and REALM_STARTUP VERIFIED, followed by: ```text ERIX_KERNEL:PROCESS_EXIT_TCB=0000000000000023 ERIX_KERNEL:PROCESS_EXIT_EP=0000000000000004 ERIX_KERNEL:PROCESS_EXIT_CODE=0000000000000005 ``` The runner ends with status 1 after serial progress stops. These TCB/endpoint numbers alone do not identify the executable, establish client startup or explain the failure. The previous first-client preparation refusal does not recur in the visible serial output; successful complete client preparation is not inferred from its absence. ## Environment and identity - Integration: `04bff04f4328676ba8205f7ac2b0df64213698a0`; original signed graph of 76 repositories. - Loaderd: `5bcf522cfec873fa29ffe1b91bde1bf67e9f5ba2`; Exsh: `f36b1f51f138f4c5c56fcac45a881731d0471c81`. - Launchd: `0b0788b5530ce7607097eee97babe22e71d8c1b9`; Posixd: `3541ce5d621bfbf32e89df1882c3b15cb5ecd707`. - Release x86_64 appliance, one CPU, two realm records/four client rows and 576 plan bytes; existing caller-owned shell routes only. - Packaged disk SHA256: `61d30816055d8e38555f1b11f1c015fb9af8655dc79e0faf9be6d0e0e8632085`; retained post-VM disk bytes have the same digest. ## Evidence and prior checks Every one of the 119 failures in each run is the known [policy-format issue](https://git.erikinkinen.fi/erix/integration/issues/96): the older producer emits 14 fields while the consumer requires 16. Admission, guarded preparation, private startup, configured clients, the pipeline descriptor script and ext4 quota all pass in both original runs. The selected Launchd `372ab70ea2ea1133a9bc3edb2cb621f9679b075d` includes the configuration-carrier GRANT correction; the earlier client stall is absent. This corroborates the existing scoped closure of [Launchd issue 10](https://git.erikinkinen.fi/erix/launchd/issues/10) and [Integration issue 92](https://git.erikinkinen.fi/erix/integration/issues/92). The unexplained older same-source pipeline pass/failure in [issue 91](https://git.erikinkinen.fi/erix/integration/issues/91) remains open; two later passes do not establish its cause. Original markers and deadlines are unchanged, and no unchanged job was rerun. Both configured-client scenarios pass the admission, preparation and startup markers, then the original mediator process 35 exits with code 5 and the VM reaches its 45-second serial-progress deadline. The selected Launchd `0b0788b5530ce7607097eee97babe22e71d8c1b9` predates the configuration-carrier GRANT correction and is the exact graph already reported in [Launchd issue 10](https://git.erikinkinen.fi/erix/launchd/issues/10) and [Integration issue 92](https://git.erikinkinen.fi/erix/integration/issues/92). Loaderd already contains its separate counted grant-destination fix. This historical evidence does not reopen the later corrected source; its component and native acceptance remain tied to their signed identities. Native configured-client acceptance — 25 September 2026: the exact signed [Integration 7a42fa26](https://git.erikinkinen.fi/erix/integration/commit/7a42fa2663bda878edcf59de01d972dd20c58ec7) graph passes the configured-client, preparation and startup VMs with all original markers, deadlines and cleanup gates. The configured witness seals two isolated realms, observes six successful byte-client exits across three realm instances, checks independent retirement, rejects stale client selectors and reuses a Launchd realm record with a new generation. This proves the native configure/seal/publish/mediated-I/O criterion; full Procd managed-row/final CPU reclamation, supervisor loss and resource ceilings remain separate requirements. [Launchd 372ab70e](https://git.erikinkinen.fi/erix/launchd/commit/372ab70ea2ea1133a9bc3edb2cb621f9679b075d) retains only the required result-carrier GRANT and preserves RECV-only client copies plus complete setup retirement. All six strict 497-test configurations and original CI179/180 pass. [Loaderd 5bcf522c](https://git.erikinkinen.fi/erix/loaderd/commit/5bcf522cfec873fa29ffe1b91bde1bf67e9f5ba2) retains the counted grant destination; four strict 145-test configurations and original CI107/108 pass. The original red tests, failed native runs and admitted host TCG diagnostic remain preserved. [Docs 6a1a9686](https://git.erikinkinen.fi/erix/docs/commit/6a1a96869736030727d0366c3e7622993dded7ff) passes the full 2,655-page manual and 45 tests locally and in original CI1085/1086; four complete CI logs total 824,660 bytes and final warnings are zero. The [configuration carrier defect](https://git.erikinkinen.fi/erix/launchd/issues/10), [counted grant defect](https://git.erikinkinen.fi/erix/loaderd/issues/5) and [configured-client stall](https://git.erikinkinen.fi/erix/integration/issues/92) are resolved by this component and native evidence. The [Exsh fixture report](https://git.erikinkinen.fi/erix/exsh/issues/10) retains its pending release compiler/frame validation despite the three native passes. Complete Integration CI1801/1802 is still queued. Whole-fleet authority/source/effect/frame acceptance, final accounting/profiler observation, rebuilding the external Rust/LLVM/runtime toolchain inside EriX and both full EriX guest-build generations remain open. Original prerequisite Integration CI1793/1794 each passes 490/491 scenarios; Rust and Markdown pass, and six complete logs total 27,432,406 bytes with zero warnings. CI1794 retains the [ext4 quota timeout](https://git.erikinkinen.fi/erix/integration/issues/18). CI1793 instead exposes a distinct [one-shot replay QMP connection loss](https://git.erikinkinen.fi/erix/integration/issues/93), recorded using the canonical bug template and phase label. No original workflow was rerun, and no deadline or authority check was relaxed. Only the existing native-witness sub-item gains credit: **15.42% weighted progress**, 413/873 sub-items. Whole acceptance remains **3.86%, 17/460 parents**. Signed configuration carrier correction — 25 September 2026: [Launchd 372ab70e](https://git.erikinkinen.fi/erix/launchd/commit/372ab70ea2ea1133a9bc3edb2cb621f9679b075d) retains exactly RECV | GRANT only for counted-client result delivery. Readiness-only RECV, mediator SEND-only routes, RECV-only client copies, original authentication and complete setup disposal before seal/retirement remain required. [Docs 6a1a9686](https://git.erikinkinen.fi/erix/docs/commit/6a1a96869736030727d0366c3e7622993dded7ff) documents the carrier/source distinction and its temporary setup lifetime. All 45 document tests, Markdown, template, link and unchanged-reference checks pass. The complete 2,655-page manual builds without final warnings, all 491,315 word bounds pass and three affected/adjacent pages are visually reviewed. Original CI1085/1086 is under observation. [Integration 7a42fa26](https://git.erikinkinen.fi/erix/integration/commit/7a42fa2663bda878edcf59de01d972dd20c58ec7) selects the two signed image sources. Both catalog policies, all 76 original signatures and complete dynamic closure pass. Helper, Rust, profiler and CI inputs remain byte-identical to their recorded validation; fresh formatting, source policy and Markdown pass. The corrected client VM is running from a fresh source-bound runner, followed by the older preparation/startup regressions. Original CI1801/1802 is queued. [Launchd issue 10](https://git.erikinkinen.fi/erix/launchd/issues/10) and [Integration issue 92](https://git.erikinkinen.fi/erix/integration/issues/92) remain open pending native closure. Earlier failures and host counter evidence are preserved. No deadline or authority gate was relaxed, and these repairs add no canonical checklist credit. The configured-client VM stalls after the same three inherited markers and a private process exit with status 5. Its original failure retains 109 artifacts and 56,130 serial bytes, zero image-warning candidates and no process-owner cleanup failure. [Integration issue 92](https://git.erikinkinen.fi/erix/integration/issues/92) preserves that observation. An admitted host TCG counter run against the identical guest disk reproduces the stall under the original 120/45-second bounds. Its complete capture contains 100,639 translations and 74,009 code groups. The top 100 groups cover 85.32% of the 65,912,963,324 translated-instruction upper bound; uniquely matching Kernel mapping-batch code accounts for 38.67% of that total. This is a performance investigation candidate, without a measured speedup, timing attribution or accepted native result. Shared code, addresses and process identity remain separate evidence questions. Exact retained executable-region comparison with original symbol-bearing artifacts narrows the failure to Posixd configuration progress and its native exit; the exit block uniquely matches Posixd across all 68 packaged executables/shared objects. The source mismatch is confirmed independently: Launchd retains only RECV on its configuration receiver, while native COMPLETE requires carrier GRANT for capability-bearing client replies. Enforcing that actual gate in the maintained complete-record pull fixture fails the original implementation. [Launchd issue 10](https://git.erikinkinen.fi/erix/launchd/issues/10) tracks the correction and original red control. The local correction retains exactly RECV | GRANT only for counted-client replies, preserves readiness-only RECV, mediator SEND-only routes and RECV-only client copies, and retains complete receiver disposal before seal/cancellation. Full component validation is running; corrected native client execution remains required. Both original Docs CI1083/1084 pass, with 45 document tests and the 2,653-page manual. All four complete logs total 824,211 bytes; normal early-pass references converge to zero final warnings. Original Integration runs remain running or queued, without unchanged reruns. No canonical checklist credit is added: configured native clients, managed-row reuse, complete authority/frame closure, profiling improvements and both full guest-build requirements remain open. The failed execution retains 109 artifacts and 56,130 serial bytes, zero image-warning candidates and no process-owner cleanup failure. Exact source, packaged artifacts, bounded runner status, serial output and writable disk are retained. Both preceding prerequisite VMs passed on the previous corrected-fixture graph; their regressions on this graph remain separate checks. Loaderd's counted destination correction passes four configurations of 145 tests, strict host/native checks and original CI107/108. That component evidence cannot establish this full scenario. The next diagnostic will preserve the original guest inputs and bounds and use explicitly selected host execution counters to narrow attribution; generic status 5 is insufficient for a source-level root-cause claim. ## Tracking [Counted destination defect](https://git.erikinkinen.fi/erix/loaderd/issues/5), [mediator fixture defect](https://git.erikinkinen.fi/erix/exsh/issues/10), [Integration PR12](https://git.erikinkinen.fi/erix/integration/pulls/12), and [phase completion](https://git.erikinkinen.fi/erix/integration/issues/65). - [x] Preserve the original failed execution and exact source/artifact identity. - [x] Identify the original failing component and operation without adding guest diagnostic authority or weakening gates. - [x] Reproduce and correct the responsible behavior with actual-code controls. - [x] Pass the configured-client scenario and older native regressions, strict affected-component validation and original CI observation.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:39:19 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/integration#92
No description provided.