[BUG] Private Posixd startup returns INTERNAL after guarded preparation #89

Closed
opened 2026-09-24 20:37:01 +02:00 by erikinkinen · 0 comments
Owner

Summary

The maintained two-Posixd startup scenario completes admission and guarded preparation, but the first public START returns INTERNAL before authenticated readiness. The original scalar result did not identify an internal boundary. Subsequent exact-image hardware-breakpoint diagnosis now attributes it to Launchd retaining a copied configuration SEND while expecting MOVE; corrected native acceptance now passes as recorded below.

Reproduction

Build the original pinned graph at Integration 8f9b7d752c4769e3017aa4116ad0a542f54f488b and invoke:

bash scripts/run-scenario.sh tests/scenarios/appliance-disk-image-realm-startup-positive.toml components.toml

One fresh exact-source execution is retained. It uses the selected signing fixtures, full-service release image and original scenario configuration. No unchanged failed-input retry or failure-frequency claim is made.

Expected and actual behavior

Expected: both actual Posixd processes reach authenticated PrivateReady; retiring realm A preserves realm B; both retired handles are rejected, then the initial shell exits successfully within the maintained limits.

Actual: admission and preparation markers pass. Request 0x13, operation 0x6d4, returns typed rejection code 5 (INTERNAL). No startup marker occurs. Exsh's TCB 0x22 / endpoint 0x520 exits 0xe5; Rootd subsequently exits 0xdc. The 45-second progress watchdog terminates QEMU. The owned scenario command exits 1 after 169.570989 seconds including image construction, with no cleanup failure.

Environment and identity

  • Integration: 8f9b7d752c4769e3017aa4116ad0a542f54f488b.
  • Launchd: cf6d4a70d5423dca65bbab5e8092fd374184c4ed.
  • Posixd: 5f54a55a5f189559352c7d0e6a2af55aaa37c44f.
  • Procd: 4f3b01bf21932e86ac542f07e6ade05138cc0473.
  • Release x86-64 QEMU/OVMF full-service disk image, one CPU, original 120-second guest hard limit and 45-second progress watchdog; Rust 1.97.1.
  • Packaged image SHA-256: 396e15c2ba4f81bf2d39a86eb11d6b536588b32ad671c02a724ced602906c65b.
  • The original scoped executable and private control/configuration routes are unchanged; no ordinary launch sidecar or extra child authority was added.
  • Corrected full startup passes at Integration 66ddc78da31dd03dd554d0677cdc2af86588d525.

Evidence and prior checks

The maintained appliance-disk-image-realm-startup-positive scenario passes unchanged: both actual Posixd processes reach authenticated PrivateReady; A retires while B stays ready; B then retires; both stale handles are rejected; the initial shell exits successfully. The original required, unique, ordered and forbidden markers, expected guest exit 0x10, 120-second guest hard limit and 45-second serial-progress limit all pass.

This exercises counted minimum startup, actual capability census, setup-grant consumption and copied SEND disposal before Start, followed by native child registration, exact-origin readiness and the controller challenge. The packaged Posixd alias selects the same authenticated executable; no ordinary sidecar, extra child endpoint or fallback is introduced. Source-bound exercised gates are the evidence; no separate global inventory dump is claimed. Posixd's Ready caller remains its separately owned obligation until retirement. The successful explicit-retirement path does not exercise Launchd's spontaneous WAIT decoder, final CPU/profiler handoff, repeated row reuse, loss paths or capacity exhaustion.

Launchd's six strict local configurations pass 329 library and 69 runtime tests each; all 48 required commands are warning-free. The corrected native-faithful 14-test group includes COPY source retention, post-Provision caller retirement, separate source disposal/absence verification and failure preservation. Original Launchd CI155/156 also passes: four complete logs, 148,673 bytes, zero warning/failure candidates. Original Integration CI1791/1792 remains queued; earlier failures and regression reports remain preserved. The copied-SEND defect and its native startup regression are resolved at this scoped boundary. Relevant implementation reviews remain Launchd PR2, Procd PR2, Posixd PR5 and Integration PR12.

Configuration/seal/publication, four real clients and mediated byte I/O, complete running retirement and profiler disposition, rebuilding the external Rust/LLVM toolchain inside EriX and both complete EriX guest builds remain required and unproved. The WIP reviews remain unfinished.

Attributed native boundary — 24 September 2026: a bounded, changed-debugger execution of the same immutable image captures Launchd's first SourceAuthority failure in phase Provision, before native Start. The owner has no prior failure and retains the successful 224-byte Provision acknowledgment. Exact selected disassembly places the failure at verify_column(2): the actual local query finds Launchd's original SEND copy. Kernel owned invocation copies ordinary endpoint capabilities; the host fixture incorrectly consumed this source as though it were a unique grant. The owning Launchd report tracks the correction and host regression.

All packaged PT_LOAD bytes and geometry match the original symbol carriers after the maintained packaging capacity adjustment; running executable bytes also match. Hardware breakpoints and a private host Unix socket add no guest authority or source changes. The source image and diagnostic copy retain SHA-256 396e15c2ba4f81bf2d39a86eb11d6b536588b32ad671c02a724ced602906c65b. The diagnostic ends at the unchanged 45-second progress watchdog, exit 1 after 95.831832 seconds. Its GDB index-cache warning and repeated-SIGTERM wrapper traceback remain recorded separately; both child processes are reaped with no cleanup error. This is causal evidence, not native acceptance or a performance measurement. The original failed scenario below remains authoritative for acceptance.

All 107 evidence files, 56,013 serial bytes, original command status, image and source identities are retained. Source membership is unchanged through execution, and image compilation has zero warning candidates. Kernel/frame checks and the full image closure remain enabled.

The preceding graph returned NOT_FOUND (3); this changed graph includes the independently reproduced Launchd empty-intake fix, Posixd empty-intake fix, Procd registration wait and exact retirement-event disposition. Their strict local matrices and original component CI pass. They are not proof of this scenario's acceptance.

Public START includes both private Provision and Start; its scalar error alone did not identify an internal stage. The attributed diagnostic below supplements that original observation without changing the failed acceptance receipt. Earlier failures and their original evidence remain separate.

Tracking

Original historical CI1790 follow-up — 25 September 2026: Integration CI1790 at 8f9b7d752c4769e3017aa4116ad0a542f54f488b finishes with 490 passed and one failed out of 491 scenarios. Worker 8's realm-startup VM repeats START 0x6D4 result 5/INTERNAL after Guarded preparation, followed by the 45-second serial-progress deadline. This is the older graph already diagnosed in the scoped startup report; that scoped correction stays closed. The observation does not establish a regression in a later revision. All three complete logs independently verify 13,610,151 bytes with zero warning candidates; Rust and Markdown pass. The initially budget-limited Integration log was completed with a bounded request for only its missing suffix, preserving the original verified prefix. Original CI1791 and CI1792 remain running. No unchanged rerun or relaxed deadline is used, and full native client execution remains unaccepted.

Original historical CI1788/1789 follow-up — 25 September 2026: run1789, original 8f9b7d752c4769e3017aa4116ad0a542f54f488b, reproduces START0x6D4 result5/INTERNAL after Guarded preparation and then the45-second serial watchdog; its catalog passes490/491. This is the source graph already diagnosed and corrected by the later accepted native startup witness. Run1788, original fa857c0964774def6777142afeb7a3c16440abee, instead reports START result3/NOT_FOUND; it is not evidence of the same cause. Its catalog passes483/491 with separate filesystem and pipeline failures. The scoped correction stays closed; full client-execution acceptance remains open. All original logs are preserved without retries.

Older original CI follow-up — 25 September 2026: run1787 at fa857c0964774def6777142afeb7a3c16440abee reports realm START NOT_FOUND (operation0x6D4, result3), then the unchanged45-second progress watchdog expires. This is not the same reported result as the issue’s START/INTERNAL observation; cause remains unclassified. The graph predates the accepted66ddc78/3d39c74 startup correction. Its other failure is the separately tracked ext4 quota120-second overall timeout;489/491 scenarios pass. No original job was rerun.

Track diagnosis and correction through Integration PR 12 and Phase 6 completion.

  • Identify the original failing boundary from source-bound native evidence (attributed exact-image diagnostic; runtime acceptance remains separate).
  • Reproduce and correct the identified defect with meaningful regression coverage.
  • Pass applicable formatting, strict Clippy, host/native tests and warning-free image checks.
  • Pass the maintained two-mediator startup scenario with every original marker, identity, authority, cleanup and timing gate.

Configuration/seal, four actual clients, byte I/O, full resource retirement, the external toolchain guest rebuild and both complete EriX guest builds remain open.

## Summary The maintained two-Posixd startup scenario completes admission and guarded preparation, but the first public START returns INTERNAL before authenticated readiness. The original scalar result did not identify an internal boundary. Subsequent exact-image hardware-breakpoint diagnosis now attributes it to [Launchd retaining a copied configuration SEND while expecting MOVE](https://git.erikinkinen.fi/erix/launchd/issues/7); corrected native acceptance now passes as recorded below. ## Reproduction Build the original pinned graph at Integration `8f9b7d752c4769e3017aa4116ad0a542f54f488b` and invoke: ```sh bash scripts/run-scenario.sh tests/scenarios/appliance-disk-image-realm-startup-positive.toml components.toml ``` One fresh exact-source execution is retained. It uses the selected signing fixtures, full-service release image and original scenario configuration. No unchanged failed-input retry or failure-frequency claim is made. ## Expected and actual behavior Expected: both actual Posixd processes reach authenticated PrivateReady; retiring realm A preserves realm B; both retired handles are rejected, then the initial shell exits successfully within the maintained limits. Actual: admission and preparation markers pass. Request `0x13`, operation `0x6d4`, returns typed rejection code `5` (INTERNAL). No startup marker occurs. Exsh's TCB `0x22` / endpoint `0x520` exits `0xe5`; Rootd subsequently exits `0xdc`. The 45-second progress watchdog terminates QEMU. The owned scenario command exits 1 after 169.570989 seconds including image construction, with no cleanup failure. ## Environment and identity - Integration: `8f9b7d752c4769e3017aa4116ad0a542f54f488b`. - Launchd: `cf6d4a70d5423dca65bbab5e8092fd374184c4ed`. - Posixd: `5f54a55a5f189559352c7d0e6a2af55aaa37c44f`. - Procd: `4f3b01bf21932e86ac542f07e6ade05138cc0473`. - Release x86-64 QEMU/OVMF full-service disk image, one CPU, original 120-second guest hard limit and 45-second progress watchdog; Rust 1.97.1. - Packaged image SHA-256: `396e15c2ba4f81bf2d39a86eb11d6b536588b32ad671c02a724ced602906c65b`. - The original scoped executable and private control/configuration routes are unchanged; no ordinary launch sidecar or extra child authority was added. - Corrected full startup passes at Integration `66ddc78da31dd03dd554d0677cdc2af86588d525`. ## Evidence and prior checks The maintained `appliance-disk-image-realm-startup-positive` scenario passes unchanged: both actual Posixd processes reach authenticated PrivateReady; A retires while B stays ready; B then retires; both stale handles are rejected; the initial shell exits successfully. The original required, unique, ordered and forbidden markers, expected guest exit `0x10`, 120-second guest hard limit and 45-second serial-progress limit all pass. This exercises counted minimum startup, actual capability census, setup-grant consumption and copied SEND disposal before Start, followed by native child registration, exact-origin readiness and the controller challenge. The packaged Posixd alias selects the same authenticated executable; no ordinary sidecar, extra child endpoint or fallback is introduced. Source-bound exercised gates are the evidence; no separate global inventory dump is claimed. Posixd's Ready caller remains its separately owned obligation until retirement. The successful explicit-retirement path does not exercise Launchd's spontaneous WAIT decoder, final CPU/profiler handoff, repeated row reuse, loss paths or capacity exhaustion. Launchd's six strict local configurations pass 329 library and 69 runtime tests each; all 48 required commands are warning-free. The corrected native-faithful 14-test group includes COPY source retention, post-Provision caller retirement, separate source disposal/absence verification and failure preservation. Original Launchd CI155/156 also passes: four complete logs, 148,673 bytes, zero warning/failure candidates. Original Integration CI1791/1792 remains queued; earlier failures and regression reports remain preserved. [The copied-SEND defect](https://git.erikinkinen.fi/erix/launchd/issues/7) and [its native startup regression](https://git.erikinkinen.fi/erix/integration/issues/89) are resolved at this scoped boundary. Relevant implementation reviews remain [Launchd PR2](https://git.erikinkinen.fi/erix/launchd/pulls/2), [Procd PR2](https://git.erikinkinen.fi/erix/procd/pulls/2), [Posixd PR5](https://git.erikinkinen.fi/erix/posixd/pulls/5) and [Integration PR12](https://git.erikinkinen.fi/erix/integration/pulls/12). Configuration/seal/publication, four real clients and mediated byte I/O, complete running retirement and profiler disposition, rebuilding the external Rust/LLVM toolchain inside EriX and both complete EriX guest builds remain required and unproved. The WIP reviews remain unfinished. Attributed native boundary — 24 September 2026: a bounded, changed-debugger execution of the same immutable image captures Launchd's first `SourceAuthority` failure in phase Provision, before native Start. The owner has no prior failure and retains the successful 224-byte Provision acknowledgment. Exact selected disassembly places the failure at `verify_column(2)`: the actual local query finds Launchd's original SEND copy. Kernel owned invocation copies ordinary endpoint capabilities; the host fixture incorrectly consumed this source as though it were a unique grant. [The owning Launchd report](https://git.erikinkinen.fi/erix/launchd/issues/7) tracks the correction and host regression. All packaged PT_LOAD bytes and geometry match the original symbol carriers after the maintained packaging capacity adjustment; running executable bytes also match. Hardware breakpoints and a private host Unix socket add no guest authority or source changes. The source image and diagnostic copy retain SHA-256 `396e15c2ba4f81bf2d39a86eb11d6b536588b32ad671c02a724ced602906c65b`. The diagnostic ends at the unchanged 45-second progress watchdog, exit 1 after 95.831832 seconds. Its GDB index-cache warning and repeated-SIGTERM wrapper traceback remain recorded separately; both child processes are reaped with no cleanup error. This is causal evidence, not native acceptance or a performance measurement. The original failed scenario below remains authoritative for acceptance. All 107 evidence files, 56,013 serial bytes, original command status, image and source identities are retained. Source membership is unchanged through execution, and image compilation has zero warning candidates. Kernel/frame checks and the full image closure remain enabled. The preceding graph returned NOT_FOUND (`3`); this changed graph includes the independently reproduced [Launchd empty-intake fix](https://git.erikinkinen.fi/erix/launchd/issues/5), [Posixd empty-intake fix](https://git.erikinkinen.fi/erix/posixd/issues/7), [Procd registration wait](https://git.erikinkinen.fi/erix/procd/issues/8) and [exact retirement-event disposition](https://git.erikinkinen.fi/erix/procd/issues/7). Their strict local matrices and original component CI pass. They are not proof of this scenario's acceptance. Public START includes both private Provision and Start; its scalar error alone did not identify an internal stage. The attributed diagnostic below supplements that original observation without changing the failed acceptance receipt. Earlier failures and their original evidence remain separate. ## Tracking Original historical CI1790 follow-up — 25 September 2026: [Integration CI1790](https://git.erikinkinen.fi/erix/integration/actions/runs/1790) at `8f9b7d752c4769e3017aa4116ad0a542f54f488b` finishes with 490 passed and one failed out of 491 scenarios. Worker 8's realm-startup VM repeats START `0x6D4` result 5/INTERNAL after Guarded preparation, followed by the 45-second serial-progress deadline. This is the older graph already diagnosed in [the scoped startup report](https://git.erikinkinen.fi/erix/integration/issues/89); that scoped correction stays closed. The observation does not establish a regression in a later revision. All three complete logs independently verify 13,610,151 bytes with zero warning candidates; Rust and Markdown pass. The initially budget-limited Integration log was completed with a bounded request for only its missing suffix, preserving the original verified prefix. Original [CI1791](https://git.erikinkinen.fi/erix/integration/actions/runs/1791) and [CI1792](https://git.erikinkinen.fi/erix/integration/actions/runs/1792) remain running. No unchanged rerun or relaxed deadline is used, and full native client execution remains unaccepted. Original historical CI1788/1789 follow-up — 25 September 2026: [run1789](https://git.erikinkinen.fi/erix/integration/actions/runs/1789), original `8f9b7d752c4769e3017aa4116ad0a542f54f488b`, reproduces START0x6D4 result5/INTERNAL after Guarded preparation and then the45-second serial watchdog; its catalog passes490/491. This is the source graph already diagnosed and corrected by the later accepted native startup witness. [Run1788](https://git.erikinkinen.fi/erix/integration/actions/runs/1788), original `fa857c0964774def6777142afeb7a3c16440abee`, instead reports START result3/NOT_FOUND; it is not evidence of the same cause. Its catalog passes483/491 with separate filesystem and pipeline failures. The scoped correction stays closed; full client-execution acceptance remains open. All original logs are preserved without retries. Older original CI follow-up — 25 September 2026: [run1787](https://git.erikinkinen.fi/erix/integration/actions/runs/1787) at `fa857c0964774def6777142afeb7a3c16440abee` reports realm START NOT_FOUND (operation0x6D4, result3), then the unchanged45-second progress watchdog expires. This is not the same reported result as the issue’s START/INTERNAL observation; cause remains unclassified. The graph predates the accepted66ddc78/3d39c74 startup correction. Its other failure is the separately tracked ext4 quota120-second overall timeout;489/491 scenarios pass. No original job was rerun. Track diagnosis and correction through [Integration PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12) and [Phase 6 completion](https://git.erikinkinen.fi/erix/integration/issues/65). - [x] Identify the original failing boundary from source-bound native evidence (attributed exact-image diagnostic; runtime acceptance remains separate). - [x] Reproduce and correct the identified defect with meaningful regression coverage. - [x] Pass applicable formatting, strict Clippy, host/native tests and warning-free image checks. - [x] Pass the maintained two-mediator startup scenario with every original marker, identity, authority, cleanup and timing gate. Configuration/seal, four actual clients, byte I/O, full resource retirement, the external toolchain guest rebuild and both complete EriX guest builds remain open.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:39:19 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/integration#89
No description provided.