[BUG] Multi-PTY VM stops during initial-shell cleanup #125

Open
opened 2026-10-15 02:56:21 +02:00 by erikinkinen · 11 comments
Owner

Summary

The multi-PTY VM case stops during initial-shell launch, before it can exercise either terminal session.

Reproduction

At Integration 94802d9c1650821f8cdda36a49d84b91ffbf4c5e, run:

./scripts/run-scenario.sh tests/scenarios/appliance-disk-image-tty-multi-pty-positive.toml components.toml

One execution reproduced the failure.

Expected and actual behavior

The shell should process the physical-input fixture and produce MULTIPTY and the session-retirement witnesses. Instead, the guest reaches service readiness, prints its EriX banner and reports ERIX_ROOTD:INITIAL_EXSH:CLEANUP_UNCERTAIN. No QMP input is sent. The unchanged 45-second serial watchdog then stops QEMU, and the scenario exits with status 1.

Environment and identity

Integration 94802d9c, Rootd af0b4261, Exsh 6ffb6994; compiled orchestration source 32fb0a10. The original case selects the Rust release profile, development runtime configuration, one VM CPU and the tty-multi-pty physical-input recipe.

Evidence and prior checks

The original image, source identities, serial output and QEMU output are retained. The case took 339.78269 seconds and its outer cleanup completed. The final MetadataFailure message wraps the scenario failure; it does not establish a Cargo metadata error. That original run did not identify the failing launch or cleanup operation.

Tracking

Phase completion. The separate Exsh frame-checker issue is not an established cause.

  • Identify the failing launch or cleanup operation.
  • Repair it while preserving authority checks and failure cleanup.
  • Complete the unchanged multi-PTY case on the repaired source graph.
## Summary The multi-PTY VM case stops during initial-shell launch, before it can exercise either terminal session. ## Reproduction At Integration `94802d9c1650821f8cdda36a49d84b91ffbf4c5e`, run: ```sh ./scripts/run-scenario.sh tests/scenarios/appliance-disk-image-tty-multi-pty-positive.toml components.toml ``` One execution reproduced the failure. ## Expected and actual behavior The shell should process the physical-input fixture and produce `MULTIPTY` and the session-retirement witnesses. Instead, the guest reaches service readiness, prints its EriX banner and reports `ERIX_ROOTD:INITIAL_EXSH:CLEANUP_UNCERTAIN`. No QMP input is sent. The unchanged 45-second serial watchdog then stops QEMU, and the scenario exits with status 1. ## Environment and identity Integration `94802d9c`, Rootd `af0b4261`, Exsh `6ffb6994`; compiled orchestration source `32fb0a10`. The original case selects the Rust release profile, development runtime configuration, one VM CPU and the `tty-multi-pty` physical-input recipe. ## Evidence and prior checks The original image, source identities, serial output and QEMU output are retained. The case took 339.78269 seconds and its outer cleanup completed. The final `MetadataFailure` message wraps the scenario failure; it does not establish a Cargo metadata error. That original run did not identify the failing launch or cleanup operation. ## Tracking [Phase completion](https://git.erikinkinen.fi/erix/integration/issues/65). The separate [Exsh frame-checker issue](https://git.erikinkinen.fi/erix/exsh/issues/4) is not an established cause. - [x] Identify the failing launch or cleanup operation. - [ ] Repair it while preserving authority checks and failure cleanup. - [ ] Complete the unchanged multi-PTY case on the repaired source graph.
Author
Owner

Rootd 9c5de279 now records native-error and call-framing context through the existing Logd route. The next run on the updated source graph will use these diagnostics to identify the failing operation. The original failure remains preserved; no repaired guest result is claimed.

Rootd [9c5de279](https://git.erikinkinen.fi/erix/rootd/commit/9c5de279a04fe31c869bad70040df9b0cf4b98e1) now records native-error and call-framing context through the existing Logd route. The next run on the updated source graph will use these diagnostics to identify the failing operation. The original failure remains preserved; no repaired guest result is claimed.
Author
Owner

The run on ff0b5c29 identifies the failed operation: the first bootstrap-session request succeeds, then the second call to TTYd returns native NOT_FOUND. Cleanup cannot obtain a reply from the first session’s lifecycle endpoint either. The original watchdog stops the VM, with outer cleanup confirmed.

I’m tracing TTYd’s path after the first reply. The reason for the lost service route remains unknown; the repair and unchanged-scenario criteria stay open.

The run on [ff0b5c29](https://git.erikinkinen.fi/erix/integration/commit/ff0b5c29da030fa80a717db36db76e74733fb717) identifies the failed operation: the first bootstrap-session request succeeds, then the second call to TTYd returns native `NOT_FOUND`. Cleanup cannot obtain a reply from the first session’s lifecycle endpoint either. The original watchdog stops the VM, with outer cleanup confirmed. I’m tracing TTYd’s path after the first reply. The reason for the lost service route remains unknown; the repair and unchanged-scenario criteria stay open.
Author
Owner

TTYd 6109bec2 fixes a diagnostic gap: ordinary failures now retain their runtime phase instead of always exiting with generic code 5. Readiness and serial input have separate codes. The existing process-exit trace can use those codes once the catalog selects this revision; the underlying guest failure is still open.

TTYd [6109bec2](https://git.erikinkinen.fi/erix/ttyd/commit/6109bec2b6685397db31dca5b4dc9802b46282f3) fixes a diagnostic gap: ordinary failures now retain their runtime phase instead of always exiting with generic code 5. Readiness and serial input have separate codes. The existing process-exit trace can use those codes once the catalog selects this revision; the underlying guest failure is still open.
Author
Owner

The process-exit trace on Integration a30ea110 with TTYd 6109bec2 records TTYd’s endpoint 0xA6 exiting with status 0xA5 (call handling), before the same bootstrap NOT_FOUND and uncertain cleanup. The earlier endpoint 0xAC exit with status zero is a separate task lifetime.

This narrows the failure to call handling; the current status does not distinguish a panic from an ordinary failure. The scenario still fails, so the cause and repair remain open.

The process-exit trace on Integration `a30ea110` with TTYd `6109bec2` records TTYd’s endpoint `0xA6` exiting with status `0xA5` (call handling), before the same bootstrap `NOT_FOUND` and uncertain cleanup. The earlier endpoint `0xAC` exit with status zero is a separate task lifetime. This narrows the failure to call handling; the current status does not distinguish a panic from an ordinary failure. The scenario still fails, so the cause and repair remain open.
Author
Owner

TTYd 56f9cdf2 adds panic-location and timestamp-error diagnostics through its existing exit path. The earlier 0xA5 trace identifies call handling only; it does not distinguish panic from ordinary failure. Catalog adoption and the next guest observation remain pending.

TTYd [56f9cdf2](https://git.erikinkinen.fi/erix/ttyd/commit/56f9cdf24e3bb2ffaef6be4b997d74d043c7b43a) adds panic-location and timestamp-error diagnostics through its existing exit path. The earlier `0xA5` trace identifies call handling only; it does not distinguish panic from ordinary failure. Catalog adoption and the next guest observation remain pending.
Author
Owner

Integration 63bb744d now selects TTYd 56f9cdf2. The next diagnostic run can distinguish a panic location from ordinary call failure and retain the full timestamp error. The earlier 0xA5 trace remains inconclusive; the cause and guest repair are still open.

Integration `63bb744d` now selects TTYd `56f9cdf2`. The next diagnostic run can distinguish a panic location from ordinary call failure and retain the full timestamp error. The earlier `0xA5` trace remains inconclusive; the cause and guest repair are still open.
Author
Owner

The unchanged multi-PTY case still fails on Integration 63bb744d with TTYd 56f9cdf2. The new trace records TTYd endpoint 0xA6 exiting with raw status 5, followed by the same missing bootstrap endpoint and uncertain cleanup. This is not a panic-location word. Status 5 still has more than one reachable source, so it does not yet identify the cause. The first failed run and both packaged TTY variants are preserved; no retry was made.

The unchanged multi-PTY case still fails on Integration `63bb744d` with TTYd `56f9cdf2`. The new trace records TTYd endpoint `0xA6` exiting with raw status `5`, followed by the same missing bootstrap endpoint and uncertain cleanup. This is not a panic-location word. Status `5` still has more than one reachable source, so it does not yet identify the cause. The first failed run and both packaged TTY variants are preserved; no retry was made.
Author
Owner

The failed image contains the current TTY executable at /lib/erix/dynlink/executables/ttyd.elf. Reading that file from the retained disk image gives the packaged executable byte for byte; its instruction section also matches the current native build and differs from the previous diagnostic build.

That excludes stale TTY code as an explanation for the latest failure. Raw status 5 still does not identify the failing operation: it can come from several explicit error paths. The separate Timed reply paths also use their respective framing correctly. No cause or repair is established yet.

The failed image contains the current TTY executable at `/lib/erix/dynlink/executables/ttyd.elf`. Reading that file from the retained disk image gives the packaged executable byte for byte; its instruction section also matches the current native build and differs from the previous diagnostic build. That excludes stale TTY code as an explanation for the latest failure. Raw status `5` still does not identify the failing operation: it can come from several explicit error paths. The separate Timed reply paths also use their respective framing correctly. No cause or repair is established yet.
Author
Owner

94f14f6c adds a host regression joining Timed’s legacy clock service to LibIPC’s retained CALL and wall observer. It covers repeated requests, backpressure, malformed replies and receipt cleanup, and keeps service refusal distinct from transport error 5. The guest exit remains under investigation in #125.

[94f14f6c](https://git.erikinkinen.fi/erix/integration/commit/94f14f6cedd9427522074b8155288786b5cf21d1) adds a host regression joining Timed’s legacy clock service to LibIPC’s retained CALL and wall observer. It covers repeated requests, backpressure, malformed replies and receipt cleanup, and keeps service refusal distinct from transport error `5`. The guest exit remains under investigation in [#125](https://git.erikinkinen.fi/erix/integration/issues/125).
Author
Owner

TTYd 17477bf6 adds hosted coverage for two bootstrap sessions with separate clock observations, routes and metadata. A refused second receiver must preserve the first live session and leave the unpublished slots reusable. The guest startup failure remains open in Integration #125.

TTYd [17477bf6](https://git.erikinkinen.fi/erix/ttyd/commit/17477bf6bad97db4810fa8f48ff5cbe6f7c220da) adds hosted coverage for two bootstrap sessions with separate clock observations, routes and metadata. A refused second receiver must preserve the first live session and leave the unpublished slots reusable. The guest startup failure remains open in [Integration #125](https://git.erikinkinen.fi/erix/integration/issues/125).
Author
Owner

CI 2846 also stalls for 45 seconds in the multi-PTY scenario, at 02:14:13 UTC on October 18. Its retained serial output ends at rootd readiness.

The earlier bootstrap-call diagnosis still needs to be checked against this run’s inner evidence. The workflow is still running, and this issue remains open.

[CI 2846](https://git.erikinkinen.fi/erix/integration/actions/runs/2846/jobs/1) also stalls for 45 seconds in the multi-PTY scenario, at 02:14:13 UTC on October 18. Its retained serial output ends at rootd readiness. The earlier bootstrap-call diagnosis still needs to be checked against this run’s inner evidence. The workflow is still running, and this issue remains open.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:34:16 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/integration#125
No description provided.