[BUG] Development serial-input CI probe loses its character device after the VM catalog passes #30

New issue

Open

opened 2026-09-12 19:17:44 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-12 19:17:44 +02:00

Owner

Summary

The positive development-image serial-input probe fails because its QEMU serial character device closes before the injector observes the required command result. Integration PR run 1576 passed all 486 VM catalog scenarios, then failed this later probe. This is a retained CI failure; the reason for the transport closure is not established.

Reproduction

Observed once in Integration run 1576 at 96f1c21848d534683fc077779dd92d332e3e4779, job 19487, attempt 1. The existing workflow builds its development image and invokes:

./tests/dev-image-native-launch-vm.sh

The wrapper first validates physical-input native ls, then uses a separate development-disk copy for serial input. The latter waits for $ , injects ls;/bin/echo SERIAL_INPUT_STATUS$? followed by carriage return, and requires the SERIAL_INPUT_STATUS0 result. No fresh reproduction, CI retry or deadline change has been performed; frequency and a minimal deterministic trigger remain unknown.

Expected and actual behavior

Expected: the serial probe completes its native command and exact result checks, then stops its owned QEMU through QMP and allows the later workflow gates to run.

Actual at 2026-09-12 17:05:47 UTC:

error: serial input injector failed with status 1
error: serial input injection failed: serial character device closed

The step exits 1 and the job/run fails. This message comes from a zero-byte socket receive. It does not show whether the prompt was reached, the command was submitted, or QEMU exited because of a guest, watchdog or other condition. The wrapper checks injector failure before its QEMU timeout/stall/status branches, so absence of those later messages does not establish their underlying status.

Environment and identity

Integration: 96f1c21848d534683fc077779dd92d332e3e4779; PR event, run 1576/API 9060, job 19487 attempt 1.
Image: workflow-built development image; emitted component-resolution rows are retained in the full job log, but the image digest, final guest output and QEMU stderr bytes were not published through the artifact catalog available to this audit.
Selected wrapper defaults: x86_64 QEMU, one vCPU, serial socket input, 120-second VM limit, 45-second progress-stall limit and 10-second command limit. These are source-selected defaults; unreported effective host/firmware/tool executable bytes are not inferred.
The wrapper, runner and serial injector are byte-identical at the next Integration commit 194f01c61f61de9b9fcd41589c14d556cb6d59e3; its separate runs were still active when this failure was classified.
Earlier run 1573 passed the corresponding workflow, while its PR peer 1574 failed an earlier host test and did not establish this probe's outcome. No introducing revision is claimed.

Evidence and prior checks

The terminal run 1576 job log contains the exact real catalog summary passed=486 failed=0 total=486 at 16:58:25.8086840 UTC. Later development disk-rebuild, canonical interactive development VM and physical-input native ls checks succeeded; native ls reported 6.373670918866992 seconds. The serial-input failure follows, before later interactive/editor gates. Rust and Markdown jobs succeeded independently.

The complete failing job log is 13,136,260 bytes with SHA-256 cf5125e1d12aa4c76ba06a02361e9a66b65ef4cab9a2629551683a775c59c246. The final evidence listing names serial and QEMU-stderr files, but those contents were unavailable; the artifact catalog was empty. No cause, guest failure, host contention or threshold defect is inferred. No new warning diagnostic or real catalog timeout was observed.

Current 74f16acc4e0dbac80b437b0f13edcb9b213409d8 adds a distinct private runner log and correlated status receipt to each development physical/serial native-launch probe. It emits only allowlisted scalar failure diagnostics, preserves the actual runner exit and existing injector precedence, and stops before the second stage after first-stage failure. Missing, malformed, stale or false-success receipts cannot manufacture success. The vm_exit field describes the waited watchdog/timeout/QEMU command status, which may be 124/125 rather than QEMU's direct exit. Existing command markers and 120/45/10-second bounds are unchanged. All 19 host diagnostic tests pass, including the original 13 cases, both-stage success, missing markers, independent status failures and private evidence. The original wrapper fails the new omission control. This establishes diagnostic coverage, not a cause or correction for #30. The signed checkpoint is available in PR #12; current-head Integration CI remains pending. The matching manual is signed and published as Docs 0912e7313c0212cd3be6f06fae7d932174ce087a in Docs PR #4: all 44 documentation tests and the complete 2317-page local render pass with clean final warnings/geometry and actual visual review. Both exact-revision push CI 819 and PR CI 820 pass. Each passes 44 unit tests, 55 Markdown files and the complete 2317-page manual. All four terminal job logs are retained (744804 bytes); reference/rerun diagnostics resolve across LaTeX passes, leaving zero final warnings. CI reports a 4050835-byte PDF; its bytes were not downloaded, and no local PDF hash is attributed to CI. This issue remains open until the transport failure has its own cause and runtime evidence.

Tracking

This is a different observed gate from the release physical editor failure, catalog quota timeout, other positive filesystem timeouts, and positive host-watchdog fixture failure. Those observations do not establish a shared cause. Preserve this failure and obtain bounded runner-owned status and the selected probe's final evidence before changing runtime behavior or acceptance limits.

Implementation and regression coordination: WIP PR #12.

## Summary The positive development-image serial-input probe fails because its QEMU serial character device closes before the injector observes the required command result. Integration PR run 1576 passed all 486 VM catalog scenarios, then failed this later probe. This is a retained CI failure; the reason for the transport closure is not established. ## Reproduction Observed once in [Integration run 1576](https://git.erikinkinen.fi/erix/integration/actions/runs/1576) at `96f1c21848d534683fc077779dd92d332e3e4779`, job 19487, attempt 1. The existing workflow builds its development image and invokes: ```sh ./tests/dev-image-native-launch-vm.sh ``` The wrapper first validates physical-input native `ls`, then uses a separate development-disk copy for serial input. The latter waits for `$ `, injects `ls;/bin/echo SERIAL_INPUT_STATUS$?` followed by carriage return, and requires the `SERIAL_INPUT_STATUS0` result. No fresh reproduction, CI retry or deadline change has been performed; frequency and a minimal deterministic trigger remain unknown. ## Expected and actual behavior Expected: the serial probe completes its native command and exact result checks, then stops its owned QEMU through QMP and allows the later workflow gates to run. Actual at 2026-09-12 17:05:47 UTC: ```text error: serial input injector failed with status 1 error: serial input injection failed: serial character device closed ``` The step exits 1 and the job/run fails. This message comes from a zero-byte socket receive. It does not show whether the prompt was reached, the command was submitted, or QEMU exited because of a guest, watchdog or other condition. The wrapper checks injector failure before its QEMU timeout/stall/status branches, so absence of those later messages does not establish their underlying status. ## Environment and identity - Integration: `96f1c21848d534683fc077779dd92d332e3e4779`; PR event, run 1576/API 9060, job 19487 attempt 1. - Image: workflow-built development image; emitted component-resolution rows are retained in the full job log, but the image digest, final guest output and QEMU stderr bytes were not published through the artifact catalog available to this audit. - Selected wrapper defaults: x86_64 QEMU, one vCPU, serial socket input, 120-second VM limit, 45-second progress-stall limit and 10-second command limit. These are source-selected defaults; unreported effective host/firmware/tool executable bytes are not inferred. - The wrapper, runner and serial injector are byte-identical at the next Integration commit `194f01c61f61de9b9fcd41589c14d556cb6d59e3`; its separate runs were still active when this failure was classified. - Earlier run 1573 passed the corresponding workflow, while its PR peer 1574 failed an earlier host test and did not establish this probe's outcome. No introducing revision is claimed. ## Evidence and prior checks The terminal [run 1576](https://git.erikinkinen.fi/erix/integration/actions/runs/1576) job log contains the exact real catalog summary `passed=486 failed=0 total=486` at 16:58:25.8086840 UTC. Later development disk-rebuild, canonical interactive development VM and physical-input native `ls` checks succeeded; native `ls` reported 6.373670918866992 seconds. The serial-input failure follows, before later interactive/editor gates. Rust and Markdown jobs succeeded independently. The complete failing job log is 13,136,260 bytes with SHA-256 `cf5125e1d12aa4c76ba06a02361e9a66b65ef4cab9a2629551683a775c59c246`. The final evidence listing names serial and QEMU-stderr files, but those contents were unavailable; the artifact catalog was empty. No cause, guest failure, host contention or threshold defect is inferred. No new warning diagnostic or real catalog timeout was observed. Current `74f16acc4e0dbac80b437b0f13edcb9b213409d8` adds a distinct private runner log and correlated status receipt to each development physical/serial native-launch probe. It emits only allowlisted scalar failure diagnostics, preserves the actual runner exit and existing injector precedence, and stops before the second stage after first-stage failure. Missing, malformed, stale or false-success receipts cannot manufacture success. The `vm_exit` field describes the waited watchdog/timeout/QEMU command status, which may be 124/125 rather than QEMU's direct exit. Existing command markers and 120/45/10-second bounds are unchanged. All 19 host diagnostic tests pass, including the original 13 cases, both-stage success, missing markers, independent status failures and private evidence. The original wrapper fails the new omission control. This establishes diagnostic coverage, not a cause or correction for #30. The signed checkpoint is available in [PR #12](https://git.erikinkinen.fi/erix/integration/pulls/12); current-head Integration CI remains pending. The matching manual is signed and published as Docs `0912e7313c0212cd3be6f06fae7d932174ce087a` in [Docs PR #4](https://git.erikinkinen.fi/erix/docs/pulls/4): all 44 documentation tests and the complete 2317-page local render pass with clean final warnings/geometry and actual visual review. Both exact-revision [push CI 819](https://git.erikinkinen.fi/erix/docs/actions/runs/819) and [PR CI 820](https://git.erikinkinen.fi/erix/docs/actions/runs/820) pass. Each passes 44 unit tests, 55 Markdown files and the complete 2317-page manual. All four terminal job logs are retained (744804 bytes); reference/rerun diagnostics resolve across LaTeX passes, leaving zero final warnings. CI reports a 4050835-byte PDF; its bytes were not downloaded, and no local PDF hash is attributed to CI. This issue remains open until the transport failure has its own cause and runtime evidence. ## Tracking This is a different observed gate from [the release physical editor failure](https://git.erikinkinen.fi/erix/integration/issues/13), [catalog quota timeout](https://git.erikinkinen.fi/erix/integration/issues/18), [other positive filesystem timeouts](https://git.erikinkinen.fi/erix/integration/issues/20), and [positive host-watchdog fixture failure](https://git.erikinkinen.fi/erix/integration/issues/25). Those observations do not establish a shared cause. Preserve this failure and obtain bounded runner-owned status and the selected probe's final evidence before changing runtime behavior or acceptance limits. Implementation and regression coordination: [WIP PR #12](https://git.erikinkinen.fi/erix/integration/pulls/12).

erikinkinen added the

bug

phase-6

labels

2026-09-12 19:17:44 +02:00

~~erikinkinen referenced this issue 2026-09-12 19:20:38 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

~~erikinkinen referenced this issue from erix/docs 2026-09-12 20:00:19 +02:00~~

WIP: Document native authority and verify workspace and command evidence #4

erikinkinen referenced this issue

2026-09-12 20:17:10 +02:00

[BUG] Release repeated-interrupt CI probe fails with vm-stall receipt #31