[BUG] Release repeated-interrupt CI probe fails with vm-stall receipt #31

New issue

Open

opened 2026-09-12 20:17:10 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-12 20:17:10 +02:00

Owner

Summary

The release image's repeated foreground-interrupt probe fails after the complete VM catalog and earlier interactive probes pass. Integration reports a bounded vm-stall diagnostic and stops before the remaining post-catalog checks. The underlying guest or host cause is not established.

Reproduction

Observed once in Integration run 1577 at 194f01c61f61de9b9fcd41589c14d556cb6d59e3. The existing workflow builds the release image, then invokes:

./tests/interactive-interrupt-reuse-vm.sh out/appliance/erix-release.img

The maintained positive probe selects physical input and nine cycles. Each cycle starts /bin/cat, verifies live echo/output, sends Ctrl+C, checks status 130 and the returned prompt, then runs a following command. The wrapper requires every ordered milestone. This CI observation has not been replayed; frequency, a minimal deterministic trigger and clean-rebuild reproducibility remain unknown.

Expected and actual behavior

Expected: all nine cycles complete within the existing limits, their exact command/prompt witnesses pass, and the wrapper allows later checks to run.

Actual at 2026-09-12 18:04:40 UTC:

vm diagnostic: stage=vm-stall runner_exit=1 vm_exit=125 expected_vm_exit=0 physical_input_exit=unknown serial_input_exit=unknown physical_cleanup_requested=1

The step exits 1 and the main job fails. vm_exit is the waited watchdog/timeout/QEMU command status; it is not necessarily QEMU's direct exit. The runner classifies 125 as a stall when its stall limit is enabled. Physical input cleanup was requested, but no independently waited injector terminal result is available in the summary. Neither the last guest milestone nor the specific origin of 125 can be recovered from this public job log.

Environment and identity

Integration 194f01c61f61de9b9fcd41589c14d556cb6d59e3, push run 1577/API 9065, main job 19498 attempt 1; API terminal failure at 18:05:40 UTC.
Release image built by that workflow. The original 72-row component catalog and emitted resolution records define the selected sources; no current branch head is substituted. The image digest and inner serial/input/QEMU log contents were unavailable through the observed artifact catalog.
Source-selected x86_64 QEMU, one vCPU, physical QMP keyboard input, 120-second VM limit, 15-second serial-progress limit and 10-second command limit. Effective acceleration, firmware and compiler executable bytes are not inferred from local tools.
The selected workflow and probe helpers are retained from original Git blobs. Native-stage return values are distinct from image identity and runtime correctness.
No introducing revision or exact-source last-known-working comparison is established. The development repeated-interrupt case passed earlier in this same run.

Evidence and prior checks

The complete main log is 13,140,137 bytes, SHA-256 a96b903705b0ca51ceb1a6b1415719376d0fb61f4d6340934b16e7932444fd78. It records 486 distinct real catalog worker passes and passed=486 failed=0 total=486 at 17:53:39 UTC, including the positive quota case. The 21-case watchdog and 43-case process/source host groups passed. The separate Rust and Markdown jobs succeeded.

After the catalog, the development disk rebuild, both canonical images, both native-launch probes, four interactive-lines cases, both single-interrupt cases and development repeated-interrupt case passed. The release repeated-interrupt invocation starts at 18:04:21 UTC and reports the failure at 18:04:40 UTC. The observed artifact catalog is empty; an output listing alone does not recover the inner evidence. No warning diagnostic or real catalog timeout was found. No retry, deadline relaxation, guest-cause or contention claim follows.

At preceding signed 74f16acc4e0dbac80b437b0f13edcb9b213409d8, push CI 1579 terminates in failure on September 12 at 19:44:02 UTC. Rust and Markdown succeed, and all 486 distinct VM catalog scenarios pass. The development repeated-interrupt probe passes at 19:42:02.137 UTC; the following release nine-cycle probe starts at 19:42:02.343 UTC and fails at 19:42:55.837 UTC. Its receipt reports stage=physical-input runner_exit=1 vm_exit=0 expected_vm_exit=0 physical_input_exit=1 serial_input_exit=unknown physical_cleanup_requested=0. This is the same gate tracked in #31 with a different diagnostic shape from run 1577; it does not establish a shared cause. The complete main log is 13140239 bytes, SHA-256 2587f752ad9e7e304ea93124eef28c2aa63b3b6f9d6867bdb2c2602a9eef8bee. All three job logs are retained, without warning diagnostics. The failed command phase, inner probe evidence, image digest and raw QEMU wait remain unknown from those logs. No artifact catalog was queried for this follow-up; no absence claim or retry is made. The separate PR CI 1580 subsequently fails the development native-directory gate; see #35.

At the same preceding revision, PR CI 1580 terminates in failure at September 12, 20:17:29 UTC. All 486 distinct VM catalog scenarios, both repeated-interrupt probes and all four Powerbox prompt probes pass. The following development physical native-directory probe fails at 20:16:28.687 UTC with physical-input exit 1, composed VM-command exit 0 and no requested physical cleanup. Rust and Markdown succeed. All three complete logs are retained; the main log is 13141904 bytes, SHA-256 313b4956715cea70e9f97331677ce3e07cd8e94e6277afb6ca108e445030b4b6. This separate gate is tracked in #35. The exact command/consent/cleanup failure, inner probe artifacts, image digest and raw QEMU wait are unknown from those logs. No artifact query, retry or acceptance change was made. The current Integration 9e8 push CI 1581 and PR CI 1582 are both running.

Tracking

This is a different observed probe from the release editor failure, positive filesystem timeouts, the host watchdog fixture failure, and the development serial EOF. The development serial probe passed in run 1577; that does not close the earlier intermittent observation or establish a common cause.

Coordinate through Integration PR 12. Preserve this terminal failure and obtain the exact inner progress/status evidence before proposing a runtime or acceptance change. The separate PR run 1578 and successor workflows remain independent evidence.

## Summary The release image's repeated foreground-interrupt probe fails after the complete VM catalog and earlier interactive probes pass. Integration reports a bounded `vm-stall` diagnostic and stops before the remaining post-catalog checks. The underlying guest or host cause is not established. ## Reproduction Observed once in [Integration run 1577](https://git.erikinkinen.fi/erix/integration/actions/runs/1577) at `194f01c61f61de9b9fcd41589c14d556cb6d59e3`. The existing workflow builds the release image, then invokes: ```sh ./tests/interactive-interrupt-reuse-vm.sh out/appliance/erix-release.img ``` The maintained positive probe selects physical input and nine cycles. Each cycle starts `/bin/cat`, verifies live echo/output, sends Ctrl+C, checks status 130 and the returned prompt, then runs a following command. The wrapper requires every ordered milestone. This CI observation has not been replayed; frequency, a minimal deterministic trigger and clean-rebuild reproducibility remain unknown. ## Expected and actual behavior Expected: all nine cycles complete within the existing limits, their exact command/prompt witnesses pass, and the wrapper allows later checks to run. Actual at 2026-09-12 18:04:40 UTC: ```text vm diagnostic: stage=vm-stall runner_exit=1 vm_exit=125 expected_vm_exit=0 physical_input_exit=unknown serial_input_exit=unknown physical_cleanup_requested=1 ``` The step exits 1 and the main job fails. `vm_exit` is the waited watchdog/timeout/QEMU command status; it is not necessarily QEMU's direct exit. The runner classifies 125 as a stall when its stall limit is enabled. Physical input cleanup was requested, but no independently waited injector terminal result is available in the summary. Neither the last guest milestone nor the specific origin of 125 can be recovered from this public job log. ## Environment and identity - Integration `194f01c61f61de9b9fcd41589c14d556cb6d59e3`, push run 1577/API 9065, main job 19498 attempt 1; API terminal failure at 18:05:40 UTC. - Release image built by that workflow. The original 72-row component catalog and emitted resolution records define the selected sources; no current branch head is substituted. The image digest and inner serial/input/QEMU log contents were unavailable through the observed artifact catalog. - Source-selected x86_64 QEMU, one vCPU, physical QMP keyboard input, 120-second VM limit, 15-second serial-progress limit and 10-second command limit. Effective acceleration, firmware and compiler executable bytes are not inferred from local tools. - The selected workflow and probe helpers are retained from original Git blobs. Native-stage return values are distinct from image identity and runtime correctness. - No introducing revision or exact-source last-known-working comparison is established. The development repeated-interrupt case passed earlier in this same run. ## Evidence and prior checks The complete main log is 13,140,137 bytes, SHA-256 `a96b903705b0ca51ceb1a6b1415719376d0fb61f4d6340934b16e7932444fd78`. It records 486 distinct real catalog worker passes and `passed=486 failed=0 total=486` at 17:53:39 UTC, including the positive quota case. The 21-case watchdog and 43-case process/source host groups passed. The separate Rust and Markdown jobs succeeded. After the catalog, the development disk rebuild, both canonical images, both native-launch probes, four interactive-lines cases, both single-interrupt cases and development repeated-interrupt case passed. The release repeated-interrupt invocation starts at 18:04:21 UTC and reports the failure at 18:04:40 UTC. The observed artifact catalog is empty; an output listing alone does not recover the inner evidence. No warning diagnostic or real catalog timeout was found. No retry, deadline relaxation, guest-cause or contention claim follows. At preceding signed `74f16acc4e0dbac80b437b0f13edcb9b213409d8`, [push CI 1579](https://git.erikinkinen.fi/erix/integration/actions/runs/1579) terminates in failure on September 12 at 19:44:02 UTC. Rust and Markdown succeed, and all 486 distinct VM catalog scenarios pass. The development repeated-interrupt probe passes at 19:42:02.137 UTC; the following release nine-cycle probe starts at 19:42:02.343 UTC and fails at 19:42:55.837 UTC. Its receipt reports `stage=physical-input runner_exit=1 vm_exit=0 expected_vm_exit=0 physical_input_exit=1 serial_input_exit=unknown physical_cleanup_requested=0`. This is the same gate tracked in #31 with a different diagnostic shape from run 1577; it does not establish a shared cause. The complete main log is 13140239 bytes, SHA-256 `2587f752ad9e7e304ea93124eef28c2aa63b3b6f9d6867bdb2c2602a9eef8bee`. All three job logs are retained, without warning diagnostics. The failed command phase, inner probe evidence, image digest and raw QEMU wait remain unknown from those logs. No artifact catalog was queried for this follow-up; no absence claim or retry is made. The separate PR CI 1580 subsequently fails the development native-directory gate; see [#35](https://git.erikinkinen.fi/erix/integration/issues/35). At the same preceding revision, [PR CI 1580](https://git.erikinkinen.fi/erix/integration/actions/runs/1580) terminates in failure at September 12, 20:17:29 UTC. All 486 distinct VM catalog scenarios, both repeated-interrupt probes and all four Powerbox prompt probes pass. The following development physical native-directory probe fails at 20:16:28.687 UTC with physical-input exit 1, composed VM-command exit 0 and no requested physical cleanup. Rust and Markdown succeed. All three complete logs are retained; the main log is 13141904 bytes, SHA-256 `313b4956715cea70e9f97331677ce3e07cd8e94e6277afb6ca108e445030b4b6`. This separate gate is tracked in [#35](https://git.erikinkinen.fi/erix/integration/issues/35). The exact command/consent/cleanup failure, inner probe artifacts, image digest and raw QEMU wait are unknown from those logs. No artifact query, retry or acceptance change was made. The current Integration 9e8 push CI 1581 and PR CI 1582 are both running. ## Tracking This is a different observed probe from [the release editor failure](https://git.erikinkinen.fi/erix/integration/issues/13), [positive filesystem timeouts](https://git.erikinkinen.fi/erix/integration/issues/20), [the host watchdog fixture failure](https://git.erikinkinen.fi/erix/integration/issues/25), and [the development serial EOF](https://git.erikinkinen.fi/erix/integration/issues/30). The development serial probe passed in run 1577; that does not close the earlier intermittent observation or establish a common cause. Coordinate through [Integration PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12). Preserve this terminal failure and obtain the exact inner progress/status evidence before proposing a runtime or acceptance change. The separate PR run 1578 and successor workflows remain independent evidence.

erikinkinen added the

bug

phase-6

labels

2026-09-12 20:17:10 +02:00

~~erikinkinen referenced this issue 2026-09-12 20:20:35 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

~~erikinkinen referenced this issue from erix/docs 2026-09-12 20:52:05 +02:00~~

WIP: Document native authority and verify workspace and command evidence #4

erikinkinen referenced this issue

2026-09-12 20:55:51 +02:00

[BUG] Development physical command-line CI probe fails after the VM catalog passes #32