[BUG] Development physical command-line CI probe fails after the VM catalog passes #32

New issue

Open

opened 2026-09-12 20:55:51 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-12 20:55:51 +02:00

Owner

Summary

The positive development-image physical command-line probe fails after all 486 VM catalog scenarios pass. Integration PR run 1578 reports physical injector exit 1 while the composed watchdog/timeout/QEMU command returns 0. The specific injector or guest failure is not established by the available log.

An additional observation at this same development physical three-line gate occurs in PR CI 1592, at 9f004461b284493b7b53d386c8727435b02abe0f. Its runner reports a VM-stall stage with unknown injector status. This differs from the original run 1578 status combination and does not establish a common cause.

Reproduction

Observed in Integration run 1578, at 194f01c61f61de9b9fcd41589c14d556cb6d59e3, main job 19501 attempt 1. After the workflow prepares its pinned component graph and development image, it invokes:

./tests/interactive-shell-lines-vm.sh physical out/appliance/erix-dev.img

The default interactive-lines profile sends three separate native command lines: ls, ls, and /bin/echo INTERACTIVESENTINEL. Each requires its own new output and subsequent prompt before the next command starts. No fresh reproduction, CI retry or deadline change was performed. Frequency and a minimal deterministic trigger remain unknown.

Expected and actual behavior

Expected: the physical injector observes all three command results and prompt returns, retires its owned QEMU through QMP, and allows the later acceptance steps to run.

Actual: the invocation starts at 18:38:44.6407968UTC on 2026-09-12 and reports at 18:39:10.1245075UTC:

vm diagnostic: stage=physical-input runner_exit=1 vm_exit=0 expected_vm_exit=0 physical_input_exit=1 serial_input_exit=unknown physical_cleanup_requested=0

The step returns 1 and the workflow fails before the wrapper's final event-list assertions. The runner observed physical injector exit 1 without requesting its cleanup. The vm_exit value is the waited composed watchdog/timeout/QEMU command status, not independently recovered QEMU exit evidence.

The injector attempts bounded QMP quit in finally after either successful commands or an earlier error. Cleanup errors can also make it fail. Therefore this status combination does not identify the failed command, prove successful guest execution, or establish a guest crash, command deadline or QMP cleanup cause.

Environment and identity

Integration: 194f01c61f61de9b9fcd41589c14d556cb6d59e3; PR event, run 1578/API9066, main job 19501 attempt 1; terminal failure at 18:39:54UTC.
Image: workflow-built development image, copied privately for the physical probe. Component-resolution rows are retained, but the failed invocation's image digest and inner raw evidence were unavailable through the terminal artifact catalog.
Source-selected defaults: x86_64 QEMU, one vCPU, physical keyboard input, 120-second VM limit, 15-second progress-stall limit and 10-second per-command limit. Unreported effective accelerator, firmware and tool executable bytes are not inferred.
The same Integration commit's push run 1577 passed this development physical three-line gate and later failed a different release repeated-interrupt gate. This does not identify an introducing commit or establish a common cause.

Evidence and prior checks

The complete terminal main log is 13,137,731 bytes, SHA-256 5f0673b2cd07b57e968d6fc076745dea37726aa159dfaa320533108258feab41. Its real worker-bound catalog contains 486 distinct scenario passes and the summary passed=486 failed=0 total=486 at 18:32:05.6912931UTC. The positive quota case passed at 18:09:32.1317537UTC.

The development disk equality check, canonical development VM, development native physical and serial probes, and canonical release VM also pass before this failure. Rust and Markdown jobs succeed independently. The 43-case immutable-source/process observer group and 21-case watchdog group pass. Earlier malformed-option and zero-hash fetch errors are intentional passing negative fixtures; no new warning diagnostic or real catalog timeout was observed.

All three terminal job logs were retained completely. The terminal Actions artifact catalog is empty, so the injector event stream, final serial bytes, QEMU stderr and image identity cannot be recovered from those logs alone. No runtime fix or acceptance relaxation is claimed.

A diagnostic checkpoint is now signed and pushed as 9e8e2917de855d6bf42285c0f0cfabb116790b26 in Integration PR 12. It preserves the first attempted host operation and command index separately from cleanup failures, and emits only correlated scalar evidence after validating the runner's actual physical-input exit. All 70 focused host cases, two shell syntax checks and 113 Markdown files pass. The original omission assertion fails as expected; an earlier fixture executable-mode setup failure is retained separately. Existing command gates and deadlines are unchanged. This supplies future evidence and does not establish a guest cause or fix; its own Integration CI was pending at publication. The matching manual is signed and pushed as Docs 31953048287aea47edb25e57c22ee8bff22d190b in Docs PR #4. All 44 documentation tests, 55 Markdown files and the complete 2317-page local render pass, with zero final warnings and all 417771 word boxes within page bounds. Both changed pages pass visual inspection. Docs CI 823/824 both pass: each has 44 tests, 55 Markdown files and a complete 2317-page manual with zero final warnings. All four job logs are retained (746634 bytes); CI PDF bytes were not downloaded. At that manual publication, Integration CI 1581/1582 remained pending.

Additional observation on 2026-09-13: PR run 1592/API 9102 at 9f004461b284493b7b53d386c8727435b02abe0f, main job 19585 attempt 1, fails the same physical out/appliance/erix-dev.img default three-line command. The main log contains 486 distinct original worker-bound scenario passes and the 486/0/486 summary at 04:22:08.0462316 UTC. Development disk byte equality, canonical development, development native physical/serial probes and canonical release all pass before the failure. Later separate-line and interrupt gates are not reached. Rust and Markdown jobs succeed independently.

At 04:30:15.6548202 UTC the wrapper emits:

vm diagnostic: stage=vm-stall runner_exit=1 vm_exit=125 expected_vm_exit=0 physical_input_exit=unknown serial_input_exit=unknown physical_cleanup_requested=1

The workflow reports terminal failure at 04:31:06 UTC. The complete main log is 13,147,054 bytes, SHA-256 eab90238d0cd5b1f328243f388f58664b4f4e383c1715f25da551a9697f878f1; all three complete job logs total 13,251,721 bytes. Source selects the existing 120-second VM, 15-second serial-progress and 10-second command limits. The composed 125 status and forced cleanup do not recover the raw QEMU wait, failed command, injector exit or cause. No failed-invocation artifact bytes or inner logs were obtained, and no replay was attempted.

The existing scalar input-phase diagnostic deliberately requires the runner's physical-input stage, a known independently waited failing input status and no forced cleanup; this vm-stall receipt does not satisfy it. The newer directory/editor progress formatter is not selected for the default three-line profile. Their absent extra output therefore does not itself establish a diagnostic regression. Preserve this observation separately from the original input 1/composed 0 result. Same-revision push 1591 was still running when observed.

Tracking

This is a different profile and gate from the release physical native-editor failure, development serial transport closure, and release repeated-interrupt stall. These separate observations do not establish separate or common causes. All current Integration bug issues were checked for an equivalent gate.

Preserve this failure and recover bounded selected-probe evidence before changing runtime behavior or assertions. Coordinate the regression through Integration PR 12.

## Summary The positive development-image physical command-line probe fails after all 486 VM catalog scenarios pass. Integration PR run 1578 reports physical injector exit 1 while the composed watchdog/timeout/QEMU command returns 0. The specific injector or guest failure is not established by the available log. An additional observation at this same development physical three-line gate occurs in [PR CI 1592](https://git.erikinkinen.fi/erix/integration/actions/runs/1592), at `9f004461b284493b7b53d386c8727435b02abe0f`. Its runner reports a VM-stall stage with unknown injector status. This differs from the original run 1578 status combination and does not establish a common cause. ## Reproduction Observed in [Integration run 1578](https://git.erikinkinen.fi/erix/integration/actions/runs/1578), at `194f01c61f61de9b9fcd41589c14d556cb6d59e3`, main job 19501 attempt 1. After the workflow prepares its pinned component graph and development image, it invokes: ```sh ./tests/interactive-shell-lines-vm.sh physical out/appliance/erix-dev.img ``` The default `interactive-lines` profile sends three separate native command lines: `ls`, `ls`, and `/bin/echo INTERACTIVESENTINEL`. Each requires its own new output and subsequent prompt before the next command starts. No fresh reproduction, CI retry or deadline change was performed. Frequency and a minimal deterministic trigger remain unknown. ## Expected and actual behavior Expected: the physical injector observes all three command results and prompt returns, retires its owned QEMU through QMP, and allows the later acceptance steps to run. Actual: the invocation starts at 18:38:44.6407968UTC on 2026-09-12 and reports at 18:39:10.1245075UTC: ```text vm diagnostic: stage=physical-input runner_exit=1 vm_exit=0 expected_vm_exit=0 physical_input_exit=1 serial_input_exit=unknown physical_cleanup_requested=0 ``` The step returns 1 and the workflow fails before the wrapper's final event-list assertions. The runner observed physical injector exit 1 without requesting its cleanup. The `vm_exit` value is the waited composed watchdog/timeout/QEMU command status, not independently recovered QEMU exit evidence. The injector attempts bounded QMP quit in `finally` after either successful commands or an earlier error. Cleanup errors can also make it fail. Therefore this status combination does not identify the failed command, prove successful guest execution, or establish a guest crash, command deadline or QMP cleanup cause. ## Environment and identity - Integration: `194f01c61f61de9b9fcd41589c14d556cb6d59e3`; PR event, run 1578/API9066, main job 19501 attempt 1; terminal failure at 18:39:54UTC. - Image: workflow-built development image, copied privately for the physical probe. Component-resolution rows are retained, but the failed invocation's image digest and inner raw evidence were unavailable through the terminal artifact catalog. - Source-selected defaults: x86_64 QEMU, one vCPU, physical keyboard input, 120-second VM limit, 15-second progress-stall limit and 10-second per-command limit. Unreported effective accelerator, firmware and tool executable bytes are not inferred. - The same Integration commit's push run 1577 passed this development physical three-line gate and later failed a different release repeated-interrupt gate. This does not identify an introducing commit or establish a common cause. ## Evidence and prior checks The complete terminal main log is 13,137,731 bytes, SHA-256 `5f0673b2cd07b57e968d6fc076745dea37726aa159dfaa320533108258feab41`. Its real worker-bound catalog contains 486 distinct scenario passes and the summary `passed=486 failed=0 total=486` at 18:32:05.6912931UTC. The positive quota case passed at 18:09:32.1317537UTC. The development disk equality check, canonical development VM, development native physical and serial probes, and canonical release VM also pass before this failure. Rust and Markdown jobs succeed independently. The 43-case immutable-source/process observer group and 21-case watchdog group pass. Earlier malformed-option and zero-hash fetch errors are intentional passing negative fixtures; no new warning diagnostic or real catalog timeout was observed. All three terminal job logs were retained completely. The terminal Actions artifact catalog is empty, so the injector event stream, final serial bytes, QEMU stderr and image identity cannot be recovered from those logs alone. No runtime fix or acceptance relaxation is claimed. A diagnostic checkpoint is now signed and pushed as `9e8e2917de855d6bf42285c0f0cfabb116790b26` in [Integration PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12). It preserves the first attempted host operation and command index separately from cleanup failures, and emits only correlated scalar evidence after validating the runner's actual physical-input exit. All 70 focused host cases, two shell syntax checks and 113 Markdown files pass. The original omission assertion fails as expected; an earlier fixture executable-mode setup failure is retained separately. Existing command gates and deadlines are unchanged. This supplies future evidence and does not establish a guest cause or fix; its own Integration CI was pending at publication. The matching manual is signed and pushed as Docs `31953048287aea47edb25e57c22ee8bff22d190b` in [Docs PR #4](https://git.erikinkinen.fi/erix/docs/pulls/4). All 44 documentation tests, 55 Markdown files and the complete 2317-page local render pass, with zero final warnings and all 417771 word boxes within page bounds. Both changed pages pass visual inspection. Docs CI 823/824 both pass: each has 44 tests, 55 Markdown files and a complete 2317-page manual with zero final warnings. All four job logs are retained (746634 bytes); CI PDF bytes were not downloaded. At that manual publication, Integration CI 1581/1582 remained pending. Additional observation on 2026-09-13: PR run 1592/API 9102 at `9f004461b284493b7b53d386c8727435b02abe0f`, main job 19585 attempt 1, fails the same `physical out/appliance/erix-dev.img` default three-line command. The main log contains 486 distinct original worker-bound scenario passes and the 486/0/486 summary at 04:22:08.0462316 UTC. Development disk byte equality, canonical development, development native physical/serial probes and canonical release all pass before the failure. Later separate-line and interrupt gates are not reached. Rust and Markdown jobs succeed independently. At 04:30:15.6548202 UTC the wrapper emits: ```text vm diagnostic: stage=vm-stall runner_exit=1 vm_exit=125 expected_vm_exit=0 physical_input_exit=unknown serial_input_exit=unknown physical_cleanup_requested=1 ``` The workflow reports terminal failure at 04:31:06 UTC. The complete main log is 13,147,054 bytes, SHA-256 `eab90238d0cd5b1f328243f388f58664b4f4e383c1715f25da551a9697f878f1`; all three complete job logs total 13,251,721 bytes. Source selects the existing 120-second VM, 15-second serial-progress and 10-second command limits. The composed 125 status and forced cleanup do not recover the raw QEMU wait, failed command, injector exit or cause. No failed-invocation artifact bytes or inner logs were obtained, and no replay was attempted. The existing scalar input-phase diagnostic deliberately requires the runner's `physical-input` stage, a known independently waited failing input status and no forced cleanup; this `vm-stall` receipt does not satisfy it. The newer directory/editor progress formatter is not selected for the default three-line profile. Their absent extra output therefore does not itself establish a diagnostic regression. Preserve this observation separately from the original input 1/composed 0 result. Same-revision push 1591 was still running when observed. ## Tracking This is a different profile and gate from [the release physical native-editor failure](https://git.erikinkinen.fi/erix/integration/issues/13), [development serial transport closure](https://git.erikinkinen.fi/erix/integration/issues/30), and [release repeated-interrupt stall](https://git.erikinkinen.fi/erix/integration/issues/31). These separate observations do not establish separate or common causes. All current Integration bug issues were checked for an equivalent gate. Preserve this failure and recover bounded selected-probe evidence before changing runtime behavior or assertions. Coordinate the regression through [Integration PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12).

erikinkinen added the

bug

phase-6

labels

2026-09-12 20:55:51 +02:00

~~erikinkinen referenced this issue 2026-09-12 20:57:01 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

~~erikinkinen referenced this issue from erix/docs 2026-09-12 22:17:55 +02:00~~

WIP: Document native authority and verify workspace and command evidence #4

erikinkinen referenced this issue

2026-09-12 22:26:22 +02:00

[BUG] Development physical directory CI probe fails after catalog and consent probes pass #35