[BUG] Development physical native-ls CI probe fails before its serial-input check #39

New issue

Open

opened 2026-09-13 03:00:10 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-13 03:00:10 +02:00

Owner

Summary

The development image's physical native-ls probe fails in Integration CI 1585 after all 486 catalog scenarios and the canonical development-image boot pass. The runner observes physical injector exit 1 with composed VM-command exit 0. The following serial-input half and later input probes are not reached. The failed input operation and underlying cause are unknown.

Reproduction

This is one retained CI observation, not a newly reproduced failure. At original Integration 2604971484facb4910d8f030818dc01f83e57816, the workflow prepares its pinned graph and development image, then invokes:

./tests/dev-image-native-launch-vm.sh

The selected workflow step is “Validate native launch latency and serial input.” Its first half uses emulated physical input with the native-ls profile; the serial half follows only after that succeeds. Frequency, a smaller deterministic trigger and independent clean-image reproducibility remain unknown. No retry or new VM execution was performed.

Expected and actual behavior

The physical probe should enter its native echo/status/listing sequence, observe LSSTATUS0 within its command deadline, complete bounded QMP shutdown and pass the wrapper's literal output/status checks. The serial half and later CI checks should then run.

Instead, the canonical development-image boot passed at 00:35:35.3954319 UTC on 2026-09-13. At 00:36:08.1627967 UTC the next step reported:

stage=physical-input runner_exit=1 vm_exit=0 expected_vm_exit=0 physical_input_exit=1 serial_input_exit=unknown physical_cleanup_requested=0

The step returned 1 before its physical-probe success line and output assertions. The runner did not request physical cleanup. This does not rule out injector-owned QMP shutdown after either a command error or success. vm_exit=0 is the waited composed watchdog/timeout/QEMU command status, not independently recovered raw QEMU evidence or proof of successful guest commands.

Environment and identity

The original source was 2604971484facb4910d8f030818dc01f83e57816, push run 1585/API9087, main job 19548, attempt 1. The selected source disk is the workflow-built development image; the wrapper gives the physical probe a private disk copy and native-ls profile. The command limit is explicitly 10 seconds. The wrapper defaults to a 120-second hard limit, 45-second stall limit and one vCPU unless its documented environment overrides them. No threshold was changed during this investigation.

The original component catalogs, native-launch wrapper, injector, runner and diagnostic decoder are correlated to original Git blobs. Exact image bytes and private guest, QMP and input artifacts were not downloaded. A last known working source/image pair for this specific failure is not established.

Evidence and prior checks

Additional retained observations: CI 1596 at 37299e719d7662698593c6d3be24752309bed7ec, CI 1598 at 84e876b0a025159e77e8a62f44947209f098ca3d, and CI 1599 at 5a593a5b308aa781bc796cd145ba7a49a4a559d9 fail at the same first physical native-ls gate after 486 catalog scenarios and canonical development boot pass. Each receipt reports physical-input, runner 1, composed VM-command 0, physical input 1, unknown serial input and no requested physical cleanup. Later serial native-ls is not reached. These are complete original CI logs, not newly reproduced failures; no common cause is established.

All three complete logs are retained, totaling 13,248,721 bytes, with no compiler-warning candidates. Rust and Markdown jobs pass; the main job fails and the API records terminal failure at 00:36:59 UTC. Its complete main log has 13,144,056 bytes and SHA-256 f96452657feda61e0437fe9f130e149b41c07bf4ad33d587391af169389ddf47.

The 486 explicit worker-scoped catalog success records match original scenario blobs. The development disk's reproducible rebuild and canonical boot pass before this failure. Source ordering and the physical-input receipt identify the first native-ls half, without identifying a failed key, output marker, deadline, QMP operation or guest/host cause. The later serial half, canonical release boot and separate directory/editor/utility probes are not reached in this run.

Tracking

Track this route separately from development serial-input loss #30, development three-line physical input #32, physical directory input #35 and release physical utility input #38. Similar scalar outcomes do not establish a common cause. Coordination remains in Integration PR #12.

Keep this issue open until source-bound diagnosis and appropriate unchanged-limit runtime validation establish a correction. The directory/editor progress formatter does not cover or fix this separate probe.

## Summary The development image's physical `native-ls` probe fails in [Integration CI 1585](https://git.erikinkinen.fi/erix/integration/actions/runs/1585) after all 486 catalog scenarios and the canonical development-image boot pass. The runner observes physical injector exit 1 with composed VM-command exit 0. The following serial-input half and later input probes are not reached. The failed input operation and underlying cause are unknown. ## Reproduction This is one retained CI observation, not a newly reproduced failure. At original Integration `2604971484facb4910d8f030818dc01f83e57816`, the workflow prepares its pinned graph and development image, then invokes: ```sh ./tests/dev-image-native-launch-vm.sh ``` The [selected workflow step](https://git.erikinkinen.fi/erix/integration/src/commit/2604971484facb4910d8f030818dc01f83e57816/.github/workflows/ci.yml#L651) is “Validate native launch latency and serial input.” Its first half uses emulated physical input with the `native-ls` profile; the serial half follows only after that succeeds. Frequency, a smaller deterministic trigger and independent clean-image reproducibility remain unknown. No retry or new VM execution was performed. ## Expected and actual behavior The physical probe should enter its native echo/status/listing sequence, observe `LSSTATUS0` within its command deadline, complete bounded QMP shutdown and pass the wrapper's literal output/status checks. The serial half and later CI checks should then run. Instead, the canonical development-image boot passed at 00:35:35.3954319 UTC on 2026-09-13. At 00:36:08.1627967 UTC the next step reported: ```text stage=physical-input runner_exit=1 vm_exit=0 expected_vm_exit=0 physical_input_exit=1 serial_input_exit=unknown physical_cleanup_requested=0 ``` The step returned 1 before its physical-probe success line and output assertions. The runner did not request physical cleanup. This does not rule out injector-owned QMP shutdown after either a command error or success. `vm_exit=0` is the waited composed watchdog/timeout/QEMU command status, not independently recovered raw QEMU evidence or proof of successful guest commands. ## Environment and identity The original source was `2604971484facb4910d8f030818dc01f83e57816`, push run 1585/API9087, main job 19548, attempt 1. The selected source disk is the workflow-built development image; the wrapper gives the physical probe a private disk copy and `native-ls` profile. The command limit is explicitly 10 seconds. The wrapper defaults to a 120-second hard limit, 45-second stall limit and one vCPU unless its documented environment overrides them. No threshold was changed during this investigation. The original component catalogs, [native-launch wrapper](https://git.erikinkinen.fi/erix/integration/src/commit/2604971484facb4910d8f030818dc01f83e57816/tests/dev-image-native-launch-vm.sh), [injector](https://git.erikinkinen.fi/erix/integration/src/commit/2604971484facb4910d8f030818dc01f83e57816/scripts/qmp-physical-input.py), runner and diagnostic decoder are correlated to original Git blobs. Exact image bytes and private guest, QMP and input artifacts were not downloaded. A last known working source/image pair for this specific failure is not established. ## Evidence and prior checks Additional retained observations: [CI 1596](https://git.erikinkinen.fi/erix/integration/actions/runs/1596) at `37299e719d7662698593c6d3be24752309bed7ec`, [CI 1598](https://git.erikinkinen.fi/erix/integration/actions/runs/1598) at `84e876b0a025159e77e8a62f44947209f098ca3d`, and [CI 1599](https://git.erikinkinen.fi/erix/integration/actions/runs/1599) at `5a593a5b308aa781bc796cd145ba7a49a4a559d9` fail at the same first physical native-ls gate after 486 catalog scenarios and canonical development boot pass. Each receipt reports physical-input, runner 1, composed VM-command 0, physical input 1, unknown serial input and no requested physical cleanup. Later serial native-ls is not reached. These are complete original CI logs, not newly reproduced failures; no common cause is established. All three complete logs are retained, totaling 13,248,721 bytes, with no compiler-warning candidates. Rust and Markdown jobs pass; the main job fails and the API records terminal failure at 00:36:59 UTC. Its complete main log has 13,144,056 bytes and SHA-256 `f96452657feda61e0437fe9f130e149b41c07bf4ad33d587391af169389ddf47`. The 486 explicit worker-scoped catalog success records match original scenario blobs. The development disk's reproducible rebuild and canonical boot pass before this failure. Source ordering and the physical-input receipt identify the first `native-ls` half, without identifying a failed key, output marker, deadline, QMP operation or guest/host cause. The later serial half, canonical release boot and separate directory/editor/utility probes are not reached in this run. ## Tracking Track this route separately from [development serial-input loss #30](https://git.erikinkinen.fi/erix/integration/issues/30), [development three-line physical input #32](https://git.erikinkinen.fi/erix/integration/issues/32), [physical directory input #35](https://git.erikinkinen.fi/erix/integration/issues/35) and [release physical utility input #38](https://git.erikinkinen.fi/erix/integration/issues/38). Similar scalar outcomes do not establish a common cause. Coordination remains in [Integration PR #12](https://git.erikinkinen.fi/erix/integration/pulls/12). Keep this issue open until source-bound diagnosis and appropriate unchanged-limit runtime validation establish a correction. The directory/editor progress formatter does not cover or fix this separate probe.

erikinkinen added the

bug

phase-6

labels

2026-09-13 03:00:10 +02:00

~~erikinkinen referenced this issue 2026-09-13 03:01:44 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

erikinkinen referenced this issue

2026-09-13 03:39:33 +02:00

[BUG] Canonical development-image CI gate fails before evidence parsing #40

~~erikinkinen referenced this issue from erix/docs 2026-09-13 04:02:00 +02:00~~

WIP: Document native authority and verify workspace and command evidence #4