[BUG] Canonical development-image CI gate fails before evidence parsing #40

New issue

Open

opened 2026-09-13 03:39:33 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-13 03:39:33 +02:00

Owner

Summary

The first canonical development-image VM gate returns status 1 in Integration CI 1586, after all 486 catalog scenarios and the development disk reproducibility check pass. The canonical wrapper exits before its evidence parser and all subsequent native-input gates. The failed runner stage and underlying cause are unknown.

Reproduction

This is one retained CI observation, not a newly reproduced failure. At original Integration 2604971484facb4910d8f030818dc01f83e57816, the workflow prepares the pinned product graph and packaged development disk, then invokes:

./tests/dev-image-interactive-vm.sh

The selected workflow step is “Boot canonical interactive development image.” It delegates to canonical-image-vm.sh dev. Frequency, a smaller deterministic trigger and an independently reproduced source/image pair remain unknown. No retry or new VM execution was performed.

Expected and actual behavior

The canonical development-image runner should succeed and allow its evidence parser to validate the captured output. Later native-launch and input checks should then run.

Instead, on 2026-09-13 the wrapper announced its evidence directory at 01:16:29.7747058 UTC and reported “canonical image VM failed with status 1” at 01:16:55.0296627 UTC. It returned that recorded runner status after checking that its source image, identity sidecar and firmware inputs were unchanged. The evidence parser was not reached.

The wrapper redirects the runner's full output to a private log. That log and private guest, QMP and input artifacts were not downloaded. The complete outer CI log contains no runner diagnostic stage. Status 1 therefore identifies the waited runner command, without identifying a physical injector exit, a silence deadline, a failed guest operation or raw QEMU termination.

Environment and identity

The exact source was 2604971484facb4910d8f030818dc01f83e57816, pull-request run 1586/API9088, main job 19551, attempt 1. The canonical wrapper selects the workflow-built development image, makes a private writable disk copy, uses physical input with the canonical-shell profile, and disables serial injection. It passes explicit limits of 120 seconds hard, 15 seconds stall and 10 seconds per command, with one vCPU. No limit was changed during this investigation.

The workflow, catalogs, canonical wrappers and runner are correlated to original Git blobs. Exact image, firmware and identity-sidecar bytes were not independently inspected. A last known working source/image pair for this specific failure is not established.

Evidence and prior checks

All three complete job logs are retained, totaling 13,248,678 bytes, with no observer compiler-warning candidates. The Rust job passes 320 default and 321 all-feature unit cases; the Markdown job passes. The main job fails, and the API records terminal failure at 01:17:42 UTC. Its complete log has 13,143,997 bytes and SHA-256 5dfad622e6f94eb5ca088678fc43fec6c6a7a34d465102a973a3e793bdde62d3.

The 486 distinct worker-scoped catalog success records match original scenario blobs; their final summary is at 01:10:24.8278542 UTC. The byte-identical development disk rebuild passes at 01:16:08.3415855 UTC. These are separate successful stages and do not establish acceptance of the subsequent canonical image boot. The native-launch, canonical release-image and directory/editor/utility gates are not reached in this run.

Tracking

Keep this first canonical CI gate distinct from the later development physical native-ls failure #39, whose same-source push run passed canonical development boot before failing. The controlled local startup silence observation #33 has separate image, capture and diagnostic evidence; a shared cause is not established. Coordination remains in Integration PR #12.

Keep this issue open until source-bound diagnosis and appropriate unchanged-limit runtime validation establish a correction. Neither the generic nonzero status nor later input diagnostic improvements locate or fix this failure.

## Summary The first canonical development-image VM gate returns status 1 in [Integration CI 1586](https://git.erikinkinen.fi/erix/integration/actions/runs/1586), after all 486 catalog scenarios and the development disk reproducibility check pass. The canonical wrapper exits before its evidence parser and all subsequent native-input gates. The failed runner stage and underlying cause are unknown. ## Reproduction This is one retained CI observation, not a newly reproduced failure. At original Integration `2604971484facb4910d8f030818dc01f83e57816`, the workflow prepares the pinned product graph and packaged development disk, then invokes: ```sh ./tests/dev-image-interactive-vm.sh ``` The [selected workflow step](https://git.erikinkinen.fi/erix/integration/src/commit/2604971484facb4910d8f030818dc01f83e57816/.github/workflows/ci.yml#L648) is “Boot canonical interactive development image.” It delegates to `canonical-image-vm.sh dev`. Frequency, a smaller deterministic trigger and an independently reproduced source/image pair remain unknown. No retry or new VM execution was performed. ## Expected and actual behavior The canonical development-image runner should succeed and allow its evidence parser to validate the captured output. Later native-launch and input checks should then run. Instead, on 2026-09-13 the wrapper announced its evidence directory at 01:16:29.7747058 UTC and reported “canonical image VM failed with status 1” at 01:16:55.0296627 UTC. It returned that recorded runner status after checking that its source image, identity sidecar and firmware inputs were unchanged. The evidence parser was not reached. The wrapper redirects the runner's full output to a private log. That log and private guest, QMP and input artifacts were not downloaded. The complete outer CI log contains no runner diagnostic stage. Status 1 therefore identifies the waited runner command, without identifying a physical injector exit, a silence deadline, a failed guest operation or raw QEMU termination. ## Environment and identity The exact source was `2604971484facb4910d8f030818dc01f83e57816`, pull-request run 1586/API9088, main job 19551, attempt 1. The [canonical wrapper](https://git.erikinkinen.fi/erix/integration/src/commit/2604971484facb4910d8f030818dc01f83e57816/tests/canonical-image-vm.sh) selects the workflow-built development image, makes a private writable disk copy, uses physical input with the `canonical-shell` profile, and disables serial injection. It passes explicit limits of 120 seconds hard, 15 seconds stall and 10 seconds per command, with one vCPU. No limit was changed during this investigation. The workflow, catalogs, canonical wrappers and runner are correlated to original Git blobs. Exact image, firmware and identity-sidecar bytes were not independently inspected. A last known working source/image pair for this specific failure is not established. ## Evidence and prior checks All three complete job logs are retained, totaling 13,248,678 bytes, with no observer compiler-warning candidates. The Rust job passes 320 default and 321 all-feature unit cases; the Markdown job passes. The main job fails, and the API records terminal failure at 01:17:42 UTC. Its complete log has 13,143,997 bytes and SHA-256 `5dfad622e6f94eb5ca088678fc43fec6c6a7a34d465102a973a3e793bdde62d3`. The 486 distinct worker-scoped catalog success records match original scenario blobs; their final summary is at 01:10:24.8278542 UTC. The byte-identical development disk rebuild passes at 01:16:08.3415855 UTC. These are separate successful stages and do not establish acceptance of the subsequent canonical image boot. The native-launch, canonical release-image and directory/editor/utility gates are not reached in this run. ## Tracking Keep this first canonical CI gate distinct from [the later development physical native-ls failure #39](https://git.erikinkinen.fi/erix/integration/issues/39), whose same-source push run passed canonical development boot before failing. The [controlled local startup silence observation #33](https://git.erikinkinen.fi/erix/integration/issues/33) has separate image, capture and diagnostic evidence; a shared cause is not established. Coordination remains in [Integration PR #12](https://git.erikinkinen.fi/erix/integration/pulls/12). Keep this issue open until source-bound diagnosis and appropriate unchanged-limit runtime validation establish a correction. Neither the generic nonzero status nor later input diagnostic improvements locate or fix this failure.

erikinkinen added the

bug

phase-6

labels

2026-09-13 03:39:33 +02:00

~~erikinkinen referenced this issue 2026-09-13 03:43:49 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

~~erikinkinen referenced this issue from erix/docs 2026-09-13 04:02:00 +02:00~~

WIP: Document native authority and verify workspace and command evidence #4

erikinkinen referenced this issue

2026-09-14 07:36:09 +02:00

[BUG] Canonical release-image CI gate fails before evidence parsing #47