[BUG] Positive ext4 truncate VM scenario reaches serial-progress deadline #44

New issue

Open

opened 2026-09-13 05:30:53 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-13 05:30:53 +02:00

Owner

Summary

The positive ext4 truncate VM scenario fails at the existing 45-second serial-progress deadline in Integration PR CI 1590. The complete catalog finishes with 485 passed and one failed out of 486 scenarios. Later image and native-input gates are not established by this run.

Reproduction

Observed in the maintained complete-catalog workflow step:

./scripts/run-ci-vm-scenarios.sh out/ci-scenarios.txt

This assumes the workflow's preceding source, build, media and catalog preparation. Worker 9 fails tests/scenarios/subsystem-ext4-truncate-positive.toml and continues its remaining cases. No isolated reproduction or failure frequency is established, and no guest replay or deadline change was attempted for this report.

Expected and actual behavior

The scenario is subsystem-positive and requires successful ext4 shrink/grow behavior, completion markers and subsequent host filesystem verification. It requests debug exit 0x10 within the configured 120-second overall QEMU allowance and 45-second serial-progress allowance.

At 02:41:33 UTC on 2026-09-13 the runner reports error: qemu stalled with no serial progress for 45s. Its retained watchdog stderr independently reports no progress for 45 seconds; QEMU reports termination by signal 15 from the Python watchdog. The runner interprets the composed command status 125 as a stall and returns 1. This does not independently recover QEMU's raw waited exit status.

The failure dump reaches ERIX_ROOTD:READY:VFS_MOUNT. It does not contain the required ERIX_ROOTD:READY:VFS_IO, ERIX_ROOTD:TRACE:VFS:EXT_TRUNCATE_OK or standalone ERIX_ROOTD:READY completion lines. The declared forbidden panic/fatal/service-failure markers are absent from this dump. The dump includes further diagnostic state; it does not establish which guest operation failed or why progress stopped.

Worker 9 reports 39 passed and one failed; the complete catalog reports 485/1 at 03:02:28 UTC. Because the VM command failed, the scenario's subsequent marker acceptance and host filesystem verification do not establish success.

Environment and identity

Integration run 1590, PR event, API run 9098, Integration revision 07c41b6b082ceb4e5a7fb3af0f4b3a5148b37905.
Integration job 19575, attempt 1, terminal failure; Rust job 19574 and Markdown job 19576 both succeed. The API records workflow completion at 03:03:06 UTC on 2026-09-13.
Scenario configuration: release, full-system x86-64 VM, one vCPU, Rootd e2fs-fat-ext-truncate, writable fs-ext4-extent-tree media and runtime.dev.storage-writable-fat.toml.
Catalog configuration: 12 workers, one Cargo job per worker,120-second QEMU ceiling and 45-second serial-progress deadline.
Selected source catalog includes kernel 4330175e27718490f06b93098e95e3c1553cb765, Rootd ac3c1847af9915164d4a819f5f15e1827884e8b1, E2fsd 63ff1883732069aa10a7333b6bbf1d53557b2101, lib-ipc b452a46803982c57e474adcc318ce40bd281d621 and Bootloader 8cb4383699c47812f64f086f54da6b2a7b9e9133.
Per-worker image digests and complete compiler, firmware and QEMU executable identities were not independently established. Source selection does not prove complete execution-environment equality or identify a causal change.

Evidence and prior checks

All three complete terminal job logs were retained, totaling 13,752,095 bytes, with no diagnostic warning candidates. The main log SHA-256 is 13e5d82d60dcb4e8724085f7bb5d4265aed418088522c36238a49b33f4a95b32. Fourteen original Integration source blobs bind the workflow, catalog/VM runners, unchanged watchdog, scenario and configuration. The 485 distinct worker pass witnesses and failed scenario are bound to their original scenario blobs; repeated failure-tail output and earlier synthetic fixture summaries are excluded.

Host timestamps on the dumped guest text are replay timestamps, not original guest event timing. The catalog result and observed stall do not establish a filesystem, scheduling, toolchain or contention cause. An unchanged retry is not a correction.

Tracking

Coordinate through Integration PR 12. Issue 18 covers the quota scenario, and issue 20 covers nine other ext3/ext4 scenarios that reached the overall 120-second deadline. This observation concerns a different scenario, the 45-second serial-progress deadline and different last readiness evidence; no shared cause is claimed.

A correction requires source-grounded causal evidence and acceptance of the unchanged truncate assertions and deadlines, followed by the complete current catalog and later workflow gates. This report preserves the terminal failure without claiming a fix or runtime recovery.

## Summary The positive ext4 truncate VM scenario fails at the existing 45-second serial-progress deadline in Integration PR CI 1590. The complete catalog finishes with 485 passed and one failed out of 486 scenarios. Later image and native-input gates are not established by this run. ## Reproduction Observed in the maintained complete-catalog workflow step: ```sh ./scripts/run-ci-vm-scenarios.sh out/ci-scenarios.txt ``` This assumes the workflow's preceding source, build, media and catalog preparation. Worker 9 fails `tests/scenarios/subsystem-ext4-truncate-positive.toml` and continues its remaining cases. No isolated reproduction or failure frequency is established, and no guest replay or deadline change was attempted for this report. ## Expected and actual behavior The scenario is `subsystem-positive` and requires successful ext4 shrink/grow behavior, completion markers and subsequent host filesystem verification. It requests debug exit `0x10` within the configured 120-second overall QEMU allowance and 45-second serial-progress allowance. At 02:41:33 UTC on 2026-09-13 the runner reports `error: qemu stalled with no serial progress for 45s`. Its retained watchdog stderr independently reports no progress for 45 seconds; QEMU reports termination by signal 15 from the Python watchdog. The runner interprets the composed command status 125 as a stall and returns 1. This does not independently recover QEMU's raw waited exit status. The failure dump reaches `ERIX_ROOTD:READY:VFS_MOUNT`. It does not contain the required `ERIX_ROOTD:READY:VFS_IO`, `ERIX_ROOTD:TRACE:VFS:EXT_TRUNCATE_OK` or standalone `ERIX_ROOTD:READY` completion lines. The declared forbidden panic/fatal/service-failure markers are absent from this dump. The dump includes further diagnostic state; it does not establish which guest operation failed or why progress stopped. Worker 9 reports 39 passed and one failed; the complete catalog reports 485/1 at 03:02:28 UTC. Because the VM command failed, the scenario's subsequent marker acceptance and host filesystem verification do not establish success. ## Environment and identity - [Integration run 1590](https://git.erikinkinen.fi/erix/integration/actions/runs/1590), PR event, API run 9098, Integration revision `07c41b6b082ceb4e5a7fb3af0f4b3a5148b37905`. - Integration job 19575, attempt 1, terminal failure; Rust job 19574 and Markdown job 19576 both succeed. The API records workflow completion at 03:03:06 UTC on 2026-09-13. - Scenario configuration: release, full-system x86-64 VM, one vCPU, Rootd `e2fs-fat-ext-truncate`, writable `fs-ext4-extent-tree` media and `runtime.dev.storage-writable-fat.toml`. - Catalog configuration: 12 workers, one Cargo job per worker,120-second QEMU ceiling and 45-second serial-progress deadline. - Selected source catalog includes kernel `4330175e27718490f06b93098e95e3c1553cb765`, Rootd `ac3c1847af9915164d4a819f5f15e1827884e8b1`, E2fsd `63ff1883732069aa10a7333b6bbf1d53557b2101`, lib-ipc `b452a46803982c57e474adcc318ce40bd281d621` and Bootloader `8cb4383699c47812f64f086f54da6b2a7b9e9133`. - Per-worker image digests and complete compiler, firmware and QEMU executable identities were not independently established. Source selection does not prove complete execution-environment equality or identify a causal change. ## Evidence and prior checks All three complete terminal job logs were retained, totaling 13,752,095 bytes, with no diagnostic warning candidates. The main log SHA-256 is `13e5d82d60dcb4e8724085f7bb5d4265aed418088522c36238a49b33f4a95b32`. Fourteen original Integration source blobs bind the workflow, catalog/VM runners, unchanged watchdog, scenario and configuration. The 485 distinct worker pass witnesses and failed scenario are bound to their original scenario blobs; repeated failure-tail output and earlier synthetic fixture summaries are excluded. Host timestamps on the dumped guest text are replay timestamps, not original guest event timing. The catalog result and observed stall do not establish a filesystem, scheduling, toolchain or contention cause. An unchanged retry is not a correction. ## Tracking Coordinate through [Integration PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12). [Issue 18](https://git.erikinkinen.fi/erix/integration/issues/18) covers the quota scenario, and [issue 20](https://git.erikinkinen.fi/erix/integration/issues/20) covers nine other ext3/ext4 scenarios that reached the overall 120-second deadline. This observation concerns a different scenario, the 45-second serial-progress deadline and different last readiness evidence; no shared cause is claimed. A correction requires source-grounded causal evidence and acceptance of the unchanged truncate assertions and deadlines, followed by the complete current catalog and later workflow gates. This report preserves the terminal failure without claiming a fix or runtime recovery.

erikinkinen added the

bug

phase-6

labels

2026-09-13 05:30:53 +02:00

~~erikinkinen referenced this issue 2026-09-13 05:32:02 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

erikinkinen referenced this issue

2026-09-13 06:06:09 +02:00

[BUG] Release single foreground-interrupt CI probe fails with vm-stall receipt #45

~~erikinkinen referenced this issue from erix/docs 2026-09-13 06:48:58 +02:00~~

WIP: Document native authority and verify workspace and command evidence #4