[BUG] Existing ext4 VM scenarios exceed their original QEMU deadlines #73

New issue

Open

opened 2026-09-22 04:08:51 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-22 04:08:51 +02:00

Owner

Summary

Existing ext4 positive VM scenarios reach their original 120-second QEMU deadline in two completed Integration CI runs. Both complete catalog runs fail, while the Rust and Markdown jobs pass. The failing scenario sets differ; this report records the observed failures without attributing them to filesystem semantics or host load.

Reproduction

The original push and pull-request CI executions at Integration fd8a5cf0dbcf9a9cd3ddb6038370295e6ec2c8fa run the maintained workflow and all 489 VM scenarios. Run 1705 reports seven failures and run 1706 reports four. These are two original executions, with no rerun, timeout increase or changed success oracle. An isolated clean-image reproduction has not yet been performed.

Expected and actual behavior

Every positive scenario must reach its existing required witnesses inside its declared limit. Run 1705 reports passed=482 failed=7 total=489; run 1706 reports passed=485 failed=4 total=489. Each listed failure has actual qemu timed out after 120s output and scenario status 1; the workflow ends with failed VM workers.

Ext4 scenario suffix	Run 1705	Run 1706
geometry-positive	timeout	passes
mmp-positive	timeout	passes
orphan-file-positive	timeout	passes
orphan-list-positive	timeout	timeout
fast-commit-htree-positive	timeout	passes
quota-positive	timeout	timeout
fast-commit-linear-positive	timeout	timeout
links-positive	passes	timeout

All suffixes belong to tests/scenarios/subsystem-e2fs-fat-ext4-<suffix>.toml. Retained timeout transcripts end their EriX markers at ERIX_KERNEL:ROOTD_SMOKE:LOGD_REPLY_WAKE_OK=0000000000000001. That last printed marker does not identify the instruction executing at the deadline or prove a Logd defect.

Environment and identity

Integration revision: fd8a5cf0dbcf9a9cd3ddb6038370295e6ec2c8fa; original source graph and artifact construction are recorded in each linked run.
Workflow: ci.yml, original push run 1705 and pull-request run 1706, with 489 maintained VM scenarios and the original 120-second QEMU ceiling.
Architecture: x86_64 with the workflow-selected QEMU, firmware and image inputs; no replacement image is used for this classification.
Input and authority: original scenario-selected boot image and test disk. The observer only reads completed CI status/logs; it grants no guest capability or additional runtime control.
The complete original six job logs are retained with verified lengths and SHA-256 digests. Individual image digests have not yet been independently recovered from CI artifacts.
Last known passing exact graph and a common root cause are not established.

Evidence and prior checks

Original push run 1705 and original pull-request run 1706 retain the observed failures. Their Rust jobs each pass 320 and 321 tests, and Markdown passes. All six complete logs have no warning candidates. Expected negative-helper error output is distinct from the final VM failures. Do not treat successful reruns as a correction or widen the original limits.

Exact original-contract comparison: ten timeout serial dumps lack their required ERIX_ROOTD:READY:VFS_IO marker; the links case also lacks ERIX_ROOTD:READY:VFS_EXT_LINKS. The geometry dump already contains its declared required substrings, but still reaches the deadline instead of the required exit. Marker presence cannot replace the exit and timeout checks. None of the eleven dumps contains its scenario's forbidden Kernel fault markers. This narrows the observed stage without proving which operation stalls or why.

Tracking

Track under Phase 6 completion and the coordinated implementation review. The separate startup timing report and FAT32 timing report remain open; shared causation has not been established. Preserve original evidence, identify the failing stage with bounded diagnostics and validate any correction against these unchanged scenario oracles and the complete maintained suite.

Later original CI comparison — 22 September 2026: original Integration 1707 and 1708, at 9c62af2d2484b1ab0cadba768adc025dfd017485, both pass all 489 VM scenarios, Rust 320/321 tests and Markdown. All six complete job logs are retained and hash-verified, totaling 26,966,330 bytes with zero warning candidates. Both runs explicitly select Kernel a9bdf6163813d378e0b4a164bceb839e24fbb6b7. These are later original runs on a different source graph, not retries of 1705/1706. Their success does not identify the cause or demonstrate a correction for the eleven previously retained ext4 timeouts; this report remains open.

## Summary Existing ext4 positive VM scenarios reach their original 120-second QEMU deadline in two completed Integration CI runs. Both complete catalog runs fail, while the Rust and Markdown jobs pass. The failing scenario sets differ; this report records the observed failures without attributing them to filesystem semantics or host load. ## Reproduction The original push and pull-request CI executions at Integration `fd8a5cf0dbcf9a9cd3ddb6038370295e6ec2c8fa` run the maintained workflow and all 489 VM scenarios. Run 1705 reports seven failures and run 1706 reports four. These are two original executions, with no rerun, timeout increase or changed success oracle. An isolated clean-image reproduction has not yet been performed. ## Expected and actual behavior Every positive scenario must reach its existing required witnesses inside its declared limit. Run 1705 reports `passed=482 failed=7 total=489`; run 1706 reports `passed=485 failed=4 total=489`. Each listed failure has actual `qemu timed out after 120s` output and scenario status 1; the workflow ends with failed VM workers. | Ext4 scenario suffix | Run 1705 | Run 1706 | | --- | --- | --- | | geometry-positive | timeout | passes | | mmp-positive | timeout | passes | | orphan-file-positive | timeout | passes | | orphan-list-positive | timeout | timeout | | fast-commit-htree-positive | timeout | passes | | quota-positive | timeout | timeout | | fast-commit-linear-positive | timeout | timeout | | links-positive | passes | timeout | All suffixes belong to `tests/scenarios/subsystem-e2fs-fat-ext4-<suffix>.toml`. Retained timeout transcripts end their EriX markers at `ERIX_KERNEL:ROOTD_SMOKE:LOGD_REPLY_WAKE_OK=0000000000000001`. That last printed marker does not identify the instruction executing at the deadline or prove a Logd defect. ## Environment and identity - Integration revision: `fd8a5cf0dbcf9a9cd3ddb6038370295e6ec2c8fa`; original source graph and artifact construction are recorded in each linked run. - Workflow: `ci.yml`, original push run 1705 and pull-request run 1706, with 489 maintained VM scenarios and the original 120-second QEMU ceiling. - Architecture: x86_64 with the workflow-selected QEMU, firmware and image inputs; no replacement image is used for this classification. - Input and authority: original scenario-selected boot image and test disk. The observer only reads completed CI status/logs; it grants no guest capability or additional runtime control. - The complete original six job logs are retained with verified lengths and SHA-256 digests. Individual image digests have not yet been independently recovered from CI artifacts. - Last known passing exact graph and a common root cause are not established. ## Evidence and prior checks [Original push run 1705](https://git.erikinkinen.fi/erix/integration/actions/runs/1705) and [original pull-request run 1706](https://git.erikinkinen.fi/erix/integration/actions/runs/1706) retain the observed failures. Their Rust jobs each pass 320 and 321 tests, and Markdown passes. All six complete logs have no warning candidates. Expected negative-helper error output is distinct from the final VM failures. Do not treat successful reruns as a correction or widen the original limits. Exact original-contract comparison: ten timeout serial dumps lack their required `ERIX_ROOTD:READY:VFS_IO` marker; the links case also lacks `ERIX_ROOTD:READY:VFS_EXT_LINKS`. The geometry dump already contains its declared required substrings, but still reaches the deadline instead of the required exit. Marker presence cannot replace the exit and timeout checks. None of the eleven dumps contains its scenario's forbidden Kernel fault markers. This narrows the observed stage without proving which operation stalls or why. ## Tracking Track under [Phase 6 completion](https://git.erikinkinen.fi/erix/integration/issues/65) and [the coordinated implementation review](https://git.erikinkinen.fi/erix/integration/pulls/12). The separate [startup timing report](https://git.erikinkinen.fi/erix/integration/issues/72) and [FAT32 timing report](https://git.erikinkinen.fi/erix/integration/issues/70) remain open; shared causation has not been established. Preserve original evidence, identify the failing stage with bounded diagnostics and validate any correction against these unchanged scenario oracles and the complete maintained suite. Later original CI comparison — 22 September 2026: original Integration [1707](https://git.erikinkinen.fi/erix/integration/actions/runs/1707) and [1708](https://git.erikinkinen.fi/erix/integration/actions/runs/1708), at `9c62af2d2484b1ab0cadba768adc025dfd017485`, both pass all 489 VM scenarios, Rust 320/321 tests and Markdown. All six complete job logs are retained and hash-verified, totaling 26,966,330 bytes with zero warning candidates. Both runs explicitly select Kernel `a9bdf6163813d378e0b4a164bceb839e24fbb6b7`. These are later original runs on a different source graph, not retries of 1705/1706. Their success does not identify the cause or demonstrate a correction for the eleven previously retained ext4 timeouts; this report remains open.

erikinkinen added the

bug

phase-6

labels

2026-09-22 04:08:51 +02:00

erikinkinen referenced this issue

2026-09-22 04:08:56 +02:00

[FEATURE] Freeze the complete source, build, ABI and self-hosting inventory #1

erikinkinen referenced this issue

2026-09-22 04:08:57 +02:00

WIP: Coordinate native authority, shell workspace and source transport #12

erikinkinen referenced this issue