[BUG] FAT32 mirrored and Unicode VM scenarios exceed their QEMU deadlines #58

New issue

Open

opened 2026-09-17 09:01:21 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-17 09:01:21 +02:00

Owner

Summary

Two existing FAT32 storage scenarios exceed their 120-second QEMU deadlines in the full Integration catalog. These failures prevent catalog acceptance. The paired run at the same Integration revision passes both scenarios, so the underlying cause and a deterministic isolated reproduction remain unestablished.

Reproduction

Observed in Integration CI 1645, exact revision df8147dfa437194cd091747fd7147941bd2793c4, in the ordinary full catalog workflow:

Worker 8: tests/scenarios/subsystem-e2fs-fat-fat32-mirrored-positive.toml.
Worker 11: tests/scenarios/subsystem-e2fs-fat-fat32-vfat-unicode-positive.toml.

Both report qemu timed out after 120s on 16 September 2026. The original run was not restarted. Preserve the source selection, scenario inputs, build profile, original limits and failing execution artifacts when constructing the next bounded reproduction. Do not retry unchanged inputs until green or increase the deadline to claim a correction.

Expected and actual behavior

Expected: each scenario reaches its declared storage/VFS readiness markers and expected guest exit 0x10, with its post-VM checks passing, within the configured 120 seconds.

Actual: the QEMU overall deadline expires and both scenarios return failure. CI 1645 finishes with 405 passing and 81 failing scenarios out of 486: the 78 known layout refusals in #57, the previously tracked quota timeout in #18, and these two FAT32 timeouts. The full workflow fails. Paired CI 1646, at the identical Integration revision, records 408 passes and only the 78 layout refusals; both FAT32 scenarios and quota pass there. This passing observation does not resolve the retained failures.

Environment and identity

Integration source: df8147dfa437194cd091747fd7147941bd2793c4; push run 1645 and paired review run 1646.
Full-system x86-64 VM scenarios, release profile, one guest CPU, writable block media, original QEMU timeout of 120 seconds.
Runtime configuration: tests/runtime-configs/runtime.dev.storage-writable-fat.toml; root storage test mode e2fs-fat.
Selected media: fs-fat32-mirrored and fs-fat32-vfat-unicode respectively.
CI package log: QEMU 1:10.0.13+ds-0+deb13u1, OVMF 2025.02-8+deb13u1. Package versions do not prove identical emulator/firmware bytes or host scheduling.
Exact original image, firmware and complete toolchain digests are not available from the artifact API; execution-environment equivalence is not established. No broader last-known-working revision is asserted.
No new input route or capability grant is introduced by this report. A guest authority failure has not been demonstrated.

Evidence and prior checks

All six terminal logs for runs 1645/1646 are complete and hash-bound in retained validation evidence, totaling 27,222,183 bytes with zero warning candidates. Rust, Markdown and preceding host controls pass in both runs. The complete scenario summaries independently identify each failure; each timeout belongs to its named worker. Host timestamps around replayed guest dumps are not guest event timing.

The Forgejo artifact endpoints for these original runs return no artifacts. An independent cause, exact image equivalence and deterministic minimal reproduction have not been established. The shared timing symptom does not establish that these failures have the same cause as #18. No source change, timeout workaround or unchanged rerun is claimed as a fix.

Original transcript analysis, 17 September 2026: the mirrored FAT32 guest reached ERIX_ROOTD:PROGRESS:VFS:ESP_IO_DONE; the Unicode guest additionally reached DATA_IO_DONE. Both transcripts show earlier FAT creation, truncate, short-name and rename completion. Neither records final VFS I/O readiness or final root readiness. These observations narrow the uncompleted portion of the storage pipeline but do not identify a stalled instruction or an underlying defect.

The paired passing run's host runner intervals are approximately 41.26 seconds (mirrored) and 43.45 seconds (Unicode); the failing intervals are approximately 120.45 and 120.96 seconds. These are QEMU-launch-to-runner-result intervals, not guest operation durations. The retained source shows serial dumps are printed only after QEMU terminates, so timestamps on those dumped markers cannot be used as arrival times.

The next diagnostic is designed but has not been executed: bind fresh source, scenario/configuration, image, emulator and firmware identities, then attach the existing bounded serial observer before the VM creates its fresh log. Retain raw per-marker arrival windows, including ESP, DATA and HTree completion, under the existing observer and VM limits. Its shell-oriented success summary does not apply to a storage scenario and must remain incomplete; do not alter the guest to manufacture a prompt. An isolated fresh observation cannot erase the original catalog failures. Original artifact absence still prevents an exact image replay.

Original Integration catalog completion — 18 September 2026: At exact revision 0db8a679f5d1acb50d9ff92e003cec91f145dfc8, CI 1659 finishes with 396 passes and 90 failures out of 486; CI 1660 finishes with 393 passes and 93 failures. Each retains the 78 known Exsh layout refusals. The remaining 12 and 15 failures are original 120-second QEMU storage-scenario timeouts. All six terminal logs are complete (36,570,659 bytes), with zero warning candidates. Rust/unit and Markdown jobs pass. No workflow was restarted, no deadline was enlarged and no failed scenario is accepted. A shared guest or host cause is unestablished.

Original positive case	CI 1659	CI 1660
`fat32-directory-metadata`	timeout	timeout
`fat32-primary-only`	pass	timeout

A paired pass does not close a retained failure. These are observed complete-catalog results; isolated reproduction and failure frequency remain unestablished. Preserve the original source graph and limits for diagnosis.

Tracking

Related retained failures: quota #18 and coherent Exsh layout adoption #57. Coordinated validation: #1 and #12. Keep this issue open until the original scenario identities and authority/cleanup behavior are preserved in a bounded reproduction, a cause is identified, and the correction passes both focused checks and the complete regression catalog under the unchanged deadlines.

Original regression follow-up — 20 September 2026:

Original Integration 1681/1682 at a62d1381f56a01afc692112d9b427205eaeb6a2e finished with 479/489 and 487/489 VM scenario passes. Six complete logs total 31,617,835 bytes with zero warning candidates. The following failures reach the unchanged 120-second QEMU deadline; none was retried or cancelled.

Run 1681: subsystem-e2fs-fat-fat32-active-fat-positive, subsystem-e2fs-fat-fat32-directory-metadata-positive, subsystem-e2fs-fat-fat32-stress-positive, subsystem-e2fs-fat-fat32-primary-only-positive, subsystem-e2fs-fat-fat32-metadata-clean-positive.

Run 1682: subsystem-e2fs-fat-fat32-vfat-unicode-positive.

Each timeout is correlated with its own worker and serial dump, stopping before that worker starts another scenario. The retained dumps show bootstrap/VFS progress and native logging activity; these observations do not identify a common cause or prove a filesystem implementation fault. Full service, filesystem and interactive acceptance remain open. Later manifest/catalog mismatches in issue 68 stop before these VMs and are tracked separately.

## Summary Two existing FAT32 storage scenarios exceed their 120-second QEMU deadlines in the full Integration catalog. These failures prevent catalog acceptance. The paired run at the same Integration revision passes both scenarios, so the underlying cause and a deterministic isolated reproduction remain unestablished. ## Reproduction Observed in [Integration CI 1645](https://git.erikinkinen.fi/erix/integration/actions/runs/1645), exact revision `df8147dfa437194cd091747fd7147941bd2793c4`, in the ordinary full catalog workflow: - Worker 8: `tests/scenarios/subsystem-e2fs-fat-fat32-mirrored-positive.toml`. - Worker 11: `tests/scenarios/subsystem-e2fs-fat-fat32-vfat-unicode-positive.toml`. Both report `qemu timed out after 120s` on 16 September 2026. The original run was not restarted. Preserve the source selection, scenario inputs, build profile, original limits and failing execution artifacts when constructing the next bounded reproduction. Do not retry unchanged inputs until green or increase the deadline to claim a correction. ## Expected and actual behavior Expected: each scenario reaches its declared storage/VFS readiness markers and expected guest exit `0x10`, with its post-VM checks passing, within the configured 120 seconds. Actual: the QEMU overall deadline expires and both scenarios return failure. CI 1645 finishes with 405 passing and 81 failing scenarios out of 486: the 78 known layout refusals in #57, the previously tracked quota timeout in #18, and these two FAT32 timeouts. The full workflow fails. [Paired CI 1646](https://git.erikinkinen.fi/erix/integration/actions/runs/1646), at the identical Integration revision, records 408 passes and only the 78 layout refusals; both FAT32 scenarios and quota pass there. This passing observation does not resolve the retained failures. ## Environment and identity - Integration source: `df8147dfa437194cd091747fd7147941bd2793c4`; push run 1645 and paired review run 1646. - Full-system x86-64 VM scenarios, release profile, one guest CPU, writable block media, original QEMU timeout of 120 seconds. - Runtime configuration: `tests/runtime-configs/runtime.dev.storage-writable-fat.toml`; root storage test mode `e2fs-fat`. - Selected media: `fs-fat32-mirrored` and `fs-fat32-vfat-unicode` respectively. - CI package log: QEMU `1:10.0.13+ds-0+deb13u1`, OVMF `2025.02-8+deb13u1`. Package versions do not prove identical emulator/firmware bytes or host scheduling. - Exact original image, firmware and complete toolchain digests are not available from the artifact API; execution-environment equivalence is not established. No broader last-known-working revision is asserted. - No new input route or capability grant is introduced by this report. A guest authority failure has not been demonstrated. ## Evidence and prior checks All six terminal logs for runs 1645/1646 are complete and hash-bound in retained validation evidence, totaling 27,222,183 bytes with zero warning candidates. Rust, Markdown and preceding host controls pass in both runs. The complete scenario summaries independently identify each failure; each timeout belongs to its named worker. Host timestamps around replayed guest dumps are not guest event timing. The Forgejo artifact endpoints for these original runs return no artifacts. An independent cause, exact image equivalence and deterministic minimal reproduction have not been established. The shared timing symptom does not establish that these failures have the same cause as #18. No source change, timeout workaround or unchanged rerun is claimed as a fix. Original transcript analysis, 17 September 2026: the mirrored FAT32 guest reached `ERIX_ROOTD:PROGRESS:VFS:ESP_IO_DONE`; the Unicode guest additionally reached `DATA_IO_DONE`. Both transcripts show earlier FAT creation, truncate, short-name and rename completion. Neither records final VFS I/O readiness or final root readiness. These observations narrow the uncompleted portion of the storage pipeline but do not identify a stalled instruction or an underlying defect. The paired passing run's host runner intervals are approximately 41.26 seconds (mirrored) and 43.45 seconds (Unicode); the failing intervals are approximately 120.45 and 120.96 seconds. These are QEMU-launch-to-runner-result intervals, not guest operation durations. The retained source shows serial dumps are printed only after QEMU terminates, so timestamps on those dumped markers cannot be used as arrival times. The next diagnostic is designed but has not been executed: bind fresh source, scenario/configuration, image, emulator and firmware identities, then attach the existing bounded serial observer before the VM creates its fresh log. Retain raw per-marker arrival windows, including ESP, DATA and HTree completion, under the existing observer and VM limits. Its shell-oriented success summary does not apply to a storage scenario and must remain incomplete; do not alter the guest to manufacture a prompt. An isolated fresh observation cannot erase the original catalog failures. Original artifact absence still prevents an exact image replay. Original Integration catalog completion — 18 September 2026: At exact revision `0db8a679f5d1acb50d9ff92e003cec91f145dfc8`, [CI 1659](https://git.erikinkinen.fi/erix/integration/actions/runs/1659) finishes with 396 passes and 90 failures out of 486; [CI 1660](https://git.erikinkinen.fi/erix/integration/actions/runs/1660) finishes with 393 passes and 93 failures. Each retains the 78 known Exsh layout refusals. The remaining 12 and 15 failures are original 120-second QEMU storage-scenario timeouts. All six terminal logs are complete (36,570,659 bytes), with zero warning candidates. Rust/unit and Markdown jobs pass. No workflow was restarted, no deadline was enlarged and no failed scenario is accepted. A shared guest or host cause is unestablished. | Original positive case | CI 1659 | CI 1660 | | --- | --- | --- | | `fat32-directory-metadata` | timeout | timeout | | `fat32-primary-only` | pass | timeout | A paired pass does not close a retained failure. These are observed complete-catalog results; isolated reproduction and failure frequency remain unestablished. Preserve the original source graph and limits for diagnosis. ## Tracking Related retained failures: quota #18 and coherent Exsh layout adoption #57. Coordinated validation: #1 and #12. Keep this issue open until the original scenario identities and authority/cleanup behavior are preserved in a bounded reproduction, a cause is identified, and the correction passes both focused checks and the complete regression catalog under the unchanged deadlines. Original regression follow-up — 20 September 2026: Original Integration 1681/1682 at `a62d1381f56a01afc692112d9b427205eaeb6a2e` finished with 479/489 and 487/489 VM scenario passes. Six complete logs total 31,617,835 bytes with zero warning candidates. The following failures reach the unchanged 120-second QEMU deadline; none was retried or cancelled. Run 1681: `subsystem-e2fs-fat-fat32-active-fat-positive`, `subsystem-e2fs-fat-fat32-directory-metadata-positive`, `subsystem-e2fs-fat-fat32-stress-positive`, `subsystem-e2fs-fat-fat32-primary-only-positive`, `subsystem-e2fs-fat-fat32-metadata-clean-positive`. Run 1682: `subsystem-e2fs-fat-fat32-vfat-unicode-positive`. Each timeout is correlated with its own worker and serial dump, stopping before that worker starts another scenario. The retained dumps show bootstrap/VFS progress and native logging activity; these observations do not identify a common cause or prove a filesystem implementation fault. Full service, filesystem and interactive acceptance remain open. Later manifest/catalog mismatches in [issue 68](https://git.erikinkinen.fi/erix/integration/issues/68) stop before these VMs and are tracked separately.

erikinkinen added the

performance

bug

phase-6

labels

2026-09-17 09:01:21 +02:00

erikinkinen referenced this issue

2026-09-17 09:02:19 +02:00

[BUG] Existing ext4 quota VM scenario exceeds its QEMU deadline #18

erikinkinen referenced this issue

2026-09-17 09:02:19 +02:00

[BUG] Positive serial-progress host fixture expires before completion #53