generated from erix/meta
[BUG] Additional positive ext3/ext4 VM scenarios exceed the QEMU deadline #20
Labels
No labels
bug
ci
docs
duplicate
enhancement
help wanted
invalid
performance
phase-6
question
refactor
security
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
erix/integration#20
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Positive ext3/ext4 cases exceed the existing 120-second QEMU deadline in original Integration catalogs. The earlier observations and the expanded CI 1659/1660 matrix below remain failures with unestablished causes. The quota case is tracked separately in #18 and FAT32 cases in #58.
Reproduction
Observed in the maintained Run complete VM scenario catalog workflow step:
This command assumes the workflow's preceding source-resolution, build, media and catalog preparation. The observations come from the complete CI runs below; an isolated clean-checkout reproduction and failure frequency are not established. The affected scenario files are
tests/scenarios/subsystem-e2fs-fat-<case>-positive.toml:Expected and actual behavior
Each scenario is declared
subsystem-positive, requests debug exit0x10, and requires successful VFS readiness markers. It should complete within the configured 120 seconds.Each failed worker instead reports
error: qemu timed out after 120s; its runner treats the timeout status as failure and returns 1. The retained guest output reachesERIX_ROOTD:PROGRESS:VFS:DATA_IO_DONE, followed by the kernel'sLOGD_REPLY_BEGIN,LOGD_REPLY_WRITE_OKandLOGD_REPLY_WAKE_OK=0000000000000001markers. QEMU then reports SIGTERM from the timeout process. The finalERIX_ROOTD:READY:VFS_IOand standaloneERIX_ROOTD:READYlines are absent. There is no reported serial-stall diagnostic or forbidden panic/failure marker in these worker segments.These are positive-case failures, not intentional negative fixtures. The shared final markers do not establish a common cause, a caller resumption or the completion of subsequent guest work.
Environment and identity
5bf0fd2c84dd7cd838655cd8abc6915997868a04, PR event, API run 8990, integration job 19333, attempt 1; terminal failure.2ae6c9cd34cd456c568e5f61976879902b73d9f6, push event, API run 9003, integration job 19360, attempt 1; terminal failure.4330175e27718490f06b93098e95e3c1553cb765, Rootdac3c1847af9915164d4a819f5f15e1827884e8b1, E2fsd63ff1883732069aa10a7333b6bbf1d53557b2101and lib-ipcb452a46803982c57e474adcc318ce40bd281d621. This is the reported component selection, not independently verified per-worker image provenance.e2fs-fat; scenario-specific writable filesystem media;runtime.dev.storage-writable-fat.toml. The catalog reports 12 workers, one Cargo job per worker, QEMU deadline/ceiling 120 seconds and serial-stall deadline 45 seconds.Evidence and prior checks
Complete terminal logs were retained for both runs. Run 1562 records its first additional timeout at 10:41:26 UTC and its catalog summary at 11:34:16 UTC on 2026-09-12; run 1563 records its first additional timeout at 10:54:52 UTC and its catalog summary at 11:40:14 UTC.
The earlier terminal run 1554, Integration
c86c0d89e51ed9818d62afa799d358768646c953, passes 486/486. Terminal run 1560, Integratione7995dc7a2cb32bcc13a923c473e1d5c0e731303, and run 1561, Integration5bf0fd2c84dd7cd838655cd8abc6915997868a04, each pass 485/486, with only the known quota case failing. All nine additional cases pass in both of those runs. Their original scenario definitions are unchanged across all five Integration heads, and all five logs report the same component graph. Run 1561 and run 1562 share the exact Integration head, so these observations do not identify a causal commit.Catalog totals above use the terminal catalog summary, corroborated by distinct scenario pass names. Counting every explicit pass line would include earlier helper-fixture passes and repeated worker-tail output. No deadline changes, VM reruns or source fixes were attempted during this read-only classification.
Original archive-head regression classification — 17 September 2026: CI 1653 and CI 1654, both at
77921a7d97874e3f44aae66ae9bef6e6196af2f5, each finish with 406 passing and 80 failing scenarios out of 486. All six terminal logs are complete (27,648,897 bytes), with no warning candidates; Rust and Markdown pass. Both retain the 78 layout refusals and quota timeout. The additional original failures are ext4-xattr in push CI 1653 and ext4-metadata in PR CI 1654, each at the unchanged 120-second QEMU deadline. The opposite case passes in the paired run. No serial-stall report, deterministic isolated reproduction or common cause is established. No workflow is restarted.Original Integration catalog completion — 18 September 2026: At exact revision
0db8a679f5d1acb50d9ff92e003cec91f145dfc8, CI 1659 finishes with 396 passes and 90 failures out of 486; CI 1660 finishes with 393 passes and 93 failures. Each retains the 78 known Exsh layout refusals. The remaining 12 and 15 failures are original 120-second QEMU storage-scenario timeouts. All six terminal logs are complete (36,570,659 bytes), with zero warning candidates. Rust/unit and Markdown jobs pass. No workflow was restarted, no deadline was enlarged and no failed scenario is accepted. A shared guest or host cause is unestablished.ext3-fast-commit-mutatingext3-geometryext3-htree-depth0ext3-htree-depth1ext3-jbd2-legacyext3-journal-scaleext3-linksext3-many-journalext3-renameext4-encrypted-casefoldext4-encryptedext4-extent-treeext4-mmpext4-renameext4-xattrA paired pass does not close a retained failure. These are observed complete-catalog results; isolated reproduction and failure frequency remain unestablished. Preserve the original source graph and limits for diagnosis.
Original CI observation — 19 September 2026: CI 1671 finishes with 471 passes/16 failures and CI 1672 with 465 passes/22 failures across all 487 scenarios at
c479813423fb71c606fa30e5cf7d4dfb16f557d0. Every failed scenario is an ext3/ext4 positive filesystem case reportingerror: qemu timed out after 120s, with scenario status 1. The existing filesystem deadline report retains the expanded matrix; quota issue 18 applies to the quota failure in 1672. All six complete logs are hashed (41,592,478 bytes), with zero warning candidates. Rust, Markdown and the full helper step pass. The zero-realm startup and exact-selector race scenarios explicitly pass in both runs. No common cause, source regression boundary or performance diagnosis is inferred; all actual deadlines and failure statuses remain unchanged, and neither workflow was rerun.Tracking
The ext4 quota failure remains tracked separately in Integration #18. Similar final progress is an observation, not evidence that the quota case and these additional failures share a cause. Resolution requires explaining the additional positive failures and demonstrating their unchanged assertions and deadlines pass; later workflow gates skipped by a failed catalog remain unvalidated for these runs.
Original corrected-source CI checkpoint — 19 September 2026: at signed Integration
9139c6c5fa, original push CI 1675 passes all 488 VM scenarios. Original PR CI 1676 passes 487/488 and fails onlysubsystem-e2fs-fat-ext3-htree-positiveat the unchanged 120-second QEMU limit. Both passappliance-disk-image-realm-admission-positive; both Rust and Markdown jobs pass. Six complete hashed logs contain 27,276,890 bytes and no warnings. No rerun or cancellation supplies either result. The ext3 timeout remains tracked in issue 20; its cause is not inferred from the passing sibling run. The private script-route admission defect is corrected and natively demonstrated, while full realm lifecycle, complete frame proof and guest builds remain open.Original regression follow-up — 20 September 2026:
Original Integration 1681/1682 at
a62d1381f56a01afc692112d9b427205eaeb6a2efinished with 479/489 and 487/489 VM scenario passes. Six complete logs total 31,617,835 bytes with zero warning candidates. The following failures reach the unchanged 120-second QEMU deadline; none was retried or cancelled.Run 1681:
subsystem-e2fs-fat-ext4-orphan-file-positive,subsystem-e2fs-fat-ext4-links-positive,subsystem-e2fs-fat-ext4-quota-positive,subsystem-e2fs-fat-ext4-rename-positive,subsystem-e2fs-fat-ext4-metadata-positive.Run 1682:
subsystem-e2fs-fat-ext4-quota-positive.Each timeout is correlated with its own worker and serial dump, stopping before that worker starts another scenario. The retained dumps show bootstrap/VFS progress and native logging activity; these observations do not identify a common cause or prove a filesystem implementation fault. Full service, filesystem and interactive acceptance remain open. Later manifest/catalog mismatches in issue 68 stop before these VMs and are tracked separately.
Installer return-slot regression — 21 September 2026:
Both original corrected-catalog runs are now classified: 1695 passes 431/489 VM scenarios and 1696 passes 430/489. All six complete logs are retained and hashed, 32,723,390 bytes, zero warning candidates. Rust, Markdown and full dependency equality pass. Catalog mismatch issue 68 is corrected; this does not establish full consumer acceptance. Each run has 57 initial-shell uncertain-disposition failures and a separately retained release-appliance stall. Run 1696 also retains the ext4 quota timeout under its original 120-second bound, tracked in issue 20.
Procd bug 4 records a concrete producer/consumer mismatch. The final installer was handed off from VSpace scratch slot 1056 while later TTY provisioning requires managed grant slot 1040. The added producer regression fails on the original code. Signed Procd 10d972b652297fd656e9a6ac6dbdf197f362c7ce in WIP PR 2 derives the exact GRANT-only result, disposes its delegating source and uniquely relocates the result back to the empty managed grant slot. Refusals preserve original-stage rollback and remaining custody. Four strict 291/296-unit configurations, four native builds, host/native Clippy, formatting, private rustdoc and policies pass with zero warnings. Corrected full-service VM validation and original CI remain open; bug 4 remains open.
Acceptance remains 15/460 leaves, 3.48% weighted. Full consumer lifecycle, terminal accounting, source/effect/frame proof, Pagerd, profiler attribution, native external Rust/LLVM/runtime rebuilding and both full in-EriX build generations remain required.
Original completed installer-handoff regressions — 21 September 2026:
Original Integration 1697 and 1698, at signed
648fbd5614d3d0b82223b1c3bb7f1b5a0c81ea7d, finish with 488/489 and 487/489 passing VM scenarios. Rust and Markdown jobs pass. All six complete logs are retained with verified hashes: 28,115,674 bytes and zero warning candidates.subsystem-e2fs-fat-ext3-links-positivereaches the unchanged 120-second QEMU deadline in run(s) 1698. Each timeout is correlated with its own worker, scenario start and serial dump, stopping before that worker begins another scenario. The dumps reach VFS progress and native logging activity; this does not establish a shared cause or prove a filesystem implementation defect. These original failures remain unresolved; no unchanged run was retried or cancelled and no deadline was relaxed. The subsequent acknowledged terminal-accounting service graph still requires separate adoption and VM validation.Original full-regression observation — 21 September 2026:
Original Integration 1699/1700, source
fcd7b4a9608f629a12de78c53da5c3615d906b46, each finish at 487/489 VM passes. All six complete logs retain 28,483,276 bytes with zero warning candidates; Rust and Markdown jobs pass. The original worker boundaries identify quota's 120-second timeout in 1699 and 45-second serial stall in 1700, ext4 links' 120-second timeout in 1700, and FAT32 directory metadata's 120-second timeout in 1699. The latter has a distinct canonical bug report. Existing quota 18 and links 20 remain open. No shared root cause is inferred, and no unchanged rerun, cancellation or threshold relaxation is requested.Original full-regression follow-up — 22 September 2026: Integration run 1701 and run 1702, source b06dfad00202, both pass. Six complete original logs total 26,962,712 bytes with zero warning candidates. Each run explicitly records passing ext4 quota, ext4 links and FAT32 directory-metadata scenarios in the full catalog. These are observations of the original submitted runs; no job was restarted or cancelled and no deadline was changed for this follow-up. The prior failures remain retained, and these passes do not establish their cause or failure frequency. This report stays open while the independent timing investigation and current-source regressions continue.
Additional original full-regression observation — 22 September 2026: Integration 1703 and 1704, source
dff878dd35, both pass. Six complete logs total 26,964,178 bytes with zero warnings and explicit ext4 quota/links and FAT32 directory passes in each run. These are observations of the original submitted handles, with no retry or deadline change. They do not establish the cause or frequency of the retained earlier failures, so this report remains open.erikinkinen referenced this issue2026-09-12 14:15:15 +02:00