erix/integration

Watch

Fork 0

generated from erix/meta

[BUG] Existing ext4 quota VM scenario exceeds its QEMU deadline #18

New issue

Open

opened 2026-09-12 11:15:01 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-12 11:15:01 +02:00

Owner

Summary

The existing ext4 quota VM scenario timed out in the full Integration catalog. The workflow continued to later scenarios with a recorded failure, so a completed catalog cannot be accepted as passing.

Reproduction

Observed in PR CI run 1560 at Integration e7995dc7a2, worker 6, scenario subsystem-e2fs-fat-ext4-quota-positive. The bounded catalog's bootloader VM stage 3/3 reached the QEMU overall 120-second deadline and terminated QEMU through the timeout wrapper.

The same quota scenario also timed out at 09:33:48 UTC in push run 1561, revision 5bf0fd2c84dd7cd838655cd8abc6915997868a04. Both observations are retained; an isolated deterministic reproduction and the underlying cause are not yet established. Do not retry unchanged inputs until green or increase the deadline to claim a correction.

Expected and actual behavior

Expected: the existing scenario completes its required success/cleanup checks within the unchanged declared deadline.

Actual: QEMU overall timeout at 09:07:23 UTC on 2026-09-12. No serial-silence timeout was reported. Run 1560 is now a terminal failure: all twelve worker summaries total 485 passed and one failed out of 486 catalog scenarios. Later dependent integration steps did not establish acceptance. Run 1561 is also a terminal failure: all twelve worker summaries independently total 485 passed and one failed out of 486. It repeats the same quota timeout. Neither failed catalog establishes acceptance for subsequent dependent steps.

Environment and identity

Integration revision: e7995dc7a2.
PR run 1560, Integration job, worker 6.
Scenario: subsystem-e2fs-fat-ext4-quota-positive; bootloader VM stage 3/3.
Recorded limit: QEMU overall 120 seconds; preserve the configured limits.
The reported 72-component revision map, source catalogs, quota scenario, runtime profile and 120-second deadline match passing workflow 1554. The scenario runner source is also unchanged.
Outer workflow/helper code and VM networking arguments differ. QEMU changed from 1:10.0.11+ds-0+deb13u1 to 1:10.0.13+ds-0+deb13u1; both logs report OVMF 2025.02-8+deb13u1. Package version equality is not firmware-byte equality.
Exact matching image/firmware digests and full compiler/support provenance are unavailable; complete execution-environment equivalence and the cause remain unestablished.

Evidence and prior checks

Additional retained observation: CI 1597 at 84e876b0a025159e77e8a62f44947209f098ca3d reports worker 6 QEMU timeout after 120 seconds for subsystem-e2fs-fat-ext4-quota-positive.toml. The final catalog is 485 passed, one failed, 486 total. Later image gates are not reached. Rust and Markdown pass. Complete original logs identify the existing scenario and deadline, without establishing the guest cause or a newly reproduced frequency.

CI run 1560 retains the timeout and catalog continuation. The observed complete log-response snapshot had no warning candidates. Guest output is replayed by the failure handler; its host dump timestamps cannot establish the original guest event timing.

Complete terminal logs for run 1560 confirm the 485/1 catalog result and failure exit. The complete terminal logs for run 1561 independently confirm the same 485/1 catalog result. Both failed guest dumps end after DATA_IO_DONE and successful LOGD reply/wake markers; no subsequent nested-directory completion is shown. These host replay timestamps do not establish guest event times.

The last guest dump contains DATA_IO_DONE and LOGD reply/wake markers. These identify progress but do not prove that the waiting process resumed or that the subsequent directory operations completed.

The next scenario started after this failure. Separate run 1559 failed earlier in a host scenario-lock fixture; that is tracked independently and does not explain this guest timeout. No attribution to contention, filesystem code, the observer or a recent change is established.

Endpoint construction checkpoint — 14 September 2026: Completed older Integration 1605/1606 at 7cc0593e101b2e0ca42f24d4fa189dd96cb1455c each reports 485 passes and one failure across all 486 scenarios. The sole failed scenario is the existing ext4 quota timeout tracked in #18. All six complete logs total 27,546,527 bytes with zero warning candidates; Rust and Markdown pass. There are no VM-worker dependency-feature compiler failures. Subsequent dependent image and terminal gates remain unestablished. The allocator cohort 1607/1608 is running; current endpoint-construction cohort 1609/1610 is waiting. Both failure-handler guest dumps end after DATA_IO_DONE and successful LOGD reply/wake markers, repeating the recorded boundary. Dump timestamps are host replay times. No new causal attribution, unchanged rerun or deadline increase is claimed.

Completed CI checkpoint — 15 September 2026: Signed Integration a869a81eb406a4f027a2b5db573b6330ca114d94 passes both 1607/1608; signed endpoint-construction checkpoint 989d44d604d07c4c9bcc23264912bdf2076b99eb passes both 1609/1610. Each run reports all 486 distinct VM scenarios passing, followed by the actual native lifetime scenario, physical/serial interactive checks and all four console modes. Rust and Markdown pass. The two six-log cohorts contain 26,769,289 and 26,769,316 bytes respectively, with zero warning candidates. The previously observed intermittent quota timeout remains documented without a causal-fix claim. These results establish the selected source checkpoints, not owned invocation transport, complete authority closure or either full in-EriX build generation.

Original memory-receipt cohort observation — 16 September 2026: CI 1639 at db80a2c1dfc669bd724762d476f5f5c0da005ff6 is terminal with 407 passes and 79 failures across 486 scenarios. Exactly 78 preparation refusals report the incompatible Exsh workspace layout tracked in #57. The additional failed scenario is the existing ext4 quota timeout tracked in #18: worker 6 reports QEMU timeout after 120 seconds and the scenario records status 1. Its guest dump ends after DATA_IO_DONE and successful LOGD reply/wake markers; these host replay timestamps do not establish guest event times or a cause.

All three complete terminal logs total 13,431,206 bytes with no compiler/lint warning candidates. Rust and Markdown pass. The run artifact API returns no original image/firmware bytes or inner receipts, so exact environment equivalence remains unverified. Original paired CI 1640 is still running. No unchanged rerun, relaxed deadline, guest correction or acceptance of later image gates is claimed.

Second original memory-receipt run — 16 September 2026: CI 1640, exact db80a2c1dfc669bd724762d476f5f5c0da005ff6, is now terminal. It reports 408 passes and 78 failures across 486 scenarios, with exactly 78 incompatible Exsh layout refusals. The ext4 quota scenario explicitly passes in this original paired run. That observation does not explain or fix the 120-second timeout in original CI 1639; #18 remains open, as does the layout adoption issue #57.

All three logs are complete (13,033,398 bytes) with zero compiler/lint warnings; Rust and Markdown pass. Both original runs together retain six complete logs totaling 26,464,604 bytes. The artifact API still exposes no original image/firmware or inner receipts. The cohort is fully classified without restarting it, changing deadlines or weakening mixed-layout rejection. Full source/frame/runtime prerequisites still gate coherent catalog adoption.

First matching watchdog correction run — 16 September 2026: CI 1641, exact 01daadd3d775079a77dd3278f6b07ecbaebe3e84, passes its 13 serial watchdog controls and remaining host checks. The full catalog run then records 407 passes and 79 failures across 486 scenarios: 78 known Exsh layout refusals and one 120-second ext4 quota timeout. This is another recurrence of #18, not evidence that the deterministic fixture correction changed guest timing. #57 still tracks coherent layout adoption. All three terminal logs are complete (13,436,715 bytes), with zero compiler/lint warnings; Rust and Markdown pass. The artifact API exposes no original images or inner receipts. Paired CI 1642 remains running; retain those same handles and original deadlines. No unchanged rerun or full runtime acceptance is claimed.

Further original CI results — 16 September 2026: watchdog CI 1642, exact 01daadd3d775079a77dd3278f6b07ecbaebe3e84, records 408 passes and 78 known layout failures across 486 scenarios; the quota scenario passes in that run. Permission-validation CI 1643, exact 89d15e4536004c087dcebd63c87216c0fd58958d, records 407 passes and 79 failures: the same 78 layout refusals plus the original 120-second quota timeout. Both runs pass their host controls, Rust and Markdown checks. All six terminal logs are complete, totaling 26,457,909 bytes with zero warning candidates. Artifact APIs expose no original images or inner receipts. This preserves the timing regression in #18 and coherent layout adoption in #57 as open obligations. Permission CI 1644 and release-input CI 1645 are running; CI 1646 waits. No run was restarted or assigned a larger deadline.

Remaining original CI results — 17 September 2026: permission CI 1644, exact 89d15e4536004c087dcebd63c87216c0fd58958d, finishes with 408 passing and 78 known layout refusals. Release-input CI 1645, exact df8147dfa437194cd091747fd7147941bd2793c4, finishes with 405 passes and 81 failures: those 78 layout refusals, the original quota timeout, and new 120-second FAT32 mirrored/Unicode timeouts. Paired CI 1646, at the same source revision, records 408 passes and only the 78 layout refusals. The separate FAT32 observations are tracked with the canonical bug template in #58; a common cause with quota is not established.

All nine terminal logs are complete, totaling 40,256,559 bytes with zero warning candidates; host/Rust/Markdown checks pass. Original artifact APIs expose no images or inner receipts. Both formerly pending cohorts are fully classified, with their original failures retained. No run was restarted, no limit was increased and no complete product or guest-build acceptance is asserted.

Original Integration regression classification — 17 September 2026: Original runs 1647–1650 are fully classified. Runs 1648, 1649 and 1650 each pass 408 of 486 scenarios and fail the same 78 known shell-layout refusals in issue 57. Run 1647 passes 407 and additionally repeats the existing ext4 quota timeout in issue 18. All twelve complete terminal logs total 52,533,051 bytes and contain no warnings. Rust and Markdown jobs pass. No additional failed scenario is identified; the complete catalogs remain failed. The separate local automatic-KVM preboot observation is tracked in issue 60.

Original archive-head regression classification — 17 September 2026: CI 1653 and CI 1654, both at 77921a7d97874e3f44aae66ae9bef6e6196af2f5, each finish with 406 passing and 80 failing scenarios out of 486. All six terminal logs are complete (27,648,897 bytes), with no warning candidates; Rust and Markdown pass. Both retain the 78 layout refusals and quota timeout. The additional original failures are ext4-xattr in push CI 1653 and ext4-metadata in PR CI 1654, each at the unchanged 120-second QEMU deadline. The opposite case passes in the paired run. No serial-stall report, deterministic isolated reproduction or common cause is established. No workflow is restarted.

Original linker-selection regression classification — 18 September 2026: Integration bc0e67f927ded76dd3ff5790180a7bded0886885 finishes original CI 1655 and CI 1656 failed. Each passes 407/486 VM scenarios and retains 78 known Exsh layout refusals plus the existing ext4 quota 120-second timeout. Rust and Markdown pass. All six logs are complete (26,854,856 bytes), with zero warning candidates. No unchanged workflow was retried, and no new common cause or guest-build acceptance is inferred.

Compiler-metadata original CI closure — 18 September 2026: CI 1657 and CI 1658, exact 56e035d46a11dc803d654bf2151e484fab307682, each finish with 407 passes and 79 failures across 486 scenarios. Each retains 78 known Exsh capacity-layout refusals (#57) and one ext4 quota timeout at the original 120-second deadline (#18). Rust and Markdown pass. All six terminal logs are complete: 26,867,392 bytes, no compiler/lint warnings. These observations establish recurrence, without establishing a new guest timing cause. Both known bugs and full runtime acceptance remain open; no unchanged rerun or deadline change was made.

Original Integration catalog completion — 18 September 2026: At exact revision 0db8a679f5d1acb50d9ff92e003cec91f145dfc8, CI 1659 finishes with 396 passes and 90 failures out of 486; CI 1660 finishes with 393 passes and 93 failures. Each retains the 78 known Exsh layout refusals. The remaining 12 and 15 failures are original 120-second QEMU storage-scenario timeouts. All six terminal logs are complete (36,570,659 bytes), with zero warning candidates. Rust/unit and Markdown jobs pass. No workflow was restarted, no deadline was enlarged and no failed scenario is accepted. A shared guest or host cause is unestablished.

Original relocation catalog CI closure — 18 September 2026: CI 1663 and CI 1664, exact 8e66e54c86d93f257b6051da0381ff8cdc84fe2f, each finish with 407 passes and 79 failures across 486 scenarios. Each retains 78 known Exsh capacity-layout refusals (#57) and one ext4 quota timeout at the original 120-second deadline (#18). Rust and Markdown pass. All six terminal logs are complete: 26,859,606 bytes, no compiler/lint warnings. These observations establish recurrence, without establishing a new guest timing cause. Both known bugs and full runtime acceptance remain open; no unchanged rerun or deadline change was made.

Original coherent-graph CI classification — 19 September 2026: CI 1669 records 485 passes/1 failure; CI 1670 records 484 passes/2 failures, each across all 486 scenarios at 8b1c037aed2503e1f2a4b17c8a8be0666d62a305. Both retain the original 120-second ext4 quota timeout in bug 18; the latter also fails physical input observation in the exact-selector race scenario, now bug 64. That scenario explicitly passes in the paired run. All six terminal logs are complete (27,557,577 bytes), with no warning candidates; Rust and Markdown pass. No incompatible shell-layout refusal remains. The layout issue’s full source/frame prerequisites and later native-image gates remain unproved. Neither workflow was rerun, and no deadline or admission check was relaxed.

Original CI observation — 19 September 2026: CI 1671 finishes with 471 passes/16 failures and CI 1672 with 465 passes/22 failures across all 487 scenarios at c479813423fb71c606fa30e5cf7d4dfb16f557d0. Every failed scenario is an ext3/ext4 positive filesystem case reporting error: qemu timed out after 120s, with scenario status 1. The existing filesystem deadline report retains the expanded matrix; quota issue 18 applies to the quota failure in 1672. All six complete logs are hashed (41,592,478 bytes), with zero warning candidates. Rust, Markdown and the full helper step pass. The zero-realm startup and exact-selector race scenarios explicitly pass in both runs. No common cause, source regression boundary or performance diagnosis is inferred; all actual deadlines and failure statuses remain unchanged, and neither workflow was rerun.

Further original CI observations — 19 September 2026: push run 1679, original Integration b65183ddb93eb4396d4140a002c3727fec42ff87, passes 488 of 489 catalog scenarios and fails only subsystem-e2fs-fat-ext4-quota-positive. Worker 9 reports the unchanged 120-second QEMU timeout at 18:11:12 UTC; the complete catalog finishes with one recorded failure. Later native and interactive workflow gates are not reached in this run. Rust and Markdown pass. Three complete hashed logs total 13,856,284 bytes with zero warning candidates. The main log has 13,751,641 bytes, SHA256 4f0a4dceee87449767330274b37c4f17392be12b550eff37ba726aaa31c9daf2; its timeout is at raw LF line 48,985. Companion run 1680 passes all 489 catalog scenarios before a separate development COM1 directory failure tracked in issue 56. Neither the sibling pass nor the later failure establishes a cause for this timeout. No unchanged rerun, cancellation or limit change occurred.

Tracking

Related work: #2, #3, #11 and WIP PR #12. Preserve source/image/configuration identities, timeout status and remaining catalog outcomes. Obtain bounded reproduction or sufficient causal evidence, apply a reviewed correction, and require the unchanged scenario plus full current-revision regressions to pass without warnings. Keep unexecuted or incomplete checks explicit.

Original completed installer-handoff regressions — 21 September 2026:

Original Integration 1697 and 1698, at signed 648fbd5614d3d0b82223b1c3bb7f1b5a0c81ea7d, finish with 488/489 and 487/489 passing VM scenarios. Rust and Markdown jobs pass. All six complete logs are retained with verified hashes: 28,115,674 bytes and zero warning candidates.

subsystem-e2fs-fat-ext4-quota-positive reaches the unchanged 120-second QEMU deadline in run(s) 1697, 1698. Each timeout is correlated with its own worker, scenario start and serial dump, stopping before that worker begins another scenario. The dumps reach VFS progress and native logging activity; this does not establish a shared cause or prove a filesystem implementation defect. These original failures remain unresolved; no unchanged run was retried or cancelled and no deadline was relaxed. The subsequent acknowledged terminal-accounting service graph still requires separate adoption and VM validation.

Original full-regression observation — 21 September 2026:

Original Integration 1699/1700, source fcd7b4a9608f629a12de78c53da5c3615d906b46, each finish at 487/489 VM passes. All six complete logs retain 28,483,276 bytes with zero warning candidates; Rust and Markdown jobs pass. The original worker boundaries identify quota's 120-second timeout in 1699 and 45-second serial stall in 1700, ext4 links' 120-second timeout in 1700, and FAT32 directory metadata's 120-second timeout in 1699. The latter has a distinct canonical bug report. Existing quota 18 and links 20 remain open. No shared root cause is inferred, and no unchanged rerun, cancellation or threshold relaxation is requested.

Original full-regression follow-up — 22 September 2026: Integration run 1701 and run 1702, source b06dfad00202, both pass. Six complete original logs total 26,962,712 bytes with zero warning candidates. Each run explicitly records passing ext4 quota, ext4 links and FAT32 directory-metadata scenarios in the full catalog. These are observations of the original submitted runs; no job was restarted or cancelled and no deadline was changed for this follow-up. The prior failures remain retained, and these passes do not establish their cause or failure frequency. This report stays open while the independent timing investigation and current-source regressions continue.

Additional original full-regression observation — 22 September 2026: Integration 1703 and 1704, source dff878dd35, both pass. Six complete logs total 26,964,178 bytes with zero warnings and explicit ext4 quota/links and FAT32 directory passes in each run. These are observations of the original submitted handles, with no retry or deadline change. They do not establish the cause or frequency of the retained earlier failures, so this report remains open.

## Summary The existing ext4 quota VM scenario timed out in the full Integration catalog. The workflow continued to later scenarios with a recorded failure, so a completed catalog cannot be accepted as passing. ## Reproduction Observed in PR CI run 1560 at Integration e7995dc7a2cb32bcc13a923c473e1d5c0e731303, worker 6, scenario subsystem-e2fs-fat-ext4-quota-positive. The bounded catalog's bootloader VM stage 3/3 reached the QEMU overall 120-second deadline and terminated QEMU through the timeout wrapper. The same quota scenario also timed out at 09:33:48 UTC in [push run 1561](https://git.erikinkinen.fi/erix/integration/actions/runs/1561), revision `5bf0fd2c84dd7cd838655cd8abc6915997868a04`. Both observations are retained; an isolated deterministic reproduction and the underlying cause are not yet established. Do not retry unchanged inputs until green or increase the deadline to claim a correction. ## Expected and actual behavior Expected: the existing scenario completes its required success/cleanup checks within the unchanged declared deadline. Actual: QEMU overall timeout at 09:07:23 UTC on 2026-09-12. No serial-silence timeout was reported. Run 1560 is now a terminal failure: all twelve worker summaries total 485 passed and one failed out of 486 catalog scenarios. Later dependent integration steps did not establish acceptance. Run 1561 is also a terminal failure: all twelve worker summaries independently total 485 passed and one failed out of 486. It repeats the same quota timeout. Neither failed catalog establishes acceptance for subsequent dependent steps. ## Environment and identity - Integration revision: e7995dc7a2cb32bcc13a923c473e1d5c0e731303. - PR run 1560, Integration job, worker 6. - Scenario: subsystem-e2fs-fat-ext4-quota-positive; bootloader VM stage 3/3. - Recorded limit: QEMU overall 120 seconds; preserve the configured limits. - The reported 72-component revision map, source catalogs, quota scenario, runtime profile and 120-second deadline match passing workflow 1554. The scenario runner source is also unchanged. - Outer workflow/helper code and VM networking arguments differ. QEMU changed from 1:10.0.11+ds-0+deb13u1 to 1:10.0.13+ds-0+deb13u1; both logs report OVMF 2025.02-8+deb13u1. Package version equality is not firmware-byte equality. - Exact matching image/firmware digests and full compiler/support provenance are unavailable; complete execution-environment equivalence and the cause remain unestablished. ## Evidence and prior checks Additional retained observation: [CI 1597](https://git.erikinkinen.fi/erix/integration/actions/runs/1597) at `84e876b0a025159e77e8a62f44947209f098ca3d` reports worker 6 QEMU timeout after 120 seconds for `subsystem-e2fs-fat-ext4-quota-positive.toml`. The final catalog is 485 passed, one failed, 486 total. Later image gates are not reached. Rust and Markdown pass. Complete original logs identify the existing scenario and deadline, without establishing the guest cause or a newly reproduced frequency. [CI run 1560](https://git.erikinkinen.fi/erix/integration/actions/runs/1560) retains the timeout and catalog continuation. The observed complete log-response snapshot had no warning candidates. Guest output is replayed by the failure handler; its host dump timestamps cannot establish the original guest event timing. Complete terminal logs for run 1560 confirm the 485/1 catalog result and failure exit. The complete terminal logs for run 1561 independently confirm the same 485/1 catalog result. Both failed guest dumps end after DATA_IO_DONE and successful LOGD reply/wake markers; no subsequent nested-directory completion is shown. These host replay timestamps do not establish guest event times. The last guest dump contains DATA_IO_DONE and LOGD reply/wake markers. These identify progress but do not prove that the waiting process resumed or that the subsequent directory operations completed. The next scenario started after this failure. Separate run 1559 failed earlier in a host scenario-lock fixture; that is tracked independently and does not explain this guest timeout. No attribution to contention, filesystem code, the observer or a recent change is established. Endpoint construction checkpoint — 14 September 2026: Completed older Integration [1605](https://git.erikinkinen.fi/erix/integration/actions/runs/1605)/[1606](https://git.erikinkinen.fi/erix/integration/actions/runs/1606) at `7cc0593e101b2e0ca42f24d4fa189dd96cb1455c` each reports 485 passes and one failure across all 486 scenarios. The sole failed scenario is the existing ext4 quota timeout tracked in #18. All six complete logs total 27,546,527 bytes with zero warning candidates; Rust and Markdown pass. There are no VM-worker dependency-feature compiler failures. Subsequent dependent image and terminal gates remain unestablished. The allocator cohort 1607/1608 is running; current endpoint-construction cohort 1609/1610 is waiting. Both failure-handler guest dumps end after DATA_IO_DONE and successful LOGD reply/wake markers, repeating the recorded boundary. Dump timestamps are host replay times. No new causal attribution, unchanged rerun or deadline increase is claimed. Completed CI checkpoint — 15 September 2026: Signed Integration `a869a81eb406a4f027a2b5db573b6330ca114d94` passes both [1607](https://git.erikinkinen.fi/erix/integration/actions/runs/1607)/[1608](https://git.erikinkinen.fi/erix/integration/actions/runs/1608); signed endpoint-construction checkpoint `989d44d604d07c4c9bcc23264912bdf2076b99eb` passes both [1609](https://git.erikinkinen.fi/erix/integration/actions/runs/1609)/[1610](https://git.erikinkinen.fi/erix/integration/actions/runs/1610). Each run reports all 486 distinct VM scenarios passing, followed by the actual native lifetime scenario, physical/serial interactive checks and all four console modes. Rust and Markdown pass. The two six-log cohorts contain 26,769,289 and 26,769,316 bytes respectively, with zero warning candidates. The previously observed intermittent quota timeout remains documented without a causal-fix claim. These results establish the selected source checkpoints, not owned invocation transport, complete authority closure or either full in-EriX build generation. Original memory-receipt cohort observation — 16 September 2026: [CI 1639](https://git.erikinkinen.fi/erix/integration/actions/runs/1639) at `db80a2c1dfc669bd724762d476f5f5c0da005ff6` is terminal with 407 passes and 79 failures across 486 scenarios. Exactly 78 preparation refusals report the incompatible Exsh workspace layout tracked in [#57](https://git.erikinkinen.fi/erix/integration/issues/57). The additional failed scenario is the existing ext4 quota timeout tracked in [#18](https://git.erikinkinen.fi/erix/integration/issues/18): worker 6 reports QEMU timeout after 120 seconds and the scenario records status 1. Its guest dump ends after DATA_IO_DONE and successful LOGD reply/wake markers; these host replay timestamps do not establish guest event times or a cause. All three complete terminal logs total 13,431,206 bytes with no compiler/lint warning candidates. Rust and Markdown pass. The run artifact API returns no original image/firmware bytes or inner receipts, so exact environment equivalence remains unverified. Original paired CI 1640 is still running. No unchanged rerun, relaxed deadline, guest correction or acceptance of later image gates is claimed. Second original memory-receipt run — 16 September 2026: [CI 1640](https://git.erikinkinen.fi/erix/integration/actions/runs/1640), exact `db80a2c1dfc669bd724762d476f5f5c0da005ff6`, is now terminal. It reports 408 passes and 78 failures across 486 scenarios, with exactly 78 incompatible Exsh layout refusals. The ext4 quota scenario explicitly passes in this original paired run. That observation does not explain or fix the 120-second timeout in original CI 1639; [#18](https://git.erikinkinen.fi/erix/integration/issues/18) remains open, as does the layout adoption issue [#57](https://git.erikinkinen.fi/erix/integration/issues/57). All three logs are complete (13,033,398 bytes) with zero compiler/lint warnings; Rust and Markdown pass. Both original runs together retain six complete logs totaling 26,464,604 bytes. The artifact API still exposes no original image/firmware or inner receipts. The cohort is fully classified without restarting it, changing deadlines or weakening mixed-layout rejection. Full source/frame/runtime prerequisites still gate coherent catalog adoption. First matching watchdog correction run — 16 September 2026: [CI 1641](https://git.erikinkinen.fi/erix/integration/actions/runs/1641), exact `01daadd3d775079a77dd3278f6b07ecbaebe3e84`, passes its 13 serial watchdog controls and remaining host checks. The full catalog run then records 407 passes and 79 failures across 486 scenarios: 78 known Exsh layout refusals and one 120-second ext4 quota timeout. This is another recurrence of [#18](https://git.erikinkinen.fi/erix/integration/issues/18), not evidence that the deterministic fixture correction changed guest timing. [#57](https://git.erikinkinen.fi/erix/integration/issues/57) still tracks coherent layout adoption. All three terminal logs are complete (13,436,715 bytes), with zero compiler/lint warnings; Rust and Markdown pass. The artifact API exposes no original images or inner receipts. Paired CI 1642 remains running; retain those same handles and original deadlines. No unchanged rerun or full runtime acceptance is claimed. Further original CI results — 16 September 2026: watchdog [CI 1642](https://git.erikinkinen.fi/erix/integration/actions/runs/1642), exact `01daadd3d775079a77dd3278f6b07ecbaebe3e84`, records 408 passes and 78 known layout failures across 486 scenarios; the quota scenario passes in that run. Permission-validation [CI 1643](https://git.erikinkinen.fi/erix/integration/actions/runs/1643), exact `89d15e4536004c087dcebd63c87216c0fd58958d`, records 407 passes and 79 failures: the same 78 layout refusals plus the original 120-second quota timeout. Both runs pass their host controls, Rust and Markdown checks. All six terminal logs are complete, totaling 26,457,909 bytes with zero warning candidates. Artifact APIs expose no original images or inner receipts. This preserves the timing regression in #18 and coherent layout adoption in #57 as open obligations. Permission CI 1644 and release-input CI 1645 are running; CI 1646 waits. No run was restarted or assigned a larger deadline. Remaining original CI results — 17 September 2026: permission [CI 1644](https://git.erikinkinen.fi/erix/integration/actions/runs/1644), exact `89d15e4536004c087dcebd63c87216c0fd58958d`, finishes with 408 passing and 78 known layout refusals. Release-input [CI 1645](https://git.erikinkinen.fi/erix/integration/actions/runs/1645), exact `df8147dfa437194cd091747fd7147941bd2793c4`, finishes with 405 passes and 81 failures: those 78 layout refusals, the original quota timeout, and new 120-second FAT32 mirrored/Unicode timeouts. Paired [CI 1646](https://git.erikinkinen.fi/erix/integration/actions/runs/1646), at the same source revision, records 408 passes and only the 78 layout refusals. The separate FAT32 observations are tracked with the canonical bug template in [#58](https://git.erikinkinen.fi/erix/integration/issues/58); a common cause with quota is not established. All nine terminal logs are complete, totaling 40,256,559 bytes with zero warning candidates; host/Rust/Markdown checks pass. Original artifact APIs expose no images or inner receipts. Both formerly pending cohorts are fully classified, with their original failures retained. No run was restarted, no limit was increased and no complete product or guest-build acceptance is asserted. Original Integration regression classification — 17 September 2026: Original runs 1647–1650 are fully classified. Runs 1648, 1649 and 1650 each pass 408 of 486 scenarios and fail the same 78 known shell-layout refusals in [issue 57](https://git.erikinkinen.fi/erix/integration/issues/57). Run 1647 passes 407 and additionally repeats the existing ext4 quota timeout in [issue 18](https://git.erikinkinen.fi/erix/integration/issues/18). All twelve complete terminal logs total 52,533,051 bytes and contain no warnings. Rust and Markdown jobs pass. No additional failed scenario is identified; the complete catalogs remain failed. The separate local automatic-KVM preboot observation is tracked in [issue 60](https://git.erikinkinen.fi/erix/integration/issues/60). Original archive-head regression classification — 17 September 2026: [CI 1653](https://git.erikinkinen.fi/erix/integration/actions/runs/1653) and [CI 1654](https://git.erikinkinen.fi/erix/integration/actions/runs/1654), both at `77921a7d97874e3f44aae66ae9bef6e6196af2f5`, each finish with 406 passing and 80 failing scenarios out of 486. All six terminal logs are complete (27,648,897 bytes), with no warning candidates; Rust and Markdown pass. Both retain the 78 layout refusals and quota timeout. The additional original failures are ext4-xattr in push CI 1653 and ext4-metadata in PR CI 1654, each at the unchanged 120-second QEMU deadline. The opposite case passes in the paired run. No serial-stall report, deterministic isolated reproduction or common cause is established. No workflow is restarted. Original linker-selection regression classification — 18 September 2026: Integration `bc0e67f927ded76dd3ff5790180a7bded0886885` finishes original [CI 1655](https://git.erikinkinen.fi/erix/integration/actions/runs/1655) and [CI 1656](https://git.erikinkinen.fi/erix/integration/actions/runs/1656) failed. Each passes 407/486 VM scenarios and retains 78 known Exsh layout refusals plus the existing ext4 quota 120-second timeout. Rust and Markdown pass. All six logs are complete (26,854,856 bytes), with zero warning candidates. No unchanged workflow was retried, and no new common cause or guest-build acceptance is inferred. Compiler-metadata original CI closure — 18 September 2026: [CI 1657](https://git.erikinkinen.fi/erix/integration/actions/runs/1657) and [CI 1658](https://git.erikinkinen.fi/erix/integration/actions/runs/1658), exact `56e035d46a11dc803d654bf2151e484fab307682`, each finish with 407 passes and 79 failures across 486 scenarios. Each retains 78 known Exsh capacity-layout refusals ([#57](https://git.erikinkinen.fi/erix/integration/issues/57)) and one ext4 quota timeout at the original 120-second deadline ([#18](https://git.erikinkinen.fi/erix/integration/issues/18)). Rust and Markdown pass. All six terminal logs are complete: 26,867,392 bytes, no compiler/lint warnings. These observations establish recurrence, without establishing a new guest timing cause. Both known bugs and full runtime acceptance remain open; no unchanged rerun or deadline change was made. Original Integration catalog completion — 18 September 2026: At exact revision `0db8a679f5d1acb50d9ff92e003cec91f145dfc8`, [CI 1659](https://git.erikinkinen.fi/erix/integration/actions/runs/1659) finishes with 396 passes and 90 failures out of 486; [CI 1660](https://git.erikinkinen.fi/erix/integration/actions/runs/1660) finishes with 393 passes and 93 failures. Each retains the 78 known Exsh layout refusals. The remaining 12 and 15 failures are original 120-second QEMU storage-scenario timeouts. All six terminal logs are complete (36,570,659 bytes), with zero warning candidates. Rust/unit and Markdown jobs pass. No workflow was restarted, no deadline was enlarged and no failed scenario is accepted. A shared guest or host cause is unestablished. Original relocation catalog CI closure — 18 September 2026: [CI 1663](https://git.erikinkinen.fi/erix/integration/actions/runs/1663) and [CI 1664](https://git.erikinkinen.fi/erix/integration/actions/runs/1664), exact `8e66e54c86d93f257b6051da0381ff8cdc84fe2f`, each finish with 407 passes and 79 failures across 486 scenarios. Each retains 78 known Exsh capacity-layout refusals ([#57](https://git.erikinkinen.fi/erix/integration/issues/57)) and one ext4 quota timeout at the original 120-second deadline ([#18](https://git.erikinkinen.fi/erix/integration/issues/18)). Rust and Markdown pass. All six terminal logs are complete: 26,859,606 bytes, no compiler/lint warnings. These observations establish recurrence, without establishing a new guest timing cause. Both known bugs and full runtime acceptance remain open; no unchanged rerun or deadline change was made. Original coherent-graph CI classification — 19 September 2026: [CI 1669](https://git.erikinkinen.fi/erix/integration/actions/runs/1669) records 485 passes/1 failure; [CI 1670](https://git.erikinkinen.fi/erix/integration/actions/runs/1670) records 484 passes/2 failures, each across all 486 scenarios at `8b1c037aed2503e1f2a4b17c8a8be0666d62a305`. Both retain the original 120-second ext4 quota timeout in [bug 18](https://git.erikinkinen.fi/erix/integration/issues/18); the latter also fails physical input observation in the exact-selector race scenario, now [bug 64](https://git.erikinkinen.fi/erix/integration/issues/64). That scenario explicitly passes in the paired run. All six terminal logs are complete (27,557,577 bytes), with no warning candidates; Rust and Markdown pass. No incompatible shell-layout refusal remains. The layout issue’s full source/frame prerequisites and later native-image gates remain unproved. Neither workflow was rerun, and no deadline or admission check was relaxed. Original CI observation — 19 September 2026: [CI 1671](https://git.erikinkinen.fi/erix/integration/actions/runs/1671) finishes with 471 passes/16 failures and [CI 1672](https://git.erikinkinen.fi/erix/integration/actions/runs/1672) with 465 passes/22 failures across all 487 scenarios at `c479813423fb71c606fa30e5cf7d4dfb16f557d0`. Every failed scenario is an ext3/ext4 positive filesystem case reporting `error: qemu timed out after 120s`, with scenario status 1. The existing [filesystem deadline report](https://git.erikinkinen.fi/erix/integration/issues/20) retains the expanded matrix; [quota issue 18](https://git.erikinkinen.fi/erix/integration/issues/18) applies to the quota failure in 1672. All six complete logs are hashed (41,592,478 bytes), with zero warning candidates. Rust, Markdown and the full helper step pass. The zero-realm startup and exact-selector race scenarios explicitly pass in both runs. No common cause, source regression boundary or performance diagnosis is inferred; all actual deadlines and failure statuses remain unchanged, and neither workflow was rerun. Further original CI observations — 19 September 2026: [push run 1679](https://git.erikinkinen.fi/erix/integration/actions/runs/1679), original Integration `b65183ddb93eb4396d4140a002c3727fec42ff87`, passes 488 of 489 catalog scenarios and fails only `subsystem-e2fs-fat-ext4-quota-positive`. Worker 9 reports the unchanged 120-second QEMU timeout at 18:11:12 UTC; the complete catalog finishes with one recorded failure. Later native and interactive workflow gates are not reached in this run. Rust and Markdown pass. Three complete hashed logs total 13,856,284 bytes with zero warning candidates. The main log has 13,751,641 bytes, SHA256 `4f0a4dceee87449767330274b37c4f17392be12b550eff37ba726aaa31c9daf2`; its timeout is at raw LF line 48,985. Companion run 1680 passes all 489 catalog scenarios before a separate development COM1 directory failure tracked in [issue 56](https://git.erikinkinen.fi/erix/integration/issues/56). Neither the sibling pass nor the later failure establishes a cause for this timeout. No unchanged rerun, cancellation or limit change occurred. ## Tracking Related work: #2, #3, #11 and WIP PR #12. Preserve source/image/configuration identities, timeout status and remaining catalog outcomes. Obtain bounded reproduction or sufficient causal evidence, apply a reviewed correction, and require the unchanged scenario plus full current-revision regressions to pass without warnings. Keep unexecuted or incomplete checks explicit. Original completed installer-handoff regressions — 21 September 2026: Original Integration [1697](https://git.erikinkinen.fi/erix/integration/actions/runs/1697) and [1698](https://git.erikinkinen.fi/erix/integration/actions/runs/1698), at signed `648fbd5614d3d0b82223b1c3bb7f1b5a0c81ea7d`, finish with 488/489 and 487/489 passing VM scenarios. Rust and Markdown jobs pass. All six complete logs are retained with verified hashes: 28,115,674 bytes and zero warning candidates. `subsystem-e2fs-fat-ext4-quota-positive` reaches the unchanged 120-second QEMU deadline in run(s) 1697, 1698. Each timeout is correlated with its own worker, scenario start and serial dump, stopping before that worker begins another scenario. The dumps reach VFS progress and native logging activity; this does not establish a shared cause or prove a filesystem implementation defect. These original failures remain unresolved; no unchanged run was retried or cancelled and no deadline was relaxed. The subsequent acknowledged terminal-accounting service graph still requires separate adoption and VM validation. Original full-regression observation — 21 September 2026: Original Integration [1699](https://git.erikinkinen.fi/erix/integration/actions/runs/1699)/[1700](https://git.erikinkinen.fi/erix/integration/actions/runs/1700), source `fcd7b4a9608f629a12de78c53da5c3615d906b46`, each finish at 487/489 VM passes. All six complete logs retain 28,483,276 bytes with zero warning candidates; Rust and Markdown jobs pass. The original worker boundaries identify quota's 120-second timeout in 1699 and 45-second serial stall in 1700, ext4 links' 120-second timeout in 1700, and FAT32 directory metadata's 120-second timeout in 1699. The latter has a distinct [canonical bug report](https://git.erikinkinen.fi/erix/integration/issues/70). Existing [quota 18](https://git.erikinkinen.fi/erix/integration/issues/18) and [links 20](https://git.erikinkinen.fi/erix/integration/issues/20) remain open. No shared root cause is inferred, and no unchanged rerun, cancellation or threshold relaxation is requested. Original full-regression follow-up — 22 September 2026: Integration [run 1701](https://git.erikinkinen.fi/erix/integration/actions/runs/1701) and [run 1702](https://git.erikinkinen.fi/erix/integration/actions/runs/1702), source [b06dfad00202](https://git.erikinkinen.fi/erix/integration/commit/b06dfad00202765491a64552dde29eaca1c24838), both pass. Six complete original logs total 26,962,712 bytes with zero warning candidates. Each run explicitly records passing ext4 quota, ext4 links and FAT32 directory-metadata scenarios in the full catalog. These are observations of the original submitted runs; no job was restarted or cancelled and no deadline was changed for this follow-up. The prior failures remain retained, and these passes do not establish their cause or failure frequency. This report stays open while the independent timing investigation and current-source regressions continue. Additional original full-regression observation — 22 September 2026: Integration [1703](https://git.erikinkinen.fi/erix/integration/actions/runs/1703) and [1704](https://git.erikinkinen.fi/erix/integration/actions/runs/1704), source dff878dd3545c4751b3c05d37b2bdd5e21cce548, both pass. Six complete logs total 26,964,178 bytes with zero warnings and explicit ext4 quota/links and FAT32 directory passes in each run. These are observations of the original submitted handles, with no retry or deadline change. They do not establish the cause or frequency of the retained earlier failures, so this report remains open.

erikinkinen added the

bug

phase-6

labels

2026-09-12 11:15:01 +02:00

~~erikinkinen referenced this issue 2026-09-12 11:24:43 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

erikinkinen referenced this issue from erix/e2fsd

2026-09-12 12:14:34 +02:00

[BUG] Standalone service build can succeed without its declared entry point #2

erikinkinen referenced this issue from erix/e2fsd

2026-09-12 12:37:52 +02:00

WIP: Validate filesystem media and explicit build inputs #3

Rows
Columns