[BUG] Positive serial-progress host fixture expires before completion #53

New issue

Open

opened 2026-09-15 23:54:58 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-15 23:54:58 +02:00

Owner

Summary

The positive serial-progress watchdog host fixture fails in Integration CI 1633, stopping the remaining integration suite before the scenario catalog runs.

Reproduction

Observed once in the retained original CI at the revision below; failure frequency is unknown. The relevant host command is python3 tests/progress_watchdog_test.py. Its positive fixture writes the arming marker followed by five appends separated by 100 ms sleeps, under a 250 ms stall deadline and 20 ms polling interval. No unchanged retry was performed.

Expected and actual behavior

Expected: test_post_arm_progress_refreshes_the_stall_deadline returns child status 0 with no watchdog error. Actual: the watchdog returns 125 and the assertion reports 125 != 0. The group runs eight tests with one failure. The terminal log does not retain accepted-progress timestamps or establish host contention as the cause.

Environment and identity

Original Integration 2a16b1a7ac5514a43582025e5b9ba0a2f2ce2a29, push CI 1633, integration job 20350, attempt 1.
Host Python fixture in the x86-64 Linux container; this failing test does not execute a guest image or grant guest authority.
The workflow retains its original pinned component graph. Rust and Markdown jobs pass. No last known working revision for this exact observed timing failure has been established.

Evidence and prior checks

CI 1633 reports the failure at 21:25:31 UTC on September 15. All three complete terminal job logs total 245,076 bytes with no compiler/lint warning candidates. Earlier QEMU wrapper watchdog fixtures pass. The paired review run was incomplete at that observation; its later terminal result is recorded below.

The fixture currently relies on real host scheduling to keep each positive write within the short deadline. Replace positive timing assertions with a finite logical child/clock schedule around the actual watchdog loop and real serial-file bytes. Preserve the original thresholds, marker detection, status propagation and real negative subprocess tests. Include missing refresh, split marker, exact expiry and interruption/cleanup controls. Do not modify production timing policy to make this fixture pass.

Serial watchdog fixture checkpoint — 16 September 2026: signed Integration 01daadd3d775079a77dd3278f6b07ecbaebe3e84 replaces the two positive host timing schedules with deterministic child/clock schedules around real serial bytes and the actual watchdog loop. All 13 serial and 21 console tests pass with warnings denied. Four independently selected wrong refresh, exact-expiry, marker and status decisions are rejected. The real subprocess refusal cases and production watchdog code/deadlines are unchanged. The strict default/all-feature development/release Rust matrix, host/native Clippy, private rustdoc, formatting and four native builds pass; the unchanged helper inputs retain their prior coverage. Bug #53 remains open pending matching CI. The original CI1633 failure and earlier incomplete paired-review observations are retained; neither establishes a guest defect or a historical scheduling cause.

First matching watchdog correction run — 16 September 2026: CI 1641, exact 01daadd3d775079a77dd3278f6b07ecbaebe3e84, passes its 13 serial watchdog controls and remaining host checks. The full catalog run then records 407 passes and 79 failures across 486 scenarios: 78 known Exsh layout refusals and one 120-second ext4 quota timeout. This is another recurrence of #18, not evidence that the deterministic fixture correction changed guest timing. #57 still tracks coherent layout adoption. All three terminal logs are complete (13,436,715 bytes), with zero compiler/lint warnings; Rust and Markdown pass. The artifact API exposes no original images or inner receipts. Paired CI 1642 remains running; retain those same handles and original deadlines. No unchanged rerun or full runtime acceptance is claimed.

Further original CI results — 16 September 2026: watchdog CI 1642, exact 01daadd3d775079a77dd3278f6b07ecbaebe3e84, records 408 passes and 78 known layout failures across 486 scenarios; the quota scenario passes in that run. Permission-validation CI 1643, exact 89d15e4536004c087dcebd63c87216c0fd58958d, records 407 passes and 79 failures: the same 78 layout refusals plus the original 120-second quota timeout. Both runs pass their host controls, Rust and Markdown checks. All six terminal logs are complete, totaling 26,457,909 bytes with zero warning candidates. Artifact APIs expose no original images or inner receipts. This preserves the timing regression in #18 and coherent layout adoption in #57 as open obligations. Permission CI 1644 and release-input CI 1645 are running; CI 1646 waits. No run was restarted or assigned a larger deadline.

Remaining original CI results — 17 September 2026: permission CI 1644, exact 89d15e4536004c087dcebd63c87216c0fd58958d, finishes with 408 passing and 78 known layout refusals. Release-input CI 1645, exact df8147dfa437194cd091747fd7147941bd2793c4, finishes with 405 passes and 81 failures: those 78 layout refusals, the original quota timeout, and new 120-second FAT32 mirrored/Unicode timeouts. Paired CI 1646, at the same source revision, records 408 passes and only the 78 layout refusals. The separate FAT32 observations are tracked with the canonical bug template in #58; a common cause with quota is not established.

All nine terminal logs are complete, totaling 40,256,559 bytes with zero warning candidates; host/Rust/Markdown checks pass. Original artifact APIs expose no images or inner receipts. Both formerly pending cohorts are fully classified, with their original failures retained. No run was restarted, no limit was increased and no complete product or guest-build acceptance is asserted.

Tracking

Integration PR #12 coordinates the correction. Earlier console-progress fixture #25 uses a separate verified-console path; a shared historical cause is not established. Keep original failures and close this report only after focused regression checks and matching CI.

Continued same-run observation — 16 September 2026: the original paired CI 1634 is now terminal. Its host fixtures and all 486 catalog scenarios pass before the separate release repeated-interruption failure in issue 54. All three logs are complete (13,299,168 bytes), without warnings. Original CI 1633 still retains its positive serial-fixture failure. Matching correction CI 1641/1642 remains waiting at observation 05; this report remains open.

## Summary The positive serial-progress watchdog host fixture fails in Integration CI 1633, stopping the remaining integration suite before the scenario catalog runs. ## Reproduction Observed once in the retained original CI at the revision below; failure frequency is unknown. The relevant host command is `python3 tests/progress_watchdog_test.py`. Its positive fixture writes the arming marker followed by five appends separated by 100 ms sleeps, under a 250 ms stall deadline and 20 ms polling interval. No unchanged retry was performed. ## Expected and actual behavior Expected: `test_post_arm_progress_refreshes_the_stall_deadline` returns child status 0 with no watchdog error. Actual: the watchdog returns 125 and the assertion reports `125 != 0`. The group runs eight tests with one failure. The terminal log does not retain accepted-progress timestamps or establish host contention as the cause. ## Environment and identity - Original Integration `2a16b1a7ac5514a43582025e5b9ba0a2f2ce2a29`, push CI 1633, integration job 20350, attempt 1. - Host Python fixture in the x86-64 Linux container; this failing test does not execute a guest image or grant guest authority. - The workflow retains its original pinned component graph. Rust and Markdown jobs pass. No last known working revision for this exact observed timing failure has been established. ## Evidence and prior checks [CI 1633](https://git.erikinkinen.fi/erix/integration/actions/runs/1633) reports the failure at 21:25:31 UTC on September 15. All three complete terminal job logs total 245,076 bytes with no compiler/lint warning candidates. Earlier QEMU wrapper watchdog fixtures pass. The paired review run was incomplete at that observation; its later terminal result is recorded below. The fixture currently relies on real host scheduling to keep each positive write within the short deadline. Replace positive timing assertions with a finite logical child/clock schedule around the actual watchdog loop and real serial-file bytes. Preserve the original thresholds, marker detection, status propagation and real negative subprocess tests. Include missing refresh, split marker, exact expiry and interruption/cleanup controls. Do not modify production timing policy to make this fixture pass. Serial watchdog fixture checkpoint — 16 September 2026: signed Integration `01daadd3d775079a77dd3278f6b07ecbaebe3e84` replaces the two positive host timing schedules with deterministic child/clock schedules around real serial bytes and the actual watchdog loop. All 13 serial and 21 console tests pass with warnings denied. Four independently selected wrong refresh, exact-expiry, marker and status decisions are rejected. The real subprocess refusal cases and production watchdog code/deadlines are unchanged. The strict default/all-feature development/release Rust matrix, host/native Clippy, private rustdoc, formatting and four native builds pass; the unchanged helper inputs retain their prior coverage. [Bug #53](https://git.erikinkinen.fi/erix/integration/issues/53) remains open pending matching CI. The original CI1633 failure and earlier incomplete paired-review observations are retained; neither establishes a guest defect or a historical scheduling cause. First matching watchdog correction run — 16 September 2026: [CI 1641](https://git.erikinkinen.fi/erix/integration/actions/runs/1641), exact `01daadd3d775079a77dd3278f6b07ecbaebe3e84`, passes its 13 serial watchdog controls and remaining host checks. The full catalog run then records 407 passes and 79 failures across 486 scenarios: 78 known Exsh layout refusals and one 120-second ext4 quota timeout. This is another recurrence of [#18](https://git.erikinkinen.fi/erix/integration/issues/18), not evidence that the deterministic fixture correction changed guest timing. [#57](https://git.erikinkinen.fi/erix/integration/issues/57) still tracks coherent layout adoption. All three terminal logs are complete (13,436,715 bytes), with zero compiler/lint warnings; Rust and Markdown pass. The artifact API exposes no original images or inner receipts. Paired CI 1642 remains running; retain those same handles and original deadlines. No unchanged rerun or full runtime acceptance is claimed. Further original CI results — 16 September 2026: watchdog [CI 1642](https://git.erikinkinen.fi/erix/integration/actions/runs/1642), exact `01daadd3d775079a77dd3278f6b07ecbaebe3e84`, records 408 passes and 78 known layout failures across 486 scenarios; the quota scenario passes in that run. Permission-validation [CI 1643](https://git.erikinkinen.fi/erix/integration/actions/runs/1643), exact `89d15e4536004c087dcebd63c87216c0fd58958d`, records 407 passes and 79 failures: the same 78 layout refusals plus the original 120-second quota timeout. Both runs pass their host controls, Rust and Markdown checks. All six terminal logs are complete, totaling 26,457,909 bytes with zero warning candidates. Artifact APIs expose no original images or inner receipts. This preserves the timing regression in #18 and coherent layout adoption in #57 as open obligations. Permission CI 1644 and release-input CI 1645 are running; CI 1646 waits. No run was restarted or assigned a larger deadline. Remaining original CI results — 17 September 2026: permission [CI 1644](https://git.erikinkinen.fi/erix/integration/actions/runs/1644), exact `89d15e4536004c087dcebd63c87216c0fd58958d`, finishes with 408 passing and 78 known layout refusals. Release-input [CI 1645](https://git.erikinkinen.fi/erix/integration/actions/runs/1645), exact `df8147dfa437194cd091747fd7147941bd2793c4`, finishes with 405 passes and 81 failures: those 78 layout refusals, the original quota timeout, and new 120-second FAT32 mirrored/Unicode timeouts. Paired [CI 1646](https://git.erikinkinen.fi/erix/integration/actions/runs/1646), at the same source revision, records 408 passes and only the 78 layout refusals. The separate FAT32 observations are tracked with the canonical bug template in [#58](https://git.erikinkinen.fi/erix/integration/issues/58); a common cause with quota is not established. All nine terminal logs are complete, totaling 40,256,559 bytes with zero warning candidates; host/Rust/Markdown checks pass. Original artifact APIs expose no images or inner receipts. Both formerly pending cohorts are fully classified, with their original failures retained. No run was restarted, no limit was increased and no complete product or guest-build acceptance is asserted. ## Tracking [Integration PR #12](https://git.erikinkinen.fi/erix/integration/pulls/12) coordinates the correction. [Earlier console-progress fixture #25](https://git.erikinkinen.fi/erix/integration/issues/25) uses a separate verified-console path; a shared historical cause is not established. Keep original failures and close this report only after focused regression checks and matching CI. Continued same-run observation — 16 September 2026: the original paired CI 1634 is now terminal. Its host fixtures and all 486 catalog scenarios pass before the separate release repeated-interruption failure in [issue 54](https://git.erikinkinen.fi/erix/integration/issues/54). All three logs are complete (13,299,168 bytes), without warnings. Original CI 1633 still retains its positive serial-fixture failure. Matching correction CI 1641/1642 remains waiting at observation 05; this report remains open.

erikinkinen added the

bug

phase-6

labels

2026-09-15 23:54:58 +02:00

~~erikinkinen referenced this issue 2026-09-16 00:01:58 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12

erikinkinen referenced this issue

2026-09-16 00:02:01 +02:00

[FEATURE] Freeze the complete source, build, ABI and self-hosting inventory #1