[BUG] Host tests exhaust the fixed CI deadline before the final modules #175

Closed
opened 2026-10-21 00:55:14 +02:00 by erikinkinen · 4 comments
Owner

Summary

The host test namespace has grown to 199 modules. Current CI reaches its fixed
1,200-second limit before the final modules finish, so VM checks never start.

Reproduction

Run the Integration push workflow at 6ec5c74542a7aa4a3a859a3c1946d7a099f85d57.
CI run 2942
expires during vm_failure_diagnostics_test.py.

Expected and actual behavior

Every maintained host module should finish under an original process owner
before VM checks start. Instead, run-python-tests.py reports that its hard or
stall deadline expired. The console timestamps match the 1,200-second total
limit; they do not establish a failed assertion in the interrupted module.

Environment and identity

Forgejo push workflow, Linux x86-64 container and the revision above. Each
module has a fresh Python interpreter, a 600-second maximum and a 90-second
stall interval; Cargo runs with one job.

Evidence and prior checks

The complete first CI log is retained. A separate local run completed all 199
modules; its socket fixture error is tracked in #174.
The CI log records about 174 seconds for Rust documentation policy checks,
85 seconds for scenario locking and 65 seconds for live-output tests. These are
elapsed observations, not CPU profiles. The live-output test deliberately runs
beyond sixty seconds and must retain that coverage.

Tracking

Split host checks into two deterministic CI shards with separate containers
and original process owners. Keep the existing test assertions, output bounds,
per-module limits and cleanup rules. Require both complete shards before the
single VM job, and schedule them after Rust checks to bound compiler overlap.
Related work: PR 12.

  • Retain the original deadline failure and its exact source.
  • Freeze the complete module inventory and prove disjoint, complete assignments.
  • Preserve unsharded use, original ownership and all existing test limits.
  • Pass both host shards and let the affected workflow reach VM checks.
    • Pass both complete host shards and the downstream coverage check.
    • Reach VM checks after the remaining helper checks.
## Summary The host test namespace has grown to 199 modules. Current CI reaches its fixed 1,200-second limit before the final modules finish, so VM checks never start. ## Reproduction Run the Integration push workflow at `6ec5c74542a7aa4a3a859a3c1946d7a099f85d57`. [CI run 2942](https://git.erikinkinen.fi/erix/integration/actions/runs/2942) expires during `vm_failure_diagnostics_test.py`. ## Expected and actual behavior Every maintained host module should finish under an original process owner before VM checks start. Instead, `run-python-tests.py` reports that its hard or stall deadline expired. The console timestamps match the 1,200-second total limit; they do not establish a failed assertion in the interrupted module. ## Environment and identity Forgejo push workflow, Linux x86-64 container and the revision above. Each module has a fresh Python interpreter, a 600-second maximum and a 90-second stall interval; Cargo runs with one job. ## Evidence and prior checks The complete first CI log is retained. A separate local run completed all 199 modules; its socket fixture error is tracked in [#174](https://git.erikinkinen.fi/erix/integration/issues/174). The CI log records about 174 seconds for Rust documentation policy checks, 85 seconds for scenario locking and 65 seconds for live-output tests. These are elapsed observations, not CPU profiles. The live-output test deliberately runs beyond sixty seconds and must retain that coverage. ## Tracking Split host checks into two deterministic CI shards with separate containers and original process owners. Keep the existing test assertions, output bounds, per-module limits and cleanup rules. Require both complete shards before the single VM job, and schedule them after Rust checks to bound compiler overlap. Related work: [PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12). - [x] Retain the original deadline failure and its exact source. - [x] Freeze the complete module inventory and prove disjoint, complete assignments. - [x] Preserve unsharded use, original ownership and all existing test limits. - [x] Pass both host shards and let the affected workflow reach VM checks. - [x] Pass both complete host shards and the downstream coverage check. - [x] Reach VM checks after the remaining helper checks.
Author
Owner

The host-suite split is pushed in 355487ac, with the failure fixtures corrected in 8dcca3f2.

Two separate jobs keep the existing 1,200-second suite, 600-second module and 90-second silence limits. The VM job checks that their source manifests agree and that every module ran exactly once. The 25 focused controls pass; complete runs of the 101-module and 100-module shards are still pending. This issue remains open.

The host-suite split is pushed in [355487ac](https://git.erikinkinen.fi/erix/integration/commit/355487acec5c611175c7f7338ed8e3bccb3ee594), with the failure fixtures corrected in [8dcca3f2](https://git.erikinkinen.fi/erix/integration/commit/8dcca3f296330c1140bf4fefbea2c9475c92d32e). Two separate jobs keep the existing 1,200-second suite, 600-second module and 90-second silence limits. The VM job checks that their source manifests agree and that every module ran exactly once. The 25 focused controls pass; complete runs of the 101-module and 100-module shards are still pending. This issue remains open.
Author
Owner

CI 2951 passed both host jobs: 101 modules / 1,222 tests and 100 modules / 997 tests. The integration job accepted their matching manifests and complete coverage of 201 modules / 2,219 tests. The existing deadlines are unchanged.

A later helper check stopped VM entry; that separate regression is tracked in #177. Its fix is pushed, and CI 2954 is running. The remaining VM-entry criterion stays open.

[CI 2951](https://git.erikinkinen.fi/erix/integration/actions/runs/2951) passed both host jobs: 101 modules / 1,222 tests and 100 modules / 997 tests. The integration job accepted their matching manifests and complete coverage of **201 modules / 2,219 tests**. The existing deadlines are unchanged. A later helper check stopped VM entry; that separate regression is tracked in [#177](https://git.erikinkinen.fi/erix/integration/issues/177). Its fix is pushed, and [CI 2954](https://git.erikinkinen.fi/erix/integration/actions/runs/2954) is running. The remaining VM-entry criterion stays open.
Author
Owner

The complete 201-module/2,219-case host result from CI 2951 remains confirmed. The later CI 2954 completed both module lists but shard one failed because the cancellation test required SIGTERM where the detached child had validly been retired with SIGKILL.

That assertion and a forced-retirement control are fixed in 5eed43c1; see #179. The VM-entry criterion remains open alongside the source-layout repair and catalog selection.

The complete 201-module/2,219-case host result from CI 2951 remains confirmed. The later CI 2954 completed both module lists but shard one failed because the cancellation test required SIGTERM where the detached child had validly been retired with SIGKILL. That assertion and a forced-retirement control are fixed in [5eed43c1](https://git.erikinkinen.fi/erix/integration/commit/5eed43c1c0931740f1fe7719eaacdffa3a261445); see [#179](https://git.erikinkinen.fi/erix/integration/issues/179). The VM-entry criterion remains open alongside [the source-layout repair](https://git.erikinkinen.fi/erix/integration/issues/177) and [catalog selection](https://git.erikinkinen.fi/erix/integration/issues/178).
Author
Owner

CI 2963 passes both complete host shards: 201 modules and 2,229 cases, including the downstream coverage check. Its integration job also passes all helper checks and reaches Run complete VM scenario catalog. The host-suite split now meets this issue’s remaining acceptance criterion; the VM catalog itself is still running.

[CI 2963](https://git.erikinkinen.fi/erix/integration/actions/runs/2963) passes both complete host shards: 201 modules and 2,229 cases, including the downstream coverage check. Its integration job also passes all helper checks and reaches `Run complete VM scenario catalog`. The host-suite split now meets this issue’s remaining acceptance criterion; the VM catalog itself is still running.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:34:20 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/integration#175
No description provided.