[BUG] Sequential scenario checks can encounter an active-run lock after success #16

New issue

Open

opened 2026-09-12 10:54:56 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-12 10:54:56 +02:00

Owner

Summary

A sequence of scenario-runner regression checks stopped with an active-run lock error immediately after the preceding scenario had reported success. This prevents the rest of the Integration regression workflow from running.

Reproduction

Observed once in Integration push CI run 1559 at e7995dc7a2cb32bcc13a923c473e1d5c0e731303, during bash tests/run-scenario-bootloader-negative-timeout.sh. The fixture invokes the actual scenario runner sequentially in its own temporary harness. Two default-timeout invocations passed; the next invocation exited before executing its scenario.

A deterministic host fixture now holds the actual catalog producer after it emits rows. Original runner bytes return success before that producer exits; a following invocation receives exit 75 while the helper retains the lock. The same original runner also hides producer exit 37 after plausible output. This establishes a matching defect, while the exact historical CI process schedule remains unrecorded.

Expected and actual behavior

Each completed invocation should leave the sequential fixture able to acquire its run lock. Concurrent invocations must still be excluded.

Actual result: the runner reported error: another integration scenario run is active and exited 75. The workflow stopped before the complete VM catalog. There was no reported VM timeout in this fixture; the fixture uses original test stubs for its bootloader command.

Environment and identity

Integration revision: e7995dc7a2cb32bcc13a923c473e1d5c0e731303, feature/posix-compat.
CI run: push 1559, Integration job; Linux shell fixture with a private temporary harness.
Image, firmware and guest execution: not applicable to this failing fixture.
Exact runner/kernel/tool versions at the failure: not yet established from the retained log.
Input and synchronization: sequential scenario invocations and the maintained scenario lock; exclusion must remain enforced.
Last known successful workflow: main run 1554 at c86c0d89; this does not establish the first faulty revision or rule out a pre-existing race.

Evidence and prior checks

Failing CI run 1559 reports two successful default-timeout scenarios, the lock error and exit 75. The complete retained job log contains no build warnings. Other intentional negative-fixture errors are distinct from this terminal failure. Companion Rust and Markdown jobs passed.

The correction at signed revision 2ae6c9cd34cd456c568e5f61976879902b73d9f6 synchronously captures the complete catalog and propagates producer status before lookup. It preserves standalone/group lock ownership and concurrent exclusion. Four new tests pass; exact original runner bytes fail three of them. Twelve focused existing runner/schema/worker and syntax/documentation checks pass without warnings. Test cleanup uses bounded owned-child reaping. Full new-head CI is queued and remains required; no lock bypass, cancellation, production delay or retry is used.

Tracking

Related work: #2, #11 and WIP PR #12. Acceptance requires a deterministic regression protecting lock lifetime and concurrent exclusion, a reviewed correction preserving cleanup/error behavior, warning-free relevant checks and full exact-revision CI. Keep previous failing evidence and unexecuted regressions explicit.

## Summary A sequence of scenario-runner regression checks stopped with an active-run lock error immediately after the preceding scenario had reported success. This prevents the rest of the Integration regression workflow from running. ## Reproduction Observed once in Integration push CI run 1559 at `e7995dc7a2cb32bcc13a923c473e1d5c0e731303`, during `bash tests/run-scenario-bootloader-negative-timeout.sh`. The fixture invokes the actual scenario runner sequentially in its own temporary harness. Two default-timeout invocations passed; the next invocation exited before executing its scenario. A deterministic host fixture now holds the actual catalog producer after it emits rows. Original runner bytes return success before that producer exits; a following invocation receives exit 75 while the helper retains the lock. The same original runner also hides producer exit 37 after plausible output. This establishes a matching defect, while the exact historical CI process schedule remains unrecorded. ## Expected and actual behavior Each completed invocation should leave the sequential fixture able to acquire its run lock. Concurrent invocations must still be excluded. Actual result: the runner reported `error: another integration scenario run is active` and exited 75. The workflow stopped before the complete VM catalog. There was no reported VM timeout in this fixture; the fixture uses original test stubs for its bootloader command. ## Environment and identity - Integration revision: `e7995dc7a2cb32bcc13a923c473e1d5c0e731303`, `feature/posix-compat`. - CI run: push 1559, Integration job; Linux shell fixture with a private temporary harness. - Image, firmware and guest execution: not applicable to this failing fixture. - Exact runner/kernel/tool versions at the failure: not yet established from the retained log. - Input and synchronization: sequential scenario invocations and the maintained scenario lock; exclusion must remain enforced. - Last known successful workflow: main run 1554 at `c86c0d89`; this does not establish the first faulty revision or rule out a pre-existing race. ## Evidence and prior checks [Failing CI run 1559](https://git.erikinkinen.fi/erix/integration/actions/runs/1559) reports two successful default-timeout scenarios, the lock error and exit 75. The complete retained job log contains no build warnings. Other intentional negative-fixture errors are distinct from this terminal failure. Companion Rust and Markdown jobs passed. The correction at signed revision `2ae6c9cd34cd456c568e5f61976879902b73d9f6` synchronously captures the complete catalog and propagates producer status before lookup. It preserves standalone/group lock ownership and concurrent exclusion. Four new tests pass; exact original runner bytes fail three of them. Twelve focused existing runner/schema/worker and syntax/documentation checks pass without warnings. Test cleanup uses bounded owned-child reaping. Full new-head CI is queued and remains required; no lock bypass, cancellation, production delay or retry is used. ## Tracking Related work: #2, #11 and WIP PR #12. Acceptance requires a deterministic regression protecting lock lifetime and concurrent exclusion, a reviewed correction preserving cleanup/error behavior, warning-free relevant checks and full exact-revision CI. Keep previous failing evidence and unexecuted regressions explicit.

erikinkinen added the

bug

phase-6

labels

2026-09-12 10:54:56 +02:00

~~erikinkinen referenced this issue 2026-09-12 11:24:43 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12