[BUG] Parallel cleanup tests race on shared quarantine state #4

New issue

Closed

opened 2026-09-19 22:55:23 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-19 22:55:23 +02:00

Owner

Summary

Deviced's host cleanup tests share the global service-driver quarantine flag without serializing reset, operation and observation. Parallel test execution can report a spurious failure in successful_cleanup_preserves_the_operation_error. This is a test-fixture race; no native service failure is established.

Reproduction

Run cargo +1.97.1 test --locked with the default parallel test harness during the coordinated explicit install-grant update. One retained local run failed; no unchanged retry was used. The original cleanup implementation and both tests match the published base. Clean-checkout reproduction and failure frequency are not yet established.

The conflicting tests are cleanup_failure_quarantines_the_service and successful_cleanup_preserves_the_operation_error. Both reset the same atomic flag; the first sets it while the second expects it to remain clear. Atomic loads/stores do not serialize those multi-operation assertions.

Expected and actual behavior

Expected: each test observes its own quarantine transition independently, and successful cleanup preserves the original operation error without setting quarantine.

Actual: 128 tests passed and one failed. successful_cleanup_preserves_the_operation_error panicked at src/catalog_service_driver/cleanup.rs:183 on assert!(!service_driver_cleanup_quarantined()); Cargo exited 101. The error-result assertion preceding it passed.

Environment and identity

Component: Deviced, published base 2a387fa1ef0bc06fb68217b5003cf8efd65dc522, feature branch with coordinated grant-receipt and original dependency-pin updates under development.
Configuration: ordinary host unit tests; no image or guest execution.
Architecture/toolchain: x86_64 Linux, Rust 1.97.1, warnings denied, default parallel test harness.
Authority: test-only access to the component's shared quarantine atomic; no process authority or runtime quarantine relaxation is proposed.
Last known working revision: not established for this nondeterministic fixture defect.

Evidence and prior checks

The complete failing stdout, stderr, true exit status and exact dependency metadata are retained. Inspection identifies an admissible interleaving between the two unchanged tests, but the failing run did not record a thread schedule. The correction will serialize only tests touching the shared flag, preserve all assertions, and rerun the affected unit and strict validation matrices. No workflow retry, timeout increase or test exclusion is proposed.

Tracking

Related: component audit and phase completion. Acceptance requires all affected test configurations to pass with the shared-state controls retained; native quarantine behavior remains unchanged.

Verified grant-rights checkpoint — 20 September 2026:

Signed commit 2c60b5d607997e671601044ae0f47dd0d8f69664 requires exact GRANT-only final installer receipts and selects the original shared dependency graph. Four strict 135/136-unit configurations, four native builds with the maintained linker layout, host/native Clippy, formatting and private rustdoc pass without warnings. Original CI 206/207 passes from four complete hashed logs (210,344 bytes), with zero warning candidates. Bug 4 is corrected by test-only serialization of the shared quarantine flag; production fail-stop semantics are unchanged. The original failure is retained.

Both maintained isolated native scenarios pass on their first attempts under the unchanged 60-second scenario limits, with no build warnings and empty QEMU stderr. Lifetime now exercises 27 ordinary CPL3 grant-right controls and requires INSTALL_GRANT_RIGHTS_OK before its existing cleanup assertions. Its 2,025-byte serial stream has SHA256 6c685f1ceaf9080bf0bec628a4fac512bf9049d0fbcfe2b3737666adf414ca3a; owned invocation retains 1,587 bytes. Normal stripping exactly matches both packaged kernels to retained original artifacts. All fifteen selected original source signatures and clean trees verify. These minimal native scenarios establish neither full service-image acceptance nor a guest build.

Full coordinated consumer acceptance remains open under Kernel design 19 and phase completion.

## Summary Deviced's host cleanup tests share the global service-driver quarantine flag without serializing reset, operation and observation. Parallel test execution can report a spurious failure in `successful_cleanup_preserves_the_operation_error`. This is a test-fixture race; no native service failure is established. ## Reproduction Run `cargo +1.97.1 test --locked` with the default parallel test harness during the coordinated explicit install-grant update. One retained local run failed; no unchanged retry was used. The original cleanup implementation and both tests match the published base. Clean-checkout reproduction and failure frequency are not yet established. The conflicting tests are `cleanup_failure_quarantines_the_service` and `successful_cleanup_preserves_the_operation_error`. Both reset the same atomic flag; the first sets it while the second expects it to remain clear. Atomic loads/stores do not serialize those multi-operation assertions. ## Expected and actual behavior Expected: each test observes its own quarantine transition independently, and successful cleanup preserves the original operation error without setting quarantine. Actual: 128 tests passed and one failed. `successful_cleanup_preserves_the_operation_error` panicked at `src/catalog_service_driver/cleanup.rs:183` on `assert!(!service_driver_cleanup_quarantined())`; Cargo exited 101. The error-result assertion preceding it passed. ## Environment and identity - Component: Deviced, published base `2a387fa1ef0bc06fb68217b5003cf8efd65dc522`, feature branch with coordinated grant-receipt and original dependency-pin updates under development. - Configuration: ordinary host unit tests; no image or guest execution. - Architecture/toolchain: x86_64 Linux, Rust 1.97.1, warnings denied, default parallel test harness. - Authority: test-only access to the component's shared quarantine atomic; no process authority or runtime quarantine relaxation is proposed. - Last known working revision: not established for this nondeterministic fixture defect. ## Evidence and prior checks The complete failing stdout, stderr, true exit status and exact dependency metadata are retained. Inspection identifies an admissible interleaving between the two unchanged tests, but the failing run did not record a thread schedule. The correction will serialize only tests touching the shared flag, preserve all assertions, and rerun the affected unit and strict validation matrices. No workflow retry, timeout increase or test exclusion is proposed. ## Tracking Related: [component audit](https://git.erikinkinen.fi/erix/deviced/issues/1) and [phase completion](https://git.erikinkinen.fi/erix/integration/issues/65). Acceptance requires all affected test configurations to pass with the shared-state controls retained; native quarantine behavior remains unchanged. Verified grant-rights checkpoint — 20 September 2026: Signed commit [2c60b5d607997e671601044ae0f47dd0d8f69664](https://git.erikinkinen.fi/erix/deviced/commit/2c60b5d607997e671601044ae0f47dd0d8f69664) requires exact GRANT-only final installer receipts and selects the original shared dependency graph. Four strict 135/136-unit configurations, four native builds with the maintained linker layout, host/native Clippy, formatting and private rustdoc pass without warnings. Original CI 206/207 passes from four complete hashed logs (210,344 bytes), with zero warning candidates. [Bug 4](https://git.erikinkinen.fi/erix/deviced/issues/4) is corrected by test-only serialization of the shared quarantine flag; production fail-stop semantics are unchanged. The original failure is retained. Both maintained isolated native scenarios pass on their first attempts under the unchanged 60-second scenario limits, with no build warnings and empty QEMU stderr. Lifetime now exercises 27 ordinary CPL3 grant-right controls and requires INSTALL_GRANT_RIGHTS_OK before its existing cleanup assertions. Its 2,025-byte serial stream has SHA256 `6c685f1ceaf9080bf0bec628a4fac512bf9049d0fbcfe2b3737666adf414ca3a`; owned invocation retains 1,587 bytes. Normal stripping exactly matches both packaged kernels to retained original artifacts. All fifteen selected original source signatures and clean trees verify. These minimal native scenarios establish neither full service-image acceptance nor a guest build. Full coordinated consumer acceptance remains open under [Kernel design 19](https://git.erikinkinen.fi/erix/kernel/issues/19) and [phase completion](https://git.erikinkinen.fi/erix/integration/issues/65).

erikinkinen added the

bug

phase-6

labels

2026-09-19 22:55:23 +02:00

erikinkinen referenced this issue from erix/kernel

2026-09-19 23:04:17 +02:00

[FEATURE] Bind staged child cleanup to its native supervisor lifetime #19

erikinkinen referenced this issue from erix/kernel

2026-09-19 23:04:17 +02:00

[FEATURE] Supply authorized memory, TLS, threads and race-free blocking for runtimes #1

erikinkinen referenced this issue from erix/kernel