CI: parallel unoptimized sanitizer cells exceed the per-test timeout #653

Closed
opened 2026-10-21 02:21:25 +02:00 by erikinkinen · 0 comments
Owner

Reproduction

PR #652 at signed head eded66b72c7f95fb54e40bd9d9f4df2f2203acbe
expands the complete suite to 192 tests and requests four concurrent CTest
workers. Both remote Clang 19 Debug ASan/UBSan jobs time out on the first four
128-leaf lineage-swap cells, each at the unchanged 120-second per-test limit.

  • Push run 23823, job 49879: tests 67–70 timed out at 120.09–120.17 seconds.
  • Pull-request run 23826, job 49883: the same tests timed out at 120.07–120.25 seconds.
  • Both jobs report four failures after 66 passing tests; remaining tests did
    not complete. These are failed gates, not successful partial validation.
  • The same implementation passed all 192 tests locally in GCC Debug, GCC
    Release and Clang 22 unoptimized Debug ASan/UBSan. The exact final head also
    passes all 192 locally with -g -O1, both sanitizers and one CTest worker
    (538.57 seconds for tests). Local timing does not establish remote capacity.

The remote jobs show substantially slower cell execution with four workers.
Available logs establish the timeout; they do not establish a CPU quota or a
unique underlying contention cause. Preserve both failure logs and the local
candidate evidence before changing the workflow.

Fix

Configure the sanitizer Debug build with -g -O1, retaining assertions,
AddressSanitizer, UndefinedBehaviorSanitizer, frame pointers, leak detection,
and halt-on-error. Use one explicit CTest worker per job in both complete
suites. Preserve the pinned Clang 19 container/toolchain, 120-second per-test
limit, stop-on-failure and 30-minute overall sanitizer budget. Record CPU count
and cgroup quota when readable for future capacity diagnosis. Document the
profile and its limits; do not claim native scientific performance from CI.

Steps

  • Preserve both exact-head timeout logs and verify the complete local candidate-profile evidence.
  • Apply and document the bounded optimized sanitizer profile with one test worker per job.
  • Validate the workflow, all repository hooks and all eight final-head remote checks.

Definition of done

The separate signed fix PR passes all eight checks on its exact head. Then
integrate the fix into #652, preserve the original acceptance evidence with an
explicit integration record, and require its own complete final-head CI before
checking #103 or merging it. No test, timeout, assertion, workload semantics or
scientific protocol is relaxed.

## Reproduction PR #652 at signed head `eded66b72c7f95fb54e40bd9d9f4df2f2203acbe` expands the complete suite to 192 tests and requests four concurrent CTest workers. Both remote Clang 19 Debug ASan/UBSan jobs time out on the first four 128-leaf lineage-swap cells, each at the unchanged 120-second per-test limit. - Push run 23823, job 49879: tests 67–70 timed out at 120.09–120.17 seconds. - Pull-request run 23826, job 49883: the same tests timed out at 120.07–120.25 seconds. - Both jobs report four failures after 66 passing tests; remaining tests did not complete. These are failed gates, not successful partial validation. - The same implementation passed all 192 tests locally in GCC Debug, GCC Release and Clang 22 unoptimized Debug ASan/UBSan. The exact final head also passes all 192 locally with `-g -O1`, both sanitizers and one CTest worker (538.57 seconds for tests). Local timing does not establish remote capacity. The remote jobs show substantially slower cell execution with four workers. Available logs establish the timeout; they do not establish a CPU quota or a unique underlying contention cause. Preserve both failure logs and the local candidate evidence before changing the workflow. ## Fix Configure the sanitizer Debug build with `-g -O1`, retaining assertions, AddressSanitizer, UndefinedBehaviorSanitizer, frame pointers, leak detection, and halt-on-error. Use one explicit CTest worker per job in both complete suites. Preserve the pinned Clang 19 container/toolchain, 120-second per-test limit, stop-on-failure and 30-minute overall sanitizer budget. Record CPU count and cgroup quota when readable for future capacity diagnosis. Document the profile and its limits; do not claim native scientific performance from CI. ## Steps - [x] Preserve both exact-head timeout logs and verify the complete local candidate-profile evidence. - [x] Apply and document the bounded optimized sanitizer profile with one test worker per job. - [x] Validate the workflow, all repository hooks and all eight final-head remote checks. ## Definition of done The separate signed fix PR passes all eight checks on its exact head. Then integrate the fix into #652, preserve the original acceptance evidence with an explicit integration record, and require its own complete final-head CI before checking #103 or merging it. No test, timeout, assertion, workload semantics or scientific protocol is relaxed.
erikinkinen added this to the Phase 2 milestone 2026-10-21 02:21:25 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
erikinkinen/AES#653
No description provided.