CI: parallel unoptimized sanitizer cells exceed the per-test timeout #653
Labels
No labels
bug
cli
core
docs
event
experiment
figure
invariant
metrics
oracle
phase-0
phase-1
phase-2
phase-3
phase-4
phase-5
phase-6
provenance
revocation
tests
workload
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Blocks
#103 Phase 2: Generate lineage-swap paired tapes
erikinkinen/AES
Reference
erikinkinen/AES#653
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Reproduction
PR #652 at signed head
eded66b72c7f95fb54e40bd9d9f4df2f2203acbeexpands the complete suite to 192 tests and requests four concurrent CTest
workers. Both remote Clang 19 Debug ASan/UBSan jobs time out on the first four
128-leaf lineage-swap cells, each at the unchanged 120-second per-test limit.
not complete. These are failed gates, not successful partial validation.
Release and Clang 22 unoptimized Debug ASan/UBSan. The exact final head also
passes all 192 locally with
-g -O1, both sanitizers and one CTest worker(538.57 seconds for tests). Local timing does not establish remote capacity.
The remote jobs show substantially slower cell execution with four workers.
Available logs establish the timeout; they do not establish a CPU quota or a
unique underlying contention cause. Preserve both failure logs and the local
candidate evidence before changing the workflow.
Fix
Configure the sanitizer Debug build with
-g -O1, retaining assertions,AddressSanitizer, UndefinedBehaviorSanitizer, frame pointers, leak detection,
and halt-on-error. Use one explicit CTest worker per job in both complete
suites. Preserve the pinned Clang 19 container/toolchain, 120-second per-test
limit, stop-on-failure and 30-minute overall sanitizer budget. Record CPU count
and cgroup quota when readable for future capacity diagnosis. Document the
profile and its limits; do not claim native scientific performance from CI.
Steps
Definition of done
The separate signed fix PR passes all eight checks on its exact head. Then
integrate the fix into #652, preserve the original acceptance evidence with an
explicit integration record, and require its own complete final-head CI before
checking #103 or merging it. No test, timeout, assertion, workload semantics or
scientific protocol is relaxed.