CI: unoptimized 292-test smoke suite exceeds the 90-minute job deadline #661

Closed
opened 2026-10-21 05:50:31 +02:00 by erikinkinen · 0 comments
Owner

Reproduction

PR #657 at 3b13d6ccb61d9277374dbd5b707e64e69674cb14 combines
both witness-generator families in the complete 292-test suite. Both GCC
Debug smoke jobs reached the overall 90-minute deadline:

  • Push run 23865 (UI run 829), job 49941: 190 tests passed before
    context deadline exceeded at 2026-10-21 03:48:23 UTC.
  • PR run 23868 (UI run 832), job 49945: 189 tests passed before
    the same deadline at 2026-10-21 03:48:26 UTC.
  • Neither complete log shows an individual failed test or per-test timeout.
    The unexecuted remainder prevents either attempt from being accepted.
  • Both complete 292-test Clang Debug -g -O1 ASan/UBSan jobs passed, in
    approximately 53 minutes. The largest unoptimized smoke cases took roughly
    85–100 seconds each. These observations do not identify a unique hardware
    or scheduling cause.

The allowance introduced in #655 was validated with 192 tests. It does not
establish capacity for this expanded unoptimized 292-test consumer.

Fix and scope

Configure the GCC smoke Debug build with explicit -g -O1, matching the
optimization level already used for the instrumented Debug suite. Retain
Debug/assertion semantics: no NDEBUG, no test filtering or shortened fixtures,
no changed scientific inputs or expected results. Keep the compiler/container,
complete CTest inventory, one test worker, sanitizer settings and both bounded
90-minute job deadlines. Record the optimization change explicitly.

Check generated compile flags and locally execute the complete consumer suite.
Then require every remote check on both the fix and the expanded consumer;
local timing or the successful sanitizer build alone is not the acceptance gate.

Steps

  • Preserve both complete deadline logs and exact-head status.
  • Implement explicit Debug -g -O1 smoke compilation and verify retained assertions and unchanged complete test inventory.
  • Pass hooks and all required exact-head checks on both the fix and the 292-test consumer before merging the fix.

Definition of done

Use one signed branch and PR for this bug. Integrate the fix candidate into
#657 without changing its witness implementations, preserve the earlier failed
attempts and replace integration provenance explicitly. Require all eight
contexts for both heads before this bug's checklist/merge; #657 additionally
requires its native prerequisites closed. Reconcile all stacked descendants
to the new predecessor hashes and require their full final checks. No checksum,
previous success or cancelled old attempt is substituted for a final-head gate.

## Reproduction PR #657 at `3b13d6ccb61d9277374dbd5b707e64e69674cb14` combines both witness-generator families in the complete 292-test suite. Both GCC Debug smoke jobs reached the overall 90-minute deadline: - Push run 23865 (UI run 829), job 49941: 190 tests passed before `context deadline exceeded` at 2026-10-21 03:48:23 UTC. - PR run 23868 (UI run 832), job 49945: 189 tests passed before the same deadline at 2026-10-21 03:48:26 UTC. - Neither complete log shows an individual failed test or per-test timeout. The unexecuted remainder prevents either attempt from being accepted. - Both complete 292-test Clang Debug `-g -O1` ASan/UBSan jobs passed, in approximately 53 minutes. The largest unoptimized smoke cases took roughly 85–100 seconds each. These observations do not identify a unique hardware or scheduling cause. The allowance introduced in #655 was validated with 192 tests. It does not establish capacity for this expanded unoptimized 292-test consumer. ## Fix and scope Configure the GCC smoke Debug build with explicit `-g -O1`, matching the optimization level already used for the instrumented Debug suite. Retain Debug/assertion semantics: no `NDEBUG`, no test filtering or shortened fixtures, no changed scientific inputs or expected results. Keep the compiler/container, complete CTest inventory, one test worker, sanitizer settings and both bounded 90-minute job deadlines. Record the optimization change explicitly. Check generated compile flags and locally execute the complete consumer suite. Then require every remote check on both the fix and the expanded consumer; local timing or the successful sanitizer build alone is not the acceptance gate. ## Steps - [x] Preserve both complete deadline logs and exact-head status. - [x] Implement explicit Debug `-g -O1` smoke compilation and verify retained assertions and unchanged complete test inventory. - [x] Pass hooks and all required exact-head checks on both the fix and the 292-test consumer before merging the fix. ## Definition of done Use one signed branch and PR for this bug. Integrate the fix candidate into #657 without changing its witness implementations, preserve the earlier failed attempts and replace integration provenance explicitly. Require all eight contexts for both heads before this bug's checklist/merge; #657 additionally requires its native prerequisites closed. Reconcile all stacked descendants to the new predecessor hashes and require their full final checks. No checksum, previous success or cancelled old attempt is substituted for a final-head gate.
erikinkinen added this to the Phase 2 milestone 2026-10-21 05:50:31 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
erikinkinen/AES#661
No description provided.