CI: full Phase 2 sanitizer suite exceeds the overall 30-minute job budget #655

Closed
opened 2026-10-21 03:16:45 +02:00 by erikinkinen · 0 comments
Owner

Reproduction

PR #652 at f9b114fa0d5d3d9ce1530d7941af61083bd9c0fb integrates
accepted #653, running all 192 tests with Clang 19 Debug -g -O1, ASan/UBSan
and one CTest worker. Both sanitizer jobs hit the overall job deadline:

  • Push run 23841, job 49910: 119 passing tests, followed by runner
    context deadline exceeded at 2026-10-21 01:12:31 UTC.
  • PR run 23844, job 49914: 120 passing tests, followed by the same deadline
    at 2026-10-21 01:12:30 UTC.
  • No individual test failure or 120-second test timeout appears in these
    attempts. Remaining tests were not executed, so neither run is accepted.
  • The expensive 128-leaf cases now pass individually, approximately 25 seconds
    each under sanitizers. Cold setup/build and the enlarged sequential suite
    together exceed the 30-minute allowance. Smoke cases take approximately
    54–58 seconds each. These observations do not identify a unique slowdown cause.

This is a separate overall-capacity regression from #653's per-test timeouts.
Retain complete logs and exact-head status before changing the workflow.

Fix and scope

Set an explicit 90-minute overall budget for both full-suite CI jobs. The
observed 192-test run already needs more than 30 minutes; the prepared second
100-test generator family adds comparable work. Ninety minutes is a bounded
engineering allowance for these complete sequential suites and cold builds,
not an empirical runtime guarantee. Keep the complete inventory, one CTest
worker, compiler/container, -g -O1, assertions, ASan/UBSan and the sanitizer
120-second per-test timeout unchanged. No test filtering, retries, disabled
checks, scientific changes or shortened workloads are allowed.

Steps

  • Preserve both deadline logs and exact-head status; verify the failure is the overall job deadline.
  • Set and document the bounded 90-minute full-job budget without changing test selection or instrumentation.
  • Pass hooks and all eight exact-head checks for this fix and for the 192-test consumer before merging the fix.

Definition of done

Use one signed branch and PR for this bug. Integrate its final candidate into
#652 with explicit source/provenance checks, and run that expanded consumer's
full remote checks. Require all eight checks on both heads before checking this
bug or merging its fix. Then require #652's closed prerequisites and exact-head
checks before checking #103 and merging. Preserve original failed evidence and
new successful logs. The baseline 92-test checks alone do not demonstrate
expanded-suite capacity. Any further real failure remains a failed gate and
receives a separate diagnosis/issue.

## Reproduction PR #652 at `f9b114fa0d5d3d9ce1530d7941af61083bd9c0fb` integrates accepted #653, running all 192 tests with Clang 19 Debug `-g -O1`, ASan/UBSan and one CTest worker. Both sanitizer jobs hit the overall job deadline: - Push run 23841, job 49910: 119 passing tests, followed by runner `context deadline exceeded` at 2026-10-21 01:12:31 UTC. - PR run 23844, job 49914: 120 passing tests, followed by the same deadline at 2026-10-21 01:12:30 UTC. - No individual test failure or 120-second test timeout appears in these attempts. Remaining tests were not executed, so neither run is accepted. - The expensive 128-leaf cases now pass individually, approximately 25 seconds each under sanitizers. Cold setup/build and the enlarged sequential suite together exceed the 30-minute allowance. Smoke cases take approximately 54–58 seconds each. These observations do not identify a unique slowdown cause. This is a separate overall-capacity regression from #653's per-test timeouts. Retain complete logs and exact-head status before changing the workflow. ## Fix and scope Set an explicit 90-minute overall budget for both full-suite CI jobs. The observed 192-test run already needs more than 30 minutes; the prepared second 100-test generator family adds comparable work. Ninety minutes is a bounded engineering allowance for these complete sequential suites and cold builds, not an empirical runtime guarantee. Keep the complete inventory, one CTest worker, compiler/container, `-g -O1`, assertions, ASan/UBSan and the sanitizer 120-second per-test timeout unchanged. No test filtering, retries, disabled checks, scientific changes or shortened workloads are allowed. ## Steps - [x] Preserve both deadline logs and exact-head status; verify the failure is the overall job deadline. - [x] Set and document the bounded 90-minute full-job budget without changing test selection or instrumentation. - [x] Pass hooks and all eight exact-head checks for this fix and for the 192-test consumer before merging the fix. ## Definition of done Use one signed branch and PR for this bug. Integrate its final candidate into #652 with explicit source/provenance checks, and run that expanded consumer's full remote checks. Require all eight checks on both heads before checking this bug or merging its fix. Then require #652's closed prerequisites and exact-head checks before checking #103 and merging. Preserve original failed evidence and new successful logs. The baseline 92-test checks alone do not demonstrate expanded-suite capacity. Any further real failure remains a failed gate and receives a separate diagnosis/issue.
erikinkinen added this to the Phase 2 milestone 2026-10-21 03:16:45 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
erikinkinen/AES#655
No description provided.