Expanded 496/497-test sanitizer suites exceed aggregate CI deadline #670

Closed
opened 2026-10-30 04:26:50 +01:00 by erikinkinen · 0 comments
Owner

Reproduced failure

PRs #667, #668 and #669 each exhausted the 90-minute sanitizer job deadline in both push and pull-request jobs. Runs 941/945 completed 279/280 of 496 tests; runs 949/953 completed 284/288 of 497; runs 969/970 completed 373/373 of 497. All completed tests passed. The six preserved full logs contain no ASan/LSan/UBSan finding or individual CTest timeout. This establishes an aggregate deadline failure, not a unique hardware cause or a successful full suite.

Required correction

Increase only the sanitizer job overall timeout to 180 minutes. Keep the pinned image, Clang 19, assertion-enabled Debug -g -O1, ASan/UBSan options, complete inventory, one CTest worker, build parallelism four, 120-second per-test timeout and stop-on-failure unchanged. Retain the smoke budget at 90 minutes. Document the new bound and original failures.

Acceptance

  • Preserve the six failed-run identities and log hashes; verify the workflow diff changes only the sanitizer overall timeout and documentation.
  • Validate the complete expanded consumer locally with the pinned CI container and unchanged sanitizer settings; preserve commands, environment, inventory and results. A timeout or finding remains failure.
  • Require all ten exact-head remote contexts on both the fix and the expanded consumer before merging the fix; then require each dependent PR's own final checks and predecessor acceptance. Preserve failed attempts; check each item once.

This is CI capacity qualification, not scientific runtime evidence. No scientific, source-profile or workload acceptance is inferred from a budget change. The original source validation evidence remains immutable.

<!-- aes-bug-key: expanded-497-sanitizer-job-budget --> ## Reproduced failure PRs #667, #668 and #669 each exhausted the 90-minute sanitizer job deadline in both push and pull-request jobs. Runs 941/945 completed 279/280 of 496 tests; runs 949/953 completed 284/288 of 497; runs 969/970 completed 373/373 of 497. All completed tests passed. The six preserved full logs contain no ASan/LSan/UBSan finding or individual CTest timeout. This establishes an aggregate deadline failure, not a unique hardware cause or a successful full suite. ## Required correction Increase only the sanitizer job overall timeout to 180 minutes. Keep the pinned image, Clang 19, assertion-enabled Debug -g -O1, ASan/UBSan options, complete inventory, one CTest worker, build parallelism four, 120-second per-test timeout and stop-on-failure unchanged. Retain the smoke budget at 90 minutes. Document the new bound and original failures. ## Acceptance - [x] Preserve the six failed-run identities and log hashes; verify the workflow diff changes only the sanitizer overall timeout and documentation. - [x] Validate the complete expanded consumer locally with the pinned CI container and unchanged sanitizer settings; preserve commands, environment, inventory and results. A timeout or finding remains failure. - [x] Require all ten exact-head remote contexts on both the fix and the expanded consumer before merging the fix; then require each dependent PR's own final checks and predecessor acceptance. Preserve failed attempts; check each item once. This is CI capacity qualification, not scientific runtime evidence. No scientific, source-profile or workload acceptance is inferred from a budget change. The original source validation evidence remains immutable.
erikinkinen added this to the Phase 2 milestone 2026-10-30 04:26:50 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
erikinkinen/AES#670
No description provided.