CI: sanitizer workflow deadline truncates the expanded passing suite #649

Closed
opened 2026-10-21 00:30:20 +02:00 by erikinkinen · 0 comments
Owner

Reproduction

PR #648 at signed head 885bb62f8037be033b57b9eebae173656b51e416
expanded the complete suite to 91 tests. Both sanitizer jobs hit the existing
15-minute workflow deadline on 2026-10-20, after passing tests 1–80 and while
running test 81 (aes_phase1_figures_tests). No test assertion failed.

  • Push run 23778, sanitizer job 49807: started 22:13:14 UTC and was cancelled
    at 22:28:14 UTC with context deadline exceeded.
  • Pull-request run 23781, sanitizer job 49811: the same deadline and test boundary.
  • Both independent compressed-artifact reference checks passed before timeout.
  • The same implementation source passed all 91 tests locally in Debug, Release
    and Clang ASan/UBSan. Remote smoke jobs also completed successfully.

The cold build, dependency setup and full instrumented suite share one deadline.
The expanded suite now exhausts it under the normal concurrent push/PR jobs.
This is a CI capacity regression; successful partial output is not a passing gate.

Fix

Use a bounded 30-minute sanitizer job budget, preserving the 120-second per-test
timeout, stop-on-failure, full unfiltered suite, sanitizer settings, compiler pin,
and all eight required checks. Document the reason and preserve both failure logs.
No modeled semantics, data protocol, source tests or assertion thresholds change.

Steps

  • Preserve both exact-head timeout logs and local complete-suite evidence references.
  • Increase the bounded job budget and document its distinction from the per-test limit.
  • Validate the workflow change, all repository hooks and all eight final-head CI checks.

Definition of done

The independently signed fix PR passes its full CI suite. P2-09 remains blocked
until this fix is merged, integrated and its own final-head checks pass; its
checklist must not be completed on the timed-out head.

## Reproduction PR #648 at signed head `885bb62f8037be033b57b9eebae173656b51e416` expanded the complete suite to 91 tests. Both sanitizer jobs hit the existing 15-minute workflow deadline on 2026-10-20, after passing tests 1–80 and while running test 81 (`aes_phase1_figures_tests`). No test assertion failed. - Push run 23778, sanitizer job 49807: started 22:13:14 UTC and was cancelled at 22:28:14 UTC with `context deadline exceeded`. - Pull-request run 23781, sanitizer job 49811: the same deadline and test boundary. - Both independent compressed-artifact reference checks passed before timeout. - The same implementation source passed all 91 tests locally in Debug, Release and Clang ASan/UBSan. Remote smoke jobs also completed successfully. The cold build, dependency setup and full instrumented suite share one deadline. The expanded suite now exhausts it under the normal concurrent push/PR jobs. This is a CI capacity regression; successful partial output is not a passing gate. ## Fix Use a bounded 30-minute sanitizer job budget, preserving the 120-second per-test timeout, stop-on-failure, full unfiltered suite, sanitizer settings, compiler pin, and all eight required checks. Document the reason and preserve both failure logs. No modeled semantics, data protocol, source tests or assertion thresholds change. ## Steps - [x] Preserve both exact-head timeout logs and local complete-suite evidence references. - [x] Increase the bounded job budget and document its distinction from the per-test limit. - [x] Validate the workflow change, all repository hooks and all eight final-head CI checks. ## Definition of done The independently signed fix PR passes its full CI suite. P2-09 remains blocked until this fix is merged, integrated and its own final-head checks pass; its checklist must not be completed on the timed-out head.
erikinkinen added this to the Phase 2 milestone 2026-10-21 00:30:20 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
erikinkinen/AES#649
No description provided.