Phase 1: Batch runner for Phase 1 sweeps #77

Merged

erikinkinen merged 4 commits from 1-batch-runner-for-phase-1-sweeps into main

2026-03-06 12:02:47 +01:00

erikinkinen commented

2026-03-06 12:00:03 +01:00

Owner

Summary

Implements Issue #37 by adding a deterministic Phase 1 batch runner via a new aes sweep CLI subcommand.

Closes #37.

What’s Included

Added strict JSON v1 sweep config parsing and validation.
Added deterministic matrix expansion in locked order:
1. workload axis order
2. revoker axis order
3. seed axis order
Added aes sweep --config <path> [--out-dir <path>].
Implemented sequential-only batch execution for all expanded runs.
Reused existing simulation/event-log/metrics semantics through shared runner logic.
Added deterministic run IDs: r000001, r000002, ...
Added per-run artifacts:
- <out_dir>/runs/<run_id>/event_log.jsonl
- <out_dir>/runs/<run_id>/metrics_v1.json
Implemented continue-on-failure behavior:
- all runs attempt execution
- failures are recorded per run
- command returns non-zero when any run fails
Added atomic sweep-level tidy exports:
- <out_dir>/runs.csv
- <out_dir>/metrics_long.csv
Added deterministic flattening/order rules for metrics_long.csv, including unobserved latency encoding with observed=0 and empty value.
Updated Phase 1 docs with schema, ordering, execution policy, exit codes, and CSV schemas.
Added comprehensive sweep CLI tests for config strictness, ordering, determinism, failure continuation, out-dir override, and CSV shape/order/reproducibility.

Exit Codes

0: all runs succeeded
1: sweep completed with one or more run failures
2: CLI/config validation error

Backward Compatibility

No core/event-log schema changes.
Existing simulate behavior preserved.
Existing metrics/event-log/revoker regression suites remain green.

Verification

cmake --build _build --target aes_cli_sweep_tests aes_cli_simulate_tests aes_metrics_runner_tests aes_revocation_outcome_metrics_tests
ctest --test-dir _build --output-on-failure -R "aes_cli_sweep_tests|aes_cli_simulate_tests|aes_metrics_runner_tests|aes_revocation_outcome_metrics_tests"
ctest --test-dir _build --output-on-failure -R "aes_event_log_reader_tests|aes_event_log_replay_tests|aes_revocation_strategy_tests|aes_strategy_equivalence_tests|aes_invalid_event_determinism_tests"

## Summary Implements Issue #37 by adding a deterministic Phase 1 batch runner via a new `aes sweep` CLI subcommand. Closes #37. ## What’s Included - Added strict JSON v1 sweep config parsing and validation. - Added deterministic matrix expansion in locked order: 1. workload axis order 2. revoker axis order 3. seed axis order - Added `aes sweep --config <path> [--out-dir <path>]`. - Implemented sequential-only batch execution for all expanded runs. - Reused existing simulation/event-log/metrics semantics through shared runner logic. - Added deterministic run IDs: `r000001`, `r000002`, ... - Added per-run artifacts: - `<out_dir>/runs/<run_id>/event_log.jsonl` - `<out_dir>/runs/<run_id>/metrics_v1.json` - Implemented continue-on-failure behavior: - all runs attempt execution - failures are recorded per run - command returns non-zero when any run fails - Added atomic sweep-level tidy exports: - `<out_dir>/runs.csv` - `<out_dir>/metrics_long.csv` - Added deterministic flattening/order rules for `metrics_long.csv`, including unobserved latency encoding with `observed=0` and empty `value`. - Updated Phase 1 docs with schema, ordering, execution policy, exit codes, and CSV schemas. - Added comprehensive sweep CLI tests for config strictness, ordering, determinism, failure continuation, out-dir override, and CSV shape/order/reproducibility. ## Exit Codes - `0`: all runs succeeded - `1`: sweep completed with one or more run failures - `2`: CLI/config validation error ## Backward Compatibility - No core/event-log schema changes. - Existing `simulate` behavior preserved. - Existing metrics/event-log/revoker regression suites remain green. ## Verification - `cmake --build _build --target aes_cli_sweep_tests aes_cli_simulate_tests aes_metrics_runner_tests aes_revocation_outcome_metrics_tests` - `ctest --test-dir _build --output-on-failure -R "aes_cli_sweep_tests|aes_cli_simulate_tests|aes_metrics_runner_tests|aes_revocation_outcome_metrics_tests"` - `ctest --test-dir _build --output-on-failure -R "aes_event_log_reader_tests|aes_event_log_replay_tests|aes_revocation_strategy_tests|aes_strategy_equivalence_tests|aes_invalid_event_determinism_tests"`