Phase 1: Phase 1 revocation outcome metrics #76

Merged

erikinkinen merged 3 commits from 1-revocation-outcome-metrics into main

2026-03-06 11:21:43 +01:00

erikinkinen commented

2026-03-06 11:20:57 +01:00

Owner

Closes #36

Summary

This PR adds metrics_v1 revocation-outcome reporting to quantify strategy trade-offs from replayed event logs.

It introduces deterministic, per-strategy metrics for:

oracle completeness vs residuals,
request-indexed revocation cost curves,
latency-to-effect for both selector roots and strategy-resolved targets.

metrics_v0 remains unchanged and default.
No revocation runtime behavior or event-log schema/version is changed.

Scope

Included

New metrics_v1 report model and JSON encoder
compute_revocation_outcome_metrics_v1(...) over validated event-log records
run_metrics_from_event_log_jsonl_v1(...) runner API
Per-strategy completeness counters via post-revocation oracle
Per-strategy cost curve points with cumulative counters
Per-strategy latency points and summaries (root and resolved scopes)
aes_metrics CLI --report <v0|revocation_v1> support
Docs update for #36 metrics semantics
New/extended tests for metrics API, runner, CLI report parsing, determinism, and latency behavior

Explicitly excluded

Any revocation strategy logic changes
Any event-log version/schema bump
Any changes to metrics_v0 output semantics

Design notes

Completeness policy is uniform immediate oracle evaluation for all strategies.
Oracle selector is strategy-resolved targets (revocation_result.targeted_edges).
Cost curves are keyed by strategy-local revoke request index.
Latency effect is first subsequent denied UseCap with reason:
- missing_edge, or
- invalidated
insufficient_rights does not count as revocation effect.
CLI remains backward-compatible: default report is still v0.

Verification

cmake --build _build --target aes_revocation_outcome_metrics_tests aes_metrics_tests aes_metrics_runner_tests aes_metrics_api_tests aes_metrics_cli
ctest --test-dir _build --output-on-failure -R "aes_revocation_outcome_metrics_tests|aes_metrics_tests|aes_metrics_runner_tests|aes_metrics_api_tests"
ctest --test-dir _build --output-on-failure -R "aes_event_log_reader_tests|aes_event_log_replay_tests|aes_revocation_oracle_tests|aes_revocation_strategy_tests|aes_strategy_equivalence_tests|aes_cli_simulate_tests"

Review focus

Completeness accounting correctness across eager and lazy/epoch strategies
Cost curve request-index/cumulative semantics
Latency capture and denial-reason filtering
Backward compatibility of aes_metrics default behavior

Closes #36 --- ## Summary This PR adds `metrics_v1` revocation-outcome reporting to quantify strategy trade-offs from replayed event logs. It introduces deterministic, per-strategy metrics for: - oracle completeness vs residuals, - request-indexed revocation cost curves, - latency-to-effect for both selector roots and strategy-resolved targets. `metrics_v0` remains unchanged and default. No revocation runtime behavior or event-log schema/version is changed. --- ## Scope ### Included - New `metrics_v1` report model and JSON encoder - `compute_revocation_outcome_metrics_v1(...)` over validated event-log records - `run_metrics_from_event_log_jsonl_v1(...)` runner API - Per-strategy completeness counters via post-revocation oracle - Per-strategy cost curve points with cumulative counters - Per-strategy latency points and summaries (root and resolved scopes) - `aes_metrics` CLI `--report <v0|revocation_v1>` support - Docs update for #36 metrics semantics - New/extended tests for metrics API, runner, CLI report parsing, determinism, and latency behavior ### Explicitly excluded - Any revocation strategy logic changes - Any event-log version/schema bump - Any changes to `metrics_v0` output semantics --- ## Design notes - Completeness policy is uniform immediate oracle evaluation for all strategies. - Oracle selector is strategy-resolved targets (`revocation_result.targeted_edges`). - Cost curves are keyed by strategy-local revoke request index. - Latency effect is first subsequent denied `UseCap` with reason: - `missing_edge`, or - `invalidated` - `insufficient_rights` does not count as revocation effect. - CLI remains backward-compatible: default report is still `v0`. --- ## Verification - [x] `cmake --build _build --target aes_revocation_outcome_metrics_tests aes_metrics_tests aes_metrics_runner_tests aes_metrics_api_tests aes_metrics_cli` - [x] `ctest --test-dir _build --output-on-failure -R "aes_revocation_outcome_metrics_tests|aes_metrics_tests|aes_metrics_runner_tests|aes_metrics_api_tests"` - [x] `ctest --test-dir _build --output-on-failure -R "aes_event_log_reader_tests|aes_event_log_replay_tests|aes_revocation_oracle_tests|aes_revocation_strategy_tests|aes_strategy_equivalence_tests|aes_cli_simulate_tests"` --- ## Review focus - Completeness accounting correctness across eager and lazy/epoch strategies - Cost curve request-index/cumulative semantics - Latency capture and denial-reason filtering - Backward compatibility of `aes_metrics` default behavior