Phase 1: Plot and figure generation pipeline #78

Merged

erikinkinen merged 3 commits from 1-plot-and-figure-generation-pipeline into main

2026-03-06 12:39:52 +01:00

erikinkinen commented

2026-03-06 12:36:54 +01:00

Owner

Closes #38

Summary

This PR adds a deterministic, script-driven Phase 1 figure pipeline that
generates all required figures directly from sweep raw tables
(runs.csv + metrics_long.csv) with no manual plotting steps.

It introduces:

a Python + Matplotlib plotting entrypoint,
canonical sweep configs for figure datasets,
a one-command tools/regen-figures.sh workflow,
fixture/contract/determinism tests,
an end-to-end figure smoke test wired into CI.

Scope

Included

tools/figures/phase1/plot_phase1_figures.py
- CLI:
  python3 .../plot_phase1_figures.py --cost-dir <dir> --depth-dir <dir> --fanout-dir <dir> --out-dir <dir> [--formats png,svg]
- strict input contract checks (runs.csv, metrics_long.csv, required columns)
- run-level metric derivation:
  - completeness ratio:
    completeness_holds_total / revocation_requests_total
  - cost scalar:
    max edges_scanned_cumulative
- deterministic plot generation for:
  - cost_vs_completeness_by_workload
  - depth_sensitivity_completeness
  - depth_sensitivity_cost
  - fanout_sensitivity_completeness
  - fanout_sensitivity_cost
- deterministic row ordering and fixed plotting/style metadata knobs
Canonical configs under tools/figures/phase1/:
- sweep_cost_completeness.json
- sweep_depth_sensitivity.json
- sweep_fanout_sensitivity.json
- plus small _smoke variants for CI smoke generation
tools/regen-figures.sh
- defaults:
  - raw root: artifacts/phase1/raw
  - figure output: artifacts/phase1/figures
- validates required raw files exist
- if missing, prints exact aes sweep --config ... --out-dir ... commands
- runs plotting entrypoint
Tests:
- tests/phase1_figures_tests.py
  - missing file/column contract failures
  - missing required cost rows failure
  - fixture-based derivation correctness
  - deterministic hash comparison across reruns
- tests/phase1_figures_smoke.sh
  - generates tiny raw datasets from smoke sweep configs
  - runs tools/regen-figures.sh
  - asserts expected PNG/SVG outputs exist and are non-empty
  - reruns and verifies identical hashes
- CTest wiring in tests/CMakeLists.txt
CI update:
- .forgejo/workflows/ci.yml installs python3 and python3-matplotlib
  so figure smoke runs in CI.
Docs:
- docs/phase1.md now documents the figure pipeline, input contract,
  canonical generation commands, one-command regen, and expected outputs.

Explicitly excluded

No figure image artifacts are committed.
No sweep execution is embedded inside plotting code.
No changes to event-log schema or metrics schema.

Design intent

The figure pipeline is a pure, deterministic transform from sweep raw tables
to figure files. This keeps experiment execution and presentation decoupled:
sweeps produce canonical raw data, and plotting consumes only those tables.

Determinism is enforced via stable sorting, fixed strategy ordering, fixed
Matplotlib rcParams, and stable save metadata configuration.

Phase discipline

This is Phase 1 tooling only. It builds directly on Phase 1 sweep outputs
without altering core simulation semantics or introducing a new metrics
version.

Verification

aes_phase1_figures_tests passes
aes_phase1_figures_smoke_test passes
Existing sweep/simulate/metrics tests remain green:
- aes_cli_sweep_tests
- aes_cli_simulate_tests
- aes_metrics_runner_tests
- aes_revocation_outcome_metrics_tests
Documentation updated
Markdown lint passes for updated docs

Notes

Review focus:

data contract validation behavior and error clarity,
metric derivation correctness (especially missing cost-row failure mode),
deterministic output behavior and smoke hash reproducibility.

Closes #38 --- ## Summary This PR adds a deterministic, script-driven Phase 1 figure pipeline that generates all required figures directly from sweep raw tables (`runs.csv` + `metrics_long.csv`) with no manual plotting steps. It introduces: - a Python + Matplotlib plotting entrypoint, - canonical sweep configs for figure datasets, - a one-command `tools/regen-figures.sh` workflow, - fixture/contract/determinism tests, - an end-to-end figure smoke test wired into CI. --- ## Scope ### **Included** - `tools/figures/phase1/plot_phase1_figures.py` - CLI: `python3 .../plot_phase1_figures.py --cost-dir <dir> --depth-dir <dir> --fanout-dir <dir> --out-dir <dir> [--formats png,svg]` - strict input contract checks (`runs.csv`, `metrics_long.csv`, required columns) - run-level metric derivation: - completeness ratio: `completeness_holds_total / revocation_requests_total` - cost scalar: max `edges_scanned_cumulative` - deterministic plot generation for: - `cost_vs_completeness_by_workload` - `depth_sensitivity_completeness` - `depth_sensitivity_cost` - `fanout_sensitivity_completeness` - `fanout_sensitivity_cost` - deterministic row ordering and fixed plotting/style metadata knobs - Canonical configs under `tools/figures/phase1/`: - `sweep_cost_completeness.json` - `sweep_depth_sensitivity.json` - `sweep_fanout_sensitivity.json` - plus small `_smoke` variants for CI smoke generation - `tools/regen-figures.sh` - defaults: - raw root: `artifacts/phase1/raw` - figure output: `artifacts/phase1/figures` - validates required raw files exist - if missing, prints exact `aes sweep --config ... --out-dir ...` commands - runs plotting entrypoint - Tests: - `tests/phase1_figures_tests.py` - missing file/column contract failures - missing required cost rows failure - fixture-based derivation correctness - deterministic hash comparison across reruns - `tests/phase1_figures_smoke.sh` - generates tiny raw datasets from smoke sweep configs - runs `tools/regen-figures.sh` - asserts expected PNG/SVG outputs exist and are non-empty - reruns and verifies identical hashes - CTest wiring in `tests/CMakeLists.txt` - CI update: - `.forgejo/workflows/ci.yml` installs `python3` and `python3-matplotlib` so figure smoke runs in CI. - Docs: - `docs/phase1.md` now documents the figure pipeline, input contract, canonical generation commands, one-command regen, and expected outputs. ### **Explicitly excluded** - No figure image artifacts are committed. - No sweep execution is embedded inside plotting code. - No changes to event-log schema or metrics schema. --- ## Design intent The figure pipeline is a pure, deterministic transform from sweep raw tables to figure files. This keeps experiment execution and presentation decoupled: sweeps produce canonical raw data, and plotting consumes only those tables. Determinism is enforced via stable sorting, fixed strategy ordering, fixed Matplotlib rcParams, and stable save metadata configuration. --- ## Phase discipline This is Phase 1 tooling only. It builds directly on Phase 1 sweep outputs without altering core simulation semantics or introducing a new metrics version. --- ## Verification - [x] `aes_phase1_figures_tests` passes - [x] `aes_phase1_figures_smoke_test` passes - [x] Existing sweep/simulate/metrics tests remain green: - `aes_cli_sweep_tests` - `aes_cli_simulate_tests` - `aes_metrics_runner_tests` - `aes_revocation_outcome_metrics_tests` - [x] Documentation updated - [x] Markdown lint passes for updated docs --- ## Notes Review focus: - data contract validation behavior and error clarity, - metric derivation correctness (especially missing cost-row failure mode), - deterministic output behavior and smoke hash reproducibility.