Phase 6: Implement held out evaluation #457

Open
opened 2026-10-20 15:05:05 +02:00 by erikinkinen · 0 comments
Owner

Plan ID: P6-11
Record type: Tracking parent; no implementation PR
Scientific protocol: 2.2 · Backlog revision: 2.2.1
Specification: sections 10.5, 10.6.
Source: AES_PHASES_2_6_PLAN.md, issue heading at line 8296.
Plan SHA-256: 8acdaeb52d6a94fcad7de64d64521cd465f608344d63560b45f9202f5af71212
Protocol snapshot: Download the complete governing plan.

Goal

assess discrimination, calibration and usefulness on unseen groups.

Retained requirements

  • P6-11.R1: Add AP, AUROC, threshold metrics, calibration and weighted prevalence.
  • P6-11.R2: Add event recall, healthy at-risk exposure and conditional lead distributions.
  • P6-11.R3: Emit prediction/model/alert tables for primary and all transfer fits.

Child issues

Closure

This is an AND tracking parent, not an implementation task. Close it only after every child is accepted and the original parent acceptance below is verified. A permitted not_applicable child needs the signed route decision required by section 17.4.

every model uses identical eligible comparison rows and all
denominators, absent classes and undefined metrics are recorded.

Execution and acceptance contract

Apply the governing semantic/data/statistical contracts in sections 3–5 and the relevant phase section, plus the section-11 checks applicable to this task. Section 17.4 defines the required acceptance evidence. Store the outcome and immutable prerequisite/input/output hashes at

artifacts/backlog/2.2.1/P6-11/acceptance.json.

Only explicit dependency and parent-closure edges govern readiness; numbering adds no predecessor dependency. Checkboxes are work/acceptance items, not commit quotas. Keep implementation changes independently reviewable using the existing PR template. Execution, review and gate tasks close with their evidence/decision records and a PR only when tracked artifacts change.

An unsupported scientific effect is a valid result. Missing correctness validation, invalid inference or absent required review is not completion. Do not change endpoints, matrices, margins, inference rules or sample sizes outside the protocol change-control process.

Native blocking dependencies below represent the child-closure requirements; the parent must never block its own children.

<!-- aes-plan-id: P6-11; backlog-revision: 2.2.1 --> **Plan ID:** `P6-11` **Record type:** Tracking parent; no implementation PR **Scientific protocol:** 2.2 · **Backlog revision:** 2.2.1 **Specification:** sections 10.5, 10.6. **Source:** `AES_PHASES_2_6_PLAN.md`, issue heading at line 8296. **Plan SHA-256:** `8acdaeb52d6a94fcad7de64d64521cd465f608344d63560b45f9202f5af71212` **Protocol snapshot:** [Download the complete governing plan](https://git.erikinkinen.fi/attachments/31cb768b-4788-4e38-aba9-9d7f0d1ee16d). ## Goal assess discrimination, calibration and usefulness on unseen groups. ## Retained requirements - **P6-11.R1:** Add AP, AUROC, threshold metrics, calibration and weighted prevalence. - **P6-11.R2:** Add event recall, healthy at-risk exposure and conditional lead distributions. - **P6-11.R3:** Emit prediction/model/alert tables for primary and all transfer fits. ## Child issues - [ ] [P6-11-A (#458)](https://git.erikinkinen.fi/erikinkinen/AES/issues/458) — Compute held-out discrimination and calibration - [ ] [P6-11-B (#459)](https://git.erikinkinen.fi/erikinkinen/AES/issues/459) — Compute operational held-out outcomes ## Closure This is an AND tracking parent, not an implementation task. Close it only after every child is accepted and the original parent acceptance below is verified. A permitted `not_applicable` child needs the signed route decision required by section 17.4. every model uses identical eligible comparison rows and all denominators, absent classes and undefined metrics are recorded. ## Execution and acceptance contract Apply the governing semantic/data/statistical contracts in sections 3–5 and the relevant phase section, plus the section-11 checks applicable to this task. Section 17.4 defines the required acceptance evidence. Store the outcome and immutable prerequisite/input/output hashes at `artifacts/backlog/2.2.1/P6-11/acceptance.json`. Only explicit dependency and parent-closure edges govern readiness; numbering adds no predecessor dependency. Checkboxes are work/acceptance items, not commit quotas. Keep implementation changes independently reviewable using the existing PR template. Execution, review and gate tasks close with their evidence/decision records and a PR only when tracked artifacts change. An unsupported scientific effect is a valid result. Missing correctness validation, invalid inference or absent required review is not completion. Do not change endpoints, matrices, margins, inference rules or sample sizes outside the protocol change-control process. Native blocking dependencies below represent the child-closure requirements; the parent must never block its own children.
erikinkinen added this to the Phase 6 milestone 2026-10-20 15:05:05 +02:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
erikinkinen/AES#457
No description provided.