Skip to content

Algo2015 Verification Execution Roadmap

Status: Active
Owner: BionicLoop engineering
Last updated: 2026-03-09 16:09 UTC

1. Purpose

Convert the Algo2015 verification strategy into an executable, auditable work program with clear progress tracking.

This roadmap is the operational companion to: - Docs/Planning/Algo2015VerificationPlan.md - Docs/Quality/SoftwareVerificationAndValidationPlan.md - Docs/Quality/TraceabilityMatrix.md (RA-014, TV-ALG-*)

2. Legacy V&V Methods Harvested (From ALG_Main_VandV)

Reviewed package: /Users/jcostik/Downloads/ALG_Main_VandV

Methods worth carrying forward: - Scenario-family structure (Input Fields, Core Reqs, Coverage, Tool Verification). - Procedure-first evidence discipline (protocol + execution + archived outputs). - Release-tagged output trees with copied source inputs and generated logs. - Explicit static-analysis/code-review quality lanes plus conditional MISRA decision linkage.

Observed legacy gaps to avoid: - Manual golden comparison as primary oracle. - Screenshot/docx-only acceptance with limited machine-readable assertions. - Tooling variation not always captured as immutable run manifest.

3. End-State Architecture

Primary orchestration remains shell-based, but expanded into a full verification pipeline:

  1. prepare: environment checks + immutable run manifest.
  2. run: execute deterministic suites from scenario manifests.
  3. evaluate: assertion engine (golden + property/metamorphic checks).
  4. coverage: structural metrics and branch delta.
  5. package: STR bundle with trace links and exception package.

4. Work Breakdown and Progress Tracker

Legend: - [ ] not started - [-] in progress - [x] done

Phase A: Orchestrator Foundation

  • [x] A1. Clean evidence reset and fresh baseline run captured (clean-01).
  • [x] A2. Refactor runner into command stages: prepare/run/evaluate/coverage/package.
  • [x] A3. Add machine-readable run manifest (manifest.json) with:
  • git SHA
  • toolchain versions
  • compile flags
  • suite list
  • source/input checksums
  • [x] A4. Add deterministic seed and run-id policy to prevent ambiguous reruns.

Deliverables: - Scripts/run_algo2015_verification.sh staged orchestrator (prepare/coverage/run/evaluate/package/all). - Docs/Quality/Evidence/Working/STR-ALG-001/2026-02-19-tv-alg-phase-i-i6-rerun/manifest.json.

Phase B: Requirement-Aware Scenario Suites

  • [x] B1. Implement InputFields suite with automated warning-surrogate and boundary assertions.
  • [x] B2. Implement CoreReqs suite with requirement-tagged assertion checks.
  • [x] B3. Implement ToolVerification boundary-transfer assertions (no manual-only review).
  • [x] B4. Produce per-suite results.json and human-readable summary.

Deliverables: - Scenario manifests (YAML/JSON) under version control. - Structured per-suite pass/fail output in STR evidence tree (suites/inputfields/results.json, suites/core-reqs/results.json, suites/tool-verification/results.json, evaluation-summary.json).

Phase D: Branch-Closure Campaign

Target functions (top residual branch misses): - Set_Target - Highs_Lows - Trim_Arrays - Pumps_CGM_UI_Fields - SaveData

  • [x] D1. Generate branch-target catalog from coverage-export.json.
  • [x] D2. Add directed vector packs for each target function.
  • [x] D3. Iterate until branch target met or justified exception remains.
  • [x] D4. Minimize/normalize reproducer vectors for long-term regression use.

Deliverables: - Updated coverage report. - Updated uncovered-branch-gap-map.md. - Updated branch-exception-package.md. - New branch-target-catalog.{json,md} artifact. - New deterministic vector-pack reference (branch-target-vector-pack.json copied from Docs/Quality/TestVectors/ALG2015/BranchTargetVectorPack.json).

Phase C: Oracle Model Upgrade

  • [x] C1. Golden-vector oracle framework (stable expected snapshots).
  • [x] C2. Property/metamorphic oracle framework for broad-state safety checks.
  • [x] C3. Differential replay mode for pregnancy config changes (TV-ALG-009).

Deliverables: - Oracle spec docs + encoded assertions. - Differential report artifact in STR bundle. - New staged suite output: suites/differential/results.json. - Differential replay artifact: suites/differential/differential-report.json.

Phase E: Exception Package Hardening

  • [x] E1. For each residual uncovered branch, provide:
  • reachability rationale
  • safety impact
  • mitigation/monitoring
  • retirement or acceptance disposition
  • [x] E2. Add reviewer sign-off fields and decision date.

Deliverables: - branch-exception-package.md with sign-off metadata. - branch-exception-package.json with reviewer decision fields and machine-readable entry records.

Phase F: Quality Lane Integration

  • [x] F1. Integrate static-analysis lane into the same execution package.
  • [x] F2. Integrate MISRA decision/results linkage (risk-based conditional lane).
  • [x] F3. Link code-review record to run SHA and impacted SRS-*.

Deliverables: - Unified STR package with structural + behavioral + static checks. - New staged suite output: suites/static-analysis/results.json. - Package-level quality lane linkage in manifest.json (qualityLanes.codeReviewLinkage, qualityLanes.misraLinkage, where MISRA is conditionally satisfied by linked evidence or explicit N/A rationale).

Phase G: Submission-Grade STR Packaging

  • [x] G1. Standardize STR folder template (README, manifest, results, coverage, exceptions).
  • [x] G2. Add trace map linking each suite/assertion to TV-ALG-* and SRS-ALG-*.
  • [x] G3. Add reproducibility recipe (single command + expected artifacts list).

Deliverables: - Repeatable IDE-ready package schema for each run. - str-template-check.json artifact with required/missing counters. - suite-assertion-trace-map.{json,md} generated from staged suite assertions. - reproducibility-recipe.md with command/integrity/required-artifact contract.

Phase H: Branch-Coverage 100% Campaign (Post-G)

Formal tracked baseline remains 88.02% with 127 missed branches. Latest local working run (not committed as formal STR evidence) reached 90.58% with 100 missed branches.

Immediate, do-now scope (no product behavior change required): - Needs directed vectors (60 missed branches) - Environment-dependent diagnostics (5 missed branches; via deterministic fault-injection seams in harness)

Decision-gated scope (requires code-owner sign-off before closure): - Legacy set-point path (14 missed branches; SetPtKnob path currently hard-disabled in runtime flow) - Legacy trial path (4 missed branches; ExpmtOver call path currently disabled) - Constraint-implied unreachable (12 missed branches) - Mutually exclusive condition residual (3 missed branches) - Guardrail path (2 missed branches; malformed-state forcing only)

  • [x] H1. Add deterministic vector packs for top residual functions (Trim_Arrays, Pumps_CGM_UI_Fields, Glucagon_PD, Reconcile_I_Dose, TemporaryDropout, Extract_CGM_Adapt, loadState, Constraints, OL_Controller).
  • [x] H2. Add deterministic fault-injection hooks in coverage harness for SaveData mixed open-fail/open-success retry branches.
  • [x] H3. Re-run staged verification after each vector pack, track branch delta in STR working evidence, and update gap map.
  • [x] H4. Produce a decision package for gated branches (Set_Target, ExpmtOver, constraint-implied residuals) with recommendation: retire dead paths or add test-only seams.
  • [x] H5. Close with either:
  • 100% branch achieved, or
  • formal exception package with signed justification for each residual branch.

Deliverables: - Updated branch-target-vector-pack.json and staged suite vectors. - Updated coverage-report.txt and uncovered-branch-gap-map.md. - Decision memo: Docs/Planning/Algo2015PhaseHDecisionMemo.md. - Updated branch-exception-package.{md,json} with final dispositions.

Phase I: CPP-Grounded Safety Gap Closure (Current-Surface Parity)

Purpose: - Close the real safety-relevant verification gaps identified from the actual current algorithm/bridge surfaces (not legacy assumptions). - Treat legacy-only items (for example raw algorithmWarning parity in host-visible outputs) as out-of-scope unless we explicitly add a bridge seam for them.

Execution order (required): - I1 -> I2 -> I3 -> I4 -> I5 -> I6 - Complete surface-contract matrix and characterization before adding closure assertions.

  • [x] I1. Build a machine-readable "current surface contract" matrix from actual interfaces (AlgorithmInterface.h, bridge C layer, Swift runtime mapping), including expected handling for in-range/out-of-range permutations.
  • Include subject ID parser tokens that affect target behavior.
  • Include meal-time/meal-status inputs and domain boundaries.
  • Include pump unavailable sentinels and bridge normalization rules.
  • Mark legacy/non-surfaced fields as N/A with rationale.
  • [x] I2. Expand staged inputfields and core-reqs suites to execute this matrix and emit row-level observed-vs-expected results (JSON/TSV).
  • [x] I3. Add targeted behavior characterization tests for real high-risk logic currently under-asserted:
  • meal-time validation behavior (including invalid meal type values)
  • pump reconciliation/under-delivery and over-delivery handling
  • offline/forced-open/BG-fallback transition behavior
  • subject ID parser impact on target/set-point behavior
  • [x] I4. Add bridge-contract checks for bool/sentinel normalization and document fixed bridge assumptions (for example forced bgCal and glucagon-unavailable mappings).
  • [x] I5. Classify dead/legacy input/path surfaces (active, legacy-disabled, runtime-unreachable) with decision-owner sign-off and RTM linkage.
  • [x] I6. Run staged verification and publish updated artifacts/trace links as the new baseline.

Deliverables: - Current-surface matrix spec under version control (source of truth for verification vectors): - Docs/Planning/Algo2015CurrentSurfaceContractMatrix.json - Updated staged suite artifacts: - suites/inputfields/results.json - suites/core-reqs/results.json - row-level matrix observation artifact (TSV/JSON) - Updated gap closure report for the six real cpp-grounded areas (meal validation, reconciliation, transitions, subject parsing, bridge normalization, dead-path classification). - Updated assertion trace map (suite-assertion-trace-map.{json,md}) linking these checks to TV-ALG-008 and related SRS-ALG-* requirements. - Updated quality docs status snapshot after staged run.

5. Current Baseline Snapshot (Starting Point)

From: - Docs/Quality/Evidence/Working/STR-ALG-001/2026-02-19-tv-alg-phase-i-i6-rerun/suites/coverage/coverage-report.txt

Metrics: - Formal tracked baseline (2026-02-18): Algorithm_2015_10_13.cpp function 100.00%, line 95.13%, branch 88.02% - Latest local working run (2026-02-19): Algorithm_2015_10_13.cpp function 100.00%, line 97.33%, branch 90.58% - Algo2015Bridge.c: function 100.00%, line 100.00%, branch 100.00% (both runs)

Current position: - Line/function targets achieved. - Branch target (>=90%) achieved in working run (90.58%). - Phase H closed via formal exception-package path for residual branches. - A2/A3/A4/B1 implementation evidence: - Docs/Quality/Evidence/Working/STR-ALG-001/2026-02-19-tv-alg-phase-i-i6-rerun/ - B2/B3 implementation evidence: - Docs/Quality/Evidence/Working/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/ - Phase-I staged rerun baseline: - Docs/Quality/Evidence/Working/STR-ALG-001/2026-02-19-tv-alg-phase-i-i6-rerun/ - evaluation-summary.json: all suites pass; coverage metrics algo line 97.33%, algo branch 90.58%, bridge line/branch 100%.

6. Execution Rules

  • Always run from a clean evidence destination.
  • Every run emits immutable manifest + command log.
  • Every uncovered branch is either:
  • covered by directed vectors, or
  • explicitly justified in exception package.
  • Every roadmap phase update must also update:
  • Docs/Planning/Algo2015VerificationPlan.md (status snapshot)
  • Docs/Quality/TraceabilityMatrix.md (RA-014 evidence links when changed)
  • Docs/Quality/CodeReviewLog.md when review/fixes occur

7. Immediate Next Actions

  1. Run algorithm-owner + clinical lead review for uncovered-branch rationale using Docs/Planning/Algo2015UncoveredBranchReviewPacket.md.
  2. After signoff, run a formal (tracked) STR execution in Docs/Quality/Evidence/Formal/STR-ALG-001/... using signoff-aware packaging.
  3. Update Docs/Quality/TraceabilityMatrix.md RA-014 row with formal STR package links once generated.
  4. Continue non-blocking branch-reduction workstream for remaining directed-vector opportunities.