Algo2015 Verification Plan (FDA IDE Evidence)
Status: Draft v0.7
Owner: BionicLoop engineering
Last updated: 2026-02-19 14:18 UTC
1. Purpose
Define the verification campaign required to provide high-confidence, traceable Algo2015 evidence for FDA IDE submission.
This plan is specifically for:
- structural coverage of current Algo2015 C++ implementation
- deterministic behavioral regression protection
- bridge-contract validation between Swift/C and C++
- reproducible evidence packaging (STP/STR + coverage reports)
Execution and progress tracking for this plan are maintained in:
- Docs/Planning/Algo2015ExecutionRoadmap.md
2. Scope
In scope source surfaces:
- Algo2015/Algorithm_2015_10_13.cpp
- Algo2015/AlgorithmInterface.h
- BionicLoopCore/Sources/Algo2015Bridge/Algo2015Bridge.c
- BionicLoopCore/Sources/Algo2015Bridge/include/Algo2015Bridge.h
- BionicLoopCore/Sources/BionicLoopCore/Algorithms/RealBUDosingAlgorithm.swift
Out of scope for this campaign: - BLE transport reliability (covered by integration/system testing) - UI behavior (covered by app/UI test plans)
3. Coverage Targets
| Metric | Target | Notes |
|---|---|---|
| Function coverage (C bridge) | 100% | Algo2015Bridge.c must be fully exercised. |
| Line coverage (C bridge) | 100% | Null guards + all mapping paths covered. |
| Branch coverage (C bridge) | 100% | Includes pointer guards and request-time mapping branch. |
| Function coverage (Algo2015 C++) | >= 98% | All callable algorithm units exercised via deterministic scenarios. |
| Line coverage (Algo2015 C++) | >= 95% | Remaining uncovered lines require explicit rationale. |
| Branch coverage (Algo2015 C++) | >= 90% | Remaining uncovered branches require explicit rationale and risk disposition. |
Any uncovered region must be documented in STR evidence with one of: - not reachable in deployed runtime path - debug/legacy path planned for retirement - risk accepted with documented mitigation
3.1 Current Measured Baseline (Formal + Working)
Formal source:
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/coverage/coverage-report.txt
| Metric | Target | Current | Gap |
|---|---|---|---|
| Function coverage (C bridge) | 100% | 100.00% | None |
| Line coverage (C bridge) | 100% | 100.00% | None |
| Branch coverage (C bridge) | 100% | 100.00% | None |
| Function coverage (Algo2015 C++) | >= 98% | 100.00% | None |
| Line coverage (Algo2015 C++) | >= 95% | 95.13% | None |
| Branch coverage (Algo2015 C++) | >= 90% | 88.02% | 1.98% |
Notes:
- Formal tracked baseline remains below branch target and is retained for STR traceability.
- Formal baseline residual branch rationale:
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/coverage/uncovered-branch-gap-map.md
Latest local working run (2026-02-19, non-formal/untracked evidence path):
| Metric | Target | Current | Gap |
|---|---|---|---|
| Function coverage (C bridge) | 100% | 100.00% | None |
| Line coverage (C bridge) | 100% | 100.00% | None |
| Branch coverage (C bridge) | 100% | 100.00% | None |
| Function coverage (Algo2015 C++) | >= 98% | 100.00% | None |
| Line coverage (Algo2015 C++) | >= 95% | 97.33% | None |
| Branch coverage (Algo2015 C++) | >= 90% | 90.58% | None |
Working-run notes:
- H1/H2/H3 implementation increased branch coverage from 88.02% to 90.58%.
- Remaining misses in working run are concentrated in decision-gated and directed-vector residual paths.
- Phase H closeout decision package is captured in:
- Docs/Planning/Algo2015PhaseHDecisionMemo.md
3.2 Branch 100% Closure Campaign (Supplemental Goal)
Submission threshold remains Section 3 target (>=90% branch with justified exceptions), but current engineering objective is to pursue full closure where feasible.
Current missed-branch composition (working run, 100 total):
- Needs directed vectors: 60
- Environment-dependent diagnostics: 5
- Legacy set-point path: 14
- Legacy trial path: 4
- Constraint-implied unreachable: 12
- Mutually exclusive condition residual: 3
- Guardrail path: 2
Work partition:
- Immediate executable scope (no behavior changes): directed vectors + deterministic harness fault injection for diagnostic retry branches.
- Decision-gated scope: dead/legacy or structurally constrained paths (Set_Target, ExpmtOver, mutually exclusive/constraint-implied branches) requiring owner sign-off for retirement, seam injection, or formal exception acceptance.
Closure contract:
- If all gated decisions resolve to testable paths, continue to 100% branch closure.
- If not, capture signed residual exceptions in branch-exception-package.{json,md} with explicit reachability/safety/mitigation/disposition.
Current Phase-H closeout decision:
- Closed via exception-package path (branch target met; 100% deferred).
- Formal tracked execution still required to capture submission-grade signed exception artifacts in Docs/Quality/Evidence/Formal/STR-ALG-001/....
3.3 Current Staged Verification Runner Baseline (2026-02-18)
Execution command:
- Scripts/run_algo2015_verification.sh all Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final --seed 20260218 --suites coverage,inputfields,core-reqs,differential,tool-verification,static-analysis
Current generated STR artifacts:
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/run-context.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/evaluation-summary.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/manifest.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/inputfields/results.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/core-reqs/results.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/differential/results.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/differential/differential-report.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/tool-verification/results.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suites/static-analysis/results.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/str-template-check.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/suite-assertion-trace-map.json
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-012-verification-b2-b3-final/reproducibility-recipe.md
Current status:
- staged command architecture implemented (prepare, coverage, run, evaluate, package, all)
- deterministic run-id and seed policy implemented
- InputFields automated suite implemented and passing (9/9 assertions)
- CoreReqs requirement-tagged suite implemented and passing (5/5 assertions)
- ToolVerification boundary-transfer suite implemented and passing (9/9 assertions)
- Differential suite implemented and passing (5/5 assertions) with deterministic JSON report output
- StaticAnalysis suite implemented and passing (6/6 assertions) with clang build/analyze logs, MISRA-policy linkage metadata, and CodeReviewLog run-SHA linkage
- submission-grade packaging artifacts now generated per run:
- str-template-check.json (required artifact completeness)
- suite-assertion-trace-map.{json,md} (assertion-level TV-ALG-*/SRS-ALG-* linkage)
- reproducibility-recipe.md (single-command rerun contract)
- Golden-vector tests are now backed by reusable oracle helpers (Algo2015OracleSupport) with explicit snapshot assertions
- New metamorphic/property suite (Algo2015MetamorphicTests) verifies deterministic replay and key monotonic properties
- coverage suite now emits branch-target-catalog.{json,md} and hardened exception artifacts (branch-exception-package.{json,md}) directly from coverage-export.json
- exception package artifacts now include per-symbol reachability/safety/mitigation/disposition fields and reviewer sign-off metadata (reviewerName, reviewerRole, decisionDate, decisionStatus, decisionNotes)
- directed vector-pack reference is versioned at Docs/Quality/TestVectors/ALG2015/BranchTargetVectorPack.json and copied into coverage artifacts as branch-target-vector-pack.json
- structural coverage currently measures 95.13% line and 88.02% branch for Algorithm_2015_10_13.cpp
- package manifest now carries quality-lane metadata (qualityLanes.codeReviewLinkage, qualityLanes.misraLinkage) for audit linkage; MISRA lane closure is risk-based/conditional and resolved by either linked MISRA evidence + deviations or explicit not-applicable rationale in decision artifacts
3.4 Phase I Alignment (CPP-Grounded Current-Surface Closure)
This plan adopts the same Phase I sequence defined in:
- Docs/Planning/Algo2015ExecutionRoadmap.md
Required execution order:
- I1 -> I2 -> I3 -> I4 -> I5 -> I6
- Current-surface contract matrix closure must complete before gap-closure assertions are locked.
Phase I tasks:
- I1 Build a machine-readable current-surface contract matrix from actual interfaces (AlgorithmInterface.h, bridge C layer, Swift runtime mapping), including in-range/out-of-range permutations and expected handling.
- I2 Expand staged inputfields and core-reqs suites to execute the matrix and emit row-level observed-vs-expected results.
- I3 Add targeted characterization/closure assertions for real high-risk logic:
- meal-time validation behavior for invalid meal-type values
- pump reconciliation behavior (under/over-delivery handling)
- offline/forced-open/BG-fallback transition behavior
- subject ID parser impact on target/set-point behavior
- I4 Add bridge-contract checks for bool/sentinel normalization and document fixed bridge assumptions (bgCal mapping and glucagon-unavailable mapping).
- I5 Classify dead/legacy input/path surfaces (active, legacy-disabled, runtime-unreachable) with decision-owner sign-off.
- I6 Run staged verification and publish updated trace/package artifacts as the new baseline.
Current progress:
- I1 complete: Docs/Planning/Algo2015CurrentSurfaceContractMatrix.json added as machine-readable source of truth.
- I2 complete: staged suites now emit row-level artifacts:
- suites/inputfields/matrix-row-observations.{json,tsv}
- suites/core-reqs/matrix-row-observations.{json,tsv}
- I3 complete: targeted characterization suite added and wired into staged core-reqs execution:
- BionicLoopCore/Tests/BionicLoopCoreTests/Algo2015CurrentSurfaceCharacterizationTests.swift
- suites/core-reqs/current-surface-characterization.log
- I4 complete: bridge normalization tests and fixed-assumption checks added:
- BionicLoopCore/Tests/BionicLoopCoreTests/Algo2015BridgeNormalizationTests.swift
- suites/core-reqs/bridge-normalization.log
- matrix rows CSM-011 and CSM-012 explicitly tracked in suites/core-reqs/matrix-row-observations.{json,tsv}
- I5 complete: legacy/dead-path decision register and RTM linkage added:
- Docs/Planning/Algo2015LegacyPathClassification.md
- Docs/Quality/TraceabilityMatrix.md (RA-014 evidence text includes I5 classification register)
- I6 complete: staged full-suite rerun baseline published:
- Docs/Quality/Evidence/Working/STR-ALG-001/2026-02-19-tv-alg-phase-i-i6-rerun/
- all staged suites pass (coverage, inputfields, core-reqs, differential, tool-verification, static-analysis)
- coverage summary: Algo2015 C++ line 97.33%, branch 90.58%; bridge line/branch 100%
Phase I deliverables:
- Machine-readable current-surface contract matrix spec under version control:
- Docs/Planning/Algo2015CurrentSurfaceContractMatrix.json
- Updated staged artifacts:
- suites/inputfields/results.json
- suites/core-reqs/results.json
- field-by-field matrix observation artifact (TSV/JSON)
- Updated gap-closure report for the six real cpp-grounded areas (meal validation, reconciliation, transitions, subject parsing, bridge normalization, dead-path classification).
- Updated suite-assertion-trace-map.{json,md} linking these checks to TV-ALG-008 and related SRS-ALG-* tags.
4. Test Campaign Structure
4.1 Bridge Contract Suite (TV-ALG-001..003)
Goals: - validate null-pointer and malformed state handling - validate input field mapping and sentinel transformation - validate output and state handoff behavior
Core checks:
- NULL input/state/output short-circuit behavior
- stateData == NULL && timeStep > 0 reset behavior
- insulin request time 0 -> INVALID_REQUEST_TIME mapping
- subjectId nil and truncation behavior (MAX_SUBJECT_ID_LEN)
- state pointer/time-step increment and ownership expectations
4.2 Deterministic Golden Vector Suite (TV-ALG-004..006)
Goals: - lock behavior for canonical step-by-step scenarios - detect algorithm drift after code/config changes
Scenario classes:
- nominal in-range CGM sequence
- CGM unavailable/sentinel sequence (-1 path)
- meal-announcement scenarios by meal type/relative meal size
- pump available vs unavailable feedback inputs
- manual BG present/absent combinations
- target selection values (90,100,110,120,130)
Artifacts per vector: - scenario input CSV/JSON - expected output snapshot - actual output snapshot - diff summary
4.3 Stateful Sequence and Lifecycle Suite (TV-ALG-007)
Goals: - verify step/state continuity over long runs - verify persistence/reload boundaries
Checks:
- multi-step continuity (timeStep progression)
- state save/load across algorithm instance recreation
- restart-from-persisted-state determinism
- reset-to-fresh behavior
Current baseline implementation:
- Algo2015GoldenVectorTests.testPersistedStateReloadMatchesContinuousExecution
- Algo2015GoldenVectorTests.testResetToFreshStateProducesDeterministicStepZeroOutput
- evidence path: Docs/Quality/Evidence/STR-ALG-001/2026-02-18-tv-alg-007/
4.4 Boundary and Safety Sentinel Suite (TV-ALG-008)
Goals: - exercise clinically relevant boundaries and sentinel behavior
Checks:
- CGM range boundary (39, 401) and outside (<39, >401)
- pump unavailable sentinel mapping for requested/delivered insulin
- manual BG range boundaries used by runtime path
- invalid/missing history handling without unsafe command intent
Current transition note: - Prior staged implementation used alert-signal surrogates where raw warning codes were not exposed on the current bridge surface. - Phase I now treats raw warning-code parity as optional/separate bridge-seam work and instead closes cpp-grounded behavioral gaps on the exposed runtime contract.
4.5 Differential Configuration Suite (TV-ALG-009)
Goals: - verify pregnancy configuration controls alter algorithm input as intended - ensure no unintended regression in unchanged pathways
Checks:
- target variants map and execute deterministically
- meal upfront options (75%, 90%) impact path as expected
- TMAX variants (40...70, step 5) map as expected
- baseline-vs-changed comparison report for identical replay streams
4.6 Stress and Soak Regression Suite (TV-ALG-010)
Goals: - increase confidence for long-session runtime behavior
Checks: - overnight-length replay traces - repeated degraded/recovery transitions - repeated meal/BG intervention sequences
5. Coverage Measurement Method
Planned tooling:
- instrumented macOS static build of Algo2015 (clang coverage flags)
- deterministic test runner in BionicLoopCoreTests or dedicated harness target
- coverage extraction with llvm-profdata + llvm-cov
- staged verification orchestrator: Scripts/run_algo2015_verification.sh
Planned outputs:
- coverage.profraw
- merged coverage.profdata
- line/branch/function reports for Algo2015 + bridge sources
- machine-readable export (json/lcov) for archival and trend tracking
6. IDE Evidence Packaging
Evidence folder convention:
- Docs/Quality/Evidence/STR-ALG-001/<date>-algo2015-coverage/
Consolidated STR rollup:
- Docs/Quality/Evidence/STR-ALG-001/2026-02-18-rollup-v1/
Required contents:
- test protocol/version (STP reference)
- execution command log (tool versions + environment)
- pass/fail summary by TV-ALG-*
- coverage summary + full report artifacts
- uncovered-lines rationale log
- differential report for any config-change run
7. Completion Gates
Campaign is complete when all are true:
- all TV-ALG-* tests pass
- coverage targets in Section 3 are met (or justified exceptions approved)
- STR evidence package is generated and linked in RTM
- impacted RA/SRS/SDD/SVVP/RTM rows are updated
8. Traceability Links
- SRS:
SRS-ALG-001..007 - SDD:
SDD-ALG-001,SDD-QA-001 - SVVP:
TV-ALG-001..011 - RTM:
RA-014
9. Open Decisions
- Final threshold values for branch coverage acceptance if specific branches are unreachable in production path.
- Whether to treat this campaign as release-gating for every algorithm-change commit or for designated integration milestones only.
- Whether to archive raw replay vector payloads in-repo or in external controlled evidence storage.