Skip to content

Medium-Fidelity Harness Plan

Last updated: 2026-02-19 13:59 ET Owner: BionicLoop engineering Status: Planned

Goal

Build a deterministic, medium-fidelity simulation harness that exercises runtime logic using protocol-conformant mock CGM and pump services, without BLE transport dependency.

Primary outcomes:

  • reproducible safety-path regression testing
  • stronger pre-merge verification for cadence and degraded-mode logic
  • auditable STR-style evidence artifacts for quality traceability

Scope and Non-Goals

In scope:

  • deterministic scenario replay into runtime ports
  • virtual-clock control of step cadence and timing windows
  • scenario-based verification for TV-SIM-001 ... TV-SIM-005
  • STR-SIM evidence packaging (scenario, expected, actual, diff, results)

Out of scope (this phase):

  • BLE packet/session emulation
  • hardware transport timing fidelity
  • replacing hardware-in-the-loop verification

Design Anchors

  • Design source: /Users/jcostik/BionicLoop/Docs/Quality/SoftwareDesignDescription.md (SDD-SIM-001, Section 9)
  • Verification source: /Users/jcostik/BionicLoop/Docs/Quality/SoftwareVerificationAndValidationPlan.md (TV-SIM-001 ... TV-SIM-005)
  • Execution tracking source: /Users/jcostik/BionicLoop/Docs/Planning/ExecutionPlan.md (Workstream H)

Target Architecture

Core components:

  • VirtualClock
  • deterministic time progression, no wall-clock dependency
  • ScenarioRunner
  • ordered event dispatch into runtime interfaces
  • MockCGMService
  • protocol-conformant CGM event source
  • MockPumpService
  • protocol-conformant pump status/command-result source
  • RuntimeProbe
  • captures executed step, skip reason, algorithm I/O snapshots, command-apply decisions, alert lifecycle state

Planned placement:

  • test-only harness under app/core test targets (no production gating by default)
  • deterministic script entrypoint for repeatable execution in CI/local runs

Scenario Model (Draft)

Each scenario defines:

  • metadata: id, name, seed, description, requirements
  • initial state: runtime/session baseline + persisted state snapshot
  • events: timestamped list of domain events
  • expected outputs:
  • expectedStep, executedStep, skipReason
  • algorithm input/output expectations
  • command-application expectation (applied/blocked)
  • alert state expectations (active/recent)

Event classes:

  • CGM reading (value, timestamp, reliability flags, availability transitions)
  • pump status (idle, delivering, unknown, reconnect/disconnect)
  • command result (success, failure, reconciliation payload)
  • user action (mealAnnounce, bgCheck)
  • alert issue/retract events

Phase Plan

Phase 0: Contract and Artifact Lock

  • [ ] Approve scope, non-goals, and deterministic guarantees.
  • [ ] Freeze scenario schema and versioning rules.
  • [ ] Freeze STR-SIM artifact bundle layout and naming.
  • [ ] Define run context metadata fields (git SHA, seed, scenario set, generated timestamp, tool versions).

Exit criteria:

  • schema version v1 approved and documented
  • artifact template accepted by quality owner

Phase 1: Harness Core Scaffold

  • [x] Implement VirtualClock.
  • [x] Implement ScenarioRunner with deterministic event ordering.
  • [x] Implement protocol-conformant MockCGMService.
  • [x] Implement protocol-conformant MockPumpService.
  • [x] Implement RuntimeProbe recorder and export format.
  • [x] Add deterministic runner command for local/CI execution.

Implemented baseline:

  • /Users/jcostik/BionicLoop/BionicLoopCore/Tests/BionicLoopCoreTests/SimulationHarnessSupport.swift
  • /Users/jcostik/BionicLoop/BionicLoopCore/Tests/BionicLoopCoreTests/SimulationHarnessTests.swift

Exit criteria:

  • deterministic replay of a smoke scenario is stable across reruns
  • no wall-clock dependency in harness logic

Phase 2: First Campaigns (TV-SIM-001, TV-SIM-002)

  • [x] Implement cadence/anchor scenarios (TV-SIM-001).
  • [x] Implement step-0 gate + step>0 degraded CGM scenarios (TV-SIM-002).
  • [x] Assert step continuity and skip-reason correctness.
  • [x] Assert algorithm input mapping for degraded CGM sentinel behavior.

Exit criteria:

  • TV-SIM-001 and TV-SIM-002 passing in deterministic replay

Phase 3: Safety Path Campaigns (TV-SIM-003, TV-SIM-004)

  • [x] Implement pump unknown/unavailable command-block scenarios (TV-SIM-003).
  • [x] Implement meal/BG trigger interplay scenarios with missed steps and reconnect churn (TV-SIM-004).
  • [x] Assert no false command application in blocked conditions.

Implemented campaigns:

  • /Users/jcostik/BionicLoop/BionicLoopCore/Tests/BionicLoopCoreTests/SimulationHarnessTests.swift
  • testTVSIM001_AnchoredCadenceAcrossReconnectAndRelaunch
  • testTVSIM002_StepZeroGateAndStepGreaterThanZeroDegradedCGMExecution
  • testTVSIM003_PumpUnavailableAndUnknownStatesBlockCommandApplication
  • testTVSIM004_MealAndBGInterplayAcrossMissedStepsAndReconnectChurn

Exit criteria:

  • TV-SIM-003 and TV-SIM-004 passing with deterministic evidence

Phase 4: Alert Lifecycle Campaign (TV-SIM-005)

  • [x] Implement alert churn scenarios (issue/retract/precedence/dedupe).
  • [x] Validate time-sensitive countdown text progression at minute cadence.
  • [x] Validate clear/ack behavior and active/recent transitions under churn.

Implemented campaign:

  • /Users/jcostik/BionicLoop/BionicLoopTests/BionicLoopAlertTests.swift
  • testTVSIM005_AlertLifecycleChurnCountdownDedupeAndClearTransitions

Exit criteria:

  • TV-SIM-005 passing with deterministic evidence and stable ordering

Phase 5: Evidence Packaging and Merge Gating

  • [x] Emit STR-SIM-* package for each campaign run.
  • [x] Add RTM linkage for scenario IDs to SRS/RA/TV entries.
  • [x] Define merge gate requiring scenario pass for high-risk runtime changes.
  • [x] Add CodeReviewLog linkage for harness-impacting merges.

Implemented baseline:

  • Script: /Users/jcostik/BionicLoop/Scripts/run_sim_harness_verification.sh
  • Merge-gate helper: /Users/jcostik/BionicLoop/Scripts/check_sim_merge_gate.sh
  • Run output lanes:
  • Working: /Users/jcostik/BionicLoop/Docs/Quality/Evidence/Working/STR-SIM-001/...
  • Formal: /Users/jcostik/BionicLoop/Docs/Quality/Evidence/Formal/STR-SIM-001/...
  • Bundle artifacts:
  • run-context.json, evaluation-summary.json, manifest.json
  • suite-assertion-trace-map.json (TV-SIM-* -> SRS-*)
  • suites/core-sim/*, suites/alert-sim/*
  • Policy linkage:
  • /Users/jcostik/BionicLoop/Docs/Quality/CodeReviewLog.md now requires STR-SIM linkage in review entries for high-risk runtime merges.

Exit criteria:

  • reproducible STR-SIM package from one command
  • traceable evidence links in quality docs

Phase 6: High-Fidelity Emulator (Future Track)

  • [ ] Define BLE/session emulation requirements (transport timing and ACK/failure patterns).
  • [ ] Keep high-fidelity work out of near-term critical path.
  • [ ] Promote to active implementation only after medium-fidelity harness stability.

Exit criteria:

  • approved design proposal and effort estimate

Verification Mapping

  • TV-SIM-001: cadence anchoring and continuity
  • TV-SIM-002: CGM gate/degraded handling
  • TV-SIM-003: pump unavailable command-block behavior
  • TV-SIM-004: meal/BG trigger interplay
  • TV-SIM-005: alert lifecycle and countdown behavior

Risks and Mitigations

Risk: harness diverges from production runtime behavior.

  • Mitigation: only use production protocols and runtime entry points; avoid harness-only decision logic.

Risk: nondeterministic output due to implicit timing dependencies.

  • Mitigation: virtual clock only; explicit timestamps; deterministic event ordering.

Risk: evidence sprawl and poor traceability.

  • Mitigation: fixed artifact contract, run-context metadata, RTM linkage requirements.

Open Decisions

  • exact scenario DSL format (JSON vs typed Swift builder as source-of-truth)
  • CI lane placement and runtime budget for full campaign
  • threshold for mandatory pre-merge scenario set by change type