Skip to main content

​
Coverage

Coverage in Roboticks is requirement coverage, not source-line coverage. The question is “is every requirement confirmed by a passing test on this SHA?”, not “how many source lines did the test suite touch?”. Both are useful; only the first is what an auditor asks about.

​
The state machine

Every requirement is in exactly one of five states per SHA:
StateDefined as
gapNo RequirementLink rows exist for this requirement on this SHA.
unconfirmed≥1 link exists; no run has produced a result yet.
confirmed≥1 link exists; all linked tests passed on this SHA; no prior-confirmed state was lost.
partial≥1 link exists; at least one linked test passed and at least one failed.
regressionThe most recent run on the parent branch (e.g. main) had this requirement confirmed; this SHA shows at least one failure on a confirming test.
State is computed deterministically — same inputs, same SHA, same answer. The audit pack includes a re-computation script that proves it.

​
Calculation inputs

For a given SHA, the inputs are:
  1. The requirement set in scope (the current roboticks/requirements.yaml or the snapshot for a release SHA).
  2. The RequirementLink rows in scope (the union of source-code-derived, manual, and accepted LLM links).
  3. The JUnit XML output of the test run on this SHA.
  4. The prior coverage state on the parent branch’s last successful run (for regression detection).
There are no inputs from outside these four buckets — coverage is reproducible from artifacts.

​
Examples

​
Single requirement, single test

EventRequirement state
REQ-014 created, no test existsgap
@confirms("REQ-014") added on test_estop (not yet run)unconfirmed
Run passesconfirmed
Code change breaks the testregression
Fix lands, test passes againconfirmed

​
Single requirement, two tests

EventRequirement state
Both tests passingconfirmed
One passes, one failspartial
Both failpartial (or regression if the requirement was confirmed previously)
One is removed, the other passesconfirmed

​
Requirement that has never been confirmed

EventRequirement state
Two confirming tests; both fail on first ever runpartial
One then passespartial (still one failing)
The failing one is fixedconfirmed
regression is reserved for requirements that were once confirmed. A first-time partial is just partial — no regression alarm.

​
Regression detection

Regression detection runs against the parent branch’s last successful Roboticks run, defined as:
  • For a PR, the base branch.
  • For a push to a release branch, the prior commit on the same branch.
  • For the first run on a new branch, the inherited state from the branch’s fork point.
If a requirement was confirmed in that reference run and is partial or has a failing confirming test in the current run, it becomes regression. This is the strongest negative signal Roboticks emits — it shows up red in the Check Run and the matrix, and is called out separately in the summary.

​
Hierarchy rollup

Parent requirements (those with descendants) get their state from their children:
All descendants are…Parent is…
confirmed (with non-test verification methods excluded)confirmed
Any gapgap
Any regressionregression
Any partial (no gap or regression)partial
Mixed unconfirmed + confirmed (no failures)partial
All unconfirmedunconfirmed
Verification methods other than test (i.e. inspection, analysis, demonstration) are excluded from rollup — they don’t drag a parent down to gap just because Roboticks can’t verify them by test. See Hierarchy.

​
Multi-repo rollup

A Roboticks project can host requirements that are confirmed by tests in multiple repos. The rollup is the same — a requirement is confirmed if any confirming test in any linked repo passes on the current SHA of its repo. A push to one repo only re-runs its tests; results from the other repos are read from their last-known-good run. See Multi-repo projects for the freshness rules.

​
Coverage percentage

The dashboard summary shows a per-requirement-type percentage: 
Safety:        42 / 51 confirmed  (82%)
Functional:   127 /137 confirmed  (93%)
Performance:    8 / 12 confirmed  (67%)
The denominator includes only requirements with verification_method: test. Inspection / analysis / demonstration requirements are reported separately as out-of-band verification. A weighted variant — ”% of ASIL-D requirements confirmed” — drives some auditor-facing dashboards. See Standards overview for the integration.

​
Live vs snapshot coverage

The matrix has a snapshot selector. The numbers it shows depend on which mode:
ModeWhat “confirmed” means
LiveCurrent state on the default branch’s latest run. Updated continuously.
SnapshotState at the moment the snapshot was cut. Immutable, never updates.
This is what makes the evidence pack reliable — the snapshot’s coverage numbers don’t drift between when you ship and when the auditor reads them three years later.

​
Where coverage updates trigger

TriggerWhat re-runs
New test run completes (PR or push)Update RequirementLink rows from JUnit, recompute states for the touched repo, recompute rollups.
Inline YAML edited and pushedRe-ingest, then recompute states (a new requirement starts at gap).
ReqIF re-importSame as above.
Link manually added in dashboardRecompute coverage for the affected requirement.
Snapshot cutFreeze the current state into the snapshot. Live state continues from there.

​
Re-computing coverage

For investigation, the CLI can recompute coverage from artifacts: 
rbtk coverage compute --sha abcd1234
This is a read-only operation — it doesn’t mutate the database — and is mostly useful for sanity-checking the platform against a custom expectation in an audit.

​
Next

Matrix

The UI that renders these states.

Gaps

What to do with a gap.

Change-impact

Affected-requirements detection per PR.

Snapshotting

Per-release coverage immutability.