Troubleshooting

Most runner issues fall into one of five buckets. Start with rbtk-runner doctor, then walk the list below.

Runner never picks up a job

A runner that heartbeats ONLINE but never receives jobs is almost always a capability mismatch.

Diagnose

# Get the job's required capabilities
rbtk test inspect <test-run-id>
# requires: { ros: humble, sim: gazebo-harmonic, gpu: true }

# Get the runner's declared capabilities
rbtk pool runners --pool prod-gpu-farm
# gpu-host-04: ros:humble,iron · sim:gazebo · gpu:T4   ← OK
# cpu-host-02: ros:humble        · sim:none  · gpu:no  ← skipped (no sim, no gpu)

Common mismatches

Job needs	Runner declares	Fix
`ros: humble`	`ros_distros: [iron]` only	Install Humble; add `humble` to `runner.yaml`
`sim: gazebo-harmonic`	`sim_engines: [gazebo-classic]`	Install Harmonic; switch the declaration
`gpu: true`	`gpu.enabled: false`	See GPU setup
label `ldra-licensed`	label not present	Add the label to the runner’s `capabilities.labels`

Reload runner.yaml or restart the service after changes.

MCAP upload fails

The runner uploads MCAP files via S3 presigned URLs that the platform mints just-in-time.

Symptom

[error] mcap upload failed: 403 SignatureDoesNotMatch
[error] mcap upload failed: 403 The request signature we calculated does not match

Causes and fixes

Symptom	Cause	Fix
`403 SignatureDoesNotMatch` immediately on first chunk	Host clock skew > 5 minutes	`sudo timedatectl set-ntp true`; verify with `timedatectl status`
`403 Request has expired` mid-upload	MCAP > 5 GB and upload slower than the 1-hour presign TTL	Switch to multipart upload (enabled by default in v2.2+); upgrade if pinned older
`connection refused` to S3 host	Outbound firewall blocking S3 endpoint	Allow `.s3.amazonaws.com` and `.s3.<region>.amazonaws.com`. Air-gapped: allow your on-prem object store
`407 Proxy authentication required`	Corporate proxy in the path	Set `HTTPS_PROXY` and `NO_PROXY=api.roboticks.io,s3.internal` in the systemd unit

Heartbeat lapse

The platform marks a runner OFFLINE after 60 s without a heartbeat.

Diagnose

# On the runner host
journalctl -u rbtk-runner --since "10 minutes ago" | grep -E '(heartbeat|error)'

# Network reachability
curl -fsS https://api.roboticks.io/healthz && echo OK

Common causes

Cause	Symptom	Fix
Runner token revoked	`401 unauthorized` on heartbeat	Re-register with a fresh registration token
Platform unreachable	`connection timed out`	Check firewall, DNS, proxy
Process killed (OOM)	`dmesg` shows `Out of memory: Killed process rbtk-runner`	Lower `max_concurrent_jobs` or add memory
Clock skew	Heartbeat 401s with `clock skew detected`	Sync NTP
Docker daemon down	Logs show `cannot connect to docker daemon`	`sudo systemctl start docker`

Version skew with the platform

rbtk-runner v2.x is wire-compatible with the latest platform. If the platform advances a wire-contract minor version, the runner emits a deprecation warning, and after 90 days a hard error.

Diagnose

rbtk-runner status

Version           v2.1.0
Latest stable     v2.4.1
Platform wire     v2.4.x (incompatible with runner v2.1.0 — upgrade by 2026-06-30)

Fix

Upgrade via the install path you used originally:

# One-liner reinstall
curl -fsSL https://get.roboticks.io/runner | bash

# Homebrew
brew upgrade rbtk-runner

# Chocolatey
choco upgrade rbtk-runner

Then drain and replace — see Run as a service.

Registration token issues

$ rbtk-runner register --token rbtk_pool_reg_xx...
Error: registration failed: token expired or already used

Registration tokens are single-use and live for 1 hour. Mint a new one:

rbtk pool register-runner --project warehouse --pool prod-gpu-farm

If you need to register many runners (e.g., autoscaling), pre-mint a batch with --ttl 24h and --uses 50 — these flags require Enterprise tier.

Job timeouts

A job killed at the job_timeout boundary surfaces as failed with reason runner_timeout.

Diagnose

rbtk test logs <test-run-id> --tail 100
# ... last visible test stdout ...
[runner] timeout after 30m, sending SIGTERM
[runner] container did not exit cleanly, sending SIGKILL

Fix

Increase resources.job_timeout in runner.yaml:

resources:
  job_timeout: 90m

Or override per-test with timeout: in .roboticks/test.yaml.

Docker permission denied

[error] failed to start container: permission denied while trying to connect to the Docker daemon socket

The runner user must be in the docker group:

sudo usermod -aG docker rbtk
sudo systemctl restart rbtk-runner

Disk space exhaustion

The runner cleans up containers and the work-dir between jobs, but it does not prune Docker images. Periodically:

docker system prune -af --filter "until=168h"

Or wire it into a weekly systemd timer.

Getting more signal

Crank logs to debug and re-run:

ROBOTICKS_LOG_LEVEL=debug rbtk-runner start

debug includes every HTTP request to the platform, the full docker run command for each job, and S3 multipart upload chunk timings.

Job stays `queued` indefinitely

A test job that never moves past queued almost always means the pool is at capacity. The scheduler counts every HostedRunner row in PENDING, LAUNCHING, or RUNNING against the pool’s max_concurrent_runners. If a row is stuck — usually because the underlying EC2 was terminated out-of-band, the spot request failed silently, or the runner crashed before reporting terminal status — the slot stays occupied forever and new jobs sit in the queue. The platform runs a reaper every 60 s that reconciles every active HostedRunner against EC2 reality and unblocks the pool when it finds a dead row:

Reaper observation	What it does
EC2 instance state is `terminated`, `stopping`, `stopped`, or `shutting-down`	Marks the runner row `STOPPED`, flips the linked `SelfHostedRunner` to `OFFLINE`, closes out the assignment, and fails the riding test job with attribution `"runner reaper marked job ERROR"`
EC2 instance is missing from `describe_instances` entirely	Same cleanup as above
EC2 has been alive >2 h	Hard age cap: terminates the EC2, then reaps the row
Tagged `Application=roboticks Purpose=test-execution` but no active runner row references it (orphan)	Terminates the EC2 directly — cost protection backstop

Operationally this means:

Wait one reaper tick (≤ 60 s) after the failure before assuming the pool is permanently stuck.
A job that the reaper marked ERROR will have its error message say so. That’s the signal to look at the runner’s CloudWatch logs (cloud) or journalctl -u rbtk-runner (self-hosted) for the root cause.
If the same root cause keeps recurring (your test_command references a binary that isn’t in the docker image, for example), the reaper will keep cleaning up while you keep submitting failing jobs. Fix the image, not the reaper.

Verifying the reaper is running

Backend ECS task logs include Starting hosted-runner reaper (interval: 60s, ...) at startup and Reaper tick N: reaped=X orphans_terminated=Y whenever it cleans up something. If you see Job queue worker stopped gracefully without a matching started line on a newer container, the deployment is in flux — wait for it to settle.

Still stuck?

Collect rbtk-runner doctor, rbtk-runner status, and the last 200 lines of the service log.
File at github.com/roboticks-io/roboticks-runner/issues with the bundle.
For paid plans, open a support ticket at support.roboticks.io referencing your org slug.

Configuration

Capabilities, resource limits, log level.

GPU setup

NVIDIA driver and container-toolkit issues.

​Troubleshooting

​Runner never picks up a job

​Diagnose

​Common mismatches

​MCAP upload fails

​Symptom

​Causes and fixes

​Heartbeat lapse

​Diagnose

​Common causes

​Version skew with the platform

​Diagnose

​Fix

​Registration token issues

​Job timeouts

​Diagnose

​Fix

​Docker permission denied

​Disk space exhaustion

​Getting more signal

​Job stays queued indefinitely

​Verifying the reaper is running

​Still stuck?

​Related

Configuration

GPU setup

Troubleshooting

Runner never picks up a job

Diagnose

Common mismatches

MCAP upload fails

Symptom

Causes and fixes

Heartbeat lapse

Diagnose

Common causes

Version skew with the platform

Diagnose

Fix

Registration token issues

Job timeouts

Diagnose

Fix

Docker permission denied

Disk space exhaustion

Getting more signal

Job stays `queued` indefinitely

Verifying the reaper is running

Still stuck?

Related