Skip to main content

​
Troubleshooting

Most runner issues fall into one of five buckets. Start with rbtk-runner doctor, then walk the list below.

​
Runner never picks up a job

A runner that heartbeats ONLINE but never receives jobs is almost always a capability mismatch.

​
Diagnose

# Get the job's required capabilities
rbtk test inspect <test-run-id>
# requires: { ros: humble, sim: gazebo-harmonic, gpu: true }

# Get the runner's declared capabilities
rbtk pool runners --pool prod-gpu-farm
# gpu-host-04: ros:humble,iron · sim:gazebo · gpu:T4   ← OK
# cpu-host-02: ros:humble        · sim:none  · gpu:no  ← skipped (no sim, no gpu)

​
Common mismatches

Job needsRunner declaresFix
ros: humbleros_distros: [iron] onlyInstall Humble; add humble to runner.yaml
sim: gazebo-harmonicsim_engines: [gazebo-classic]Install Harmonic; switch the declaration
gpu: truegpu.enabled: falseSee GPU setup
label ldra-licensedlabel not presentAdd the label to the runner’s capabilities.labels
Reload runner.yaml or restart the service after changes.

​
MCAP upload fails

The runner uploads MCAP files via S3 presigned URLs that the platform mints just-in-time.

​
Symptom

[error] mcap upload failed: 403 SignatureDoesNotMatch
[error] mcap upload failed: 403 The request signature we calculated does not match

​
Causes and fixes

SymptomCauseFix
403 SignatureDoesNotMatch immediately on first chunkHost clock skew > 5 minutessudo timedatectl set-ntp true; verify with timedatectl status
403 Request has expired mid-uploadMCAP > 5 GB and upload slower than the 1-hour presign TTLSwitch to multipart upload (enabled by default in v2.2+); upgrade if pinned older
connection refused to S3 hostOutbound firewall blocking S3 endpointAllow *.s3.amazonaws.com and *.s3.<region>.amazonaws.com. Air-gapped: allow your on-prem object store
407 Proxy authentication requiredCorporate proxy in the pathSet HTTPS_PROXY and NO_PROXY=api.roboticks.io,s3.internal in the systemd unit

​
Heartbeat lapse

The platform marks a runner OFFLINE after 60 s without a heartbeat.

​
Diagnose

# On the runner host
journalctl -u rbtk-runner --since "10 minutes ago" | grep -E '(heartbeat|error)'

# Network reachability
curl -fsS https://api.roboticks.io/healthz && echo OK

​
Common causes

CauseSymptomFix
Runner token revoked401 unauthorized on heartbeatRe-register with a fresh registration token
Platform unreachableconnection timed outCheck firewall, DNS, proxy
Process killed (OOM)dmesg shows Out of memory: Killed process rbtk-runnerLower max_concurrent_jobs or add memory
Clock skewHeartbeat 401s with clock skew detectedSync NTP
Docker daemon downLogs show cannot connect to docker daemonsudo systemctl start docker

​
Version skew with the platform

rbtk-runner v2.x is wire-compatible with the latest platform. If the platform advances a wire-contract minor version, the runner emits a deprecation warning, and after 90 days a hard error.

​
Diagnose

rbtk-runner status

Version           v2.1.0
Latest stable     v2.4.1
Platform wire     v2.4.x (incompatible with runner v2.1.0 — upgrade by 2026-06-30)

​
Fix

Upgrade via the install path you used originally: 
# One-liner reinstall
curl -fsSL https://get.roboticks.io/runner | bash

# Homebrew
brew upgrade rbtk-runner

# Chocolatey
choco upgrade rbtk-runner
Then drain and replace — see Run as a service.

​
Registration token issues

$ rbtk-runner register --token rbtk_pool_reg_xx...
Error: registration failed: token expired or already used
Registration tokens are single-use and live for 1 hour. Mint a new one: 
rbtk pool register-runner --project warehouse --pool prod-gpu-farm
If you need to register many runners (e.g., autoscaling), pre-mint a batch with --ttl 24h and --uses 50 — these flags require Enterprise tier.

​
Job timeouts

A job killed at the job_timeout boundary surfaces as failed with reason runner_timeout.

​
Diagnose

rbtk test logs <test-run-id> --tail 100
# ... last visible test stdout ...
[runner] timeout after 30m, sending SIGTERM
[runner] container did not exit cleanly, sending SIGKILL

​
Fix

Increase resources.job_timeout in runner.yaml: 
resources:
  job_timeout: 90m
Or override per-test with timeout: in .roboticks/test.yaml.

​
Docker permission denied

[error] failed to start container: permission denied while trying to connect to the Docker daemon socket
The runner user must be in the docker group: 
sudo usermod -aG docker rbtk
sudo systemctl restart rbtk-runner

​
Disk space exhaustion

The runner cleans up containers and the work-dir between jobs, but it does not prune Docker images. Periodically: 
docker system prune -af --filter "until=168h"
Or wire it into a weekly systemd timer.

​
Getting more signal

Crank logs to debug and re-run: 
ROBOTICKS_LOG_LEVEL=debug rbtk-runner start
debug includes every HTTP request to the platform, the full docker run command for each job, and S3 multipart upload chunk timings.

​
Job stays queued indefinitely

A test job that never moves past queued almost always means the pool is at capacity. The scheduler counts every HostedRunner row in PENDING, LAUNCHING, or RUNNING against the pool’s max_concurrent_runners. If a row is stuck — usually because the underlying EC2 was terminated out-of-band, the spot request failed silently, or the runner crashed before reporting terminal status — the slot stays occupied forever and new jobs sit in the queue. The platform runs a reaper every 60 s that reconciles every active HostedRunner against EC2 reality and unblocks the pool when it finds a dead row:
Reaper observationWhat it does
EC2 instance state is terminated, stopping, stopped, or shutting-downMarks the runner row STOPPED, flips the linked SelfHostedRunner to OFFLINE, closes out the assignment, and fails the riding test job with attribution "runner reaper marked job ERROR"
EC2 instance is missing from describe_instances entirelySame cleanup as above
EC2 has been alive >2 hHard age cap: terminates the EC2, then reaps the row
Tagged Application=roboticks Purpose=test-execution but no active runner row references it (orphan)Terminates the EC2 directly — cost protection backstop
Operationally this means:
  • Wait one reaper tick (≤ 60 s) after the failure before assuming the pool is permanently stuck.
  • A job that the reaper marked ERROR will have its error message say so. That’s the signal to look at the runner’s CloudWatch logs (cloud) or journalctl -u rbtk-runner (self-hosted) for the root cause.
  • If the same root cause keeps recurring (your test_command references a binary that isn’t in the docker image, for example), the reaper will keep cleaning up while you keep submitting failing jobs. Fix the image, not the reaper.

​
Verifying the reaper is running

Backend ECS task logs include Starting hosted-runner reaper (interval: 60s, ...) at startup and Reaper tick N: reaped=X orphans_terminated=Y whenever it cleans up something. If you see Job queue worker stopped gracefully without a matching started line on a newer container, the deployment is in flux — wait for it to settle.

​
Still stuck?

​
Related

Configuration

Capabilities, resource limits, log level.

GPU setup

NVIDIA driver and container-toolkit issues.