> ## Documentation Index > Fetch the complete documentation index at: https://docs.roboticks.io/llms.txt > Use this file to discover all available pages before exploring further. # Runner Troubleshooting > Diagnose common runner failures — capability mismatch, MCAP upload errors, heartbeat lapse, version skew, registration token expiry. # Troubleshooting Most runner issues fall into one of five buckets. Start with `rbtk-runner doctor`, then walk the list below. ## Runner never picks up a job A runner that heartbeats `ONLINE` but never receives jobs is almost always a **capability mismatch**. ### Diagnose ```bash theme={null} # Get the job's required capabilities rbtk test inspect # requires: { ros: humble, sim: gazebo-harmonic, gpu: true } # Get the runner's declared capabilities rbtk pool runners --pool prod-gpu-farm # gpu-host-04: ros:humble,iron · sim:gazebo · gpu:T4 ← OK # cpu-host-02: ros:humble · sim:none · gpu:no ← skipped (no sim, no gpu) ``` ### Common mismatches | Job needs | Runner declares | Fix | | ---------------------- | ------------------------------- | --------------------------------------------------- | | `ros: humble` | `ros_distros: [iron]` only | Install Humble; add `humble` to `runner.yaml` | | `sim: gazebo-harmonic` | `sim_engines: [gazebo-classic]` | Install Harmonic; switch the declaration | | `gpu: true` | `gpu.enabled: false` | See [GPU setup](/runners/gpu-setup) | | label `ldra-licensed` | label not present | Add the label to the runner's `capabilities.labels` | Reload `runner.yaml` or restart the service after changes. ## MCAP upload fails The runner uploads MCAP files via S3 presigned URLs that the platform mints just-in-time. ### Symptom ``` [error] mcap upload failed: 403 SignatureDoesNotMatch [error] mcap upload failed: 403 The request signature we calculated does not match ``` ### Causes and fixes | Symptom | Cause | Fix | | ------------------------------------------------------ | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `403 SignatureDoesNotMatch` immediately on first chunk | Host clock skew > 5 minutes | `sudo timedatectl set-ntp true`; verify with `timedatectl status` | | `403 Request has expired` mid-upload | MCAP > 5 GB and upload slower than the 1-hour presign TTL | Switch to multipart upload (enabled by default in v2.2+); upgrade if pinned older | | `connection refused` to S3 host | Outbound firewall blocking S3 endpoint | Allow `*.s3.amazonaws.com` and `*.s3..amazonaws.com`. Air-gapped: allow your on-prem object store | | `407 Proxy authentication required` | Corporate proxy in the path | Set `HTTPS_PROXY` and `NO_PROXY=api.roboticks.io,s3.internal` in the systemd unit | ## Heartbeat lapse The platform marks a runner `OFFLINE` after 60 s without a heartbeat. ### Diagnose ```bash theme={null} # On the runner host journalctl -u rbtk-runner --since "10 minutes ago" | grep -E '(heartbeat|error)' # Network reachability curl -fsS https://api.roboticks.io/healthz && echo OK ``` ### Common causes | Cause | Symptom | Fix | | -------------------- | --------------------------------------------------------- | ------------------------------------------- | | Runner token revoked | `401 unauthorized` on heartbeat | Re-register with a fresh registration token | | Platform unreachable | `connection timed out` | Check firewall, DNS, proxy | | Process killed (OOM) | `dmesg` shows `Out of memory: Killed process rbtk-runner` | Lower `max_concurrent_jobs` or add memory | | Clock skew | Heartbeat 401s with `clock skew detected` | Sync NTP | | Docker daemon down | Logs show `cannot connect to docker daemon` | `sudo systemctl start docker` | ## Version skew with the platform `rbtk-runner` v2.x is wire-compatible with the latest platform. If the platform advances a wire-contract minor version, the runner emits a deprecation warning, and after 90 days a hard error. ### Diagnose ```bash theme={null} rbtk-runner status Version v2.1.0 Latest stable v2.4.1 Platform wire v2.4.x (incompatible with runner v2.1.0 — upgrade by 2026-06-30) ``` ### Fix Upgrade via the install path you used originally: ```bash theme={null} # One-liner reinstall curl -fsSL https://get.roboticks.io/runner | bash # Homebrew brew upgrade rbtk-runner # Chocolatey choco upgrade rbtk-runner ``` Then **drain and replace** — see [Run as a service](/runners/service#drain-then-replace-upgrades). ## Registration token issues ``` $ rbtk-runner register --token rbtk_pool_reg_xx... Error: registration failed: token expired or already used ``` Registration tokens are **single-use** and live for **1 hour**. Mint a new one: ```bash theme={null} rbtk pool register-runner --project warehouse --pool prod-gpu-farm ``` If you need to register many runners (e.g., autoscaling), pre-mint a batch with `--ttl 24h` and `--uses 50` — these flags require Enterprise tier. ## Job timeouts A job killed at the `job_timeout` boundary surfaces as `failed` with reason `runner_timeout`. ### Diagnose ```bash theme={null} rbtk test logs --tail 100 # ... last visible test stdout ... [runner] timeout after 30m, sending SIGTERM [runner] container did not exit cleanly, sending SIGKILL ``` ### Fix Increase `resources.job_timeout` in `runner.yaml`: ```yaml theme={null} resources: job_timeout: 90m ``` Or override per-test with `timeout:` in `.roboticks/test.yaml`. ## Docker permission denied ``` [error] failed to start container: permission denied while trying to connect to the Docker daemon socket ``` The runner user must be in the `docker` group: ```bash theme={null} sudo usermod -aG docker rbtk sudo systemctl restart rbtk-runner ``` ## Disk space exhaustion The runner cleans up containers and the work-dir between jobs, but it does **not** prune Docker images. Periodically: ```bash theme={null} docker system prune -af --filter "until=168h" ``` Or wire it into a weekly systemd timer. ## Getting more signal Crank logs to `debug` and re-run: ```bash theme={null} ROBOTICKS_LOG_LEVEL=debug rbtk-runner start ``` `debug` includes every HTTP request to the platform, the full `docker run` command for each job, and S3 multipart upload chunk timings. ## Job stays `queued` indefinitely A test job that never moves past `queued` almost always means the pool is at capacity. The scheduler counts every `HostedRunner` row in `PENDING`, `LAUNCHING`, or `RUNNING` against the pool's `max_concurrent_runners`. If a row is stuck — usually because the underlying EC2 was terminated out-of-band, the spot request failed silently, or the runner crashed before reporting terminal status — the slot stays occupied forever and new jobs sit in the queue. The platform runs a **reaper** every 60 s that reconciles every active `HostedRunner` against EC2 reality and unblocks the pool when it finds a dead row: | Reaper observation | What it does | | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | EC2 instance state is `terminated`, `stopping`, `stopped`, or `shutting-down` | Marks the runner row `STOPPED`, flips the linked `SelfHostedRunner` to `OFFLINE`, closes out the assignment, and fails the riding test job with attribution `"runner reaper marked job ERROR"` | | EC2 instance is missing from `describe_instances` entirely | Same cleanup as above | | EC2 has been alive >2 h | Hard age cap: terminates the EC2, then reaps the row | | Tagged `Application=roboticks Purpose=test-execution` but no active runner row references it (orphan) | Terminates the EC2 directly — cost protection backstop | Operationally this means: * **Wait one reaper tick (≤ 60 s)** after the failure before assuming the pool is permanently stuck. * A job that the reaper marked `ERROR` will have its error message say so. That's the signal to look at the runner's CloudWatch logs (cloud) or `journalctl -u rbtk-runner` (self-hosted) for the root cause. * If the same root cause keeps recurring (your `test_command` references a binary that isn't in the docker image, for example), the reaper will keep cleaning up while you keep submitting failing jobs. Fix the image, not the reaper. ### Verifying the reaper is running Backend ECS task logs include `Starting hosted-runner reaper (interval: 60s, ...)` at startup and `Reaper tick N: reaped=X orphans_terminated=Y` whenever it cleans up something. If you see `Job queue worker stopped gracefully` without a matching `started` line on a newer container, the deployment is in flux — wait for it to settle. ## Still stuck? * Collect `rbtk-runner doctor`, `rbtk-runner status`, and the last 200 lines of the service log. * File at [github.com/roboticks-io/roboticks-runner/issues](https://github.com/roboticks-io/roboticks-runner/issues) with the bundle. * For paid plans, open a support ticket at [support.roboticks.io](https://support.roboticks.io) referencing your org slug. ## Related Capabilities, resource limits, log level. NVIDIA driver and container-toolkit issues.