Skip to main content

​
Testing Framework

The Roboticks testing framework enables automated end-to-end testing of your robotics deployments. Tests run on actual hardware or simulated devices, validating that your modules and compositions work correctly.

​
Architecture

The test framework runs inside a Docker container alongside your compositions: 
+------------------------------------------------------------------+
|                        Docker Container                           |
|                                                                   |
|  +-------------------------------------------------------------+  |
|  |                  runner.py (Test Orchestrator)              |  |
|  |                                                             |  |
|  |  1. Start Device Manager (via Python bindings)              |  |
|  |  2. Wait for device registration                            |  |
|  |  3. Discover & start compositions                           |  |
|  |  4. Execute pytest                                          |  |
|  |  5. Report progress to platform                             |  |
|  |  6. Cleanup                                                 |  |
|  +-------------------------------------------------------------+  |
|                               |                                   |
|                               v                                   |
|  +-------------------------------------------------------------+  |
|  |                   Device Manager Process                    |  |
|  |                                                             |  |
|  |  +--------------+  +--------------+  +--------------+       |  |
|  |  | Composition  |  | Composition  |  |     ...      |       |  |
|  |  |      A       |  |      B       |  |              |       |  |
|  |  +--------------+  +--------------+  +--------------+       |  |
|  |                                                             |  |
|  |  ZMQ Publisher (port 5556) ---------------------------+     |  |
|  +-------------------------------------------------------------+  |
|                                                          |        |
|  +-------------------------------------------------------------+  |
|  |                       Your Tests                            |  |
|  |                                                             |  |
|  |  ZmqMonitor subscribes to 5556 <--------------------------+    |
|  |  Asserts on message content                                 |  |
|  |  Validates infrastructure                                   |  |
|  +-------------------------------------------------------------+  |
+------------------------------------------------------------------+

​
Key Components

Test Runner (runner.py)

The orchestrator that manages the entire test lifecycle - starts device manager, deploys compositions, runs pytest, and reports results.

ZMQ Monitor

Subscribe to ZeroMQ messages from your modules and assert on their content, timing, and structure.

Pytest Plugin

Automatically reports test progress to the Roboticks platform in real-time.

Packaging Script

Package your tests and the framework for upload to the platform.

​
Files from roboticks-tests Repository

The roboticks-tests repository provides the testing framework. The source files are located at src/roboticks_tests/:
FilePurposeWhen to Use
runner.pyMain test orchestrator - manages device manager lifecycle, runs pytestRequired for test-framework mode. Copy and customize for your needs.
zmq_monitor.pyZMQ message capture utility for test assertionsWhen testing module ZMQ output
pytest_roboticks_progress.pyPytest plugin for progress reporting to backendFor real-time test progress tracking
__init__.pyPackage exportsAlways included
All files in the roboticks-tests/src/roboticks_tests/ directory can be reused and customized. The runner.py is the most commonly customized file - you can modify it to add pre-test setup, custom device configuration, or integration with external systems.

​
Python Bindings (roboticks_device.so)

The test runner uses Python bindings to control the Device Manager in-process. These bindings are built from the Device Manager source and provide direct control over composition lifecycle.

​
Where the Bindings Come From

The roboticks_device.so file is built as part of the Device Manager release and included in the device manager artifact: 
device-manager-{version}.tar.gz
|-- bin/
|   +-- device-manager                 # Device manager binary
+-- lib/
    +-- python3.X/site-packages/
        +-- roboticks_device.so        # Python bindings

​
What the Bindings Provide

import roboticks_device

dm = roboticks_device.DeviceManager()
dm.start("/path/to/config.yaml")           # Start device manager
dm.wait_for_registration(timeout=30)       # Wait for backend registration
dm.discover_compositions("/opt/roboticks/deployment")  # Find compositions
dm.start_all_compositions(mode="prod")     # Start all compositions
dm.stop_all_compositions()                 # Stop compositions
dm.stop(timeout_seconds=5)                 # Stop device manager
The Python bindings are version-specific. Ensure the bindings version matches your Device Manager version. The bindings are automatically included when you download the device manager artifact from the platform.

​
Test Package Structure

When using the test framework as an orchestrator, your test package must follow this directory layout: 
/opt/roboticks/                            # ROBOTICKS_ROOT
|-- install/                               # Device Manager & Python bindings
|   |-- bin/
|   |   +-- device-manager                 # Device manager binary
|   +-- lib/
|       +-- python3.X/site-packages/
|           +-- roboticks_device.so        # Python bindings (REQUIRED)
|-- deployment/                            # Compositions and modules
|   |-- HelloWorldComposition/
|   |   +-- composition.yaml
|   +-- HelloWorldModule/
|       +-- HelloWorldModule
|-- device/                                # Device configuration
|   +-- device-config.yaml
|-- test-framework/                        # Test framework files
|   |-- runner.py                          # Entry point (REQUIRED)
|   |-- zmq_monitor.py                     # ZMQ utilities
|   +-- pytest_roboticks_progress.py       # Progress plugin
|-- tests/                                 # Your test files
|   |-- conftest.py
|   +-- test_*.py
+-- test-output/                           # Results (writable)
The test runner detects test-framework mode by checking for the existence of test-framework/runner.py. If this file is missing, the runner falls back to direct mode.

​
Execution Modes

The test runner supports two execution modes: When test-framework/runner.py exists, the runner delegates all orchestration to this script:
  1. runner.py starts the device manager via Python bindings (in-process)
  2. Waits for device registration with the backend
  3. Discovers and starts compositions from /opt/roboticks/deployment/
  4. Runs pytest with your test command
  5. Reports progress via the pytest plugin
  6. Cleans up device manager
This mode provides:
  • In-process device manager control via Python bindings
  • Automatic composition lifecycle management
  • Integrated progress reporting
  • Graceful cleanup on failure

​
Direct Mode

When test-framework/runner.py is absent but a device manager binary exists, the test runner uses a default execution script inside the Docker container:
  1. Test runner starts the Docker container with a default entrypoint script
  2. The script starts the device manager process
  3. Device manager discovers compositions automatically from the deployment directory using its config
  4. The script executes ROBOTICKS_TEST_COMMAND (e.g., pytest tests/ -v)
  5. The script stops the device manager when tests complete
Use this mode for:
  • Simpler test scenarios where device manager handles composition lifecycle
  • When you don’t need custom Python-based orchestration
  • Legacy test setups

​
Quick Start

​
1. Write Your Tests

Create test files in a tests/ directory. See the example at tests/test_example_hello_world.py: 
# tests/test_my_module.py
import pytest
import sys
from pathlib import Path

# Add test framework to path for ZMQ utilities
test_framework = Path("/opt/roboticks/test-framework")
if test_framework.exists():
    sys.path.insert(0, str(test_framework))

from zmq_monitor import ZmqMonitor

class TestMyModule:
    @pytest.fixture
    def zmq_monitor(self):
        monitor = ZmqMonitor()
        monitor.subscribe("tcp://localhost:5556", "/my/topic")
        monitor.start()
        yield monitor
        monitor.stop()

    def test_module_publishes_messages(self, zmq_monitor):
        """Verify module publishes expected messages."""
        msg = zmq_monitor.wait_for_message(
            topic="/my/topic",
            timeout=30.0
        )
        assert msg is not None, "Expected message not received"
        assert "data" in msg.payload

​
2. Package Your Tests

Use the packaging script to bundle your tests with the framework: 
./tools/package_tests.sh \
    --name my-integration-tests \
    --version 1.0.0
This creates build/test-package.tar.gz containing:
  • Test framework from roboticks-tests submodule
  • Your test files from tests/

​
3. Upload to Platform

roboticks test-package push \
    --file build/test-package.tar.gz \
    --name my-integration-tests \
    --version 1.0.0

​
4. Run Tests

roboticks simulate run \
    --fleet <fleet-id> \
    --test <test-package-id> \
    --deployment <deployment-id>

​
Environment Variables

The test runner uses these environment variables:
VariableDescriptionDefault
ROBOTICKS_ROOTBase path for roboticks installation/opt/roboticks
ROBOTICKS_PROJECT_SECRETProject secret for device registrationRequired
ROBOTICKS_API_URLBackend API URLhttps://api.roboticks.io
ROBOTICKS_TEST_JOB_IDTest job ID for progress reporting-
ROBOTICKS_COMPOSITIONTarget composition to test (optional)All compositions
ROBOTICKS_COMPOSITION_MODEComposition mode (prod/dev)prod
ROBOTICKS_TEST_COMMANDCustom pytest commandpytest /opt/roboticks/tests -v
ROBOTICKS_TEST_TIMEOUTTest timeout in seconds3600
ZMQ_TEST_ENDPOINTZMQ endpoint for monitoringtcp://localhost:5556

​
Next Steps

Writing Tests

Learn how to write effective tests using ZMQ monitoring and other techniques.

Test Runner Details

Deep dive into the runner.py orchestrator and customization options.

Packaging Tests

How to package and upload your tests to the platform.

Self-Hosted Runners

Set up runners to execute your tests.