Skip to main content

Device File Upload Integration Guide

This document describes how devices can upload large files (images, logs, recordings, etc.) to Roboticks sessions using the AWS IoT large payload ingestion pattern.

Overview

The file upload mechanism allows devices to upload files directly to S3 without going through MQTT message size limits (128KB). The device requests a presigned URL via MQTT, then uploads directly to S3 using HTTPS.

Architecture

┌─────────┐  1. Request URL   ┌──────────┐  3. Validate    ┌──────────┐
│ Device  │ ────────────────> │ IoT Rule │ ──────────────> │  Lambda  │
│         │                   │          │                 │          │
│         │  2. Subscribe     └──────────┘  4. Generate    └────┬─────┘
│         │ <────────────────────────────────── URL ────────────┘
│         │                                                      │
│         │  5. Upload File                                      │
│         │ ──────────────────────────────> ┌───────────┐       │
│         │                                 │    S3     │       │
└─────────┘                                 └───────────┘       │
                                                  │              │
                                            6. Validate session  │
                                               in PostgreSQL ────┘

MQTT Topics

Request Topic

Topic: roboticks/fleet/{device_id}/file_upload/request Where {device_id} is your device’s DSN (e.g., ROBOT-ABC123) Payload (JSON):
{
  "session_id": 123,
  "filename": "image_001.jpg",
  "content_type": "image/jpeg",
  "size_mb": 5
}
Required Fields:
  • session_id (integer): Active session ID for this device
  • filename (string): Target filename in S3
Optional Fields:
  • content_type (string): MIME type, defaults to application/octet-stream
  • size_mb (integer): Max file size in MB, defaults to 100MB

Response Topic

Topic: roboticks/fleet/{device_id}/file_upload/response Subscribe to this topic to receive the presigned URL response. Success Response:
{
  "success": true,
  "timestamp": "2025-01-12T10:30:00.000Z",
  "url": "https://roboticks-storage-123456.s3.amazonaws.com",
  "fields": {
    "key": "sessions/ROBOT-ABC123/123/image_001.jpg",
    "x-amz-algorithm": "AWS4-HMAC-SHA256",
    "x-amz-credential": "...",
    "x-amz-date": "...",
    "x-amz-signature": "...",
    "policy": "...",
    "Content-Type": "image/jpeg"
  },
  "object_key": "sessions/ROBOT-ABC123/123/image_001.jpg",
  "expires_in": 3600
}
Error Response:
{
  "success": false,
  "timestamp": "2025-01-12T10:30:00.000Z",
  "error": "Invalid session or session does not belong to device"
}

S3 Upload

Once you receive the presigned URL response, upload the file using HTTPS POST with multipart/form-data.

Using curl:

curl -X POST "${url}" \
  -F "key=${fields.key}" \
  -F "x-amz-algorithm=${fields['x-amz-algorithm']}" \
  -F "x-amz-credential=${fields['x-amz-credential']}" \
  -F "x-amz-date=${fields['x-amz-date']}" \
  -F "policy=${fields.policy}" \
  -F "x-amz-signature=${fields['x-amz-signature']}" \
  -F "Content-Type=${fields['Content-Type']}" \
  -F "file=@/path/to/your/file.jpg"

Using Python (boto3):

import requests

# After receiving presigned URL response
response = {
    "url": "https://...",
    "fields": {...}
}

# Upload file
with open('image.jpg', 'rb') as f:
    files = {'file': f}
    http_response = requests.post(
        response['url'],
        data=response['fields'],
        files=files
    )

if http_response.status_code == 204:
    print("Upload successful!")
else:
    print(f"Upload failed: {http_response.text}")

Using Python (requests):

import requests
import json

# Read presigned URL from MQTT response
presigned_response = json.loads(mqtt_payload)

if not presigned_response.get('success'):
    print(f"Error: {presigned_response.get('error')}")
    return

# Prepare multipart form data
files = {
    'file': open('image.jpg', 'rb')
}
data = presigned_response['fields']

# Upload to S3
response = requests.post(
    presigned_response['url'],
    data=data,
    files=files
)

if response.status_code == 204:
    print(f"File uploaded to: {presigned_response['object_key']}")

Security & Validation

The backend performs multiple security checks:
  1. Certificate Validation: Device must authenticate with valid X.509 certificate
  2. Device ID Validation: device_id in topic must match the certificate
  3. Session Validation: Session must exist and belong to the device (PostgreSQL query)
  4. S3 Path Isolation: Files are stored in device-specific paths: sessions/{device_id}/{session_id}/
All validations happen before presigned URL is generated. Invalid requests receive error responses on the response topic.

Required IoT Policy Permissions

Devices must have the following permissions in their IoT policy (already configured in RoboticksDevicePolicy):
{
  "Effect": "Allow",
  "Action": ["iot:Publish"],
  "Resource": [
    "arn:aws:iot:region:account:topic/roboticks/fleet/${iot:Connection.Thing.ThingName}/file_upload/request"
  ]
},
{
  "Effect": "Allow",
  "Action": ["iot:Subscribe"],
  "Resource": [
    "arn:aws:iot:region:account:topicfilter/roboticks/fleet/${iot:Connection.Thing.ThingName}/file_upload/response"
  ]
},
{
  "Effect": "Allow",
  "Action": ["iot:Receive"],
  "Resource": [
    "arn:aws:iot:region:account:topic/roboticks/fleet/${iot:Connection.Thing.ThingName}/file_upload/response"
  ]
}
The policy uses AWS IoT policy variables ${iot:Connection.Thing.ThingName} which automatically resolves to the device’s Thing Name at connection time, ensuring:
  • Devices can only publish/subscribe to their own file upload topics
  • Device isolation - no device can interfere with another device’s file uploads
  • Automatic authorization based on certificate

S3 File Structure

Files are organized by device and session:
s3://roboticks-storage/
└── sessions/
    └── {device_id}/          # e.g., ROBOT-ABC123
        └── {session_id}/     # e.g., 123
            ├── image_001.jpg
            ├── image_002.jpg
            └── logs/
                └── system.log

Frontend Access

Uploaded files are immediately visible in the Roboticks web interface:
  1. Navigate to Sessions page
  2. Click on a session
  3. Click the Files button
  4. Browse and download files
Files are secured - users must be authenticated and authorized to access session files.

Error Codes

Status CodeMeaning
200Presigned URL generated successfully
400Missing required fields or invalid format
403Authentication failed or unauthorized access
500Internal server error

Best Practices

  1. Subscribe before publishing: Subscribe to the response topic before publishing the request
  2. Handle timeouts: Set a timeout for waiting for the response (e.g., 5 seconds)
  3. Verify uploads: Check S3 POST response (204 = success)
  4. Reuse connections: Keep MQTT connection alive for multiple uploads
  5. File naming: Use unique filenames to avoid collisions (e.g., timestamp-based)
  6. Content types: Specify accurate content types for proper frontend display
  7. Size limits: Respect the size_mb parameter (default 100MB max)

Example: Complete Upload Flow

import json
import time
import requests
import paho.mqtt.client as mqtt

class FileUploader:
    def __init__(self, device_id, mqtt_client):
        self.device_id = device_id
        self.client = mqtt_client
        self.presigned_url = None

        # Subscribe to response topic
        response_topic = f"roboticks/fleet/{device_id}/file_upload/response"
        self.client.subscribe(response_topic)
        self.client.on_message = self._on_message

    def _on_message(self, client, userdata, msg):
        """Handle presigned URL response"""
        response = json.loads(msg.payload)
        if response.get('success'):
            self.presigned_url = response
        else:
            print(f"Error: {response.get('error')}")

    def upload_file(self, session_id, filepath, content_type="application/octet-stream"):
        """Upload file to session"""
        import os
        filename = os.path.basename(filepath)

        # Step 1: Request presigned URL
        request_topic = f"roboticks/fleet/{self.device_id}/file_upload/request"
        request_payload = {
            "session_id": session_id,
            "filename": filename,
            "content_type": content_type
        }

        self.presigned_url = None
        self.client.publish(request_topic, json.dumps(request_payload))

        # Step 2: Wait for response
        timeout = 5  # seconds
        start = time.time()
        while self.presigned_url is None and (time.time() - start) < timeout:
            time.sleep(0.1)

        if self.presigned_url is None:
            raise TimeoutError("No response from server")

        # Step 3: Upload to S3
        with open(filepath, 'rb') as f:
            files = {'file': f}
            response = requests.post(
                self.presigned_url['url'],
                data=self.presigned_url['fields'],
                files=files
            )

        if response.status_code == 204:
            print(f"✓ Uploaded: {self.presigned_url['object_key']}")
            return True
        else:
            print(f"✗ Upload failed: {response.status_code}")
            return False

# Usage
uploader = FileUploader("ROBOT-ABC123", mqtt_client)
uploader.upload_file(
    session_id=123,
    filepath="/tmp/image.jpg",
    content_type="image/jpeg"
)

Testing

Test the file upload flow using MQTT command-line tools:
# Subscribe to response topic
mosquitto_sub -h your-iot-endpoint.iot.us-east-1.amazonaws.com \
  --cert device.crt --key device.key --cafile root-CA.crt \
  -t "roboticks/fleet/ROBOT-ABC123/file_upload/response"

# Publish request (in another terminal)
mosquitto_pub -h your-iot-endpoint.iot.us-east-1.amazonaws.com \
  --cert device.crt --key device.key --cafile root-CA.crt \
  -t "roboticks/fleet/ROBOT-ABC123/file_upload/request" \
  -m '{"session_id": 123, "filename": "test.jpg", "content_type": "image/jpeg"}'

Support

For issues or questions:
  • Check CloudWatch Logs for Lambda execution logs
  • Verify session exists and is active
  • Ensure device certificate is valid and attached to IoT Thing
  • Contact support at support@roboticks.io