Skip to main content

​
DynamoDB Cost Optimization - Device Logs

​
🔴 The Problem

Current Implementation: Device log Lambda writes EACH log entry individually 
# OLD CODE (Lines 80-97)
for log_entry in logs:
    item = { ... }
    table.put_item(Item=item)  # 1 write request per log!

​
Cost Impact

If a device sends 100 logs in a batch, this makes 100 separate DynamoDB write requests. DynamoDB On-Demand Pricing: $1.25 per million write request units (WRUs)
ScenarioLogs/Device/DayTotal Writes/MonthCost/Month
1 device, 1 log/min1,44043K$0.054
1 device, 1 log/sec86,4002.6M$3.24
5 devices, 1 log/sec432,00013M$16.20
50 devices, 1 log/sec4.3M130M$162.00
**This is why your Team Plan (99)hasterriblemargins∗∗−99) has terrible margins** - 162 in log costs alone!

​
✅ The Solution: Batch Writes

Use DynamoDB’s batch_write_item() API to write up to 25 items per request.

​
New Implementation

# NEW CODE - Batch writes (Lines 79-128)
# 1. Prepare all items
items_to_write = []
for log_entry in logs:
    items_to_write.append({ ... })

# 2. Write in batches of 25
for i in range(0, len(items_to_write), 25):
    batch = items_to_write[i : i + 25]
    dynamodb_client.batch_write_item(RequestItems={...})

​
Cost Savings

ScenarioOld WritesNew Writes (÷25)Old CostNew CostSavings
5 devices @ 1 log/sec13M520K$16.20$0.65$15.55 (96%) ✅
50 devices @ 1 log/sec130M5.2M$162.00$6.50$155.50 (96%) ✅
Cost reduction: ~96% (25x fewer writes)

​
Updated Plan Costs

​
Free Plan

ResourceOld CostNew CostSavings
DynamoDB Logs$1.32$0.053$1.27 (96%)
Other$0.43$0.43-
Total$1.75$0.48$1.27 ✅
New margin: Still lose $0.48/user but 73% better!

​
Personal Plan ($29/month)

ResourceOld CostNew CostSavings
DynamoDB Logs$6.60$0.26$6.34 (96%)
Other$1.46$1.46-
Total$8.06$1.72$6.34 ✅
Old Profit: 20.94(7220.94 (72% margin)** **New Profit: 27.28 (94% margin) ✅✅✅

​
Team Plan ($99/month)

ResourceOld CostNew CostSavings
DynamoDB Logs$66.00$2.64$63.36 (96%)
Other$12.10$12.10-
Total$78.10$14.74$63.36 ✅
Old Profit: 20.90(2120.90 (21% margin) ❌** **New Profit: 84.26 (85% margin) ✅✅✅ This is HUGE! Team plan went from barely profitable to highly profitable!

​
Real-World Impact

​
Scenario: 20 Personal + 10 Team Users

Before Optimization:
  • Fixed Costs: $116.63
  • Variable Costs: (20 × 8.06)+(10×8.06) + (10 × 78.10) = $942.20
  • Revenue: $1,570
  • Net Profit: $511.17 (32.6% margin)
After Optimization:
  • Fixed Costs: $116.63
  • Variable Costs: (20 × 1.72)+(10×1.72) + (10 × 14.74) = $181.80
  • Revenue: $1,570
  • Net Profit: $1,271.57 (81% margin) 🚀
Profit increase: +$760.40/month (+149%) ✅

​
Implementation Details

​
Key Changes to device-logs/index.py

  1. Added _convert_to_dynamodb_item() helper (Lines 26-68)
    • Converts Python types to DynamoDB format (S, N, M, L types)
    • Required for batch_write_item API
  2. Replaced individual writes with batch writes (Lines 79-128)
    • Collects all items first
    • Writes in chunks of 25 (DynamoDB limit)
    • Handles unprocessed items with retry
    • Falls back to individual writes on error

​
Error Handling

  • If batch write fails, automatically retries unprocessed items
  • If retry fails, falls back to individual put_item() for that batch
  • Ensures no log data is lost

​
Testing Recommendations

  1. Test with small batch (1-5 logs): Verify single batch write works
  2. Test with large batch (50+ logs): Verify chunking into multiple batches
  3. Test with invalid data: Verify fallback to individual writes
  4. Monitor CloudWatch logs: Check for any batch write errors

​
Additional Optimization Ideas

​
1. Log Sampling for Free Users

Only write every 10th log for free users: 
if is_free_tier and random.randint(1, 10) != 1:
    continue  # Skip 90% of logs
Additional savings: 90% fewer writes for free users

​
2. Aggregate Logs Before Writing

Combine multiple logs into a single item: 
{
  "device_id": "robot-123",
  "timestamp_us": 1234567890,
  "log_batch": [log1, log2, ..., log100],  # Store array
  "batch_size": 100
}
Savings: 100 logs = 1 write instead of 100 writes

​
3. Time-based Buffering

Buffer logs for 5 minutes, then write in one batch:
  • Use SQS to queue logs
  • Lambda processes queue every 5 minutes
  • Savings: Reduce Lambda invocations + more efficient batching

​
4. Use S3 for Log Archive, DynamoDB for Recent Only

  • Write last 24 hours to DynamoDB (queryable)
  • Archive older logs to S3 (cheap storage)
  • Use DynamoDB TTL (already configured)
  • Savings: ~50% DynamoDB storage costs

​
Cost Summary

MetricBeforeAfterImprovement
Free Plan Loss-$1.75/user-$0.48/user73% better
Personal Margin72%94%+22 pp
Team Margin21% ❌85% ✅+64 pp
20P+10T Profit$511$1,271+149%

​
Deployment

The fix has been applied to:
  • /infrastructure/lambda/device-logs/index.py
Next Steps:
  1. Test locally or in staging environment
  2. Deploy with cdk deploy
  3. Monitor CloudWatch logs for any batch write errors
  4. Check DynamoDB metrics to confirm 25x reduction in writes
Expected result: DynamoDB costs should drop by ~96% within 24 hours! 🎉