Skip to main content

📦 Export: Streaming COCO Dataset Generation

Summary​

Export reviewed events as COCO-format datasets for ML training. Downloads stream immediately as ZIP archives—no waiting for the full export to build. The system uses parallel workers to maximize throughput while bounded queues prevent memory overflow. Client disconnects trigger clean shutdown of all workers.

Use exports to:

  • Download training data for ML models
  • Archive reviewed events with annotations
  • Generate COCO datasets filtered by date/camera/category

API Usage​

Endpoint: GET /api/export/coco

Authentication: Required. Include Authorization: Bearer <token> header.

Example:

curl -H "Authorization: Bearer <token>" "https://api.example.com/api/export/coco?reviewed_at_start=2026-01-01T00:00:00+00:00&reviewed_at_end=2026-01-26T00:00:00+00:00&start_utc=2026-01-01T00:00:00+00:00&end_utc=2026-01-26T00:00:00+00:00&annotation_categories=fire&annotation_categories=smoke&annotation_categories=structure&limit=10000" -o export.zip

Response: ZIP archive containing coco.json + images/ directory

Parameters:

  • reviewed_at_start/end – Date range when events were reviewed (required)
  • annotation_categories – Annotation types to include (fire, smoke, structure, etc.)
  • start_utc/end_utc – Date range when events occurred (optional)
  • detection_categories – Filter by AI detection types (optional)
  • camera_ids – Filter by cameras (optional)
  • limit – Max events, default 10,000 (optional)

Single-Event Window Export​

Endpoint: GET /api/export/coco/window

Exports all events from a single camera in a relative time window around one event.

Parameters:

  • event_id – Central event ID (required)
  • secondsBefore – Seconds before the central event (required, 0-3600)
  • secondsAfter – Seconds after the central event (required, 0-3600)

At least one of secondsBefore or secondsAfter must be greater than 0.

Examples:

  • secondsBefore=600&secondsAfter=0 → export 600 seconds before the event timestamp
  • secondsBefore=0&secondsAfter=600 → export 600 seconds after the event timestamp
curl -H "Authorization: Bearer <token>" "https://api.example.com/api/export/coco/window?event_id=12345&secondsBefore=600&secondsAfter=0" -o window_export.zip

Architecture​

Flow:

  1. Database queries split into time chunks → GetEventsWorker threads (N workers)
  2. Workers fetch events → split metadata into 3 queues (images for download, images for JSON, annotations for JSON)
  3. Parallel processing:
    • ImagesDownloadWorker (M workers) downloads from blob storage
    • CocoJsonWorker (1 worker) generates COCO JSON file
  4. Both feed StreamFileRequestQueue → ZIP streamed to client as files arrive
  5. ShutdownCoordinatorWorker signals completion when all event workers finish

Backpressure: Image and stream queues are bounded (configurable via EXPORT_IMAGE_REQUEST_QUEUE_SIZE and EXPORT_STREAM_FILE_QUEUE_SIZE, both default to 100) to prevent memory overflow.