Drone Detection with Intel RealSense D435

Detect a drone with an Intel RealSense D435 and extract rich information per detection by combining a YOLO object detector with the camera's depth stream:

• Distance to the drone (meters), from the aligned depth map
• 3D position (X, Y, Z) in the camera frame, via depth deprojection
• Real-world size (width × height, meters) estimated from depth + intrinsics
• Tracking with a persistent ID across frames
• Velocity & speed (m/s) from the 3D track history
• Logging & recording: per-detection CSV + JSONL, annotated video, snapshots

Runs on a laptop (live OpenCV window) or headless on a Jetson (--headless).

---

1. How it works

D435 ─┬─ color (BGR) ─────────────► YOLO detect + track ─► boxes + track IDs
      └─ depth (aligned to color) ─► robust median depth in box ─► distance
                                          │
                                          ├─ deproject center px ─► (X, Y, Z)
                                          ├─ intrinsics + depth  ─► size (m)
                                          └─ track history       ─► velocity/speed
                                                                     │
                                            overlay + CSV/JSONL/video/snapshots

Depth is aligned to the color frame so a detected pixel maps to the correct depth. Distance uses the median of valid depths in the central part of the box, which rejects background and edge outliers.

2. The detection model (important)

COCO-pretrained YOLO has no drone class. To detect a real drone you need drone-trained weights:

• Recommended: obtain or train a YOLO model for drones and save it to models/drone.pt (this is the default detector.model_path). Public drone-detection datasets/models exist (e.g. on Roboflow Universe and Kaggle); export/train to an Ultralytics .pt file.
• To train your own on your specific drone, collect ~200+ labeled images and run Ultralytics training:

  yolo detect train data=drone.yaml model=yolov8n.pt epochs=100 imgsz=640

  then copy runs/detect/train/weights/best.pt to models/drone.pt.

If models/drone.pt is missing, the app falls back to a COCO model (yolov8n.pt, auto-downloaded) and loosely treats airplane/bird/kite as drone stand-ins — this just lets you verify the full pipeline end-to-end before you have real weights. It is not accurate drone detection.

3. Setup

Laptop (Linux/Windows/macOS)

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Connect the D435 to a USB 3 port, then verify it:

python scripts/check_camera.py

NVIDIA Jetson

• Install librealsense + pyrealsense2 from the JetPack build or from source.
• Install a Jetson-compatible PyTorch wheel (NVIDIA's index) before pip install ultralytics.
• Run with --headless if there's no display.

4. Run

# Live window (laptop)
python -m src.app

# Use your drone weights explicitly
python -m src.app --model models/drone.pt

# Headless (Jetson / SSH), also record annotated video
python -m src.app --headless --record

Press q to quit the GUI window, or Ctrl+C in headless mode.

5. Configuration

Edit config/config.yaml. Key options:

Section	Key	Meaning
camera	width/height/fps	D435 stream resolution and frame rate
detector	model_path	Your drone weights (.pt)
detector	conf / iou	Confidence and NMS thresholds
detector	drone_classes	Class names to keep (empty = all)
tracking	history_len	Samples per track used for velocity
output	record_video	Save annotated .mp4
output	snapshot_on_detect	Save a JPG the first time each track appears
runtime	headless	No GUI window

CLI flags --headless, --record, and --model override the config.

6. Output

• data/logs/detections_<timestamp>.csv — one row per detected drone per frame (timestamp, track_id, box, distance, X/Y/Z, size, speed, velocity)
• data/logs/detections_<timestamp>.jsonl — same data as JSON lines
• data/recordings/annotated_<timestamp>.mp4 — with --record
• data/snapshots/ — with snapshot_on_detect: true

7. Project layout

config/config.yaml     # all tunables
src/camera.py          # D435 pipeline, alignment, deprojection
src/detector.py        # YOLO detection + tracking (Ultralytics)
src/geometry.py        # robust depth, size estimation
src/tracking.py        # per-track history -> velocity/speed
src/recorder.py        # CSV/JSONL logs, video, snapshots
src/visualize.py       # overlays / HUD
src/app.py             # main loop (entry point: python -m src.app)
scripts/check_camera.py# verify the D435 is detected and streaming
scripts/selftest.py    # hardware-free math checks

8. Accuracy notes & limits

• The D435's depth range is roughly 0.3–10 m (best under ~6 m). Beyond that, distance/size/velocity get noisy or unavailable (reported as blank).
• Small, fast, distant drones are hard for both depth and detection — get real drone-trained weights and consider a larger YOLO model (yolov8s/m) if your hardware allows.
• Velocity is relative to the camera; if the camera moves, so does the frame.