Dance Dataset Builder

Build a training-ready dance dataset from a directory of source videos. The pipeline extracts audio and 3D poses, aligns them on a shared timing grid, segments each video on phrase boundaries, computes motion and audio descriptors, clusters segments, assigns train/val/test splits, and writes metadata for downstream model training.

The intended V2 consumer is an audio-conditioned motion model trained on pairs of mel spectrogram windows and normalized pose sequences.

Requirements

• Python 3.10+
• ffmpeg and ffprobe available on PATH
• MediaPipe-compatible CPU environment

Installation

uv sync --extra dev
uv run pre-commit install

Install training dependencies for the mobile baseline:

uv sync --extra train --extra dev

Configuration

The default config lives at configs/v1.yaml.

Key defaults:

• audio sample rate: 22050
• mel hop length: 512
• mel bins: 128
• phrase fallback length: 4 bars
• phrase stride: 1
• minimum mel energy: -40.0 dB
• minimum motion activity: 0.02
• clustering: 16 clusters
• dataset split ratios: 0.8 / 0.1 / 0.1

output_root controls where artefacts, arrays, and dataset metadata are written.

Pipeline Stages

The CLI orchestrates these stages in order:

1. 01 ingest video metadata into artifacts/01_video_manifest.jsonl
2. 02 extract WAV audio and video-level mel/beat/downbeat features
3. 03 extract MediaPipe world-coordinate poses and normalized poses
4. 04 run alignment checks between audio and pose timelines
5. 05 segment videos into phrase-aligned training examples and filter out low-energy / low-motion windows
6. 08 write per-segment arrays and dataset metadata
7. 06 compute motion features for each segment
8. 07 compute segment-level audio features
9. 09 cluster segments
10. 10 run dataset QA

The current run-all order in code is 01 -> 02 -> 03 -> 05 -> 08 -> 06 -> 07 -> 09 -> 10, with split assignment and weak text labels applied after clustering and before QA.

Each stage writes a checkpoint sentinel under <output_root>/artifacts/.checkpoints/ and is skipped on rerun unless --force is passed.

CLI Usage

Run the full pipeline:

uv run dance-pipeline run-all --config configs/v1.yaml --video-root ./videos

Run a single stage through the orchestrator:

uv run dance-pipeline run-all --config configs/v1.yaml --video-root ./videos --stage 05

Force a stage or full rerun:

uv run dance-pipeline run-all --config configs/v1.yaml --video-root ./videos --force

Launch the local preview app for generated segments:

uv run dance-pipeline review --config configs/v1.yaml

The review app uses one synchronized segment timeline for pose playback, mel playhead rendering, and audio playback. By default it plays the materialized segment clip from audio.clip_url; the UI can also switch to the parent-track context source from audio.context_url without changing the shared segment clock.

Pose playback defaults to world-travel view, where the stick figure moves across the canvas using the segment's absolute trajectory (hip midpoint in world coordinates). A toggle switches to body-centered view, which keeps the hip at the canvas center. When no trajectory artifact is available the UI degrades to body-centered rendering and labels the panel accordingly.

The review app serves on http://127.0.0.1:8000 by default. Override the bind address or port with --host and --port:

uv run dance-pipeline review --config configs/v1.yaml --host 0.0.0.0 --port 8123

This command expects generated dataset output under output_root, especially output/dataset/metadata.jsonl, so run the pipeline first.

If a pose or mel asset is missing, only that panel is expected to degrade while the rest of the review page remains usable. If timing metadata is missing from the review payload, synchronized playback is no longer guaranteed and the preferred behavior is to disable timeline-driven playback instead of falling back to browser repaint timing.

Standalone entrypoints are also exposed:

• scan-videos --config ... --video-root ...
• extract-audio --config ...
• extract-pose --config ...
• align --config ...
• segment --config ...
• build-dataset --config ...
• build-training-dataset --config ...
• train-baseline --config ...
• compute-features --config ...
• cluster --config ...
• qa --config ...

Training Sample Dataset

The assembled segment dataset is the source corpus, not yet the final model-facing contract. To build fixed-window samples for a mobile-oriented audio-to-dance model, materialize a separate training dataset from dataset/metadata.jsonl:

uv run dance-pipeline build-training-dataset \
  --config configs/v1.yaml \
  --audio-context-seconds 2.0 \
  --audio-frame-count 86 \
  --pose-history-seconds 1.0 \
  --pose-history-frame-count 20 \
  --pose-target-seconds 1.0 \
  --pose-target-frame-count 20 \
  --sample-stride-seconds 0.5

This writes:

• output/training/metadata.jsonl
• output/training/spec.json
• output/training/audio_context/*.npy
• output/training/pose_history/*.npy
• output/training/pose_target/*.npy

Each sample represents a fixed contract:

• audio input: prior mel window
• optional pose input: prior normalized pose window
• pose target: future normalized pose window

Current default contract:

• audio_context: [86, 128] for about 2.0s of mel context
• pose_target: [20, 33, 3] for about 1.0s of future motion at 20 FPS
• pose_history: [20, 33, 3] for about 1.0s of recent normalized motion context

Operational notes:

• build-training-dataset is deterministic for the same inputs and spec
• it overwrites current metadata and matching sample paths
• it does not yet clean stale sample files left behind by older specs
• it shows a segment-level progress bar and logs a final segment/sample summary

Mobile Baseline Training

The first baseline model is a mel-to-pose generator intended to stay small enough for eventual phone deployment:

• input: recent mel window from output/training/audio_context/
• input: recent pose history from output/training/pose_history/
• output: next normalized pose chunk from output/training/pose_target/

The training stack depends on PyTorch and is kept behind the train extra:

uv sync --extra train --extra dev
uv run dance-pipeline train-baseline --config configs/v1.yaml
uv run dance-pipeline eval-baseline --config configs/v1.yaml --split val --max-samples 16

By default this trains a lightweight Conv1D audio encoder plus GRU decoder. It uses pose_history as an anchor and predicts per-frame pose deltas, which are then accumulated into the future pose chunk. Checkpoints and metrics are written under output/models/mobile_baseline/.

The intended first deployment path is React Native -> ONNX Runtime Mobile. That is why the baseline sticks to simple export-friendly layers instead of heavier research-first architectures, while still using recent pose history as part of the conditioning signal.

Training notes:

• PyTorch is required only for baseline training, not for the dataset pipeline
• train-baseline uses epoch-level progress bars and logs per-epoch losses
• default training objective combines pose loss, velocity loss, and delta loss
• downstream.training.delta_loss_weight controls how strongly the model is pushed to move
• eval-baseline writes prediction/target .npy pairs plus metrics.json for inspection
• batch-level progress and ONNX export commands are not implemented yet

Recommended first run:

uv sync --extra train --extra dev
uv run dance-pipeline run-all --config configs/v1.yaml --video-root ./videos
uv run dance-pipeline build-training-dataset --config configs/v1.yaml
uv run dance-pipeline train-baseline --config configs/v1.yaml
uv run dance-pipeline eval-baseline --config configs/v1.yaml --split val --max-samples 16

Acceptance check after retraining:

• eval-baseline metrics should improve beyond the static-mean baseline
• exported predictions/*.npy should vary across frames, not stay almost constant
• the review viewer should show prediction moving instead of freezing in one pose

If that check passes, the next engineering steps are:

• add ONNX export for the current baseline checkpoint
• load the exported model from a small React Native inference spike
• measure on-device latency and memory for a single forward pass
• only then decide whether beat/downbeat features or a stronger decoder are worth the added mobile cost

Output Layout

Typical output tree under output_root:

output/
  artifacts/
    01_video_manifest.jsonl
    02_audio_index.jsonl
    03_pose_index.jsonl
    03_pose_quality.jsonl
    05_segment_index.jsonl
    motion_features.jsonl
    audio_segment_features.jsonl
    09_cluster_report.json
    10_qa_report.json
    10_qa_samples.json
    .checkpoints/
  audio/
  poses/raw/
  poses/normalized/
  trajectories/
  features/audio/
  features/motion/
  features/segment_audio/
  clusters/assignments.json
  dataset/metadata.jsonl
  training/
    metadata.jsonl
    spec.json
    audio_context/
    pose_history/
    pose_target/
  models/mobile_baseline/
  segments/

Dataset Record Format

Each line in dataset/metadata.jsonl is a DatasetRecord with:

• identity: segment_id, video_id
• timing: time_start, time_end, bpm, beat_timestamps
• phrase metadata: bars_per_phrase on segment records in artifacts/05_segment_index.jsonl
• array paths: mel_path, pose_raw_path, pose_norm_path, trajectory_path (optional)
• derived annotations: motion_features, audio_features, cluster_id, split
• QA and labeling: qa, segment_text_summary, cluster_text_summary

Tensor conventions:

• mel: [T_mel, 128]
• pose raw: [T_pose, 33, 3]
• pose normalized: [T_pose, 33, 3]

T_mel and T_pose are not resampled to match each other. Alignment is preserved through phrase timing plus beat-index metadata.

Development

uv run pytest tests/
uv run ruff check src tests
uv run mypy src

See docs/developer.md for module and schema details, and docs/runbook.md for operational guidance.