Skip to content

Add Echo-TTS (community model) — draft, opening early per #54 - #180

Draft
5uck1ess wants to merge 8 commits into
0xShug0:mainfrom
5uck1ess:echo-tts-port
Draft

Add Echo-TTS (community model) — draft, opening early per #54#180
5uck1ess wants to merge 8 commits into
0xShug0:mainfrom
5uck1ess:echo-tts-port

Conversation

@5uck1ess

@5uck1ess 5uck1ess commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Draft — opening early per #54 to flag the area and get direction before writing the bulk of the port.

Adds Echo-TTS as a community model: an English
zero-shot voice-cloning TTS model. A 2.8B diffusion transformer generates 80-D latents in PCA space,
decoded to 44.1 kHz by the Fish S1-DAC autoencoder. Cloning needs a reference wav, no transcript.

Echo-TTS is on the candidate list in #34 (struck through, "contributions are welcome").

What's in this commit

Scaffolding only — no model math yet:

  • model_specs/echo_tts.json, schema_version: 1, using the generic spec-backed loader (no loader.cpp)
  • Session stub returning silence, so the family registers and links
  • docs/community_models/echo_tts.md

Verified on top of current main: registered_loaders goes 44 → 45, and echo_tts: clon (offline) appears in --list-loaders.

Plan

Each milestone is gated on cosine ≥ 0.999 parity against the reference PyTorch implementation on a
fixed seed, plus a listening check.

Milestone Scope
M0 Family registration + spec (this commit)
M1 GGUF conversion, DiT, PCA inverse, Fish decode
M2 Native speaker encoding (Fish encoder + RVQ) + long-form chunking
M3 Q8_0, RTF and VRAM evidence, docs

This PR stays in draft until the full evidence pack exists: build/run commands, parity numbers,
path-test results across safetensors/F16/Q8_0, RTF < 1.0, stable VRAM across repeated requests, and
generated wavs.

How long text is handled

Echo generates at most 640 latents (640 x 2048 / 44100 = 29.7215 s). That is a property of the
model, not the port: longer text is spoken faster to fit rather than truncated, and the tokenizer
hard-truncates past 768 UTF-8 bytes. Upstream's blockwise sampler subdivides that window rather
than extending it (sum(block_sizes) + continuation_length < 640), and upstream notes it "hasn't
been thoroughly tested".

So long text goes through the framework chunker, exactly as the rest of the repo does:

const int64_t text_chunk_size =
    engine::text::parse_text_chunk_size_override(request.options).value_or(kDefaultTextChunkSize);
const auto chunk_requests = runtime::chunk_text_request(request, text_chunk_size);
for (const auto & chunk_request : chunk_requests) { /* synthesize with cached conditioning */ }

engine::text::split_text_chunks is already consumed by 19 session.cpp files, and chatterbox
is the closest analogue — a clone family with cached speaker conditioning that concatenates chunk
audio via append_audio_buffer. Echo caches its speaker latent once per session, so timbre is
stable across seams by construction. Planned kDefaultTextChunkSize is 300 codepoints (~20 s
of English), leaving headroom before the model starts compressing; overridable at runtime.

I've deliberately not claimed long_form in capabilities — it appears nowhere in the C++, and
only 5 of 22 TTS/clone families declare it. The 17 that don't include chatterbox, fish_audio,
higgs_audio_tts, index_tts2, qwen3_tts, voxcpm2, and pocket_tts. Happy to add it if you'd
rather it be declared.

One question

Anything you'd want structured differently before there's a lot of code to restructure — file
layout, option naming, or whether this belongs in community_models at all.

Licence

Echo-TTS is CC-BY-NC-SA-4.0, and the restriction covers generated audio, not just the weights
(inherited from the Fish S1-DAC dependency). Flagging it explicitly rather than leaving it to be
inferred. There's precedent in-tree — fish_audio carries the identical restriction from the
identical dependency — and audio.cpp's own Apache 2.0 licence is unaffected, since weights are a
separate download. Documented in the model doc so users don't ship product on non-commercial output.

Why this model

Picked by comparing every model tracked in tts-bench — 62
local TTS models benchmarked across speed, objective scores, and blind human preference — against the
existing support table.

Measure Echo-TTS Field
Blind cloning Elo 1162 #3 of 40 (35 games, 738 cloning votes)
Speaker similarity 0.836 2nd of 41
UTMOS / WER 4.21 / 7.45 %
Frozen pairwise study 21-1-6 near-tied 1st of 28

Caveats worth stating: the cloning arena averages ~30 games per model, so gaps under ~100 Elo are
noise, and the ranking uses a single reference clip. Echo is top-3 on votes and 2nd on objective
SIM, which are independent measurements.

Compute profile suits a GGUF port — ~2.8B at 1.35× RTFx and 9.4 GB VRAM in PyTorch on an RTX 3090,
so there's real work to amortise.

Design for porting Echo-TTS (jordand/echo-tts-base, 2.8B DiT + Fish
S1-DAC) into audio.cpp as a community model.

Architecture verified against upstream source and safetensors headers,
not inferred. Key findings:
- EchoDiT: 24 blocks, d=2048, joint attention, adaLN, byte-level text
- Fixed 640-latent / 29.72s generation window
- Blockwise path subdivides that window, does not extend it
- Decode and encode need near-disjoint Fish submodules
- 303.6M of the Fish checkpoint is regenerable buffers, not weights

Staged M0-M4 with per-milestone gates and a hard Definition of Ready
before the PR leaves draft.
- Resolve RoPE theta open question (10000.0, complex-valued, model.py:9)
- Add timestep embedding formula
- Warn RTF vs RTFx are inverses (tts-bench vs audio.cpp conventions)
- Cite the actual schema validator for the M0 gate
- Define 'cosine' precisely (flattened 1-D, with max-abs-error)
- Add decomposition note: M0+M1 in one plan, M2/M3/M4 separate
13 tasks, each gated on executed evidence:
- M0 (T1-2): spec v1 registration + draft PR
- M1 (T3-13): converter, parity dumps, GGUF, assets, tokenizer,
  text/speaker encoders, 24-block DiT, dual-CFG Euler sampler,
  PCA inverse + Fish decode, crop, warm bench

Every stage gates on cosine >= 0.999 vs PyTorch before the next
begins. Task 12 requires a human ear check - tensor parity cannot
catch a wrong flattening-point crop.

PR stays draft through M1; cloning still needs an injected .npy
until M2 lands native speaker encoding.
- Repo has no CMakePresets.json; --preset would fail. Use
  scripts/build_linux.sh or cmake --build build/linux-cuda-release.
- Existing build tree pins CMAKE_CUDA_ARCHITECTURES=75 (Turing) on an
  sm_86 card. Task 13 now reconfigures to 86 before measuring RTF,
  otherwise the number is invalid.
- Note AUDIOCPP_MODEL_SET=full so the family compiles in automatically.
Baseline on this branch is registered_loaders=42, verified. Also note
that a 'requires a schema v1 model contract' failure means a stale
binary, not a broken tree.
Plan Task 1. Spec-backed loader (no loader.cpp), schema_version 1,
capabilities.clone deliberately omits long_form until M3 earns it.

Verified by execution: registered_loaders 42 -> 43, echo_tts appears
as 'clon (offline)', spec parses.

Fix over Codex's draft: guard used VoiceTaskKind::Tts, but the family
registers as a clone task, so every real invocation would have thrown.
Corrected to VoiceCloning, matching confucius4_tts:185. The
registration gate could not catch this - --list-loaders enumerates
loaders without constructing a session.
Documents the fixed 29.72s window, why blockwise does not extend it,
the CC-BY-NC-SA output restriction (with the fish_audio precedent),
benchmark provenance, options, and the WhisperD text format.

Also drops an unrelated .gitignore change that was accidentally
swept into an earlier docs commit, so the PR diff stays scoped.
The design spec and implementation plan are our working process, not
content for audio.cpp. Preserved on the local echo-tts-planning branch
and still on disk; just untracked here so the PR diff stays scoped to
the actual contribution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant