Speech and voice
Speech is made on your own machine from text, with a model's own voices or one cloned from a short clip, and clips are turned back into text. Both run as generation jobs, and both are reachable through the OpenAI-compatible Audio endpoints.
Available models
| Model | Codec | Rate | Size | Built-in voices | Registries |
|---|
The same catalog dppan models --tts reads. What a model can do follows from its codec:
qwen3tts | orpheus | f5 | |
|---|---|---|---|
| Voice comes from | a reference clip, required | a built-in name, or nothing | a reference clip and its transcript, both required |
| Languages | ten, chosen per request | fixed by the checkpoint | fixed by the checkpoint — eleven Indian languages for IndicF5 |
| Pull needs | --speech | --speech and --speech-decoder | --speech |
GET /v1/generations/capabilities reports each installed model's voices, languages and limits.
Getting a model
# Qwen3-TTS
dppan pull hf:ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF --speech
# Orpheus: name the decoder too, or the pull fails
dppan pull hf:PkmX/orpheus-3b-0.1-ft-Q8_0-GGUF --speech \
--speech-decoder hf:cstr/snac-24khz-GGUF:24khzA second Orpheus model needs its own decoder path, or it replaces the first one's pairing and the first disappears from dppan models --tts:
dppan pull hf:lex-au/Orpheus-3b-Hindi-FT-Q8_0.gguf --speech \
--speech-decoder hf:cstr/snac-24khz-GGUF:24khz \
--projector-out ~/.dppan/models/projectors/orpheus-hindi/snac-24khz.gguf
# Veena — Hindi and English from one checkpoint, including code-mixed input
dppan pull hf:Mungert/Veena-GGUF:Q8_0 --speech \
--speech-decoder hf:cstr/snac-24khz-GGUF:24khz \
--projector-out ~/.dppan/models/projectors/veena/snac-24khz.gguf--projector-out names the decoder file, in a directory of its own under projectors/.
Some models are converted while they install. The instruction-tuned Qwen3-TTS variants are published only as PyTorch checkpoints; --speech converts them, with nothing else to install:
dppan pull hf:Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --speechThe conversion needs about 8.4 GB of free disk and 4.8 GB of memory while it runs, takes about a minute, and shows its progress. Too little disk is refused before anything downloads.
IndicF5 is converted too, to one 0.93 GB file (2.3 GB free while it converts). Its repository is gated: accept the terms on the model page and set HF_TOKEN first.
dppan pull hf:ai4bharat/IndicF5 --speechAny other F5 repository needs its generation named once, at pull time: --f5-variant v0 for fine-tunes of F5TTS_Base, --f5-variant v1 for fine-tunes of F5TTS_v1_Base. The wrong one produces fluent, wrong audio, so a pull never guesses.
Installed models are served on the orchestrator's next start, with no configuration. List them against the catalog:
dppan models --ttsMODEL PIPELINE RATE SIZE SOURCES LOCAL
qwen3-tts-1.7b qwen3tts 24000Hz 2.3G hf ready (hf-ggml-org-Qwen3-TTS-12Hz-1.7B-Base-GGUF-Q8_0-full)
qwen3-tts-1.7b-customvoice qwen3tts 24000Hz 2.5G hf · converts ready (hf-Qwen-Qwen3-TTS-12Hz-1.7B-CustomVoice-full)
orpheus-3b-en orpheus 24000Hz 4.0G hf ready (hf-PkmX-orpheus-3b-0.1-ft-Q8_0-GGUF-Q8_0-full)
indicf5 f5 24000Hz 0.9G hf 🔒 · converts ready (ai4bharat-IndicF5)converts marks a model converted at install; 🔒 a gated repository. ready means the orchestrator will serve it, and the name in brackets is the id a request uses.
To serve a model from outside the models directory, name both halves:
export DPPAN_SPEECH_MODEL=/path/to/backbone.gguf
export DPPAN_SPEECH_MMPROJ=/path/to/mmproj.ggufSetting only one is ignored, with a warning.
Generating
curl -X POST http://localhost:8001/v1/generations \
-H 'Content-Type: application/json' \
-d '{"model":"qwen3-tts","modality":"audio",
"params":{"text":"Hello world","lang":"en"}}'voice ryan, lang en: “Welcome back. Today we're looking at three small ways to make your morning coffee better, starting with the water: fresh, filtered, and just off the boil.”Poll the job; when state is completed, output_artifacts[0] is a WAV at /v1/artifacts/:id. To follow it live, open GET /v1/generations/:id/events; disconnecting does not cancel the job.
With an OpenAI client, use POST /v1/audio/speech instead: it waits and returns the audio directly. See REST API → Audio.
Request parameters
| Field | |
|---|---|
text | What to speak. Required |
lang | Language — see Languages |
voice | A built-in voice name, or one of the deployment's named clips |
speaker_artifact | ID of an uploaded clip to clone the voice from |
reference_text | Exactly what the speaker_artifact clip says. Required by f5 models, refused by the others |
temperature, top_k, top_p, repetition_penalty | Sampling. Absent means the deployment's default. temperature at most 4, and 0 or below is greedy; top_k 0 or below and top_p 1 or above turn those stages off; top_p and repetition_penalty above 0, and repetition_penalty 1 turns it off. The dashboard holds its fields to temperature 0 to 4, top-k 0 to 1000 and top-p up to 1, and marks a value outside |
instruct | A sentence of style direction, e.g. "Very happy.". Only instruction-tuned models act on it |
max_duration_seconds | Ceiling on the finished audio |
speed | Pace: 2.0 is twice as fast. Only models that report a speed_range honour it (IndicF5: 0.3–2.0) |
Send voice or speaker_artifact, not both. Whether one is required depends on the model: requires_speaker and requires_reference_text in GET /v1/generations/capabilities say so per generator. Unknown fields are ignored.
An f5 model continues the reference clip's speech with your text, so it needs the clip's exact transcript:
curl -X POST http://localhost:8001/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model": "ai4bharat-IndicF5",
"input": "नमस्ते! संगीत की तरह जीवन भी खूबसूरत होता है।",
"speaker_artifact": "66815206-...",
"reference_text": "exactly what my-voice.wav says"}' \
-o hindi.wavreference_text. IndicF5 speaks in its voice, with lang mr: “हा आवाज एका छोट्या नमुन्यातून तयार केला आहे, आणि आता तो मराठीत बोलत आहे.” (“This voice was made from a short sample, and now it is speaking Marathi.”)- The clip: 5–10 seconds of clear speech, never over 15 s.
- The transcript: word for word, punctuation included. A wrong one costs quality.
- The text: a single character usually comes out silent — write
अ।rather thanअ. - Errors say what to fix: unsupported characters (emoji, Chinese) are named, and a silent result fails with the likely cause instead of returning an empty file.
Voices
A request gets a voice in one of three ways, all through voice or speaker_artifact:
Built into the model. Orpheus's English checkpoint has eight: tara, leah, jess, leo, dan, mia, zac, zoe. They are listed as builtin_voices in capabilities.
curl -X POST http://localhost:8001/v1/generations \
-H 'Content-Type: application/json' \
-d '{"model":"hf-PkmX-orpheus-3b-0.1-ft-Q8_0-GGUF-Q8_0-full","modality":"audio",
"params":{"text":"Hello world","voice":"tara"}}'Cloned, per request. Upload a clip and pass its ID as speaker_artifact:
curl -X POST http://localhost:8001/v1/files \
-F purpose=user_data -F [email protected]
# {"id":"66815206-...","filename":"my-voice.wav",...}Named clips, per deployment. Register clips under names; they appear in the dashboard and in chat:
export DPPAN_SPEECH_VOICES="warm=/srv/voices/warm.wav,clear=/srv/voices/clear.wav"Cloned and named voices need a qwen3tts model (or an f5 one, per request with its transcript). A few seconds of clean, single-speaker wav, mp3 or flac is enough; sample rate and channels do not matter. A clip sent to an orpheus model is refused, and an unknown voice is refused with the available names listed.
Speech in chat
Chat can speak only with a voice it needs no clip for: a model with built-in voices (Orpheus), or named clips registered with DPPAN_SPEECH_VOICES. A qwen3tts model on its own works on the Generate page and the API, but its chat speech control stays greyed out until you register a clip:
export DPPAN_SPEECH_VOICES="narrator=/srv/voices/narrator.wav"A wrong path shows in the startup log as ignoring speech voice ….
Languages
Qwen3-TTS takes ten: zh, en, de, it, pt, es, ja, ko, fr, ru; anything else fails the job. Read the list from capabilities rather than assuming it. An empty list means the language is fixed by the checkpoint (Orpheus), so the dashboard shows no language control.
Long text
Long text is split at sentence boundaries and spoken in pieces, joined into one WAV: one request, one artifact. The voice may shift slightly where pieces meet. max_duration_seconds caps the whole result.
Several models at once
Every installed model is served; the dashboard shows a picker when there is more than one. Models share a memory budget (DPPAN_SPEECH_MEMORY_BUDGET_MB, half of RAM by default): loading one that does not fit unloads the least recently used, which reloads on its next request.
Configuration
| Variable | Default | |
|---|---|---|
DPPAN_SPEECH_MODEL | unset | Backbone GGUF. Overrides discovery |
DPPAN_SPEECH_MMPROJ | unset | Its decoder GGUF, required alongside it |
DPPAN_SPEECH_MODEL_ID | qwen3-tts | The id requests use for that pair |
DPPAN_SPEECH_MAX_FRAMES | 512 | Length ceiling per piece (~41 s on qwen3tts, ~44 s on orpheus) — see Long text |
DPPAN_SPEECH_MEMORY_BUDGET_MB | half of RAM | Memory all loaded generators may hold together |
DPPAN_SPEECH_VOICES | unset | Named clips: name=/path/clip.wav, comma-separated |
DPPAN_SPEECH_TEMPERATURE | 0.9 | Default sampling temperature |
DPPAN_SPEECH_TOP_K | 50 | <= 0 disables it |
DPPAN_SPEECH_TOP_P | 1.0 | >= 1.0 disables it |
DPPAN_SPEECH_REPETITION_PENALTY | 1.05 | 1.0 disables it |
DPPAN_SPEECH_MAX_ATTEMPTS | 1 | Retries for a model that does not stop — see below |
DPPAN_SPEECH_SYNC_TIMEOUT_SECS | 120 | How long /v1/audio/speech waits before returning a job ID to poll |
DPPAN_MODELS_DIR | ~/.dppan/models | Where installed models are found |
Speech needs artifact storage to be available.
When a model will not stop
Some checkpoints occasionally never finish an utterance and repeat it. Such a job fails with did_not_terminate instead of returning the loop. A retry with a new seed usually succeeds; DPPAN_SPEECH_MAX_ATTEMPTS retries automatically, at the cost of several normal generations per failure. A model that does this often is better replaced.
Cancelling a running generation stops it at once and publishes nothing.
Speech to text
If the chat model understands audio (Gemma 4's E-series does), a voice clip goes straight to it. If it is text-only, add a transcriber such as qwen3-asr:1.7b — pull its projector too:
dppan pull hf:ggml-org/Qwen3-ASR-1.7B-GGUF:Q8_0
dppan pull hf:ggml-org/Qwen3-ASR-1.7B-GGUF:Q8_0 --projector-onlyFor a transcript on its own, use POST /v1/audio/transcriptions (OpenAI-compatible) — see REST API. language is a filter, not a hint: if the model reports a different language than the one you asked for, the transcript comes back empty. This catches near-silent clips, which otherwise come back as an invented sentence.
In the dashboard, the microphone attaches one clip to a message; the voice control beside it starts a hands-free conversation, and replies are read aloud when a speech model is installed.