Skip to content

Speech and voice ​

Speech is made on your own machine from text, with a model's own voices or one cloned from a short clip, and clips are turned back into text. Both run as generation jobs, and both are reachable through the OpenAI-compatible Audio endpoints.

Available models ​

ModelCodecRateSizeBuilt-in voicesRegistries

The same catalog dppan models --tts reads. What a model can do follows from its codec:

qwen3ttsorpheusf5
Voice comes froma reference clip, requireda built-in name, or nothinga reference clip and its transcript, both required
Languagesten, chosen per requestfixed by the checkpointfixed by the checkpoint — eleven Indian languages for IndicF5
Pull needs--speech--speech and --speech-decoder--speech

GET /v1/generations/capabilities reports each installed model's voices, languages and limits.

Getting a model ​

bash
# Qwen3-TTS
dppan pull hf:ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF --speech

# Orpheus: name the decoder too, or the pull fails
dppan pull hf:PkmX/orpheus-3b-0.1-ft-Q8_0-GGUF --speech \
  --speech-decoder hf:cstr/snac-24khz-GGUF:24khz

A second Orpheus model needs its own decoder path, or it replaces the first one's pairing and the first disappears from dppan models --tts:

bash
dppan pull hf:lex-au/Orpheus-3b-Hindi-FT-Q8_0.gguf --speech \
  --speech-decoder hf:cstr/snac-24khz-GGUF:24khz \
  --projector-out ~/.dppan/models/projectors/orpheus-hindi/snac-24khz.gguf

# Veena — Hindi and English from one checkpoint, including code-mixed input
dppan pull hf:Mungert/Veena-GGUF:Q8_0 --speech \
  --speech-decoder hf:cstr/snac-24khz-GGUF:24khz \
  --projector-out ~/.dppan/models/projectors/veena/snac-24khz.gguf

--projector-out names the decoder file, in a directory of its own under projectors/.

Some models are converted while they install. The instruction-tuned Qwen3-TTS variants are published only as PyTorch checkpoints; --speech converts them, with nothing else to install:

bash
dppan pull hf:Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --speech

The conversion needs about 8.4 GB of free disk and 4.8 GB of memory while it runs, takes about a minute, and shows its progress. Too little disk is refused before anything downloads.

IndicF5 is converted too, to one 0.93 GB file (2.3 GB free while it converts). Its repository is gated: accept the terms on the model page and set HF_TOKEN first.

bash
dppan pull hf:ai4bharat/IndicF5 --speech

Any other F5 repository needs its generation named once, at pull time: --f5-variant v0 for fine-tunes of F5TTS_Base, --f5-variant v1 for fine-tunes of F5TTS_v1_Base. The wrong one produces fluent, wrong audio, so a pull never guesses.

Installed models are served on the orchestrator's next start, with no configuration. List them against the catalog:

bash
dppan models --tts
MODEL                      PIPELINE       RATE      SIZE  SOURCES          LOCAL
qwen3-tts-1.7b             qwen3tts    24000Hz      2.3G  hf               ready (hf-ggml-org-Qwen3-TTS-12Hz-1.7B-Base-GGUF-Q8_0-full)
qwen3-tts-1.7b-customvoice qwen3tts    24000Hz      2.5G  hf · converts    ready (hf-Qwen-Qwen3-TTS-12Hz-1.7B-CustomVoice-full)
orpheus-3b-en              orpheus     24000Hz      4.0G  hf               ready (hf-PkmX-orpheus-3b-0.1-ft-Q8_0-GGUF-Q8_0-full)
indicf5                    f5          24000Hz      0.9G  hf 🔒 · converts  ready (ai4bharat-IndicF5)

converts marks a model converted at install; 🔒 a gated repository. ready means the orchestrator will serve it, and the name in brackets is the id a request uses.

To serve a model from outside the models directory, name both halves:

bash
export DPPAN_SPEECH_MODEL=/path/to/backbone.gguf
export DPPAN_SPEECH_MMPROJ=/path/to/mmproj.gguf

Setting only one is ignored, with a warning.

Generating ​

bash
curl -X POST http://localhost:8001/v1/generations \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-tts","modality":"audio",
       "params":{"text":"Hello world","lang":"en"}}'
Qwen3-TTS, voice ryan, lang en: “Welcome back. Today we're looking at three small ways to make your morning coffee better, starting with the water: fresh, filtered, and just off the boil.”

Poll the job; when state is completed, output_artifacts[0] is a WAV at /v1/artifacts/:id. To follow it live, open GET /v1/generations/:id/events; disconnecting does not cancel the job.

With an OpenAI client, use POST /v1/audio/speech instead: it waits and returns the audio directly. See REST API → Audio.

Request parameters ​

Field
textWhat to speak. Required
langLanguage — see Languages
voiceA built-in voice name, or one of the deployment's named clips
speaker_artifactID of an uploaded clip to clone the voice from
reference_textExactly what the speaker_artifact clip says. Required by f5 models, refused by the others
temperature, top_k, top_p, repetition_penaltySampling. Absent means the deployment's default. temperature at most 4, and 0 or below is greedy; top_k 0 or below and top_p 1 or above turn those stages off; top_p and repetition_penalty above 0, and repetition_penalty 1 turns it off. The dashboard holds its fields to temperature 0 to 4, top-k 0 to 1000 and top-p up to 1, and marks a value outside
instructA sentence of style direction, e.g. "Very happy.". Only instruction-tuned models act on it
max_duration_secondsCeiling on the finished audio
speedPace: 2.0 is twice as fast. Only models that report a speed_range honour it (IndicF5: 0.3–2.0)

Send voice or speaker_artifact, not both. Whether one is required depends on the model: requires_speaker and requires_reference_text in GET /v1/generations/capabilities say so per generator. Unknown fields are ignored.

An f5 model continues the reference clip's speech with your text, so it needs the clip's exact transcript:

bash
curl -X POST http://localhost:8001/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model": "ai4bharat-IndicF5",
       "input": "नमस्ते! संगीत की तरह जीवन भी खूबसूरत होता है।",
       "speaker_artifact": "66815206-...",
       "reference_text": "exactly what my-voice.wav says"}' \
  -o hindi.wav
The clip (Veena, Hindi)
The result (IndicF5, Marathi)
The clip says “नमस्ते! आज हम सुबह की कॉफ़ी को थोड़ा बेहतर बनाने के तीन आसान तरीक़े देखेंगे।”, and that is its reference_text. IndicF5 speaks in its voice, with lang mr: “हा आवाज एका छोट्या नमुन्यातून तयार केला आहे, आणि आता तो मराठीत बोलत आहे.” (“This voice was made from a short sample, and now it is speaking Marathi.”)
  • The clip: 5–10 seconds of clear speech, never over 15 s.
  • The transcript: word for word, punctuation included. A wrong one costs quality.
  • The text: a single character usually comes out silent — write अ। rather than अ.
  • Errors say what to fix: unsupported characters (emoji, Chinese) are named, and a silent result fails with the likely cause instead of returning an empty file.

Voices ​

A request gets a voice in one of three ways, all through voice or speaker_artifact:

Built into the model. Orpheus's English checkpoint has eight: tara, leah, jess, leo, dan, mia, zac, zoe. They are listed as builtin_voices in capabilities.

bash
curl -X POST http://localhost:8001/v1/generations \
  -H 'Content-Type: application/json' \
  -d '{"model":"hf-PkmX-orpheus-3b-0.1-ft-Q8_0-GGUF-Q8_0-full","modality":"audio",
       "params":{"text":"Hello world","voice":"tara"}}'

Cloned, per request. Upload a clip and pass its ID as speaker_artifact:

bash
curl -X POST http://localhost:8001/v1/files \
  -F purpose=user_data -F [email protected]
# {"id":"66815206-...","filename":"my-voice.wav",...}

Named clips, per deployment. Register clips under names; they appear in the dashboard and in chat:

bash
export DPPAN_SPEECH_VOICES="warm=/srv/voices/warm.wav,clear=/srv/voices/clear.wav"

Cloned and named voices need a qwen3tts model (or an f5 one, per request with its transcript). A few seconds of clean, single-speaker wav, mp3 or flac is enough; sample rate and channels do not matter. A clip sent to an orpheus model is refused, and an unknown voice is refused with the available names listed.

Speech in chat ​

Chat can speak only with a voice it needs no clip for: a model with built-in voices (Orpheus), or named clips registered with DPPAN_SPEECH_VOICES. A qwen3tts model on its own works on the Generate page and the API, but its chat speech control stays greyed out until you register a clip:

bash
export DPPAN_SPEECH_VOICES="narrator=/srv/voices/narrator.wav"

A wrong path shows in the startup log as ignoring speech voice ….

Languages ​

Qwen3-TTS takes ten: zh, en, de, it, pt, es, ja, ko, fr, ru; anything else fails the job. Read the list from capabilities rather than assuming it. An empty list means the language is fixed by the checkpoint (Orpheus), so the dashboard shows no language control.

Long text ​

Long text is split at sentence boundaries and spoken in pieces, joined into one WAV: one request, one artifact. The voice may shift slightly where pieces meet. max_duration_seconds caps the whole result.

Several models at once ​

Every installed model is served; the dashboard shows a picker when there is more than one. Models share a memory budget (DPPAN_SPEECH_MEMORY_BUDGET_MB, half of RAM by default): loading one that does not fit unloads the least recently used, which reloads on its next request.

Configuration ​

VariableDefault
DPPAN_SPEECH_MODELunsetBackbone GGUF. Overrides discovery
DPPAN_SPEECH_MMPROJunsetIts decoder GGUF, required alongside it
DPPAN_SPEECH_MODEL_IDqwen3-ttsThe id requests use for that pair
DPPAN_SPEECH_MAX_FRAMES512Length ceiling per piece (~41 s on qwen3tts, ~44 s on orpheus) — see Long text
DPPAN_SPEECH_MEMORY_BUDGET_MBhalf of RAMMemory all loaded generators may hold together
DPPAN_SPEECH_VOICESunsetNamed clips: name=/path/clip.wav, comma-separated
DPPAN_SPEECH_TEMPERATURE0.9Default sampling temperature
DPPAN_SPEECH_TOP_K50<= 0 disables it
DPPAN_SPEECH_TOP_P1.0>= 1.0 disables it
DPPAN_SPEECH_REPETITION_PENALTY1.051.0 disables it
DPPAN_SPEECH_MAX_ATTEMPTS1Retries for a model that does not stop — see below
DPPAN_SPEECH_SYNC_TIMEOUT_SECS120How long /v1/audio/speech waits before returning a job ID to poll
DPPAN_MODELS_DIR~/.dppan/modelsWhere installed models are found

Speech needs artifact storage to be available.

When a model will not stop ​

Some checkpoints occasionally never finish an utterance and repeat it. Such a job fails with did_not_terminate instead of returning the loop. A retry with a new seed usually succeeds; DPPAN_SPEECH_MAX_ATTEMPTS retries automatically, at the cost of several normal generations per failure. A model that does this often is better replaced.

Cancelling a running generation stops it at once and publishes nothing.

Speech to text ​

If the chat model understands audio (Gemma 4's E-series does), a voice clip goes straight to it. If it is text-only, add a transcriber such as qwen3-asr:1.7b — pull its projector too:

bash
dppan pull hf:ggml-org/Qwen3-ASR-1.7B-GGUF:Q8_0
dppan pull hf:ggml-org/Qwen3-ASR-1.7B-GGUF:Q8_0 --projector-only

For a transcript on its own, use POST /v1/audio/transcriptions (OpenAI-compatible) — see REST API. language is a filter, not a hint: if the model reports a different language than the one you asked for, the transcript comes back empty. This catches near-silent clips, which otherwise come back as an invented sentence.

In the dashboard, the microphone attaches one clip to a message; the voice control beside it starts a hands-free conversation, and replies are read aloud when a speech model is installed.

Free to run · Proprietary