How Inclavate works
The few ideas the rest of these docs build on.
The orchestrator, nodes and the dashboard
Inclavate runs three kinds of process:
- The orchestrator (
dppan-orchestrator) takes every request: chat, generation jobs and uploads. It sends the work to the machines that do it, runs a text model's first and last steps (its embeddings and output layer) itself, and reads the images, audio and video sent in chat. - Nodes (
dppan join) run a model's layers. A node dials out to the orchestrator, so it needs no open ports and no public address. - The dashboard (
dppan ui) is a web page for chat, generation and running the cluster. See The dashboard.
Speech and music models run on the orchestrator's machine, each loaded when a job needs it. Image and video models run there too, each in a process of its own, or on nodes that offer them.
A model split across machines
A text model is a stack of layers, and each node serves a range of them: a request passes through the nodes in order, and the orchestrator assembles the answer. When a node joins, the orchestrator gives it a range sized to its GPU, or you name the range with --layers. A node can download only the layers it serves, so a model bigger than any one machine still runs: see Sharded download. Machines need only outbound TCP to the orchestrator's ports 8001 and 9001. Setting it up: Multi-machine setup.
Generation jobs
Making media is asynchronous: submit a job, follow it, download what it produced. Speech, music, images, video and reading media as a job all run through the same queue, and a job survives a restart of the orchestrator.
state is queued, running, completed, failed, cancelled, or expired. Only completed fills output_artifacts, which you download as artifacts.
429 means the queue is full: retry later. 422 means nothing in this deployment makes that modality with that model: retrying will not help.
Cancelling stops a running job cleanly, leaving nothing half-written. A job past its deadline is stopped.
Each machine runs one music, image or video job at a time. Speech, voice cloning and transcription jobs go ahead of them in the queue and run beside one that is already running: transcription at once, speech and voice cloning on a Mac when their model fits in the memory left. Otherwise, and on a machine whose GPU has memory of its own, they wait for it to finish. A music, image or video job in turn starts once a reply already running has finished.
The endpoints are in the Generation and artifacts API; the queue's settings are under Configuration › Jobs.
Which kind of generator
GET /v1/generations/capabilities lists every generator this deployment serves, each with a modality and a kind:
{ "modality": "audio", "kind": "music", "model": "minimax",
"limits": { "max_duration_seconds": 10.0 } }Branch on kind, not modality: speech and music are both audio but take different parameters. A server that omits kind serves speech. Image and video generators have kind image or video and a visual block with the model's settings: see Images and Video.
Artifacts
An artifact is a stored file: an upload you reference later by ID, or a generator's result. It stays on the orchestrator; other machines in the cluster never receive it unless a job they run needs it.
state is pending, ready or failed; only ready can be read. digest is the SHA-256 of the file. Someone else's artifact answers 404, the same as a missing one. One upload is capped at 512 MiB. If the artifact directory cannot be used, artifact endpoints answer 503 and everything else keeps working.
The endpoints are in the Generation and artifacts API; where they are kept, and for how long, under Configuration › Artifacts.
What stays on which machine
- Every model runs on hardware you control, and nothing is sent outside your network.
- Images, audio and video sent in chat are read on the orchestrator; the other machines in the cluster never receive them.
- An image or video node is sent the prompts and input files of the jobs it runs, and what it returns is published as the result. Turn image and video nodes on only where you trust every machine that can reach the orchestrator: see Image and video nodes.