Hardware and memory
What runs where
Apple Silicon Macs use Metal, and NVIDIA GPUs use CUDA 12 on Linux and Windows. Everything also runs on the CPU, more slowly; Intel Macs run CPU-only. The operating system versions, drivers and GPU generations supported are under Installation › Platform requirements and Supported GPUs.
How a model is spread over machines
Each node detects its GPU when it joins, and the orchestrator gives it a range of layers sized to it; you can also name the range yourself with --layers. The machine running the orchestrator also does the model's first and last steps (its embeddings and output layer), so give it the smaller slice when your machines differ. How to plan a split: Sharded download › Picking a split.
Memory for chat
Every chat session keeps its context in memory on the nodes that serve it, for as long as the session lasts.
MAX_CONTEXT_TOKENSon the orchestrator (default 2048) is how much of a conversation is kept: past it, the oldest turns are dropped. Raise it for models made for long contexts.DPPAN_CTXon a node caps the context a session is given memory for. Lower it on a machine short of memory: each session then takes less. Keep it at least as high asMAX_CONTEXT_TOKENS, and raise the two together.
Memory for speech, music, images and video
- One shared budget: loaded speech, music, image and video models share
DPPAN_SPEECH_MEMORY_BUDGET_MB(half of RAM by default). When a model needs room, the least recently used idle one is unloaded, and loads again on its next request. - Images and video: each model runs in a process of its own, stays loaded between jobs for a while, and can be capped with
DPPAN_VISUAL_MAX_VRAM_GIB. How much a model needs, and how to fit one to your machine: Memory and the worker process, What to expect and Tuning a model for this machine. - Music:
--f16halves a model's size when you install it; see Smaller and faster. - Chat beside images and video: a machine serving a text model as well keeps memory free for its chat; see Image and video nodes.
Every setting named here is listed in Configuration.