Skip to content

Hardware and memory ​

What runs where ​

Apple Silicon Macs use Metal, and NVIDIA GPUs use CUDA 12 on Linux and Windows. Everything also runs on the CPU, more slowly; Intel Macs run CPU-only. The operating system versions, drivers and GPU generations supported are under Installation › Platform requirements and Supported GPUs.

How a model is spread over machines ​

Each node detects its GPU when it joins, and the orchestrator gives it a range of layers sized to it; you can also name the range yourself with --layers. The machine running the orchestrator also does the model's first and last steps (its embeddings and output layer), so give it the smaller slice when your machines differ. How to plan a split: Sharded download › Picking a split.

Memory for chat ​

Every chat session keeps its context in memory on the nodes that serve it, for as long as the session lasts.

  • MAX_CONTEXT_TOKENS on the orchestrator (default 2048) is how much of a conversation is kept: past it, the oldest turns are dropped. Raise it for models made for long contexts.
  • DPPAN_CTX on a node caps the context a session is given memory for. Lower it on a machine short of memory: each session then takes less. Keep it at least as high as MAX_CONTEXT_TOKENS, and raise the two together.

Memory for speech, music, images and video ​

  • One shared budget: loaded speech, music, image and video models share DPPAN_SPEECH_MEMORY_BUDGET_MB (half of RAM by default). When a model needs room, the least recently used idle one is unloaded, and loads again on its next request.
  • Images and video: each model runs in a process of its own, stays loaded between jobs for a while, and can be capped with DPPAN_VISUAL_MAX_VRAM_GIB. How much a model needs, and how to fit one to your machine: Memory and the worker process, What to expect and Tuning a model for this machine.
  • Music: --f16 halves a model's size when you install it; see Smaller and faster.
  • Chat beside images and video: a machine serving a text model as well keeps memory free for its chat; see Image and video nodes.

Every setting named here is listed in Configuration.

Free to run · Proprietary