More than one machine

Register machines once, then run a tested recipe, pool replicas, or shard one model across them.

A model can span machines in three ways, and all three start from the same registry of nodes.

Register the machines

node records a machine once; pool, shard and recipes launch then refer to it by name. Every name passed to --nodes must be registered, including the machine you run the command on. A node whose host is this machine is recognized as local and run without SSH.

vllm-helper node add spark1 --host http://spark1:8001 --ssh coder@spark1
vllm-helper node add spark2 --host http://spark2:8001 --ssh coder@spark2
vllm-helper node ls
vllm-helper node rm spark2

• --host is where the node answers OpenAI requests.
• --ssh is how this machine orchestrates it.

The registry lives in nodes.json beside your config.

Run a tested recipe

A recipe is a configuration that was run and measured on named hardware, with its image pinned by digest:

vllm-helper recipes list                                  # what is tested, and on what
vllm-helper recipes show <id>                             # image, argv, provenance, measured numbers
vllm-helper recipes launch <id> --nodes spark1,spark2     # rank 0, the first node, serves the API
vllm-helper recipes stop <id> --nodes spark1,spark2

recipes launch starts the containers over SSH on the nodes you name, writes a profile of the same name, and points Coder at it. From then on status, logs, restart and stop act on the recipe instead of the native service unit, and switch stops it before starting another profile. --dry-run prints the commands, with the API key redacted, and changes nothing.

Editing that profile is allowed and reaches the command line, but the entry then reports itself as the recipe plus local overrides and stops claiming the verified numbers. An override known to abort on the recipe's hardware is refused before any node is touched.

To keep a recipe serving across reboots:

vllm-helper recipes install-service

This installs vllm-cluster.service, which runs whatever recipe the active profile names, and stands the native vllm.service down because it cannot serve a container. A watchdog timer relaunches the cluster when the endpoint stops answering. A deliberate stop (vllm-helper stop, recipes stop, or stopping the unit) is remembered, so the watchdog leaves it stopped until the next launch.

A pool is replicas

Every node serves the same model, and a router in front of them spreads requests across the healthy ones:

vllm-helper pool serve --nodes spark1,spark2 --bind :8011 --sticky auto
vllm-helper pool install-service --nodes spark1,spark2   # the same thing, as a systemd unit
vllm-helper pool status                                  # per-backend health

Stickiness keeps a conversation on the node that already has its prefix cached. A repeated long prompt is nearly free on a warm node and expensive on a cold one. auto keys on an X-Session-ID header when the client sends one and falls back to the Authorization header, so a plain OpenAI client is still sticky per API key. authorization always uses the API key, and header:Name uses a header of your choosing.

A shard is one model split

For a model too large for any single node, shard runs one vLLM server across several:

vllm-helper shard install-ray --nodes spark1,spark2       # once, for the default Ray backend
vllm-helper shard start --profile big --nodes spark1,spark2 --tp 2 --pp 1
vllm-helper shard status --nodes spark1,spark2
vllm-helper shard stop --nodes spark1,spark2

--tp splits each layer across GPUs and --pp splits the model by layer. Their product is the number of GPUs the server expects, so on two single-GPU machines it must be 2. --pp defaults to 2 and --tp to 1. Which split is faster depends on the workload, so measure both.

shard start runs in the foreground against a native vLLM install and writes no state. That makes it the command for trying a split before there is a recipe for it.

• Run shard start on the head node: the first in --nodes, unless --head names another. The head's part runs locally and every other node is driven over SSH (pass --ssh-key if that takes a key).
• --backend mp uses vLLM's multiprocessing executor instead of Ray. It needs the head first in --nodes.
• --stop-services first stops the single-node unit (--service-name, vllm.service by default) on every node, and needs passwordless sudo there.
• --dry-run prints the commands without running them.

DGX Spark prerequisites

On a DGX Spark, vllm-helper spark doctor checks the packages vLLM's JIT builds need, and vllm-helper spark install-prereqs installs them with apt-get.

Next steps

• Recipes: the tested multi-machine configurations.
• CLI reference: every flag for node, pool and shard.