How to Run Cosmos 3 for Robotics Training

Getting started with NVIDIA Cosmos 3 for robotics? This hands-on tutorial walks through running Cosmos3-Nano — the 16B model in NVIDIA's Cosmos 3 family of omnimodal world models for Physical AI. By the end you'll know how to install the framework, download the model, and run multiple modes including world generation, forward dynamics, and a robot policy model that predicts actions from a video and a text prompt. Cosmos 3 is built on a unified Mixture-of-Transformers (MoT) architecture that combines an autoregressive transformer for reasoning with a diffusion transformer for generation. A single model handles world generation (text-to-image, text-to-video, image-to-video), vision-language reasoning, forward dynamics, inverse dynamics, and policy — making it one foundation model for both generating physically plausible video and predicting robot actions. In this video I run it hands-on across real hardware, including an NVIDIA H100, and show what each mode actually produces. Learn more about Cosmos: https://nvda.ws/4x09UVX Download the model on HuggingFace: https://huggingface.co/collections/nv... What you'll learn: -What NVIDIA Cosmos 3 is and why it matters for Physical AI and robotics -A high-level look at the MoT architecture: the autoregressive reasoning tower and the diffusion generation tower -How to install and set up the Cosmos 3 framework -How to run world generation (text-to-video) -How to run forward dynamics on robot data -How to run the policy model to predict robot actions from an observation and a language instruction Prerequisites: NVIDIA GPU (Ampere, Hopper, or Blackwell); H100 80GB used in this video CUDA 13.0 (CUDA 12.8 also supported) Python 3.10+ A Hugging Face account with an access token (Read permission) Chapters 00:00 What Is Cosmos 3? One Model, Five Modes 00:50 Mixture-of-Transformers Architecture: Reasoner & Generator Towers 02:17 Nano vs. Super: Model Sizes and Where to Run 03:18 Setup: Install, CUDA 13, and Running on Brev with an H100 05:54 Demo 1: Text-to-Video Generation 13:10 Demo 2: Forward Dynamics (Predicting Future Video from Actions) 17:16 Demo 3: Robot Policy Model (Predicting Actions to Close a Drawer) 21:18 Recap: Why the Policy Model Matters for Robotics Data 22:33 Cosmos 3 Benchmarks and Leaderboards 26:31 Final Thoughts and How to Get Started FAQ Q: What is NVIDIA Cosmos 3? A: Cosmos 3 is a family of omnimodal world models from NVIDIA for Physical AI, built on a unified Mixture-of-Transformers architecture. A single model handles world generation (text-to-image, text-to-video, image-to-video), vision-language reasoning, forward dynamics, inverse dynamics, and robot policy. This tutorial uses Cosmos3-Nano, the 16B model. Q: What is the policy mode in Cosmos 3? A: Policy mode takes a starting observation (a video) and a language instruction, then predicts the sequence of robot actions needed to accomplish the task. It does not require an action file as input — the actions are the output. Q: What hardware do I need to run Cosmos3-Nano? A: An NVIDIA GPU with Ampere, Hopper, or Blackwell architecture. In this tutorial I run Cosmos3-Nano (16B) on an NVIDIA H100 80GB. Q: Where do I download the Cosmos 3 model? A: The model checkpoints are hosted on Hugging Face in the NVIDIA Cosmos 3 collection. You'll need a Hugging Face account and an access token with Read permission to download them. Q: What is the difference between forward dynamics and policy in Cosmos 3? A: Forward dynamics predicts future video given a starting observation and a sequence of actions. Policy does the reverse direction of reasoning — given an observation and a language instruction, it predicts the actions a robot should take.