Build autonomous robots
Develop robots that perceive, plan and act in changing surroundings by training and testing their behavior in simulation, then running perception and control on an onboard edge computer.
The problem
Fixed automation works when parts and paths never change. Mobile robots, arms that pick mixed items and humanoids must cope with clutter, people and variation, and collecting enough real-world data to train them is slow, expensive and sometimes unsafe.
Teams need a repeatable loop: simulate the robot and its surroundings, train or tune its policies, test them in software and with real hardware in the loop, then deploy to a compact onboard computer that runs perception and control within a tight power budget.
The approach
NVIDIA describes this as three computers: data center systems to train robot models, simulation systems to test them, and an onboard computer to run them. In its ecosystem, Isaac Sim builds simulated scenes from CAD or URDF and generates synthetic sensor data; Isaac Lab trains policies with reinforcement and imitation learning; Cosmos world models add synthetic video and scene reasoning; Isaac GR00T provides an open humanoid foundation model; and Isaac ROS supplies GPU-accelerated ROS 2 packages that run on Jetson.
ROS 2 itself, open source simulators such as Gazebo or MuJoCo, and classical motion planning remain valid choices and are often enough for well-structured tasks. Learned policies are worth the extra effort when tasks vary too much for hand-written rules.123456
Conceptual architecture
Diagram as a list
Applications & solutions
- Simulation and synthetic data (Isaac Sim)Runs the robot in physically based scenes and renders sensor dataConnects to Policy training (Isaac Lab) on data center GPUs
Models & frameworks
- Robot and scene models (CAD or URDF converted to USD)Describe the robot, its sensors and its surroundingsConnects to Simulation and synthetic data (Isaac Sim)
- World foundation models (Cosmos)Generate varied synthetic video and reason about scenesConnects to Policy training (Isaac Lab) on data center GPUs
- Robot policy or foundation model (for example Isaac GR00T)Maps observations to actionsConnects to Onboard computer (Jetson) with Isaac ROS
Operations & orchestration
- Software- and hardware-in-the-loop testsChecks policies against scenarios before field trialsConnects to Onboard computer (Jetson) with Isaac ROS
Accelerated computing
- Policy training (Isaac Lab) on data center GPUsTrains policies with reinforcement and imitation learningConnects to Robot policy or foundation model (for example Isaac GR00T)
- Onboard computer (Jetson) with Isaac ROSRuns perception, planning and control on the robot
Technologies and their roles
isaac1
Simulation, learning and robot software
Isaac Sim, Isaac Lab, Isaac ROS and Isaac GR00T cover simulation, policy training, onboard packages and humanoid models.
cosmos4
World models
Generate synthetic data and support reasoning and planning for physical AI.
jetson7
Onboard compute
Edge modules up to Jetson Thor run perception and policies within robot power budgets.
omniverse2
Simulation libraries
Isaac Sim is built on Omniverse libraries for rendering, physics and OpenUSD scenes.
dgx1
Training compute
NVIDIA lists DGX for building robot models in its three-computer approach.
What you need first
- Robot CAD or URDF models and sensor specifications
- ROS 2 and Python skills; reinforcement learning skills for learned policies
- RTX GPUs for simulation and training GPUs for policies
- A safety case and test plan for the physical robot
- Real-world data to measure the gap between simulation and reality
Risks and how to reduce them
- Policies trained in simulation fail on the real robot
- Randomize simulated conditions, run hardware-in-the-loop tests and fine-tune with real data.
- Physical safety around people
- Keep certified safety controllers and standards-based risk assessments; learned behavior does not replace them.
- Artifacts in generated training data4
- NVIDIA notes Cosmos 3 output can show temporal inconsistency and implausible dynamics; filter and review synthetic data.
- Early-stage components change2
- Some parts, such as the Newton physics engine, are in beta; pin versions and plan for API changes.
Related
Sources
- NVIDIA Isaac (developer page) (opens in a new tab)
- NVIDIA Isaac Sim (developer page and FAQ) (opens in a new tab)
- NVIDIA Isaac Lab (developer page) (opens in a new tab)
- NVIDIA Cosmos repository README (opens in a new tab)
- NVIDIA Isaac GR00T (developer page) (opens in a new tab)
- NVIDIA Isaac ROS (developer page) (opens in a new tab)
- NVIDIA Jetson embedded systems (product page) (opens in a new tab)
Thank you. Your correction was sent.
The editors check it against the sources. If you left an email address, they may reply about it.