Researchers use generative simulation to unlock infinite training data for robots


Researchers present RoboGen, a generative robotic agent that automatically learns new skills in a generative simulation.

The work by researchers from CMU, Tsinghua IIIS, MIT CSAIL, UMass Amherst, and the MIT-IBM AI Lab aims to leverage recent advances in generative AI to generate infinite training data for automated robot learning.

According to the team, RoboGen is a generative robotic agent that learns various robotic tasks automatically and en masse through generative simulation. The team is using existing foundation models, such as OpenAI’s GPT-4, to “automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision.”

RoboGen uses GPT-4 for generative simulation

Specifically, RoboGen has an automated pipeline with four steps:



Task proposal: RoboGen selects a robot model and object and uses GPT-4 to generate tasks based on this information.

Scene generation: RoboGen creates simulation environments for the proposed tasks. This is done by selecting 3D objects from the Objaverse database and placing them in realistic arrangements using GPT-4.

Training supervision generation: RoboGen then uses GPT-4 to break down all the tasks into small steps. According to the team, this decomposition leads to less demanding tasks that can then be solved by algorithms such as reinforcement learning. After this step, GPT-4 selects an appropriate algorithm for each task.

Skill learning: Once the task, the scene in the simulation environment, the subtasks, and the algorithms have been selected, the scene and training in the simulation are started.

Video: Wang, Xian, Chen et al.



Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top