Show Don’t Tell: Teaching Robots to Detect Objects via Human Demonstration

How does a robot learn to recognize an object it's never encountered? In this case, a demo can be worth a thousand words. Short human demonstrations can be used to create fully automated training datasets, sidestepping the prompting limitations that hold back vision-language models. Rather than describing objects with language, the system tracks what a person touches and manipulates during a demo, follows those objects through time, and clusters detections to handle objects merging or splitting apart in the scene. This bypasses a core weakness of VLMs, which struggle to reliably detect unusual or novel objects even with repeated, carefully engineered prompts. The result? Task-specific detectors that are trained in minutes with zero manual labeling and outperform state-of-the-art vision-language models on both detection accuracy and real-world task success. Down the line, this work can help folks who are domain experts in other areas - mechanics, assembly line workers, etc. - more easily program robots without needing to become robotics experts themselves. Learn more about the “Show, Don’t Tell” approach here: https://rai-inst.com/resources/blog/s...