Smart Pick-and-Place on OMY with GraspNet and Gemini

<p><a href="https://www.youtube.com/watch?v=OxMBVGkFJHM">AI MANIPULATOR #14: Smart Pick-and-Place on OMY with GraspNet and Gemini</a></p> <p>An end-to-end smart pick-and-place pipeline built on the OMY robotic arm, integrating Gemini and GraspNet.</p> <p>Here is how the pipeline works:</p> <ol> <li>Scene Reasoning: Gemini interprets natural language instructions and identifies target objects from camera inputs.</li> <li>Grasp Generation: GraspNet processes 3D point cloud data to generate stable 6-DoF grasp candidates.</li> <li>Motion Execution: Cyclo Control, our QP-based motion controller, plans collision-free trajectories and executes inverse kinematics in real time.</li> </ol> <p>By combining VLM reasoning, data-driven grasp pose synthesis, and constrained motion control in ROS 2, OMY reliably picks arbitrary objects even in unstructured setups.</p> <p>Check out the full workflow, demo video, and open-source code below.</p> <p><img alt=":clapper_board:" class="emoji" height="20" src="https://emoji.discourse-cdn.com/noto/clapper_board.png?v=15" title=":clapper_board:" width="20" /> Video: <a href="https://youtu.be/OxMBVGkFJHM" rel="noopener nofollow ugc">https://youtu.be/OxMBVGkFJHM</a></p> <p><img alt=":robot:" class="emoji" height="20" src="https://emoji.discourse-cdn.com/noto/robot.png?v=15" title=":robot:" width="20" /> Code:<br /> • <a class="inline-onebox" href="https://github.com/ROBOTIS-GIT/cyclo_control" rel="noopener nofollow ugc">GitHub - ROBOTIS-GIT/cyclo_control: Control packages for the ROBOTIS Physical AI lineup · GitHub</a><br /> • <a class="inline-onebox" href="https://github.com/ROBOTIS-GIT/open_manipulator" rel="noopener nofollow ugc">GitHub - ROBOTIS-GIT/open_manipulator: AI Manipulator and Open Manipulator · GitHub</a></p> <p><img alt=":memo:" class="emoji" height="20" src="https://emoji.discourse-cdn.com/noto/memo.png?v=15" title=":memo:" width="20" /> Docs: <a class="inline-onebox" href="https://docs.robotis.com/docs/systems/omy/resources/technical_story/graspnet_pick_and_place/" rel="noopener nofollow ugc">Pick-and-Place on OMY with GraspNet and Gemini | ROBOTIS Docs</a></p> <p><small>1 post - 1 participant</small></p> <p><a href="https://discourse.openrobotics.org/t/smart-pick-and-place-on-omy-with-graspnet-and-gemini/57718">Read full topic</a></p>