[Announcement] ROS2 Visual Servoing Controllers

<p>Hello ROS community!</p> <p>Over the past few months, I have been working on the visual_servoing_controllers repository, a set of controllers for Visual Servoing tasks in ROS2 Jazzy. Specifically, it includes a Position-Based Visual Servoing and an Image-Based Visual Servoing controller. The controllers include an action server providing a parametrizable PBVS/IBVS task, mimicking the joint trajectory controller’s functionality.</p> <blockquote> <p><strong><img alt=":warning:" class="emoji" height="20" src="https://emoji.discourse-cdn.com/noto/warning.png?v=15" title=":warning:" width="20" /> Important:</strong> The controllers only manage the kinematic part of the Visual Servoing tasks, leaving the visual detection outside the controller. The controllers rely on a stream of 2D/6D poses received through topics to calculate the robot movements, decoupling the target detection and tracking to facilitate the integration of different implementations.</p> </blockquote> <p>Thanks for reading, below there is some additional information about the controller:</p> <h2><a class="anchor" href="https://discourse.openrobotics.org#p-117266-motivation-1" name="p-117266-motivation-1"></a>Motivation</h2> <p>Visual Servoing is a well-established technique that facilitates robot positioning in classic applications with stationary objects, as well as dynamic environments with moving objects. The encapsulation of Visual Servoing control in ROS2 controllers would facilitate the development of these types of applications, enabling the deployment of a simple pipeline consisting of a detection module that streams the estimated poses directly to ROS2 controllers.</p> <p>Additionally, the capability to parametrize Visual Servoing processes with custom task-oriented parameters such as destination, final tolerances, or timeouts, enables the utilization of the same controllers in distinct operations.</p> <h2><a class="anchor" href="https://discourse.openrobotics.org#p-117266-features-2" name="p-117266-features-2"></a>Features</h2> <p>Both the Position-Based Visual Servoing and the Image-Based Visual Servoing controllers provide the following general features:</p> <ul> <li>ROS2 controllers including an action server that executes PBVS and IBVS tasks.</li> <li>The control law calculates a twist value, which is internally managed to generate joint positions using KDL to calculate the Jacobian matrix and Eigen to estimate the pseudo-inverse using SVD.</li> <li>Allows an <strong>open-loop</strong> mode in which the previously commanded joint positions are used instead of the joint positions from the state interfaces, avoiding the injection of hardware feedback latency and transport delays into the command generation loop.</li> <li>The controllers include the option to <strong>enable the action feedback</strong> to continuously publish the pose error and the twist both in camera and tip link of the kinematic chain.</li> </ul> <p>Additionally, the IBVS controller has this specific feature:</p> <ul> <li>Due to kinematic redundancies in Image-Based Visual Servoing (IBVS), this controller includes a feature to enable or disable specific camera motion axes during 2D image-plane error correction (e.g., restricting movement to X/Y translation or pure rotation only). This ensures a predictable, task-tailored motion and prevents undesired robot movements.</li> </ul> <h2><a class="anchor" href="https://discourse.openrobotics.org#p-117266-testing-3" name="p-117266-testing-3"></a>Testing</h2> <p>The controllers have been tested on an industrial setup consisting of a UR16e robot, an Igus linear track, and a Realsense D435 RGBD camera. These tests included the use of the controllers in multiple configurations using an ArUco marker and the <a href="https://github.com/fictionlab/ros_aruco_opencv" rel="noopener nofollow ugc"><strong>aruco_opencv</strong></a> package.</p> <h3><a class="anchor" href="https://discourse.openrobotics.org#p-117266-pbvs-controller-4" name="p-117266-pbvs-controller-4"></a>PBVS Controller</h3> <p>Position-Based Visual Servoing task following an ArUco marker with a UR16e robot.</p> <p><img alt="pbvs" class="animated" height="240" src="https://us1.discourse-cdn.com/flex022/uploads/ros/original/3X/a/7/a77921e7759d17c9bc0df5024500f916cbaf245a.gif" width="320" /></p> <p>Position-Based Visual Servoing task following an ArUco marker with a 7 DoF setup consisting of a UR16e robot mounted on a linear axis.</p> <p><img alt="pbvs_7_axis" class="animated" height="240" src="https://us1.discourse-cdn.com/flex022/uploads/ros/original/3X/5/e/5e077e19214f5f5d9108dfa528b26a79b0369080.gif" width="320" /></p> <h3><a class="anchor" href="https://discourse.openrobotics.org#p-117266-ibvs-controller-5" name="p-117266-ibvs-controller-5"></a>IBVS Controller</h3> <p>Image-Based Visual Servoing task following an ArUco marker with a UR16e robot, moving only on the X/Y translation axes of the camera.</p> <p><img alt="IBVS-trans" class="animated" height="240" src="https://us1.discourse-cdn.com/flex022/uploads/ros/original/3X/2/5/25f32cb81eb5f9d5aa141d3399cb6145b71b79cf.gif" width="320" /></p> <p>Image-Based Visual Servoing task following an ArUco marker with a UR16e robot, moving only on the X/Y rotation axes of the camera.</p> <p><img alt="IBVS-rot" class="animated" height="240" src="https://us1.discourse-cdn.com/flex022/uploads/ros/original/3X/0/6/06071777aac96109bc3f99a2f30310d49b702345.gif" width="320" /></p> <p>IBVS controller chained with the <a href="https://github.com/aitor-ibarguren/adaptive_admittance_controller" rel="noopener nofollow ugc"><strong>adaptive_admittence_controller</strong></a>, creating a compliant Visual Servoing.</p> <p><img alt="IBVS-admittance" class="animated" height="240" src="https://us1.discourse-cdn.com/flex022/uploads/ros/original/3X/5/7/573009b682e2fd719583c2b710540b6dabbdbe5c.gif" width="320" /></p> <h2><a class="anchor" href="https://discourse.openrobotics.org#p-117266-repo-6" name="p-117266-repo-6"></a>Repo</h2> <p>GitHub: <a class="inline-onebox" href="https://github.com/aitor-ibarguren/visual_servoing_controllers" rel="noopener nofollow ugc">GitHub - aitor-ibarguren/visual_servoing_controllers: ROS2 Visual Servoing controllers designed to provide actions for vision-guided tasks. · GitHub</a></p> <p>The README includes information about the configuration of the controllers.</p> <h2><a class="anchor" href="https://discourse.openrobotics.org#p-117266-feedback-7" name="p-117266-feedback-7"></a>Feedback</h2> <p>I would appreciate feedback on:</p> <ul> <li>API design</li> <li>Additional features <ul> <li>Multiple target IBVS to enable a 6D positioning by staking image Jacobians</li> </ul> </li> </ul> <p><small>3 posts - 2 participants</small></p> <p><a href="https://discourse.openrobotics.org/t/announcement-ros2-visual-servoing-controllers/57661">Read full topic</a></p>