Combine AI with UR5

Hallo everyone,
In my project, I’m using UR5 to get the good tools that the camera classifies with AI.
Running out of ideas on how to combine UR5 and AI.
I need advice where to start.
my question is if the camera classifies the tools are good, how can I send a signal to the robot to understand that it will move the tools to the correct output positions??
Can this be done simply by connecting the camera as an input to the robot’s microcontroller and getting an output as a signal for the robot to move?

Thank you

Hello there,

I have seen most of these projects in research-related. As far as I’m aware, the most common approach in these projects is with ROS. You connect a separate Linux computer to the UR, which hosts the AI and ROS. Then you install the ROS-driver URCap on the UR, so it can be moved through a ROS node.

If you have never used ROS, I would recommend you to go through a crash or small course, so you learn how to use the nodes, services and additional packages, like Move it!, very common for moving the robot from point A to point B.
Knowledge on C++ and Python are a mandatory requirement in advance.

Here you have a master thesis using AI for Human-Robot-Interaction, which moves a two-UR5 arms, which behave as a single physical robot.

Here you have a crash online course with ROS, using simulation to go from a basic hello world to a path-planning exercise.

Thank you very much for your answer,
Do you think there are other ways to do this other than ROS.
I don’t know if it is possible to find multiple sources of ROS so that it can be learned.

I think there are other ways alternative to ROS, since AI and robotics are common nowadays. However, I don’t work in AI field professionally, so I cannot provide more information.
About learning ROS, it is open-source and a lot of AI projects research-related are also available under public licensing, so it is for sure easier to find tutorials/documentation/similar projects, there is a bigger community and their code can be re-used.

If you’re just looking to have your code talk to the robot, you probably don’t need to go the ROS route. The robot can talk to your code through a bunch of protocols including TCP socket (Most languages have socker servers), xmlrpc (super easy in python), plus most industrial protocols (I’m assuming you’re probably not dealing with these).

Does this sound like what you need?

  1. Robot polls a camera/AI server asking for the okay to do something
  2. Camera takes a picture, classifies object
  3. On success, the AI server now allows the robot to continue to do whatever it is you need it to

If you’re running this stuff on a device with digital outputs, you could use the digital output of your device to trigger a relay which would connect the robots 24V into one of it’s digital inputs. In this case the robot just sits there waiting for a high signal on that input.