I’m starting a new project where I going to use a UR10 for picking objects that are in random places. So for that I have to use a camera, and the camera that I have at the moment is a basler aca1300gm-30. I was able to put the robot moving according to the detected object, i.e., tracking or following the object, but it is only in a 2D space. I am using python to program the robot and YOLO for object detection. What’s missing now is to have the robot go to the object and pick it up, but for that, I believe I need information about depth. So, my question is, how can I obtain that? How can I make the robot pick up objects from 2D images? Is there any example project or documentation available?
You won’t be able to achieve any depth measurements with a 2D camera, as you say yourself.
A vision setup like that most often use a “fixed” height, which means the camera sends X and Y coordinates to the robot, and then the robot moves to the X and Y with a Z coordinate above the fixed pick up height and the moves downwards towards the object to the fixed height.
Alternatively, if you’re using a UR10e, you can use the built-in FT sensor to move downwards until the robot “touches” the object.
Hi @efn
The problem is that the objects are at different distances to the camera. They can be at 0.5meters to 2. If I know the lens that I am using could it be possible to know that measurement?
I don’t know of a way to do that. You can maybe mount an optical sensor on the tool. Then when the robot moves downwards, it will register the object with the sensor and then stop and pick it up at that height?
Based on some research I did a a year or so ago. A Terabee 3d camera has a ToF sensor that would allow you to get the depth measurement you would need I believe you can run python programs with it as well. I’m just not sure how accurate the depth measurement is or how accurate you would need it to be. I hope this helps.
A fixed heigth along with the use of the -z direction UNTIL CONTACT, could be useful here, as the robot then would find the element with the X and Y-axis data from the camera, and move down on the z axis until it contacts with an object.
Universal Robots has a video on this application use, but without the camera.
Hello Diego. Theoretically, you would be able to calculate the depth from the 2D image. There a few libraries that can help you. This approach, of course, has some limitation. First of all, it is a camera (lens) setup. You are talking about the quite huge depth of the field. You can double check your setting e.g here Calculation depth of field . The second one, you need to be also sure that object has significant edges that are always on top, fairly illuminated, and robust contrast, that could be also painstaking task. Another think is the number of object types, I mean how many models you need to create to cover the objects portfolio. I know that Halcon (commercialized image processing library), also for Python import, is able to extract all 6DoF from 2D image but it has conditions. Take a look and inspire. Alternatively, I would suggest aby 3D system.
hi @ales.zikmund thank you for the reply and the help. The objects are approximately spherical. I’m going to go for an Intel RealSense D450 series camera. I think it’s the most suitable for this application.
Hey Diogo, have you managed to perform smooth tracking? I’m doing something similar and in my case, I want to center the detection with the camera but the robot jitters a lot due to the continuous update of the target location.
Use Stereo Cameras: Set up two cameras for depth calculation.
Depth Camera: Switch to a depth camera like Intel RealSense for direct depth data.
Estimate Depth Use techniques like Structure from Motion with a single camera (more complex).
After detecting objects with YOLO, combine the depth data to guide the robot for picking. Check ROS tutorials or GitHub for example projects.https://iqratechnology.com/
