C281A3: Robot State Machine: Shoulder joint eneterd the Fault State

Technical Questions URScript
1

Hey there,

we are using an UR5e to assemble lithium-ion batteries by using the forces measured in its joints.
I have written a state machine in URScript telling the robot how to find the hole and center the battery.
But unfortunately some issues arise here and then - not every time:

  1. Sometimes while the robot is looking for the hole and trying to mount it, the following error message appears:
    C281A3: Robot State Machine: Shoulder joint eneterd the Fault State
    Additionally, the breaks are activated and the robot needs to be manually activated again. The whole programm stopped and I need to start it again.

  2. Furthermore, sometimes the robot enters a Protective Stop telling one of the threads used too much time. But I have tried to add sync() to the threads and functions where necessary. Although the popup is telling me I can find further information which thread is affected but I can not find any helpful information in the logs.

Some usefull information:

  • For the movements I have used servoj(get_inverse_kin(interpolated_pose), 0, 0, t=steptime, lookahead_time=10*steptime, gain=500) for example in

      def move_interpolated_path(pose_from, pose_to, speed=0.004):
      # param1 pose_from: p[m,m,m,rad,rad,rad], the starting pose of the path
      # param2 pose_to:   p[m,m,m,rad,rad,rad], the destination pose of the path
      # param3 speed:     [m/s], the speed the robot travels the path with
      # returns:          nothing
      # description:      move_interpolated_path takes a given start and destination pose.
      #                   It then interpolates the path between the two poses and moves
      #                   the robot accordingly.
      steptime = get_steptime()
      distance = point_dist(pose_from, pose_to)
      number_pose_interpolate = distance / (speed * steptime)
      delta_alpha = 1.0 / number_pose_interpolate
      alpha = 0.0
      
      interpolated_pose = p[0,0,0,0,0,0]
      while (alpha <= 1.0):        
          alpha = alpha + delta_alpha
          interpolated_pose = interpolate_pose(pose_from, pose_to, alpha)
          servoj(get_inverse_kin(interpolated_pose), 0, 0, t=steptime, lookahead_time=10*steptime, gain=500)
      end
      return None
  end

  • there are in total 6 threads running

As I said it works most of the time and only sometimes the errors above arise. Does anyone know how to deal with them.

Best regards,

Moritz

2

Hi Moritz,

We have been experiencing exactly the same problem on some robots we installed just before christmas which use an external force torque sensor and gripper. We have been communicating with UR support here in the UK over the last few weeks about the issue and it sems to be caused by the robot simply not having enough computational power to process everything it needs to in the required time. We have tried simplifying our program down, combining threads together, cutting out polyscope nodes which generate threads behind the scenes (eg. pulsed outputs, timer nodes, check expression continuously boxes, and unchecking the thread generating options in URCap nodes (such as continous grip detection from the grippers we are using).

We have configured the cells in which this issue is occuring with some additional safety, a light guard on one, and an area scanner on the other. The robot switches between a reduced velocity mode in which it does not monitor the safety, and full automatic mode, during which the safety is monitored and the robot is allowed to move faster. On one of the cells, we only experience this error when the cell is in automatic, we can run the robot in reduced velocity mode just fine. We are currently working with the developers of the URCap for the FT sensor and gripper we use to try and minimise the computational power used by the cap but have not yet been successful in eliminating the issue.

We hope this can be resolved soon as it is currently preventing production on the customer site from running at the required speeds.

3

Hi Sam,

thank you so much for the hint concerning the limited computational power of the UR5e! That indeed could be causing the issue, because additionally to our own threads, functions and state machines, we use the Robotiq Force Copilot URCap which is also running extensive code behind the scenes. Unfortunately I could not find an option in the Force Copilot to deactivate some threads we are not depending on in our application. Because from this URCap we involve only a fraction of its extent like:

  • Zero FT Sensor
  • Linear insert
  • rq_wait_ft_sensor_steady
  • rq_linear_search_urcap
  • rq_express_force_in_frame

I think I will contact the Robotiq support and check the URCap for further updates.

Have a good weekend,

Moritz

4

Your use case continues to back up the fact that this is the robot being overloaded with work. The gripper and FT sensor we encountered the problems with were not robotiq equipment, and as a result I suspect that this is a problem with polyscope. This is not a problem we have encountered with previous versions of polyscope and as a result we hope UR can make changers to reduce the load on the robots system resources. For reference, we have found this problem on UR10es, we haven’t looked to see if this still occurs on a CB3 robot.

5

Ok, thank you so much, Sam! Then I think I will contact UR itself.

6

We found the solution.
As you said, Sam, simplifiying the program did the trick

  • simplifying threads

  • merging several threads together

  • temporaily disable part of an extensive thread with a global variable e.g. reduce_cpu_effort:

    while True:
  if not reduce_cpu_effort:
      do some extensive stuff
      sync()
      do some extensive stuff
      sync()
      do some extensive stuff
  end	
  do some simple stuff
  sync()

    end

  • use sync to split up code parts

And even if the threads work in the first couple of minutes without causing a Protective Stop error, I noticed that they still can cause a Protective Stop error after 45 minutes or so. Then you have to split the thread by using sync()

Hope this helps somebody