Failures with multiple RTDE connections

Technical Questions Robot Communication
1

We are using UR10 running firmware 3.5.3.10825 and have hit a problem.

We have a main program which works fine and communicates with the robot in several ways:

  • We upload a script on port 30002 and read the replies for some time to see if the program started.
  • We have two RTDE connections - one to read/write registers to talk to the script and one to read joint states etc. for generic purposes.
  • We are using ATI force-torque sensor, which also mentions RTDE in some logs, but I don’t know the details.

As mentioned, this works fine. However, if we add the following program running from a different terminal (where RTDE is a class from a UR sample program):

connection = RTDE(hostname=robot_ip, port=30004)
connection.connect()
assert connection.send_output_setup(variables=('safety_status_bits',), types=('UINT32',), frequency=1)
assert connection.send_start()
while True:
    print(connection.receive().safety_status_bits)

things start to break in strange ways:

  • Sometimes it’s robot status “NO CONTROLLER”. What does it mean? Does it mean that the controller process crashed?
  • Sometimes it’s a protective stop due to “fieldbus input disconnected”. Our ATI F/T sensor used to give such errors from time to time, but now it’s happening all the time on the first move and the problem disappears after killing the first program.
  • Sometimes it’s just our URscript uploaded by the first program stopping prematurely.

when we stop this program, things return to normal.

If we remove reading the status from port 30002 whether the program started, it seems this simple program works fine, but when we add another RTDE class to read more data, things break again. Hence, is there some limit of how many RTDE connections we can have? Where can we start debugging what is going wrong?

2

The “NO CONTROLLER” is shown every time you start the robot, this indicates generally that the robot is not powered on.
If this happens randomly, this typically indicates that the URControl real-time process crashed. This may be due to overload.

“Fieldbus input disconnected” can happen, if a watchdog is configured in RTDE, but the dog is not kicked often enough, causing a watchdog timer reset. It could also happen if e.g. Ethernet IP or Profinet is enabled, but no client is active with either of these.

3

We have Ethernet IP and Profinet disabled and no RTDE watchdog configured. However, I don’t know if ATI F/T sensor doesn’t set up something? If there a way to check it?

The NO CONTROLLER happens randomly, so I guess this means a URControl overload? What contributes most to the load? The number of RTDE connections, the number of RTDE variables, the number of URscript threads, something else? What can we do to decrease the load?

4

The controller doesn’t seem to be overloaded (the CPU usage is 9% max), however, after a “Fieldbus disconnected” protective stop, an ATI scripts uses 100% of one core, so maybe it’s responsible for the overload - I had one NO CONTROLLER in such a situation.

A new development is that, I have a nice proof that our robot supports only 3 RTDE connections:

    def start_rtde(thread_id):
        from rtde.rtde import RTDE
        connection = RTDE(hostname=robot_ip, port=30004)
        connection.connect()
        assert connection.send_output_setup(variables=('safety_status_bits',), types=('UINT32',), frequency=1)
        assert connection.send_start()
        while True:
            try:
                print(connection.receive().safety_status_bits)
            except:
                print('In thread {}'.format(thread_id))
                traceback.print_exc()

    import time
    for i in range(10):
        print('Starting thread {}'.format(i))
        t = threading.Thread(target=start_rtde,
                             args=(i,),
                             daemon=True)
        t.start()
        time.sleep(5)

This code works fine for the first 3 threads, but then fails the 4th thread fails, with receive() returning None.

Thus, our current hypothesis is:

  • There is a limit of 3 RTDE connections.
  • When we reach this limit, ATI F/T sensor has RTDE connectivity problems, leading to failures like “Fieldbus input disconnected”.
  • It also starts a script using 100% CPU, sometimes overloading the controller, leading to “NO CONTROLLER”.

The question that is important for us - is the 3 RTDE connection limit by design?

1 Like
5

We verified this on 2 separate robots and they behave in the same way (only 3 RTDE clients allowed). Can someone please explain whether it’s expected behavior? We need to quickly make some architectural decisions / changes based on this.

6

It is indeed correct, that the RTDE currently has a limit of only 3 active clients.
Ethernet/IP and Profinet also counts as a client, if they are enabled.