Strange I2C communication errors lead to Controller Disconnect

Hello,

We’ve encountered an issue in which our UR10e will stop safeguard stop at seemingly random times.

The logs (below) show our Program (“recpXX”) start and complete as expected but every now and then the log shows Error Code C290A10. This eventually results in a “No Controller” message and a loss of function.

The code below shows a handful of these messages, but sometimes they appear consecutively many more times over several hours (e.g. when the robot running for many repeat cycles)

Any help is appreciated!

Robot Info:
UR10e
Polyscope Version: 5.11.10.1010502


The Error Code Guide shows the following:

C290A10 I2C peripheral issue, unhandled events: {hex}

Logs Output:

3.5 :: 0319d11h43m31.713s :: 2023-02-10 15:57:04.341 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program manager starting... (Last saved: 2022-05-12 17:24:45) :: null
3.5 :: 0319d11h43m48.029s :: 2023-02-10 15:57:04.396 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager started :: null
3.5 :: 0319d11h43m48.039s :: 2023-02-10 15:57:04.397 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager stopped :: null
3.5 :: 0319d11h43m32.918s :: 2023-02-10 15:57:05.553 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program recp12 starting... (Last saved: 2022-03-31 01:22:53) :: null
3.5 :: 0319d11h43m49.335s :: 2023-02-10 15:57:05.698 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 started :: null
3.5 :: 0319d11h43m57.949s :: 2023-02-10 15:57:14.337 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h43m58.213s :: 2023-02-10 15:57:14.541 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 paused :: null
3.5 :: 0319d11h44m06.863s :: 2023-02-10 15:57:23.273 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h44m06.863s :: 2023-02-10 15:57:23.275 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 resumed :: null
3.5 :: 0319d11h45m35.931s :: 2023-02-10 15:58:52.346 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h47m40.431s :: 2023-02-10 16:00:56.765 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h48m48.431s :: 2023-02-10 16:02:04.747 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h49m21.899s :: 2023-02-10 16:02:38.288 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 stopped :: null
3.5 :: 0319d11h49m34.679s :: 2023-02-10 16:02:51.032 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h49m42.691s :: 2023-02-10 16:02:59.059 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h49m42.691s :: 2023-02-10 16:02:59.061 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 resumed :: null
3.5 :: 0319d11h50m37.049s :: 2023-02-10 16:03:53.355 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h50m53.369s :: 2023-02-10 16:04:09.713 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h50m53.369s :: 2023-02-10 16:04:09.716 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 resumed :: null
3.5 :: 0319d11h50m57.826s :: 2023-02-10 16:04:30.421 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program manager starting... (Last saved: 2022-05-12 17:24:45) :: null
3.5 :: 0319d11h51m14.129s :: 2023-02-10 16:04:30.490 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager started :: null
3.5 :: 0319d11h51m14.139s :: 2023-02-10 16:04:30.490 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager stopped :: null
3.5 :: 0319d11h50m59.031s :: 2023-02-10 16:04:31.635 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program recp11 starting... (Last saved: 2022-03-31 01:22:43) :: null
3.5 :: 0319d11h51m15.427s :: 2023-02-10 16:04:31.793 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 started :: null
3.5 :: 0319d11h51m21.201s :: 2023-02-10 16:04:37.516 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h51m21.465s :: 2023-02-10 16:04:37.820 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 paused :: null
3.5 :: 0319d11h51m30.595s :: 2023-02-10 16:04:46.955 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h51m30.595s :: 2023-02-10 16:04:46.958 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 resumed :: null
3.5 :: 0319d11h55m29.931s :: 2023-02-10 16:08:46.319 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h56m35.431s :: 2023-02-10 16:09:51.799 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h57m29.769s :: 2023-02-10 16:10:46.141 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 stopped :: null