Strange I2C communication errors lead to Controller Disconnect

Hello,

We’ve encountered an issue in which our UR10e will stop safeguard stop at seemingly random times.

The logs (below) show our Program (“recpXX”) start and complete as expected but every now and then the log shows Error Code C290A10. This eventually results in a “No Controller” message and a loss of function.

The code below shows a handful of these messages, but sometimes they appear consecutively many more times over several hours (e.g. when the robot running for many repeat cycles)

Any help is appreciated!

Robot Info:
UR10e
Polyscope Version: 5.11.10.1010502


The Error Code Guide shows the following:

C290A10 I2C peripheral issue, unhandled events: {hex}

Logs Output:

3.5 :: 0319d11h43m31.713s :: 2023-02-10 15:57:04.341 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program manager starting... (Last saved: 2022-05-12 17:24:45) :: null
3.5 :: 0319d11h43m48.029s :: 2023-02-10 15:57:04.396 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager started :: null
3.5 :: 0319d11h43m48.039s :: 2023-02-10 15:57:04.397 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager stopped :: null
3.5 :: 0319d11h43m32.918s :: 2023-02-10 15:57:05.553 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program recp12 starting... (Last saved: 2022-03-31 01:22:53) :: null
3.5 :: 0319d11h43m49.335s :: 2023-02-10 15:57:05.698 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 started :: null
3.5 :: 0319d11h43m57.949s :: 2023-02-10 15:57:14.337 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h43m58.213s :: 2023-02-10 15:57:14.541 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 paused :: null
3.5 :: 0319d11h44m06.863s :: 2023-02-10 15:57:23.273 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h44m06.863s :: 2023-02-10 15:57:23.275 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 resumed :: null
3.5 :: 0319d11h45m35.931s :: 2023-02-10 15:58:52.346 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h47m40.431s :: 2023-02-10 16:00:56.765 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h48m48.431s :: 2023-02-10 16:02:04.747 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h49m21.899s :: 2023-02-10 16:02:38.288 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 stopped :: null
3.5 :: 0319d11h49m34.679s :: 2023-02-10 16:02:51.032 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h49m42.691s :: 2023-02-10 16:02:59.059 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h49m42.691s :: 2023-02-10 16:02:59.061 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 resumed :: null
3.5 :: 0319d11h50m37.049s :: 2023-02-10 16:03:53.355 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h50m53.369s :: 2023-02-10 16:04:09.713 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h50m53.369s :: 2023-02-10 16:04:09.716 :: -3 :: C0A0:7 :: null :: 1 :: recp12 :: Program recp12 resumed :: null
3.5 :: 0319d11h50m57.826s :: 2023-02-10 16:04:30.421 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program manager starting... (Last saved: 2022-05-12 17:24:45) :: null
3.5 :: 0319d11h51m14.129s :: 2023-02-10 16:04:30.490 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager started :: null
3.5 :: 0319d11h51m14.139s :: 2023-02-10 16:04:30.490 :: -3 :: C0A0:7 :: null :: 1 :: manager :: Program manager stopped :: null
3.5 :: 0319d11h50m59.031s :: 2023-02-10 16:04:31.635 :: -5 :: C0A0:7 :: null :: 1 ::  :: Program recp11 starting... (Last saved: 2022-03-31 01:22:43) :: null
3.5 :: 0319d11h51m15.427s :: 2023-02-10 16:04:31.793 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 started :: null
3.5 :: 0319d11h51m21.201s :: 2023-02-10 16:04:37.516 :: -2 :: C0A0:5 :: 5 :: 1 ::  ::  :: 0
3.5 :: 0319d11h51m21.465s :: 2023-02-10 16:04:37.820 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 paused :: null
3.5 :: 0319d11h51m30.595s :: 2023-02-10 16:04:46.955 :: -2 :: C0A0:5 :: 1 :: 1 ::  ::  :: 0
3.5 :: 0319d11h51m30.595s :: 2023-02-10 16:04:46.958 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 resumed :: null
3.5 :: 0319d11h55m29.931s :: 2023-02-10 16:08:46.319 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h56m35.431s :: 2023-02-10 16:09:51.799 :: 20 :: *C290A10*:6 :: null :: 128 ::  ::  :: 30000
3.5 :: 0319d11h57m29.769s :: 2023-02-10 16:10:46.141 :: -3 :: C0A0:7 :: null :: 1 :: recp11 :: Program recp11 stopped :: null

polyscope를 업데이트 해보시기 바랍니다. 우리는 OnRobot Eyes를 함께 사용하는 고객에게서 C290A10 문제가 발생했으며, 이에 대해 최신 Polyscope로 업데이트하니 해당 문제가 발생하지 않았습니다.