Often when we are called to look at an issue with the robot, a reboot solves the problem.
I’ve asked our operators to shut the units down at the end of the week, which seems to help.
Today, for example:
Normal operator has been out for the last 1-1/2 weeks, so I don’t think the robot has been shut down the last couple weekends.
We run a setup program where the operator enters part dimensions and sets up the gage.
After that, they run the actual part program.
However, after setup program finished, all the part dimension variables were back at the values from the previous job. This obviously resulted in strange behavior.
After a reboot everything seems to work fine.
We run 2 or 3 shifts on these, so they typically don’t shut them down.
UR robots absolutely have issues when you don’t reboot for a while.
That might be a bit much, but a week seems reasonable. I boot mine “as necessary” when it starts getting stupid and forgetting things (couple weeks).
I see weird things when I don’t reboot, including and especially a system slow-down mostly noticeable in the File Dialog.
If I don’t reboot every couple weeks or so the File Dialog slows way down. We’re talking literally almost 10 seconds for the File dialog box just to open. I use that box sometimes a dozen times a day or more.
I’m not sure if your issue is related to the same thing as mine but I do know that I have the same issue on two UR10e’s and it is absolutely related to the number of files in the file dialog.
Every time you edit a file and save it, Polyscope creates a backup file for it, numbered from 0 to 9, so there are as many as 10 backup files for each URP file. I probably have 800 or 900 files at this point (I have dozens of different jobs, many with multiple programs.
It needs to be addressed. UR is looking at losing at least one customer over this nonsense.
Interestingly enough, I just got back from another UR5e with the same problem - this one is in a different area of the plant, and is the most recent system we released, compared to the other which we’ve been using for about 1-1/2 years.
Tried multiple times to enter new specs - the active setup program uses the new values, but as soon as that program stops, the old values are still in place.
The older unit is pretty clean - has about 125 files under the /programs folder and subfolders, totalling about 17MB - most of which is in the 2 versions of OnRobot URCaps stored there.
The newer robot hasn’t been cleaned out of all the old revs, so it has more like 200 files and 16 MB (also has a couple big URCap files there which probably don’t need to stay).
It’s possible this issue has come up before and we didn’t realize what the issue was - it tends to manifest itself as the robot going where it’s not supposed to, so in the past someone may have just rebooted without digging.
Similar things seem to happen with other components - a few weeks ago one of our cameras decided to switch an axis orientation in the values it was sending.
I/O units and stepper drivers also periodically just get lost. We need to put some switches in those power lines to keep people from having to crawl under desks & figure out which plugs to pull.
I’d like to be proactive and give all the operators some straightforward guidelines - either stick with the end of week shutdown or have 2nd shift shut them down every night, or maybe every new setup.
Now we’re starting to see this same issue (not retaining variable values) on additional machines, and repeatedly on a couple of them. One machine we’ve had to restart for almost every job (2-3 times per day).
Guess it’s time to go through channels to see what we can do.
They know there’s an issue, but it’s probably difficult to find. We had this happen several times, then it quit being a problem for a couple weeks.
Typically we were finding the issue between the setup program and actual run program - my take on it is that while in the setup program, variables are kept/used from a temporary memory location, and are supposed to update permanent memory locations, but for some reason don’t. Once the program stops, the next program start pulls the old values from permanent memory.
That was reasonably easy to catch, as clearly funky things would happen when the robot had bad part dimension values.
Today we ran into a more insidious variation - Setup was done yesterday, and parts obviously ran fine for some period.
After finishing 2 lots of the same part, and almost finished with the 3rd (~1000 pcs total), operators noticed something wrong with the part sorting.
We measure parts and sort them into trays - trays have a display which shows the part size, which is assigned on the fly. A couple arrays track the size and qty associated with each tray.
When a tray fills up, an operator pulls the parts and clears the info for that tray by pressing its associated button.
The display would clear, but the underlying variables were not clearing - leading to some major confusion.
Not knowing when this problem arose, we’ll probably have to re-run all of them.
I’ve been trying to figure a way to write some code which would check to see if the problem is occurring, then notify the operator.
However, if it’s working the way I think, then it would take multiple program start/stops with some variable changes & checks to confirm whether the variables are being properly stored in the long-term memory locations.
We had an open ticket with UR on this issue, but we saw several occurrences on multiple machines within a couple day period, then all was well for a while, so the ticket was closed. Didn’t think to pull a support file off the machine before I restarted it, so probably can’t tell them anything useful.
I’m thinking we may yet just move back to an older version of Polyscope.
I don’t think anything I do within a program will tell me anything.
It seems to be that when a program stops, all the variable changes which happened during the run don’t get written back to the permanent variable location. As far as I can tell, it affects all of the variables. It requires looking at a variable before & after stopping the program to see whether the problem has occurred.
In our case, it’s common for the operator to stop & restart the program, so this could happen any time.
Short term, I’ve created a script which looks at a variable & shows it to the operator, then assigns a new random number to the variable and halts the program.
The operator runs it again, and looks to see if the number is different than the first time. If they’re the same, it means that the new value didn’t overwrite the original value.
As I was developing that program, it happened again, so I was able to verify that my program worked, as well as pull a support file off which may help UR to ID the problem.
I"ve stuck this as an optional operation at the beginning of the setup & run programs - the operator can do a fairly quick check to see if the problem is currently active.
I’m also asking them to shut down the machine more often - at least on weekends, and possibly at the beginning of a run if they’re at all concerned.
They’d rather do that than have to re-run hundreds of parts.
