Periodic 100% cpu usage

[rocketuc]

@zumiani @RussellPTown @rocketuc @Gunnar Brynjólfsson

Were you all using the Instance Manager with the affected machines connected to a Reseller account?

Yep, but I did turn that off a few days ago and it made no difference, just did it again, and it's still at 100%. Will give it a bit of time to settle.

 

[Gunnar Brynjólfsson]

@zumiani @RussellPTown @rocketuc @Gunnar Brynjólfsson

Were you all using the Instance Manager with the affected machines connected to a Reseller account?

I was at least, the problem was resolved after I deactivated the Instance Manager. I like the instance manager a lot, love the feature and would like to see it fixed.

 

[Gunnar Brynjólfsson]

Yep, but I did turn that off a few days ago and it made no difference, just did it again, and it's still at 100%. Will give it a bit of time to settle.

I had to kill the apt-cache and salt-minion processes manually to get it to go down to normal CPU load. I think reloading the container should do the same. As I have shell access to my containers I just killed the affected processes.

 

[JohnS_3CX]

Thank you for your feedback, the issue reported is under investigation.

For the time being I would suggest to disable the instance manager on the affected machines, and then reboot to clear any stuck processes while we look further into the matter.

 

[RussellPTown]

@zumiani @RussellPTown @rocketuc @Gunnar Brynjólfsson

Were you all using the Instance Manager with the affected machines connected to a Reseller account?

Yes, we were using the instance manager - and the instancemanager user account was generating the high CPU

 

[JohnS_3CX]

Thanks Russell, please do as above for now while we investigate

 

[rocketuc]

I had to kill the apt-cache and salt-minion processes manually to get it to go down to normal CPU load. I think reloading the container should do the same. As I have shell access to my containers I just killed the affected processes.

Hey Gunnar, mine did settle down eventually without killing anything, I guess the service eventually finished doing something, and just wasn't an issue after!

 

[NickD_3CX]

Hi everyone,

Just an update on this. We made a few adjustments to the scripts that are ran in the background by the Instance Manager which should prevent this kind of behavior.
To get the adjustments, all you should need to do is either:

• Stop the Instance Manager, then start it again

or

• Restart the whole machine 3CX is running on

(whatever is easier for you)

I think we haven't missed something, but looking forward to any feedback you may have.

 

[RussellPTown]

We've re-enabled Instance Manager, and I'm afraid the 100% CPU is back...

Have disabled Instance Manager and killed all the (many) apt-cache processes and performance is back to normal again.

 

[rocketuc]

Same here, rebooted the server, re-enabled instance manager, CPU back up to 100%.

Going to give it some time to see if it settles.

 

[NickD_3CX]

@RussellPTown @rocketuc
Thanks for the feedback guys! We have updated the scripts and added some reasonable timeouts when some actions take too long (e.g. 10 seconds), after which the delayed process is terminated.
This should counter things like stale apt-cache calls.

Could you try the following?

1. Turn off Instance Manager
2. Reboot the server
3. Turn on Instance Manager
4. Wait 10-15 minutes
5. Check CPU usage

The above process should update the Instance Manager scripts. I'd be interested to hear how it goes.

 

[rocketuc]

@RussellPTown @rocketuc
Thanks for the feedback guys! We have updated the scripts and added some reasonable timeouts when some actions take too long (e.g. 10 seconds), after which the delayed process is terminated.
This should counter things like stale apt-cache calls.

Could you try the following?

1. Turn off Instance Manager
2. Reboot the server
3. Turn on Instance Manager
4. Wait 10-15 minutes
5. Check CPU usage

The above process should update the Instance Manager scripts. I'd be interested to hear how it goes.

Mine has actually settled back down now after the initial spike, rolling along at 1 or 2%.