Queue Manager Service "crash"

[lancerichards]

Hi Team

We've noticed a rise in tickets where customers complain "I'm in the queue but my phone isn't ringing".
We have done the whole, remove and read to queue, but have tracked it down to the Queue Manager Service is ultimately crashing, hanging, not processing etc.

Is there some sort of internal watchdog that can go on this Service to avoid these sorts of issues.
It's kind of critical to customer call flow

A restart of it seems to make it come good, but monitoring for fault and proactively looking at it is better than grumpy customers.

Best
Lance

 

[KyriacosS_3CX]

Hello,

Could (if you have not - I couldn't directly spot one) open a ticket with support on this, so logs etc can be reviewed properly? And to be sure we are following up correctly, this are all U8 and fully updated? When did you start noticing this?

 

[lancerichards]

Hi @KyriacosS_3CX

We've probably had calls on it for about 2 months. So it would have been U7 and U8. All fully patched.
I just managed to intercept a support ticket today and was able to assist the tech on it before they went too far down the rabbit hole.

Are logs retrospectively going to assist? Ticket 2299363
All we did was restarted the queue service.

Best
Lance

 

[KyriacosS_3CX]

Hi Lance,

Support should be contacting you soon if they have not already.

 

[Rohit Subhedar]

Hi Team

We've noticed a rise in tickets where customers complain "I'm in the queue but my phone isn't ringing".
We have done the whole, remove and read to queue, but have tracked it down to the Queue Manager Service is ultimately crashing, hanging, not processing etc.

Is there some sort of internal watchdog that can go on this Service to avoid these sorts of issues.
It's kind of critical to customer call flow

A restart of it seems to make it come good, but monitoring for fault and proactively looking at it is better than grumpy customers.

Best
Lance

Hello @lancerichards,

If you've already confirmed that removing/re-adding agents to the queue only provides temporary relief and have identified that the Queue Manager Service is becoming unresponsive or crashing, then the behavior you're seeing is consistent with a service-level issue rather than a queue configuration issue.

At present, there is no built-in watchdog service that automatically restarts the Queue Manager Service when it becomes unresponsive. As a workaround, some customers implement OS-level service monitoring or scheduled health checks to detect and restart the service if required.

To help determine the root cause, we'd recommend reviewing the Queue Manager Service logs around the time the issue occurs and checking for any related application, system, or resource events (CPU, memory, disk, etc.). If the service is consistently hanging or crashing, please raise a support case and provide the relevant logs so further investigation can be performed.

In the meantime, monitoring the service state and alerting on failures would be a good proactive measure to minimize customer impact.

 

[lancerichards]

Hello @lancerichards,

If you've already confirmed that removing/re-adding agents to the queue only provides temporary relief and have identified that the Queue Manager Service is becoming unresponsive or crashing, then the behavior you're seeing is consistent with a service-level issue rather than a queue configuration issue.

At present, there is no built-in watchdog service that automatically restarts the Queue Manager Service when it becomes unresponsive. As a workaround, some customers implement OS-level service monitoring or scheduled health checks to detect and restart the service if required.

To help determine the root cause, we'd recommend reviewing the Queue Manager Service logs around the time the issue occurs and checking for any related application, system, or resource events (CPU, memory, disk, etc.). If the service is consistently hanging or crashing, please raise a support case and provide the relevant logs so further investigation can be performed.

In the meantime, monitoring the service state and alerting on failures would be a good proactive measure to minimize customer impact.

Thanks mate - we're already doing that but not seeing any deadlocks, OOM, memory leaks, dead processes.
It's literally one or two people just stop getting calls - they're in the queue, everything looks AOK.

Remove, re-add - All OK.
So today when I intercepted a support case, we restarted the service, presto, all good.

So it's something in the service which is getting unhappy - but can't debug the binary and leaving verbose on all the time isn't practical either.

Support case is open. Hopefully we can get to the bottom of it.

 

[Alphabetic]

I've had this issue before on our system actually. Cant remember what the fix was though.. What spec is your server? Got enough?

 

[lancerichards]

I've had this issue before on our system actually. Cant remember what the fix was though.. What spec is your server? Got enough?

Yeah, more than enough processor, RAM, disk.
HTOP shows it's sitting there twiddling it's thumbs most of the time.

 

[KyriacosS_3CX]

HTOP shows it's sitting there twiddling it's thumbs most of the time.

There you go, it's just looking to make its days and ours more exiting

 

[lancerichards]

There you go, it's just looking to make its days and ours more exiting

They're the worst kind of bugs to trace. You sit there and wait and wait and wait and you give up, turn of verbose logging and it happens. Always the way