In case anybody else has the same problem: we figured out it was related to some phones not being provisioned correctly. Apparently they were flooding the system with subscribe (and other) messages and 3CX didn't handle it well and so the memory usage increased until it was no more. We now provisioned those phones in another way (phones in question are older aastra models) and now the problem is gone.
Thanks Thomas for the update. I have to jump in to correct you on some key points here because the way you report things make 3CX look like it SHOULD handle this by design. This is completely wrong.
What you describe here is the exact same case of a person that tries to drive a car with the handbrake still in parking position and is complaining to the car manufacturer why the car is moving slowly or not moving at all..
Overall the driver expects the car to handle this case properly!!!!
People need to use supported phones. This company decided to keep on using old, legacy phones and admins took the decade old provisioning file for granted - dumping it into these phones as is and expecting everything to work without side-effects.
As a result we can learn from this case how unsupported, old or legacy phones can be turned into Authenticated and Admin Approved, Denial of service Demon Bots with one .,. actually 2 scopes in life - 1) Make a call from A to B and 2) FLOOD / DOS the server with unnecessary subscribes and sip messages causing the server to eat up all the operating system's memory resources ..
The most ironic part is here - These Subscribe messages were designed to be ONLY UNDERSTOOD (proprietary) by a discontinued Aastra pbx.. but we complain that 3CX does not handle this well. Yes of course not - they are proprietary..
Administrators who are administering these types of phones should never dump a phone on the network and move on to the next task like we do when we work with supported phones.
Unfortunately this is the privilege you lose when you work with unsupported or old legacy phones.
You need erase and disable everything on the phone, put one, test it rigorously and monitor for weeks to ensure this phone behaves correctly to the standard you need. All features should be disabled and the phones should be used for basic phone calls only.
So yes, the memory usage increased because the admin let the phones flood the server until the OS has no more memory to work with.
This is a problem caused by wrong admin decisions. 3CX has nothing to handle here. The phones cannot be blocked because this is all lawful traffic. The admin tagged this as lawful traffic.