The phones will generally work, you might never even notice an issue until someone points it out. The modem the customer has will remap the ports and allow audio and SIP to pass through. But after some inactivity, it might purge those ports and the next call might not come in until you make a call. Some other customer with a different modem may keep the ports open for longer, or only reuse them when needed. Others may handle it perfectly and you will always have perfect calls. There is no standard behavior on this though, and there are thousands of types of modems/routers one may come across. Others may block SIP altogether, or have some reserved ports and will not work at all. Others may have SIP ALG that cannot be disabled, and can break the SIP messages by rewriting the packets.
There are many unknowns out in the wild, so by making an effort to control them, you ensure reliability. That's the gist of it. How you implement a level of control can depend on the conditions (i.e. SBC, VPN, STUN+port forwards, use an app instead etc)