I wonder if there's a way to un-nice of whatever phone processes there is while increasing the nice factor for the processes that are say... taking over 60% CPU load. That would likely make the phone app more responsive.

As it is right now, it seems to be treated as another application. And if your system is under load, it, like all other processes compete for the strained resources leading to a bunch of problems like missed phone calls, the UI being laggy, the race condition some people experience with custom ringtones.