As you have a lot of things going on in your network, you will need to start some place as a base line.
Not only do you need to have QoS on the routers, it is also required on any data switch’s the VoIP network’s pass through.
As for MPLS this would be a good place to start for base line. I would set up something like Ether-real at each site to monitor packets to compare what is being sent and when/if it is being received at the other end.
If a smart data switch is installed you may be able to do port mirroring, this allows you to monitor both way traffic from another port. Or install a hub just for test.
I had a site with MPLS 2 miles apart (in FL), after vendor set up above monitoring, the provider admitted to porting some traffic to Texas then back exceeding 150ms.
This is just a starting point, with a network your size it will take time to isolate all issues.
Good luck!
