What handsets are you using?
Forward the MPS ports or disable it
Are you running private with public?
Disable SIP ALG in the router since you have the ports forwarded manually

I recently had a similar issue where the caller could hear fine, but we couldn't hear them, turned out to be DOS (denial of service) option in the router was causing the issue, disabled it and all was fine.