My guess is your VPN client is not using the remote gateway and is instead using your local one. Can you please confirm the direction the one way path is working? From the office to you or from you to the office?

My guess is it's from the office to you if my suspicion about the gateway is correct.