Sounds like the switch. I had this same thing happen with a os7200s where the oas card was connected to a 10mb switch. Moved it to a 10/100 and the problem went away.

You may also need to leave a pc with wireshark to see how much network traffic is hitting the system/oas cards.
They don't like large amounts of traffic. Though if the vlan is setup correctly it shouldn't happen.