I assumed you were using the standard Panasonic KX-T30865 doorphone -- but it now sounds more like you have an aftermarket unit that is connected to the CO line ahead of the system rather than to a doorphone card.

As I don't have much experience with that kind of setup, at this point I'll have to defer to anyone else that cares to chime in. Providing the make/model of the door box may help.