It's 8P8C, but incompatible with RJ45

I'm research has possibly lead me to ATT Merlin

RJ31 for alarm panels. Amazon link

I thought RJ31 was physically compatible with Cat5

Well, yes. I didn't pay attention to your picture when I posted. So, I don't know why it's narrower than the standard 8p8c jack and it looks keyed on both sides, opposite the clip end. Have you tried a keyed connector? As in a true RJ45 plug end, I don't know it they're narrower or not. That's all I've got.
I looked around and I don't have any RJ45 jacks or connector to try.

If it is a true RJ31x then there are shorting pins between 1 and 4, 5 and 8. With no connections for 2, 3, 6 or 7.

Telco in on terminals 4, 5 and out on 1, 8.

What does the wiring of the actual jack look like?

Here is a neat USOC Guide. I did see anything with two keys listed in it, but it might give you a starting point

AT&T used to use a "miniature" 8P8C that was keyed at the top like that. The large operator consoles for the Merlin Legend had an expansion port for an additional button field (sidecar). The connector for both looked very similar and would also not accept a standard width plug. I would fix broken ones by crimping a standard plug on and then take a minute with my Dremel to shave down one side. Ghetto...I know. But on those, I think the key was just on one side--not both.

I also feel like this could be some old serial or IBM thing. The family of DEC connectors were a bit odd too, but only because the key skewed to the side. Either way, the housing is clearly designed to keep the riff-raff out!

What is that jack on, where is it? Looks like a wall jack which supports what Tim said about IBM.

-Hal