I have no first-hand knowledge of the 3-Com NBX system but what I have heard is that it uses a somewhat-customized version of sip. My understanding is that it uses a few specialized commands in order to maintain a "heartbeat" of phones and that aftermarket phones (Polycom, Snom, etc), after registering, will not work after 5 or 10 minutes (whenever the heartbeat is sent). I haven't confirmed this so take it with a grain of salt.
Version 5.0 and earlier use a highly-customized protocol that was taken from h.323. Dont expect much of anything to work with this.
The 3-com system requires 3-com switches and server gear similar to that of the shoretel system. This has to do with the way that 3com approached system features was to use IP Multicasting for shared line appearances and other messaging functions.
The phones also do not retain any system programming information or software. Every time you plug the phone in it runs it's bootloader then has to redownload the firmware and configuration settings from tftp from the server. While this is a trivial concern for LAN users it presents a problem on large divided networks and roaming internet-based users. The phone will re-provision itself every time it powers up, and if your network is not set-up to adequately provide tftp to the phone, the phone turns into a brick until provisioned instead of running on the last good provisioning. Provisioning also must take place from the actual phone server and cant be placed on another server.
Because of it's use of IP Numbering it is not designed to really co-exist with other IP infrastructure. The scope of this is beyond my ability to look it all back up
I do not know how the licensing structure of this system is set-up.
Asterisk is hard to compare to most systems because it has no preprogrammed feature list that it comes with. You make your own features. That being said there are a few things that Asterisk can not be made to readily do.
1) Asterisk v.1.2 (what is considered stable and production-ready) does not support share line apperances. This means that when the boss picks up the phone, the secretary wont see an LED go red. Asterisk v.1.4 does offer support for this feature but it is still undergoing revamps to it's core functionality.
2) Asterisk does not have a Page/Intercom function. Use of certain phones (like Polycom) will allow you to add a sip header that will cause the phone to play a beep then auto-answer on speakerphone. However, there is no group-page feature. The closest I have seen to this is sending all phones an auto-answer sip header then connecting them all to a conference. It doesn't really work well.
3) Asterisk is not pre-programmed, pre-conceptualized, pre-anything. It shows up as a box full of legos (or whatever euphemism you like) and no instructions. You will need to learn how to write dial-plans in order to make it even make a phone ring. There are pre-programmed asterisk distributions out there such as Trixbox and AsteriskNOW but these are primarily targetted at soho and small business, not really something you would use in a medium sized business or larger.
Probably the most noteworthy thing you need to understand about Asterisk is that you are the engineer, programmer, technician, installer, and sales rep. There is no large corporate backing that you can call with T1 issues or hardware problems. You are that person.
The plus-side to asterisk is that it runs on regular network gear, will connect to TDM/SIP/MGCP/H323/IAX/SCCP/Bluetooth, uses standard computer hardware, has no licensing restrictions, and is limited to your ability to program it.