atcomsystems.ca/forum Forums Other TDM Phone Systems ESI Telephones & Systems Getting Diagnostic Info/Logs from a Communications Server 1000

Hi,

We have an ALL IP ESI Communications Server 1000 system. We have experienced a few outages that may be hardware or maybe software but to narrow it down we would like to get access to any diagnostic logs on our CS1000.

In working with ESI TS they speak of remoting in and looking at diagnostic info. So far all of TS's looking has resulted in very little in the way of RCAs for the outages.

We certainly see very little in the GUI in the way of diagnostic info/logs but when ESI TS remotes in it is either via telnet or ssh.

We have access to our Administrator and Installer passwords but I do not believe our reseller is not yet a super cool ESI Certified Technician so I am not sure if even with both of these passwords that they can logon via telnet or ssh and see what the ESI TS see.

Ideally when we have an outage again, we or the reseller would open a PuTTY session to the Communication Server 1000 and capture the diagnostic logs to the scroll-back buffer and share them with ESI TS. We are 24x7 and have very few times where we can 'go without' our phones for an hour or two while everyone gets rounded up to diagnose an issue as it happens.

So if any of the members who are ESI Certified Technicians can confirm if the Installer password 'should' let you remote in with telnet or ssh that would be a big help. Also is the usernames Administrator and Installer or some other, security-through-obscurity, username. [I hope not 'root']. Please PM me if you chose to share the username.

Also I would ask if you can share the names and file locations of the diagnostic logs/files. I can take a stab at it assuming a *NIX OS.

Thanks in advance!

Your installer has access to all the diagnostic data. Tell him to to look at F7.

When you say "outages" what do you mean? Are you losing connectivity or are you losing outside line access? Do the phones display "Connecting" or are the access to outside lines, which I assume are SIP not there? The biggest problem with ALL IP based systems is usually the network. If you have just ONE device, be it phone or computer, that is throwing out garbage on the network, it can bring an entire system down. You can have a flaky POE switch that is dipping the power below the usable threshold or you may have a managed switch which has become "unmanaged."

When the problem with the telephone system occurs, are you having other network or data related problems?

Rcaman

Hi Rcaman,

Thank you for your reply and questions. I always prefer problems like the ones you described as I can sort them our quickly. Unfortunately they are not.

Also sorry for not including some details on the outages so you had to try to fill in the information gap I left. I originally had a lot more detail in the post. I pulled the details out because the post wandered well beyond the intended scope of my post.

Originally I wanted to ask:

"Is there a diagnostic file or files on the CS1000 we can access, through some manner of CLI connection, using either the Administrator or Installer level passwords?"

Another related question:

"Does the CS1000 support SNMP [pref v3] poling or the sending of SNMP traps, or some form of log shipping or SysLog functionality?"


Answering your questions:

The outages we have had seem to be internal to the CS1000.

For reference the CS1000 has:
5 shelves including the base management shelf [with NSP]
5 SIP24 cards
3 ASC cards
16 IVC24 cards
380 60 Gig IP phones

We are staffed 24x7 and have high round the clock call volume.

A couple example outages are:

All 7 IVC24 cards in Shelf 3 stopped communicating with the CS1000
evidence:
- no activity on each cards red LED
- each of the IVC24 card NICs had network layer-2 PDU activity but they did not respond to ping
- the extensions assigned to the 7 IVC24 cards were all showing 'connection lost...' [the phones are connected to different switches throughout the two buildings]
- moving the Ethernet cables to other ports on the same switch and its hot standby twin switch did not restore service to the given IVC24 card
- the IVC24 cards in other shelves to include shelves 4 and 5 continued to functioning without issue
- - - - even after moving some of their Ethernet cables into ports used by the non-communicating IVC24 cards


Twice the SIP24 card 5 spontaneously stopped servicing its outbound SIP trunks 97-120
evidence:
- The SIP24 card's Primary Processor IP address [SIP signalling] did not respond to ping to where it had previously.
- However, the SIP24 card's Secondary Processor IP address [RTP media] continued to respond to ping
- Both of the SIP24 card's NICs showed link and network layer-2 PDU activity
- We moved the SIP trunks that SIP24 card 5 serviced to an unused Outbound Group to temporarily disable the use of SIP trunks 97-120
- Unseating the card and re-seating it allowed it to restart, reload its config and also respond to ping on its Primary Processor IP address


Best regards!

OK. To answer some of your questions:

All Comm servers can collect trap data. Your installer must contact ESI and have them turn on the trap. A master tech or your ESI installer in contact with ESI Technical can review the trap data.

The data is not available unless ESI technical releases it. There is some proprietary information contained in the trap data and ESI needs to weed out what can be shared and what they prefer to hide.

To be honest, you really need to have your installer elevate this situation to ESI programmers and engineering. What you describe is a loss of layer communication and since it is happening on the IVC and SIP modules, it appears to me that there's a software "rabbit hole" that needs to be poked and fixed. We had a similar situation with a 600 and ESI banged their collective heads for a few months until one of the engineers took a look and, within 5 minutes had the problem solved. There is a lot of inter module communication going on and it looks, to me, like the CPU is losing track of the "sign posts and mile markers."

It's like allowing the Minions to "Run Amok." Not good for them and, certainly, not good for you.

Rcaman

@Rcaman,

Thank you for your reply. Sorry I just got to it. The reseller and I have put together our issues list and they are taking it to ESI Technical Support today.

On a related note, I saw was a third-party tool to monitor SNMP on an ESI IP900 and send alerts. from BrainHerd:

https://www.brainherd.com/products-2/snmp-monitor-for-esi-ip-server-900/

I will ask them if they have gotten it to work on a CS1000.

The IP900 was designed for network management. The CS systems have the capability for network management, however, I am sure ESI has to get in an enable that AND it may be a license item.

Rcaman

Thanks Rcaman!

I will let my reseller know to ask. They are now making progress with ESI support.