atcomsystems.ca/forum
Posted By: Rhonda Disaster Recovery - 08/28/08 10:53 AM
My company is under another audit and the topic at the moment is Disaster Recovery.

I am wondering what each of you are doing for your own companies and/or for the companies you service.

What are customers doing or asking for in the way of having a new PBX switch up and working.

How long is a company willing to be without dialtone?

I know there are plans out there with various vendors to have a cold switch built and ready to go. Is that a true reality with the expected costs?

If any of you could give me some insite on what you are doing, I can proceed with the best routes to take to satisfy future audits, dial tone needs and my sanity.

Thank you. Rhonda
Posted By: tampasteve Re: Disaster Recovery - 08/28/08 11:39 AM
Depends on the customer and their desire to be covered. Some just go with a regular support contract, others keep a complete spare on site, but most that are concerned with disaster recovery keep a smaller version of the system on site in case they need to start up ASAP but with limited staff.

Steve
Posted By: hitechcomm Re: Disaster Recovery - 08/28/08 11:40 AM
This is a question you should talk to your maintenance provider for answers. Ask them what there plans are for you if you have such a disaster.
We have every maintenance customer backed up on software in our office and a secure location. Also we keep enough parts on hand to restore service with in hours of our notification.
Posted By: Silversam Re: Disaster Recovery - 08/28/08 11:55 AM
Back a while ago I was the Disaster Recovery Specialist for my division of GTE.

I'll try to dig up some stuff for you tonight.

But until then -

First questions are:

1-How big is your opperation (Personnel)?
2- How many locations?
3- Voice only or Data too? Anything else?

And most important question of all:

4- What disaster are you trying to protect against?

Are we talking a switch crash? A loss of Outside trunks? A cracked pipe that destroys the switchroom? A Flood that takes out the building? Something bigger?

How crucial and "timely" is Business Recovery? (When do you absolutely have to be back up and running)

How serious is Management? Are they just looking to respond to an audit? Are they willing to spend real money?

I've done Disaster Recovery (I used to like to call them "Disaster Avoidance") Studies for some real big operations. Each of them had completely different needs.

Keep in mind that an Air Force base, a NASA location, a major Railroad, a Hospital, a Bank, a Mail Order house and a "generic" business all have different needs and different requirements. And the amount of money they are willing to spend are completely different.

PM me if you like.

Sam
Posted By: bf6b5yr Re: Disaster Recovery - 08/28/08 12:15 PM
This is a big topic and to give a better answer I think we'll need more details. I'll throw some general thoughts out just to get things started.

· Make sure that you have accurate and current records for your system stored in a secure location. Ideally these types of records and backups should be at an offsite location.
· A big concern is getting incoming calls answered as soon as possible. Even a voice message indicating that your company has suffered a major phone failure and asking the caller to leave a message is far better than a line ringing unanswered. Getting the main numbers routed to voicemail as quickly as possible is a step in the right direction. Messages could be retrieved and calls place via cell phones.
· How long a customer is willing to wait for dialtone depends on what type of business they are. Some businesses are totally dependant on their phone service. I have dealt with some large credit card operations where no outage is acceptable.
· Part of any disaster recovery plan should be disaster avoidance. I recommend that you or any business assess what their needs are as far as communications go. I’d try to determine the risks that are out there. I’d try to look at what things could possibly fail. Out of these things, I’d try to determine what I had some degree of control over and what I can do to prevent or minimize a failure. Do I have spare parts available? Do I have the telephone numbers of any supplier or technical support that I may require? I’d look at who else I am dependant upon for my communications. Do I have contact information available? Who else will I need to contact? Any major customers or government agencies? All of your contact information should be in a single binder. Any plan that you develop should be in this binder as well. It should be kept in a location where it is easy to locate. It’s nice having things on computers but what if you have no power or your laptop’s battery is dead.
· What your actually disaster recovery plan will consist of is dependant upon how much money a company is will to spend and dependant upon the size of the company and the number of locations that it has.
Posted By: liquidvw Re: Disaster Recovery - 08/28/08 01:02 PM
I usually keep a crash kit on site for high profile sites. This would contain a processor, power supply, and some boards that are used in the switch, ie clan, medpro, digital, analog, etc. As far as trunking goes, the site should get dial tone for 2 different types of trunks. This is usually PRI and POTS. If a site has 1 PRI for both local and LD calls I will have a few POTS are necessary for backup. If the site has 2 PRI's then split the carriers, ie get the a local PRI for DID service and local calling from Verizon and a LD T1 for LD. Having them served out of different CO's is also a plus if you can get it. Redundancy in your routes will allow you to use either PRI if one were to go down. IP trunks are also very important. If your local dialtone is down but your WAN is up, you outbound traffic can flow over the WAN to another site where it can find the PSTN there. As far as DID service goes, you can order services called CLAR or Custom Redirect, or whatever your LEC calls it to allow you to redirect your DID's in the cloud. This can be very helpful if your DID PRI goes down. This costs money. I ususaly only get it for the main number at the site. This way if the PRI goes down, I redirect the main number to another site. From there I can backhaul the traffic over the WAN to the site that has the down PRI. Or whatever...

Always have an off site and current backup. Your translations are priceless. Your hardware can be replaced, the translations cannot. If you backup to tape or flash or FTP or whatever, if you don't have an offsite backup your in alot of trouble.

To answer your question. How long is a company willing to be without dialtone? The answer to that is 0.001% of the time. Which if I did my math right is 8 hours of down time per year. Plan it wisely. smile
Posted By: Rhonda Re: Disaster Recovery - 08/28/08 02:13 PM
Wow... what a nice response, thank you. Very good detail and information to work off of.

I did contact my hardware rep and asked him to contact me to discuss.

As far as how "serious" our parent company is and how much we want to spend is completely unknown. I know they want a dollar figure on what it will take to get ourselves back up with no indication on if we would go forward and how they would like it to work.

I'm grasping at straws as far as my situation goes. I was just wondering what everyone else is doing.

I will take all of your words in when considering what to do to satisfy the audit and the parent company.

Thanks, again. Rhonda :bow:
Posted By: mbhydro Re: Disaster Recovery - 08/28/08 07:02 PM
The only thing that I would add to this is if your going to recover off site is check to see if all the incoming pairs from your telco are punched down at the backboard.

Our company did a disaster plan review a few years ago and found that while we had high count cables coming in not all locations had them punched down on the bix blocks. The idea was to split up our downtown customer care agents to the different suburban service centres if we had the lines available.

Our ILEC as part of the review suggested we pay them to punch down all incoming pairs to speed up the time to bring everything back on line.
Posted By: KLD Re: Disaster Recovery - 08/28/08 08:15 PM
Rhonda,

After your situation in Mexico, look to your DT provider to set up a re-route of primary numbers to another site ( another branch?). We normally set up for natural disasters. While most telco switch buildings are "storm proof", even underground cables fail, and with high density fiber and bundled services, all your eggs can easily be in one basket. Even the Feds had two cable routes and split the entrances. Only place you can't outrun is the loss of a C.O. unless you plan way ahead which equals $$$$$.

Any questions? E-mail or PM or Call!.
© Sundance Phone System Forums - VOIP & Cloud Phone Help