Image

Is your DR provider ripping you off?

/ /
Business Continuity passing by

 

Now that I have your attention let me expand on my statement a little further. When it comes to IT in general, I strive to take the same approaches which I do in daily life and that is to THINK for other people. Whether it’s my Disaster Recovery (DR) provider or driving my car I always think to calculate others moves and motives. You can get caught off guard if you don’t and on that same scenario it can cost you everything. Let me clarify both for instances; while driving down the highway (literal not virtual:) you may see a car flying up on the right and a small amount of distance between you and the next car in front of you. Now you may stop and think this guy wouldn’t try and cut into this space, would he?… I now try and assess his motives for speeding down the slow lane (obviously he is in a hurry) so now that’s settled either give him the space to cut in or take my chances thinking the person is smarter than that. Well after driving for a couple of decades the choice is an easy one give the preemptive space then follow the person for the rest of my journey and let them get the ticket first.

 

So when you pay for a DR Provider the same level of thinking should apply. You’ve either picked them for reputation or price depending on your situation and budget because there is some kind of compromise otherwise it would be done in house. Either way that’s where you are on the (virtual highway this time) and instead of a car fast approaching its a disaster/compliance test and the same type of fast thinking needs to apply. So your DR provider they should be constantly assessing you companies change management and modifying the current DR plan to ensure that you are recoverable to the level of expectation agreed upon. This is where motive comes into play and in this case much like someone in a hurry to get past you its now somebody in hurry to get home except of course a sales guy they’ll follow you home just to tell you everything’s OK. So then what are the choices? The choice is to play out how you want this DR to play out using your imagination at first. Visualize it, see how you want the components to move and who you want to execute the whole process. Now that the battle plan is in your mind its time to start moving the pieces into place and I’ll play some devils advocate below then try and pull out the best scenarios for you to help utilize in working towards a successful action plan.

 

Feeling out your current situation:
I have on many occasions arrived to clients sites and been faced with the current third party DR provider who is either an in-house consultant or a solution architect that has been trained to bark IT buzz words with no real understanding of them. There is an awkward feeling out period and a hi level back and forth of the current recovery plan in place usually the word cloud is said repetitively as well as virtual. However when you enter the technical deep-dive portion of the meeting things change drastically as they fumble around on their phones or can never seem to get a hold of the person who was supposed to be on call there or on their company messenger etc.. While amusing in hind site; the unfortunate part is you need to have at least 5 of these meetings to gather any type of momentum towards your DR plan and the oh S*** meter goes off in the room because everyone realizes just how vulnerable they really are.

 

How not to do DR

 

Now lets examine the Pain Points that could get us into the scenario above:


1. Your Disaster Recovery provider backs up to their location and provides no type of remediation plans in case of a real disaster.
What that boils down to is a third party company is taking your data  and backing it up somewhere and you’re not really sure what they’re even backing up in the first place. When the time for a real recovery they just hand over some tapes of a VPN to your data path and the rest is up to you. It hits you WAIT I thought since you backed it up you should help recover? The answer is usually the same No that’s not part of the agreement you signed you might as well be backing up to some free Google drive.

 

Now how to move your chess pieces. When its time for a contract renewal or when the account S.A. shows up to tell you all about the improvements on their side ask them if they have any default recovery procedures. Usually they are hesitant either because A. They don’t or B. They sell them customized for upwards of 8k each. Either way you can push for them for free and if they don’t at first just ask the sales guy how a DR provider doesn’t have any default documentation and trust me they will get something in your hand from one of their hardware vendors. If its B then just say generic procedure and then once you have that just fill in the blanks while asking specific questions to a SA.

 

2. You have a Managed Service company hosting your data with no backup plan
The first thought most people have is that when they sign up for a a company to host your servers off site is that they have a great backup policy. Another common misdirection is most managed services companies will lie and say they’re built for hi availability cut over and not truly DR. (Hopefully you read this not looking in your review mirror b/c this is something to think about when selecting provider and also something to consider when interviewing potential providers). Even the biggest companies in the world do this and they all go by the big story they have 5 Active Directory domain controllers all around the world and your data is protected yada yada yada. Who knows if this data is really syncing as its supposed to though? when the servers crash and the site goes down can you just log in with no data to new site or tombstone out of sync data and mind you none of this matters if there’s a virus outbreak because then everything is compromised. Chase bank hacked

 

The above solution wont work as above because they go into them up-selling you a DR fault plans once they manage your data. Its the old trust us and if you don’t you’re gonna pay through the nose because we now have to actually come up with a real solution. Your best bet is to weigh the cost of adding  a new backup server or tape farm to the contract or letting them do it unfortunately.  The bright side is you can usually bet against the house and put a stipulation into the contract that says if they cant recover to your expectations you get money back. The chances of them getting it right on the first few attempts is less than probable.

 

3. Your company backup admin inherited the current model that he doesn’t quite understand and you need a consultant
Lets face it most people in the IT field don’t stay with a company long enough or have been around long enough to have set up their current DR environment. Most times the processes and backup environments are inherited by the new admin coming in or sometimes just an existing employee or team that got stuck DEALING with it. You may be or have been in that situation yourself at some point and had no change control in place as well leading you to start from step 1.

*Its important to understand that just because the new admin has the necessary skills to do the job doesn’t mean they understand a new enterprise environment from jump.
While the person may have the technical skills needed to fill the role there is a great ramp up period that can be significantly helped by a contractor. Having a DR consultant in place is a great idea to team with the new admin and document all of the procedures while they ramp up on internal policies and procedures. I tend to recommend one that has no particular hardware channel partners otherwise you will just end up buying stuff you may not need. (Remember you are decreasing speed to let the driver room to pass not accelerating past) Remember to get them to document the actual recovery procedures and not just schedules and RTOs.

 

4. Your data is being backed up to a foreign land “the 24 hour support model AKA follow the sun” and you have no idea what it even looks like anymore…

 

Now is not the time to ease back and let that driver cut in front of you, its now time to speed up and block them. This is the scenario that has the worst implications because the provider does not have to comply with US government standards nor insurance company/internal audits. A DR test is now being done off site that include isolated tests that run in the middle of the night for a specified app that has all of the backend servers still running uninterrupted like DNS, AD. So they show your execs “look we cut you guys off at 2Am and your app was running within minutes isn’t it great?” While that’s all the assurance they may need we know better and that was no DR test. Another issue is DR virtual machines that are only used for these tests and have no production based use other than to show an app running that really wont be if a DR situation happens but then again most places do that.

 

How to speed up. Firstly ask if your data is being hosted in the same SAN/Tape Group as other clients. If yes and if your insurance lets that be compliant ask if its in its own slice or separate tape group. What this does is start to build a case that your data is not isolated and being remember the race to sunshine that you wont be able to recover as they said at that speed. Next find out what their DR plan is if they actually go down? Its important to question this contingency because if they give one of the excuses above then its cheaper to bring it back in house. Can they perform a DR test at your site where you see them remote into the servers being recovered from scratch? Transparency is lost when your data leaves the country and you are left with someone who doesn’t speak your native language. Your best bet there is to find a consultant that does speak that language and have them work for you understanding what your real goals are and then convey it to them. This is the real way to speed up your message, plan and expected results when dealing with a foreign provider. It takes away the mis-communication excuse as well as not allowing the native language conferences people sometimes have when you are in the room with them and they don’t want you to hear something then play it off as if they were explaining something to a coworker. TRUST ME that happens all the time. Consider that consultant your rocket fuel to getting better service or on-shoring your servers back.

 

5. A group of Channel partner Solution Architects just came into scare your boss into upgrading everything to be DR capable and overspent your budget on a SAN that wasn’t necessary…

So remember to examine your situation; firstly your budget has been overspent and now you are on the hook to outperform what someone else promised…
Step one Update your resume.
Step two Get yourself integrated into the actual implementation with the VAR. This doesn’t mean actually helping them do their work but make sure you are involved in all change point meetings whether via Physical, Skype or conference call. There is good reason for this you will be looked at by your boss as someone deeply involved and also you can ask enough questions so that they will get tired of you and send you for actual training.
Step three Constantly ask for documentation on everything because they wont want to hand it over (they can up-sell more services down the line the less you know).
Step Four Always ask for best practices and white papers for all technologies being implemented these will become your DR recovery documents with a few minor tweaks.
Step Five Before the project finished and they do a full cut-over have them run a DR test for you (you can leverage the equipment they just decommissioned)

 

I hope these 5 scenarios helped and I made them somewhat entertaining so that you can come out on top. Stay tuned when I come back with Disaster Recovery Failback part 2

Share on Tumblr

Leave a comment

TwitterFacebookGoogle