Five things I learned from losing my SharePoint farm

My name is Paul and I am a recovering SharePointer…

I work for a smallish charitable organisation in London. I am the SharePoint Admin (and also the Developer, Trainer, DR guy, Champion and so on). We are a small organisation in terms of staff and we only have 2 SharePoint deployments, one a highly customised MOSS farm running off SQL 2005 and the other a new SP2010 farm running off SQL 2008. Recently I was at work on a Saturday at the end of a long day when – without going into all the details – I realised that somehow I had managed to destroy my SharePoint 2010 farm. I couldn’t access any content, and nothing I was doing was getting it back. Suddenly I was feeling very sick and nauseous.

I went home and made the first sensible decision of the evening. I went to bed. Partly I thought I should stay up and work on the problem but I was shattered and I wasn’t going to solve anything in the state I was in. When I woke up – at 5:00AM, screaming – I got to work and thankfully by midday I had the farm back in a working state and all the data accessible. During those painful hours I learned 5 valuable lessons that I thought were worth sharing for relative newcomers – like me – to SharePoint.

1. SQL backups.

Backing up SharePoint properly requires time and attention to detail and planning. I know that now. I certainly didn’t have everything I needed (more of that later) but what I did have was recent, reliable SQL backups. These aren’t the whole story by any means, but they are a very good starting point.

Put simply, if you have your SQL Content Databases backed up then you have your users’s files backed up.

That is not everything, there is plenty that you won’t have but it’s a great start.

A bonus tip I picked up from the excellent ‘DBA Survivor: Become a Rock Star DBA’ by Thomas LaRock (@SQLRockstar on Twitter) is that just doing the backups isn’t enough, you need to periodically test that you can actually restore them. That is good advice.

Having the content databases was a good start. But I still needed to rebuild the farm before I could use them. Which leads me to my next point …

2. Documentation

Keep good documentation (I’m not inventing fire with any of these points but this is another basic message that bears repeating). Luckily in this case I had my install and build steps well documented. It was a massive relief that when I had to rebuild the farm for real I had all the steps set out clearly in front of me and wasn’t trying to hazily remember what I’d done months ago.

A word of warning though. I have recently started using Microsoft OneNote for the first time to group my SharePoint documentation, including my build and config procedures. Luckily for me, I had the OneNote document synced on my laptop. However, just two weeks previously I had my documentation saved ONLY in a SP2010 wiki. I’m a big fan of wiki’s generally but be aware:

• A SharePoint wiki cannot be synced to a SharePoint Workspace.
• A SharePoint wiki can be connected to Outlook but can’t actually be read in Outlook.
• Correct me if I am wrong but… I have found no clear way to take an offline copy of content stored in a SharePoint wiki.

In short, I realised just in time that a wiki stored in a SP2010 farm is not a good place to keep the documents you need to restore that SP2010 farm.

3. Powershell is your friend.

I am a newcomer to Powershell but I am already a big fan, I don’t think you can properly administer SharePoint 2010 without it. One massive benefit I found in the early hours of Sunday morning – as I struggled to regain some dignity and my tenuous grasp on continued employment – was that having used Powershell scripts to build and configure a lot of my farm, it made RE-building and RE-configuring my farm a LOT easier since I was able to do much of the work by running the same scripts I had used (and backed up) before.

Not only did it save me time but it reduced the scope for errors since I wasn’t re-typing in database names (they were already in the scripts) and I knew that I would be setting services up the same way. If I had relied more on using the GUI – however well documented that process was – I would have been far more likely to miss a tick box here or set a drop down incorrectly there. By re-running my Powershell build and config scripts I knew I was building the farm EXACTLY the way I had done it before.

4. Get the Disaster Recovery book BEFORE the disaster

The first thing I did once I realised how badly things had gone on the Saturday night was reach for my trusty SharePoint Disaster Recovery book. *happy trumpet noise*

The second thing I did was regret not having bought a SharePoint Disaster Recovery book. *sad trumpet noise*

There is a lot of help on-line but I really recommend getting a book and reading it cover to cover, I found out at the wrong time that my DR planning was woeful. The information I found online that was really clear and helped me enormously was written by Sean McDonough (@spmcdonough on Twitter) and I have now ordered a copy of his SharePoint 2010 Disaster Recovery Guide.

Personally I think sellers of DR guides should be able to sell the book for two prices, one for people who buy it BEFORE a disaster, and a higher price for muppets like me who buy it AFTER.

Everyone’s needs and strategy will be different, but from my experience I would urge anyone starting out in SharePoint DR to backup your SQL databases and use either Central Admin or (better) automated PowerShell backups to ensure your Web Applications are backed up too. I had the former, not the latter.

5. Sleep

Ok, really I only had 4 things for this post but no-one ever writes a list of 4 key things do they, it’s always 5, or 10.

But, as I mentioned, a sensible decision I made on Saturday night was to stop making things worse and get my head down for a few hours. A lot of things seemed far clearer in the morning and who knows how much time I would have wasted and extra damage I would have done if I had carried on blundering around when I was tired and stressed late on Saturday.

So that’s a brief recap of the disaster that befell me and how I (mostly) recovered from it. All the tips are fairly obvious but, especially in cases like mine where one or two people are responsible for all aspects of a particular project or software, it is sometimes easy to not see the wood for the trees and miss some obvious steps.

About redhotbelgian

Paul Chapman has written 1 post in this blog.

• View all posts by redhotbelgian →
• Blog
• Twitter