Are your backups good enough?

I caught this article at Ars Technica today.

http://arstechnica.com/web/news/2009/05/hackers-crack-flight-sim-community-site-ruin-it-for-everyone.ars

I'm not a flight sim guy by any stretch, but I think we can all agree: losing 13 years of work is one thing, losing 13 years of what you do in your spare time, your labor of love so to speak, is absolutely gut-wrenching.

I'm sympathetic to these guys, but unfortuntely there's just no excuse. Their backup strategy has been inappropriate and insufficent since the beginning, and noone caught it. Their entire disaster recovery scenario rested with a backup server that was online and (somehow) accessible? Wow, just wow.

So, they'll probably rebuild, and it will likely take a whole bunch of time. But if they take away one thing from this fiasco, its that they had better take their backup and disaster recovery strategy very, very seriously.

SBS 2008 and Third Party AV

Having problems loading the SBS 2008 Console? Keeps crashing a few moments after you open it? The problem might be the way the SBS 2008 console tries to use WMI to enumerate the security status of the various AV products that you have sitting on the servers and PCs on your network. Check out this procedure:

http://blogs.technet.com/sbs/archive/2009/03/12/sbs-console-crashes-when-duplicate-entries-from-av-products-are-written-into-security-center.aspx

Its a bit deep, and requires some concentration and patience, but definitely worth a look.

Free Stuff: SBS 2008 Admin Companion

I like free stuff. Check this out - Microsoft Press is offering a PDF copy of the SBS 2008 Administrator's Companion for free.

Also, Dana over at Scorpion Software has an AuthAnvil promotion going on. Check that out too.

Make sure you're doing everything you can to end up with a stable networking environment.

Stuart vs. Harry

Hyper-V and Windows Server Backup

So, this is still going on. I can't reliably use Windows Server Backup to back up Hyper-V virtual machines to another volume with Backup Integration Services enabled (translation: I can't do live backups of virtual machines).

I first became aware of this about 7 months ago. When our Hyper-V hosts started crashing.

Omar Droubi - Posted on Tuesday, October 21, 2008 9:46:03 AM
Hello All,

After much communication with PSS they have acknowledged that the hyper-v vss writer issues is due to a known Windows Bug

This bug was first discovered in Windows 7 and has now been reported in Windows Server 2008.

Now the good news.

The fix for this bug will be incorporated in Windows Server 2008 Service Pack 2 (ETA currently unknown). If they get a fix out sooner I will try to share as best I can- if they happen to released it as a separate hotfix and they notify me about it- since I am now on the list of affected clients/partners who have open cases.

For the mean time I have implemented the work around of simply disabling the VSS backup integration on each of my VMs. This has proved to work well enough for me since I backup at the physical server level using Windows backup.

My guest machines go into saved state and start back up with out issue- their downtime is only about 2 to 4 minutes total per physical host and me and my client find this acceptable as the backup runs off hours.

(I am running Win2k8 VMs only-plus a single W2k3 VM)- within each VM I have assigned its own dedicated disk for backup using Windows Server backup (NTBACKUP on W2k3) in case I need to restore at the VM level. and if the parent partition goes corrupt I can recover the entire server using the Windows backup I run to external USB drives.

Poor mans backup- but in this economy- for this particular client- 5 physical servers-plus 10 hyper-V guests- this solution works very well. (backup to external USB drives that are swapped a few times per week.

Does anyone know of issues or unsupported scenarios where putting a machine in a saved state is risky or not recommended?

Thanks,Omar

Reference: http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/d7026f87-4f58-4576-b89f-302c44da6532/#page:1

That forum thread is a good read, and there's lots of hope in there. For example, when we first ran into the issue it was on our own Hyper-V server. So our first reaction was to change freeze all of the Hyper-V hosts that we managed. This actually worked for a while - until a stupid blunder with our update management system and some 2008 updates "leaked". Then those machines started crashing too. So, our current workaround, and we're still using it, is to disable the "Backup Integration Service" on all of our guest VMs and scheduling the backup to occur at night.

What this does is save the state of the VM briefly while the VSS snapshot is taken. Then the VM wakes up and the backup of the volume continues. It works really well, but its actually a step backwards for us, since we've been doing live VM VSS backups since the Virtual Server 2005 days.

The thread goes on to mention things like issues in Hyper-V VMs that have SCSI disks/devices, etc. I don't doubt that those are issues, just not for us. The very last post of that thread sounds extremely promising: ensuring that all of the volumes that have Hyper-V VHDs on them have dedicated shadowstorage. Unfortunately my own testing has not revealed that to completely fix the problem. At best it reduces the crashing from "every backup" to "every other backup".

So, like Omar, I guess we're still waiting for Service Pack 2, or whatever. I think Hyper-V is a wonderful thing, but I sure wish they'd fix this.

Ask the Core Team

If you use Hyper-V, this blog is an absolute must. For example:

http://blogs.technet.com/askcore/archive/2009/02/16/top-issues-for-microsoft-support-for-windows-server-2008-hyper-v-q2.aspx

Kaseya Database Woes

Running Kaseya on SQL Express? Database getting too big? There's lots of good information on monkeying with SQL in Kaseya's knowledgebase and forums, but none of it talks about moving from SQL 2005 Express to SQL 2005 Standard. Also, most of it is some very ugly "take a Kaseya backup, remove MSDE, reinstall SQL/Kaseya" type stuff. Sounds like a great way to make life difficult.

0. Take a Hyper-V snapshot of your Kaseya server.

1. Get a copy of SQL Server Standard.

2. Launch the setup using "setup.exe SKUUPGRADE=1"

3. When you get to the instance selection screen, click the "Installed instances" button to find your existing SQL Express instance and select it.

4. Continue through the wizard, your SQL Express instance will be updated to full-blown SQL 2005.

5. Restart the Kaseya Service. All of your agents may currently show offline. Don't panic, get a coffee, have a donut, come back in a few.

6. Oh look, we're back. Enjoy the rest of your weekend.

Now, I can't take full credit for this, but I'll try.