Test Disk = Good

Test Disk, from the guys over at CGSecurity, gets a huge thumbs up from me today. My son had a school project due this week that I helped him with yesterday involving pics from our old digital camera. All the pictures were taken and we get to the part where we need to print them, and the SD card isn't recognized in my laptop. Figuring it was a problem with my laptop, I reboot and try again, still nothing. Placing the card back in the camera results in:

"Hey cool, a new SD card. Want me to format it?"*

*Error message paraphrased.

I'm not about to spend money on this, but I'm definitely hoping to save some time. Lifehacker had done a bit on the "best recovery tools" a while back that I checked out. I tried "Recuva" first, but was disappointed. Turns out, even though Recuva won their little showdown, it's not actually good for any sort of failure situation where the card won't mount or the partition is screwed. It's really just an undelete tool. But next in line was Test Disk. It's cousin is called PhotoRec and that's what I used to recover everything we needed off the SD card, and then some. It also pulled off some old MP3 tracks and some videos that used to live on that card.

So, it worked really well, and I highly recommend it!

Are your backups good enough?

I caught this article at Ars Technica today.

http://arstechnica.com/web/news/2009/05/hackers-crack-flight-sim-community-site-ruin-it-for-everyone.ars

I'm not a flight sim guy by any stretch, but I think we can all agree: losing 13 years of work is one thing, losing 13 years of what you do in your spare time, your labor of love so to speak, is absolutely gut-wrenching.

I'm sympathetic to these guys, but unfortuntely there's just no excuse. Their backup strategy has been inappropriate and insufficent since the beginning, and noone caught it. Their entire disaster recovery scenario rested with a backup server that was online and (somehow) accessible? Wow, just wow.

So, they'll probably rebuild, and it will likely take a whole bunch of time. But if they take away one thing from this fiasco, its that they had better take their backup and disaster recovery strategy very, very seriously.

Hyper-V and Windows Server Backup

So, this is still going on. I can't reliably use Windows Server Backup to back up Hyper-V virtual machines to another volume with Backup Integration Services enabled (translation: I can't do live backups of virtual machines).

I first became aware of this about 7 months ago. When our Hyper-V hosts started crashing.

Omar Droubi - Posted on Tuesday, October 21, 2008 9:46:03 AM
Hello All,

After much communication with PSS they have acknowledged that the hyper-v vss writer issues is due to a known Windows Bug

This bug was first discovered in Windows 7 and has now been reported in Windows Server 2008.

Now the good news.

The fix for this bug will be incorporated in Windows Server 2008 Service Pack 2 (ETA currently unknown). If they get a fix out sooner I will try to share as best I can- if they happen to released it as a separate hotfix and they notify me about it- since I am now on the list of affected clients/partners who have open cases.

For the mean time I have implemented the work around of simply disabling the VSS backup integration on each of my VMs. This has proved to work well enough for me since I backup at the physical server level using Windows backup.

My guest machines go into saved state and start back up with out issue- their downtime is only about 2 to 4 minutes total per physical host and me and my client find this acceptable as the backup runs off hours.

(I am running Win2k8 VMs only-plus a single W2k3 VM)- within each VM I have assigned its own dedicated disk for backup using Windows Server backup (NTBACKUP on W2k3) in case I need to restore at the VM level. and if the parent partition goes corrupt I can recover the entire server using the Windows backup I run to external USB drives.

Poor mans backup- but in this economy- for this particular client- 5 physical servers-plus 10 hyper-V guests- this solution works very well. (backup to external USB drives that are swapped a few times per week.

Does anyone know of issues or unsupported scenarios where putting a machine in a saved state is risky or not recommended?

Thanks,Omar

Reference: http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/d7026f87-4f58-4576-b89f-302c44da6532/#page:1

That forum thread is a good read, and there's lots of hope in there. For example, when we first ran into the issue it was on our own Hyper-V server. So our first reaction was to change freeze all of the Hyper-V hosts that we managed. This actually worked for a while - until a stupid blunder with our update management system and some 2008 updates "leaked". Then those machines started crashing too. So, our current workaround, and we're still using it, is to disable the "Backup Integration Service" on all of our guest VMs and scheduling the backup to occur at night.

What this does is save the state of the VM briefly while the VSS snapshot is taken. Then the VM wakes up and the backup of the volume continues. It works really well, but its actually a step backwards for us, since we've been doing live VM VSS backups since the Virtual Server 2005 days.

The thread goes on to mention things like issues in Hyper-V VMs that have SCSI disks/devices, etc. I don't doubt that those are issues, just not for us. The very last post of that thread sounds extremely promising: ensuring that all of the volumes that have Hyper-V VHDs on them have dedicated shadowstorage. Unfortunately my own testing has not revealed that to completely fix the problem. At best it reduces the crashing from "every backup" to "every other backup".

So, like Omar, I guess we're still waiting for Service Pack 2, or whatever. I think Hyper-V is a wonderful thing, but I sure wish they'd fix this.

Backing Up Windows Home Server

So, I haven't really been paying attention. NTBackup does not work for backing up a WHS. I mean, we've got it backing up our PCs, and that's fine. But what about the server? Vlad Mazek talked about this a bit on his blog a while back. I've already got the remote critical data backups covered, but as I mentioned before, I'm lazy and I don't want to rebuild everything from scratch if my WHS dies. Would ShadowProtect or some other image based software help us here? Its a solid maybe - WHS makes use of logical volumes and the bastard step-child of software mirroring. Maybe it wouldn't matter if you restored the entire image to one large drive. I might need to try that.

Anyway here's how I discovered this - I was futzing with the WHS web server files today as I migrated my websites from a virtualized Linux box over to IIS on the home server. I modified a silly .html file, saved too quickly, and then decided I wanted the original back. So I went to grab an old version out of my latest NTbackup. As I attempted to index the .bkf file and restore the document I wanted, I'm greeted by the above. So no love there. Really, all I want is for the semi-important stuff to be "off-box", ideally a local image, since my really critical stuff is already off-site using a remote backup service. At this point it doesn't look likely that we can replicate the PC backups off the box, but I'm not too worried about those either.

So until something better comes along, you can actually still use NTBackup if you want, you just can't back up the fancy logical volume in WHS (which makes up most of the D: drive). What you can back up is C:\, SystemState, and then your WHS shares over the network (i.e. you would add \\ server\users) to your backup selections. Could you do a bare-metal restore from this? Ehm, probably not. But's it's something I'd try, and maybe some day I will. In the meantime, it allows you to use a familiar, reliable backup tool to offload important and critical files to another device or elsewhere on your home network.

So is this enough? If your critical data is covered, then yeah, I think so. You could avoid ALL of the issues above I suppose and run WHS inside Virtual Server or Hyper-V, but then I think you lose one really nice thing that WHS lets you do - use cobbled-together hardware you already had laying around the house. The way the logical volume system works, its designed for that, with disk drives that WILL fail in mind. I've had two drives fail in mine in the last year, and in both cases the process to replace them was pretty painless. Your mileage will definitely vary.