A tale of survival - FreeNAS and ZFS - and four disks with failed sectors

Recently, one of the FreeNAS storage devices we have at the office started to generate failed sectors on two of the disks. While an eyebrow raising event in and of itself, I wasn't particularly concerned. Living in the virtual outback as we do, I ordered some more disks. About 8 days later, the third of the four disks in our NAS started to throw out errors! Uh oh, it appears that we were on a slippery slope towards Doom!

I called the supplier demanding my disks only to find out they'd ordered WD Green drives! Noooo. I amended the order to get WD Red drives (which are designed for a NAS) and was informed it would take a day or two. The next morning the final disk was generating errors. We were getting close to some serious error thresholds on two of the disks and the third and fourth were well on the way...

Impatiently waiting for the new disks to arrive, I kept a close eye on the NAS. FreeNAS emailed me with alarming frequency about disk failure, imminent apocalypse and the like. The next day the WD Red drives arrived and three of the four disks were now generating large numbers of errors. I shut the NAS down (not taking the pool offline like I should have!) and replaced the most error prone disk. Restarting the NAS I added it back to the pool, replacing one of the dead disks and let it rebuild. Gradually I replaced all the disks until the pool was degraded with a corrupted file. 

On this filesystem are all my virtual machines, so I was a bit concerned about which file was corrupt. Thankfully it was an old backup of my current Windows 7 workstation so I deleted it. Oddly, I was unable to remove two of the old disks - every time I tried it would add them back.

After a bit of head scratching I realised I needed to delete the snapshots and once I did that, I was able to remove the disks from the pool and it changed from Degraded to Online and services were all restored. I've checked over the disks and every single one has failed since. One file lost out of almost 3TB of data - thank you ZFS and FreeNAS! Note to self - sort the backups out!

How to update XenServer 6.2

Since XenServer 6.2 came out, things have changed for the Citrix developed virtualisation environment. Specifically, the free licence no longer applies - you get a fully featured system out of the box and licensing is applied per socket. What this means in a practical sense for keeping your server updated is that using XenCenter to update it is not an option unless you've licensed it.

If you're like me and running XenServer in a quasi test / virtual environment, you won't have paid for a licence (as much as you'd like to). Therefore updating your VM hosts is a bit different and you have to use the command line. Now, it's important to note a couple of things that the XenServer FAQs don't necessarily specify. You have to use the xe command line program to upload and apply the patches. Do this in the following way:

Open up a cmd process (Windows+r, then type cmd and hit enter).

Go to the correct directory: cd C:\Program Files (x86)\Citrix\XenCenter and hit enter.

In this folder is the xe.exe application we'll need. The next thing to do is download the patches from Citrix. Usually they'll land in your Downloads folder. Extract the files from within and note where you've put them - typically it ends up being c:\users\ryv\\Downloads\XS62E002\ for example

You have to use xe.exe to upload and apply these patches to your server - send it to the Pool Master using either it's IP or hostname (if you have DNS set up correctly).

The syntax is:

xe patch-upload -s <hostname / IP> -u root -pw <password> file-name=<path to file>\XS62E002.xsupdate

In real life this command might look like this:

xe patch-upload -s 10.0.0.100 -u root -pw s3cret123 file-name=c:\users\ryv\Downloads\XS62E002\XS62E002.xsupdate

Once the file is uploaded, it gives you a UUID of the hotfix. It might look like this: 59128f15-92cd-4dd9-8fbe-a0115d1b07b4

Make a note of this - we'll need it in a tick.

To apply the hotfix to your hosts in the pool, the syntax looks like this:

xe -s <hostname / IP> -u root -pw <password> patch-pool-apply uuid=<hotfix UUID> and then hit enter.

In real life it might look like this:

xe -s 10.0.0.100 -u root -pw s3cret123 patch-pool-apply uuid=59128f15-92cd-4dd9-8fbe-a0115d1b07b4

You'll really need to reboot the hosts after this. Verifying the application is easy - go to XenCentre and choose your Pool, then click the General Tab and expand Updates. Note that the update is "Fully Applied" and you're done!

Have fun!

Further adventures of XenServer on the HP N40L Microserver - a follow up

If you've read my previous post on this matter you'll know how delighted I was to get the SBS2003 server running so well under XenServer. Well it's been about 300 days since I did this work. How do I know it's been 300 days? Easy - the SBS2003 server crashed for other reasons and I was compelled to reboot both - with an uptime of 281 days! Apparently it has been running flawlessly - indeed the logs from both servers would suggest this.

Backups to a NAS were set up shortly after the server was commissioned and these have been running like a freight train - for which I'm profoundly grateful. Fortunately in this instance the server simply needed a restart and it was up, running and doing it's job quite happily.

I should also note that I blew a tonne of dust and dirt out of it and the whole time the N40L was like the little train that could :-)

If you have a low usage server and you're looking for a simply solution and doing a P2V migration doesn't scare you, then this is a fine option.

I now have two N40L's at home - one running FreeNAS and the other running Mint - both are running very well. I note that the price for them has gone up to $300 from eBay now....