A tale of survival - FreeNAS and ZFS - and four disks with failed sectors

Recently, one of the FreeNAS storage devices we have at the office started to generate failed sectors on two of the disks. While an eyebrow raising event in and of itself, I wasn't particularly concerned. Living in the virtual outback as we do, I ordered some more disks. About 8 days later, the third of the four disks in our NAS started to throw out errors! Uh oh, it appears that we were on a slippery slope towards Doom!

I called the supplier demanding my disks only to find out they'd ordered WD Green drives! Noooo. I amended the order to get WD Red drives (which are designed for a NAS) and was informed it would take a day or two. The next morning the final disk was generating errors. We were getting close to some serious error thresholds on two of the disks and the third and fourth were well on the way...

Impatiently waiting for the new disks to arrive, I kept a close eye on the NAS. FreeNAS emailed me with alarming frequency about disk failure, imminent apocalypse and the like. The next day the WD Red drives arrived and three of the four disks were now generating large numbers of errors. I shut the NAS down (not taking the pool offline like I should have!) and replaced the most error prone disk. Restarting the NAS I added it back to the pool, replacing one of the dead disks and let it rebuild. Gradually I replaced all the disks until the pool was degraded with a corrupted file. 

On this filesystem are all my virtual machines, so I was a bit concerned about which file was corrupt. Thankfully it was an old backup of my current Windows 7 workstation so I deleted it. Oddly, I was unable to remove two of the old disks - every time I tried it would add them back.

After a bit of head scratching I realised I needed to delete the snapshots and once I did that, I was able to remove the disks from the pool and it changed from Degraded to Online and services were all restored. I've checked over the disks and every single one has failed since. One file lost out of almost 3TB of data - thank you ZFS and FreeNAS! Note to self - sort the backups out!

HP N40L and FreeNAS 8.3.0

My existing HP N40L Microserver is running out of disk space. 2 TB is not enough it turns out. So I thought why not add another N40L to my network? After all, it's been a success with my existing one thus far.... So on to eBay I went, and I found an Australian company selling them for $209 delivered! I'm amazed these are so cheap - after all even low end PCs are more than this. So I ordered one up and it arrived three days later. I put a couple of 2 TB disks into the box, an 8 GB RAM DIMM and an 8 GB usb drive. Half an hour later I had FreeNAS 8.3.0 installed and a 2 TB array set up.

With an NFS share I can access the 2 TB array from my media PC and it all runs brilliantly. I've got space to add in two extra drives, and once I get two more disks I'll install them - running two 2TB mirrors and sharing out data easily. The N40L runs very quietly and efficiently and even running two of them is very quiet in the lounge room. I've used Western Digital Green Disks from 2 TB external USB drives. For $109 each plus the $209 for the N40L means that for $436 I've got a reasonable little NAS here. Another $218 and I've got a 4 TB NAS! It's stable and runs brilliantly. FreeNAS is an excellent platform for this, easy to upgrade and very stable with a wide range of network protocols available for connection to it. I'd heartily suggest using a server like this for a backup server or simple data storage. Add a couple of extra gigabit ethernet ports via the PCI Express card slots and then LAGG them together for greater through put and this simple and inexpensive NAS has even more applications in the business arena. I would strongly recommend 8 GB or more of RAM so pre-caching can be effected - this will improve data delivery.

As I type this I note an update for FreeNAS has become available so I'll grab that and install it!

FreeNAS details here: http://www.freenas.org

HP N40L Microserver details can be found here - HP N40L Microserver (URL truncated because it's awful)

Further adventures with the HP N40L and Dragonfly BSD

Since my last post, I've tried a bunch of different things with this box. These include:

• Xenserver with the following virtual machines:
• Ubuntu Server
• Windows Home Server 2011
• FreeNAS
• DragonflyBSD
• Ubuntu Server
• Linux Mint 13

Having the 2TB mirror has caused many of the problems. FreeNAS simply crashed and used 100% of CPU the as soon as I tried to copy a file. Dragonfly wouldn't install. Ubuntu took 4 days to build a software RAID and then when we had a power failure, the RAID failed to build again. There was a lot of frustration and perhaps a cranky swear word or two.

Eventually I decided to follow this route:

• Installed Linux Mint 13 on the box, set up a software RAID for the large disks, and left the other disks simply as an install disk and an archive disk. 
• moved all the data from my Netgear Stora to the new server
• I've deployed it to replace my server and my media PC, allowing me to ditch two PCs, one UPS and the Stora.

Yes I'm going a bit greener :-) Thus far it's going quite well. I've managed to also build up a Dragonfly BSD backup server as well.

I took my old server and put the disks from my Stora and from an external USB drive that had a failed power converter. With 3 x 2 TB Disks and a 1 TB disk in the system I wanted a way to effectively use all that disk space. It's not for live data, just for periodic backups. The HAMMER file system is awesome for this because there is no fsck on boot - it's live and happening straight away. Dragonfly is also very lightweight and runs well on the system I've got. The HAMMER filesystem has a lot going for it too - I was able to add the three disks into a single filesystem of 5.5TB and it does the normal snapshots built into HAMMER. The set up and the speed it created the filesystems with was excellent. I had one small issue with my USB DVD drive during the installation which caused the whole thing to fall in a heap but once I used a USB drive the installation went quickly and smoothly.

Adventures with XenServer on the N40L Microserver

Since my last post (in amongst family and work and stuff) I've been playing with Xenserver and the N40L Microserver. Here's how it's gone down:

The initial install went quite well, except I had configured two mirrored RAID arrays - one with 2 x 1TB disks, the other with 2 x 2TB disks. Xenserver saw neither of them. So I removed both arrays, set the SATA controller back to AHCI and went from there.

After a quick re-install, I had an initial local storage of 1TB. Following this I added three more local storage devices and re-thought how I was going to do the installation of my VMs. Following the exceptional directions on creating an Ubuntu 12.04LTS server on Xenserver from http://www.invalidlogic.com/2012/05/01/deploying-ubuntu-12-04-on-xenserver-made-easy I created a 12.04 Ubuntu virtual server - all went very well. In accordance with my plan, I also created a Windows Home Server 2011 using the Windows 2008R2 template.

The WHS2011 server was so slow to install, setup and run. I had assigned it 2 CPU's, 4GB of RAM and it took at least three hours longer than the Ubuntu server to install and get going. I must note that the template from invalidlogic.com was superb and perhaps that has spoilt me a bit :-)

I also set up a FreeNAS - installed to one of the 1TB drives with 2 x 2TB virtual disks presented to it. The FreeNAS server installed quite fast and everything looked good. I set up NFS and CIFS shares and thought it was all going well. Unfortunately, as soon as I started to copy data across to the FreeNAS it's CPU usage hit 100% (as did the host's CPU). This was clearly a failure and wasn't going to work.

Undeterred, I thought about Dragonfly BSD and the excellent HAMMER file system - let's give that a crack I thought. The initial boot of the disk failed.... So FreeNAS and Dragonfly aren't going to play the game - new plan! More about this tomorrow :-)

Adventures with my new HP N40L Microserver

I took delivery today of my brand new HP N40L Microserver. I plan to use it to replace my existing whitebox server, Netgear Stora and add to my network at home. The idea is to install Citrix XenServer on this little box, then virtual guests running Ubuntu 10.04LTS, Windows Server (of some variety - 2008R2 or maybe Windows Home Server) and if required FreeNAS or another *BSD product (for fun).

The default N40L comes with 2GB of RAM, 250GB HDD and a 1.5GHz processor. I've updated the RAM to 8GB of RAM and I'll put a couple of 2TB HDDs in and probably 2 1TB HDDs disks as well.   The idea is then to set up two RAID arrays - one for the install of operating systems and associated applications, one as a storage pool for data (hopefully 2TB will be enough initially). I'll look into using 4 x 2TB disks in RAID10 and see what happens. Here we go!

Experiences with Virtualisation - XenServer

I have been experimenting with different virtualisation technologies lately. At work I already run a Microsoft Hyper-V server with two Ubuntu servers running on it. One is an OTRS (ticketing system) server and the other an FTP / DNS server. This server is a 1RU rackmount box with no virtualisation supported on the hardware. Oh well... it still runs reasonably well, however, I've found that under high disk / network load, the virtual machines grind to an absolute halt and I have to reset them. So I've begun expanding my horizons.

Recently I searched for and found some small form factor desktops with support for virtualisation - namely: HP Compaq dc5750 Small Form Factor Black AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Dua. I bought two (for the princely sum of $9 each plus postage so $130 (!) delivered) - they came with 1GB of RAM and 80GB HDD. I've upgraded the RAM in both to 4GB and I've got more on order. I also turfed the disk drives - one had a dodgy sector and the other was just plain dodgy. I slammed a couple of 200GB disks in and away we went.

XenServer is produced by Citrix, a well known company for remote access solutions and now more so for their virtual server solutions. I've used VMware before on the desktop and still use the VMware Player on the desktop for various things, but I had not looked into XenServer before. I started with the Live CD and was reasonably encouraged. The information out of it looked pretty good so I thought I'd install the XenServer operating system on some machines and see how it went.

The install on my dc5750's went very smoothly - all hardware detected and accounted for. The dc5750 supports AMD virtualisation so it went very nicely and XenServer ran very happily. On the first server - xenserver1 (very imaginative naming) I neglected to set the timeserver or enable NTP and this did come back to bite me later on. After the initial set up, I installed XenCenter on my Windows 7 notebook. It's a slick interface and once I put the IP of xenserver1 in it detected it without issue. My notebook and both servers are on a gigabit interface so it all runs pretty fast. I started the install on the second dc5750 (xenserver2 - more imagination there) while I added a pool in XenCenter and put xenserver1 in to it as the master.

After xenserver2 was installed I added it to the pool and noticed that the tool wasn't reporting the RAM usage on the second server. I had fully updated both servers and XenCenter. Kind of strange - then I got messages about the clocks not being synchronised. I went back and reset the NTP servers on both machines and it turned out that xenserver1 was an hour ahead - once that was reset both servers reported CPU / RAM / Network and disk usage quite happily. So now to the installation of virtual machines - but where to put the virtual disks? Aha! I added a storage pool via NFS on our FreeNAS server - and although this in itself caused some issues until I sorted the NFS share out, eventually it was all good.

XenServer has templates that are used to create the virtual machines. There is, naturally, a blank template for unsupported operating systems (like *BSD?!). I started with an Ubuntu Server 11.10 install - the template suggested RAM usage, disk size etc and I created the the virtual machine very quickly. I had previously added a file storage pool for iso images - I pointed the server template at the appropriate ISO image and declined to point the VM at a particular xenserver, opting to allow it to choose one with the available resources. It chose xenserver2 and the installation began. I undocked the console so I could watch it and returned to the XenCenter to watch the load and usage on the servers. I also started a Windows Server 2008 R2 installation from the template for the hell of it (I love Microsoft TechNet Direct). Again, I allowed the template to set the configuration for the server and again allowed XenCentre to pick the server with the available resources - it chose xenserver1 and the installation began.

Both installs ran through their usual routines, until the Ubuntu server reached the disk partitioning stage and it stalled. The Windows 2008 R2 server install ran perfectly. It detected all the hardware properly and I installed the Xen tools on it without issue - the reporting detail in XenCentre improved markedly after that - individual cpu and RAM usage and network/disk usage too. The install was actually pretty quick across the network (I was surprised to say the least). After I restarted the Ubuntu install it ran again and finally completed. While this was happening I was updating the Windows 2008 R2 Server and I began an install of FreeBSD in a new server under the default template. It installed perfectly and once again, detected the hardware properly (detecting the network card as a RealTek device) and I was left with a fully functional FreeBSD Release 9.0 server. Eventually the Ubuntu Server finished installing too and it was working properly.

My initial impressions were good. The software was clear to understand, the virtual machines easy to manipulate and work with and the support for the hardware in the virtual machines was all good. Over the next few weeks I'll continue testing them and record my impressions here. Then, I'll take the disks out, install VMware's offering and test them too.

Microsoft's Hyper-V server is not really the system I wish to run - while it's great for Microsoft products, they aren't the only operating systems we run (for a variety of reasons). I like to be able to deploy the best suited OS to the requirement and I hate being locked in to anything - I really prefer to be flexible. I'll also cover some of the licensing costs as we go along - how much and how it's all costed out. Stay tuned!