Further adventures with the HP N40L and Dragonfly BSD

Since my last post, I've tried a bunch of different things with this box. These include:

• Xenserver with the following virtual machines:
• Ubuntu Server
• Windows Home Server 2011
• FreeNAS
• DragonflyBSD
• Ubuntu Server
• Linux Mint 13

Having the 2TB mirror has caused many of the problems. FreeNAS simply crashed and used 100% of CPU the as soon as I tried to copy a file. Dragonfly wouldn't install. Ubuntu took 4 days to build a software RAID and then when we had a power failure, the RAID failed to build again. There was a lot of frustration and perhaps a cranky swear word or two.

Eventually I decided to follow this route:

• Installed Linux Mint 13 on the box, set up a software RAID for the large disks, and left the other disks simply as an install disk and an archive disk. 
• moved all the data from my Netgear Stora to the new server
• I've deployed it to replace my server and my media PC, allowing me to ditch two PCs, one UPS and the Stora.

Yes I'm going a bit greener :-) Thus far it's going quite well. I've managed to also build up a Dragonfly BSD backup server as well.

I took my old server and put the disks from my Stora and from an external USB drive that had a failed power converter. With 3 x 2 TB Disks and a 1 TB disk in the system I wanted a way to effectively use all that disk space. It's not for live data, just for periodic backups. The HAMMER file system is awesome for this because there is no fsck on boot - it's live and happening straight away. Dragonfly is also very lightweight and runs well on the system I've got. The HAMMER filesystem has a lot going for it too - I was able to add the three disks into a single filesystem of 5.5TB and it does the normal snapshots built into HAMMER. The set up and the speed it created the filesystems with was excellent. I had one small issue with my USB DVD drive during the installation which caused the whole thing to fall in a heap but once I used a USB drive the installation went quickly and smoothly.

Adventures with XenServer on the N40L Microserver

Since my last post (in amongst family and work and stuff) I've been playing with Xenserver and the N40L Microserver. Here's how it's gone down:

The initial install went quite well, except I had configured two mirrored RAID arrays - one with 2 x 1TB disks, the other with 2 x 2TB disks. Xenserver saw neither of them. So I removed both arrays, set the SATA controller back to AHCI and went from there.

After a quick re-install, I had an initial local storage of 1TB. Following this I added three more local storage devices and re-thought how I was going to do the installation of my VMs. Following the exceptional directions on creating an Ubuntu 12.04LTS server on Xenserver from http://www.invalidlogic.com/2012/05/01/deploying-ubuntu-12-04-on-xenserver-made-easy I created a 12.04 Ubuntu virtual server - all went very well. In accordance with my plan, I also created a Windows Home Server 2011 using the Windows 2008R2 template.

The WHS2011 server was so slow to install, setup and run. I had assigned it 2 CPU's, 4GB of RAM and it took at least three hours longer than the Ubuntu server to install and get going. I must note that the template from invalidlogic.com was superb and perhaps that has spoilt me a bit :-)

I also set up a FreeNAS - installed to one of the 1TB drives with 2 x 2TB virtual disks presented to it. The FreeNAS server installed quite fast and everything looked good. I set up NFS and CIFS shares and thought it was all going well. Unfortunately, as soon as I started to copy data across to the FreeNAS it's CPU usage hit 100% (as did the host's CPU). This was clearly a failure and wasn't going to work.

Undeterred, I thought about Dragonfly BSD and the excellent HAMMER file system - let's give that a crack I thought. The initial boot of the disk failed.... So FreeNAS and Dragonfly aren't going to play the game - new plan! More about this tomorrow :-)

Adventures with my new HP N40L Microserver

I took delivery today of my brand new HP N40L Microserver. I plan to use it to replace my existing whitebox server, Netgear Stora and add to my network at home. The idea is to install Citrix XenServer on this little box, then virtual guests running Ubuntu 10.04LTS, Windows Server (of some variety - 2008R2 or maybe Windows Home Server) and if required FreeNAS or another *BSD product (for fun).

The default N40L comes with 2GB of RAM, 250GB HDD and a 1.5GHz processor. I've updated the RAM to 8GB of RAM and I'll put a couple of 2TB HDDs in and probably 2 1TB HDDs disks as well.   The idea is then to set up two RAID arrays - one for the install of operating systems and associated applications, one as a storage pool for data (hopefully 2TB will be enough initially). I'll look into using 4 x 2TB disks in RAID10 and see what happens. Here we go!

Munin for Monitoring

Lately I've been very busy with work, but I find that I still need to keep an eye on the Linux servers I have kicking around at home and at the office. I use Nagios for basic monitoring (ping, smtp, http) but wanted something a bit more fine grained on the actual host itself. I stumbled on munin when doing some Google searching.

Munin's homepage is here: http://munin-monitoring.org and I'm running it on a variety of servers at them moment (mostly Ubuntu and Debian). It supports quite a few distributions and operating systems so check it out if you're interested in seeing what's happening on your servers.

I like the many graphs that it produces to demonstrate what's happening and I also fell into the trap of trying to over-configure things. There are only a couple of basic things to change to suit yourself, restart apache2 and munin-node and it's slamming. It also monitors multiple machines - install munin-node instead of munin and munin-node and configure as recommended. I thought I'd be tricky with it and things didn't go as planned so it appeared that it wasn't working. I was over thinking it though and it did work, very well actually, once I found the spot it was creating the files in :-)

A handy app and a nice way to keep on top of your servers!

Further experiences with Virtualization - VMware ESXi and Oracle Virtualization

So continuing in the vein of trying out these virtualization options in the marketplace I looked into the big gun - VMware to see how that would go on my cheapernetwork. Alas, my attempts were doomed to fail. The initial installation of ESXi failed - it was unable to detect the hard disk drives in my HP virtual servers. It was a bit of a WTF moment - that a fairly easily available mainboard, with no special interfaces or anything like that. The disks were fairly standard SATA disks - Seagate drives both. 160GB so not small and not unusually large. At any rate, after trying different disks, going to the console prompt and looking there I gave up. dmesg didn't even detect the /dev/sdi disks at all and the more I investigated the more I found that there were some hardware limitations. Given I have limited time, I abandoned my VMware attempts and instead looked into Oracle's Virtualization offerings.

The first thing that struck me was the sheer size of the downloads for the components of Oracle VM - it's a complex system, needing not only your head units (my HP desktops in this case) but also another PC as the manager - with an install of Oracle Linux required, plus the over 2GB VM Manager and then I needed two NFS shares for more data. In terms of complexity compared with XenServer I was surprised. I could install the VM Manager as a virtual machine on another host ( I decided to use my notebook running VMware player - which I love by the way). I had downloaded the netboot option for Oracle Linux - trying to minimise the impact on my corporate network. None of this went smoothly of course. VMware Player ran fine, the Oracle Linux install needed a URL to install from - and this proved surprisingly hard to find. I spent a bit of time searching for it, but the Oracle install information I found was for Oracle Linux Release 5, not 6 and very little information was available - perhaps my google-fu was letting me down... At any rate, again - time was against me. I had to let it go for the time being.

The thing that has struck me throughout this exercise is just how easy it was to get XenServer going - Citrix are really on to something. Additionally, for supported platforms under the XenServer virtual platforms it is silly easy to transfer guest machines between hosts - right click - move to server and off it goes. The downtime is extremely small. If you pay for XenServer you can enable High Availability so guests can be restarted on host fail and you can really start to get some high uptimes. If you know that the box is about to fail - it's trivial to migrate them around - fantastic!

So we'll keep playing with XenServer for the time being and with a bit of luck I can re-visit VMware down the track.

Experiences with Virtualisation - XenServer

I have been experimenting with different virtualisation technologies lately. At work I already run a Microsoft Hyper-V server with two Ubuntu servers running on it. One is an OTRS (ticketing system) server and the other an FTP / DNS server. This server is a 1RU rackmount box with no virtualisation supported on the hardware. Oh well... it still runs reasonably well, however, I've found that under high disk / network load, the virtual machines grind to an absolute halt and I have to reset them. So I've begun expanding my horizons.

Recently I searched for and found some small form factor desktops with support for virtualisation - namely: HP Compaq dc5750 Small Form Factor Black AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Dua. I bought two (for the princely sum of $9 each plus postage so $130 (!) delivered) - they came with 1GB of RAM and 80GB HDD. I've upgraded the RAM in both to 4GB and I've got more on order. I also turfed the disk drives - one had a dodgy sector and the other was just plain dodgy. I slammed a couple of 200GB disks in and away we went.

XenServer is produced by Citrix, a well known company for remote access solutions and now more so for their virtual server solutions. I've used VMware before on the desktop and still use the VMware Player on the desktop for various things, but I had not looked into XenServer before. I started with the Live CD and was reasonably encouraged. The information out of it looked pretty good so I thought I'd install the XenServer operating system on some machines and see how it went.

The install on my dc5750's went very smoothly - all hardware detected and accounted for. The dc5750 supports AMD virtualisation so it went very nicely and XenServer ran very happily. On the first server - xenserver1 (very imaginative naming) I neglected to set the timeserver or enable NTP and this did come back to bite me later on. After the initial set up, I installed XenCenter on my Windows 7 notebook. It's a slick interface and once I put the IP of xenserver1 in it detected it without issue. My notebook and both servers are on a gigabit interface so it all runs pretty fast. I started the install on the second dc5750 (xenserver2 - more imagination there) while I added a pool in XenCenter and put xenserver1 in to it as the master.

After xenserver2 was installed I added it to the pool and noticed that the tool wasn't reporting the RAM usage on the second server. I had fully updated both servers and XenCenter. Kind of strange - then I got messages about the clocks not being synchronised. I went back and reset the NTP servers on both machines and it turned out that xenserver1 was an hour ahead - once that was reset both servers reported CPU / RAM / Network and disk usage quite happily. So now to the installation of virtual machines - but where to put the virtual disks? Aha! I added a storage pool via NFS on our FreeNAS server - and although this in itself caused some issues until I sorted the NFS share out, eventually it was all good.

XenServer has templates that are used to create the virtual machines. There is, naturally, a blank template for unsupported operating systems (like *BSD?!). I started with an Ubuntu Server 11.10 install - the template suggested RAM usage, disk size etc and I created the the virtual machine very quickly. I had previously added a file storage pool for iso images - I pointed the server template at the appropriate ISO image and declined to point the VM at a particular xenserver, opting to allow it to choose one with the available resources. It chose xenserver2 and the installation began. I undocked the console so I could watch it and returned to the XenCenter to watch the load and usage on the servers. I also started a Windows Server 2008 R2 installation from the template for the hell of it (I love Microsoft TechNet Direct). Again, I allowed the template to set the configuration for the server and again allowed XenCentre to pick the server with the available resources - it chose xenserver1 and the installation began.

Both installs ran through their usual routines, until the Ubuntu server reached the disk partitioning stage and it stalled. The Windows 2008 R2 server install ran perfectly. It detected all the hardware properly and I installed the Xen tools on it without issue - the reporting detail in XenCentre improved markedly after that - individual cpu and RAM usage and network/disk usage too. The install was actually pretty quick across the network (I was surprised to say the least). After I restarted the Ubuntu install it ran again and finally completed. While this was happening I was updating the Windows 2008 R2 Server and I began an install of FreeBSD in a new server under the default template. It installed perfectly and once again, detected the hardware properly (detecting the network card as a RealTek device) and I was left with a fully functional FreeBSD Release 9.0 server. Eventually the Ubuntu Server finished installing too and it was working properly.

My initial impressions were good. The software was clear to understand, the virtual machines easy to manipulate and work with and the support for the hardware in the virtual machines was all good. Over the next few weeks I'll continue testing them and record my impressions here. Then, I'll take the disks out, install VMware's offering and test them too.

Microsoft's Hyper-V server is not really the system I wish to run - while it's great for Microsoft products, they aren't the only operating systems we run (for a variety of reasons). I like to be able to deploy the best suited OS to the requirement and I hate being locked in to anything - I really prefer to be flexible. I'll also cover some of the licensing costs as we go along - how much and how it's all costed out. Stay tuned!

A simple script to use either robocopy or xcopy to backup files

Under various circumstances, I've found it useful to cobble together a script to do a sync backup across the network from one Windows server to another. Usually this is for files only and is for either a mirror or a daily, full backup of data. Obviously there are some great backup tools available that make something like this largely unnecessary, however, this is quick, simple and gives you an email output of what has happened. The first example below uses robocopy (Robust Copy) which is a very nice bit of kit indeed. It's a bit more useful than xcopy and handles larger numbers of files better. Don't get me wrong, I love xcopy, but it has it's limitations. I use rsync a lot on Linux servers and robocopy gives me many similar options for how I want to handle files.

The destination directory could be anything - another folder on the same PC, a removable disk, a mapped share or even a straight UNC path e.g. \\server\share - flexibility is the key for this script, once the basic variables are right and you've decided to use robocopy or xcopy then off you go.

So to the script:

Open notepad and put this info in - note where things are comments and what variables you'll have to change:

Script start:

echo on
REM Set up some time variables
for /f "Tokens=1-4 Delims=/ " %%i in ('date /t') do set dt=%%l-%%k-%%j
for /f "Tokens=1" %%i in ('time /t') do set tm=-%%i
set tm=%tm::=-%
set dtt=%dt%%tm%
REM set up variables for log files, source and destination - change this variable
set log="C:\Users\owner\Documents\Scripts\Logs\%dt%.log"
REM local stuff to be backed up - change this variable
set src="c:\documents"
REM remote location to put backups - change this variable
set dest="I:\backups\server"
REM now for the actual work - change switches as required - explanation of switches is below.
robocopy %src% %dest% /E /Z /MIR /R:1 /LOG:%log%
REM I'd like to know how it went (this file can be big if there are a lot of files copied)
echo Backup Logs attached | blat - -subject "Sync Log Report for %dt%" -to "me@mydomain.com" -attach %log% -f user@domain.com

Use blat to send the email - grab it from www.blat.net (great program!) It sends an email with a header that will look like this:

Sync Log Report for 2012-02-10

and an attachment of your log file. You can add different things to this - for example I'll often use a [servername] tag after the date.

The robocopy switches used are:

• /E = copy sub-directories, including empty ones
• /Z = copy files in restartable mode (in case the network drops out or something similar)
• /MIR = MIRror a directory tree (which is /E plus /PURGE)
• /R:1 = number of retries on failed copies. It's best to set this - by default it's 1 million (!)

I run this from the Windows Scheduler and have a mirrored copy of data files each night. It's quite a useful little tool. If you'd like to use xcopy instead there are a few things to consider:

• the src and dest variables need to have a trailing backslash and a wildcard
• set src="c:\documents\*"
• set dest="i:\backups\server\*"
• and the command to insert would be:
• xcopy %src% %dest% /C /D /E /H /Y > %log%
• where the switches are:
• /C = continue copying even if there are errors
• /D = copies files whose source is newer
• /E = copies directories and sub-directories (even if empty)
• /H = copies hidden and system files
• /Y = suppresses prompting to overwrite files
• the > redirects xcopy's output to the %log% variable we configured earlier in the script, and then blat will email the resulting file out.

If you find this useful in anyway, please let me know in the comments.