VMware nightmares

After a great couple of days away I was called urgently to work - one of my client's networks was down and my colleague was stuck. The VMsphere client couldn't see the hosts, which couldn't see their datastores and the network had zero stability. Yay. Just wanted I wanted to come home to.

The servers are running ESXi 4.1 that needs a few updates but the networks stability has always been a real issue for us. We took it over from some other chaps, solving a long series of issues with the servers and particularly the IOmega NAS. Throw in a database issue on one of the guest servers that kept the disk IO at absolute peak all the time and things were pretty tricky. All those things have been resolved, more or less, and the network had been fully functional for some months. So why did it change? Several reasons and I hope you take away from this some ideas for yourself.

The NAS appeared to have dropped a disk and while it has a RAID10 set up, it paralysed the system (not at all like the FreeNAS I've talked about before). The whole thing fell over and we shut it down before reseating the disk and powering back up. The NAS detected the disk and began a rebuild. The VMware hosts couldn't see the datastore though and the server managing the virtual environment periodically disconnected from everything. Initially we thought it was a network issue and restarted everything. The hosts were slow to come up, with errors in the logs indicating an inability to see the iSCSI disks. We could connect to the NAS via it's web interface, ping it and it looked quite happy on the network so we couldn't understand what was happening. An added complication was we have a new HP NAS on the network and while we were able to migrate several of the hosts to it, we've had problems getting then started. Don't know why, but the host's cpu goes through the roof everytime we try to start them. I thought we might have an all paths down bug and most of the documentation suggests a call to VMware and let them sort it out. At 8pm at night this isn't so great a plan, and with the client losing production time and money we had to solve it.

So with all these errors and problems left and right I was at a loss. Eventually we took inspiration from The IT Crowd and turned it all off, counted to 10 and started it back up. Would you believe that it took 3 reboots of the management server before the VMsphere client would connect to anything - especially considering it was running locally! The iSCSI shares from the NAS became available finally - there was a service issue on the iOmega NAS that was failing silently. It was alleging that all was good and it really wasn't. Now those shares were available we were able to reconnect to the datastores and boot the guest machines. The management server was still disconnecting from the hosts constantly but we were able to at least get things going. There is still quite a bit to do there but the servers are finally running.

Going forward I think we'll use XenServer and NFS shares. Simpler, fully functional and quick. Easy to backup and expand disks. Adios VMware I reckon.

The final thought is "Have you turned it off and back on again?" Something in that for all of us :-)

Further experiences with Virtualization - VMware ESXi and Oracle Virtualization

So continuing in the vein of trying out these virtualization options in the marketplace I looked into the big gun - VMware to see how that would go on my cheapernetwork. Alas, my attempts were doomed to fail. The initial installation of ESXi failed - it was unable to detect the hard disk drives in my HP virtual servers. It was a bit of a WTF moment - that a fairly easily available mainboard, with no special interfaces or anything like that. The disks were fairly standard SATA disks - Seagate drives both. 160GB so not small and not unusually large. At any rate, after trying different disks, going to the console prompt and looking there I gave up. dmesg didn't even detect the /dev/sdi disks at all and the more I investigated the more I found that there were some hardware limitations. Given I have limited time, I abandoned my VMware attempts and instead looked into Oracle's Virtualization offerings.

The first thing that struck me was the sheer size of the downloads for the components of Oracle VM - it's a complex system, needing not only your head units (my HP desktops in this case) but also another PC as the manager - with an install of Oracle Linux required, plus the over 2GB VM Manager and then I needed two NFS shares for more data. In terms of complexity compared with XenServer I was surprised. I could install the VM Manager as a virtual machine on another host ( I decided to use my notebook running VMware player - which I love by the way). I had downloaded the netboot option for Oracle Linux - trying to minimise the impact on my corporate network. None of this went smoothly of course. VMware Player ran fine, the Oracle Linux install needed a URL to install from - and this proved surprisingly hard to find. I spent a bit of time searching for it, but the Oracle install information I found was for Oracle Linux Release 5, not 6 and very little information was available - perhaps my google-fu was letting me down... At any rate, again - time was against me. I had to let it go for the time being.

The thing that has struck me throughout this exercise is just how easy it was to get XenServer going - Citrix are really on to something. Additionally, for supported platforms under the XenServer virtual platforms it is silly easy to transfer guest machines between hosts - right click - move to server and off it goes. The downtime is extremely small. If you pay for XenServer you can enable High Availability so guests can be restarted on host fail and you can really start to get some high uptimes. If you know that the box is about to fail - it's trivial to migrate them around - fantastic!

So we'll keep playing with XenServer for the time being and with a bit of luck I can re-visit VMware down the track.

Experiences with Virtualisation - XenServer

I have been experimenting with different virtualisation technologies lately. At work I already run a Microsoft Hyper-V server with two Ubuntu servers running on it. One is an OTRS (ticketing system) server and the other an FTP / DNS server. This server is a 1RU rackmount box with no virtualisation supported on the hardware. Oh well... it still runs reasonably well, however, I've found that under high disk / network load, the virtual machines grind to an absolute halt and I have to reset them. So I've begun expanding my horizons.

Recently I searched for and found some small form factor desktops with support for virtualisation - namely: HP Compaq dc5750 Small Form Factor Black AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Dua. I bought two (for the princely sum of $9 each plus postage so $130 (!) delivered) - they came with 1GB of RAM and 80GB HDD. I've upgraded the RAM in both to 4GB and I've got more on order. I also turfed the disk drives - one had a dodgy sector and the other was just plain dodgy. I slammed a couple of 200GB disks in and away we went.

XenServer is produced by Citrix, a well known company for remote access solutions and now more so for their virtual server solutions. I've used VMware before on the desktop and still use the VMware Player on the desktop for various things, but I had not looked into XenServer before. I started with the Live CD and was reasonably encouraged. The information out of it looked pretty good so I thought I'd install the XenServer operating system on some machines and see how it went.

The install on my dc5750's went very smoothly - all hardware detected and accounted for. The dc5750 supports AMD virtualisation so it went very nicely and XenServer ran very happily. On the first server - xenserver1 (very imaginative naming) I neglected to set the timeserver or enable NTP and this did come back to bite me later on. After the initial set up, I installed XenCenter on my Windows 7 notebook. It's a slick interface and once I put the IP of xenserver1 in it detected it without issue. My notebook and both servers are on a gigabit interface so it all runs pretty fast. I started the install on the second dc5750 (xenserver2 - more imagination there) while I added a pool in XenCenter and put xenserver1 in to it as the master.

After xenserver2 was installed I added it to the pool and noticed that the tool wasn't reporting the RAM usage on the second server. I had fully updated both servers and XenCenter. Kind of strange - then I got messages about the clocks not being synchronised. I went back and reset the NTP servers on both machines and it turned out that xenserver1 was an hour ahead - once that was reset both servers reported CPU / RAM / Network and disk usage quite happily. So now to the installation of virtual machines - but where to put the virtual disks? Aha! I added a storage pool via NFS on our FreeNAS server - and although this in itself caused some issues until I sorted the NFS share out, eventually it was all good.

XenServer has templates that are used to create the virtual machines. There is, naturally, a blank template for unsupported operating systems (like *BSD?!). I started with an Ubuntu Server 11.10 install - the template suggested RAM usage, disk size etc and I created the the virtual machine very quickly. I had previously added a file storage pool for iso images - I pointed the server template at the appropriate ISO image and declined to point the VM at a particular xenserver, opting to allow it to choose one with the available resources. It chose xenserver2 and the installation began. I undocked the console so I could watch it and returned to the XenCenter to watch the load and usage on the servers. I also started a Windows Server 2008 R2 installation from the template for the hell of it (I love Microsoft TechNet Direct). Again, I allowed the template to set the configuration for the server and again allowed XenCentre to pick the server with the available resources - it chose xenserver1 and the installation began.

Both installs ran through their usual routines, until the Ubuntu server reached the disk partitioning stage and it stalled. The Windows 2008 R2 server install ran perfectly. It detected all the hardware properly and I installed the Xen tools on it without issue - the reporting detail in XenCentre improved markedly after that - individual cpu and RAM usage and network/disk usage too. The install was actually pretty quick across the network (I was surprised to say the least). After I restarted the Ubuntu install it ran again and finally completed. While this was happening I was updating the Windows 2008 R2 Server and I began an install of FreeBSD in a new server under the default template. It installed perfectly and once again, detected the hardware properly (detecting the network card as a RealTek device) and I was left with a fully functional FreeBSD Release 9.0 server. Eventually the Ubuntu Server finished installing too and it was working properly.

My initial impressions were good. The software was clear to understand, the virtual machines easy to manipulate and work with and the support for the hardware in the virtual machines was all good. Over the next few weeks I'll continue testing them and record my impressions here. Then, I'll take the disks out, install VMware's offering and test them too.

Microsoft's Hyper-V server is not really the system I wish to run - while it's great for Microsoft products, they aren't the only operating systems we run (for a variety of reasons). I like to be able to deploy the best suited OS to the requirement and I hate being locked in to anything - I really prefer to be flexible. I'll also cover some of the licensing costs as we go along - how much and how it's all costed out. Stay tuned!

Review: Sun VirtualBox

Recently I've had to find a virtual machine emulator. Given my limited budget (read: zero) I was hunting around for various different pieces of software. Microsoft have their virtual machine software: Virtual PC which is a polished bit of software. It is, however, not up to par with what I want to do. Because I run a host Windows machine and want to use Linux as a guest operating system, VirtualPC has all sorts of issues with it. I tried it and didn't like it. I stumbled upon VirtualBox (www.virtualbox.org) which appears to be sponsored by Sun. Given Sun's previous support of Open Source, I decided to try it.

The installation is very simple and straightforward - it installs extra network adapters, DHCP servers for the internal NAT'd network and a few other bits and pieces. The guest OS support it offers is excellent. In a *very* short amount of time I had a copy of Debian running from the netboot cd, installed and configured an FTP server and I was very happy. My host operating system was Windows Server 2008, which isn't 100% supported but does the job very nicely anyway. I've run it under Windows XP too and it works very well (especially on my Dell D610 Latitude with a 1.6GHz Centrino processor and 2GB of RAM). I'm going to test OpenSolaris on it and see how it goes under the installation.

Some years ago I ran VMWare Desktop to have many virtual machines for testing and other purposes and I found it to be rock solid. The VMWare gear is very good. I have found, to my considerable pleasure, that VirtualBox is as solid and feature filled as VMWare was. It's free status is even better and I love the fact that it supports Linux so beautifully. I'll be experimenting with various other guest OSs in the next few days and I'll let you know my progress as I play with it.

Virtualisation in the server space

Recently I've had the chance to view VMWare (www.vmware.com) and HP's (www.hp.com.au) entries in the virtualisation market - through VMWare's ESXi server and HP's Blade servers and SAN systems. Prior to this presentation I'd only looked into virtualisation a little bit, concerns for server redundancy and the like preventing me from spending a lot of time looking into it, and also, the size and scope of the organisation I work for is insufficient for such things - or so I had thought.

I have about 10 servers under my direct control and of these, 6 could be virtualised. Thinking about it going forward, in 3 to 5 years when these servers are due for replacement, buying 2 servers and a small SAN will cost less than the 6 new servers and provide the same services. At the end of the day, I think that's what it is all about - same services (or better), with uptime being up in the 5 nines range and cost-effective hardware/software. I've been a fan of VMWare's virtualisation stuff for several years, solely through the VMWare desktop range. Running multiple operating systems and applications on a single development machine was lots of fun. Back in the day (5 odd years ago) RAM was pricey and so was disk space so the VM's were pretty little.

Now you can run whole server instances in a file and then slip that file between servers depending on load requirements and whatever other fancy things you want to do - such as bringing one physical server down for updates and maintaining all your VM's on another server. Absolutely amazing stuff - especially when you combine it with blade technology and have dozens of servers sharing one big SAN all in the same rackspace. Truly incredible. Pretty complicated too I think as well, remembering where everything is going.

The new Blade servers from HP look very impressive too - they've done a lot of work on cooling and on cabling and everything is redundant with backup from at least one module exactly the same. Of course, when I saw those big fancy Blade servers I thought to myself - why virtualise? You could have a netboot install of a server ready to rock at a moment's notice and be back up and running in a very short time, especially if you combine the server with Microsoft's ASR technology and tape backup. There is so much chance for overkill it's every enterprise geek's wet dream. There were quite a few geeks at the presentation that looked like they were in their happy place that's for sure.

The most important thing I took away from it all was that virtualisation in the server space has come a long way since I last looked into it and the offerings from both VMWare and HP are quite spectacular. Given my quite low funds though - I'm now looking into Linux's XEN virtualisation. If you're more interested in this (any virtualisation for that matter) there is a lot of stuff on the interwebs. Mind you don't clog up your tubes! :-)