How to update a single XenServer 6.2 host before adding it to a pool

You may find yourself in the position where you need to add a machine into a pool, a new host that is, and it's patching is well behind. Thankfully there is a relatively straightforward way to do this.

• You will need to have already downloaded and extracted the required files.
• On a computer with XenCentre installed run the following commands (assuming default install and Windows 7 / 8)
• Open a command prompt
• Win+r, then type cmd (or go to Start -> Run -> cmd and return)
• cd "C:\Program Files (x86)\Citrix\XenCenter"
• From here run: xe patch-upload -s server-ip -u root -pw secret-password file-name=c:\username\Downloads\XS62ESP1\XS62ESP1.xupdate and hit return. Make sure the items in italics is replaced with your actual details.
• Once this finished, you'll get a long string of characters appear and that's the patch UUID. Normally when updating a pool we make note of that, but I've found another way that works just as well.
• Download puTTY - it's an awesome terminal program and you should have it.
• Use puTTY to connect via SSH to your xenserver and get to the console. Alternatively, do this from the server console. At any rate, type the following commands and note the two UUID's they produce:
• # xe patch-list
• note the UUID of the patch
• # xe host-list
• note the UUID of the host
• # xe patch-apply UUID=patchUUID host-uuid=hostUUID and hit return. Once all the information stops flowing past and you're back to a # prompt type:
• # reboot
• if you're using puTTY, you can select the UUID's, the simply right click where you want them to be and puTTY will paste them out of the buffer. Very quick to do, rather than laboriously typing it out.

Once the machine reboots, continue to patch as applicable, and then add it to the pool.

Happy XenServering!

XenServer 6.2 - installed on a USB drive and installed on an SD Card

In this post I want to talk about recent experiences I've had with XenServer 6.2 on USB drives and SD cards. In a particular situation, I was forced initially down this path due to specific hardware and lack of appropriate drives. The upshot is that it was entirely unsuccessful and here is why:

Firstly, it's important to be aware that XenServer isn't all that happy being installed on these devices. After you install, you must boot back up on the XenServer CD and then (using ALT-F2) drop to a command line. To get XenServer to boot on USB do the following:

• From the command line do this: 

• # cat /proc/partitions
• this tells you what partitions there are that XenServer can see. Typically you will see /dev/sda and it's children - /dev/sda1, /dev/sda2 etc. We want /dev/sda1
• # mkdir /target
• create a temporary location so we can change our root directory
• # mount -t ext3 /dev/sda1 /target
• mount our existing /dev/sda1 partition to /target, giving us access to the files
• # mount proc /target/proc -t proc
• we need a proc filesystem!
• # mount sysfs /target/sys -t sysfs
• and a sysfs
• # mount --bind /dev /target/dev
• and of course somewhere for devices to be found
• # chroot /target
• change to the /target install of the system (chrooting means we are running now under the actual XenServer install, not the boot cd live filesystem)
• # cd /boot
• # ls –al

Now we need to actually tell the kernel of our installed XenServer to add USB support to it so it'll boot. Up until this point, the system will just hang, unable to find a filesystem.

• # mv initrd-lots.of-numbersxen.img initrd-lots.of-numbersxen.img.old
• this backs up the initrd ing
• # mkinitrd --with-usb initrd-lots.of-numbersxen.img lots.of-numbersxen.img
• make sure you replace lots.of-numbers with the actual kernel version. It could look like this: 2.6.18-53.1.13.el5.xs4.1.0.254.273
• # exit
• # sync
• # reboot

At this stage we are finished and the server will reboot and actually get into XenServer. After patching the pool, or indeed, the server by itself, you may need to apply the initrd update again, but with the new kernel number.

While this was cumbersome to do, particularly because I had to do it six times - 3 for initial install, 3 again after SP1 was applied, it wasn't hard to do, or particularly time consuming. I found that I had to do this for both USB drives and on the SD Card. The SD Card was significantly slower than USB. The key difference between the two was that the USB, after some magical, indeterminate time, would lose it's filesystem! XenCenter would lose connectivity to it, and the console interface was non-functional - unable to find "root"! A reboot fixed this, but these are supposed to be on all the time with minimal service loss. That was when I turned to the SD Card idea. These servers had been running an awful install of VMware 4.1 on SD Cards so I thought they would just work. Sadly, it was not to be. Once again, I'd lose connectivity to the server, but this time, I was able to get the xsconsole to respond and reboot the machine. Totally unacceptable though. I don't have time to test with this system - it's supposed to be production! At any rate, I've purchased disks and installing them. If anyone has experienced this and knows the answer - I'd love to hear it!

ShadowCraft ShadowProtect - Employee of the Month

The last two weeks have been absolutely awful. The IOmega NAS I mentioned in a previous post had a large number of things go wrong with it all at the same time, resulting in complete failure and total data loss. This also meant a significant amount of down time for a client and ridiculous amounts of stress for me.

The good thing to come out of it was the performance of ShadowProtect. This backup solution far exceeded my expectations to the point where once the recoveries were completed we had only lost 10 minutes of work. Very little data was realistically lost, only time as we worked to recover it. The Recovery Environment was easy to use and the Hardware Independent Restore allowed me to recover virtual machine from VMware 4 ESX to VMware Player, a physical machine and also to XenServer 6.2 quite a remarkable feat really. I've had this client running in borrowed hardware for a week while we get new gear in.

The snapshot capability, ease of restore and simply replication of backup files makes ShadowProtect a real winner. Check it out! 

Experiences with the IOmega ix12 NAS

One of my clients has an IOmega ix12 - a rack mounted, 12x3.5" bay NAS. This one in question is running 2.6.5.something firmware and has (or should I say had) 8 x 1TB disks in it. Today it dropped another one. The NAS has been set up with a RAID10 with total storage of 2TB (1.5 or so in reality). Although it has this redundancy, I can assure you it's not all that it's cracked up to be. In the main storage pool is a number of iSCSI drives shared out with various ESX servers. Sadly, the largest and most important of these iSCSI drives has an unfortunate tendency to vanish whenever there is a disk issue.

Today one of the drives failed. No big deal right? RAID10 is tolerant of such faults usually. This one kept going except for the aforementioned large drive that disappeared like magic. After replacing the disk, impatiently waiting for it to rebuild and then attempting to reconnect the iSCSI drives I realised I still couldn't see it. The fix? Turn the NAS completely off, count to 30 and turn it back on again. Once it booted, just like magic everything reappeared. Be wary of this. My previous post was the precursor to this. Finding out the actual issue is still to be completed.

I'm concerned about upgrading the firmware because the backups, while reliable, are never straightforward to recover and I'd hate to have to recover so many disks. Its a job for next week I reckon.

VMware nightmares

After a great couple of days away I was called urgently to work - one of my client's networks was down and my colleague was stuck. The VMsphere client couldn't see the hosts, which couldn't see their datastores and the network had zero stability. Yay. Just wanted I wanted to come home to.

The servers are running ESXi 4.1 that needs a few updates but the networks stability has always been a real issue for us. We took it over from some other chaps, solving a long series of issues with the servers and particularly the IOmega NAS. Throw in a database issue on one of the guest servers that kept the disk IO at absolute peak all the time and things were pretty tricky. All those things have been resolved, more or less, and the network had been fully functional for some months. So why did it change? Several reasons and I hope you take away from this some ideas for yourself.

The NAS appeared to have dropped a disk and while it has a RAID10 set up, it paralysed the system (not at all like the FreeNAS I've talked about before). The whole thing fell over and we shut it down before reseating the disk and powering back up. The NAS detected the disk and began a rebuild. The VMware hosts couldn't see the datastore though and the server managing the virtual environment periodically disconnected from everything. Initially we thought it was a network issue and restarted everything. The hosts were slow to come up, with errors in the logs indicating an inability to see the iSCSI disks. We could connect to the NAS via it's web interface, ping it and it looked quite happy on the network so we couldn't understand what was happening. An added complication was we have a new HP NAS on the network and while we were able to migrate several of the hosts to it, we've had problems getting then started. Don't know why, but the host's cpu goes through the roof everytime we try to start them. I thought we might have an all paths down bug and most of the documentation suggests a call to VMware and let them sort it out. At 8pm at night this isn't so great a plan, and with the client losing production time and money we had to solve it.

So with all these errors and problems left and right I was at a loss. Eventually we took inspiration from The IT Crowd and turned it all off, counted to 10 and started it back up. Would you believe that it took 3 reboots of the management server before the VMsphere client would connect to anything - especially considering it was running locally! The iSCSI shares from the NAS became available finally - there was a service issue on the iOmega NAS that was failing silently. It was alleging that all was good and it really wasn't. Now those shares were available we were able to reconnect to the datastores and boot the guest machines. The management server was still disconnecting from the hosts constantly but we were able to at least get things going. There is still quite a bit to do there but the servers are finally running.

Going forward I think we'll use XenServer and NFS shares. Simpler, fully functional and quick. Easy to backup and expand disks. Adios VMware I reckon.

The final thought is "Have you turned it off and back on again?" Something in that for all of us :-)

The pure awesomeness of Google Forms

As a Google Apps reseller it never fails to surprise me how many people love Forms. I swear, I could nearly sell Google Apps to people just based on Forms. If you've never used it, here is a quick run down of what it can do for you:

• create a web form very quickly and easily, choosing from a range of different responses including:
• time (duration and date)
• multiple choice
• checkboxes
• create separate pages based on responses
• autofill a Google Spreadsheet
• automatically create a summary of responses with a range of different graphs and information instantly
• embed it into websites
• email notifications of changes and updates
• make you more popular with friends (OK maybe not this one!)

Some examples of how I use Google Forms:

• PC Audit
• capture make / model, RAM, CPU etc
• Job Audit
• bookings for jobs
• completion of job information
• Incident responses

Seriously they are awesome and if you're trying to do any type of statistics about stuff, they are a terrific place to start. Make sure though that your questions are good and that you choose the right type of responses to get the right Summary of Responses. It's cool though - you can change it even with live data in the spreadsheet. I have taught a friend how to work Google Forms in under 15 minutes - it is that straightforward. Enjoy!

HOW TO: POP3 on Windows 8

I have a few clients still clinging to POP3 mail accounts and the newest version of the Windows Mail client doesn't work with it. Instead you will need to install the desktop version of Windows Live Mail. This can be found here: http://download.live.com

When you grab it, only install the mail component and when that is done, go into Start, find the Windows Live Mail shortcut and set up your beloved POP3 account. Enjoy