Skip to main content

A tale of survival - FreeNAS and ZFS - and four disks with failed sectors

Recently, one of the FreeNAS storage devices we have at the office started to generate failed sectors on two of the disks. While an eyebrow raising event in and of itself, I wasn't particularly concerned. Living in the virtual outback as we do, I ordered some more disks. About 8 days later, the third of the four disks in our NAS started to throw out errors! Uh oh, it appears that we were on a slippery slope towards Doom!

I called the supplier demanding my disks only to find out they'd ordered WD Green drives! Noooo. I amended the order to get WD Red drives (which are designed for a NAS) and was informed it would take a day or two. The next morning the final disk was generating errors. We were getting close to some serious error thresholds on two of the disks and the third and fourth were well on the way...

Impatiently waiting for the new disks to arrive, I kept a close eye on the NAS. FreeNAS emailed me with alarming frequency about disk failure, imminent apocalypse and the like. The next day the WD Red drives arrived and three of the four disks were now generating large numbers of errors. I shut the NAS down (not taking the pool offline like I should have!) and replaced the most error prone disk. Restarting the NAS I added it back to the pool, replacing one of the dead disks and let it rebuild. Gradually I replaced all the disks until the pool was degraded with a corrupted file. 

On this filesystem are all my virtual machines, so I was a bit concerned about which file was corrupt. Thankfully it was an old backup of my current Windows 7 workstation so I deleted it. Oddly, I was unable to remove two of the old disks - every time I tried it would add them back.

After a bit of head scratching I realised I needed to delete the snapshots and once I did that, I was able to remove the disks from the pool and it changed from Degraded to Online and services were all restored. I've checked over the disks and every single one has failed since. One file lost out of almost 3TB of data - thank you ZFS and FreeNAS! Note to self - sort the backups out!

Comments

Popular posts from this blog

Plone - the open source Content Management System - a review

One of my clients, a non-profit, has a lot of files on it's clients. They need a way to digitally store these files, securely and with availability for certain people. They also need these files to expire and be deleted after a given length of time - usually about 7 years. These were the parameters I was given to search for a Document Management System (DMS) or more commonly a Content Management System (CMS). There are quite a lot of them, but most are designed for front facing information delivery - that is, to write something, put it up for review, have it reviewed and then published. We do not want this data published ever - and some CMS's make that a bit tricky to manage. So at the end of the day, I looked into several CMS systems that looked like they could be useful. The first one to be reviewed was OpenKM ( www.openkm.com ). It looked OK, was open source which is preferable and seemed to have solid security and publishing options. Backing up the database and upgradin

Musings on System Administration

I was reading an article discussing forensic preparation for computer systems. Some of the stuff in there I knew the general theory of, but not the specifics of how to perform. As I thought about it, it occurred to me that Systems Administration is such a vast field. There is no way I can know all of this stuff. I made a list of the software and operating systems I currently manage. They include: - Windows Server 2003, Standard and Enterprise - Exchange 2003 - Windows XP - Windows Vista - Windows 2000 - Ubuntu Linux - OpenSuSE Linux - Mac OSX (10.3 and 10.4) - Solaris 8 - SQL 2005 - Various specialised software for the transport industry I have specific knowledge on some of this, broad knowledge on all of it, and always think "There's so much I *don't* know". It gets a bit down heartening sometimes. For one thing - I have no clue about SQL 2005 and I need to make it work with another bit of software. All complicated and nothing straightforward. Irritating doesn&

elementary OS 5.1 Hera - a review and a revisit

 It's been ages since I used a desktop Linux distribution - being up to my ears in the horror of implementing ISO 27001 doesn't leave you much time to play around with computers - too busy writing policies, auditing and generally trying to improve security to a formally acceptable and risk managed level. I need a quick, small OS though to do the occasional network scan, view the contents of a dodgy file on and for general, low impact activities. I remembered reviewing elementary OS ( elementary.io ) some time ago ( see  https://www.ryv.id.au/2015/01/elementary-os-review.html ) from 2015 so I thought it was worth a revisit.  I downloaded the ISO from their website, forgoing to donation for the moment while I review it. If it turns out I'm going to keep using it, I'll send them some love. The ISO is 1.38GB in size and I booted it in a VMware Player instance. From go to whoa (I won't include the install photos here) it took about 10 minutes with a dual vCPU and 4GB of