Skip to main content

The unintentional DoS

DoS - Denial of Service

Over the weekend it was very hot here - 39C over both days and air conditioning was being pushed pretty hard. My team and I had two unrelated, but linked situations evolve that could have hit us with a DoS. You see, we have a network attached storage device (NAS) that had a fan failure. While this NAS has redundant fans in it, one wasn't enough to keep the temperatures under the 55C warning threshold. So it started to complain....

Over the course of the 48 hour weekend, this NAS sent out over three and a half thousand emails! 3500+ emails! All to our logging email addresses, which then sent it out to the members of the team. 5 team members, 3500+ emails.... 17,500 emails being sent and received. That's a lot email in a short time. Most email servers will handle that and ours certainly did. Fortunately too we use G-Suite (Google Apps new fancy name) and so the volume of mail wasn't an issue.

What became an issue though - and this did have an effect on our phones and mobile devices picking up email - was that another network device - a disaster recovery server - also suffered heat stress from failed air conditioning. A sparky had unplugged our monitoring device to charge his tools and hadn't plugged it back in, so we had no idea what was happening (this was Sunday afternoon). When the A/C failed, the server turned off and the replication servers started to complain - four of them, every 30 seconds.... Over 12 hours those servers alerted our logging email address over 5,000 coming in and going back out - another 25,000 emails hitting phones plus the other emails as well.

Having had the discussion with clients about hosted email solutions versus onsite solutions, there are definite advantages to having huge servers managing your email. So if you don't have a cloud based solution, how can you mitigate this risk?

Defence in depth is a great place to start. Organise to get a mail exchanger - MXGuardDog or something similar. Westnet used to do one too. Get your MX records updated to punch mail through that. These then relay to av-relay.domainname.com. Configure your firewall to only accept emails from the IPs at MXGuardDog (for example) and drop everything else (or at least grey list it so it gets dropped and the sending server can try other MX records).

This way you can temporarily control the flow without having your ADSL or NBN connection getting flogged to death.

Configure your internal mailer to hold emails for this kind of thing - to recognise a flood of email and trickle it out where possible. The risk is that legitimate email (which these emails both are and aren't) will get lost in the flow. It's better than having your upload link fully saturated though (which will kill all internet connectivity).

DoS are bad. DDoS are worse. Let's try to avoid doing it to ourselves!

Comments

Post a Comment

Popular posts from this blog

Plone - the open source Content Management System - a review

One of my clients, a non-profit, has a lot of files on it's clients. They need a way to digitally store these files, securely and with availability for certain people. They also need these files to expire and be deleted after a given length of time - usually about 7 years. These were the parameters I was given to search for a Document Management System (DMS) or more commonly a Content Management System (CMS). There are quite a lot of them, but most are designed for front facing information delivery - that is, to write something, put it up for review, have it reviewed and then published. We do not want this data published ever - and some CMS's make that a bit tricky to manage. So at the end of the day, I looked into several CMS systems that looked like they could be useful. The first one to be reviewed was OpenKM ( www.openkm.com ). It looked OK, was open source which is preferable and seemed to have solid security and publishing options. Backing up the database and upgradin

elementary OS 5.1 Hera - a review and a revisit

 It's been ages since I used a desktop Linux distribution - being up to my ears in the horror of implementing ISO 27001 doesn't leave you much time to play around with computers - too busy writing policies, auditing and generally trying to improve security to a formally acceptable and risk managed level. I need a quick, small OS though to do the occasional network scan, view the contents of a dodgy file on and for general, low impact activities. I remembered reviewing elementary OS ( elementary.io ) some time ago ( see  https://www.ryv.id.au/2015/01/elementary-os-review.html ) from 2015 so I thought it was worth a revisit.  I downloaded the ISO from their website, forgoing to donation for the moment while I review it. If it turns out I'm going to keep using it, I'll send them some love. The ISO is 1.38GB in size and I booted it in a VMware Player instance. From go to whoa (I won't include the install photos here) it took about 10 minutes with a dual vCPU and 4GB of

Windows 10 Enterprise Eval - gotchas

After an annoying turn of events where my Windows 10 Enterprise USB drive failed, attempts to install Win10 onto a computer failed miserably. I turned to the net and managed to get my hands on Microsoft's Windows 10 Enterprise Evaluation. I have an enterprise key so I thought - cool! Here's the opportunity to get it going and to then upgrade the license later. Full install, patched etc and all is swell. Except when I try to upgrade. I straight up tried changing the licence key only to get a variety of errors, most of which are pertaining to the activation system being unavailable. The I try this: https://winaero.com/blog/upgrade-windows-10-evaluation-to-full-version-easily/ but it doesn't work either. Next I'll try this: h ttp://www.edugeek.net/forums/windows-10/174594-upgrading-windows-10-enterprise-90-evaluation-full.html And if all else fails, in goes the bootable USB I've now created. If only I'd had this in the first instance I would not be writing t