Skip to main content

The unintentional DoS

DoS - Denial of Service

Over the weekend it was very hot here - 39C over both days and air conditioning was being pushed pretty hard. My team and I had two unrelated, but linked situations evolve that could have hit us with a DoS. You see, we have a network attached storage device (NAS) that had a fan failure. While this NAS has redundant fans in it, one wasn't enough to keep the temperatures under the 55C warning threshold. So it started to complain....

Over the course of the 48 hour weekend, this NAS sent out over three and a half thousand emails! 3500+ emails! All to our logging email addresses, which then sent it out to the members of the team. 5 team members, 3500+ emails.... 17,500 emails being sent and received. That's a lot email in a short time. Most email servers will handle that and ours certainly did. Fortunately too we use G-Suite (Google Apps new fancy name) and so the volume of mail wasn't an issue.

What became an issue though - and this did have an effect on our phones and mobile devices picking up email - was that another network device - a disaster recovery server - also suffered heat stress from failed air conditioning. A sparky had unplugged our monitoring device to charge his tools and hadn't plugged it back in, so we had no idea what was happening (this was Sunday afternoon). When the A/C failed, the server turned off and the replication servers started to complain - four of them, every 30 seconds.... Over 12 hours those servers alerted our logging email address over 5,000 coming in and going back out - another 25,000 emails hitting phones plus the other emails as well.

Having had the discussion with clients about hosted email solutions versus onsite solutions, there are definite advantages to having huge servers managing your email. So if you don't have a cloud based solution, how can you mitigate this risk?

Defence in depth is a great place to start. Organise to get a mail exchanger - MXGuardDog or something similar. Westnet used to do one too. Get your MX records updated to punch mail through that. These then relay to av-relay.domainname.com. Configure your firewall to only accept emails from the IPs at MXGuardDog (for example) and drop everything else (or at least grey list it so it gets dropped and the sending server can try other MX records).

This way you can temporarily control the flow without having your ADSL or NBN connection getting flogged to death.

Configure your internal mailer to hold emails for this kind of thing - to recognise a flood of email and trickle it out where possible. The risk is that legitimate email (which these emails both are and aren't) will get lost in the flow. It's better than having your upload link fully saturated though (which will kill all internet connectivity).

DoS are bad. DDoS are worse. Let's try to avoid doing it to ourselves!

Comments

Post a comment

Popular posts from this blog

Windows 10 Enterprise Eval - gotchas

After an annoying turn of events where my Windows 10 Enterprise USB drive failed, attempts to install Win10 onto a computer failed miserably. I turned to the net and managed to get my hands on Microsoft's Windows 10 Enterprise Evaluation. I have an enterprise key so I thought - cool! Here's the opportunity to get it going and to then upgrade the license later. Full install, patched etc and all is swell. Except when I try to upgrade. I straight up tried changing the licence key only to get a variety of errors, most of which are pertaining to the activation system being unavailable. The I try this: https://winaero.com/blog/upgrade-windows-10-evaluation-to-full-version-easily/ but it doesn't work either. Next I'll try this: h ttp://www.edugeek.net/forums/windows-10/174594-upgrading-windows-10-enterprise-90-evaluation-full.html And if all else fails, in goes the bootable USB I've now created. If only I'd had this in the first instance I would not be writing t

Fixing a black screen after doing a Kali Linux update

Kali Linux is a rolling Linux distribution designed for security and penetration work. You can find details on it here: www.kali.org . We run this excellent product for a range of different security work and it's been great. I built the image in VMplayer, then shared it to the team and we've all been at it since. A recent update broke it though - black screen, no network and completely unresponsive. There are lots of posts about similar things - mostly to do with graphics adaptors, however, we found that executing the following at a root prompt fixed it. But how to get to the root prompt from a blank screen? Linux has a number of terminals available to the user - most of us use the graphical one to do our day to day, but you can access a command line prompt without much trouble. Simply hold CTRL-ALT and then F2 or F3 down at the same time and it drops you to a command line login. BOOM. Time to fix it up. For me, and for the other fellas in the team, all it too was to

Plone - the open source Content Management System - a review

One of my clients, a non-profit, has a lot of files on it's clients. They need a way to digitally store these files, securely and with availability for certain people. They also need these files to expire and be deleted after a given length of time - usually about 7 years. These were the parameters I was given to search for a Document Management System (DMS) or more commonly a Content Management System (CMS). There are quite a lot of them, but most are designed for front facing information delivery - that is, to write something, put it up for review, have it reviewed and then published. We do not want this data published ever - and some CMS's make that a bit tricky to manage. So at the end of the day, I looked into several CMS systems that looked like they could be useful. The first one to be reviewed was OpenKM ( www.openkm.com ). It looked OK, was open source which is preferable and seemed to have solid security and publishing options. Backing up the database and upgradin