DoS - Denial of Service
Over the weekend it was very hot here - 39C over both days and air conditioning was being pushed pretty hard. My team and I had two unrelated, but linked situations evolve that could have hit us with a DoS. You see, we have a network attached storage device (NAS) that had a fan failure. While this NAS has redundant fans in it, one wasn't enough to keep the temperatures under the 55C warning threshold. So it started to complain....
Over the course of the 48 hour weekend, this NAS sent out over three and a half thousand emails! 3500+ emails! All to our logging email addresses, which then sent it out to the members of the team. 5 team members, 3500+ emails.... 17,500 emails being sent and received. That's a lot email in a short time. Most email servers will handle that and ours certainly did. Fortunately too we use G-Suite (Google Apps new fancy name) and so the volume of mail wasn't an issue.
What became an issue though - and this did have an effect on our phones and mobile devices picking up email - was that another network device - a disaster recovery server - also suffered heat stress from failed air conditioning. A sparky had unplugged our monitoring device to charge his tools and hadn't plugged it back in, so we had no idea what was happening (this was Sunday afternoon). When the A/C failed, the server turned off and the replication servers started to complain - four of them, every 30 seconds.... Over 12 hours those servers alerted our logging email address over 5,000 coming in and going back out - another 25,000 emails hitting phones plus the other emails as well.
Having had the discussion with clients about hosted email solutions versus onsite solutions, there are definite advantages to having huge servers managing your email. So if you don't have a cloud based solution, how can you mitigate this risk?
Defence in depth is a great place to start. Organise to get a mail exchanger - MXGuardDog or something similar. Westnet used to do one too. Get your MX records updated to punch mail through that. These then relay to av-relay.domainname.com. Configure your firewall to only accept emails from the IPs at MXGuardDog (for example) and drop everything else (or at least grey list it so it gets dropped and the sending server can try other MX records).
This way you can temporarily control the flow without having your ADSL or NBN connection getting flogged to death.
Configure your internal mailer to hold emails for this kind of thing - to recognise a flood of email and trickle it out where possible. The risk is that legitimate email (which these emails both are and aren't) will get lost in the flow. It's better than having your upload link fully saturated though (which will kill all internet connectivity).
DoS are bad. DDoS are worse. Let's try to avoid doing it to ourselves!
Over the weekend it was very hot here - 39C over both days and air conditioning was being pushed pretty hard. My team and I had two unrelated, but linked situations evolve that could have hit us with a DoS. You see, we have a network attached storage device (NAS) that had a fan failure. While this NAS has redundant fans in it, one wasn't enough to keep the temperatures under the 55C warning threshold. So it started to complain....
Over the course of the 48 hour weekend, this NAS sent out over three and a half thousand emails! 3500+ emails! All to our logging email addresses, which then sent it out to the members of the team. 5 team members, 3500+ emails.... 17,500 emails being sent and received. That's a lot email in a short time. Most email servers will handle that and ours certainly did. Fortunately too we use G-Suite (Google Apps new fancy name) and so the volume of mail wasn't an issue.
What became an issue though - and this did have an effect on our phones and mobile devices picking up email - was that another network device - a disaster recovery server - also suffered heat stress from failed air conditioning. A sparky had unplugged our monitoring device to charge his tools and hadn't plugged it back in, so we had no idea what was happening (this was Sunday afternoon). When the A/C failed, the server turned off and the replication servers started to complain - four of them, every 30 seconds.... Over 12 hours those servers alerted our logging email address over 5,000 coming in and going back out - another 25,000 emails hitting phones plus the other emails as well.
Having had the discussion with clients about hosted email solutions versus onsite solutions, there are definite advantages to having huge servers managing your email. So if you don't have a cloud based solution, how can you mitigate this risk?
Defence in depth is a great place to start. Organise to get a mail exchanger - MXGuardDog or something similar. Westnet used to do one too. Get your MX records updated to punch mail through that. These then relay to av-relay.domainname.com. Configure your firewall to only accept emails from the IPs at MXGuardDog (for example) and drop everything else (or at least grey list it so it gets dropped and the sending server can try other MX records).
This way you can temporarily control the flow without having your ADSL or NBN connection getting flogged to death.
Configure your internal mailer to hold emails for this kind of thing - to recognise a flood of email and trickle it out where possible. The risk is that legitimate email (which these emails both are and aren't) will get lost in the flow. It's better than having your upload link fully saturated though (which will kill all internet connectivity).
DoS are bad. DDoS are worse. Let's try to avoid doing it to ourselves!
Comments
Post a Comment