Showing posts with label network monitoring. Show all posts
Showing posts with label network monitoring. Show all posts

Sunday, 10 September 2017

Using defence in depth to mitigate the risk of ransomware

I've written before about the evils of crypto locker and the spawn of that devilish state of affairs known as ransomware. Recently I came across an infection and saw first hand how defence in depth can save your data and the bitcoin.

Firstly, let's consider the perimeter of the network. What vectors for attack exist externally to the network? There are many and they include:

  • malicious emails
  • dodgy websites with malicious payload 
  • malicious actors (hackers) out to get you
The first layers of defence include (in this case):
  • an antivirus/antispam gateway for email, with the firewall at the main router allowing only connections on port 25 (smtp) from the mail scanner gateway
  • antispyware/antivirus email on the computers scanning every website that a user visits, plus using OpenDNS with a variety of restrictions on it to protect the user from themselves
  • firewalls and obfuscated ports where applicable with minimal "open-to-the-world" ports
That's the hard outer layer. Past the router / firewall and onto the network, we use:
  • firewalls on all PCs (granted only the windows ones, but supplemented with the anti-virus product's offerings)
  • WSUS to keep everything patched and up to date
  • VLANs to separate out stuff
  • usernames / passwords for access to all network resource
To further enhance security:
  • all backups go to a UNC path (i.e \\nas\backups) rather than a mapped drive (like an S:\ drive) which is important because ransomware will attack both local drives and network mapped drives - encrypted backups are 100% useless
  • users have restrictions based on principles of least privilege and this is rigorously enforced
  • servers are also patched and up to date
  • logs are maintained on a separate server
  • PRTG is used to monitor network traffic on the switches and a variety of other stuff
So what happened and how did this all help to mitigate a ransomware attack?

Well a user, let's call him Jim Bob, has a very weak password. Let's say it's Secret01 (yes if this is your password, it is shite. Change it now!)

An external attacker managed to get Jim Bob's username, and then proceeded to attack the remote desktop services to see if they could brute force their way in. What would you know, about 2 hours after starting, they got the password and were in. This could have been mitigated by having a password locking system turned on, but remember - you have to balance usability with security. This particular organisation struggles with passwords at the best of times, so locking a user out after 3 failed attempts for 10 minutes would have very high frustration levels as a result.

The attacker now had access to a server. But - Jim Bob's account was extremely limited. He just couldn't do much - he didn't need to. Basically log on and access the internet. That's him to a tee. This therefore, is all the attacker could do. Frustrating I bet!

Along comes the admin and logs on. The attacker - seeing a systems admin hit the server panics and drops a ransomware payload, probably thinking they'll get some sort of a payday somehow. Their ransomware attack manages to get a foothold on a mapped drive. The security on that mapped drive allows for only a small percentage of accessible files to be encrypted before the wily sysadmin spots it and locks Jim Bob's account down, and shuts off the file server. 

Our defence has now limited the risk by controlling the access to files and what could be attacked. And it gets better. Our snapshot backups are working as advertised and have a 15 minute old copy of the entire mapped drive's file system. With a few clicks, our intrepid sysadmin restores the whole lot over the next hour. Hundreds of files, barely out of date and only in a few instances. Within a few hours, Jim Bob's account has been restricted, password changed and the user Jim Bob given a kick in the bum for having a crappy password and the network drive is fully recovered.

Although our initial defensive line was penetrated (users can be your greatest security risk), the rest of the network's defences held firm mitigating the impact of the attack and the organisation's exposure to loss of data. No pay day for our arsehole attacker today! I like to think of how sad they must be, all that effort and no reward.

In the wash up, the sysadmin goes through and using the logs, PRTG and combing files finds the attacker's trail and mops up after them, making notes on what failed and how to improve it for next time. 

The moral of the story is this - defence always loses. Attackers will win. All we can do is to mitigate the damage and risk to the best of our capabilities and budget. Hopefully you will read this and get a few little ideas about how to perhaps enhance your existing defence, or even think about what attack vectors might exist. This pretend network is by no means perfect - it could always be better. Budget and skill restrictions come into play though and mean we have to find the best effort with whatever we've got at hand to make it work. Be smart and get margin into your security so a break in doesn't break your heart or your budget!

Friday, 7 March 2014

How to fix Nagios3 external commands error

After installing nagios3 and trying to send it a command to reschedule a check or do some other external activity you may get the following error:

Error: Could not stat() command file '/var/lib/nagios3/rw/nagios.cmd'!
The external command file may be missing, Nagios may not be running, and/or Nagios may not be checking external commands.
An error occurred while attempting to commit your command for processing.
In order to fix it, do the following actions:

Check that /etc/nagios3/nagios.cfg has:

check_external_commands=1

Also check that

command_check_interval=15s is uncommented and
command_check_interval=-1 is commented like this:
command_check_interval=15s
#command_check_interval=-1
Check the path for command_file is OK. It usually looks like this:
command_file=/var/lib/nagios3/rw/nagios.cmd
Make sure that the user www-data is part of the nagios group - this is located in /etc/group

Check permissions on the command file that we looked at above:
# ls -l /var/lib/nagios3/rw/nagios.cmd
prw-rw---- 1 nagios nagios 0 Mar  7 11:56 /var/lib/nagios3/rw/nagios.cmd
If it looks like this, we're good.

The next thing to check is that the directory that nagios.cmd resides in has executable rights for the nagios group:
# ls -l /var/lib/nagios3/
total 180
-rw------- 1 nagios www-data 176049 Mar  7 11:58 retention.dat
drwx------ 2 nagios www-data   4096 Mar  7 11:56 rw
drwxr-x--- 3 nagios nagios     4096 Jun 14  2013 spool
Uh oh - rw has no group rights! Fix it with this command:

# chmod g+x /var/lib/nagios3/rw
and then
# service nagios3 restart
And the crowd goes wild!

Friday, 8 February 2013

Traffic Monitoring using Ubuntu Linux, ntop, iftop and bridging

This is an update of an older post, as the utilities change, so has this concept of a cheap network spike - I use it to troubleshoot network issues, usually between a router and the network to understand what traffic is going where. The concept involves a transparent bridge between two network interface cards, and then looking at that traffic with a variety of tools to determine network traffic specifics. Most recently I used one to determine if a 4MB SDSL connection was saturated or not. It turned out the router was incorrectly configured and the connection had a maximum usage under 100Kb/s (!) At $1600 / month it's probably important to get this right - especially when the client was considering upgrading to a faster (and more expensive) link based on their DSL provider's advice.

Hardware requirements:


I'm using an old Dell Vostro desktop PC with a dual gigabit NIC in it - low profile and fits into the box nicely. Added a bit of extra RAM and a decent disk and that's really it. I'm also running this on an old Dell D420 with a gigabit PCMCIA adaptor - useful for the out and about jobs.

Software requirements:


  • Ubuntu 12.04 LTS - I've chosen this for longevity purposes, previously I'd used non-LTS operating systems and the updates naturally ran out. I tried this with FreeBSD 9.1 but had issues with packages and getting traffic across the network bridge effectively (probably more my screw up than FreeBSD's)
  • ntop - network traffic analysis monitor from www.ntop.org. They have version 5 available from the repositories on the site, version 4 is included in Ubuntu 12.04
  • iftop - a neat command line package that shows network usage from a terminal screen. Highly configurable
  • tcpdump or equivalent for deeper packet analysis.

Configuration

Setting up the box and the ethernet bridge
Setting up the box is straight forward - go through the usual Ubuntu installation. Use aptitude or apt-get to install bridge-utils and iftop

We want the network bridge between our ethernet adaptors to come up automatically. To do so edit /etc/rc.local and pop this into it (assuming eth1 and eth2 are the interfaces you want to bridge. I have eth0 configured statically in this instance so I can browse from other machines to it)

/etc/rc.local
brctl addbr br0
ifconfig eth1 0.0.0.0 promisc up
ifconfig eth2 0.0.0.0 promisc up
brctl addif br0 eth1
brctl addif br0 eth2
ip link set br0 up
This will bring the bridge up at boot time.

ntop

After you've added the necessary repositories to your aptitude configuration, install ntop5 using apt-get install ntop5

I run this from the command line - as a service it seems to fail fairly consistently. The command is:

ntop -P /var/lib/ntop -Q /usr/local/share/ntop/spool/ -i br0 -u ntop -m 192.168.0.0/24 -d

-P sets the database file path
-Q sets the spool file path
-i sets the interface (br0 as per /etc/rc.local)
-m sets the local subnet - in this case 192.168.0.0/24 (change to suit)
-d sets it to become a daemon freeing up your terminal
Browse to localhost:3000 to find your ntop installation, or if you have a third network card go to the address on the network e.g. 192.168.0.30:3000 and view your traffic stats.

iftop

To get what I want out of iftop, I run a script that calls it and configure the /etc/iftoprc file. The script is:
bridge_monitor.sh
#!/bin/sh
# customisable settings
LOCALNET="192.168.0.0/24"
IFACE="br0" # the bridged interface
CONF="/etc/iftoprc"
/usr/sbin/iftop -p -n -N -i $IFACE -F $LOCALNET -c $CONF

The contects of /etc/iftoprc are:
dns-resolution: yes
port-resolution: yes
show-bars: yes
promiscuous: no
port-display: source-only
#hide-source: yes
#hide-destination: yes
use-bytes: yes
sort: 2s
#line-display: one-line-both
show-totals: yes

Again customise to suit and start monitoring that network!

Saturday, 30 June 2012

Munin for Monitoring

Lately I've been very busy with work, but I find that I still need to keep an eye on the Linux servers I have kicking around at home and at the office. I use Nagios for basic monitoring (ping, smtp, http) but wanted something a bit more fine grained on the actual host itself. I stumbled on munin when doing some Google searching.

Munin's homepage is here: http://munin-monitoring.org and I'm running it on a variety of servers at them moment (mostly Ubuntu and Debian). It supports quite a few distributions and operating systems so check it out if you're interested in seeing what's happening on your servers.

I like the many graphs that it produces to demonstrate what's happening and I also fell into the trap of trying to over-configure things. There are only a couple of basic things to change to suit yourself, restart apache2 and munin-node and it's slamming. It also monitors multiple machines - install munin-node instead of munin and munin-node and configure as recommended. I thought I'd be tricky with it and things didn't go as planned so it appeared that it wasn't working. I was over thinking it though and it did work, very well actually, once I found the spot it was creating the files in :-)

A handy app and a nice way to keep on top of your servers!

Wednesday, 20 October 2010

Tips and Tricks: Monitoring with NTOP and IFTOP

For monitoring networks I have an old GX260 Dell desktop in the small form factor. Packed into this little device are a couple of network cards, added on are a wireless NIC and an extra USB network device. The purpose? To slot this in between the router and the network and see what's going backwards and forwards - very useful in the situation where a client is hemorrhaging bandwidth and doesn't know why. I'm running Ubuntu 10.04 LTS on it and it behaves very well.

Two of the main tools I use are ntop and iftop. For those of you not familiar with them, ntop monitors a particular interface and creates some nice webpages to be checked by the user in order to see what's going on through the network. iftop is similar but real time and is available through a console - which is the real appeal for me.

The server in question has two internal NICs, both are 100MB cards and are scripted to come up as a transparent bridge - br0. Basically I monitor that bridge and use one of the other interfaces to see what the server is seeing. Here is the script I use (adapted from elsewhere) to bring the bridge up:

#!/bin/bash
PATH="/sbin:/usr/sbin:/usr/local/sbin";
slaveIfs="1 2 3 4 6 7 8 9 10";
cmd="$1";
[ -z "$cmd" ] && cmd="start";
case "$cmd" in
  start)
    brctl addbr br0;
    brctl stp br0 on;
    brctl addif br0 eth1;
    brctl addif br0 eth2;
    (ifdown eth1 1>/dev/null 2>&1;);
    (ifdown eth2 1>/dev/null 2>&1;);
    ifconfig eth1 0.0.0.0 up;
    ifconfig eth2 0.0.0.0 up;
    ifconfig br0 up ### Adapt to your needs.
    ;;
  stop)
    brctl delif br0 eth1;
    brctl delif br0 eth2;
    ifconfig br0 down;
    brctl delbr br0;
    #ifup eth0; ### Adapt to your needs.
    #ifup eth1; ### Adapt to your needs.
    ;;
  restart,reload)
    $0 stop;
    sleep 3;
    $0 start;
    ;;
esac;

It resides in /etc/init.d and is called (imaginatively) bridge.sh and then has appropriate symlinks to rc2.d.

NTOP
NTOP is reasonably easy to configure on Ubuntu and is quite straightforward to get going - point it at http://localhost:3000 and set your username/password and off you go.

A tip to remember is that in the /etc/default/ntop it's a good idea to uncomment GETOPT="" and change it to read (if your network was 192.168.0.0/24):

GETOPT="--local-subnet=192.168.0.0/24"

and restart NTOP. Why is this important I hear you ask? Well on the br0 interface there is no IP assigned to it so NTOP doesn't automatically figure out what the local network is. By assigning this you can set it up to get proper info on local-remote, remote-remote and local-local traffic. Then leave it to run and see what it tells you.

IFTOP
Ah iftop is such a nice little bit of software. apt-get install it and then run it from the console with iftop -i br0 and it will tell you all sorts of things - data going from here to there and the level of bandwidth being used. I fancy pants it up a bit by creating a shell script and then launching it from there. iftop will use .iftoprc by default and if there isn't one will simply launch with it's own defaults. Here is the shell script I use (again I have adapted this from someone else):

#!/bin/sh

# customisable settings
LOCALNET="192.168.0.0/24"
IFACE="br0" # the bridged interface
CONF="/etc/iftoprc"

/usr/sbin/iftop -p -n -N -i $IFACE -F $LOCALNET -c $CONF

where /etc/iftoprc looks like this:

dns-resolution: yes
port-resolution: yes
show-bars: yes
promiscuous: no
port-display: source-only
#hide-source: yes
#hide-destination: yes
use-bytes: yes
sort: 2s
#line-display: one-line-both
show-totals: yes

Most of these are self-explanatory and I believe you should examine them more closely if you are looking to deploy it. Suffice to say, it gives me the info I want and I'm happy with that setup. So I hope that gives you some food for thought and you can take some of this away with you when you're trying to find out what the hell is chewing up all your bandwidth and download limit!

Omarchy - the new darling of the Linux world

 Omarchy is the buzzword at the moment, with it turning up on Network Chuck's YouTube channel. I came across it via DHH's blog post ...