Traffic Monitoring using Ubuntu Linux, ntop, iftop and bridging

This is an update of an older post, as the utilities change, so has this concept of a cheap network spike - I use it to troubleshoot network issues, usually between a router and the network to understand what traffic is going where. The concept involves a transparent bridge between two network interface cards, and then looking at that traffic with a variety of tools to determine network traffic specifics. Most recently I used one to determine if a 4MB SDSL connection was saturated or not. It turned out the router was incorrectly configured and the connection had a maximum usage under 100Kb/s (!) At $1600 / month it's probably important to get this right - especially when the client was considering upgrading to a faster (and more expensive) link based on their DSL provider's advice.

Hardware requirements:

I'm using an old Dell Vostro desktop PC with a dual gigabit NIC in it - low profile and fits into the box nicely. Added a bit of extra RAM and a decent disk and that's really it. I'm also running this on an old Dell D420 with a gigabit PCMCIA adaptor - useful for the out and about jobs.

Software requirements:

• Ubuntu 12.04 LTS - I've chosen this for longevity purposes, previously I'd used non-LTS operating systems and the updates naturally ran out. I tried this with FreeBSD 9.1 but had issues with packages and getting traffic across the network bridge effectively (probably more my screw up than FreeBSD's)
• ntop - network traffic analysis monitor from www.ntop.org. They have version 5 available from the repositories on the site, version 4 is included in Ubuntu 12.04
• iftop - a neat command line package that shows network usage from a terminal screen. Highly configurable
• tcpdump or equivalent for deeper packet analysis.

Configuration

Setting up the box and the ethernet bridge

Setting up the box is straight forward - go through the usual Ubuntu installation. Use aptitude or apt-get to install bridge-utils and iftop

We want the network bridge between our ethernet adaptors to come up automatically. To do so edit /etc/rc.local and pop this into it (assuming eth1 and eth2 are the interfaces you want to bridge. I have eth0 configured statically in this instance so I can browse from other machines to it)

/etc/rc.local

brctl addbr br0
ifconfig eth1 0.0.0.0 promisc up
ifconfig eth2 0.0.0.0 promisc up
brctl addif br0 eth1
brctl addif br0 eth2
ip link set br0 up

This will bring the bridge up at boot time.

ntop

After you've added the necessary repositories to your aptitude configuration, install ntop5 using apt-get install ntop5

I run this from the command line - as a service it seems to fail fairly consistently. The command is:

ntop -P /var/lib/ntop -Q /usr/local/share/ntop/spool/ -i br0 -u ntop -m 192.168.0.0/24 -d

-P sets the database file path

-Q sets the spool file path

-i sets the interface (br0 as per /etc/rc.local)

-m sets the local subnet - in this case 192.168.0.0/24 (change to suit)

-d sets it to become a daemon freeing up your terminal

Browse to localhost:3000 to find your ntop installation, or if you have a third network card go to the address on the network e.g. 192.168.0.30:3000 and view your traffic stats.

iftop

To get what I want out of iftop, I run a script that calls it and configure the /etc/iftoprc file. The script is:

bridge_monitor.sh

#!/bin/sh
# customisable settings
LOCALNET="192.168.0.0/24"
IFACE="br0" # the bridged interface
CONF="/etc/iftoprc"
/usr/sbin/iftop -p -n -N -i $IFACE -F $LOCALNET -c $CONF

The contects of /etc/iftoprc are:

dns-resolution: yes
port-resolution: yes
show-bars: yes
promiscuous: no
port-display: source-only
#hide-source: yes
#hide-destination: yes
use-bytes: yes
sort: 2s
#line-display: one-line-both
show-totals: yes

Again customise to suit and start monitoring that network!

OTRS Restore Procedure and backup script

As I note in my previous post, I managed to kill my OTRS install and as usual had to trawl around the net to remember how to restore it. In a nutshell:

# mysql -u root -p
msyql> drop database otrs;
mysql> create database otrs;
mysql> ext
# /opt/otrs/scripts/restore.pl -d path_to_backup /opt/otrs

You did back up right?

Nightly I run a script with the following in it:

otrs_backup.sh


#!/bin/bash
# Variables below - change these to suit
NOW=$(date +"%Y-%m-%d_%H-%M") # this gets the correct file name for OTRS backup
LOCAL=/root/backup # a local directory for OTRS to backup to
REMOTE="user@backupserver:~/backup/OTRS/" # remote backup dir - nfs share, ftp or cifs
/opt/otrs/scripts/backup.pl -d $LOCAL # OTRS internal backup (files and DB)
tar -cf $LOCAL/$NOW.tar $LOCAL/$NOW # creates a file from the OTRS backup folder - more efficient to copy over a network
gzip $LOCAL/$NOW.tar
rm -rf $LOCAL/$NOW # tidy up
scp -r $LOCAL/$NOW.tar.gz $REMOTE # scp to remote directory

You may wish to run this from crontab after copying otrs_backup.sh to /usr/local/bin:

0 20 * * * /usr/local/bin/otrs_backup.sh

This will run at 10pm each night - theoretically you could run it more frequently. OTRS databases will a lot of attachments get quite large though so be mindful of that (I have a couple I manage that are 1GB and are only 5 months old)

Enjoy

Upgrading OTRS 3.1 to 3.2.1

After noting that our OTRS (www.otrs.org) was complaining about a major release update pending I took the plunge this morning and set about upgrading it. Initially I ran through the normal upgrade procedure and couldn't log on. Oops. Maybe I need to pay more attention here? Turns out there are quite a few caveats about this upgrade, and I'm hoping that what I note here will assist you - especially the database upgrade stuff. That was a bit of a surprise!

Initially I ran my normal otrs_pre_upgrade.sh script which stops services and backs everything up. That script looks like this:

#!/bin/bash
service cron stop
service apache2 stop
NOW=`date +%F`mkdir /root/backup/$NOW
BDIR=/root/backup/$NOW
cp -R /opt/otrs/Kernel/Config.pm $BDIR
cp -R /opt/otrs/Kernel/Config/GenericAgent.pm $BDIR
cp -R /opt/otrs/Kernel/Config/Files/ZZZAuto.pm $BDIR
cp -R /opt/otrs/var/ $BDIR
/opt/otrs/scripts/backup.pl -d $BDIR

Usually I then ln -s otrs-new otrs and run my upgrade script - but something failed along the way. Here is what I found:

Firstly, there are a lot more PERL modules required in 3.2 - these three caught me out:

• YAML::XS
• DBD::ODBC
• JSON::XS

I added them using aptitude - my OTRS install is on Ubuntu 12.04 LTS (www.ubuntu.com) - easy enough to do and then checked the modules again. If you are following the UPGRADING documentation, you should run:

• /opt/otrs/bin/otrs.CheckModules.pl

This will tell you what modules you require. I didn't bother with the Oracle or PostgreSQL modules as I'm not using those databases, nor am I interested in the Encode::HanExtra (no Chinese characters). 

Secondly there are database changes to be made. MySQL uses INNODB as the default storage engine. I've never even thought about this before - OTRS had always just run happily without asking me fancy questions about this sort of thing. Now in 9 of the UPGRADING document I had to apply database changes, including changing the default storage engine from INNODB to MyISAM.

Fortunately I stumbled across some nice scripts to do this at Techusers.net - here they are tailored to suit OTRS:

Step 1. Get all the table names from OTRS (you'll have to put in your password and you might have to change root to something more site applicable):

mysql -u root -p -e "SHOW TABLES IN otrs;" | tail -n +2 | xargs -I '{}' echo "ALTER TABLE {} ENGINE=INNODB;" > alter_table.sql

Step 2. Updating all the tables in one go (same as above - password + username update)

perl -p -i -e 's/(search_[a-z_]+ ENGINE=)INNODB/\1MYISAM/g' alter_table.sql 

Step 3. Applying the change to your SQL Database:

mysql -u root -p otrs < alter_table.sql 

Much thanks and kudos to the writers at www.techusers.net - this saved me from doing each table by hand! The search_profile table refused to change from INNODB to MyISAM but when I checked the dbupgrade scripts, this particular table isn't mentioned. It did not seem to affect the overall upgrade.

The rest of the upgrade went fairly smoothly. It's important to note, however, that you must go to Admin -> Packages -> Update Online Repository and then upgrade your packages to get better speed from your OTRS install. I found that after I did this, I restarted the apache service (service apache2 restart) and OTRS began humming along quite nicely. I'm still exploring the new features. Enjoy