Traffic Monitoring using Ubuntu Linux, ntop, iftop and bridging

This is an update of an older post, as the utilities change, so has this concept of a cheap network spike - I use it to troubleshoot network issues, usually between a router and the network to understand what traffic is going where. The concept involves a transparent bridge between two network interface cards, and then looking at that traffic with a variety of tools to determine network traffic specifics. Most recently I used one to determine if a 4MB SDSL connection was saturated or not. It turned out the router was incorrectly configured and the connection had a maximum usage under 100Kb/s (!) At $1600 / month it's probably important to get this right - especially when the client was considering upgrading to a faster (and more expensive) link based on their DSL provider's advice.

Hardware requirements:

I'm using an old Dell Vostro desktop PC with a dual gigabit NIC in it - low profile and fits into the box nicely. Added a bit of extra RAM and a decent disk and that's really it. I'm also running this on an old Dell D420 with a gigabit PCMCIA adaptor - useful for the out and about jobs.

Software requirements:

• Ubuntu 12.04 LTS - I've chosen this for longevity purposes, previously I'd used non-LTS operating systems and the updates naturally ran out. I tried this with FreeBSD 9.1 but had issues with packages and getting traffic across the network bridge effectively (probably more my screw up than FreeBSD's)
• ntop - network traffic analysis monitor from www.ntop.org. They have version 5 available from the repositories on the site, version 4 is included in Ubuntu 12.04
• iftop - a neat command line package that shows network usage from a terminal screen. Highly configurable
• tcpdump or equivalent for deeper packet analysis.

Configuration

Setting up the box and the ethernet bridge

Setting up the box is straight forward - go through the usual Ubuntu installation. Use aptitude or apt-get to install bridge-utils and iftop

We want the network bridge between our ethernet adaptors to come up automatically. To do so edit /etc/rc.local and pop this into it (assuming eth1 and eth2 are the interfaces you want to bridge. I have eth0 configured statically in this instance so I can browse from other machines to it)

/etc/rc.local

brctl addbr br0
ifconfig eth1 0.0.0.0 promisc up
ifconfig eth2 0.0.0.0 promisc up
brctl addif br0 eth1
brctl addif br0 eth2
ip link set br0 up

This will bring the bridge up at boot time.

ntop

After you've added the necessary repositories to your aptitude configuration, install ntop5 using apt-get install ntop5

I run this from the command line - as a service it seems to fail fairly consistently. The command is:

ntop -P /var/lib/ntop -Q /usr/local/share/ntop/spool/ -i br0 -u ntop -m 192.168.0.0/24 -d

-P sets the database file path

-Q sets the spool file path

-i sets the interface (br0 as per /etc/rc.local)

-m sets the local subnet - in this case 192.168.0.0/24 (change to suit)

-d sets it to become a daemon freeing up your terminal

Browse to localhost:3000 to find your ntop installation, or if you have a third network card go to the address on the network e.g. 192.168.0.30:3000 and view your traffic stats.

iftop

To get what I want out of iftop, I run a script that calls it and configure the /etc/iftoprc file. The script is:

bridge_monitor.sh

#!/bin/sh
# customisable settings
LOCALNET="192.168.0.0/24"
IFACE="br0" # the bridged interface
CONF="/etc/iftoprc"
/usr/sbin/iftop -p -n -N -i $IFACE -F $LOCALNET -c $CONF

The contects of /etc/iftoprc are:

dns-resolution: yes
port-resolution: yes
show-bars: yes
promiscuous: no
port-display: source-only
#hide-source: yes
#hide-destination: yes
use-bytes: yes
sort: 2s
#line-display: one-line-both
show-totals: yes

Again customise to suit and start monitoring that network!

OTRS Restore Procedure and backup script

As I note in my previous post, I managed to kill my OTRS install and as usual had to trawl around the net to remember how to restore it. In a nutshell:

# mysql -u root -p
msyql> drop database otrs;
mysql> create database otrs;
mysql> ext
# /opt/otrs/scripts/restore.pl -d path_to_backup /opt/otrs

You did back up right?

Nightly I run a script with the following in it:

otrs_backup.sh


#!/bin/bash
# Variables below - change these to suit
NOW=$(date +"%Y-%m-%d_%H-%M") # this gets the correct file name for OTRS backup
LOCAL=/root/backup # a local directory for OTRS to backup to
REMOTE="user@backupserver:~/backup/OTRS/" # remote backup dir - nfs share, ftp or cifs
/opt/otrs/scripts/backup.pl -d $LOCAL # OTRS internal backup (files and DB)
tar -cf $LOCAL/$NOW.tar $LOCAL/$NOW # creates a file from the OTRS backup folder - more efficient to copy over a network
gzip $LOCAL/$NOW.tar
rm -rf $LOCAL/$NOW # tidy up
scp -r $LOCAL/$NOW.tar.gz $REMOTE # scp to remote directory

You may wish to run this from crontab after copying otrs_backup.sh to /usr/local/bin:

0 20 * * * /usr/local/bin/otrs_backup.sh

This will run at 10pm each night - theoretically you could run it more frequently. OTRS databases will a lot of attachments get quite large though so be mindful of that (I have a couple I manage that are 1GB and are only 5 months old)

Enjoy

Upgrading OTRS 3.1 to 3.2.1

After noting that our OTRS (www.otrs.org) was complaining about a major release update pending I took the plunge this morning and set about upgrading it. Initially I ran through the normal upgrade procedure and couldn't log on. Oops. Maybe I need to pay more attention here? Turns out there are quite a few caveats about this upgrade, and I'm hoping that what I note here will assist you - especially the database upgrade stuff. That was a bit of a surprise!

Initially I ran my normal otrs_pre_upgrade.sh script which stops services and backs everything up. That script looks like this:

#!/bin/bash
service cron stop
service apache2 stop
NOW=`date +%F`mkdir /root/backup/$NOW
BDIR=/root/backup/$NOW
cp -R /opt/otrs/Kernel/Config.pm $BDIR
cp -R /opt/otrs/Kernel/Config/GenericAgent.pm $BDIR
cp -R /opt/otrs/Kernel/Config/Files/ZZZAuto.pm $BDIR
cp -R /opt/otrs/var/ $BDIR
/opt/otrs/scripts/backup.pl -d $BDIR

Usually I then ln -s otrs-new otrs and run my upgrade script - but something failed along the way. Here is what I found:

Firstly, there are a lot more PERL modules required in 3.2 - these three caught me out:

• YAML::XS
• DBD::ODBC
• JSON::XS

I added them using aptitude - my OTRS install is on Ubuntu 12.04 LTS (www.ubuntu.com) - easy enough to do and then checked the modules again. If you are following the UPGRADING documentation, you should run:

• /opt/otrs/bin/otrs.CheckModules.pl

This will tell you what modules you require. I didn't bother with the Oracle or PostgreSQL modules as I'm not using those databases, nor am I interested in the Encode::HanExtra (no Chinese characters). 

Secondly there are database changes to be made. MySQL uses INNODB as the default storage engine. I've never even thought about this before - OTRS had always just run happily without asking me fancy questions about this sort of thing. Now in 9 of the UPGRADING document I had to apply database changes, including changing the default storage engine from INNODB to MyISAM.

Fortunately I stumbled across some nice scripts to do this at Techusers.net - here they are tailored to suit OTRS:

Step 1. Get all the table names from OTRS (you'll have to put in your password and you might have to change root to something more site applicable):

mysql -u root -p -e "SHOW TABLES IN otrs;" | tail -n +2 | xargs -I '{}' echo "ALTER TABLE {} ENGINE=INNODB;" > alter_table.sql

Step 2. Updating all the tables in one go (same as above - password + username update)

perl -p -i -e 's/(search_[a-z_]+ ENGINE=)INNODB/\1MYISAM/g' alter_table.sql 

Step 3. Applying the change to your SQL Database:

mysql -u root -p otrs < alter_table.sql 

Much thanks and kudos to the writers at www.techusers.net - this saved me from doing each table by hand! The search_profile table refused to change from INNODB to MyISAM but when I checked the dbupgrade scripts, this particular table isn't mentioned. It did not seem to affect the overall upgrade.

The rest of the upgrade went fairly smoothly. It's important to note, however, that you must go to Admin -> Packages -> Update Online Repository and then upgrade your packages to get better speed from your OTRS install. I found that after I did this, I restarted the apache service (service apache2 restart) and OTRS began humming along quite nicely. I'm still exploring the new features. Enjoy

Privacy in the modern times

It seems to me that with the advent of all our social media applications – Facebook, MySpace, twitter, Flickr, Tumblr etc., the ability for us to get our thoughts out there is the easiest it’s ever been. The detail that this provides to people is remarkable. Ad companies use it for focused advertising, other companies use it for various nefarious means and criminals use it to steal our identities. Less insidious I think is that people can know us in a way they never have before. The cost to our privacy seems to be one we’re happy to bear though – the most popular consumer mobile devices have Facebook and the like built in and integrated with everything – messages, photos, GPS locations etc. Our internal thoughts and feelings are now able to externalised quickly and limitlessly. Nine times out of ten this is incredibly boring stuff (let’s face it, we’re not as interesting as we’d like to be), but the fodder for bullies, abuse and misuse is extraordinary. It’s very much like posting a sticky note to the wall at school with your latest thoughts and opening yourself up to complete, uncontrolled scrutiny. We all know how much impact putting yourself out there at school can have. Now we do it on a global scale and it doesn’t seem to be an important thing to consider the value of our private lives.

Never mind the fact that once something gets to the Net it never seems to leave. Those embarrassing moments, which once passed leaving only an uncomfortable memory, now linger – sometimes that moment makes it to Youtube and it can live forever. These little moments, of often excruciating embarrassment now have the potential to harm us forever. One can make injunction to have them removed, a costly and time consuming procedure which often brings even more attention to the moment and so is only a partial remedy. It doesn’t stop people from downloading and keeping these images and movies for ever on their own personal machines. This is even exploited as people do stupid things for attention (and get it). The slapstick comedy of the Three Stooges seems to have morphed into Jackass and our collective intelligence has taken a mighty hit. But back to privacy.

I see the youth of today posting details, photos and information about themselves that as a young person I would never have done (and as an old person am even less likely). The generation older than mine are so recalcitrant about their personal feelings and life it can be like pulling teeth getting any information out of them even under the best and most appropriate of circumstances. It certainly adds to their mystery – another underrated and mostly lost commodity in the world. Whether it’s the endless tweets of a person summing up their thoughts in 160 characters or their barely there clothing, mystery is a lost art. Privacy and mystery are inextricably linked, and we don’t seem to realise that as you give up one, you give up the other. Potential partners or even potential employers can look into what you are doing, often without appropriate context, and make judgements on you and your behaviour without having other critical information, for while we do tend to post a lot of information to the net, most of it requires a certain amount of local knowledge (i.e. you had to be there type thing)

It is incumbent upon IT professionals to help non-technical people navigate this quagmire of what to do. The privacy settings of Facebook (for example) are not clear cut and there have been many times I’ve seen a profile completely exposed to all and sundry – birthdate, address, phone etc. – everything the budding identify thief needs to acquire and then sell your identity with. We need to help people understand what they can and should share to the world. There is a vulnerability to such openness and most lay people don’t understand the potential for harm. IT professionals have an obligation therefore to protect people from their potential loss through education and technical assistance. If you consider your current visibility on the Internet – where are you vulnerable?

The relevance of Microsoft Office in the Cloud based work environment

In the last few months several of my clients have made the jump from using Microsoft Exchange to using Google Apps. Initially this has just been for email, calendar and contacts but has since morphed into using Drive as well. At my office we have also made the change, but our split in staff - half telephony and half IT, means that the IT guys have really taken it on board, more so in fact than the telephony chaps.

Recently I was working with some gents from another organisation still using Office - we were using tethered laptops to exchange spreadsheets with simple data in them - at least they were doing it that way. I was getting the spreadsheet, importing it to Apps, updating it and sending it off. The beautiful thing was - I was doing it from my iPhone, HTC and my Google Nexus - same file, updating straight away and then emailing back to the guys - brilliant! And with offline files enabled it was even easier - I could still access files via Chrome and makes changes. The sync worked very well once networking was restored.

As more of my clients use Google Apps, more of them are questioning the need of having Microsoft Office on their PCs. After all, for the relatively low price of $5 per month per user for Google Apps Business software updates are included and the end user is always on the latest version of the office software - a definite advantage over the constant upgrade process one goes through with the various offices... not just Microsoft Office. Sharing files between people, both within the organisation and outside the organisation are quite easy. Even sending it to people without Google Apps is easy - files can be sent as PDF's or converted to Microsoft office files. 

Sharing and modifying files at the same time with people is easy, Apps notifies you of what is being altered and who is connected to it - a handy feature if, for example, two colleagues are updating a Sheet with details of different computers, without accidentally over writing the other person's data. Very useful. 

Anyway, this has been a musing - just something to ponder in quiet moments....

Lenovo L430 Review

Recently to replace my HP6560b notebook, I purchased a Lenovo L430. It looks like this:

Lenovo L430 Notebook

and it's specifications are:

• 14" AntiGlare LCD monitor
• i5-2520M Processor,
• 4GB 1600MHz DDR3 RAM (upgraded to 8GB),
• 320/7200 HDD,
• Multi-Burner,
• HD Graphics 3000,
• Bluetooth,
• Fingerprint reader,
• 720p Camera,
• ABGN Wireless,
• WWAN,
• Card Reader,
• Windows 7Professional 64,
• 6 Cell battery (max 7.5 hours run time),
• 1 Year return to depot warranty
• GPS 

It has the same sturdy feel and build that all Lenovo notebooks do, a legacy of the IBM heritage Lenovo has successfully perpetuated. I'm absolutely delighted by the keyboard on this device - it is the best notebook keyboard I have ever used. I'm comparing it to the following notebooks: HP 6560b, HP tc4400, Lenovo R500, MacBook Air, various ASUS laptops, various Toshiba laptops, Lenovo X1 and HP 6730b. Obviously I get around on the notebook scene :-)

The touchpad and other pointing device are efficient and work well. I was surprised and impressed also by the efficacy of the wireless - it has been more efficient and connects at higher speed than my other notebooks, including the HP6560b. The built in GPS is pretty neat too - although I'm not sure what the hell I'm going to do with it :-) Generally I remove a lot of the pre-installed software, but I have of course left the Lenovo software, which is gradually improving all the time and it's usefulness is improving as well. I installed 1.6GB of updates in the first two hours of having this laptop which is indicative of Windows 7 out of the box. I had a spare 4GB of RAM laying around so I installed that for fun.

Overall I've found this notebook to be quite fast in general usage, quick to boot and efficient in it's power usage. I've used this notebook all day, and by the time I left the office still had 20% of battery available. This is doing web browsing, Excel, file moves etc. The L430 also has USB3 and an always on USB port I used to good example as I travelled between client offices today to charge my HTC One X. I'm looking forward to my docking station arriving so I can plug it into my dual 22" LED monitors. For about $1200 for the laptop and the docking station together these laptops are great value. I'm not sure how long they'll be around for - pretty they'll be discontinued soon in this trend of everything going to Windows 8. Grab one if you can :-)

Adventures in backups and restores - StorageCraft Shadow Protect

After taking a new client over recently I have been working with StorageCraft's Shadow Protect backup solution for servers: http://www.storagecraft.com.au/  Initially I was just checking that the damn thing was backing stuff up to somewhere and trying to work through a multitude of other issues. Since these problems have settled down I've had more leisure to examine this product. 

The server solution has a wide range of backup capabilities - full backups, incremental backups and continuous incremental backups. The incremental backups can be performed every 15 minutes and up from there. These are keyed back to an initial full backup of the server and are a sector by sector backup. This means we can mount the backup image as the next free drive on the server. One can simply copy across the lost files and then dismount the backup. It's elegant, simple and works well. There is a very nice extension to this as well - the Granular Recovery for Exchange. When needing to recover a email, a folder or an entire mailbox, the ShadowProtect GRE makes it very easy. Simply mount the backup image of the drive with the Exchange database on it, tell GRE where it is, where the existing Exchange database is and then copy across the items you need to recover. It's awesome because of it's ease and speed, search functions and also as a means of migrating to a newer version of Exchange. The few times I've used this it absolutely blew me away with it's ease of use, speed and simplification of data recovery. 

ShadowProtect uses ShadowControl ImageManager software to help manage all the images and snapshots. It's another neat bit of kit and it's functionality is excellent.

The most impressive part of this backup solution is the ability it has to virtual boot a server from an image. You can boot it as an Oracle VirtualBox virtual machine, or as a HyperV VM. A failed physical server can be up and running in minutes as a virtual server. It will still be running backups as per the physical server and once the hardware is restored, the up to date virtual server can be restored back to the physical hardware and once again full services can be restored.

As with any software though it needs to be carefully set up and configured, with usual ongoing maintenance applied. The install I'm working on currently is a bit of a mess and there are a lot of images that need to be sorted out. As a backup solution it also ties in with the virtual environment, although this is something I'm still exploring - check StorageCraft's ShadowProtect out, it's well worth a look.