For monitoring networks I have an old GX260 Dell desktop in the small form factor. Packed into this little device are a couple of network cards, added on are a wireless NIC and an extra USB network device. The purpose? To slot this in between the router and the network and see what's going backwards and forwards - very useful in the situation where a client is hemorrhaging bandwidth and doesn't know why. I'm running Ubuntu 10.04 LTS on it and it behaves very well.
Two of the main tools I use are ntop and iftop. For those of you not familiar with them, ntop monitors a particular interface and creates some nice webpages to be checked by the user in order to see what's going on through the network. iftop is similar but real time and is available through a console - which is the real appeal for me.
The server in question has two internal NICs, both are 100MB cards and are scripted to come up as a transparent bridge - br0. Basically I monitor that bridge and use one of the other interfaces to see what the server is seeing. Here is the script I use (adapted from elsewhere) to bring the bridge up:
slaveIfs="1 2 3 4 6 7 8 9 10";
[ -z "$cmd" ] && cmd="start";
case "$cmd" in
brctl addbr br0;
brctl stp br0 on;
brctl addif br0 eth1;
brctl addif br0 eth2;
(ifdown eth1 1>/dev/null 2>&1;);
(ifdown eth2 1>/dev/null 2>&1;);
ifconfig eth1 0.0.0.0 up;
ifconfig eth2 0.0.0.0 up;
ifconfig br0 up ### Adapt to your needs.
brctl delif br0 eth1;
brctl delif br0 eth2;
ifconfig br0 down;
brctl delbr br0;
#ifup eth0; ### Adapt to your needs.
#ifup eth1; ### Adapt to your needs.
It resides in /etc/init.d and is called (imaginatively) bridge.sh and then has appropriate symlinks to rc2.d.
NTOP is reasonably easy to configure on Ubuntu and is quite straightforward to get going - point it at http://localhost:3000 and set your username/password and off you go.
A tip to remember is that in the /etc/default/ntop it's a good idea to uncomment GETOPT="" and change it to read (if your network was 192.168.0.0/24):
and restart NTOP. Why is this important I hear you ask? Well on the br0 interface there is no IP assigned to it so NTOP doesn't automatically figure out what the local network is. By assigning this you can set it up to get proper info on local-remote, remote-remote and local-local traffic. Then leave it to run and see what it tells you.
Ah iftop is such a nice little bit of software. apt-get install it and then run it from the console with iftop -i br0 and it will tell you all sorts of things - data going from here to there and the level of bandwidth being used. I fancy pants it up a bit by creating a shell script and then launching it from there. iftop will use .iftoprc by default and if there isn't one will simply launch with it's own defaults. Here is the shell script I use (again I have adapted this from someone else):
# customisable settings
IFACE="br0" # the bridged interface
/usr/sbin/iftop -p -n -N -i $IFACE -F $LOCALNET -c $CONF
where /etc/iftoprc looks like this:
Most of these are self-explanatory and I believe you should examine them more closely if you are looking to deploy it. Suffice to say, it gives me the info I want and I'm happy with that setup. So I hope that gives you some food for thought and you can take some of this away with you when you're trying to find out what the hell is chewing up all your bandwidth and download limit!