A few weeks ago we got a request from the application team to find out which application uses the most network traffic at the moment. Of course, iftop or NetHogs would have been most helpful here, but this wasn't installed on this SLES 11 box. We couldn't convince sar(1) either to output how much traffic there is on a certain network port.

Hm, there was tcpdump (v3.9.8) installed on that box, maybe there is some packet size information in there that we could use? Let's see:

$ sudo /usr/sbin/tcpdump -np -c 1 2>/dev/null
03:09:45.731126 IP 10.0.0.2.1502 > 10.0.0.3.9034: P 931760268:931760449(181) ack 1045182458 win 501

And there it was! According to its man page, that number in parentheses was the packet size in bytes.

Let's try this. Since each of our applications on that host is listening on a TCP socket, gather all the network ports we care for:

 $ netstat -ntl | awk '{print $4}' | awk -F: '/^[0-9]/ {print $2}' | sort -n > listening

Run tcpdump for a certain amount of time:

 $ ( sudo /usr/sbin/tcpdump -np > dump.pcap & ) && sleep 10 && sudo pkill tcpdump

Now, for every listening port, sum up the packet size for each packet found in the tcpdump output:

 $ for p in `cat listening`; do
    printf "PORT: $p     KB/s: "
    egrep "\."$p"[\ :]" dump.pcap | \
       awk '/\([0-9]*\)/ {print $7}' | \
       sed 's/.*(//;s/)//' | awk '{sum+=$1} END {print sum/10/1024}'
    done | sort -nk4 | tail -5
 PORT: 4673     KB/s: 0
 PORT: 4673     KB/s: 0
 PORT:   22     KB/s: 0.003
 PORT: 1512     KB/s: 5.371
 PORT: 1522     KB/s: 301.004

We matched for outbound and inbound packets here with that "PORT[\ :]" expression - adjust as needed.

So, there we have it: the application communicating on port 1552 transferred at a rate of 301 KB/s at the time we measured. Use netstat -p or lsof -i to find out the corresponding application for this network port.

Note: on another system a newer version of tcpdump (v4.3.0) was installed and the output changed considerably:

 $ sudo /usr/sbin/tcpdump -nnp -c 1 2>/dev/null
 18:47:19.598570 IP 10.0.0.5.5001 > 10.0.0.6.42711: Flags [P.], seq 2487026730:2487027316, ack \
  651238704, win 1040, options [nop,nop,TS val 321092974 ecr 2893804], length 586

Here, the size of the packet is at the end of the line and we can simplify our search routine a bit:

 $ for p in `cat listening`; do
    printf "PORT: $p     KB/s: "
    egrep "\."$p"[\ :]" dump.pcap | \
       awk '/length [1-9]/ {sum+=$NF} END {print sum/10/1024}'
 done

And for completeness' sake, here's a (slower) version without grep:

 $ for p in `cat listening`; do 
    awk "/\."$p":.*length [0-9]*$/ {sum+=\$NF} END {print \"PORT: $p  KB/s:  \" sum/10/1024}" dump.pcap
 done

Update: another (and maybe more elegant) solution is to create temporary iptables rules for all interesting ports and then use its builtin traffic counters (though formatting is kinda weird):

 $ for p in `cat listening`; do iptables -A INPUT -p tcp --dport $p; done
 $ iptables -n -L INPUT -v | awk '/pkts/ || /dpt:/ {print $1,$2,$NF}' | sort -nk1 | tail
 0 0 dpt:8114
 0 0 dpt:8115
 0 0 dpt:8120
 0 0 dpt:9999
 pkts bytes destination
 17      1031 dpt:4143
 25     10284 dpt:8119
 139    31000 dpt:8116
 690     121K dpt:8123
 15302    23M dpt:1234
 
 $ for p in `cat listening`; do iptables -D INPUT -p tcp --dport $p; done