Traffic analysis with Argus and Tcpdump

I recently performed some incident response using Argus for a customer who has been experiencing random site outages. With limited tools at my disposal besides a well known and somewhat useless IPS, I turned to Argus to get some insight into the traffic hitting their site. Provided to me was a libpcap file captured during an "attack" with Ethereal.

Since the capture file is so large and would take a considerable amount of time to transfer over my slow DSL connection, I decided to install Argus on a customer provided system. After installation, I generate Argus records from the pcap file:

argus -r cap.lpc -w /nsm/argus/cap.arg

Now that Argus data has been generated directly from the libpcap file,
ra (argus-client) tools can be used to work w/ the data.

Lets see the pairs of top talkers:

ramon -M Matrix -n -r cap.arg > argus_oldcap_top-pairs

24 Oct 06 15:22:26 212.243.221.227 1.1.1.2 10285 18722
583232 26384262
24 Oct 06 15:21:59 204.102.114.38 1.1.1.2 10460 18439
643850 25374248
24 Oct 06 15:21:58 206.167.78.12 1.1.1.2 8774 14051
609206 18376660
24 Oct 06 15:21:58 69.45.79.3 1.1.1.2 7877 9613
578814 11140600
24 Oct 06 15:22:00 84.53.134.34 1.1.1.2 6160 8243
505782 9621784
24 Oct 06 15:22:01 209.124.184.139 1.1.1.2 5454 8042
382142 10334766
24 Oct 06 15:21:58 128.220.254.131 1.1.1.2 6063 7197
388688 9576128
24 Oct 06 15:21:59 69.45.79.12 1.1.1.2 5777 7025
441117 7876808
24 Oct 06 15:22:00 192.91.244.226 1.1.1.2 4508 7607
266368 10625268

[root@fs3 fs3]# whois 212.243.221.227
inetnum: 212.243.221.194 - 212.243.221.255
netname: AKAMAI-NET
descr: Akamai Technologies Inc.
descr: 02139 Cambridge, MA 02139
...truncated...

The customer happens to utilize Akamai services. It's interesting that they occupy most of the top 10 talker spots during the "attack" window.
Now just top talkers:

ramon -M TopN -n -r cap.arg > argus_oldcap_top-talkers

24 Oct 06 15:21:58 1.1.1.2 241882 586948 20058262
654973106
24 Oct 06 15:21:58 204.102.114.38 18565 10583 25533978
650492
24 Oct 06 15:22:26 212.243.221.227 18722 10285 26384262
583232
24 Oct 06 15:21:58 206.167.78.12 14261 8932 18661924
617738
24 Oct 06 15:21:58 69.45.79.3 9693 7953 11245064
582918
24 Oct 06 15:22:00 84.53.134.34 8503 6320 9985540
514422
24 Oct 06 15:21:58 69.45.79.12 7882 6475 9020638
478809
24 Oct 06 15:21:59 209.124.184.139 8378 5644 10796882
392402
24 Oct 06 15:21:58 128.220.254.131 7485 6303 9980156
401648
24 Oct 06 15:22:00 192.91.244.226 7607 4508 10625268
266368
24 Oct 06 15:21:58 199.77.203.3 6799 5250 8301021
394010

Lets look at the full conversations for one of the top talkers:

ra -n -r cap.arg -L0 host 212.243.221.227

StartTime Flgs Type SrcAddr Sport Dir DstAddr
Dport SrcPkt DstPkt SrcBytes DstBytes State
24 Oct 06 15:22:26 * tcp 212.243.221.227.49629 ->
1.1.1.2.http 74 94 4910 119750 FIN
24 Oct 06 15:22:30 * tcp 212.243.221.227.49714 ->
1.1.1.2.http 36 38 3142 38738 FIN
24 Oct 06 15:22:32 * tcp 212.243.221.227.49745 ->
1.1.1.2.http 14 16 1944 5810 FIN
24 Oct 06 15:22:33 * tcp 212.243.221.227.49760 ->
1.1.1.2.http 12 14 1828 2322 FIN
24 Oct 06 15:22:33 * tcp 212.243.221.227.49761 ->
1.1.1.2.http 20 22 2244 17518 FIN
24 Oct 06 15:22:33 * tcp 212.243.221.227.49766 ->
1.1.1.2.http 14 16 1934 9144 FIN
24 Oct 06 15:22:33 * tcp 212.243.221.227.49771 ->
1.1.1.2.http 12 14 1822 2344 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49799 ->
1.1.1.2.http 14 16 1924 5256 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49800 ->
1.1.1.2.http 14 14 1944 1512 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49804 ->
1.1.1.2.http 14 16 1924 5288 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49806 ->
1.1.1.2.http 14 16 1924 5606 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49808 ->
1.1.1.2.http 14 16 1924 6508 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49811 ->
1.1.1.2.http 14 16 1924 6840 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49818 ->
1.1.1.2.http 22 24 2356 18910 FIN
24 Oct 06 15:22:35 * tcp 212.243.221.227.49820 ->
1.1.1.2.http 48 50 3760 56432 FIN
24 Oct 06 15:22:36 * tcp 212.243.221.227.49828 ->
1.1.1.2.http 12 14 1820 2258 FIN
24 Oct 06 15:22:36 * tcp 212.243.221.227.49830 ->
1.1.1.2.http 12 14 1816 2198 FIN
24 Oct 06 15:22:36 * tcp 212.243.221.227.49831 ->
1.1.1.2.http 12 14 1824 3236 FIN
24 Oct 06 15:22:36 * tcp 212.243.221.227.49833 ->
1.1.1.2.http 14 16 1924 6228 FIN
24 Oct 06 15:22:36 * tcp 212.243.221.227.49836 ->
1.1.1.2.http 12 14 1832 2526 FIN
24 Oct 06 15:23:10 * tcp 212.243.221.227.50165 ->
1.1.1.2.http 12 14 2172 1466 FIN
24 Oct 06 15:23:10 * tcp 212.243.221.227.50169 ->
1.1.1.2.http 56 60 4568 69956 FIN
24 Oct 06 15:23:21 * tcp 212.243.221.227.50274 ->
1.1.1.2.http 9819 18194 531772 25994416 CON

We can look deeper into this traffic using tcpdump w/ a BPF:

tcpdump -n -r cap.lpc -X -c 5 src 212.243.221.227

14:22:26.074184 212.243.221.227.49629 > 1.1.1.2.http: S
157012009:157012009(0) win 5840 (DF)
0x0000 4500 003c 05fb 4000 3406 78be d4f3 dde3 E..<..@.4.x..... 0x0010 aae0 6a4b c1dd 0050 095b d029 0000 0000 ..jK...P.[.).... 0x0020 a002 16d0 42a4 0000 0204 05b4 0402 080a ....B........... 0x0030 b507 d5d2 0000 0000 0103 0303 ............ 14:22:26.074186 212.243.221.227.49629 > 1.1.1.2.http: S
157012009:157012009(0) win 5840 (DF)
0x0000 4500 003c 05fb 4000 3406 78be d4f3 dde3 E..<..@.4.x..... 0x0010 aae0 6a4b c1dd 0050 095b d029 0000 0000 ..jK...P.[.).... 0x0020 a002 16d0 42a4 0000 0204 05b4 0402 080a ....B........... 0x0030 b507 d5d2 0000 0000 0103 0303 ............ 14:22:26.176200 212.243.221.227.49629 > 1.1.1.2.http: . ack
1017734966 win 5840 (DF)
0x0000 4500 0028 05fc 4000 3406 78d1 d4f3 dde3 E..(..@.4.x.....
0x0010 aae0 6a4b c1dd 0050 095b d02a 3ca9 6736 ..jK...P.[.*<.g6 0x0020 5010 16d0 916e 0000 0000 0000 0000 P....n........ 14:22:26.176203 212.243.221.227.49629 > 1.1.1.2.http: . ack 1 win
5840 (DF)
0x0000 4500 0028 05fc 4000 3406 78d1 d4f3 dde3 E..(..@.4.x.....
0x0010 aae0 6a4b c1dd 0050 095b d02a 3ca9 6736 ..jK...P.[.*<.g6 0x0020 5010 16d0 916e 0000 0000 0000 0000 P....n........ 14:22:26.176355 212.243.221.227.49629 > 1.1.1.2.http: P
0:437(437) ack 1 win 5840 (DF)
0x0000 4500 01dd 05fd 4000 3406 771b d4f3 dde3 E.....@.4.w.....
0x0010 aae0 6a4b c1dd 0050 095b d02a 3ca9 6736 ..jK...P.[.*<.g6 0x0020 5018 16d0 cfb9 0000 4745 5420 2f66 6f63 P.......GET./foc 0x0030 7573 2f72 6e61 692f 696e 6465 782e 6874 us/rnai/index.ht 0x0040 6d6c 2048 5454 502f 312e 310d 0a41 6363 ml.HTTP/1.1..Acc 0x0050 6570 743a 202a 2f2a 0d0a 5265 6665 7265 ept:.*/*..Refere 0x0060 723a 2068 7474 703a 2f2f 7363 6965 6e63 r:.http://scienc 0x0070 652e 6f72 662e 6174 2f73 6369 656e 6365 e.orf.at/science 0x0080 2f6e 6577 732f 3134 3538 3433 0d0a 4163 /news/145843..Ac 0x0090 6365 7074 2d4c 616e 6775 6167 653a 2064 cept-Language:.d 0x00a0 652d 6368 0d0a 5445 3a20 2063 6875 6e6b e-ch..TE:..chunk 0x00b0 6564 3b71 3d31 2e30 0d0a 436f 6e6e 6563 ed;q=1.0..Connec 0x00c0 7469 6f6e 3a20 5445 0d0a 4163 6365 7074 tion:.TE..Accept 0x00d0 2d45 6e63 6f64 696e 673a 2067 7a69 700d -Encoding:.gzip. 0x00e0 0a41 6b61 6d61 692d 4f72 6967 696e 2d48 .Akamai-Origin-H 0x00f0 6f70 3a20 310d 0a56 6961 3a20 312e 3120 op:.1..Via:.1.1. 0x0100 616b 616d 6169 2e6e 6574 2867 686f 7374 akamai.net(ghost 0x0110 2920 2841 6b61 6d61 6947 486f 7374 290d ).(AkamaiGHost). 0x0120 0a58 2d46 6f72 7761 7264 6564 2d46 6f72 .X-Forwarded-For 0x0130 3a20 3833 2e37 382e 3132 332e 3933 0d0a :.83.78.123.93.. 0x0140 4e50 475f 5352 435f 4144 4452 3a20 3833 NPG_SRC_ADDR:.83 0x0150 2e37 382e 3132 332e 3933 0d0a 486f 7374 .78.123.93..Host 0x0160 3a20 7777 772e 6e61 7475 7265 2e63 6f6d :.www.foo.com 0x0170 0d0a 5072 6167 6d61 3a20 6e6f 2d63 6163 ..Pragma:.no-cac 0x0180 6865 0d0a 4361 6368 652d 436f 6e74 726f he..Cache-Contro 0x0190 6c3a 206e 6f2d 6361 6368 652c 206d 6178 l:.no-cache,.max 0x01a0 2d61 6765 3d30 0d0a 436f 6e6e 6563 7469 -age=0..Connecti 0x01b0 6f6e 3a20 6b65 6570 2d61 6c69 7665 0d0a on:.keep-alive.. 0x01c0 5573 6572 2d41 6765 6e74 3a20 7377 6364 User-Agent:.swcd 0x01d0 2f35 2e32 2e30 3033 320d 0a0d 0a /5.2.0032....

Looks like non-attack traffic to me. Now here's how to collect traffic with Argus:

argus -c -d -i eth1 -w /nsm/argus/argus_eth1.out - ip

ps -ef | grep argus
root 17969 1 10 13:14 ? 00:01:07 argus -c -d -i eth1 -w /nsm/argus/argus_eth1.out - ip
root 17970 17969 0 13:14 ? 00:00:00 argus -c -d -i eth1 -w /nsm/argus/argus_eth1.out - ip
root 17971 17970 0 13:14 ? 00:00:01 argus -c -d -i eth1 -w /nsm/argus/argus_eth1.out - ip

Another Top Talkers report:

ramon -M TopN -n -r /nsm/argus/argus_eth1.out | more

07 Dec 06 13:14:43 1.1.1.2 1279856 3477698 108939939 3977099954
07 Dec 06 13:14:43 170.224.106.156 18831 103081 3548824 67919235
07 Dec 06 13:14:43 84.53.135.21 68171 46060 82953688 3681311
07 Dec 06 13:14:43 89.149.169.60 67878 37252 87414096 2788064
07 Dec 06 13:14:44 195.22.198.76 63206 34936 88545400 2042096
07 Dec 06 13:14:48 89.149.169.62 56294 37124 67945977 3006272

List flows for a specific IP:

ra -n -r /nsm/argus/argus_eth1.out -L0 host 89.149.169.60 | more

StartTime Flgs Type SrcAddr Sport Dir DstAddr Dport SrcPkt DstPkt SrcBytes DstBytes State 07 Dec 06
07 Dec 06 13:14:43 * tcp 89.149.169.60.64208 -> 1.1.1.2.http 66 82 5220 103360 FIN
07 Dec 06 13:14:45 * tcp 89.149.169.60.64261 -> 1.1.1.2.http 30 32 2824 30796 FIN
07 Dec 06 13:14:44 * tcp 89.149.169.60.64239 -> 1.1.1.2.http 12 14 1934 1754 FIN
07 Dec 06 13:14:45 * tcp 89.149.169.60.64270 -> 1.1.1.2.http 14 16 1992 3916 FIN
07 Dec 06 13:14:46 * tcp 89.149.169.60.64273 -> 1.1.1.2.http 14 16 1998 3920 FIN
07 Dec 06 13:14:44 * tcp 89.149.169.60.64244 -> 1.1.1.2.http 80 110 5886 141624 FIN
07 Dec 06 13:14:51 *S tcp 89.149.169.60.64426 -> 1.1.1.2.http 6687 17343 361846 24840880 FIN
07 Dec 06 13:14:52 * tcp 89.149.169.60.64459 -> 1.1.1.2.http 44 46 4056 50054 FIN
07 Dec 06 13:14:52 * tcp 89.149.169.60.64460 -> 1.1.1.2.http 16 16 2956 4792 FIN

Generate a top talkers report in pairs within 9am-12 generated using the following command:

ramon -M Matrix -n -r /nsm/argus/argus_eth1.out -t 13.9 - 12

While I did not find evidence of an attack, I have a much better picture of their traffic behavior. Looks like the developers have some work to do.

I first learned of Argus in Richard Bejtlich's The Tao of Network Security Monitoring.

See the Argus site for more complete documentation.