TCPdump - An introduction

As per usual, if you find any mistakes, concepts that could be explained better or have something to add, then please comment below and I'll correct it.

What is TCPdump ?

TCPdump is a command line tool that is used to look at packets on the network. TCPdump does not generate the packets itself but instead it analyzes the packets generated by other applications and from this, it can determine network behaviour and performance.

Despite being called TCPdump, you can choose from a range of protocols including IP,ARP, TCP and UDP.

How do I get TCPdump ?

From Ubuntu, I simply got it via 'sudo apt-get install tcpdump'. Otherwise the public repo is located here, along side some useful information and the manual pages.

You can also check the dependencies of TCPdump using 'apt-cache depends tcpdump'. The output that this returns for me is:

  Depends: libc6

  Depends: libpcap0.8

  Depends: libssl1.0.0

  Suggests: apparmor

  Conflicts: tcpdump:i386

 

If you get the package for the public repo, then you need will extract the content of the .tar.gz file using 'tar -zxf tcpdump-4.3.0.tar.gz' after using cd to move to the directory that you downloaded the file to, then install the program.

So I've got it.. but how do I use it ?

To start with use 'sudo tcpdump' to output to the terminal, this gives the standard output of TCPdump, using the default arguments since you have not passed TCPdump any arguments yet. To the output will be printed to the terminal, until you stop the program using Ctrl-C.

Don't be scared ... we will now begin the break this output down into understandable section (if you not scared then try 'sudo tcpdump -v' or 'sudo tcpdump -vv'  for a verbose output).

Changing network interface

When I'm using tcpdump, it listened on eth0 by default, but you can change this. Entering 'sudo tcpdump -D' will return a list of network interfaces that you can choose between. For me this returns:
        1.eth0
        2.wlan0
        3.usbmon1 (USB bus number 1)
        4.usbmon2 (USB bus number 2)
        5.any (Pseudo-device that captures on all interfaces)
        6.lo
The first item in the list is the Ethernet port, the 2nd is the Wireless card and the 3,4,5th is self explanatory. The 6th is the loopback virtual interface (more information here on wikipedia)

You can then change the interface, by calling TCPdump using 'sudo tcpdump -i <name-of-interface> '. For most of my work, I use the local Wi-Fi so can change the interface from Ethernet to wireless using 'sudo tcpdump -i wlan0'. 

Creating a file to store the tcpdump arguments

So far, we have only added one argument when we call TCPdump, but there will be a lot more to come. We can save these arguments, which filter the output of TCPdump to a file and then use the file when we call TCPdump.

Create the file using your favourite text editors (mine in gedit at the moment) and then pass the file to TCP dump using 'sudo tcpdump -F <my-file>'. The file dose not need any special file extension or layout.

You may need to change the file permissions so that TCPdump can extract the file. You can view the file premissions using 'ls -l <my-file>'. The 'ls' part is command line tool to list the files in a directory and the argument -l means use long format so the file permissions will be included. The file 10 characters are the file permissions, they are decoded as follows:

1. 'd' means this is a directory and '-' means this is a file
2. 'r' means that the file owner can read the file, '-' means they can't
3. 'w' means that the file owner can write to the file,  '-' means they can't
4. 'x'  means that the file owner can execute the file,  '-' means they can't. If this is a directory then 'x' means that the owner can list the files in directory, '-' means they can't.
5. 'r' means that the file group can read the file, '-' means they can't
6. 'w' means that the file group can write to the file,  '-' means they can't
7. 'x'  means that the file group can execute the file,  '-' means they can't. If this is a directory then 'x' means that the owner can list the files in directory, '-' means they can't.
8. 'r' means that everyone else can read the file, '-' means they can't
9. 'w' means that everyone else can write to the file,  '-' means they can't
10. 'x'  means that everyone else can execute the file,  '-' means they can't. If this is a directory then 'x' means that the owner can list the files in directory, '-' means they can't.

For me, the default file permissions are '-rw-rw-r--'. This therefore means that the file owner doesn’t have permission to execute the file. This can be changed using 'chmod u+x <my-file>'. The command line tool is chmod, this is used for changing file permissions, 'u' means we are considering the users permissions, '+' means we are granting a permission and 'x' means we can considering the execute permission. Now, if I redo 'ls -l <my-file>' the new result is -rwxrw-r--.

  

Sending the output to file

You can send the output of tcpdump to a file instead of printing it to the terminal using 'sudo tcpdump -w <my-file>'. This output is not saved as a plain text so you can't just read it in a text editor instead you need to use a special program to read the file and extract information from it. You can do this using tcpdump by 'tcpdump -r <my-file>'. Alternatively you can open it using wireshark, launch wireshark and using File>Open.

 

Filtering information by protocol

To filter packets by protocol, add the name of protocol to the arguments of tcpdump so use something like 'sudo tcpdump <name-of-protocol>'. The main protocols that I'm likely are use are IP, ARP, TCP and UDP but others are available, see the man pages for a full list

Filtering information by direction to different hosts

To filter packets by a direction and host, add the direction and then the host name. Possible direction options are src, dst, src or dst, src and dst. You specify the host using its IP address. You can use the logical operators not, or and and. For example, you can look up the local IP address of your machine using 'ifconfig <name-of-interface>'. If you don't specify the name of the interface, then all interfaces will be listed. Now if you only want to view incoming traffic you can use 'sudo udpdump dst <ip-address>'.

Change the level of detail 

Compared to the level of detail provided by the standard query, The detail can be reduced using '-q' for quiet output or increased using '-v' for verbose, '-vv' for more verbose and '-vvv' for even more verbose.

The opinion '-t' means do not print timestamp on each line and the option '-a' allows you to display each packet in ASCII or '-x' to display in hex or '-X' to display in hex and ASCII

View IP address instead of domain names

You can stop the translation of IP address to domain names using the '-n' opinion and you can stop the translation of port number too using the '-nn' opinion

Sources of Info

Wikipedia article, which contains very little information

Official public repo, including the man pages and FAQ 

Useful online tutorial at openmaniak, this site also has good tutorials on networking tools that I've previously covered including wireshark, OpenVPN, Iperf and ping.

Data Collection for Latency, Goodput & Jitter - Demo Pt 5.2

Everyone who I have spoken to about my work since yesterday, has asked me the same question. Why are you writing this in Java ? The answer is that I am going on the Google European Android Development Camp in a few weeks so I am using Java were possible in my work so that I can get familiar with the basics again.

To ensure that the server (my laptop) and the client (my android phone) can address each other I ensure that they are behind the same NAT so that private IP addresses can be used. This ensures that both devices can initialize an connect with the other

My Java code for analyzing and comparing Signpost Diagnostic Application to the results generated by Iperf and Ping, requires the following files:

• SignOutput.txt - this is generated by the Signpost Diagnostic Application, It can be generated by:
• Hard coding the server IP address into the application code, found here
• Uninstall previous versions of the application and connect the android phone via USB
• Within eclipse, load the application onto the android phone
• cd into Downloads/android-sdk-linux/platform-tools
• In another terminal, cd into the location of the demo server and start server using ./server.native
• use "./adb logcat-s SIGPST >> SignOutput.txt"
• The location of the file SignOutput.txt is now Downloads/android-sdk-linux/platform-tools, copy the file to /TestingSignpostAppOutput
• IperfOutput.txt - this is generated by a remote shell on the Android phone, it can be generated by:
• (In terminal 1) cd into Downloads/android-sdk-linux/platform-tools
• connect android phone via USB
• use "./adb shell" to start a remote shell 
• in another terminal (terminal 2), cd into /TestingSignpostAppOutput so output file is loaded straight into correct directory
• (In terminal 2) use  iperf -s -u >> IperfOutput.txt
• (In terminal 1) use iperf -u -c {insert laptop IP}
• PingDownstreamOutput.txt - this is generated by the server sending pings to the client (the android phone), it can be generated by:
• cd into  /TestingSignpostAppOutput so output file is loaded straight into correct directory
• Run a bash script containing the following:
• #!/bin/bash
  
  for i in {1..10}
  do
    ping {insert phone IP} -c 10 -n -q >> PingDownstreamOutput.txt;
  done
• PingUpstreamOutput.txt - this is generated by the client (android phone) sending pings to the server, it can be generated by:
• cd into Downloads/android-sdk-linux/platform-tools
• Save a bash script (lets call it pingUp) containing the following
• #!/bin/bash
  
  for i in {1..10}
  do
    ping {insert laptop IP} -c 10 -n -q >> PingUpstreamOutput.txt;
  done
• Connect android phone via USB and copy script to SD cards using  "./adb push pingUp / sdcard/   "
•  use "./adb shell" to start a remote shell 
• In the remote shell cd into /sdcard
• In the remote shell run the script using ./pingUp
• exit the remote shell
• cd into  /TestingSignpostAppOutput
• copy the PingUpstreamOutput.txt file to the laptop from the android phone using "./adb pull /sdcard/PingUpstreamOutput.txt  

Once you have generated all of these files then you can run my program SignpostOutputAnalysis.java found here which should output the average true and estimated latency, goodput and jitter.


The code in SignpostOutputAnalysis.java is still incomplete and untested, I also have not yet tested my instructions for generating the correct files at the correct locations for the java code to be ran. I will be doing this testing next...

Data Collection for Latency - Demo Pt 5.1

Now its time for me to collect data from the Signpost Diagnostics Application and Iperf to see if there is a statistically significant difference between the Latency and Goodput data from the two applications. [Jitter will soon be added to the application so I will need to go back the test that]

THE PLAN...

Part A - Data from Signpost Application

1. Connect the android phone (the client) and laptop (the server) to Internet in such a way that they are behind the same NAT box, so the client can initiate a connection with the server from a private IP address
2. Set up the OCaml Server running on my laptop
3. Change in client IP address in the client code and load onto android phone
4. Running client with server, collecting data whilst viewing output from log files

Part B - Latency data from Ping

1. Workout how to send pings bothways
2. Workout how to use ping to get RRT bothways
3. Collect some data on latency

Part C - Compare Data from Signposts Application and Ping

1. Extract sufficient data from both methods
2. Get data from both method into a suitable format
3. If required, convert units for latency so all data in same units
4. Compare data
5. Answer the Question: Is there a significant difference between the data collected by Ping (assumed to be accurate) and the Signpost Application

AND THE REALITY...
 

Part A

I've just been send the most recent version of the Signpost Diagnostics Application as a .apk file. I uninstall the old version of the application using the Android GUI and install the new version using:

cd /android-sdk-linux/platform-tools

./adb install ~/Downloads/SigcommDemoAndroid.apk

I quick come to realize that the above is stage is useless. It will correctly install an application from the .apk file but I need to be able to edit the code of the application so that I can set the client IP address at a later stage

I connect both the client and server to the same Wi-FI network so that the client will be able in initiate an connection with the server

As per usual, to start up the OCaml Server, I will do

cd Downloads/sebastian-SignpostDemo-53ebd3e/SignpostServerOCaml/
./server.native

I uninstalled the last Signpost Application, change the IP address in the code and load onto phone, via Eclipse using the run as dialogue.

Pressing start on the Android application triggers a connection to be initiated with the server, this is successful as the sever outputs a link detailing the android phones name, IP address and port number used.

I can view the logs for this application live, using:

cd  Downloads/android-sdk-linux/platform-tools

./adb logcat-s SIGPST

To filter the output of logcat, I use the arguments -s SIGPST, this gives me a log output such as:

I/SIGPST  ( 1520): Received Latency Upstream: 25500
I/SIGPST  ( 1520): Received Goodput Downstream: 4026
I/SIGPST  ( 1520): Received Latency Downstream: 6008
I/SIGPST  ( 1520): Received Goodput Downstream: 1262925
I/SIGPST  ( 1520): Received Latency Upstream: 49000
I/SIGPST  ( 1520): Received Goodput Downstream: 4787
I/SIGPST  ( 1520): Received Latency Downstream: 10376
I/SIGPST  ( 1520): Received Latency Upstream: 50000
I/SIGPST  ( 1520): Received Goodput Downstream: 2735
I/SIGPST  ( 1520): Received Latency Downstream: 17004
I/SIGPST  ( 1520): Received Latency Upstream: 46500
I/SIGPST  ( 1520): Received Goodput Downstream: 4461
I/SIGPST  ( 1520): Received Latency Downstream: 4463
I/SIGPST  ( 1520): Received Goodput Downstream: 799720
I/SIGPST  ( 1520): Received Latency Upstream: 51000
I/SIGPST  ( 1520): Received Goodput Downstream: 3684
I/SIGPST  ( 1520): Received Latency Downstream: 6875
I/SIGPST  ( 1520): Received Latency Upstream: 307000
I/SIGPST  ( 1520): Received Goodput Downstream: 4670
I/SIGPST  ( 1520): Received Latency Downstream: 8749
I/SIGPST  ( 1520): Received Latency Upstream: 16500
I/SIGPST  ( 1520): Received Goodput Downstream: 2652
I/SIGPST  ( 1520): Received Latency Downstream: 6795
I/SIGPST  ( 1520): Received Goodput Downstream: 24060150
I/SIGPST  ( 1520): Received Latency Upstream: 72500
I/SIGPST  ( 1520): Received Goodput Downstream: 4190
I/SIGPST  ( 1520): Received Latency Downstream: 6809

A quick code inspection, highlights that the latency values are divided by 1000 to be converted into secs, which means that the latency values here are in ms
I will look at how to convert this to a more convenient form in part 3

I can append the output to the file Signoutput.txt using ">> Signoutput.txt"

I have added a sample of Signoutput.txt here 

Part B

I open a remote shell on the android phone using ./adb shell, I then use the ping unix commend from the client to the server, and i get an output such as

 24 packets transmitted, 24 received, 0% packet loss, time 23070ms
rtt min/avg/max/mdev = 35.614/90.183/297.302/51.944 ms

The latency will be half the RRT so here it is 45.0915 ms

Then repeating the test from the server to the client and i get an output such as

43 packets transmitted, 40 received, 6% packet loss, time 42070ms
rtt min/avg/max/mdev = 33.371/93.812/241.825/45.579 ms

So the latency is similar to before at 46.906 ms

Now I want to collect the data from the ping output, so I wrote the following bash script to do this:

#!/bin/bash

for i in {1..10}
do
  ping 192.168.14.47 -c 10 -n -q >> Pingoutput.txt;
done

This sents 100 pings, in 10 sets of 10 and sent the results of the 10 test to a file called Pingoutput.txt
The first line "#!/bin/bash" mean that this is a bash script, the ping argument "-c 10" means said 10 pings and "-q" means quiet output. >> means append to file.

I've added a sample of Pingoutput.txt here

Part C

To analysis the output files, I have written the Java code here. Currently this code is untested

I'm back from brussels

From this ...

I arrived in the back in Cambridge yesterday afternoon after the IBM EMEA Best Student Recognition Event 2012 in Brussels, Belgium. The theme of this years 3 day event was "Data Analytics". The highlight for me on Wednesday 4th July was the a semenar by Prof. Bart Goethals, a lecturer at University Antwerp on data mining and machine learning in data analytics. For Thursday 5th July my highlights were a talk by Corinna Schulze from the IBM Governmental Program on privacy and a talk by Robert-Jan Sips focusing on how data analytics is being applied to water management in the Netherlands.  And for the final day was highlights was a talk by Guy Lefever on the application of Watson to future medical care. Overall I really enjoyed the event and I would throughly recommend attending if you get the opportunity to go.

It looks as if I'm going to have some new work on Signposts next week but I don't know as yet what this will be so over the weekend I intend to review some of the new material that I've learned over the my first few weeks on the Undergraduate Research Opportunities Project (UROP) here at the William Gates Building, Cambridge. My initial project brief was:

" Signposts - the virtual personal internet 

Users with multiple networked devices are having a hard time linking them up when said devices reside at more than one location. Simple example is trying to synch your android phone while on the bus to the laptop in your office and the Mac Mini at home. NATs and Firewalls and other obstacles to symmetric communication abound, and obstruct. Point solutions (Dropbox, iCloud) do not deal with many cases (different OS) and only work if you pay (with money or eyeball time on adverts). In any case, they are not acceptable to privacy conscious users.

Instead, we are building a system called signposts. This is the smallest conceivable virtual machine that sits in the public internet, and looks after your personal namespace of devices. Registration and lookup of names is "effectful", in other words, has side effects. The side effect is to use a set of possible tactics to rendezvous between the devices (various techniques for NAT traversal, for example) including possible multiple simultaneous path use, and also allowing for constraints, such as battery life, and price of use of connections. The system sits on top of DNSSEC, but more interestingly, as a piece of Computer Science, makes use of the new Cloud OS software being developed in CL, called Mirage (see http://www.openmirage.org/ and https://github.com/avsm/signpostd

This project is to help with the documentation and testing "in the wild" of signposts. We expect to uncover a lot of problems with different oddities (see http://conferences.sigcomm.org/imc/2011/docs/p181.pdf for examples) that appear, and we need to have good stories to account for what we are doing to the public and to security people. OCaml programming is pretty essential, as well as at least basic understanding of development tools (including git etc) "

Over the last few weeks, I had used a wide range of tools that are new or that I had not previously used to this extent including (in no particular order):

• Unix command line - although this is something that i typically use, I have definitely used it more and for a wider range of applications that ever before
• Git & GitHub - for my 2nd year group project PROTON, we took the (controversial) decision to use Mercurial and BitBucket instead of Git and GitHub like most other group so it been interesting to see how they compare
• Tor - As alot of my work over the last few weeks has been looking at testing the performance of tools for online anonymity, I've made extensively use of the onion router
• Orbot - Tor for android
• Tails - An OS build around tor  
• Iperf - As just explained as I've been working with testing network performance Iperf has been an invaluable tool (though I have recently discovered Torperf)
• Amazon Elastic Compute Cloud - I've been using this a server when testing network performance, its really use as it provides as global domain name and IP address and avoid pathological cases that could be created by having other cleint and server on the same network
• SHH - for remote access to the other computers I am testing with locally and the virtual machine on Amazon EC2
• dig-host - command line tools for performing DNS queries, just generally useful
• CyanogenMod - the OS on my android phone which allows easy using of proxies and tethering
• Privoxy - heavy customizable non-caching web proxy
• Hamachi - used with privoxy so set up encrypted VPN access
• OpenVPN - easy set up of VPNs
• Texmaker - a nice LaTeX editor, that I would throughly recommend which I have been using to keep the notes on my work over the last few weeks
• Mozilla Firefox & addons - previously a Chrome user myself, I've opted to use Firefox since the research project begin for a change and during this time I have experimented with a range of addons for privacy including: HTTPS Everywhere, NoScript,  AdBlock Plus, DoNotTrackPlus, Collusion ...
• Eclipse & Android Development tools - getting to grips with the basics of Android development in preparation for the Google European Android Camp 
• Telnet - for commend line control of Tor
• Wireshark - very useful for learning about computer networking, allows you to watch the packets on the network
• TraceRoute - brillient tools which used TTL to should the route that a packet takes to its destination
• Ocaml - the main project that I'm working with over the summer makes extensive use of Ocaml

I plan to use this list to visit some of the material that I've learned over the last few weeks in preparation for new work next week.