I have been planning a completely different blog post for over a week now, but I’m currently not advancing in my research. Instead, due to the many experiments, I’ve become better at capturing traffic.

Tools of the trade: Wireshark (Windows) and tcpdump (Unix). While tcpdump works on the command line and is very lightweight, Wireshark comes with a lot more options and a GUI. The two are compatible: capture files saved with tcpdump can be opened in Wireshark.

TCPdump parameters
This very simply tool is usually included by default on a Unix platform (Linux, OpenBSD, vendor systems running on a Linux kernel, …). It can have many parameters, of which I’ll list the most useful here:

  • -n – Makes sure no name resolving of addresses, and conversing of port names (e.g. 80 to www) is done. Since you’re probably troubleshooting on layer 2 and 3, it’s easier to see the actual numbers.
  • -i int – Specifies an interface to capture on. If you don’t specify it in a recent Linux kernel, it will take the first non-loopback it finds. However, when networking, devices you encounter often have multiple interfaces.
  • -s length – Sets the length that packets will be captured. By default only the first bytes are captured. Setting this value to 1514 will allow you to capture all packets on an Ethernet interface completely, which is handy for the -w option below.
  • -w file – Writes the capture to a file instead of showing it on-screen. If you specify it as a *.pcap file, it can be opened in Wireshark later on!

As an example of the above, ‘tcpdump -n -w /var/log/ethertest.pcap -i eth0’ will do a packet capture on interface eth0 and write the information to the ethertest.pcap file, without doing any name resolution. To stop a capture, press ‘Ctrl+C’. A more complete list of parameters can be checked here.

TCPdump filters
Apart from the parameters, filters are possible. Below a few handy ones:

  • host ip – Only capture packets that originate from or are destined to a certain IP address. The most common mistakes you can make with this filter are forgetting there’s NAT somewhere involved, so you don’t see anything because you’re filtering on an IP address that’s no longer present in the packets, or something is using IPv6 though dual stack and while you’re capturing on the right interface, nothing shows because you’re filtering on IPv4.
  • net prefix – Only capture traffic from or to a certain subnet, e.g. 192.168.4.0/23.
  • port number – Capture traffic with a certain port number, both UDP and TCP. Usually this is clear enough as there’s rarely both a UDP and a TCP stream on an interface with the same port number. Also counts both for destination and source port number.
  • icmp – ICMP traffic only. Great to see if your pings go through somewhere.
  • vlan number – Only frame that have an 802.1q header with the matching VLAN number will be captured. This option is very important on trunk links in combination with ‘-w’, as I’ve noticed tcpdump doesn’t always write tagged frames correctly to a file unless this filter is applied.
  • ‘not’, ‘and’, and other booleans – These allow you to negate things and make combinations.

Some examples explain these filters:
‘tcpdump -n -i eth0 not port 22’: capture all traffic except port 22 (useful when you’re connected through SSH on the same interface).
‘tcpdump -n -i eth0 host 10.0.5.3 and host 10.2.3.14’: capture all traffic between those two IP hosts.

Wireshark
Wireshark has some extra functionality compared to tcpdump, but tcpdump filters are present as well. Under ‘Capture’, ‘Options…’ you can define a capture filter, which uses the exact same parameters as tcpdump. Using ‘not port 3389’ here for example is useful if you’re trying a capture on a remote computer. You can also use a filter on all captured frames to show only those interesting to you. Difference with the pre-capture filter is that this filters out what you see, but not what is captured, which is useful when taking a raw capture to examine later. Some important ones:

  • icmp: Just ICMP traffic.
  • udp.port and tcp.port: TCP or UDP port number. Unlike tcpdump’s ‘port’ you can differentiate between UDP and TCP here.
  • ip.dst and ip.src: Source and destination IP addresses.
  • eth.dst and eth.src: Source and destination MAC addresses.
  • tcp.stream: Filter out one single TCP stream. Useful to follow a connection in a sea of frames.

Booleans can be used just like with tcpdump, and to define a variable, use ‘==’, e.g. ‘ip.dst == 10.0.0.1 and not icmp’  will show all traffic towards 10.0.0.1, except ICMP packets. Next to ‘==’ (equal to), variables can also be ‘>’ (greater than), ‘>=’ (greater or equal than), ‘!=’ (not), ‘<‘ (smaller than) and ‘<=’ (smaller or equal than).

Wireshark’s extra functionality is due to the graphical element: under ‘Statistics’ you can use ‘IO Graphs’ to show bandwidth usage during the capture. This helps visualize the traffic patterns: are there sudden bursts of traffic, or just a steady flow? Here too you can filter.
Under the same ‘Statistics’ there’s also ‘Conversations’, which makes a list of all captured traffic flows. You can sort this list to show the connections that use the most bandwidth. Very useful to find what’s causing unexpected bandwidth usage.

SPAN ports
Optimizing the filters to capture data is one thing, but optimizing the replicated data to capture on a SPAN interface can be beneficial too. A basic SPAN session on the same switch is set up as following:

Router(config)#monitor session 1 source interface G1/0/1
Router(config)#monitor session 1 destination interface G1/0/5

There are however a few tweaks possible. First, capturing on a trunk link is possible, and it’s possible to filter out only the required VLANs using the ‘monitor session 1 filter vlan vlan-list‘. A good use of this is when trying to capture traffic on a few VLANs on a gigabit trunk link, while the SPAN port is a 100 Mbps port. By filtering only the needed VLANs less traffic will be replicated and the 100 Mbps link will not saturate as quickly.

Second, while a SPAN port replicates most traffic, it does not replicate switch control frames like BPDUs and CDP frames. You can force the replication of these frames using the ‘monitor session 1 destination interface G1/0/5 encapsulation replicate’ command. A good use for this is checking why an IP Phone will not come online, for example.

The above are all just small tips and tricks, but together they make troubleshooting something a lot clearer.

Advertisements