Dangerous ARPs

Noah Davids

The ARP protocol works quietly in the background, and most of the time it works flawlessly. Unfortunately, such reliability can lead to complacency; when TCP connection problems occur, no one ever thinks of the ARP protocol. In this article, I'll provide a quick review of the ARP protocol and describe some ways in which ARP can create problems. I'll also show how to diagnose these problems using a protocol analyzer like tcpdump.

The ARP Protocol

For a TCP/IP packet to be transmitted over an Ethernet, the packet must be encapsulated in an Ethernet Frame. The first 6 bytes of that frame are the Ethernet media access control (MAC) address of the destination host. This will be either the MAC address of the destination host (if the source and destination hosts are on the same subnet), or the address of a router (if the source and destination are on different subnets).

The ARP protocol is used to find the MAC address of the destination. The sender broadcasts an "ARP Request" packet containing the IP address of the destination (or the IP address of the router) and the owner of that IP address sends an "ARP Reply" packet containing its MAC address. The sender then places the IP to MAC address association in an ARP cache, so it does not have to go through this ARP Request/Reply for every packet. Because the size of the ARP cache is limited and the IP to MAC address mappings can be changed by many things (e.g., hardware failures and DHCP renewals), entries in the ARP cache eventually time out. How long an entry stays in the ARP cache depends on the operating system and its configuration parameters.

You can display the ARP cache table with the ARP command. Note that all examples were done on a system running Red Hat 7.2 (Linux 2.4 kernel). The "?" indicates that the IP address does not resolve to a name. Name resolution, especially name resolution failure, can take time. So if the command appears to hang, try it with the -n option, which will cause the command to skip the name resolution step. See Listings 1a and 1b.

Remember that there is no provision in the ARP protocol to confirm that the host sending the ARP Reply is actually the owner of the requested IP address. Generally, this is a good thing. Many routers and hosts can provide something called "Proxy ARP". Basically, if one host sends out an ARP Request for another host on a remote network, a router can respond with an ARP Reply as if it were the destination host. This allows the source host to communicate when, for whatever reason, it believes that it and the destination host are on the same subnet but in fact they are not. One common cause of this situation is a subnet mask that has fewer bits then it should (e.g., 255.255.0.0 instead of 255.255.255.0). If you ever see an ARP cache where lots of hosts have the same MAC address, you can be pretty sure that the MAC address belongs to a router doing proxy ARPs. In Listing 2, note the entries for 192.168.2.1, 2 and 8. You should also investigate why the system you are on is ARP-ing hosts on a different subnet.

Of course, if something is generally good, there are also times when it is bad. If a router (or any host) is doing proxy ARPs when it shouldn't be, then you have a problem. As a network support specialist, I see this kind of problem once or twice a quarter. That's not very often, but if you don't look in the right place, it can take much longer to solve than typical problems and can create months' worth of headaches all on its own. On the other hand, if you know what to look for, things become much easier.

The following traces were captured in a test environment (hence the similar names and MAC and IP addresses between the two problems), but the traces accurately represent what I saw at my customer locations.

Problem #1: On Again, Off Again Connections

In the first case, the symptom is an intermittent connection between two hosts -- Top and RoadRunner. Top is running Red Hat 7.2, and its IP address is 172.16.1.2. RoadRunner is running Microsoft Windows 2000 Server and has an IP address of 192.168.1.100. Top is running the client side of the application, and RoadRunner has the server side. The connection will work for a time, and then Top will get a "Connection closed by foreign host" error message after sending some data. A new connection can be established but will eventually get the same "Connection closed by foreign host" error.

Listing 3 shows a partial trace taken on Top with tcpdump. It is unremarkable until frame #8. Frame #8 is a retransmission (from Top) of frame #7 and is immediately followed by a reset from RoadRunner in frame #9. Based on this trace, it appears that RoadRunner reset the connection for some reason.

Listing 4 shows a trace taken from another Red Hat system (Spy1) connected to the same switch as RoadRunner. The switch was configured to mirror all traffic to/from RoadRunner to the switch port connected to Spy1. Except for the timestamps, which are not in sync, the two traces are the same through frame #7. At that point, Spy1's frame #8 shows RoadRunner sending an acknowledgement frame to Top and Top sending the reset to RoadRunner in frame #9. These frames do not show up in Top's trace. The traces get back in sync at frames #10 and #11, which shows Top's retransmission frame and the reset from Roadrunner. Based on this trace, it appears that Top has reset the connection.

So, at first glance, it appears that Top has reset the connection. There are a few unusual things about this trace. First, frame #8 is sent to and frame #9 comes from a different MAC address than all the other frames. Second, the time to live (TTL) value in frame #9 is completely different from the TTL value in all the other frames that came from Top. The difference is so large (192) that it is not reasonable to assume that the packet just took another path. Third, the value in frame #9 is 255, which indicates that it did not pass through any routers (255 is the maximum TTL value). Clearly, this should not be possible. Finally, why would Top reset the connection in frame #9 and then send data in frame #10? Top should not be sending any data after resetting the connection.

It also appears that RoadRunner sent an acknowledgement packet (frame #8) to the wrong host (the different MAC address) and that that host responded with a reset (frame #9). RoadRunner then closed its connection and, when Top retransmitted frame #7 with frame #10, RoadRunner responded with a reset (frame #11) because it had already closed its connection. The trace taken on Top sees only frames #7, #10, and #11.

We still need to know why RoadRunner is sending frame #8 to the wrong MAC address. The key is to look at the entire trace, not just the connection between RoadRunner and Top. Between frame #7 and #8, RoadRunner sends an ARP request looking for the MAC address of 192.168.1.99 (see Figure 5). This is the router that connects RoadRunner's subnet to Top's subnet. A reply is received from 0:50:bf:2e:1e:56. This is the MAC address used in frame #8. After frame #9, there is another reply -- this time with the address 0:e0:1e:60:8e:c7. This is the address used in the rest of the frames.

What's happening is that, on RoadRunner, the ARP cache entry for the router times out and is purged (by default, Microsoft Windows 2000 will do this every 10 minutes). The next time a packet needs to be sent to the router, an ARP request is sent; as soon as an answer is received, the ARP cache is updated and the packet sent. When the second ARP reply is received, the ARP cache is again updated but it is too late to save the connection (because of the reset). The next connection, however, will use the updated ARP cache and work fine -- for a while.

At this point we had to identify what host had a MAC address of 0:50:bf:2e:1e:56. It turned out to belong to a firewall, which explained why it was responding with the reset. It had no knowledge of a connection between RoadRunner and Top so it shut down the connection. Unfortunately, we could not determine why it was responding to the ARP for 192.168.1.99 in the first place. To work around the problem, I set a static entry for the router in the ARP cache of RoadRunner. Setting a static ARP cache entry should be considered only a stopgap measure. Once the firewall is fixed, the entry should be deleted; otherwise, if the router NIC card were changed, we would be unable to communicate with it.

It is very important to get a complete trace of all frames when debugging a problem. If I had just done a "tcpdump host RoadRunner", I would not have seen the MAC addresses and would not have known that the packets were sent to different hosts. I also had to remove the "host RoadRunner" argument in order to see the ARP packets and learn what was really happening. Finally, getting a trace from both ends of the network at the same time can significantly cut down on finger pointing. With just the trace from Top, the problem appeared to be on RoadRunner. With just the trace from Spy1, we could easily have assumed that the problem was with Top. With both traces, however, we could see the inconsistencies in Spy1's trace.

Problem #2: One-Way Connections/Pings

In the second example, connections between hosts on the same subnet (Daffy at 192.168.1.2 and RoadRunner at 192.168.1.100) can sometimes be made. Daffy is running Red Hat 7.2, and RoadRunner is still running Microsoft Windows 2000 Server. After careful questioning, I learn that pings from Daffy to RoadRunner usually succeed, and pings from RoadRunner to Daffy usually fail. Again, tcpdump on Spy1 shows what is happening.

In Listing 6, Daffy has initiated the ping. Note that in frame #1 Daffy has sent an ARP request looking for RoadRunner; RoadRunner replies in frame #2, and Daffy sends a ping in frame #3 and gets a reply in frame #4. In frame #5, we get another ARP reply claiming to be RoadRunner but with a different Ethernet MAC address. Daffy however, ignores this reply and continues to use the address from the first reply (see frames #8 and #10). Note also that RoadRunner never sends an ARP query looking for Daffy's address; it has already learned the address from the packets that Daffy has sent.

The trace in Listing 7 starts out very much like the trace in Listing 6. Frame #1 shows RoadRunner sending an ARP query for Daffy, and Daffy responds in frame #2. Frame #3 contains RoadRunner's ping, and frame #4 contains Daffy's response. Frame #5 contains another ARP reply, but Microsoft Windows 2000 immediately starts using the MAC address in the new reply in frames #6, #7, and #8, which now go unanswered. This is a difference in how Microsoft Windows and Linux deal with gratuitous ARPs. If the ARP reply in frame #5 had arrived first, Daffy would not have been able to make a connection, but RoadRoader would have (at least until its ARP cache timed out (see problem #1)).

Again, a static ARP entry on RoadRunner resolved the problem but we were still wondering what sent the second ARP reply. We were unable to identify the second MAC address 0:50:bf:2e:1e:56, but another trace taken over a much longer time frame turned up some ARP frames looking for hosts on a completely different subnet, see Listing 8.

When we looked at the 172.16.2.0 network for the second MAC address, it was identified as a router. A check of the router showed that it thought it had a route back to our 192.168.1.0 network. Someone had bridged the two networks 172.16.2.0 and 192.168.1.0, and the router was doing proxy ARPs. The route was deleted and the proxy ARPs stopped; however, it took most of a day to trace where the two networks had been accidentally bridged and to break the connection.

Summary

The first thing to remember if you are having communications problems is to check the ARP cache of both systems (and any routers). The process is quick and easy and can prevent hours spent looking for the wrong thing. However, as we saw in example 1, the ARP cache on RoadRunner was wrong for only a very brief time. So, just because the cache is correct when you check doesn't mean it was correct when the packet was sent. Nevertheless, if it is wrong when you look, find out why.

Second, if possible, get simultaneous traces from each host participating in the connection. If you can't get a trace from both ends, remember that what you are seeing in the trace is not necessarily what the other side of the connection is seeing.

Third, filtering packets in a trace is usually necessary because of the volume of packets on a network, but sometimes the key packets don't show up in the trace because they have been filtered out. Be prepared to change the filter or, if that doesn't help, eliminate the filter altogether.

Fourth, pay attention to all the fields in the trace -- not just the IP addresses and TCP flags. Some of the secondary fields, like the TTL field, can provide critical clues to diagnose the problem.

Noah Davids (noah_davids@stratus.com) has worked as a LAN technical support specialist for a mid-sized computer company for the last 10 years. He has published numerous articles on LAN programming and troubleshooting. He has CNX, Network+, and MCSE certifications as well as a Master of Science in Computer Science from Arizona State University.