Don’t assume anything when troubleshooting! (John Modlin)

 I was working at a large network heterogeneous environment and started working on a problem of scanners at field offices being unable to transfer documents across the WAN. Working on this problem led me quite down the rabbit hole, a black hole to be more specific.

The field techs had already changed out scanners, but the site continued to be intermittently unable to send documents across the WAN. Some documents transferred, some didn’t. After checking permissions and general settings on the scanner, I started looking at the network path.
I did a traceroute to the scanner to check all the hops. Things looked OK on the surface with normal responses coming back from each hop. Taking it a step further, I started checking each hop for its capability to pass fully loaded frames. To do this I used the following:Ping x.x.x.x -f -l 1472 (-f do not fragment, -l set payload size to 1472). Assuming the standard network MTU used is 1500 bytes, then 1472 would be the total amount of data that each hop should pass. This is because the Ethernet headers including 20 bytes for the IP header and 8 bytes for the ICMP header take up 28 bytes of the 1500 available, leaving you 1472 bytes for actual payload.As I started testing each hop, I hit one router in-between the edge and my location which would not pass 1472 bytes. I reduced the payload to 1460, Ping x.x.x.x -f -l 1460, and it passed. Moving it around until I found the cutoff at 1468 bytes, where 1468 bytes would pass, but 1469 would not. Taking a capture of the traffic at the scanner, I could see the scanner was setting the Do Not Fragment Bit in the IP Header by default, so when the scanner was sending fully loaded frames, 1469-1472 bytes, the packets were being dropped at that hop, effectively black-holing the traffic. If the scanner happened to send frames that were not fully loaded, less than 1469 bytes, the traffic would pass through the router, and the document would be received.
This revelation led to troubleshooting differently other applications that were having intermittent problems at field offices, by capturing traffic, checking the “Do Not Fragment” bit was set in the IP Header, and then checking the traffic path for black holes. Defined by RFC 791, in your capture file using Wireshark, you can look for ICMP Type 3 Code 4 errors, which indicate Black Hole detection also along a path. By correlating the issues to a geographical map, I found the hand-off between two telco companies in a large MPLS was dropping 4 bytes of traffic as it traversed from one telco to the other. This ended up being an expensive month-long fix of upgrading telco equipment at many locations, while resolving countless issues 40,000+ users were experiencing intermittently.

The moral of the story?

Telco’s main job is to provide transport.

Don’t assume anything when troubleshooting!

Wireshark Interfaces and File List Tip

 Networkdatapedia.com has been asking for material that focus on knowing your network and/or knowing your tools.

Sounds pretty simple but trust me this is anything but simple or obvious. When you use the same tool and it becomes your ‘favorite’ or ‘go to tool’ you might be resistant to trying new tools.

Great example is back in the early 90’s when I was using Network General Sniffer products. I was getting very comfortable with it and was actually solving issues with no training. Through the years I heard of Lanalyzer, Capsa, Cinco, NetXRAY, Observer, Microsoft Network monitor, Protocol Inspector and of course Ethereal (aka Wireshark) as well as ton, I’ve probably forgotten.

I remember showing my Sniffer sales person Microsoft Network Analyzer and Ethereal explaining some if the features I liked. His response is “don’t waste your time on that free stuff”, followed up with “how good can it possibly be when its free”.

I soon figured out that every tool has its pluses and minuses and figuring out what works best for you is the toughest part. When you find that tool that you always reach for first, you need to take the time to learn all the nuances and what features new versions may bring – or break ;)

In this video I spend a few minutes showing you how to clear your ‘most recently used file list’ and how to hide network interfaces you won’t be using. Its important to note that hiding the interfaces does not delete, disable or affect them directly.

For example, if you hide your WiFi adapter in Wireshark, you can still use it to surf, ping, etc.. it will just be hidden from the available adapter list in Wireshark.

Enjoy.

Determining ARP Refresh Rate With Wireshark

 

There have been more than a few times where I had to illustrate that ARP was an issue.

When a suspected ARP issue, or you just need to understand how often a device ARP’s, or you need a good challenge when using your favorite packet analyzer, this is a great exercise.

There are 2 basic goals to this video:

1. Always learn how to configure your tools for the task at hand. In this specific example I turn off my colors, disable my bytes view and configure my time format.

2. Protocol analysis is largely an exercise in pattern recognition, using the correct display and filters allow you to see that pattern. Even if you are just learning, this is a great way to figure things out.

In this specific example, I was chatting with someone about ARP timeouts and mentioned that I only had up to Windows XP figured out and documented, then he asked “what about windows 8 and 10?” to which I responded, “I’ll show you how to do it so you can figure it out yourself.

Now you can figure out the ARP behavior of any device regardless if it’s a computer, appliance IOT device, etc..

In this video, I measure how often my windows 8 laptop refreshed its ARP cache while pinging my default gateway. Funny thing, as soon as we figure it out – which took about 5 minutes because we had to wait for the packets and ARP refresh, he asked “Is it different when you use TCP?”. I laughed and said “well you know how to do it, figure it out”

Tony Fortunato

Sr Network Performance Specialist

The Technology Firm

https://www.thetechfirm.com

Getting things to work better - bit by bit-

DNS Client Issues

 In this video I wanted to show you an issue that I encounter quite often. DNS is one of those protocols we all take for granted and most people believe that if its working, you can’t do much to tune it.

There are many things you can do to improve DNS performance. One of the more common techniques is to configure a device as a local DNS server, cache or relay for those scenarios where you might have slow internet connections.

I want to focus on the client configuration. I’ve seen DNS server entries (manual or DHCP assigned) that are problematic. For example, DNS servers that no longer exist, typos, slow DNS servers or DNS servers that are located on slow links or paths.In this case I highlighted what your packet trace will look like when you attempt to use a device as a DNS server that is not a DNS server. I also explain why the ICMP packets are important in this process.

Big take away is to review your DNS or any name server configurations every so often to ensure there isn’t any issues.

.