"Get Off Your Butt and Go for A Walk!"

 One of the key points I occasionally make when teaching, presenting or writing, is that sometimes you need to stop looking at your red light/green light monitoring system, stop pinging, and please stop rebooting equipment! Nothing replaces getting out there and physically looking at the problem.

In my 30+ years of troubleshooting, I can not tell you how many times the problem looked obvious when I physically got out there to investigate the problem. I've seen a ladder leaned up against a breaker, cables pressed up against power buttons, equipment hanging in a bag that was powered on causing the bag to melt around it, excess cabling wrapped around an arc welder, cables pinched in doors, cables spliced and wrapped in electrical tape, trust me, the list goes on.

In this case, some wireless equipment had a poor signal and the technicians were rebooting it, checking for noise, etc. I then found out that the signal went out of wack after a wind storm, so I climbed up (why I'm huffing and puffing a bit) and sure enough the bolts on the bracket that stabilizes the tower had come loose.

Before you comment that "sometimes its not possible to go out everywhere", i simply counter that argument with "use your tech" use Zoom, facetime, or ask someone there to take photos/videos. Trust me, its better than nothing.

SNMP DISCOVERY & INVENTORY

 The most common question I get asked is “When troubleshooting, where do you start?”.

Other than the "client interview", the answer is simple, “start with an inventory”. When I say “Inventory” it is an all-encompassing term. It covers the physical equipment, connectivity, layer 2, 3, and any other layer it takes to get from point a to point b. Many times we can get a 'visual' from the client interview but other times we need more info, or have to confirm what they believe is out there.

I always say, “We can't fix what we can't see”. If the client already has documentation, great, let's review it with respect to the issue at hand. I am not surprised how many times I am presented with out-of-date documentation. In some cases, entire sites are omitted, not added or equipment upgrades are not reflected. I never consider this scenario a negative one and explain “all we have to do is update what we have”.

It's important that this does not turn into a “who didn’t update the documentation and why witch hunt”, let's just get the documentation updated and move on. This is always the interesting part of the documentation exercise; watching the client argue with colleagues about “who moved that”, “I thought we replaced that box”, and the ever-popular “that’s an old config”. Many times, we actually find and fix the problem as we go through the documentation exercise.

When I talk about the documentation process, the most common response I get is “I don’t have time to do that”, and I respond with “how much time do you have to update the documentation during an outage or troubleshooting?”

The best place to start with is at the physical layer, at the clients' desk and the destination could be a server/application you use. The client the best place to start because you can verify the cabling, switch configuration, etc easier. Then work your way up the physical layer, to layer2, then 3, etc. If you get stuck or aren’t sure, feel free to reach out to various people/departments for clarification.

You can literally start with a piece of blank paper, Powerpoint, Draw, Impress, heck, even Paint will do. As you build the maps, and more people see the value, you can consider a different application and how to share the information.

In this video, I was doing a physical inventory at a remote office and there was no documentation, so I grabbed a couple of my Cisco switches, enabled SNMP, swapped out the client’s switch, and ran a discover with my Optiview XG. Then I moved on to the next closet.

In this case, I took photos before I started, and documented all the current switch connections. Then I had to come in when no one was in the office, run my discovery and then put everything back. The whole process took an hour and about another 30 minutes putting the documentation together using Powerpoint.

“Its Dead Jim” Looking into a SSID Issue With My NetAlly Aircheck G2

 When troubleshooting WiFi issues, the last thing I want to do is worry about having issues with my laptop and trying to figure out how to do specific things.

For example, if you have 5 access points with the same SSID, can you connect to a specific BSSID using your Windows, Linux, Android or MAC for testing?

I’ve gone through various hacks, special drivers and all sorts of technical gymnastics to get my laptop to do specific WiFi troubleshooting tasks. That might be fine for learning and playing in the lab but not very practically when you have to troubleshoot on the fly.

In this scenario, I wanted to verify that a specific SSID was working on two access points. I ask this question all the time “are all your access points supporting your SSID’s?”. I ask because it is hard to verify if this is the case.

In some environments, you can go to your wireless controller, or to specific access points to confirm who is connected to it. But then you need the mac address of the device and in some cases, the support staff doesn’t have access to the controller or access point.

This is when I prefer to use my NetAlly AirCheck G2 since I can use it to connect to specific access points and run tests. In this video, I show you how I tested and proved if both access points were working properly.

Wireshark Windows Vs Linux

 I have always enjoyed testing tools in my lab. For those of you who have followed me over the years, know that I always say that you should ‘know your tools’. I know this sounds obvious but trust me, it is anything but obvious.

For example, you have a relatively new laptop with 8 GB or RAM, i7 processor and 1 Gb Ethernet adapter. So, you would think that you should be able to capture traffic up to 1 Gbps, right? Wrong!!

In my last article, “Wireshark, Microsoft pktmon, packet testing” (https://youtu.be/pZtWAwiH7lk), I compared various command line and GUI tools and how efficient they were in capturing packets. Thank you for all the feedback and I thought I would use one of the suggestions for this next article.

The point of the article is that you need to test your tools using various packet sizes and loads to determine when it will drop packets. The secondary goal is to show you how I tested my tools so you can use a similar methodology for your testing.

In this article, I compared the Wireshark GUI performance on a Linux and Windows. I used the same laptop for both tests and neither was within a VM. The only result that shocked me was that Windows barely outperformed Linux in the 10%, 64 Byte test. See, I learned something as well.

Packet Capture Accuracy

 This is a great example of ‘knowing your tool’ or ‘tool calibration’. Regardless of what you want to call it, it is important that you test your tools to ensure their accuracy.

In most cases analysts are turning to their laptops with either Windows, Linux or MAC OS as a tool of choice since laptops are might lighter, have increased battery life, more powerful and convenient to store results.

Unfortunately, convenience might come at a price. A good example would be to ask yourself if your laptop capture packets at various rates and packet sizes. Along with this question, should be a follow up to investigate how accurate your tool of choice is.

I see many people using USB/USB-c 1 Gb Ethernet adapters with their laptops which might be an issue if you just want to capture packets but might bite you when it comes to packet.