IOS Troubleshooting Tools

Nervous

What’s more nerve-wrecking: Having your company CEO breathing down your neck asking why the network is down, or your exam proctor telling you that you only have 30 minutes left on your CCIE Lab Exam?  I somehow suspect that the answer is that they’re about the same.  I’m lucky enough to get the best of both worlds, having a company CEO breathing down my neck asking when I’m going to get my CCIE.  Ok, not really… most of the pressure to get my CCIE is really just coming from within.  (Seriously, I work for a great Dude.  He doesn’t even read this blog, so you know I must be telling the truth!)  🙂

Still, one of the most important skills you’ll develop as a Network Engineer, whether your company network just went down, or you’re trying to pass your CCIE exam, is the ability to troubleshoot.  Today, we’ll focus on some of the troubleshooting tools that are built right into IOS.  Here are today’s exam objectives:

1.3   Network troubleshooting
1.3.a   Use IOS troubleshooting tools
1.3.a (i)   debug, conditional debug
1.3.a (ii)   ping, traceroute with extended options
1.3.a (iii)   Embedded packet capture
1.3.a (iv)   Performance monitor

Debug and Conditional Debug

It’s something we learn as CCNA candidates, but still, it bears repeating: be careful with debugging commands!  Many Engineers, myself included, can tell you all about the time when they crashed a router or switch simply by turning on a debug command that ended up crushing their box.  If you can run a debugging command in a lab environment, it very well could give you an idea of just how verbose the debug output will be, and might help you know if you need to tune your debugs before turning them on.

Before we go into specifics on debugging commands, know that you can find documentation on pretty much every possible debug command here: http://www.cisco.com/c/en/us/support/ios-nx-os-software/ios-15-4m-t/products-command-reference-list.html

Conditional debugging gives you the ability to specify certain conditions that must be met to trigger a debugging message, such as an interface, IP address, MAC address, or application.  As you might imagine, this can help filter your debugging greatly!  To turn them on, simply use the privileged exec mode debug condition condition command.  For example, to debug only on interface Gig 0/0 you would use the debug condition interface Gig 0/0 command.

Cisco routers will log messages to the console by default – whether you’re actually connected to the console or not.  As a general practice, you may wish to set the no logging console command by default on all devices, and simply instruct Engineers to turn it on as needed if they do happen to connect to a device console port.  You can instruct a router to log messages to the internal logging buffer or to a syslog server with the logging buffered and logging ip-address commands, respectively.  You may also wish to add millisecond timestamps to your debug messages with the service timestamps debug datetime msec command.  (Note: for other non-debugging messages, you may also wish to turn on the service timestamps log datetime msec command)

Finally, it is possible to debug every ip packet passing through a router.  To do so, you have to first disable CEF so that each packet is process-switched.  This step alone can potentially have disastrous consequences in production, so be sure to have a backout plan ready, and that you are in a maintenance window if a router is handling business-critical traffic.  Then, you can disable CEF with the interface no ip route-cache command.  If possible, you’ll also want to use an access-list to filter interesting traffic, and keep debugging buffers more manageable, and more importantly, minimize impact to the CPU, which is already taking a huge hit because of turning off CEF.

Ping and Traceroute With Extended Options

I find that most Engineers with some time in the business know that you can type ping or traceroute at a CLI and simply hit enter rather than specify an address, and you’ll be presented with an array of options you can set.  There’s really no substitute for experience here, so I’m just going to comment on a couple of these options and tell you that you should by no means take this as an exhaustive list.  Please do yourself a favor and play with these tools a bit on your own – you can gain some very valuable familiarity with these tools with very minimal time investment!  To get you started, I’ll just point out a few of them that might be valuable.

First, you can set the ToS when using the ping command.  You can compare response times for different ToS values in order to troubleshoot or validate QoS settings.  Second, you can set the DF bit in the pings that are being sent.  This can be valuable if you suspect fragmentation issues.  If you suspect that fragmentation issues could be related to packet sizes, you can also set the DF bit in conjunction with the option to sweep a range of sizes of packets.  You can leave the DF bit on to monitor ICMP unreachable messages, or turn it off to validate that your path is indeed fragmenting packets.  (Note: it is preferable to have hosts use PMTUD and handle fragmentation themselves… see the previous post about IP Header Operations)

Embedded Packet Capture

THANK YOU CISCO!!!  When I first heard that Cisco was finally developing the ability to capture packets right from within IOS, it was a massive relief!  To tell you the truth though, many Engineers either aren’t aware of the fact that this can be done, or simply haven’t taken the time to learn how.  In other words: This is a great skill to have, and it can really help give you an edge over fellow Engineers that don’t know how to grab traffic coming right off of a router!

First, you have to create a capture buffer, which is the place where you will store the captured traffic.  Do this with the privileged exec mode monitor capture buffer buffername command.  Note that the capture buffer itself allows you to configure options like filters, circular vs linear, max-size or export format.

Second, you define a capture point, which, in most cases will be interface CEF-based.  Create this with the monitor capture point ip cef cappointname interface direction.  Note that you can capture ingress, egress or both when defining the capture point.

Next, tell the capture point that it is to put captured traffic into the capture buffer you created with the monitor capture point associate cappointname buffername command.

Lastly, tell the capture point to begin capturing traffic with the monitor capture point start cappointname command.  That’s it – you’re capturing traffic!  When enough traffic has been generated and you’re ready to collect the sample, stop the capture with the monitor capture point stop cappointname command.

Here’s a live example:

____________________________________________________________

2801-VoipFirewall#monitor capture buffer HAT-Buffer circular
2801-VoipFirewall#!
2801-VoipFirewall#$ture point ip cef HAT-CapPoint FastEthernet 0/0 both
2801-VoipFirewall#!
2801-VoipFirewall#monitor capture point associate HAT-CapPoint HAT-Buffer
2801-VoipFirewall#!
2801-VoipFirewall#monitor capture point start HAT-CapPoint
2801-VoipFirewall#!
2801-VoipFirewall#monitor capture point stop HAT-CapPoint

_____________________________________________________________

Now that you have your capture, you can actually view the capture right in IOS.  Obviously, this is not quite going to have the full feature set that a tool like Wireshark is going to have, but if a cursory glance at these packets is going to be enough, then you can use the show monitor capture buffer buffername dump command, and there you go – it’s Wireshark Lite!  (Ok, Wireshark VERY Lite!)

__________________________________________________

2801-VoipFirewall#show monitor capture buffer HAT-Buffer dump
15:25:54.539 EST Dec 26 2014 : IPv4 LES CEF    : Fa0/0 None

6BDF8610:          001759E2 AEE25823 8C6BE451      ..Yb.bX#.kdQ
6BDF8620: 08004520 00287306 40007106 3FD288B5  ..E .(s.@.q.?R.5
6BDF8630: C3240A00 00FE205B 00163392 D7C18291  C$…~ [..3.WA..
6BDF8640: 3F7E5010 FEA06C87 00000000 00000000  ?~P.~ l………
6BDF8650: 00                                   .

15:25:54.539 EST Dec 26 2014 : IPv4 LES CEF    : Fa0/0 None

6BDF8610:          001759E2 AEE25823 8C6BE451      ..Yb.bX#.kdQ
6BDF8620: 08004520 00287307 40007106 3FD188B5  ..E .(s.@.q.?Q.5
6BDF8630: C3240A00 00FE205B 00163392 D7C18291  C$…~ [..3.WA..
6BDF8640: 3FE65010 FE386C87 00000000 00000000  ?fP.~8l………
6BDF8650: 00                                   .

15:25:54.543 EST Dec 26 2014 : IPv4 LES CEF    : Fa0/0 None

6BDF8610:          001759E2 AEE25823 8C6BE451      ..Yb.bX#.kdQ
6BDF8620: 08004520 00287308 40007106 3FD088B5  ..E .(s.@.q.?P.5
6BDF8630: C3240A00 00FE205B 00163392 D7C18291  C$…~ [..3.WA..
6BDF8640: 404E5010 FDD06C87 00000000 00000000  @NP.}Pl………
6BDF8650: 00                                   .

(Output Omitted for Brevity)

___________________________________________________________

More often than not though, you’ll likely need to export this information into a PCAP file so that you can indeed view it in a protocol analyzer like Wireshark.  To do so we export the capture with the monitor capture buffer buffername export URL command.

Performance Monitor

Performance Monitor is a NetFlow-based tool for IOS.  Yes, this does mean that you need a pretty good handle on what NetFlow is and how to configure it in order to use IOS performance monitor.  NetFlow is, again, a topic unto itself in the CCIE world, so I’m going to save that overview for when we get to it.  In the meantime, if you don’t have any experience with NetFlow, GET SOME!  The visibility that NetFlow gives you into your environment, the types of traffic being sent, the top talkers in your environment, bottlenecks, etc. are all of immense value.

Performance monitor is awesome, especially for those in the voice and video fields!  A list of the performance data that can be gathered includes:

•IP Packet Count
•IP TTL
•IP TTL minimum
•IP TTL maximum
•Flow to Interface Mapping
•IP Flow destination address and port, source address and port, and protocol
•RTP Synchronization Source (SSRC)
•IP Octets Count
•Media Stream Packet Count
•Media Stream Octect Count
•Media Byte Rate
•Media Byte Count
•Media Packet Rate
•Media Packet Loss Count
•Media Packet Loss Rate
•Packets Expected Count
•Measured Rate
•Media Loss Event Count
•Round Trip Time (RTT)
•Interarrival Jitter (RFC3550) max
•Interarrival Jitter (RFC3550) min 2
•Interarrival Jitter (RFC3550) mean
•Media Rate Variation
•Monitor Event
•Media Error
•Media Stop
•IP Byte Count
•IP Byte Rate
•IP Source Mask
•IP Destination Mask
•Epoch of A Monitoring Interval
•Packet Forwarding Status
•Packet Drops
•DSCP and IPv6 Traffic Class

That list was obtained directly from the Cisco.com configuration guide.  Honestly, I it makes the most sense to simply explain what Performance Monitor is and what it does and link you directly to that configuration guide, rather than type out all of the commands here in a blog, so please do check out the config guide here:  http://www.cisco.com/c/en/us/td/docs/ios/media_monitoring/configuration/guide/15_1m_and_t/mm_15_1m_and_t/mm_pasv_mon.html

I’m going to call this a wrap for the first major exam blueprint topic, Network Principles.  Admittedly, Cisco does have the following topic all about packet captures:

1.3.c   Interpret packet capture
1.3.c (i)   Using Wireshark trace analyzer
1.3.c (ii)   Using IOS embedded packet capture
I’ll use the same approach and explanation here that I’ve used before: trying to write a post about “how to interpret Packet Captures” is too broad a topic.  So again, I don’t really feel like it makes sense to talk about this topic broadly when we’ll surely cross paths with some very specific examples.  It’s good to know that Cisco is essentially saying “Know that these might be some of the tools that we’re asking you to be familiar with for the CCIE Written exam”, but I’m sure Cisco isn’t going to write hardcore Wireshark-based questions, as they’re really not interested in testing the depth of our knowledge of someone else’s product; I feel it’s safe to assume that they really only want to know that we can use that product to prove the depth of our knowledge with Cisco routers and switches.

So… congratulations!  We’re done with the first of the 6 major exam objectives.  Next up is Layer 2 Technologies, and I can hardly wait!  Thanks for joining for our first series on network principles!

1 Comment

  1. cf53a703

    I really like those short posts about cisco certifications.
    It is well written, easy to read and I always learn, at least, one thingy !
    Thank you for sharing 🙂

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *