Traceroute: How to Diagnose & Identity Network Issues

Traceroute is a tool that helps you see the path your data takes across the network, from your device to the destination. The tool 'traces' in real time the 'route' data packets take on an IP network. This is useful for diagnosing network issues and pinpointing where data packets are getting lost.

How traceroute works

When you run a traceroute, your computer sends out packets of data with a small twist. These packets have something called TTL, which stands for Time To Live. TTL is like a countdown timer for how many hops—or routers—the packet can pass through before it gets dropped. 

When the packet's TTL hits zero, the router signals that it can't send it any further and drops it. But before it drops the packet, the router sends back a message to your computer, basically signaling that it was the last stop. The traceroute tool captures this message and records the router's IP address.

Let’s break it down with an example. Imagine you're trying to reach google.com. Your computer sends out a packet with a TTL of 1. This packet reaches the first router, TTL drops to zero, and the router sends back a "Time Exceeded" message. Traceroute logs this first router's IP address. 

Traceroute then sends out another packet, this time with a TTL of 2. This packet makes it through the first router and then gets dropped at the second, which sends back a similar message. Traceroute now logs the second router's IP address. This process repeats, increasing the TTL by one each time, until the packet finally reaches google.com, or whatever your target might be.

If you traceroute to google.com from your office, you might see that your packet hops from your local router to your ISP, then maybe through some intermediate routers, and finally to a Google server. Each hop is displayed with its own IP address along with the round-trip time, showing how quickly your packets are traveling.

This method helps you pinpoint where delays or disruptions are happening. Say the fourth router in the sequence takes much longer to respond than the others. This slowdown could indicate network congestion or a problem at that specific hop. 

Remember, though, some routers may not respond to traceroute packets. They simply drop them without sending back a message. When this happens, you might see asterisks or timeouts in your traceroute output, but you can often still glean useful info from the other hops.

The role of ICMP packets in traceroute

The Internet Control Method Protocol (ICMP) is a set of rules network deviçes use to çommunicate data transmission errors. ICMP packets in traceroute provide a step-by-step view of the network path. They help reveal each hop's identity and round-trip time, making it easier to troubleshoot network issues.

Here is how it works. Traceroute sends a series of ICMP Echo Request messages to the destination. Each packet has a different Time-To-Live (TTL) value.

When the packet reaches the first router (hop) in the path, the TTL expires. The router then sends back an ICMP Time Exceeded message. This response helps identify the first hop. 

Next, traceroute sends another packet. This process repeats, incrementing the TTL by one for each new packet.

Think of it like peeling an onion, layer by layer. At each hop, the TTL expires, and an ICMP Time Exceeded message returns to the source. By examining these ICMP responses, traceroute builds a map of the network path.

If you're tracing from New York to Los Angeles, the first few ICMP responses might come from routers in New York, then ones in Chicago, and finally in Los Angeles. Each response includes the IP address and sometimes the hostname of the router. This information helps diagnose where delays or packet losses occur.

In enterprise networks, you might see internal routers with private IP addresses in the responses. These routers manage internal traffic but behave similarly with ICMP packets. By interpreting these ICMP messages, you can pinpoint network issues within the corporate infrastructure.

Sometimes, a hop may not respond. This could be due to firewalls or security policies blocking ICMP packets. In such cases, traceroute might show an asterisk (*) instead of an IP address. This doesn't necessarily mean a problem; it could simply be a security measure.

Role of TTL (Time to Live) value in Traceroute

The TTL, or Time to Live, is a field in the IP packet header that dictates how many hops, or intermediate devices like routers, a packet can traverse before being discarded. 

When you initiate a traceroute, the tool sends out a series of packets with incrementing TTL values, starting from 1. Each time a packet hits a router, the TTL value is decremented by 1. If the TTL reaches zero, the router drops the packet and sends back a "Time Exceeded" message to me.

Suppose you perform a traceroute to www.example.com. When the first packet reaches the first router on the path, which decrements the TTL to 0, it discards the packet, and sends back the "Time Exceeded" message. This lets the traceroute tool know the IP address of the first router. 

Next, the tool sends a packet on. This packet passes through the first router and reaches the second router, where the TTL again hits zero. The second router then sends back another "Time Exceeded" message. This process continues, with the TTL value incrementing by one for each new packet sent. Each returned message provides details about a subsequent hop in the network path until the destination is reached.

This method is particularly useful in corporate networks, where multiple routers and firewalls might be in use. For instance, if you are troubleshooting a connectivity issue to a server located in a different office, the traceroute helps pinpoint where the problem occurs. 

Is the packet getting stuck at the corporate firewall? Or is it perhaps an intermediary router that’s causing the delay? By examining the sequence of returned IP addresses and the associated response times, you can identify the problematic hop.

Tying all this together, the TTL value is like a breadcrumb trail leading me through the network, hop by hop. Without it, traceroute wouldn't be able to map the path efficiently. It's this incremental increase in TTL values and the corresponding feedback from routers that make traceroute such an indispensable tool for network diagnostics.

Packet sending in traceroute

Let's imagine you're tracing the route to a server at 192.168.1.10. Your first packet reaches your company's firewall, and promptly gets dropped, sending back a "time exceeded" message from 192.168.1.1. The next packet travels to the firewall then to your core switch at 192.168.1.2, where it’s dropped again. 

By incrementing the TTL, the path unfolds one hop at a time until the packet finally reaches the server, effectively mapping the route across our network.

Now, the packets sent can be either ICMP Echo Requests, UDP packets, or even TCP SYN packets. The choice depends on the operating system and traceroute implementation. 

For instance, the Unix-based traceroute often sends out UDP packets, while the Windows version tends to use ICMP. This can be particularly useful in a corporate network where different firewall rules might block specific types of packets. If you are on a Unix machine, you might send UDP packets with a destination port that incrementally increases with each probe. 

On a Windows system, the ICMP Echo Requests will likely suffice. Each packet type has its pros and cons depending on network configurations and firewall rules.

On top of that, the payload of these packets contains unique identifiers, usually incrementing sequence numbers. This ensures that traceroute can match the returning "time exceeded" messages to the original packet sent. It’s like sending out labeled breadcrumbs and watching which ones get picked up and returned. 

This allows for accurate mapping and timing of each hop, helping to diagnose network issues or simply understand the route data takes through the corporate network.

Complex networks will usually employ MPLS (Multiprotocol Label Switching) which can sometimes mask the hops from the traceroute utility. MPLS can route packets through predetermined paths and make it tricky to accurately map the entire path. 

It might seem like the packets are teleporting from one point to another. Therefore, understanding how traceroute packets navigate these complexities can be crucial for accurate network diagnostics.

Intermediate hop responses in traceroute

When you run a traceroute, it doesn't just show you the final destination. It also gives you a peek at every step your packet takes along the way. These stops are called intermediate hops. Each hop represents a router or other network device your data travels through to reach its end point.

When looking at a traceroute output, each line usually starts with a number, which is the hop count. This tells you how many hops away this particular device is from your starting point. 

Right after the number, you’ll typically see the IP address or hostname of the hop. Sometimes, it’s just an IP address, but other times, it might resolve to a human-readable name. For example, you might see something like "4 192.0.2.1 (router.example.com)."

Following the IP address or hostname, you'll find a series of round-trip times. These times show how long it takes for a packet to go from your machine to the hop and back. Usually, you'll see three times because traceroute sends three packets to each hop by default. 

For example, you might see "20 ms 21 ms 22 ms." This gives you an idea of the latency at each step in the journey. If you notice one hop taking considerably longer than the others, that could be a sign of network congestion or a problematic device.

Now, here’s where it gets interesting. Sometimes you’ll see an asterisk (*) instead of a time. This usually means that the hop didn’t respond to the packet. It could be filtering ICMP messages, or it might just be too busy. For instance, your output might look like "5 * * *". If you see this, don't panic. It's not unusual for some corporate networks to have devices configured this way to enhance security.

In some corporate networks, traceroute might show multiple IP addresses for a single hop. This happens due to load-balancing. When routers distribute packets across multiple paths to optimize for speed and efficiency, you might see "6 10.0.0.1 10.0.0.2 10.0.0.3" with different round-trip times for each.

It’s also worth noting that in corporate environments, you might encounter private IP addresses and internal hostnames that won’t resolve in public DNS. For example, you might see an address like "192.168.1.1" or a hostname like "internal-router.local." These are part of the internal network and give you a sense of the journey within the company’s own network infrastructure.

By reading these intermediate hop responses, you can troubleshoot and understand where delays or issues might be occurring in the network. It’s like having a roadmap that shows you not just the destination, but every turn and intersection along the way.

Final destination reach in Traceroute

When running a traceroute in corporate networks, reaching the final destination feels like winning a mini lottery. But it's not always straightforward. Some destinations might have strict security policies. 

Firewalls or routers might be set to drop tracert packets. So instead of the usual stars or "Request timed out," you might see the last few hops responding and then suddenly nothing. This makes it tough to know if you've truly reached the target.

Sometimes, the final hop will show an IP address or a hostname, which can be reassuring. For instance, tracing to your office in another city, the trace might show the IP of the final destination. 

That IP may match the known address for your server in that city, confirming the path was complete. But remember, just reaching the right IP doesn’t mean there are no issues. Latency and packet loss at the final hop can indicate problems even if the trace completes.

Also, don’t be surprised if the final destination shows a private IP address. Many corporate networks use NAT (Network Address Translation). So the actual server IP might be hidden behind a firewall or gateway. The private IP is a clue about the network structure.

When running traceroute in a corporate network, each layer gives you more insight, but the core might still be hidden behind layers of security. And that's okay. Knowing what to expect and how to read those hops can help you diagnose network paths without uncovering every single node.

How to identify network bottlenecks using Traceroute

Using traceroute to identify network bottlenecks feels a bit like detective work. If you notice that your internal application is running slower than usual, for example, traceroute can help pinpoint where the slowdown happens. 

Let’s say you start by running a traceroute from your computer to the application server. First, you open your terminal and type `traceroute application-server.company.com`. 

The output shows a series of hops – each one represents a network device between my computer and the server. Each hop includes the device's IP address and the time it takes to travel to that device and back.

What you are looking for are significant increases in latency. For example, if hops 1-4 all show response times around 10ms, but hop 5 suddenly jumps to 150ms, that's a red flag. It might indicate congestion on the router at hop 5, or it could be a link between hops 4 and 5 that's causing the delay.

Sometimes you see timeouts, represented as asterisks (*), instead of response times. While a few asterisks here and there can be normal, a series of them can indicate trouble. 

For instance, if hops 7-9 all show asterisks, there may be a problem with the network segment or the devices handling those hops. Maybe the firewall is blocking ICMP packets, which can cause traceroute to show timeouts.

When interpreting traceroute results, it's essential to consider that each hop might not always respond to ICMP packets promptly. Sometimes individual device configurations can affect results. Knowing the usual response times for different parts of your network helps you distinguish between a real problem and a false alarm.

While traceroute is a powerful tool, it’s just one part of the toolkit. You can combine it with other diagnostics like ping and network monitoring software for a comprehensive view. For instance, you might use a continuous ping to monitor latency at the problematic hop identified by traceroute to confirm the issue persists over time.

By consistently using traceroute, you become more adept at recognizing patterns and anomalies in our network. This proactive approach helps catch potential issues before they escalate into major outages.

Detecting routing issues using Traceroute

Traceroute is a handy tool for diagnosing routing issues that present as slow network speeds or intermittent connectivity. It's like a map that shows the path your data packets take to reach their destination, hop by hop. Here's how it works:

Traceroute sends packets to a target and records the time taken for each hop along the way. If there's a delay or failure at any point, it pinpoints the problematic node. This visibility can reveal where packet loss or high latency occurs.

For example, say you're trying to connect to a server located in New York from your office in Los Angeles. You run a traceroute command, and it shows the packets making several hops across various ISPs and regions. Suddenly, at the 7th hop in Chicago, you notice a significant delay. This hop could be the culprit causing your slow connection.

Another scenario might involve complete packet loss. Imagine your traceroute results show failures starting at the 5th hop. If the packets don't progress past this point, you might have a dead node or a misconfiguration to investigate.

Traceroute can also help determine if the issue lies within your local network or with an external ISP. Let’s say your internal traceroute to a local server is flawless, but an external trace to Google shows delays at your ISP’s edge router. This indicates the problem might be with the ISP rather than your internal setup.

By using tools like Obkio’s Visual Traceroute, you can visualize these routes in real-time. This automated tool performs continuous traceroutes and displays them in a user-friendly manner. You can share these results with your ISP for quicker resolution.

Remember, while traceroute is powerful, it has limitations. Firewalls or routers may block ICMP packets, giving incomplete information. Despite this, it's still an invaluable first step in diagnosing network routing issues. If used correctly, traceroute helps you pinpoint exactly where your packets are hitting roadblocks.

Assessing network latency using traceroute

Traceroute is most people's go-to tool for assessing network latency in corporate networks. It lets you see the path packets take from your machine to a destination server and the delay they experience at each hop. This is crucial for diagnosing slow network responses and identifying bottlenecks.

You can start by opening your command line interface. On a Windows machine, you type `tracert` followed by the destination address. For instance, `tracert example.com`. 

On Unix-based systems, it’s `traceroute example.com`. The output will list each hop along the way, showing the IP address of each router and the time taken for the packets to get there.

Using an example, suppose you are troubleshooting a slow connection to your web server. You run `tracert webserver.company.com` and see a list of hops. 

The first few hops are internal routers. Their response times are typically under 10 milliseconds, which is normal within many local networks. But what if you see a hop where the time suddenly jumps to 150 milliseconds? That’s a red flag. It could indicate congestion or an overloaded router. 

Sometimes, a hop might show an asterisk (*) instead of a time value. This usually means the router isn't responding to ICMP requests, which is not uncommon in secure corporate networks. If you see multiple asterisks in a row, it might mean there’s a firewall blocking the requests or a serious connectivity issue beyond that point.

Traceroute also helps when comparing different network paths. For instance, if you have multiple ISPs, you might run traceroute to the same destination through each ISP. Differences in the number of hops and latency can guide you on which ISP to route critical traffic through.

Improving network performance using traceroute

As we have established already, traceroute helps in identifying bottlenecks. For instance, when experiencing slow network speeds, running a traceroute from your computer to the target server shows you each hop along the way and the time it takes. If you see a specific hop with a high latency, that's your culprit. 

Say the problem hop is a router in your internal network. You can then take action to reconfigure or replace the faulty equipment, thereby reducing delays.

Another excellent use of traceroute is in pinpointing network outages. Suppose an important service is suddenly unreachable. Traceroute can show you exactly where the connection fails. 

If, for example, the process stops at a particular gateway, you know that the issue lies either at that gateway or just beyond it. This information allows you to escalate the problem to the appropriate team, whether it's your internal IT department or an external ISP.

Traceroute also assists in optimizing routing paths. Corporate networks might have multiple routes to a destination. By comparing traceroute results from different entry points, you can determine the most efficient path. 

For example, you might find that traffic routed through your primary data center experiences less latency compared to a backup site. With this knowledge, you can adjust your routing policies to make sure critical applications use the best possible path.

Interestingly, traceroute can also be a valuable tool for proactive monitoring. You don’t have to wait for an issue to arise before running it. Setting up regular traceroute checks can help you catch potential problems before they escalate. For example, if you notice increasing latency over a period of days or weeks, you can investigate and address the issue before it impacts your users.

Moreover, traceroute is an educational tool for your team. By regularly analyzing traceroute data, your network engineers can get a better understanding of the network's behavior under different conditions. This expertise is crucial when aiming to maintain optimal performance. Think of it as giving your team x-ray vision into the network's inner workings.

In some cases, the results from a traceroute can even be used to negotiate better services from your ISP. If you consistently notice that certain hops outside of your control are causing delays, you can provide this data to your ISP. With concrete evidence, it’s easier to request improvements or compensation, ensuring that you get the quality of service you're paying for.

By incorporating traceroute into your network performance strategy, you'll have a powerful tool at your disposal. It not only illuminates the path your data takes but also empowers you to make informed decisions to keep your network running smoothly.

Running traceroute `tracert` in Windows

Running `tracert` on Windows is handy for tracing the path that data packets take from your computer to a destination, like a server or website. This tool can reveal a lot about the network path and help identify where delays or issues might be occurring.

To use `tracert`, you open the Command Prompt. You can do this by typing `cmd` into the start menu search bar and hitting Enter. Once the Command Prompt is open, type `tracert` followed by the destination IP address or domain name. For example, if you want to trace the route to Google's DNS server, you would type `tracert 8.8.8.8` and press Enter.

After running the command, `tracert` sends out a series of packets with incrementally increasing "time-to-live" (TTL) values. These packets travel through each router or hop along the path to the destination. Each hop that processes the packet sends back a response, which `tracert` displays as a list showing the route and the round-trip time (RTT) for each hop. 

It’s always interesting to see how many hops it takes to reach the destination and where the potential slowdowns are. If one of the hops shows a significantly higher RTT, it might indicate congestion or other issues at that point in the network. For instance, a hop time of 10ms followed by several hops with 100ms can lead you to investigate the network segment associated with that big jump.

But sometimes, you will encounter entries like `* * * Request timed out.` That means a particular router or hop isn't responding to the `tracert` request. This doesn’t always indicate a problem. Some corporate networks or ISPs deliberately block ICMP traffic for security reasons, so it's something to keep in mind.

If you suspect there's an issue within your corporate network, you will run `tracert` to the internal IP addresses of our routers or servers. This helps to narrow down where exactly the issue might be happening. For instance, running `tracert 192.168.1.1` could tell you if there's a problem with your internal gateway.

The `tracert` tool gives you a clearer picture of the network’s performance and helps you identify where packets are getting delayed or dropped. It’s an invaluable tool whenever you need to diagnose and troubleshoot network connectivity issues.

Using traceroute on MacOS/Linux

The Windows operating system identifies traceroute as 'tracert'. On macOS and Linux, though, it’s simply `traceroute`.

To use traceroute to diagnose network issues and understand the path your data takes through different networks on macOS and Linux, first, open up your terminal application. 

On macOS, you can find this in Applications > Utilities > Terminal. On most Linux distributions, it's just called “Terminal” and can usually be found in your system's application menu.

Once you have your terminal open, using `traceroute` is pretty straightforward. Type `traceroute` followed by the hostname or IP address you want to trace. Then, hit enter. 

You’ll start seeing the path your packets take to reach Google. This will list each hop and the time it took to get there. Each hop typically represents a router or other network device, and it shows how many milliseconds (ms) it took for the packet to travel to that hop and back.

Suppose you want to see the route to a specific IP address, say `8.8.8.8` (which is one of Google's public DNS servers). You would type:

Again, each result line corresponds to a hop along the network path. You might see something like this:

Each line starts with the hop number, followed by the hostname (if resolvable) and the IP address. The three time values represent the round-trip times for three separate trials.

There are a few options you can use with `traceroute` to get more specific information. For example, you can specify the number of queries per hop using the `-q` flag. If you wanted to limit it to just one query per hop, you would use:

Note that `traceroute` might need to be installed on some Linux distributions. If it’s not available, you can usually install it via your package manager. On Debian-based systems like Ubuntu, you’d use:

On Red Hat-based systems like CentOS, it’s:

Interpreting Traceroute results

Interpreting traceroute results can seem daunting at first, but it becomes easier with practice. When you run a traceroute in a corporate network, you get a list of hops that show the path your data takes to reach its destination. Each hop represents a device, usually a router or switch, along the way.

Look at each hop's IP address and response time. The IP addresses help you identify the devices and their locations. For instance, the first hop might be your local router, often an internal IP like 192.168.1.1. As the traceroute progresses, you'll see external IPs, which represent devices outside your local network.

If you're diagnosing a slow connection to a remote server, your traceroute output might show normal response times for the first few hops, like 1 ms or 2 ms. Then, you notice a sudden spike, and one hop jumps to 150 ms. 

That spike indicates a potential bottleneck. If the high response time persists through subsequent hops, it likely means the issue is beyond your network, possibly with your ISP or the server's network.

Sometimes, you'll see asterisks (*) instead of response times. This indicates that a hop didn't respond within the timeout period. Occasional asterisks are normal, but if you see consecutive lines of asterisks, it could mean the device isn't responding to ICMP requests. This isn't always a problem; some routers are configured to ignore these requests for security reasons.

Another thing to watch for is IP address repetition. If an IP address appears multiple times in different hops, it could signify routing loops or misconfigurations. For example, if hop 5 and hop 7 show the same IP, data might be bouncing back and forth, causing delays.

Corporate networks often use private IP addresses internally, and these won't be accessible from the outside. You might see addresses like 10.0.0.1 or 172.16.0.1 in your traceroute. These are part of your internal infrastructure and can help pinpoint issues within your network.

Occasionally, you might notice a sudden change from private to public IPs. This transition usually marks the boundary between your internal network and the ISP's network. For instance, hops 1-4 might be private IPs, and hop 5 might be a public IP like 203.0.113.1. This demarcation is useful for identifying whether a problem lies within your network or beyond.

By examining these details, you can start to build a picture of where potential issues lie. Whether it's within your corporate network, at the ISP level, or further along the route, understanding traceroute results helps you troubleshoot more effectively.

‍

Having Performance Issues? Try Netmaker

If you're diagnosing an overlay or VPN, and are having issues with speed or connectivity, consider Netmaker. Netmaker uses WireGuard, to create ultra-fast, high-performing virtual network connections.