Packet Loss Monitoring and Troubleshooting

Packets, also called network packets, are tiny units of data sent and received over a network. Everything you do on the internet is made up of packets, from answering VoIP calls to sending emails.

Packets are tiny portions of the overall message sent or function executed on the network. Packets move through a wide range of different hubs—fiber optics, wireless routes, and copper cables—to reach their destination, following the most direct route. Taking the most direct route helps the network evenly distribute its load across multiple points, which increases operational efficiency and enhances performance. When packets successfully reach their destination, they receive a timestamp.

However, packet loss can occur when packets don’t successfully make it to their destinations. A packet is considered lost when it’s been delayed or misplaced on its way through the network. When a packet is delayed, it’s timed out. Your network will continue trying to deliver the packet, but when it arrives, it will most likely be incomplete. This process is called “retransmission timeout” or RTO. Your packets will arrive eventually, but they’ll be late and not usable.

Packet loss can slow down your network, negatively impact throughput and bandwidth, and cause bottlenecks. What’s more, packet loss can also be expensive. If you don’t get a handle on packet loss you may be forced to continue buying additional bandwidth and IT infrastructure to compensate for the lag.

Packet loss has many different causes. Sometimes packet loss can be caused by bad actors trying to attack your network, but it’s far more likely for packet loss to be the result of issues within your network. Here are the top ten causes of packet loss.

1. Network or link congestion: Network congestion is the most common cause of packet loss. Networks only have so much space available, and network use ebbs and flows throughout the day. At peak times when network use hits maximum capacity, like after lunch when many employees are logging back in to their computers at the same time, packet loss can be more likely.

   When a network is trying to play catch up and accommodate for maximum traffic, packets are the first things to get dropped in favor of more critical operations. Packets must wait to be delivered. Luckily, a good packet loss monitoring tool will try to complete packet transmissions by automatically resending packets until congestion clears up or slowing down transfer speeds across the network to give more packets a chance to make it through.

2. Overloaded devices: Sometimes the devices on your network are working too hard. In this case, the packets might make it to their destinations, but by then, the network is too weak to process the packets and it sends them back. It’s good if the device has a buffer in place to automatically put packets in a holding pattern until they can be resent, but these buffers can get filled up quickly and end up dropping excess packets anyway.
3. Inadequate IT infrastructure: Many network administrators are forced to cobble together a functioning network monitoring solution from multiple disparate technologies. Unless these tools are specifically designed for packet loss monitoring, opportunities to reduce packet loss are bound to fall through the cracks.
4. Faulty network hardware: Old or glitchy hardware that cannot scale with your growing enterprise can weaken the network and make it more susceptible to packet loss.
5. Poor Wi-Fi connection: Wi-Fi packet loss is common because when data is sent through the air, it’s easier for packets to get lost in transit. This becomes even more likely when packets are sent over long-distance internet connections.
6. Software bugs: Unpatched software is vulnerable to bugs with a negative impact on network performance and prevent it from adequately transmitting packets.
7. Firewalls: Sometimes the measures you take to protect your network can end up hindering it. An upstream firewall might be filtering only certain types of traffic, which can cause some websites or services to randomly load or fail.
8. Duplex mismatch: When two ends of a link are operating at different speeds, some packets will be lost on the link.
9. Excessive hops: The packet sender can choose the maximum number of hops a packet will go through en route to its destination, which is known as Time to Travel (TTL). Each time the packet passes through a router, the TTL drops by one and eventually reaches zero, at which point the packet will be dropped. If a packet gets rerouted through too many hops, the TTL will reach zero before the packet reaches its destination and the packet will be dropped.
10. Cyberattack: It’s not very likely, but sometimes packet loss is intentionally caused by bad actors. If you notice a sharp drop in packet success or a sudden, significant slowdown in network speed, you might be experiencing a cyberattack. Unfortunately, there’s not much you can do about packet loss if your network is already under attack. If you act quickly enough, you can use an access list to block the hacker’s IP address.

If you’re experiencing packet loss, it will typically be because of an internal network issue. The best way to drill down and start troubleshooting packet loss is to leverage a comprehensive packet loss tool.

The most straightforward way to monitor packet loss is to ping the element you want to monitor, like an IP address or website, and check for evidence of packet loss or latency.

If the element takes a long time to respond to the ping, this indicates a problem needs investigating but might not amount to packet loss. If the element doesn’t respond to the ping at all, and you’ve checked to make sure everything is functioning properly on your end, you know packets aren’t making it to their destinations. You can also monitor packet loss by checking VoIP performance metrics and statistics for latency.

If you have monitored and tested your network for packet loss and determined you’re losing packets, there are a few techniques you can use to reduce packet loss.

1. Check your connections: Account for the most obvious causes of packet loss first and check your connections. Make sure all cables and ports are plugged in properly.
2. Reset everything: If you haven’t shut off your system in a while, a tired or overstressed network might be dropping packets. Restart everything and watch for any change in performance.
3. Update your software: Patch software glitches or bugs.
4. Replace old hardware: Make sure your hardware is healthy and functioning properly.
5. Look for things that could be causing static: Cameras, Bluetooth devices, wireless speakers, headphones, and other devices can eat up a lot of the bandwidth you need to deliver packets successfully. It also can help to temporarily shut off your firewall.
6. Use a cable connection instead of Wi-Fi: Switching to a fiber-optic or Ethernet connection can help prevent more packets from being lost.
7. Use QoS settings: Quality of Service (QoS) settings can help you prevent packet loss by ensuring network resources are being triaged accordingly. QoS settings will divert resources to the devices and applications that really need it first, like video calls and streamed content.

The recommendations listed above are manual ways of troubleshooting packet loss and should be sufficient for network administrators managing smaller networks or treating isolated incidents of packet loss. If you manage a large business or have widespread packet loss wreaking havoc on your network, you will need to use a packet loss monitor or packet loss analyzer.

SolarWinds VoIP & Network Quality Manager is an enterprise-grade packet loss monitor designed to take your packet loss monitoring efforts to the next level. Packet loss is one of the most prominent issues plaguing networks using VoIP, and network administrators need a specialized packet loss monitoring tool.

VNQM helps facilitate granular packet loss monitoring by allowing you to monitor, in real-time, key VoIP performance metrics indicating packet loss. Call detail records are automatically generated, stored, and compared with WAN performance to help you pinpoint performance issues caused by packet loss. You can use these historical records in conjunction with real-time data to create VoIP deployment performance baselines against which you can compare future performance.

VNQM lets you search through VoIP call records filtered according to the top packet loss monitoring metrics, which adds another dimension to troubleshooting packet loss. With this feature, you can easily find where packet loss is moving beyond acceptable levels and act accordingly before packet loss spirals into sustained VoIP call problems and network performance issues.