Troubleshoot your home network like the Pros

The TL;DR

For a quick solution, all you need is a phone or a system with a browser in order to test and troubleshoot your Internet connection speed:

On your machine (preferably connected to the LAN or Internet modem via copper), visit fast.com.

Netflix’s FAST Speed Test is a well-designed tool that offers a quick and accurate way to measure your Internet line speed. We prefer this tool because Netflix’s service is available in many countries all over the world, and they invest a lot of effort making sure customers have a high quality of service.

You can also use the handy mobile APP (Android | iPhone) to test your Internet speed and check for WiFi dead spots throughout your house. Starting at the modem or WiFi access point, execute a series of tests while walking around your house to get a feel for signal strength and line speed.

Overview

In this (part 1 of 2) handy troubleshooting guide, we’re going to cover how to (1) properly troubleshoot your Internet connection and (2) setting up a handy CLI tool you can use when you need to troubleshoot problems (NOTE: a *NIX system is required).

• Install a command line tool on your system.
• Implement a monitor to collect data that can be used to graph and troubleshoot connection issues.

Motivations:

• Anxious family: Kids complaining about connection issues impatiently wanting to get back to Netflix or playing games 🙂

• Your ISP: In early 2000, our AT&T Internet DSL went down. After a bit of troubleshooting, I was able to determine that the problem was on the provider end and suggested a few places that they could check.

  The support specialist suggested sending an engineer out to the house, and if they found an issue – I would be changed $76/hr. In a confident tone, I repeated that the issue was not on the LAN facing side of the modem, nor the last mile to the house. They insisted it was, and so we came to an impasse. After 2 weeks, AT&T confessed that a route was deleted in California (at the time I was living in Missouri).

  Figuring that this may occur again, I documented everything thoroughly (knowing they would not (despite me asking)) – hoping I would not lose another 2 weeks of my life should this occur again. Exactly a year later, the issue rears it head again. I cited my notes. Still, it took them 2 weeks to confess that the route was deleted in CA again (AT&T yearly maintenance!!!).

• Unexpected issues: A while back, I was experiencing connectivity issues to websites and without thinking too much about it, I changed the DNS servers on the firewall to get around the problem. A few days later, I learned that our ISP (Comcast) was experiencing a DNS outage that affected the entire US East Coast and parts of the Midwest. These outages still persist today.

• Academic

Although I failed to contain my frustrations with AT&T – I was able to maintain peace-of-mind knowing exactly what the issue wasn’t and what action I needed to take. This eliminates anxiety and (possibly) avoids a long wait queue if you have to call your ISP. With this tool, you will be able to document and make sure you’re getting the speed and service you’re paying for.

Before We Begin

To assist with troubleshooting, it may help to have a basic understanding of your home network topology, Here’s an oversimplified diagram of how we connect to the Internet:

Some quick basic steps we can execute prior to troubleshooting:

• Some issues may be resolved simply by checking cables and rebooting your devices one-at-a-time until the issue is resolved: Internet modem any WiFi router(s) and your machine / phone / tablet starting with the least disruptive.
• You may also have too many devices on your home network, saturating your line. When troubleshooting, ensure that all devices are quiesced – not using your Internet line.
• Try another browser on the same computer: On several occasions, Chrome stopped working for me forcing me to reboot my system to fix the problem. Install additional browsers like Firefox or Opera.
• Also check your connection from another computer or device from the same network to rule out issues on on an affected system.

Let’s begin our troubleshooting with ruling out any LAN issues:

1) Ping systems on your local network:

Can you ping your machine’s network card?

Determine your system’s IP and default gateway:

C:\Users\xomedia>ipconfig

Windows IP Configuration


Wireless LAN adapter Wi-Fi 2:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.1.7
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
[...]

You can also use the following command to filter out all of the unwanted output:

C:\Users\xomedia>ipconfig | findstr Default
   Default Gateway . . . . . . . . . :
   Default Gateway . . . . . . . . . :
   Default Gateway . . . . . . . . . : 192.168.1.1

C:\Users\xomedia>

Linux:

~]# ip addr
[...]
2: eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:99:6e:ac brd ff:ff:ff:ff:ff:ff
    altname enp0s8
    inet 192.168.1.30/24 brd 192.168.1.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
~]#

~]# ip route | grep default
default via /192.168.1.1/ dev eth1 proto static metric 100
~]#

Ping your local IP (your network card):

C:\Users\xomedia>ping 192.168.1.7

Pinging 192.168.1.7 with 32 bytes of data:
Reply from 192.168.1.7: bytes=32 time<1ms TTL=128
Reply from 192.168.1.7: bytes=32 time<1ms TTL=128
Reply from 192.168.1.7: bytes=32 time<1ms TTL=128

Ping statistics for 192.168.1.7:
    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
C:\Users\xomedia>

Now ping the default gateway your system uses to reach the Internet:

C:\Users\xomedia>ping 192.168.1.1

Pinging 192.168.1.1 with 32 bytes of data:
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64

Ping statistics for 192.168.1.1:
    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 1ms, Average = 1ms
Control-C
^C
C:\Users\xomedia>

If you’re on a system behind the Internet modem, a private IP will return.

If you’re on the device that’s connecting directly to your ISP’s upstream router, depending on your ISP’s network configuration – typically a public IP will be assigned.

Anything outside of the private range are considered public.

This result is from one of our company firewalls running OpenBSD and PF (note the public IP):

~]# netstat -rn | grep default
default            104.51.228.1       UGS         re0
~]#

~]# ping 104.51.228.1
PING 104.51.228.1 (104.51.228.1): 56 data bytes
64 bytes from 104.51.228.1: icmp_seq=0 ttl=63 time=4.004 ms
64 bytes from 104.51.228.1: icmp_seq=1 ttl=63 time=4.796 ms
64 bytes from 104.51.228.1: icmp_seq=2 ttl=63 time=1.720 ms

Now ping a more distant device:

~]# ping cisco.com
PING cisco.com (72.163.4.185): 56 data bytes
64 bytes from 72.163.4.185: icmp_seq=0 ttl=240 time=19.672 ms
64 bytes from 72.163.4.185: icmp_seq=1 ttl=240 time=18.476 ms
64 bytes from 72.163.4.185: icmp_seq=2 ttl=240 time=19.295 ms

A ping time above 300ms indicates congestion.

Ideal ping times for general Internet usage (e.g., browsing, video calls, streaming movies):

Good: < 100 ms
Acceptable: 100-200 ms
Bad: > 200 ms

For online gaming:

Excellent: < 20 ms
Good: 20-50 ms
Acceptable: 50-100 ms
Bad: 100-150 ms
Unplayable: > 150 ms

2) Execute a traceroute to an external server:

Here, we start with the default gateway IP:

~]# traceroute 104.51.228.1
traceroute to 104.51.228.1 (104.51.228.1), 64 hops max, 40 byte packets
 1  192.168.1.254 (192.168.1.254)  0.506 ms  0.329 ms  0.496 ms
 2  104-51-228-1.lightspeed.stlsmo.sbcglobal.net (104.51.228.1)  5.033 ms  4.678 ms  3.979 ms
~]#

The following traceroute is executed to Cloudflare’s DNS server.

~]# traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 40 byte packets
 1  192.168.1.254 (192.168.1.254)  0.786 ms  0.326 ms  0.494 ms
 2  108-196-176-1.lightspeed.stlsmo.sbcglobal.net (108.196.176.1)  2.537 ms  5.202 ms  3.964 ms
 3  75.14.164.80 (75.14.164.80)  3.992 ms  2.864 ms  3.482 ms
 4  * * *
 5  * * *
 6  * * *
 7  32.130.24.205 (32.130.24.205)  9.785 ms  10.880 ms  11.500 ms
 8  12.116.153.210 (12.116.153.210)  11.632 ms  14.271 ms *
 9  172.71.252.4 (172.71.252.4)  27.006 ms
    172.69.56.5 (172.69.56.5)  18.926 ms  17.919 ms
10  one.one.one.one (1.1.1.1)  11.567 ms  11.412 ms  12.522 ms
~]#

3) Finally, verify that your ISP’s DNS is functioning properly:

*NIX:

~]# cat /etc/resolv.conf
search xomedia.me
nameserver 1.1.1.1
nameserver 8.8.8.8
~]#

Windows: Use ipconfig /all to find your DNS IPs and ping them:

C:\Users\xomedia>ping 1.1.1.1

Pinging 1.1.1.1 with 32 bytes of data:
Reply from 1.1.1.1: bytes=32 time=59ms TTL=51
Reply from 1.1.1.1: bytes=32 time=56ms TTL=51
Reply from 1.1.1.1: bytes=32 time=57ms TTL=51

Ping statistics for 1.1.1.1:
    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 56ms, Maximum = 59ms, Average = 57ms
Control-C
^C
C:\Users\xomedia>

You can sometimes resolve issues by clearing your DNS cache:

To display your DNS resolver cache:

ipconfig /displaydns

To clear your DNS resolver cache:

ipconfig /flushdns

ipconfig /registerdns

Additionally, run the followig commands to reset your network:

ipconfig /release

ipconfig /renew

Installing the Tool

speedtest-cli is a Python based CLI program we can use to spot-check line speed and automate data collection that can be graphed.

We’ll be using speedtest-cli, a Python based CLI program and one of many tools available to troubleshoot Internet connection speed.

https://github.com/sivel/speedtest-cli

Read the ‘Inconsistency’ section to understand tool limitations. It’s best to use several tools to cross-check results (which we’ll cover late).

We’re going to be following these instructions:

https://github.com/sivel/speedtest-cli

Ensure you have a supported version of Python installed on your system:

~]# python --version
Python 3.10.4

~]# pip install speedtest-cli
Collecting speedtest-cli
  Downloading speedtest_cli-2.1.3-py2.py3-none-any.whl (23 kB)
Installing collected packages: speedtest-cli
Successfully installed speedtest-cli-2.1.3
WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behavior with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip

~]#

~]# speedtest-cli -h
usage: speedtest-cli [-h] [--no-download] [--no-upload] [--single] [--bytes] [--share] [--simple] [--csv] [--csv-delimiter CSV_DELIMITER] [--csv-header] [--json] [--list] [--server SERVER] [--exclude EXCLUDE] [--mini MINI] [--source SOURCE] [--timeout TIMEOUT]
                     [--secure] [--no-pre-allocate] [--version]

Command line interface for testing internet bandwidth using speedtest.net. -------------------------------------------------------------------------- https://github.com/sivel/speedtest-cli

options:
  -h, --help            show this help message and exit
  --no-download         Do not perform download test
  --no-upload           Do not perform upload test
  --single              Only use a single connection instead of multiple. This simulates a typical file transfer.
  --bytes               Display values in bytes instead of bits. Does not affect the image generated by --share, nor output from --json or --csv
  --share               Generate and provide a URL to the speedtest.net share results image, not displayed with --csv
  --simple              Suppress verbose output, only show basic information
  --csv                 Suppress verbose output, only show basic information in CSV format. Speeds listed in bit/s and not affected by --bytes
  --csv-delimiter CSV_DELIMITER
                        Single character delimiter to use in CSV output. Default ","
  --csv-header          Print CSV headers
  --json                Suppress verbose output, only show basic information in JSON format. Speeds listed in bit/s and not affected by --bytes
  --list                Display a list of speedtest.net servers sorted by distance
  --server SERVER       Specify a server ID to test against. Can be supplied multiple times
  --exclude EXCLUDE     Exclude a server from selection. Can be supplied multiple times
  --mini MINI           URL of the Speedtest Mini server
  --source SOURCE       Source IP address to bind to
  --timeout TIMEOUT     HTTP timeout in seconds. Default 10
  --secure              Use HTTPS instead of HTTP when communicating with speedtest.net operated servers
  --no-pre-allocate     Do not pre allocate upload data. Pre allocation is enabled by default to improve upload performance. To support systems with insufficient memory, use this option to avoid a MemoryError
  --version             Show the version number and exit

~]# speedtest-cli
Retrieving speedtest.net configuration...
Testing from AT&T U-verse (104.51.230.155)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Wabash Communications (Louisville, IL) [174.33 km]: 52.978 ms
Testing download speed................................................................................
Download: 370.71 Mbit/s
Testing upload speed......................................................................................................
Upload: 202.18 Mbit/s
~]#

~]# time speedtest-cli --simple
Ping: 41.793 ms
Download: 336.26 Mbit/s
Upload: 118.10 Mbit/s
   21.44s real     1.00s user     0.59s system
~]#

~]# speedtest-cli --list
Retrieving speedtest.net configuration...
55143) i3 Broadband (St. Louis, MO, United States) [28.15 km]
 8986) Fidelity Communications (Sullivan, MO, United States) [69.42 km]
50254) Rural Comm Inc (Farina, IL, United States) [154.07 km]
29413) Socket Telecom, LLC (Columbia, MO, United States) [162.55 km]
60387) Futiva (Dahlgren, IL, United States) [165.12 km]
12852) CableAmerica (St. Robert, MO, United States) [167.27 km]
61242) ProTek Fiber (Marion, IL, United States) [167.35 km]
16198) Wabash Communications (Louisville, IL, United States) [174.33 km]
27468) Metro Communications (Sullivan, IL, United States) [200.24 km]
11749) Shawnee Communications (Equality, IL, United States) [211.83 km]
~]#

~]# speedtest-cli --server 61242
Retrieving speedtest.net configuration...
Testing from AT&T U-verse (104.51.230.155)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by ProTek Fiber (Marion, IL) [167.35 km]: 41.394 ms
Testing download speed................................................................................
Download: 134.08 Mbit/s
Testing upload speed......................................................................................................
Upload: 139.10 Mbit/s
~]#

~]# speedtest-cli --csv
8986,Fidelity Communications,"Sullivan, MO",2023-12-13T02:23:17.387882Z,69.4245604015283,39.62,251853320.06110522,168998425.23618942,,104.51.230.155
~]#

#!/bin/sh
set -x

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:
CHARSET=UTF-8
MM_CHARSET=UTF-8
LANG=en_US.UTF-8

/usr/local/bin/python /usr/local/bin/speedtest-cli --csv >> /tmp/speedtest.csv

#EOF

~]# chmod 740 /root/scripts/speedtest.sh

~]# crontab -l
# Execute a line speed test every 10 minutes:
*/10 * * * * /root/scripts/speedtest.sh > /tmp/speedtest.log 2>&1
~]#

~]# wc -l speedtest.csv
     115 speedtest.csv

~]# head -3 speedtest.csv
6182,Co-Mo Connect,"Tipton, MO",2023-12-24T03:40:00.258426Z,196.73598274857156,48.343,330036112.30918926,102144260.15577576,,104.51.230.155
8165,Net Vision Communications LLC,"Springfield, MO",2023-12-24T03:50:00.151392Z,287.4611623910043,42.869,393536770.3946404,228477915.40101135,,104.51.230.155
20531,Kansas Research and Education Network,"Wichita, KS",2023-12-24T04:10:00.198762Z,604.4268737743514,57.135,46784300.52631061,253839556.72484338,,104.51.230.155
~]#

6182,Co-Mo Connect,"Tipton, MO",2023-12-24,03:40:00.258426,196.73598274857156,48.343,330036112.30918926,102144260.15577576,,104.51.230.155

6182 | Co-Mo Connect | Tipton, MO | 2023-12-24 | 03:40:00.258426 | 196.73598274857156 | 48.343 | 330036112.30918926 | 102144260.15577576 |  | 104.51.230.155

6182 | Co-Mo Connect | Tipton, MO | 2023-12-24 | 03:40:00 | 196.73598274857156 | 48.343 | 330036112 | 102144260 |  | 104.51.230.155

6182 | Co-Mo Connect | Tipton, MO | 2023-12-24 | 03:40:00 | 196.73598274857156 | 48.343 | 330,036,112 | 102,144,260 |  | 104.51.230.155

You can reference our example here: Speed test results

Additional Tools

Use these to compare results across different tools.

Cloudflare

A measurement tool that runs on the Cloudflare network, which spans data centers in over 300+ cities worldwide. This ensures you are testing against a server that is close to you, which means you are measuring only the speed of your ISP, with minimal networks in between that may impact your score.

https://speed.cloudflare.com/

Like an Internet weather report [Internet Quality Index (IQI)]:

https://radar.cloudflare.com/quality

wget

Another CLI based tool that can be used as a download-only test

A 150mb file is ideal for testing.

~]# wget --output-document=/dev/null http://link.testfile.org/150MB
--2023-12-13 11:31:29--  http://link.testfile.org/150MB
Resolving link.testfile.org (link.testfile.org)... 104.21.47.248, 172.67.174.215, 2606:4700:3037::ac43:aed7, ...
Connecting to link.testfile.org (link.testfile.org)|104.21.47.248|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://link.testfile.org/150MB [following]
--2023-12-13 11:31:29--  https://link.testfile.org/150MB
Connecting to link.testfile.org (link.testfile.org)|104.21.47.248|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://files.testfile.org/ZIPC/150MB-Corrupt-Testfile.Org.zip [following]
--2023-12-13 11:31:29--  https://files.testfile.org/ZIPC/150MB-Corrupt-Testfile.Org.zip
Resolving files.testfile.org (files.testfile.org)... 172.67.174.215, 104.21.47.248, 2606:4700:3037::ac43:aed7, ...
Connecting to files.testfile.org (files.testfile.org)|172.67.174.215|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 157286400 (150M) [application/x-zip-compressed]
Saving to: ‘/dev/null’

/dev/null                                                          100%[=============================================================================================================================================================>] 150.00M  64.2MB/s    in 2.3s

2023-12-13 11:31:32 (64.2 MB/s) - ‘/dev/null’ saved [157286400/157286400]

~]#

The following sites offer various file types and pre-generated file sizes you can test with:

https://testfile.org/

https://testfiledownload.com/

You can also create your own set of files (1gb, 10gb, 100gb) to test between systems on your private network:

~]# dd if=/dev/zero of=1gb.bin bs=1048576 count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.843310 secs (1243405407 bytes/sec)
~]# 

~]# du -hs 1gb.bin
1.0G    1gb.bin

You can also generate custom file sizes to download here:

https://testfiledownload.com/test-file-generator/

LibreSpeed

An interesting (self-hosted) tool:

https://librespeed.org/

https://github.com/librespeed/speedtest

Iperf

A tool for network performance measurement and tuning.

For serious testing you can use Iperf following this guide:

https://iperf.fr/iperf-servers.php

Network testing in your pocket

You can take this further, by installing the following mobile APPs and add them to a folder.

Conclusion

Great job! In this article, we covered a proper way to measure your Internet line speed and how to automate data collection that can be plotted in a simple graph. In part 2 (coming soon) we will cover how to troubleshoot your home network – so be sure to check back.