Monitor your Internet with a Raspberry Pi

Internet Service Providers are almost universally despised. They've pushed for the FCC to continue defining 25 Mbps as "high use" broadband, and on top of that they overstate the quality of service they provide. A recently-released map of broadband availability in the US paints a pretty dire picture:

USA map showing areas with limited high speed broadband availability

Here in St. Louis—where I guess I should count my lucky stars we have 'high use' broadband available—I have only two options: I can get 'gigabit' cable Internet from Spectrum, or 75 megabit DSL from AT&T.

That's it.

And you're probably thinking, "Gigabit Internet is great, stop complaining!"

But Spectrum's "gigabit" Internet is 930 megabits down—in ideal conditions—but only 40 megabits up. And that's the highest plan that costs about $150 a month!

Some would die for those speeds (see the map above), but much of the world is better off. And are you really getting the speeds you pay for? You probably don't know.

Grafana speedtest Internet monitoring dashboard

Well, I do, thanks to a $35 Raspberry Pi! And you don't even need a Pi, you could run the software I use on any computer. I just like having a dedicated computer to run all my Internet connection tools, so it's easy to backup or replace, and it doesn't get bogged down.

Video

I have a video that goes along with this blog post, and it's embedded below; skip past it if you prefer reading over viewing.

My Internet Pi

This is my "Internet Pi" (hostname geerli.net):

Internet Raspberry Pi in Rackmount case

I have it installed in a rack (here's a video on the Pi rack), but it could just as easily be on my desk, or sitting by my Internet router.

The most important thing for measuring a network connection is that it's wired. WiFi, especially on the Pi, fluctuates quite a bit and is terrible for monitoring anything besides maybe WiFi signal strength.

The Internet Pi runs Pi-hole for DNS privacy and ad-blocking, and Prometheus and Grafana to provide Internet connection monitoring dashboards.

Having a Pi monitoring my Internet continuously makes it easy to see trends over time, or confirm outages. If you just spot check by running a Speedtest every now and then, you don't have much data to go on.

Setting up Internet Pi

I built the free and open source internet-pi project on GitHub. It's simple to set up, and you can choose which parts of the platform you want to use.

I set up my Internet Pi from another computer, but you could also run the Internet Pi playbook on the Pi itself. Following the install instructions, you need to have a Pi set up with Raspberry Pi OS running on your network, and be able to SSH into the Pi.

Here are the steps to get it going:

  1. Make sure you have Ansible installed:
    1. (If pip3 is not installed) sudo apt-get install -y python3-pip
    2. pip3 install ansible
  2. Download the internet-pi repository to your computer: git clone https://github.com/geerlingguy/internet-pi.git && cd internet-pi
  3. Install the Ansible Galaxy content that's required to make the playbook work: ansible-galaxy install -r requirements.yml
  4. Make copies of the example.inventory.ini and example.config.yml files (dropping the example. from the filenames) and modify them for your own needs
  5. Run the playbook: ansible-playbook main.yml

The playbook installs all the tools you choose in your configuration file, and once that's done, you should be able to access your Pi's IP address in a web browser to see Pi-hole, or visit the IP address with the port :3030 on the end to see the Grafana dashboards.

Pi-hole - Geerli.net

Setting up Pi-hole

If you wanna use Pi-hole as your home network's DNS server, you should configure your router to advertise the Pi's IP address as the primary DNS server for your whole home network. For extra redundancy, you could actually configure two Pis running Pi-hole, but that's a topic for another post!

But the great thing is, that allows you to see all the DNS requests your devices are making, and make sure (for the most part) things like so-called 'smart' TVs don't send tons of information to advertisers.

Pi-hole is a great first line of defense for your Internet privacy, at least at home, and it even has some other neat tools like a simple DNS server, if you want to use them.

Internet monitoring

The Internet monitoring dashboard is automatically configured for you, but it can take up to an hour before you start seeing data.

Grafana speedtest Internet monitoring dashboard

In my case, it's been running for a couple months, so I can see the Internet connection for the past day, week, or even longer!

The monitoring configuration runs one container that connects to Speedtest.net. Every 30 minutes or so, another container running Prometheus tells the first container to run a speed test.

NOTE: If you run this monitoring on your own network, be aware that these Speedtest.net checks will consume a good amount of data, especially if you run them more frequently. If you have data caps and want to use this monitoring solution, make sure to modify the check interval so you don't get an unexpected overage. ISPs, gotta love 'em!

It stores that data on the Pi, and the Grafana dashboard displays it over time. There's another service that does simple checks on websites you can configure for uptime and HTTP request timing stats—those lightweight checks are performed every few seconds, to give a detailed view when there is more latency in your Internet connection.

One thing I found interesting was my Cable internet connection—which is supposed to be one 930 Mbps—is only really about 700 Mbps on average.

Upload speeds are more consistent, so I begrudgingly pay for this high-tier plan... But I'd be a lot happier with 50 or 100 Mbps of asymmetric data, since I'm not downloading terabytes of files every day.

Starlink monitoring

The main reason I set this stuff up was to compare my Gigabit Cable Internet to SpaceX's Starlink Internet.

Cable Spectrum ISP vs Starlink performance graph

I'll get more into Starlink in a future review, but it looks like on average, when it has a stable connection, I get over 150 Mbps down, and 15 Mbps up, with 40 milliseconds of latency.

Those numbers are less than my Cable ISP; but Starlink also costs less—it's $99/month. And Starlink is also beaming the Internet through satellites up in the sky! For someone who doesn't have a fast Cable option (like rural customers), that's a mind-blowing upgrade from dial-up speeds.

But I mentioned earlier, there's also a Starlink-specific dashboard that gives all the details you'd normally find in the Starlink App.

Starlink connection dashboard

I haven't been actively using the Starlink connection today, so it's a little barren, but it gives a ton of detail, and I'd like to thank Daniel Willcocks (DanOpsTech) for making this possible; I'm using his Docker image and dashboard, and he even adjusted it to work on the Raspberry Pi after I asked about it!

Power monitoring

I also monitor Starlink's power consumption using a Shelly Plug US and a custom Shelly Plug Prometheus exporter I wrote:

Shelly Plug power monitoring Starlink router and dishy dashboard Grafana

Starlink's dish and router use a lot of power (more than you might expect)—but I'll discuss more about that in my Starlink review coming soon! I have a blog post with more about how I set up the Shelly Plug power monitoring.

Conclusion

Now I know exactly how much my ISP is fleecing me for Internet. In Spectrum's case, it looks like they're only giving me about 3/4ths of what I pay for.

I wish I could just pay them $110 a month instead of $150, but if I did that, they'd cancel my service!

I'd like to thank Max Andersen for the original inspiration for the original monitoring configuration, the developers behind Pi-hole for making it free and open source, and especially Miguel who maintains the speedtest exporter, for his help getting some bugs fixed!

Comments

How do I get past error when I run the playbook?:
TASK [Ensure apt cache is up to date.] *****************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: W:This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details., E:Repository 'http://raspbian.raspberrypi.org/raspbian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable', W:This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details., E:Repository 'http://archive.raspberrypi.org/debian buster InRelease' changed its 'Suite' value from 'testing' to 'oldstable'"}

It looks like PiHole added a new feild that need to be processed into it. The "default_ivp4" variable. Here is my error:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.9/site-packages/ansible/plugins/action/template.py", line 138, in run
resultant = templar.do_template(template_data, preserve_trailing_newlines=True, escape_backslashes=False)
File "/home/pi/.local/lib/python3.9/site-packages/ansible/template/__init__.py", line 1131, in do_template
raise AnsibleUndefinedVariable(e)
ansible.errors.AnsibleUndefinedVariable: 'dict object' has no attribute 'default_ipv4'
fatal: [127.0.0.1]: FAILED! => {
"changed": false,
"msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'default_ipv4'"
}

PLAY RECAP **************************************************************************************************************************************************************************
127.0.0.1 : ok=13 changed=0 unreachable=0 failed=1 skipped=8 rescued=0 ignored=0

I'm not a YAML guy, but I'm willing to try if I can find the right file to add the dict_obj to.

My bet is the all the PiHole projects need this now with the latest release..

Everything seems to work fine except that it doesn't start automatically after reboot of Pi...
Any suggestions?

Raspi4B, 4GB
Latest RspberryPi OS
Wired connection

The speedtest results are nowhere near what I get using speedtest-cli. This is my typical cli result:

➜ ~ speedtest

Speedtest by Ookla

Server: Tampa Bay Fiber - Tampa, FL (id = 17026)
ISP: Comcast Business
Latency: 19.20 ms (0.06 ms jitter)
Download: 941.98 Mbps (data used: 1.5 GB)
Upload: 934.34 Mbps (data used: 1.1 GB)
Packet Loss: 0.0%

However the dashboard has a max up of 277 and down of 210. I tried both with no server specified and with the 17026 specified. It makes no difference. Why does running it in docker kill the speeds? At least it lets me know my connection is alive, but there is zero accuracy to the results. How can I troubleshoot this? I know nothing about how Prometheus or Grafana work.

Not sure if anyone has the same experience, but started using this last month and noticed that my data usage from Xfinity shot WAY up; To the point that I overran their data cap (~1.2TB). I happened to be out of town over the first of the month and noticed Xfinity had used over 200GB of data while I was still away in the new billing cycle. As soon as I disabled the Internet Pi, the usage stopped. I do have ~1Gb down, but never expected that kind of consistent throughput.

Another question that I can't figure out:

Is there a way to reset the statistics (without re-install)?

Could someone please supply a list of *all* services that could/should be stopped to stop/pause monitoring (not Pi-Hole) entirely.
( sudo systemctl stop )

Big THANKS in advance!!

You state this could run on any computer. I have an old MacBook. Could you point me to directions on setting up a MacBook to monitor my internet?note I am no computer programmer but I can follow directions (chem eng by degree). Tho this would be a good reason to get a pi and tinker a bit....

To run it on a Macbook (Intel CPU, I would guess), you'd need to run a virtualized Linux environment. You'd need to download VirtualBox, set up maybe an Ubuntu 22.04 installation in a new VM, and then run the setup instructions inside that VM.

Hopefully this is a quick question - is there a way to keep more than 2 weeks data? Whenever I login to Grafana it only shows me the latest 2 weeks? Thanks

Hi Jeff,

hope you are feeling better.
Would it be possible to explain how to add a Docker of Unbound to your build, I have tried but I am in a muddle.
I think adding in an upstream Unbound to the PiHole would make things even better.
I got your system working OK but I tried this and failed when trying to add in form this
https://fictionbecomesfact.com/unbound-container
Thanks
Fred

Hi Jeff. I hope you're feeling better. I'm not able to authenticate to Grafana anymore and don't recall setting a PW. I've seen a couple of procedures online for resetting the PW via CLI but they don't seem to work. Can you offer any insight?

Thanks,

-Rob

I've been looking for a way to monitor my ISP (Comcast) and track how often the internet drops. I've spent most of the day attempting to get this to work.

I cannot get the ansible-playbook to finish as it comes up with a pip and rust error.

I think a new, super detailed video is in order. If that's possible, I think a lot of the people having issues would appreciate it.

I am getting this when running the playbook ...

[email protected]:~/internet-pi $ ansible-playbook main.yml
[WARNING]: Unable to parse /home/pi/internet-pi/inventory.ini as an inventory
source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: Could not match supplied host pattern, ignoring: internet_pi

PLAY [internet_pi] *************************************************************
skipping: no hosts matched

PLAY RECAP ***************************************************

Any help for a total n00b plz ? :

The official Ookla Speedtest-CLI is up to version 1.2.0.84. The one in the docker container is 1.0.0.2. Any way I can update this in the docker container? I am not too familiar with docker. I ask because the 1.2 version seems better at picking a better performing server in the Speedtest network. The 1.0 version likes picking a server in another state and I get a third the performance. It does this over half the time. Thanks much.

I keep getting this error
HELP PLEASE

Building wheel for bcrypt (PEP 517): finished with status 'error'\nFailed to build bcrypt\n\n:stderr: ERROR: Command errored out with exit status 1:\n command: /usr/bin/python3 /tmp/tmp8a4e6zfx_in_process.py build_wheel /tmp/tmp1wu9g4dz\n cwd: /tmp/pip-install-zaj39q1u/bcrypt_37acdb3bfa4642f498d50f7cedd21f95

I have tried loading this software on a Raspberry Pi with the latest OS installed and keep getting this error

TASK [Ensure internet-monitoring environment is running.]
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Error connecting: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))"}
Can you direct me to a solution to this problem.

I love the concept of this project! As ISP owner/operator I was wondering if it is possible to have multiple monitoring boxes placed at selected points throughout my network to help me be more proactive about finding bandwidth and latency issues. I would really love to help further this project with funding and collaboration. It is possible to be an ISP and be noble and proactive in your efforts and intentions.

Hi ! Does this error mean anything to you?
My installation went well, I think I configured the inventory.ini file correctly, but I can't.
TASK [Install Docker Compose using Pip.] ***************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": ["/usr/bin/pip3", "inst all", "docker-compose"], "msg": "stdout: Looking in indexes: https://pypi.org/si

Hi jeff and thank you for your work !
i can tun internet pi on rasbp pi 4 but on mi Pi ZeroW i always get this message during install:
fatal: [192.168.1.29]: FAILED! => {"changed": false, "errors": [], "module_stderr": "", "module_stdout": "", "msg": "Error starting project 500 Server Error: Internal Server Error (\"Get \"https://registry-1.docker.io/v2/\": net/http: TLS handshake timeout\")"}

Is there anything i mess in my config file ?
Thank's a lot !

Hi,
I try to install this on a vm with Debian and i try to run ansible-playbook main.yml i got an error: Failed to connect to the host via ssh: [email protected]: Permission denied.
Where can i specify the right credentials?

I installed the project, and after working through a couple of issues, I could get Grafna to let me log in, but there were no graphs showing. I'm at a loss because the last thing I did without errors was run the install command again with passwords required to get past a permission error near the end. All worked successfully. I've rebooted the server to clear anything pending install. Can you tell me how to get the graphs installed?

You need to go to the 'Dashboards' section then 'Browse' to get to the dashboards that display data. And some of them may take 30+ minutes before you'll start getting data in.