Monitor your Internet with a Raspberry Pi

Internet Service Providers are almost universally despised. They've pushed for the FCC to continue defining 25 Mbps as "high use" broadband, and on top of that they overstate the quality of service they provide. A recently-released map of broadband availability in the US paints a pretty dire picture:

USA map showing areas with limited high speed broadband availability

Here in St. Louis—where I guess I should count my lucky stars we have 'high use' broadband available—I have only two options: I can get 'gigabit' cable Internet from Spectrum, or 75 megabit DSL from AT&T.

That's it.

And you're probably thinking, "Gigabit Internet is great, stop complaining!"

But Spectrum's "gigabit" Internet is 930 megabits down—in ideal conditions—but only 40 megabits up. And that's the highest plan that costs about $150 a month!

Some would die for those speeds (see the map above), but much of the world is better off. And are you really getting the speeds you pay for? You probably don't know.

Grafana speedtest Internet monitoring dashboard

Well, I do, thanks to a $35 Raspberry Pi! And you don't even need a Pi, you could run the software I use on any computer. I just like having a dedicated computer to run all my Internet connection tools, so it's easy to backup or replace, and it doesn't get bogged down.

Video

I have a video that goes along with this blog post, and it's embedded below; skip past it if you prefer reading over viewing.

My Internet Pi

This is my "Internet Pi" (hostname geerli.net):

Internet Raspberry Pi in Rackmount case

I have it installed in a rack (here's a video on the Pi rack), but it could just as easily be on my desk, or sitting by my Internet router.

The most important thing for measuring a network connection is that it's wired. WiFi, especially on the Pi, fluctuates quite a bit and is terrible for monitoring anything besides maybe WiFi signal strength.

The Internet Pi runs Pi-hole for DNS privacy and ad-blocking, and Prometheus and Grafana to provide Internet connection monitoring dashboards.

Having a Pi monitoring my Internet continuously makes it easy to see trends over time, or confirm outages. If you just spot check by running a Speedtest every now and then, you don't have much data to go on.

Setting up Internet Pi

I built the free and open source internet-pi project on GitHub. It's simple to set up, and you can choose which parts of the platform you want to use.

I set up my Internet Pi from another computer, but you could also run the Internet Pi playbook on the Pi itself. Following the install instructions, you need to have a Pi set up with Raspberry Pi OS running on your network, and be able to SSH into the Pi.

Here are the steps to get it going:

  1. Make sure you have Ansible installed:
    1. (If pip3 is not installed) sudo apt-get install -y python3-pip
    2. pip3 install ansible
  2. Download the internet-pi repository to your computer: git clone https://github.com/geerlingguy/internet-pi.git && cd internet-pi
  3. Install the Ansible Galaxy content that's required to make the playbook work: ansible-galaxy install -r requirements.yml
  4. Make copies of the example.inventory.ini and example.config.yml files (dropping the example. from the filenames) and modify them for your own needs
  5. Run the playbook: ansible-playbook main.yml

The playbook installs all the tools you choose in your configuration file, and once that's done, you should be able to access your Pi's IP address in a web browser to see Pi-hole, or visit the IP address with the port :3030 on the end to see the Grafana dashboards.

Pi-hole - Geerli.net

Setting up Pi-hole

If you wanna use Pi-hole as your home network's DNS server, you should configure your router to advertise the Pi's IP address as the primary DNS server for your whole home network. For extra redundancy, you could actually configure two Pis running Pi-hole, but that's a topic for another post!

But the great thing is, that allows you to see all the DNS requests your devices are making, and make sure (for the most part) things like so-called 'smart' TVs don't send tons of information to advertisers.

Pi-hole is a great first line of defense for your Internet privacy, at least at home, and it even has some other neat tools like a simple DNS server, if you want to use them.

Internet monitoring

The Internet monitoring dashboard is automatically configured for you, but it can take up to an hour before you start seeing data.

Grafana speedtest Internet monitoring dashboard

In my case, it's been running for a couple months, so I can see the Internet connection for the past day, week, or even longer!

The monitoring configuration runs one container that connects to Speedtest.net. Every 30 minutes or so, another container running Prometheus tells the first container to run a speed test.

NOTE: If you run this monitoring on your own network, be aware that these Speedtest.net checks will consume a good amount of data, especially if you run them more frequently. If you have data caps and want to use this monitoring solution, make sure to modify the check interval so you don't get an unexpected overage. ISPs, gotta love 'em!

It stores that data on the Pi, and the Grafana dashboard displays it over time. There's another service that does simple checks on websites you can configure for uptime and HTTP request timing stats—those lightweight checks are performed every few seconds, to give a detailed view when there is more latency in your Internet connection.

One thing I found interesting was my Cable internet connection—which is supposed to be one 930 Mbps—is only really about 700 Mbps on average.

Upload speeds are more consistent, so I begrudgingly pay for this high-tier plan... But I'd be a lot happier with 50 or 100 Mbps of asymmetric data, since I'm not downloading terabytes of files every day.

Starlink monitoring

The main reason I set this stuff up was to compare my Gigabit Cable Internet to SpaceX's Starlink Internet.

Cable Spectrum ISP vs Starlink performance graph

I'll get more into Starlink in a future review, but it looks like on average, when it has a stable connection, I get over 150 Mbps down, and 15 Mbps up, with 40 milliseconds of latency.

Those numbers are less than my Cable ISP; but Starlink also costs less—it's $99/month. And Starlink is also beaming the Internet through satellites up in the sky! For someone who doesn't have a fast Cable option (like rural customers), that's a mind-blowing upgrade from dial-up speeds.

But I mentioned earlier, there's also a Starlink-specific dashboard that gives all the details you'd normally find in the Starlink App.

Starlink connection dashboard

I haven't been actively using the Starlink connection today, so it's a little barren, but it gives a ton of detail, and I'd like to thank Daniel Willcocks (DanOpsTech) for making this possible; I'm using his Docker image and dashboard, and he even adjusted it to work on the Raspberry Pi after I asked about it!

Power monitoring

I also monitor Starlink's power consumption using a Shelly Plug US and a custom Shelly Plug Prometheus exporter I wrote:

Shelly Plug power monitoring Starlink router and dishy dashboard Grafana

Starlink's dish and router use a lot of power (more than you might expect)—but I'll discuss more about that in my Starlink review coming soon! I have a blog post with more about how I set up the Shelly Plug power monitoring.

Conclusion

Now I know exactly how much my ISP is fleecing me for Internet. In Spectrum's case, it looks like they're only giving me about 3/4ths of what I pay for.

I wish I could just pay them $110 a month instead of $150, but if I did that, they'd cancel my service!

I'd like to thank Max Andersen for the original inspiration for the original monitoring configuration, the developers behind Pi-hole for making it free and open source, and especially Miguel who maintains the speedtest exporter, for his help getting some bugs fixed!

Comments

Just a couple of questions.

Would this install work the dietpi distro?

How to do recursive DNS with pihole and the docker setup here? And how to change the pihole password if you're a dumbass like me and missed the config file edit, again I'm assuming because of the docker setup. If you can't tell I'm not familiar with docker but am learning. Slowly. :)

Hi Jeff i managed to get it started thank you. I have one concern when i logged into my Pi i checked network connetions in the top corner and found 7 different connections in ther pop up when you hover over the arrows. my Pi os was a fresh clean install on a new memory card. I was wondering if you know what these connections were to and why they are there they only showed up afrer installing all this. sorry if its a simple answer im a newb to all this i have a picture of the connections but not sure how to add it to this thread.

Many thanks
Wez

I found this after watching your excellent YouTube on StarLink. But after installing I get no default dashboard for the speedtest. PiHole works but no default on ip:3030
What can I look at sir?

How do you clear/reset the collected data for internet-connection? I was monitoring one ISP and went to another. I don't want the data from the first ISP to dirty the other one. I'm new to dockers, grafana, and prometheus, so a little patience and guidance is appreciated.

BTW, using Raspi 3B+. Thanks.

THIS is why folks don't like Linux et.al. I have run in to so many errors. I wonder if the author actually did the steps they suggested and nothing more. I am going to reinitialize my pi and document the nightmare. I have Python3.9.5 now installed but something called pip fails.

Can't even get past step 3:

pi@raspberrypi:~/internet-pi $ ansible-galaxy install -r requirements.yml
Traceback (most recent call last):
File "/home/pi/.local/bin/ansible-galaxy", line 34, in
from ansible import context
ModuleNotFoundError: No module named 'ansible'

Trying to locate it I get:
pi@raspberrypi:/ $ find . -name "ansible" 2>/dev/null
./home/pi/.local/lib/python3.7/site-packages/ansible_collections/netbox/netbox/docs/js/ansible
./home/pi/.local/lib/python3.7/site-packages/ansible_collections/ansible
./home/pi/.local/lib/python3.7/site-packages/ansible
./home/pi/.local/lib/python3.7/site-packages/ansible_test/_data/injector/ansible
./home/pi/.local/bin/ansible

Why didn't it install on the $PATH? I even rebooted and still no joy.

This set of commands work on what version of raspbian, or other compatible OS?

pihole just had an update. How do we get about updating the pihole component? Doing pihole -up in docker results to an error.
```
pi at raspberrypi:~/internet-pi $ docker exec -it pihole /bin/bash
root at pihole:/# pihole -v
Pi-hole version is v5.5 (Latest: v5.6)
AdminLTE version is v5.7 (Latest: v5.8)
FTL version is v5.10.2 (Latest: v5.11)
root at pihole:/# pihole -up
Function not supported in Docker images
root at pihole:/#
```

so i am not too versed with linux and i pretty much know how to copy paste commands into the terminal so be easy on me
i tried to update the pihole version with pihole -up and the command ran but just said that i am up to date (although in the interface it didnt update and still says there's an update available) also tried to run ansible-playbook main.yml which basically did nothing
would really like some help updating the pihole

Morning,

i'm running raspberry os 64 bit. the task is getting hung up on installing docker compose using pip.

any idea's

I'm close, so close. It's running. But I cannot seem to change the interval that it runs a speed test of pings the 3 web sites. I change the intervals in 'internet-pi/config.yml', then rerun 'ansible-playbook main.yml --ask-pass', then reboot. But it continues to run the speed test and ping(s) same as before. What am I missing?

Can this be install on a running PiHole system? I would like to keep the PIHole out of the container.

Yes, you can just set the pihole_enabled variable to false before deploying it. It will just leave its Pi-hole install out of the mix.

I've got this set up and running but it seems that the data only goes back 15-16 days. I've tried to find a setting that lets me hold on to the data longer but I can't seem to figure it out. Any ideas where I should be looking?

After flashing a usb stick 4 ou 5 times, I finally managed to install this on a rpi4. The new version of raspberry os (bullseye) eliminated all of the problemas I had previoulsy.

Was following the instructions on a brand new "Raspberry Pi OS Lite" install (from the main raspberry pi site), surprisingly git didn't seem to be installed by default so I had to add that step in order to do the git clone, don't know if this is a recent update to their image but it might be worth adding to the instructions (here and on github)?

Im lost on the 4th step on setting this up on my pi. any help?

pi@raspberrypi:~/internet-pi $ ansible-playbook main.yml
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the
controller starting with Ansible 2.12. Current version: 3.7.3 (default, Jul 25
2020, 13:03:44) [GCC 8.3.0]. This feature will be removed from ansible-core in
version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
[WARNING]: Unable to parse /home/pi/internet-pi/inventory.ini as an inventory
source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: Could not match supplied host pattern, ignoring: internet_pi

This is what im getting for the 5th step
im trying to do this on my pi locally

Anyone can help me with this error, tried now on 2 devices and not sure what im doing wrong. all installed ok but everytime i run playbook i get this error at the end.
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Error connecting: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))"}

On Github page of this project you have this comment:
If running locally on the Pi: You may encounter an error like "Error while fetching server API version". If you do, please either reboot or log out and log back in, then run the playbook again.
so try rebooting the pi and try again. It worked for me.

Is there a way to replace pi-hole with AdGuard and use your dashboard?

Hey there,
thank you for your work, great project!
I do however have one issue...
everything works great except the internet speeds and latency. If I go to the query inspector for download or upload speeds I get the following:
Object
request:Object
url:"api/datasources/proxy/3/api/v1/query_exemplars"
method:"POST"
data:Object
query:"speedtest_upload_bits_per_second{}"
start:"1636368445"
end:"1636973280"
hideFromInspector:false
response:Object
message:"404 page not found
"
error:"Not Found"
response:"404 page not found
"

which I think might be the problem?
Anyone has any ideas maybe? Unfortunately I am totally new to docker containers and prometheus...

I'm trying to get this working on my raspberry pi, I have counted this error;

fatal: [192.168.1.4]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible_user' is undefined\n\nThe error appears to be in '/home/pi/internet-pi/tasks/docker.yml': line 58, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Ensure {{ ansible_user }} user is added to the docker group.\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"}

Just wanted to say this install was done very well. I had the logoff/logon issue but everything appears to be working after that.
I did have a question though. Where does Grafana pick up the data from. I am not too Docker savvy and didn't see a definition for the logs (which I assume are json). Thanks!

Hey Jeff and all,
I was able to install everything and have everything running smoothly until I start digging and play around a little bit.
if you run:
```
cat tasks/internet-monitoring.yml or
cat tasks/pi-hole.yml
```
You can clearly see that something needs to be written in the home folder, example:
- name: Create Pi-hole folder on Pi.
```
ansible.builtin.file:
path: "{{ config_dir }}/pi-hole"
state: directory
mode: 0755
become: false
```
Unfortunately nothing gets written for pi-hole and I suspect also fro Grafana and I can't figure out why everything is working nevertheless.
if I check sudo docker container ls:
```
f616cddee2e4 pihole/pihole:latest "/s6-init" 14 hours ago Up 13 hours (healthy)
```
I am wondering if the problem might be related to the fact I don't have a user `pi` in my machine....

Hi there,
As far as I can tell i have successfully completed the install however on the Internet speed dashboard there appears to be no data (completed install yesterday so should be something right?). And by no data I mean there are no dials or anything, the page keeps suggesting me to add/create a dashboard otherwise it is empty?
Should i try re-installing?

Jeff, First off thank you for taking the time to put together such a good video/blog post. I've setup my Internet Pi without Pi Hole, and everything is showing data correctly, however I am still getting a modal that says "Annotation Query Failed - Cannot read properties of undefined (reading 'to')" Any ideas what might be causing that, or how I can get rid of that error? Thank you!

Have you thought of adding this to Home-Assistant? I'm new to HA, and would like to add something like this, but not sure how.