Monitoring my ASUS RT-AX86U Router with Prometheus and Grafana

I've been running my Internet Monitoring Pi for a year or so, and it's nice to collect data on Internet performance from inside my network.

But my router—currently an ASUS RT-AX86U—also tracks its own metrics for inbound and outbound traffic, among other things:

ASUSWRT-Merlin System Status Dashboard metrics

Sometimes having the raw data from the router that's on the edge of the network can tell a different story than measuring things behind the router. So I want to grab this data and put it into Prometheus.

With the stock ASUS firmware, this isn't really possible. But after reading a blog post about someone else monitoring an RT-AC86U with Prometheus, I decided to give it a shot on mine. I already run the ASUSWRT-Merlin firmware on my router, since I like having SSH access to it and can install some network utilities on it via a USB stick.

Installing node_exporter on the router

The first step is to get node-exporter running on the router, to expose data that my Prometheus instance can scrape.

And to do that, I needed to grab the URL of the latest arm64 Linux release on the node-exporter Releases page.

I have an USB flash drive plugged in, and it's mounted at /mnt/SANDISK. I created a node_exporter directory, and downloaded the binary inside:

mkdir -p /mnt/SANDISK/node_exporter
cd /mnt/SANDISK/node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-arm64.tar.gz
tar xzf node_exporter-1.3.1.linux-arm64.tar.gz
mv node_exporter-1.3.1.linux-arm64/node_exporter ./node_exporter
rm -rf node_exporter-1.3.1.linux-arm64*

To make sure it's working, I ran:

admin@RT-AX86U-FAC0:/tmp/mnt/SANDISK/node_exporter# ./node_exporter --version
node_exporter, version 1.3.1 (branch: HEAD, revision: a2321e7b940ddcff26873612bccdf7cd4c42b6b6)
  build user:       root@243aafa5525c
  build date:       20211205-11:10:22
  go version:       go1.17.3
  platform:         linux/arm64

NOTE: I initially tried node_exporter version 1.4.0, but was having trouble getting all the network statistics. I opened this issue to investigate: netdev collector failing with 'couldn't get netstats: incorrect size' since 1.4.0. Versions 1.3.1 and all earlier versions I tried down to 1.0.0 worked fine.

To make sure it starts running and will always run after a reboot, I added this shell script, based on this other blog post, and saved it as /mnt/SANDISK/node_exporter/node_exporter_start.sh:

#!/bin/bash

pidof node_exporter

if [[ $? -ne 0 ]] ; then
    /mnt/SANDISK/node_exporter/node_exporter --web.listen-address=":9100" >>/mnt/SANDISK/node_exporter/node_exporter.log 2>&1 &
fi

I gave it execute permissions with chmod +x node_exporter_start.sh.

You can use cru a to create a cron job, but with the ASUS firmware, the system boots into a RAM-only filesystem, meaning anything you change that's not inside a special persistent area will get wiped on reboot.

So I needed to add a 'user script' in the persistent JFFS filesystem. First, make sure JFFS scripts are enabled; to do that, in the router's UI, visit Administration > System, and make sure you have the 'Yes' option selected for "Enable JFFS custom scripts and configs".

Then SSH into the router and place a file named /jffs/scripts/init-start, with the contents:

#!/bin/sh
cru a node-exporter '* * * * * sh /tmp/mnt/SANDISK/node_exporter/node_exporter_start.sh'

Make that script executable (chmod +x /jffs/scripts/init-start), then to start node_exporter in the background, either manually run the cru command above, or reboot the router and it should be started. You can confirm it's running by visiting your router's IP at port 9100 in the browser, or run ps w | grep node_exporter via SSH and make sure it's running.

Configuring Prometheus to scrape the data

Now that node_exporter is running, it's time to set Prometheus to scrape data from the router. Inside the scrape_configs in your prometheus.yml file, add the following job:

scrape_configs:
  [...]
  - job_name: 'node'
    scrape_interval: 5s
    static_configs:
      - targets: ['10.0.100.1:9100']

If you already have a job named node, you could just add the router's IP/port to the list of targets. That's what I'm doing to get node information from all my monitored nodes in my homelab, in my internet-pi playbook.

Restart Prometheus, and if you check in Prometheus' UI, you should see the node target as 'up' after a few seconds—assuming Prometheus can see your router and node_exporter is running:

Prometheus node exporter up and running

The final step is adding a dashboard in Grafana to view the statistics in a clean way, over time. And for that, I'm relying on the Node Exporter Full dashboard—for some reason I couldn't just download the JSON using the button on Grafana's website, I had to go to the source repo and download the raw source for node-exporter-full.json.

After importing that into Grafana, I could see all the metrics for my router:

Grafana dashboard for node exporter from ASUS router

All those little spikes you see are the traffic flows in and out of various ports on the router during every-half-hour speedtests I use to monitor my available Internet bandwidth over time.

I'll likely work on a custom dashboard that gives me only the metrics I'm interested in at a glance, like in/out Internet bandwidth over time (just through the eth0 WAN connection), device CPU and memory usage, and individual graphs for each of the other interface types (e.g. 2.4 GHz and 5 GHz WiFi bands).

The ASUS RT-AX86U is actually quite a capable router, as far as up-to-1 Gbps connections are concerned. If I ever have a better ISP and can go beyond a gigabit down, or beyond 35 Mbps up, I'll probably build out a beefier router.

Comments

Hi Jeff! Love the work you did here - dashboard looks great!

Since node-exporter was (recently?) added to pfSense's public package depot I built and published a dashboard to the public Grafana repository myself! A few things would need to be adapted from BSD for your Asus - but wanted to offer as some possible inspiration, or even an as-is solution if you want to try it on any pfSense instances you may be running.

#16877 - avoided posting a direct link to not appear as spam

Hi Jeff!
Thanks for your Guide!
The node_exporter is running well on a RT-AC5300 as well. Had to use the Version 1.3.1 armv5.
Only getting node_exporter running on boot was a bit tricky.

Thanks a Lot!

Hi Jeff, thanks for the comprehensive instructions. To get running on my Asus RT-AC86U, I had to do the following extra things:

* Change the 9100 port or instead, disable "Network Printer Server" from Router UI -> USB Applications
* Pass in the following extra flag to `node_exporter` which is `--no-collector.netdev.netlink`. Issue #2502 on node-exporter Github has the details.

Good tip on the `--no-collector.netdev.netlink` flag, I was having, I assume, the same issue with "couldn't get netstats: incorrect LinkMessage size, want: 92 or 96, got: 116" errors and this corrected it for me.

Hi!

Just a beginner at linux.

When you write "pidof node_explorer" do I replace that with the pid of node_explorer, or is that content a command for the script?

I followed your instructions exactly on my RT-AX88U Merlin 388.1

After a reboot and inputting ps w | grep node_exporter into the terminal I get the result:

8317 admin 5780 S grep node_exporter

And I go to 192.168.50.1:9100

... and nothing. I've tried it from my iPad and Windows Desktop. I also tried changing the port number in the script but that doesn't help.

Any suggestions?

#!/bin/bash

pidof node_exporter

if [[ $? -ne 0 ]] ; then
/mnt/Sandisk/node_exporter/node_exporter --web.listen-address=":9100" >>/tmp/mnt/MerlinUSB/node_exporter/node_exporter.log 2>&1 &

I had the script wrong, working now! Thanks.
fi

I followed this and it's working all fine. Strangely it isn't reporting my 'RootFS Used' and 'RootFS Total' though.
Only difference between mine and yours is that my ASUS router is an AX86U_Pro, but surely this shouldn't affect node not being able to scrape this data?

The other thing I notice is that temp of the ASUS is not reported. Clearly it is able to be "got at" as the ASUS web-interface reports the core temp. Does anyone know how node_exporter can scrape that one too?

First I'd make sure it's not in node_exporter under a different name and check the log file to see if there is any issues there, but if those don't work, here's how (I think) you can add it manually.

First, you need to find your temp sensor. It is usually under '/sys/class/thermal' listed as something like thermal_zone0. To verify, can run 'cat /sys/class/thermal/thermal_zone0/temp' it should return the temperature in C multiplied by 1000 and that should match what's in the web UI.

Once you've found that, we just need a bash script that writes the information to a .prom file on the USB drive in node_exporter's format. Here's roughly what I think the bash script would be, but I didn't bother to test it on my router with node_exporter, so YMMV. Also, note you'll either need to wrap it in an infinite loop with a sleep at the end, or schedule it in cron to make it update periodically.

~~
#!/bin/bash

#define the sensor to be monitored
thermalSensor="/sys/class/thermal/thermal_zone0"

#text file output location, must match the directory defined in the collector.textfile.directory flag
outputFile=/mnt/SANDISK/node_exporter/textfiles/temp.prom

#get zone number, temperature, and sensor type
zoneNumber=${thermalSensor//[!0-9]/}
sensorTemp=$(cat $thermalSensor/temp)
sensorType=$(cat $thermalSensor/type)

#generate output string matching node_exporter format and write to file
echo $(printf 'node_thermal_zone_temp{type=\"%s\"},zone=\"%i\"} %.3f' "$sensorType" "$zoneNumber" $(($sensorTemp))e-3) > $outputFile

~~

Last but not least, you'll just need to add this flag: '--collector.textfile.directory=/mnt/SANDISK/node_exporter/textfiles' to node_exporter_start.sh where it calls node_exporter so node_exporter knows to look for the text file(s) and create the textfiles directory.

You can probably use a similar approach with something like 'df' to manually grab your RootFS data.

Hi Jeff,

Thank you for this guide! Couple questions
1. Will the node-exporter log file fill up my drive space or does it automatically clear old data?
2. Are there any other exporters you recommend setting up? Especially curious about your recommendations for security monitoring

Thanks!

it will keep filling up the drive as it's just a pipe to file (no log rotate)
however, if configured properly you shouldn't have any errors when it's running (so the log shouldn't even fill)

I had to add --no-collector.arp to my command line since it was spewing out errors every 5 seconds about unable to collect arp

Some additional information for anyone reading this in 2024
You can dig through the source code on github to pull how they grab certain stats from the router.

I've pulled all the nvram get entries from the vdsl fork of the firmware (since im using vdsl, so you will see some more commands which may not work on your variant) and put them here.

https://pastebin.com/axrQrV2B

you can also add librespeedtest-cli to the usb
https://github.com/librespeed/speedtest-cli/releases
make sure to get the linux version and not freebsd :)