During my most recent hospitalization, besides making a video about the IV pump, I was messing around with a Raspberry Pi 4G LTE router project I could stash in my hospital go-bag (I am hospitalized about 2.5x per year, on average).
I was testing out a new 4G LTE modem, a Sierra Wireless AirPrime EM7565, which is rated at Cat 12 and theoretically gets hundreds of megabits down. A huge improvement over my previous modem of choice, the Quectel EC25-A.
I already know how annoying it is to work with 4G modems—you basically have to learn a new language, sending magic
AT commands over a serial connection to the modem. And most of the guts are proprietary, usually controlled by Qualcomm. And every guide on the Internet differs a bit, because some people use QMI mode, some MBIM, and some really old guides even talk about PPP!
This blog post is a condensed summary of the process of bringing up this modem. For all the gory details, check out this issue in my Pi Router GitHub repo: Work on initial LuCI and hardware configs.
I learned how to use
minicom to connect to the modem (e.g.
minicom --device /dev/ttyUSB0 --baudrate 115200), and using that utility is a blast from the past! If you thought exiting vim was hard, try exiting
minicom while SSH'ed into a Raspberry Pi from a Mac using Terminal!
Alternatively I could have one terminal session open to
cat /dev/ttyUSB0 and in another, run AT commands like
echo -ne 'AT\r\n' > /dev/ttyUSB0. But that often led to weird characters being inserted when I did that via SSH, causing mayhem. In either case, it feels like computing in the 80s.
I bought my EM7565 used because higher-end 4G and 5G modems are expensive, and I'm from the midwest. I don't buy things new if there's a buck to be saved.
Running the AT command
AT!PRIID?, I found out the modem I purchased seems to have come from a Lenovo laptop, probably running Windows.
AT!PRIID? PRI Part Number: 9907220 Revision: 001.004 Customer: Lenovo-Laptop Carrier PRI: 9999999_9907259_SWI9X50C_00.06.02.00_00_GENERIC_001.004_000
Windows prefers these modems run in a mode called
MBIM, and sometimes they don't even appear in Linux at all. In my case, the modem's USB Product ID was
90b1, which meant Linux's
qmi_wwan driver didn't even pick up the modem and register it as a serial device.
[email protected]:~# dmesg ... [ 7.329244] usb 2-3: New USB device found, idVendor=1199, idProduct=90b1, bcdDevice= 0.06
Thanks to Sierra Wireless forum user rspmn, I learned I could force it to use a USB Product ID the
qmi_wwan Linux driver will recognize:
# modprobe option # echo 1199 90b1 > /sys/bus/usb-serial/drivers/option1/new_id
After doing that, the
ttyUSB* devices appeared in
/dev. But it's a temporary fix: you have to force the modem to persist the new PID using the
AT!ENTERCND="A710"? OK AT!USBPID? !USBPID: APP : 90B1 BOOT: 90B0 AT!USBPID=9091 OK AT!USBPID? !USBPID: APP : 9091 BOOT: 9090
Good. Now the USB PID persists across reboots. But the process of getting this modem to work with Linux was just beginning. Little did I know I'd spend days trying to get this modem into a working state!
My first attempt to get it to work was to reset it to factory defaults. I did that with the commands:
// Unlock password-protected commands (A710 is the default pw). AT!ENTERCND="A710" OK // Restore the modem to factory defaults. AT!NVRESTORE=0 !NVRESTORE: Items Restored: 906 Items Deleted: 272 Items Skipped: 4 OK // Reset the modem. AT!RESET OK
But after resetting the modem, it would no longer show up with a USB TTY device! Apparently when I reset the modem, it started using that old USB PID again. So I had to go through the steps of forcing ID
Once I did that, I compiled a custom build of OpenWRT, and configured a
wwan interface in OpenWRT to connect using my SixFab SIM card. This seemed to work... except that I was only seeing TX traffic, and no RX traffic:
Every time I reset the modem, it would connect for 45 seconds, then disconnect:
Sat Aug 20 22:36:24 2022 daemon.info : <info> [modem0] state changed (connected -> registered) Sat Aug 20 22:36:24 2022 daemon.info : <info> [modem0/bearer5] connection #1 finished: duration 45s, tx: 0 bytes, rx: 0 bytes Sat Aug 20 22:36:25 2022 user.notice modemmanager: interface wwan (network device wwan0) disconnected Sat Aug 20 22:36:25 2022 daemon.notice netifd: Interface 'wwan' has lost the connection Sat Aug 20 22:36:25 2022 daemon.notice netifd: Network device 'wwan0' link is down
After tons of debugging, a few other users (notably, rjocoleman) noticed the firmware on my EM7565 was ancient. Indeed, checking on it, it was from 2017:
AT+CGMR SWI9X50C_00.06.02.00 f50f0a jenkins 2017/08/25 05:33:58
That set me off on a long rabbit hole trying to update the firmware—first trying Sierra Wireless's
fwdwl-litearm64le utility, then using
qmi-firmware-update. I had to first try upgrading to the
01.05.01.00 version due to changes in the way Sierra signs their code, but no matter what, it would get to the point of writing the firmware
.nvu file... then stop.
And after reading through a ton of forum posts, we came to the conclusion that USB 3.0 (xHCI) was unstable when writing older firmware to the EM7565.
So it was off to Amazon to buy an external NGFF M.2 B-key to USB adapter that I could use to plug the EM7565 into a USB 2.0 port on my Raspberry Pi 4, to try flashing the firmware again.
I placed the modem in the board, tried
qmi-firmware-update from the Pi 4, and what do you know? It worked!
[email protected]:/home/pi/swi_fw0105# qmi-firmware-update -t /dev/ttyUSB0 --update SWI9X50C_01.05.01.00.cwe SWI9X50C_01.05.01.00_GENERIC_001.028_000.nvu loading device information before the update... setting firmware preference: firmware version: '01.05.01.00' config version: '001.028_000' carrier: 'GENERIC' rebooting in download mode... download mode detected downloading cwe image: SWI9X50C_01.05.01.00.cwe (80.2 MB)... finalizing download... (may take several minutes, be patient) successfully downloaded in 99.67s (804.8 kB/s) downloading cwe image: SWI9X50C_01.05.01.00_GENERIC_001.028_000.nvu (3.9 kB)... finalizing download... (may take several minutes, be patient) successfully downloaded in 0.05s (79.2 kB/s) rebooting in normal mode... normal mode detected ...
...but it also reset the USB PID again, so I had to re-reset that so I could then update the firmware again to the latest version. I did that twice because apparently if you update from the
01.05.01.00 version to the latest version only one time, it could get the modem stuck in 'low power mode'. Why? No clue.
Sometimes with these modems you don't question it: just roll with what the people who seem to know what they're doing say.
So now, the modem is finally running the latest firmware:
new firmware revision is: SWI9X50C_01.14.13.00 883709 jenkins 2022/01/05 07:08:35 new running firmware details: Model: EM7565-9 Boot version: SWI9X50C_01.14.13.00 AMSS version: SWI9X50C_01.14.13.00 Carrier ID: 1 Config version: 002.048_000 new firmware preference details: image 'modem': unique id '002.048_000', build id '01.14.13.00_GENERIC' image 'pri': unique id '002.048_000', build id '01.14.13.00_GENERIC'
Does it work? Can I get a custom compiled OpenWRT build working (there's a great OpenWRT compile guide video here, btw)? Don't know yet. I'll keep working on it, and hopefully will have it ready for a video on my YouTube channel soon!