During my most recent hospitalization, besides making a video about the IV pump, I was messing around with a Raspberry Pi 4G LTE router project I could stash in my hospital go-bag (I am hospitalized about 2.5x per year, on average).
I was testing out a new 4G LTE modem, a Sierra Wireless AirPrime EM7565, which is rated at Cat 12 and theoretically gets hundreds of megabits down. A huge improvement over my previous modem of choice, the Quectel EC25-A.
I already know how annoying it is to work with 4G modems—you basically have to learn a new language, sending magic AT commands over a serial connection to the modem. And most of the guts are proprietary, usually controlled by Qualcomm. And every guide on the Internet differs a bit, because some people use QMI mode, some MBIM, and some really old guides even talk about PPP!
This blog post is a condensed summary of the process of bringing up this modem. For all the gory details, check out this issue in my Pi Router GitHub repo: Work on initial LuCI and hardware configs.
I learned how to use minicom to connect to the modem (e.g. minicom --device /dev/ttyUSB0 --baudrate 115200), and using that utility is a blast from the past! If you thought exiting vim was hard, try exiting minicom while SSH'ed into a Raspberry Pi from a Mac using Terminal!
Alternatively I could have one terminal session open to cat /dev/ttyUSB0 and in another, run AT commands like echo -ne 'AT\r\n' > /dev/ttyUSB0. But that often led to weird characters being inserted when I did that via SSH, causing mayhem. In either case, it feels like computing in the 80s.
I bought my EM7565 used because higher-end 4G and 5G modems are expensive, and I'm from the midwest. I don't buy things new if there's a buck to be saved.
Running the AT command AT!PRIID?, I found out the modem I purchased seems to have come from a Lenovo laptop, probably running Windows.
AT!PRIID?
PRI Part Number: 9907220
Revision: 001.004
Customer: Lenovo-Laptop
Carrier PRI: 9999999_9907259_SWI9X50C_00.06.02.00_00_GENERIC_001.004_000
Windows prefers these modems run in a mode called MBIM, and sometimes they don't even appear in Linux at all. In my case, the modem's USB Product ID was 90b1, which meant Linux's qmi_wwan driver didn't even pick up the modem and register it as a serial device.
root@OpenWrt:~# dmesg
...
[ 7.329244] usb 2-3: New USB device found, idVendor=1199, idProduct=90b1, bcdDevice= 0.06
Thanks to Sierra Wireless forum user rspmn, I learned I could force it to use a USB Product ID the qmi_wwan Linux driver will recognize:
# modprobe option
# echo 1199 90b1 > /sys/bus/usb-serial/drivers/option1/new_id
After doing that, the ttyUSB* devices appeared in /dev. But it's a temporary fix: you have to force the modem to persist the new PID using the USBPID command.
AT!ENTERCND="A710"?
OK
AT!USBPID?
!USBPID:
APP : 90B1
BOOT: 90B0
AT!USBPID=9091
OK
AT!USBPID?
!USBPID:
APP : 9091
BOOT: 9090
Good. Now the USB PID persists across reboots. But the process of getting this modem to work with Linux was just beginning. Little did I know I'd spend days trying to get this modem into a working state!
My first attempt to get it to work was to reset it to factory defaults. I did that with the commands:
// Unlock password-protected commands (A710 is the default pw).
AT!ENTERCND="A710"
OK
// Restore the modem to factory defaults.
AT!NVRESTORE=0
!NVRESTORE:
Items Restored: 906
Items Deleted: 272
Items Skipped: 4
OK
// Reset the modem.
AT!RESET
OK
But after resetting the modem, it would no longer show up with a USB TTY device! Apparently when I reset the modem, it started using that old USB PID again. So I had to go through the steps of forcing ID 9091 again.
Once I did that, I compiled a custom build of OpenWRT, and configured a wwan interface in OpenWRT to connect using my SixFab SIM card. This seemed to work... except that I was only seeing TX traffic, and no RX traffic:
Every time I reset the modem, it would connect for 45 seconds, then disconnect:
Sat Aug 20 22:36:24 2022 daemon.info [3589]: <info> [modem0] state changed (connected -> registered)
Sat Aug 20 22:36:24 2022 daemon.info [3589]: <info> [modem0/bearer5] connection #1 finished: duration 45s, tx: 0 bytes, rx: 0 bytes
Sat Aug 20 22:36:25 2022 user.notice modemmanager: interface wwan (network device wwan0) disconnected
Sat Aug 20 22:36:25 2022 daemon.notice netifd: Interface 'wwan' has lost the connection
Sat Aug 20 22:36:25 2022 daemon.notice netifd: Network device 'wwan0' link is down
Quite annoying.
After tons of debugging, a few other users (notably, rjocoleman) noticed the firmware on my EM7565 was ancient. Indeed, checking on it, it was from 2017:
AT+CGMR
SWI9X50C_00.06.02.00 f50f0a jenkins 2017/08/25 05:33:58
That set me off on a long rabbit hole trying to update the firmware—first trying Sierra Wireless's fwdwl-litearm64le utility, then using qmi-firmware-update. I had to first try upgrading to the 01.05.01.00 version due to changes in the way Sierra signs their code, but no matter what, it would get to the point of writing the firmware .nvu file... then stop.
And after reading through a ton of forum posts, we came to the conclusion that USB 3.0 (xHCI) was unstable when writing older firmware to the EM7565.
So it was off to Amazon to buy an external NGFF M.2 B-key to USB adapter that I could use to plug the EM7565 into a USB 2.0 port on my Raspberry Pi 4, to try flashing the firmware again.
I placed the modem in the board, tried qmi-firmware-update from the Pi 4, and what do you know? It worked!
root@waveshare:/home/pi/swi_fw0105# qmi-firmware-update -t /dev/ttyUSB0 --update SWI9X50C_01.05.01.00.cwe SWI9X50C_01.05.01.00_GENERIC_001.028_000.nvu
loading device information before the update...
setting firmware preference:
firmware version: '01.05.01.00'
config version: '001.028_000'
carrier: 'GENERIC'
rebooting in download mode...
download mode detected
downloading cwe image: SWI9X50C_01.05.01.00.cwe (80.2 MB)...
finalizing download... (may take several minutes, be patient)
successfully downloaded in 99.67s (804.8 kB/s)
downloading cwe image: SWI9X50C_01.05.01.00_GENERIC_001.028_000.nvu (3.9 kB)...
finalizing download... (may take several minutes, be patient)
successfully downloaded in 0.05s (79.2 kB/s)
rebooting in normal mode...
normal mode detected
...
...but it also reset the USB PID again, so I had to re-reset that so I could then update the firmware again to the latest version. I did that twice because apparently if you update from the 01.05.01.00 version to the latest version only one time, it could get the modem stuck in 'low power mode'. Why? No clue.
Sometimes with these modems you don't question it: just roll with what the people who seem to know what they're doing say.
So now, the modem is finally running the latest firmware:
new firmware revision is:
SWI9X50C_01.14.13.00 883709 jenkins 2022/01/05 07:08:35
new running firmware details:
Model: EM7565-9
Boot version: SWI9X50C_01.14.13.00
AMSS version: SWI9X50C_01.14.13.00
Carrier ID: 1
Config version: 002.048_000
new firmware preference details:
image 'modem': unique id '002.048_000', build id '01.14.13.00_GENERIC'
image 'pri': unique id '002.048_000', build id '01.14.13.00_GENERIC'
Does it work? Can I get a custom compiled OpenWRT build working (there's a great OpenWRT compile guide video here, btw)? Don't know yet. I'll keep working on it, and hopefully will have it ready for a video on my YouTube channel soon!
Comments
When I had my BBS in the 90s, I had to burn EPROMs for the firmware upgrades of the modems, and I though that that was a PITA. When EEPROMS finally came around (and later flash memory), I never thought that I'd miss burning EPROMS, but this shows that newer is not necessarily better
The AT commands used for controlling GSM modems are supposed to be standardised under 3GPP, but many of the modems use extensions and then don't implement the some of the same functions under the standard. It is a mess.
https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDeta…
Interesting post!
1. How did you choose the EM7565? Could you share the approximate cost? CAT12 sounds nice, but if CAT6 is enough, then for $80 (eBay) to $130 (refurbisher with some warranty) you could get a Cradlepoint CBA850 with the MC400LP6 modem. The Cradlepoint is a very capable device by itself, but the MC400LP6 plugs into the Cradlepoint using USB. And if you were to open the MC400LP6, you'd find a mPCI card with the modem on it. So it has all the connectivity options.
2. I've been using T-Mobile Home Internet for $50 a month (it's unlimited, Verizon has a similar service but not as geographically widespread). They updated a tower near me and now I get n41/B66 with 400-900 MBit down and 30-60 MBit up (depends on time of day). Supposedly T-Mobile will unofficially allow you to use their device away from your home (not tried that myself), and there is a pre-paid version. I looked at alternative data only plans so I'd have a backup for the T-Mobile, and the best I could find was Redpocket on AT&T for $11/1GB (a far cry from the $50/unlimited). The SixFab is even more expensive. Assuming you only use this for very little data? Btw the RedPocket SIM gets ~25 MBit in the CBA850, which is far from the CAT6 maximums. How fast is the SixFab SIM? And are you on T-Mobile or AT&T?
SixFab seems to be on Twilio, which seems to roam on T-Mobile (maybe?). Not quite sure but it's better for IoT-style deployments, not as full time high speed access.
I'm looking at getting a T-Mobile or AT&T SIM and plan for more speed and coverage though.
I have been sitting on one of these modems for the better part of a year. In a USB adapter and plugged into a windows 10 box I had no issues getting it to work on my Verizon account. No such luck under pfSense, openRT, or DDWRT. Clearly I have some things to try to get it a bit closer. I have a PC Engines APU4d4 for the job so I might have issues there as well... At least I know that the modem works so I guess I got that going for me.