Automatically recover a failing USB LTE modem
written on Tuesday, September 19, 2017
My home network consists of a wireless router running LEDE 17.01 and a ZTE MF831 LTE USB modem for Internet connectivity. From time to time the Internet connection fails and the only way to recover was to physically reconnect the USB modem. So each time it failed, I had to get to my wireless router, pull the USB modem and reconnect it. This post describes the steps I took to work around this issue.
The lost connection doesn't seem to follow a pattern. Sometimes it happens every day and sometimes the connection works for weeks without any issues. But still, the problem exists and when it kicks in, the system is not able to recover itself. When the connection dies, logread contains the following log entries:
Interestingly, the devices in /dev/ttyUSB* are still there and the logs don't contain anything USB related.
PPPD notices that the serial connection with the modem is broken and shuts down. Simply restarting the interface afterwards (ifdown/ifup, web interface) does not work. The first step of the workaround is to restart the USB modem via software. Fortunately, this Stack Exchange post pointed me into the right direction. A simple unbind followed by a bind on the correct USB port works fine. On unbind, the modem disappears and all /dev/ttyUSB* devices are removed by the kernel. On bind, the kernel re-initializes the modem, does some mode switching and a few seconds later, the /dev/ttyUSB* devices reappear. After this unbind/bind cycle, PPPD is started automatically and Internet connectivity is restored. A list of USB ports may be obtained via:
There is one problem though, I want the reconnection steps to trigger automatically when PPPD detects that the serial link stopped working. Fortunately, PPPD offers various hooks that one can leverage. In my case, the ip-down hook is the correct one. It is called with various arguments and with some environment variables. To enable a ip-down hook on OpenWRT/LEDE, create the directory /etc/ppp/ip-down.d and place your executable ip-down script in this directory. All ip-down scripts in /etc/ppp/ip-down.d executed each time PPPD had a working IP connectivity and is in the process of shutting down. The last part of the puzzle is to only trigger the reconnection when the serial link is faulty. Especially, do not trigger when:
- The user requested to shutdown the interface (ifdown, web interface).
- The USB modem is physically disconnected.
The complete solution is the shell script listed below. It leverages the OpenWRT/LEDE logging system and the fact that PPPD sets the environment variable PPPD_PID. I only need to inspect log entries produced by the currently running PPPD and find log entries that indicate a faulty serial link.