0€

CarTFT.com: Professional CarPCs and Displays

«   Change to: MiniPC.de – Small, flexible, powerful

Sprache ändern:

Deutsch
Shop for mobile PC- and GPS-solutions

LTE/4G/5G modems in Linux in QMI-mode


Einführung
In a project, we took some 5G M.2 modem cards and tested to what extent they can be used under Linux. As the main router, we use a pfSense virtualized under Xenserver/XCP-NG that is connected to a fiber optic connection. What was missing was a backup line in the form of a 4G/5G connection. Unfortunately, compatibility beyond slow PPP connections under pfSense (FreeBSD-based) is terrible or almost nonexistent. Therefore, a Linux router must be used for the 4G/5G/LTE connection. This can then act as a "feeder" for pfSense or at the same time as an independent router for a subnet or other network areas. Here we chose "OpenWRT" as the router software because it requires hardly any resources (not even 1GB RAM and 1GB HDD), offers very good support for modem cards and extensive setting and expansion options.

In terms of hardware, it is of course difficult to get M.2 modems into the virtual machine in a virtualized environment. To do this, we use a PCIe USB3 interface card and pass this on to the OpenWRT VM via PCI passthrough. We connect our USB3 M.2 adapter box to the USB card. The advantage is that the modem is then outside the server/system, the SIM card is also inserted here and the 4G/5G antennas are connected. In our case, we also used a 15m USB3 active extension to get the USB3 modem box close to the 4G/5G roof antenna, as it can often be difficult to get a server close to an antenna. With a 15m USB cable (longer lengths are probably possible, or even in optical AOC cable version) and a 5-10m antenna cable, you have enough room to position the modem box appropriately.

Verwendete Hardware
In our tests we used the following modem cards:
Integrated into the USB3-M.2-Adapterbox.

All of these modems offer a QMI interface under Linux. In principle, our instructions can of course also be reproduced in a non-virtualized bare metal environment, i.e. directly on an OpenWRT system or on any Linux system.

Due to the location, unfortunately only 4G is possible at our test location so far, so 5G could not be tested. The 4G speeds achieved are already very good, however. We use a "Wittenberg LAT 60 Duo" as an antenna. The Telekom antenna mast aimed at it is 3.2 km away as the crow flies.

Installation OpenWRT (unter Xenserver/XCP-NG)
OpenWRT is not available as an ISO but only as a system image. This means that an intermediate step is required for installation.
We first create a new virtual machine with these settings : 

VM template : Other install media

Install from ISO : Here we use any Linux system with rescue mode. For example, Debian or Ubuntu.

Boot Mode : BIOS Boot

vCPUs : 1 (possibly also 2, but no additional performance is to be expected)

Memory : 1GB (possibly only 512 MB)

Virtual disk : 1GB (possibly only 512 MB)

Network : e.g. 1x LAN and 1x internal interface as a quasi WAN port for pfSense
We then boot the system into rescue mode of a Linux boot CD.



Now we download the latest OpenWRT image, unpack it and copy it to the virtual hard disk :

wget https://downloads.openwrt.org/releases/23.05.5/targets/x86/64/openwrt-23.05.5-x86-64-generic-ext4-combined-efi.img.gz

gunzip openwrt-*.img.gz

dd if=openwrt-23.05.5-x86-64-generic-ext4-combined.img bs=1M of=/dev/xvda (dauert ca. 1 Sekunde)
Now boot the system into a partition helper such as gparted or pmagic in order to be able to use the available HDD space. (Caution: adjust the boot order so that the DVD is booted before the HDD). In gparted, the message “The backup GPT table is corrupt, but the primary appears OK, so that will be used” appears. This can be confirmed with “Ok”. Confirm the message “Not all of the space available to /dev/xvda appears to be used, you can fix the GPT to use all of the space (an extra xxxx blocks) or continue with the current setting?” with “Fix”.
Then enlarge the last partition /dev/xvda2 to the maximum available storage space and save/confirm.


Shut down the system.
Now we assign the PCI USB3 controller exclusively to the virtual machine.
On the xenserver/xcp-ng console (e.g. via SSH) we first find out the PCI device number of the controller.
This usually works fastest with this command :

# lspci |grep USB

00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05)

00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05)

03:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)

42:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)

In our case, the USB controller card is 42:00.0
We will now remove this from the system in Xenserver/XCP-NG.

# /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:42:00.0) "
Then the system needs to be restarted.
Then we assign the PCI card exclusively to the OpenWRT VM.

# xe vm-param-set other-config:pci=0/0000:42:00.0 uuid=[UUID-der-OpenWRT-VM]

Now we connect the USB3 M.2 modem adapter to the USB PCI controller card and boot OpenWRT.

In OpenWRT, a few packages are first installed via SSH that are necessary for modem operation :

# opkg update

# opkg install kmod-usb-net-qmi-wwan uqmi luci-proto-qmi kmod-usb-serial-option picocom

The modem should now be visible in the system:

# ls -l /dev/cdc* (->/dev/cdc-wdm0)

The Simcom modem looks like this (# dmesg):

[ 4561.625960] usb 2-1.2.2.2: new high-speed USB device number 12 using xhci_hcd

[ 4561.846720] option 2-1.2.2.2:1.0: GSM modem (1-port) converter detected

[ 4561.862118] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB0

[ 4561.876032] option 2-1.2.2.2:1.1: GSM modem (1-port) converter detected

[ 4561.892369] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB1

[ 4561.909181] option 2-1.2.2.2:1.2: GSM modem (1-port) converter detected

[ 4561.922440] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB2

[ 4561.938091] option 2-1.2.2.2:1.3: GSM modem (1-port) converter detected

[ 4561.958481] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB3

[ 4561.980857] option 2-1.2.2.2:1.4: GSM modem (1-port) converter detected

[ 4562.005352] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB4

[ 4562.022201] qmi_wwan 2-1.2.2.2:1.5: cdc-wdm0: USB WDM device

[ 4562.038126] qmi_wwan 2-1.2.2.2:1.5 wwan0: register 'qmi_wwan' at usb-0000:00:06.0-1.2.2.2, WWAN/QMI device, 82:d6:60:94:70:6e

Quectel : 

[   11.810573] qmi_wwan 2-1.2.2.2:1.4: cdc-wdm0: USB WDM device

[   11.823345] qmi_wwan 2-1.2.2.2:1.4 wwan0: register 'qmi_wwan' at usb-0000:00:06.0-1.2.2.2, WWAN/QMI device, c2:ff:5d:91:66:c3

[   11.849161] usbcore: registered new interface driver qmi_wwan

[   11.867334] usbcore: registered new interface driver cdc_mbim

[   11.880668] usbcore: registered new interface driver option

[   11.893929] usbserial: USB Serial support registered for GSM modem (1-port)

[   11.913040] option 2-1.2.2.2:1.0: GSM modem (1-port) converter detected

[   11.931437] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB0

[   11.960033] option 2-1.2.2.2:1.1: GSM modem (1-port) converter detected

[   11.989900] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB1

[   12.007687] option 2-1.2.2.2:1.2: GSM modem (1-port) converter detected

[   12.023923] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB2

[   12.040403] option 2-1.2.2.2:1.3: GSM modem (1-port) converter detected

[   12.058821] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB3

MeiG : 

[ 2554.558037] usb 2-1.2.2.2: new high-speed USB device number 12 using xhci_hcd

[ 2555.586917] usb 2-1.2.2.2: USB disconnect, device number 12

[ 2726.117623] usb 2-1.2.2.2: new high-speed USB device number 13 using xhci_hcd

[ 2727.314051] usb 2-1.2.2.2: USB disconnect, device number 13

[ 2932.597482] usb 2-1.2.2.2: new high-speed USB device number 14 using xhci_hcd

[ 2932.821444] option 2-1.2.2.2:1.0: GSM modem (1-port) converter detected

[ 2932.840390] usb 2-1.2.2.2: GSM modem (1-port) converter now attached to ttyUSB0

[ 2932.863235] qmi_wwan 2-1.2.2.2:1.5: cdc-wdm0: USB WDM device

[ 2932.879384] qmi_wwan 2-1.2.2.2:1.5 wwan0: register 'qmi_wwan' at usb-0000:00:06.0-1.2.2.2, WWAN/QMI device, 06:dd:e6:da:53:0a

In the OpenWRT web interface we now add the modem connection (in our example with Telekom data) :
Network > Add new interface > Name : 5GMODEM, Protocol : QMI Cellular, APN: internet.telekom, Auth : PAP (telekom/tm), PDP : Ipv4

Please note that you should switch off the PIN query of the SIM card before use.
To do this, temporarily install the SIM card in any mobile phone and switch off the PIN.

OpenWRT should now be able to establish the connection automatically. This worked for us straight away with the Simcom and MeiG modem
. Only Quectel required a little more work :

# picocom /dev/ttyUSB2 (vorher picocom installieren mit opkg install picocom)

AT+QCFG="usbnet" (Ausgabe sollte 0 sein für QMI, ansonsten setzen mit : AT+QCFG="usbnet",0)

AT+CGDCONT? (Ausgabe sollte „at+cgdcont=1,“IP“,“internet.telekom““ ergeben, sonst setzen mit :

„+CGDCONT: 1,"IP","internet.telekom","0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0",0,0,0,0,,,,,,,,,"",,,,0“)
Then the output looks like this :

+CGDCONT: 1,"IP","internet.telekom","0.0.0.0",0,0,0,0,,,,,,,,,"",,,,0

Reset : AT+cfun=1,1
Then OpenWRT “5GMODEM” interface “Restart” button
And after a short time a virtual dynamic interface (DHCP client) with IP address appears or the IP address directly in the modem interface.



So that the modem connection can also be routed (NAT), we assign the “5G Modem” interface to the “wan” firewall zone and the “lan” and possibly “wan” interfaces (as a connection to other routers) to the “lan” zone. DHCP servers may also need to be activated on the “lan” interface (depending on local conditions) and ideally also on “wan” so that OpenWRT and pfSense can negotiate the addresses directly and automatically.

Under Network > Firewall are the firewall settings, which are kept quite simple for our purposes. With 4G/5G connections, there are usually no incoming connections or port forwarding.


If you now use the IP address of OpenWRT as a gateway in your LAN, you can access the Internet via the modem.
We were able to achieve the following speeds with 4G in our setup:


Simcom : Ping 18ms, 271,79 Mbit Download, 61,21 Mbit Upload
Quectel : Ping 19ms, 277,72 Mbit Download, 60,91 Mbit Upload
MeiG : Ping 21ms, 178,23 Mbit Download, 68,89 Mbit Upload


pfSense Einstellungen
In pfSense, you first create another WAN interface. If a DHCP server is set up on the interface in OpenWRT, then you only need to set DHCP client on the pfSense side.
We then create a group under System > Routing > Gateway Groups. So that the modem takes over if the main line fails, we set the trigger level to "Packet Loss", the main gateway to "Tier 1" and the modem WAN to "Tier 2".


Under System > Routing > Gateways we now set the "Default gateway Ipv4" to this gateway group.
Caution: To monitor the gateway you should use a monitor IP such as "1.1.1.1" on the new WAN port.
Google IPs are not recommended as they may restrict (ping) access.


With these settings, the modem gateway is switched over if the main line fails.
Of course, this is only useful for outgoing connections. Because port forwarding and static IP are traditionally difficult or cost extra for 4G/5G connections.

A nice side function of this setup is that in pfSense, the modem access can now be issued directly as a gateway to certain clients using static DHCP mapping.

The setup in the form shown here (OpenWRT before pfSense) is of course a double NAT scenario. This is not so easy to avoid when using modem cards, however. In this case, you would have to use M.2-to-Ethernet adapters which, however, must still be considered experimental at the moment. It is also the case that with sufficiently high-performance router hardware, you cannot really notice any difference compared to a single NAT implementation.

We are happy to answer any questions/remarks/comments.

Oliver Aigner (07.11.2024)