One commonly-stated problem with PPPoE, especially done on x86-based routerslike pfSense and OPNsense is they’re “single-threaded”.
The reason why they’re single-threaded is because of how NICs are designed.These NICs know how to sort IPv4 and IPv6 traffic, but not PPP traffic.combined with both Linux and FreeBSD processing PPPoE in the thread thatprocess packets.
However, using virtualization and bridge interfaces (not PCIe passthrough),you can mitigate this issue and get real multi-threaded PPPoE.
So what do you need?
Some routers work very well. Despite the terrible reputation, FreeBSD-basedrouters like OPNsense and pfSense can work with multi-threaded PPPoE if yourWAN uses a paravirtualized NIC like virtio. VMware and Hyper-V NICs shouldalso work but I haven’t tested those.
However, if you use FreeBSD, you will need the following in your/boot/loader.conf.local:
net.isr.numthreads=Xnet.isr.maxthreads=Xnet.isr.dispatch=deferredIf you’re using a Linux-based router distro, like OpenWrt or VyOS, you willneed to look intoReceive Packet Steering,which does the same thing above on Linux.
Some options to avoid include MikroTik CHR and OpenBSD. Both havesingle-threaded PPPoE due to their design. On MikroTik, you can’t configureReceive Packet Steering that I’m aware of. OpenBSD seems to not have kernelPPPoE or any packet steering option. If you manage to get RPS with CHR, hitme up at neel AT neelc DOT org.
Did I mention you need virtualization? Well, you do. And not only that, youalso need to use a bridge interface on the PPPoE WAN.
You may be tempted to use PCIe passthrough, but if you’re dealing with PPPoE,don’t. If you do, you will still have a single-threaded WAN.
While I’m no Linux kernel expert, using a bridge interface means the host OSwill not process the packet on a single core, but instead just forward it tothe bridge which will then balance it across multiple CPUs.
At least on my host, Rocky Linux 9 with KVM, PPPoE is more-or-less evenlybalanced with OPNsense. However, I have not tested ESXi, Windows, or FreeBSDvirtualization hosts, but I believe it should be the same also.
As a side unrelated note, in the past I was a big FreeBSD user, but moved onover a year ago outside of OPNsense (but it was more desktop-drive for me).
You must use a paravirtualized NIC. On KVM and bhyve, it’s virtio. onESXi it’s `vmxnet``. Hyper-V and Xen also have their own respectiveparavirtualized NICs. Using this allows the guest to have multi-threaded PPPoE.
There are a few things to keep in mind with this setup:
- Host CPU usage will be higher than the guest
- Fortunately, it’s via multiple threads (via
htop)
- You should disable IPv4 and IPv6 on the PPPoE WAN interface
- If your ISP uses 6rd and you wish to use it, as I do with CenturyLink in Seattle, DO NOT USE Linux-based firewalls like OpenWrt, CHR or VyOS.
- This is because 6rd on Linux-based routers is SLOW when compared to FreeBSD.
- However, a Linux host OS will be fine.
Also, not related to the technicalities of this article, but around August 2024,I will be moving to NYC, so I will be unable to test PPPoE after that. This isbecause Verizon FiOS doesn’t use PPPoE, neither does Spectrum or Optimum ifFiOS is unavailable.
I have multi-threaded PPPoE on the following system:
- HPE ProLiant ML110 Gen11
- CPU: Intel Xeon Scalable 4410Y (intial), 5412U (current)
- Host RAM: 32GB (initial), 64GB (current)
- Guest RAM: 1.5GB RAM
- Host Cores: 12 (initial), 24 (current)
- Guest Cores: 3
- Host OS: Rocky Linux 9.3 (Linux kernel 5.14.0-362.8.1.el9_3)
- Guest OS: OPNsense 23.07 (FreeBSD 13.2-p1)
- Virtualization: KVM
- Host NIC: Broadcom 57416 OCP3 (dual 10GB)
- two bridges: one WAN and one LAN
- Guest NIC:
virtio
You could probably use a much smaller RAM allocation if you just do “typical”stuff with a few connections. I have 1.5GB RAM is because of my high-bandwidthTor relays which really crunch the NAT state table: I have ~65000 states at thetime of writing, and it went up to ~78000.
This is what I get:
Look ma, balanced CPU usage.
