tomjepp.uk

APU2 and Debian Buster as a home router

This (terse!) guide shows:

This guide doesn’t cover WiFi in any form - for WiFi I recommend a separate AP that you can position for best coverage.

The router

The PC Engines APU2 is a single-board x86 computer with a reasonable 64-bit quad core CPU, up to 4GB of RAM, storage via mSATA or SD card, and up to 4 Intel NICs.

It draws <10W in normal operation - which allows it to be passively cooled, and it is fairly unusual in x86 platforms in that it doesn’t have any display out - only a serial terminal.

I have the apu2c4 model - which has 4GB of ECC DDR3 and 3x Intel i210AT NICs. I’ve fitted a 250G mSATA SSD for persistent storage.

I got mine from linitx.com - a good UK distributor of PC Engines hardware.

The network setup

I use Andrews & Arnold as my internet provider. They’re a small ISP, but they provide a high quality service with static IPv6 and easy access to static blocks of IPv4. This guide should be reasonably useful for most UK internet connections, however.

I have a FTTP service using PPPoE that’s presented as a gigabit ethernet port - but I’ve also used this router with a FTTC service using a separate FTTC modem with exactly the same setup. If your modem doesn’t support RFC4638 (mini jumbo frames for PPPoE), then you’ll need to adjust the MTU appropriately. A good modem that does is the Openreach branded Huawei HG612.

My network is connected as follows:

Installing Debian

  1. Grab the netinstall image from https://www.debian.org/CD/netinst/. Make sure it’s the amd64 image. Once downloaded, write it to a flash drive. I use dd for this:
     # replace /dev/sdX with the device node for your USB drive
     dd if=debian-10.6.0-amd64-netinst.iso bs=1M of=/dev/sdX
    
  2. Connect up your serial console, and start your terminal emulator using 115200 baud, 8 data bits, no parity, one start bit and one stop bit.
  3. Apply power to the APU2, and use the serial console to boot from USB - press F10 to enter the boot menu, then pick your flash drive from the list.
  4. When booting from the flash drive, you’ll need to provide some extra boot parameters. Add console=ttyS0,115200u8 to the command line, and press enter to boot.
  5. Follow the installer through as normal.

Initial post-install configuration

  1. I set /etc/apt/sources.list to the following:
    deb http://deb.debian.org/debian/ buster main
    deb-src http://deb.debian.org/debian/ buster main
       
    deb http://deb.debian.org/debian-security/ buster/updates main
    deb-src http://deb.debian.org/debian-security/ buster/updates main
       
    deb http://deb.debian.org/debian/ buster-updates main
    deb-src http://deb.debian.org/debian/ buster-updates main
       
    deb http://deb.debian.org/debian/ buster-backports main
    deb-src http://deb.debian.org/debian/ buster-backports main
    

    And then run:

    apt update
    
  2. Install packages we’ll need for this to work:
    # a kernel from buster-backports so we have up to date cake, nftables, and wireguard modules available
    apt install -t buster-backports linux-image-amd64
    
    # flashrom is used for firmware updates for the apu2
    apt install flashrom
    
    # a variety of useful tools for this setup
    apt install chrony curl ethtool irqbalance isc-dhcp-server jq mtr-tiny nftables ppp pppoe radvd unbound
    
    # if you intend to use bridges or vlans (optional)
    apt install bridge-utils vlan
    

    Reboot into the new kernel now.

Networking setup

  1. Set up basic networking - set /etc/network/interfaces to:
     # This file describes the network interfaces available on your system
     # and how to activate them. For more information, see interfaces(5).
    
     source /etc/network/interfaces.d/*
    
     # The loopback network interface
     auto lo
     iface lo inet loopback
    
     # WAN
     auto enp1s0
     iface enp1s0 inet manual
         description wan
         # this allows us to do full 1500 byte frames inside the PPPoE tunnel
         mtu 1508
         # this stops NIC hangs under some 5.x kernels
         up ethtool -K enp1s0 tx off rx off gso off gro off tso off
    
     auto pppoe-aaisp
     iface pppoe-aaisp inet ppp
         provider aaisp
         # ensure the underlying ethernet interface is up and give it some time for everything to settle
         pre-up ifup enp1s0
         pre-up sleep 5
    
     # LAN
     auto enp2s0
     iface enp2s0 inet static
         description lan
         address 192.168.1.1/24
         mtu 1500
         # this does some basic fair queuing
         up tc qdisc add dev enp2s0 root sfq perturb 10
         # this stops NIC hangs under some 5.x kernels
         up ethtool -K enp2s0 tx off rx off gso off gro off tso off
        
     iface enp2s0 inet6 static
         address 2001:8b0:db8::1/64
         mtu 1500
        
     # LIVE
     auto enp3s0
     iface enp3s0 inet static
         description live
         address 192.0.2.1/29
         mtu 1500
         # this does some basic fair queuing
         up tc qdisc add dev enp3s0 root sfq perturb 10
         # this stops NIC hangs under some 5.x kernels
         up ethtool -K enp3s0 tx off rx off gso off gro off tso off
        
     iface enp3s0 inet6 static
         description live
         address 2001:8b0:db8:1::1/64
         mtu 1500
    
  2. Set /etc/ppp/peers/aaisp to:
     user example@a.1
     plugin rp-pppoe.so enp1s0
     noipdefault
     defaultroute
     replacedefaultroute
     hide-password
     lcp-echo-interval 1
     lcp-echo-failure 10
     noauth
     persist
     maxfail 0
     mtu 1500
     noaccomp
     default-asyncmap
     +ipv6
     ipv6cp-accept-local
     ifname pppoe-aaisp
    
     logfile /var/log/ppp-aaisp.log
    

    This sets up PPP for connecting to A&A - it sets up a connection that will:

    • set the IPv4 default route
    • does a LCP ping every second and reconnects after 10 failures
    • uses a MTU of 1500 bytes
    • enables IPv6
    • renames the PPP interface to pppoe-aaisp
  3. Set /etc/ppp/chap-secrets to:
     # Secrets for authentication using CHAP
     # client	server	secret			IP addresses
     example@a.1	*	ExampleSecret
    
  4. Create /etc/ppp/ipv6-ip.d/0000-defaultroute with these contents:
     #!/bin/bash
    
     # add a default v6 route
     ip -6 route add default dev $1
    

    And make it executable:

     chmod +x /etc/ppp/ipv6-ip.d/0000-defaultroute
    
  5. Create /etc/sysctl.d/router.conf with these contents:
     # Enable reverse-path filter (source address spoofing protection)
     net.ipv4.conf.default.rp_filter = 1
     net.ipv4.conf.all.rp_filter = 1
    
     # Enable syncookies
     net.ipv4.tcp_syncookies = 1
    
     # Enable IPv4 forwarding
     net.ipv4.ip_forward = 1
    
     # Enable IPv6 forwarding
     net.ipv6.conf.all.forwarding = 1
    
     # Disable ICMP redirects
     net.ipv4.conf.all.accept_redirects = 0
     net.ipv6.conf.all.accept_redirects = 0
    
  6. Set /etc/resolv.conf to:
     nameserver 127.0.0.1
    
  7. Set /etc/nftables.conf to:
     #!/usr/sbin/nft -f
    
     flush ruleset
    
     table inet firewall {
         chain input {
             type filter hook input priority 0; policy drop;
    
             # allow localhost
             iif "lo" accept
    
             # allow from wan interface
             iifname "enp1s0" accept
    
             # allow return traffic
             ct state established,related accept
    
             # allow ICMP
             meta l4proto { icmp, ipv6-icmp } accept
    
             # allow DHCP from LAN
             iifname { "enp2s0" } udp dport 67 accept
    
             # allow DNS from LAN and live
             iifname { "enp2s0", "enp3s0" } tcp dport 53 accept
             iifname { "enp2s0", "enp3s0" } udp dport 53 accept
    
             # allow NTP
             iifname { "enp2s0", "enp3s0" } udp dport 123 accept
    
             reject with icmpx type admin-prohibited
         }
    
         chain forward-internet-to-lan {
             # allow ICMP
             meta l4proto { icmp, ipv6-icmp } accept
    
             # allow return traffic
             ct state established,related accept
    
             reject with icmpx type admin-prohibited
         }
    
         chain forward-internet-to-live {
             # allow ICMP
             meta l4proto { icmp, ipv6-icmp } accept
    
             # allow all traffic
             accept
         }
    
         chain forward {
             type filter hook forward priority 0; policy drop;
    
             # allow ICMP
             meta l4proto { icmp, ipv6-icmp } accept
    
             iifname "pppoe-aaisp" oifname "enp2s0" jump forward-internet-to-lan
             iifname "pppoe-aaisp" oifname "enp3s0" jump forward-internet-to-live
    
             # allow from lan to internet, live
             iifname "enp2s0" oifname { "pppoe-aaisp", "enp3s0" } accept
    
             # allow return traffic
             ct state established,related accept
    
             reject with icmpx type admin-prohibited
         }
     }
    
     table ip nat {
         chain postrouting {
             type nat hook postrouting priority 0;
    
             ip saddr 192.168.0.0/16 oifname "pppoe-aaisp" masquerade fully-random
         }
     }
    
  8. Enable nftables:
     systemctl enable nftables
    

The easiest way to apply this all is to reboot - once you’ve rebooted you should have basic routing (but no DHCP)!

Setting up network services

DNS

  1. Create /etc/unbound/unbound.conf.d/local.conf with these contents:
     server:
         interface: 0.0.0.0
         interface: ::
         interface-automatic: yes
         prefer-ip6: yes
         access-control: 192.168.0.0/16 allow
         access-control: 192.0.2.0/29 allow
         access-control: 2001:8b0:db8::/48 allow
    
  2. Restart unbound:
     systemctl restart unbound
    

NTP

  1. Edit /etc/chrony/chrony.conf and add this to the end:
     allow 192.168.0.0/16
     allow 192.0.2.0/29
     allow 2001:8b0:db8::/48
    
  2. Restart chrony:
     systemctl restart chrony
    

SLAAC for IPv6

  1. Set /etc/radvd.conf to:
     interface enp2s0
     {
             AdvSendAdvert on;
             MinRtrAdvInterval 3;
             MinRtrAdvInterval 10;
    
             AdvManagedFlag off;
             AdvOtherConfigFlag off;
    
             prefix 2001:8b0:db8::/64
             {
                     AdvOnLink on;
                     AdvAutonomous on;
             };
    
             RDNSS 2001:8b0:db8::1 {
             };
    
             DNSSL your.domain.here {
             };
     };
    

    This enables distributing DNS server and DNS search list via SLAAC, and allows devices to configure their own IP addresses from the advertised prefixes.

  2. Restart radvd:
     systemctl restart radvd
    

DHCP

  1. Edit /etc/dhcp/dhcpd.conf and add this to the end:
     # DNS settings
     option domain-name "your.domain.here";
     option domain-name-servers 192.168.1.1;
    
     # The length of the DHCP lease - 10 minutes
     default-lease-time 600;
     max-lease-time 7200;
    
     # This DHCP server is authoritative for the local network
     authoritative;
    
     subnet 192.168.1.0 netmask 255.255.255.0 {
         option routers 192.168.1.1;
         option domain-name-servers 192.168.1.1;
         option ntp-servers 192.168.1.1;
         range 192.168.1.100 192.168.1.199;
     }
    
  2. Restart the DHCP server:
     systemctl restart isc-dhcp-server
    

QoS

Setting up CAKE on both ingress and egress provides us with a responsive internet connection that provides fair bandwidth allocation where possible - not allowing any one device to abuse the connection and maintaining low-latency conditions for interactive services like SSH.

In addition, A&A provide an API that can be used to get your current line rate. This is very useful for setting up CAKE - especially if you use FTTC!

We use an ifb interface to bring ingress traffic into so we can apply CAKE to it.

  1. Create /usr/local/sbin/qos-setup with the contents:
     #!/bin/bash
    
     # Your A&A control panel credentials
     LOGIN="example@a"
     PASSWORD="ExampleControlPanelPassword"
    
     # Your line ID - this ID is the one at the end of the URL when you view circuit details.
     # for example: https://control.aa.net.uk/editline.cgi?ID=12345 has ID 12345
     SERVICE="12345"
    
     # 8 bytes for pppoe + 4 bytes for BT VLAN. This seems to be correct for FTTP, experimentally.
     # For FTTC you'll want to tweak this!
     OVERHEAD="12"
    
     # load ifb
     modprobe ifb
    
     # clear existing tc qdiscs
     tc qdisc del dev pppoe-aaisp root
     tc qdisc del dev pppoe-aaisp handle ffff: ingress
     tc qdisc del dev ifb0 root
     tc qdisc del dev ifb0 ingress
     ip link set dev ifb0 down
    
     echo "fetching service info"
     INFO="$(curl -s "https://chaos2.aa.net.uk/broadband/info?control_login=${LOGIN}&control_password=${PASSWORD}&service=${SERVICE}" | jq '.info[0]')"
     # these rates are from A&A's PoV, so tx = them transmitting to us, rx = them receiving from us
     INGRESS_RATE=$(echo "$INFO" | jq .tx_rate_adjusted -r)
     EGRESS_RATE=$(echo "$INFO" | jq .rx_rate -r)
     echo "ingress $INGRESS_RATE egress $EGRESS_RATE"
    
     # bring up ifb0
     ip link set dev ifb0 up
    
     # create ingress filter
     echo "create ingress filter"
     tc qdisc add dev pppoe-aaisp handle ffff: ingress
    
     # forward all ingress traffic to ifb0
     echo "forward all ingress traffic to ifb0"
     tc filter add dev pppoe-aaisp parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0
    
     # set up ingress cake
     echo "ingress cake"
     tc qdisc add dev ifb0 root cake bandwidth $INGRESS_RATE nat overhead ${OVERHEAD} ingress dual-dsthost diffserv4 regional
    
     # egress
     echo "egress cake"
     tc qdisc add dev pppoe-aaisp root cake bandwidth $EGRESS_RATE nat overhead ${OVERHEAD} dual-srchost diffserv4 regional
    

    And make it executable:

     chmod +x /usr/local/sbin/qos-setup
    
  2. Create /etc/ppp/ipv6-ip.d/0001-qos with these contents:
     #!/bin/bash
    
     # add a default v6 route
     /usr/local/sbin/qos-setup
    

    And make it executable:

     chmod +x /etc/ppp/ipv6-ip.d/0001-qos
    
  3. Execute it once to test:
     /usr/local/sbin/qos-setup
    

Conclusion

At this point, you should have a nicely working Linux-based router, with working IPv4 and IPv6, DHCP for IPv4, SLAAC for IPv6, DNS, NTP, and QoS!

You might like to add: