Skip to content

SRV‐M QoS

Jump to bottom
rgaudin edited this page Jan 28, 2026 · 1 revision

SRV-M, our master server and mirror only has 1Gbps bandwidth.

It serves the following services, in order of importance:

  • Rsync source for all our official mirrors
    • whitelisted
    • anonymous syncing (1 slot for kiwix, 1 slot for openZIM)
  • HTTP download mirror
    • single source for kiwix and openZIM binary releases and nightlies (not mirrored)
    • mirror (amongst others but gets it first) of ZIM files
  • FTP download mirror
    • kiwix release/nightlies
    • openZIM release/nightlies
    • anonymous only (25 slots)

QoS Strategy

QoS SRV-M
  • no limit on those services if there's available bandwidth
  • limit to 10mbps for any traffic outside those services (so it can be identified and classified if legitimate)
  • in case it's saturated with downloads on all those services at the same time (theoretical) we'd get:
    • 660mbps (82MB/s) bandwidth for rsync
    • 240mbps (30MB/s) bandwidth for HTTP
    • 80mbps (10MB/s) bandwidth for FTP
    • 10mbps for the rest
    • always a working 5mbps for SSH and k8s internals
  • all lanes have an SFQ discipline to ensure users within them are treated fairly.

Setup

  • setup-qos.sh run by master-qos.service systemd service on startup.
  • relies on iptables for marking traffic (we flag by port)
  • tc then assigns is to our lanes
  • we identify IP-based traffic using tc filter and not iptables so we can use prio.

At any time, you can reset to what's in the script with systemctl restart master-qos.service

Disable

systemctl stop master-qos.service

or manually

# clear QoS root (everything goes away)
tc qdisc del dev eno1 root

# clear POSTROUTING chain in mangle table of the firewall
iptables -t mangle -F POSTROUTING

Check that it's working

# list qdiscs
tc qdisc show dev eno1
# list classes
tc -graph class show dev eno1
# list filters
tc filter show dev eno1

# show stats per class.
tc -s -d class show dev eno1

Temporarily limit one lane

# limit downloads all-together to 10mbit
tc class replace dev eno1 parent 1: classid 1:30 htb rate 10mbit ceil 10mbit prio 3

Temporarily increase speed for an IP

By assigning it to a specific lane

# assigns any traffic from 196.200.90.182 to 1:10 lane
# notice the 1 and 2 numbers indicating to place those rules on top of the chain.
iptables -t mangle -I POSTROUTING 1 -o eno1 -p tcp -d 196.200.90.182 -j MARK --set-mark 10
iptables -t mangle -I POSTROUTING 2 -o eno1 -p tcp -d 196.200.90.182 -j RETURN

Remove rule once done

# check the rules numbers first
iptables -t mangle -L POSTROUTING --line-numbers -n
# remove the first two rules (assuming from previous example)
iptables -t mangle -D POSTROUTING 2
iptables -t mangle -D POSTROUTING 1

Create a new lane

Prio0 was intentionnaly left so you can create ones for hot-fixes

tc class add dev eno1 parent 1:1 classid 1:90 htb rate 1gbit ceil 1gbit prio 0

You would want to combine this with other tweaks like limiting others or assigning one's traffic to this lane

References

Clone this wiki locally