(→Performance) |
(→Performance) |
||
Line 41: | Line 41: | ||
=== Performance === | === Performance === | ||
− | Passed performance tests to transmit to | + | Passed performance tests to transmit to 40K connections |
=== Configuration === | === Configuration === |
UDPTX is BrandMeister-own UDP communication library, used to transmit and receive UDP traffic fast. It is very important for BrandMeister to spend less time to send and receive packets, it makes transmission (and finally sound) more smooth.
At this moment BrandMeister provides several backends (options) to send outgoing UDP:
This is standard default backend that uses Berkley sockets for sending a traffic. It tries to send the data in non-blocking mode and has special transmission thread to re-send failed packets or offload main thread on high load (> 50% CPU core).
Passed performance tests to transmit to 22K connections
transmitter = "socket";
This is fast forwarding backend that uses RAW (PACKET_MMAP) socket of Ethernet interface for sending a traffic. It allows to save up to 50% CPU time and has great compatibility.
Passed performance tests to transmit to 40K connections
transmitter = "raw:<interface name> <module parameters>";
transmitter = "raw:eth0 --count 2";
All these parameters are optional and override default settings of XDP module
(-b) --bypass - bypass QDisc traffic control (PACKET_QDISC_BYPASS) (-c) --count <n> - count of transmitter cores (default is 1)
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)
This is faster forwarding backend that uses AF_XDP socket of Ethernet interface for sending a traffic and in most cases communicates directly with Linux network interface driver.
Passed performance tests to transmit to 30K connections
transmitter = "xdp:<interface name> <module parameters>";
transmitter = "xdp:eth0 --core-ratio 64";
All these parameters are optional and override default settings of XDP module
(-w) --wakeup - use XDP in wakeup mode (XDP_USE_NEED_WAKEUP) (-c) --core-ratio <n> - ratio between NIC queues and transmitter cores (instead of default value of 64) (-q) --queue-count <n> - set transmission queue count manually (instead of all available queues) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 1024) (-r) --reference-interface <s> - reference interface to monitor routes, gateways
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)
This is fastest forwarding backend that uses kernel-bypass NIC driver for sending a traffic. It allows to save much more CPU time due to direct poll communications to the NIC and CRC offload features of some NIC models. In some tests we got up to 75% acceleration. List of supported NIC models can be found here.
Passed performance tests to transmit to 40K connections
transmitter = "Modules/DPDK-edge.so:<reference interface> <EAL parameters> [--] <module parameters>";
transmitter = "Modules/DPDK-edge.so:eth0 -a 0000:af:00.0 --file-prefix bm --lcores '(0-8)@1,3,5,7,9' -- -c 1 -q 2048 -b 1 -l 4096";
# /etc/systemd/system/brandmeister@.service.d/override.conf [Service] User=root
All these parameters are optional and override default settings of PMD or DPDK module
(-c) --core-ratio <n> - ratio between NIC queues and DPDK cores (instead of default value of 64) (-q) --queue-size <n> - set PMD queue size to <n> slots (instead of automatically generated) (-b) --batch-size <n> - set maximum batch size to <n> slots (instead of automatically generated) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 2048) (-p) --pthresh <n> | (-h) --hthresh <n> | PMD specific threshold values: (-w) --wthresh <n> | https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html#configuration-of-transmit-queues (-r) --rs-thresh <n> | (-f) --free-thresh <n> | (-s) --software-crc - force software CRC calculation
Ixy is very experimental and light user-space network driver. At this moment it supports Intel 82599ES family (aka Intel X520) and virtio. Please read Ixy documentation.
Passed performance tests to transmit 10K connections
transmitter = "Modules/Dixie.so:[reference interface] <PCI address> [<queue-count> [<buffer-length> <batch-length>]]";
transmitter = "Modules/Dixie.so:eth0 0000:af:00.0 1 128 32";'
We found Ixy has a bug with queue count > 1 on ixgbe, so we do not recommend to use multiple queue.
In reception part UDPTX's driver works in parallel with socket receiver. All it does, is accelerate reception of UDP packets on particular interface.
This is modern method to accelerate UDP reception in BrandMeister Core. It allows to save up to 30% CPU time.
receiver = "Modules/ExpressFilter.o:<interface name>[ --reference-interface <interface name>]";
receiver = "Modules/ExpressFilter.o:eth0 --reference-interface eth0.1";
All these parameters are optional and override default settings of XDP module
(-w) --wakeup - use XDP in wakeup mode (XDP_USE_NEED_WAKEUP) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 1024) (-r) --reference-interface <s> - reference interface to monitor routes, gateways
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)
This is method is fully the same as eBPF + AF_XDP but uses small additional daemon XDPHelper to load and share eBPF program between several BrandMeister Core instances. XDPHelper is supplied with BrandMeister Core and starts automatically only when required (thanks to systemd and D-BUS activation). By default XDPHelper uses eBPF program ExpressFilter.o (see xdphelper.service).
receiver = "xdp:<interface name>[ --reference-interface <interface name>]";
receiver = "xdp:eth0 --reference-interface eth0.1";
All these parameters are optional and override default settings of XDP module
(-w) --wakeup - use XDP in wakeup mode (XDP_USE_NEED_WAKEUP) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 1024) (-r) --reference-interface <s> - reference interface to monitor routes, gateways
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)
UDPTX is BrandMeister-own UDP communication library, used to transmit and receive UDP traffic fast. It is very important for BrandMeister to spend less time to send and receive packets, it makes transmission (and finally sound) more smooth.
At this moment BrandMeister provides several backends (options) to send outgoing UDP:
This is standard default backend that uses Berkley sockets for sending a traffic. It tries to send the data in non-blocking mode and has special transmission thread to re-send failed packets or offload main thread on high load (> 50% CPU core).
Passed performance tests to transmit to 22K connections
transmitter = "socket";
This is fast forwarding backend that uses RAW (PACKET_MMAP) socket of Ethernet interface for sending a traffic. It allows to save up to 50% CPU time and has great compatibility.
Passed performance tests to transmit to 20K connections
transmitter = "raw:<interface name> <module parameters>";
transmitter = "raw:eth0 --count 2";
All these parameters are optional and override default settings of XDP module
(-b) --bypass - bypass QDisc traffic control (PACKET_QDISC_BYPASS) (-c) --count <n> - count of transmitter cores (default is 1)
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)
This is faster forwarding backend that uses AF_XDP socket of Ethernet interface for sending a traffic and in most cases communicates directly with Linux network interface driver.
Passed performance tests to transmit to 30K connections
transmitter = "xdp:<interface name> <module parameters>";
transmitter = "xdp:eth0 --core-ratio 64";
All these parameters are optional and override default settings of XDP module
(-w) --wakeup - use XDP in wakeup mode (XDP_USE_NEED_WAKEUP) (-c) --core-ratio <n> - ratio between NIC queues and transmitter cores (instead of default value of 64) (-q) --queue-count <n> - set transmission queue count manually (instead of all available queues) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 1024) (-r) --reference-interface <s> - reference interface to monitor routes, gateways
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)
This is fastest forwarding backend that uses kernel-bypass NIC driver for sending a traffic. It allows to save much more CPU time due to direct poll communications to the NIC and CRC offload features of some NIC models. In some tests we got up to 75% acceleration. List of supported NIC models can be found here.
Passed performance tests to transmit to 40K connections
transmitter = "Modules/DPDK-edge.so:<reference interface> <EAL parameters> [--] <module parameters>";
transmitter = "Modules/DPDK-edge.so:eth0 -a 0000:af:00.0 --file-prefix bm --lcores '(0-8)@1,3,5,7,9' -- -c 1 -q 2048 -b 1 -l 4096";
# /etc/systemd/system/brandmeister@.service.d/override.conf [Service] User=root
All these parameters are optional and override default settings of PMD or DPDK module
(-c) --core-ratio <n> - ratio between NIC queues and DPDK cores (instead of default value of 64) (-q) --queue-size <n> - set PMD queue size to <n> slots (instead of automatically generated) (-b) --batch-size <n> - set maximum batch size to <n> slots (instead of automatically generated) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 2048) (-p) --pthresh <n> | (-h) --hthresh <n> | PMD specific threshold values: (-w) --wthresh <n> | https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html#configuration-of-transmit-queues (-r) --rs-thresh <n> | (-f) --free-thresh <n> | (-s) --software-crc - force software CRC calculation
Ixy is very experimental and light user-space network driver. At this moment it supports Intel 82599ES family (aka Intel X520) and virtio. Please read Ixy documentation.
Passed performance tests to transmit 10K connections
transmitter = "Modules/Dixie.so:[reference interface] <PCI address> [<queue-count> [<buffer-length> <batch-length>]]";
transmitter = "Modules/Dixie.so:eth0 0000:af:00.0 1 128 32";'
We found Ixy has a bug with queue count > 1 on ixgbe, so we do not recommend to use multiple queue.
In reception part UDPTX's driver works in parallel with socket receiver. All it does, is accelerate reception of UDP packets on particular interface.
This is modern method to accelerate UDP reception in BrandMeister Core. It allows to save up to 30% CPU time.
receiver = "Modules/ExpressFilter.o:<interface name>[ --reference-interface <interface name>]";
receiver = "Modules/ExpressFilter.o:eth0 --reference-interface eth0.1";
All these parameters are optional and override default settings of XDP module
(-w) --wakeup - use XDP in wakeup mode (XDP_USE_NEED_WAKEUP) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 1024) (-r) --reference-interface <s> - reference interface to monitor routes, gateways
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)
This is method is fully the same as eBPF + AF_XDP but uses small additional daemon XDPHelper to load and share eBPF program between several BrandMeister Core instances. XDPHelper is supplied with BrandMeister Core and starts automatically only when required (thanks to systemd and D-BUS activation). By default XDPHelper uses eBPF program ExpressFilter.o (see xdphelper.service).
receiver = "xdp:<interface name>[ --reference-interface <interface name>]";
receiver = "xdp:eth0 --reference-interface eth0.1";
All these parameters are optional and override default settings of XDP module
(-w) --wakeup - use XDP in wakeup mode (XDP_USE_NEED_WAKEUP) (-l) --buffer-length <n> - set workers buffer length to <n> slots (instead of default value of 1024) (-r) --reference-interface <s> - reference interface to monitor routes, gateways
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)