(UDP Transmitter)
(Module parameters)
(23 intermediate revisions by 2 users not shown)
Line 15: Line 15:
 
== socket ==
 
== socket ==
  
This is standard default backend that uses Berkley sockets for sending a traffic. It tries to send the data in non-blocking mode and has special transmission thread to re-send failed packets.
+
This is standard default backend that uses Berkley sockets for sending a traffic. It tries to send the data in non-blocking mode and has special transmission thread to re-send failed packets or offload main thread on high load (> 50% CPU core).
  
 
=== You have to use it if you have ===
 
=== You have to use it if you have ===
Line 24: Line 24:
 
=== Performance ===
 
=== Performance ===
  
Passed performance tests to transmit to 5K connections
+
Passed performance tests to transmit to 10K connections
  
 
=== Configuration ===
 
=== Configuration ===
Line 65: Line 65:
 
=== Configuration ===
 
=== Configuration ===
  
''transmitter = "xdp:<interface name>";''
+
''transmitter = "xdp:<interface name> <module parameters>";''
  
''transmitter = "xdp:eth0";''
+
''transmitter = "xdp:eth0 --core-ratio 64";''
 +
 
 +
==== Module parameters ====
 +
 
 +
All these parameters are optional and override default settings of XDP module
 +
 
 +
<pre>
 +
(-c) --core-ratio          <n>  - ratio between NIC queues and transmitter cores  (instead of default value of 64)
 +
(-l) --buffer-length      <n>  - set workers buffer length to <n> slots (instead of default value of 1024)
 +
(-r) --reference-interface <s>  - reference interface to monitor routes, gateways
 +
</pre>
 +
 
 +
Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use '''--reference-interface''' key to pass the name of VLAN interface (eth0.1 for example)
  
 
== DPDK ==
 
== DPDK ==
Line 88: Line 100:
 
''transmitter = "Modules/DPDK-edge.so:<reference interface> <EAL parameters> [--] <module parameters>";''
 
''transmitter = "Modules/DPDK-edge.so:<reference interface> <EAL parameters> [--] <module parameters>";''
  
''transmitter = "Modules/DPDK-edge.so:eth0 -w 0000:af:00.0 --file-prefix bm --lcores '(0-3)@0' -- --queue-size 64 --batch-size 32";''
+
''transmitter = "Modules/DPDK-edge.so:eth0 -w 0000:af:00.0 --file-prefix bm --lcores '(0-64)@0' -- -c 1 -q 2048 -b 1 -l 4096";''
  
 
* Reference interface is a kernel attached interface used for normal communications (please read about raw mode). DPDK will reuse its IPs and default gateway
 
* Reference interface is a kernel attached interface used for normal communications (please read about raw mode). DPDK will reuse its IPs and default gateway
 
* For EAL parameters please [https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html read this documentation]
 
* For EAL parameters please [https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html read this documentation]
* It will use so many NIC queues as defined amount of slave logical cores minus one in startup parameters.
+
* It will use so many NIC queues as defined amount of slave logical cores minus one and multiplied by core ratio (so ratio means how may queues each core should handle)
 
* The best performance on NUMA machines could be reached by using the same CPU as NIC connected by pinning logical cores via EAL's parameter ''lcores'' and BrandMeister's parameter ''affinity''
 
* The best performance on NUMA machines could be reached by using the same CPU as NIC connected by pinning logical cores via EAL's parameter ''lcores'' and BrandMeister's parameter ''affinity''
 
* Also in [http://doc.dpdk.org/guides/linux_gsg/enable_func.html#running-dpdk-applications-without-root-privileges most cases it requires to run BrandMeister with root privileges], you can do this by overriding systemd configuration ([https://askubuntu.com/questions/659267/how-do-i-override-or-configure-systemd-services beandmeister@.service.d]):
 
* Also in [http://doc.dpdk.org/guides/linux_gsg/enable_func.html#running-dpdk-applications-without-root-privileges most cases it requires to run BrandMeister with root privileges], you can do this by overriding systemd configuration ([https://askubuntu.com/questions/659267/how-do-i-override-or-configure-systemd-services beandmeister@.service.d]):
Line 107: Line 119:
  
 
<pre>
 
<pre>
 +
(-c) --core-ratio    <n>  - ratio between NIC queues and DPDK cores (instead of default value of 64)
 
(-q) --queue-size    <n>  - set PMD queue size to <n> slots (instead of automatically generated)
 
(-q) --queue-size    <n>  - set PMD queue size to <n> slots (instead of automatically generated)
 
(-b) --batch-size    <n>  - set maximum batch size to <n> slots (instead of automatically generated)
 
(-b) --batch-size    <n>  - set maximum batch size to <n> slots (instead of automatically generated)
Line 115: Line 128:
 
(-r) --rs-thresh  <n> |
 
(-r) --rs-thresh  <n> |
 
(-f) --free-thresh <n> |
 
(-f) --free-thresh <n> |
 +
(-s) --software-crc      - force software CRC calculation
 
</pre>
 
</pre>
  
 
== Ixy ==
 
== Ixy ==
  
Ixy is very experimental and light [https://github.com/emmericp/ixy user-space network driver]. At this moment it supports Intel 82599ES family (aka Intel X520) and [https://www.linux-kvm.org/page/Virtio virtio]
+
Ixy is very experimental and light [https://github.com/emmericp/ixy user-space network driver]. At this moment it supports Intel 82599ES family (aka Intel X520) and [https://www.linux-kvm.org/page/Virtio virtio]. Please read [https://github.com/emmericp/ixy Ixy documentation].
 +
 
 +
=== Limitations ===
 +
* Similar to DPDK but experimental
 +
* Single thread
 +
* Software CRC only
 +
 
 +
=== Performance ===
 +
 
 +
Passed performance tests to transmit 10K connections
 +
 
 +
=== Configuration ===
 +
 
 +
''transmitter = "Modules/Dixie.so:[reference interface] <PCI address> <MAC address> [<queue-count> [<buffer-length> <batch-length>]]";''
 +
 
 +
''transmitter = "Modules/Dixie.so:eth0 0000:af:00.0 00:1b:21:a5:a0:2c 1 128 32";'
 +
 
 +
Ixy doesn't provide method to resolve MAC address directly from NIC, so address has to be defined in configuration.
 +
We found Ixy has a bug with queue count > 1 on ixgbe, so we do not recommend to use multiple queue.
  
 
= UDP Receiver =
 
= UDP Receiver =

Revision as of 21:33, 3 October 2020

UDPTX is BrandMeister-own UDP communication library, used to transmit and receive UDP traffic fast. It is very important for BrandMeister to spend less time to send and receive packets, it makes transmission (and finally sound) more smooth.

UDP Transmitter

At this moment BrandMeister provides several backends (options) to send outgoing UDP:

  • socket
  • raw
  • AF_XDP
  • DPDK
  • Ixy


UDPTX.png

socket

This is standard default backend that uses Berkley sockets for sending a traffic. It tries to send the data in non-blocking mode and has special transmission thread to re-send failed packets or offload main thread on high load (> 50% CPU core).

You have to use it if you have

  • non-ethernet interfaces
  • more than one interface for outgoing traffic (such as public + AMPR, or one for IPv4 and another one for IPv6)
  • huge routing tables

Performance

Passed performance tests to transmit to 10K connections

Configuration

transmitter = "socket";

raw

This is fast forwarding backend that uses RAW (PACKET_MMAP) socket of Ethernet interface for sending a traffic. It allows to save up to 50% CPU time and has great compatibility.

Limitations

  • Requires to use a single ethernet interface for BrandMeister's traffic (IPv4 and IPv6, local site connectivity will not work)
  • All traffic will be routed via default gateway (except loopback, see next bullet)
  • Loopback addresses (127.0.0.1 and ::1) are handled by using Berkley sockets

Performance

Passed performance tests to transmit to 20K connections

Configuration

transmitter = "raw:<interface name>";

transmitter = "raw:eth0";

AF_XDP

This is faster forwarding backend that uses AF_XDP socket of Ethernet interface for sending a traffic and in most cases communicates directly with Linux network interface driver.

Limitations

  • The same set of limitations as raw
  • Requires Linux kernel >= 4.18
  • Interface have to be configured to use the same count of TX and RX queues (please read man ethtool)
  • May have compatibility problems (NIC may have no support of XDP)

Performance

Passed performance tests to transmit to 30K connections

Configuration

transmitter = "xdp:<interface name> <module parameters>";

transmitter = "xdp:eth0 --core-ratio 64";

Module parameters

All these parameters are optional and override default settings of XDP module

(-c) --core-ratio          <n>  - ratio between NIC queues and transmitter cores  (instead of default value of 64)
(-l) --buffer-length       <n>  - set workers buffer length to <n> slots (instead of default value of 1024)
(-r) --reference-interface <s>  - reference interface to monitor routes, gateways

Note: in case when you need to use VLANs, you have pass a physical interface (eth0 for example) as interface to bind and use --reference-interface key to pass the name of VLAN interface (eth0.1 for example)

DPDK

This is fastest forwarding backend that uses kernel-bypass NIC driver for sending a traffic. It allows to save much more CPU time due to direct poll communications to the NIC and CRC offload features of some NIC models. In some tests we got up to 75% acceleration. List of supported NIC models can be found here.

Limitations

  • The same set of limitations as raw
  • You have to have separate NIC or virtual detachable NIC port allowed to use for the DPDK transmission
  • Only DPDK port #0 will be used
  • We added support of dpdk-proc-info and dpdk-pdump
  • Very hard in configuration and performance tuning!

Performance

Passed performance tests to transmit to 40K connections

Configuration

transmitter = "Modules/DPDK-edge.so:<reference interface> <EAL parameters> [--] <module parameters>";

transmitter = "Modules/DPDK-edge.so:eth0 -w 0000:af:00.0 --file-prefix bm --lcores '(0-64)@0' -- -c 1 -q 2048 -b 1 -l 4096";

  • Reference interface is a kernel attached interface used for normal communications (please read about raw mode). DPDK will reuse its IPs and default gateway
  • For EAL parameters please read this documentation
  • It will use so many NIC queues as defined amount of slave logical cores minus one and multiplied by core ratio (so ratio means how may queues each core should handle)
  • The best performance on NUMA machines could be reached by using the same CPU as NIC connected by pinning logical cores via EAL's parameter lcores and BrandMeister's parameter affinity
  • Also in most cases it requires to run BrandMeister with root privileges, you can do this by overriding systemd configuration (beandmeister@.service.d):
# /etc/systemd/system/brandmeister@.service.d/override.conf
[Service]
User=root

Module parameters

All these parameters are optional and override default settings of PMD or DPDK module

(-c) --core-ratio    <n>  - ratio between NIC queues and DPDK cores (instead of default value of 64)
(-q) --queue-size    <n>  - set PMD queue size to <n> slots (instead of automatically generated)
(-b) --batch-size    <n>  - set maximum batch size to <n> slots (instead of automatically generated)
(-l) --buffer-length <n>  - set workers buffer length to <n> slots (instead of default value of 2048)
(-p) --pthresh     <n> |
(-h) --hthresh     <n> | PMD specific threshold values:
(-w) --wthresh     <n> | https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html#configuration-of-transmit-queues
(-r) --rs-thresh   <n> |
(-f) --free-thresh <n> |
(-s) --software-crc       - force software CRC calculation

Ixy

Ixy is very experimental and light user-space network driver. At this moment it supports Intel 82599ES family (aka Intel X520) and virtio. Please read Ixy documentation.

Limitations

  • Similar to DPDK but experimental
  • Single thread
  • Software CRC only

Performance

Passed performance tests to transmit 10K connections

Configuration

transmitter = "Modules/Dixie.so:[reference interface] <PCI address> <MAC address> [<queue-count> [<buffer-length> <batch-length>]]";

transmitter = "Modules/Dixie.so:eth0 0000:af:00.0 00:1b:21:a5:a0:2c 1 128 32";'

Ixy doesn't provide method to resolve MAC address directly from NIC, so address has to be defined in configuration. We found Ixy has a bug with queue count > 1 on ixgbe, so we do not recommend to use multiple queue.

UDP Receiver

In reception part UDPTX's driver works in parallel with socket receiver. All it does, is accelerate reception of UDP packets on particular interface.

eBPF + AF_XDP

This is modern method to accelerate UDP reception in BrandMeister Core. It allows to save up to 30% CPU time.

Limitations

  • Requires Linux kernel >= 4.18
  • Interface have to be configured to use the same count of TX and RX queues (please read man ethtool)
  • eBPF handles traffic before iptables
  • Works with only a single instance of BrandMeister Core on the single machine

Configuration

receiver = "Modules/ExpressFilter.o:<interface name>";

receiver = "Modules/ExpressFilter.o:eth0";

eBPF + AF_XDP + XDPHelper

This is method is fully the same as eBPF + AF_XDP but uses small additional daemon XDPHelper to load and share eBPF program between several BrandMeister Core instances. XDPHelper is supplied with BrandMeister Core and starts automatically only when required (thanks to systemd and D-BUS activation). By default XDPHelper uses eBPF program ExpressFilter.o (see xdphelper.service).

Limitations

  • The same list of limitations as in case of eBPF + AF_XDP except support of multiple instances
  • All instances of BrandMeister Core should use the same network interface

Configuration

receiver = "xdp:<interface name>";

receiver = "xdp:eth0";

UDPTX is BrandMeister-own UDP communication library, used to transmit and receive UDP traffic fast. It is very important for BrandMeister to spend less time to send and receive packets, it makes transmission (and finally sound) more smooth.

UDP Transmitter[edit]

At this moment BrandMeister provides several backends (options) to send outgoing UDP:


UDPTX.png

socket[edit]

This is standard default backend that uses Berkley sockets for sending a traffic. It tries to send the data in non-blocking mode and has special transmission thread to re-send failed packets.

You have to use it if you have[edit]

Performance[edit]

Passed performance tests to transmit to 5K connections

Configuration[edit]

transmitter = "socket";

raw[edit]

This is fast forwarding backend that uses RAW (PACKET_MMAP) socket of Ethernet interface for sending a traffic. It allows to save up to 50% CPU time and has great compatibility.

Limitations[edit]

Performance[edit]

Passed performance tests to transmit to 20K connections

Configuration[edit]

transmitter = "raw:<interface name>";

transmitter = "raw:eth0";

AF_XDP[edit]

This is faster forwarding backend that uses AF_XDP socket of Ethernet interface for sending a traffic and in most cases communicates directly with Linux network interface driver.

Limitations[edit]

Performance[edit]

Passed performance tests to transmit to 30K connections

Configuration[edit]

transmitter = "xdp:<interface name>";

transmitter = "xdp:eth0";

DPDK[edit]

This is fastest forwarding backend that uses kernel-bypass NIC driver for sending a traffic. It allows to save much more CPU time due to direct poll communications to the NIC and CRC offload features of some NIC models. In some tests we got up to 75% acceleration. List of supported NIC models can be found here.

Limitations[edit]

Performance[edit]

Passed performance tests to transmit to 40K connections

Configuration[edit]

transmitter = "Modules/DPDK-edge.so:<reference interface> <EAL parameters> [--] <module parameters>";

transmitter = "Modules/DPDK-edge.so:eth0 -w 0000:af:00.0 --file-prefix bm --lcores '(0-3)@0' -- --queue-size 64 --batch-size 32";

# /etc/systemd/system/brandmeister@.service.d/override.conf
[Service]
User=root

Module parameters[edit]

All these parameters are optional and override default settings of PMD or DPDK module

(-q) --queue-size    <n>  - set PMD queue size to <n> slots (instead of automatically generated)
(-b) --batch-size    <n>  - set maximum batch size to <n> slots (instead of automatically generated)
(-l) --buffer-length <n>  - set workers buffer length to <n> slots (instead of default value of 2048)
(-p) --pthresh     <n> |
(-h) --hthresh     <n> | PMD specific threshold values:
(-w) --wthresh     <n> | https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html#configuration-of-transmit-queues
(-r) --rs-thresh   <n> |
(-f) --free-thresh <n> |

Ixy[edit]

Ixy is very experimental and light user-space network driver. At this moment it supports Intel 82599ES family (aka Intel X520) and virtio

UDP Receiver[edit]

In reception part UDPTX's driver works in parallel with socket receiver. All it does, is accelerate reception of UDP packets on particular interface.

eBPF + AF_XDP[edit]

This is modern method to accelerate UDP reception in BrandMeister Core. It allows to save up to 30% CPU time.

Limitations[edit]

Configuration[edit]

receiver = "Modules/ExpressFilter.o:<interface name>";

receiver = "Modules/ExpressFilter.o:eth0";

eBPF + AF_XDP + XDPHelper[edit]

This is method is fully the same as eBPF + AF_XDP but uses small additional daemon XDPHelper to load and share eBPF program between several BrandMeister Core instances. XDPHelper is supplied with BrandMeister Core and starts automatically only when required (thanks to systemd and D-BUS activation). By default XDPHelper uses eBPF program ExpressFilter.o (see xdphelper.service).

Limitations[edit]

Configuration[edit]

receiver = "xdp:<interface name>";

receiver = "xdp:eth0";