ice Linux* Base Driver for the Intel(R) Ethernet Controller 800 Series =============================================================================== November 5, 2020 =============================================================================== Contents -------- - Overview - Identifying Your Adapter - Important Notes - Building and Installation - Command Line Parameters - Additional Features & Configurations - Performance Optimization - Known Issues/Troubleshooting Overview ======== This driver supports kernel versions 3.10.0 and newer. However, some features may require a newer kernel version. The associated Virtual Function (VF) driver for this driver is iavf. Driver information can be obtained using ethtool, lspci, and ip. Instructions on updating ethtool can be found in the section Additional Configurations later in this document. This driver is only supported as a loadable module at this time. Intel is not supplying patches against the kernel source to allow for static linking of the drivers. For questions related to hardware requirements, refer to the documentation supplied with your Intel adapter. All hardware requirements listed apply to use with Linux. This driver supports XDP (Express Data Path) on kernel 4.14 and later and AF_XDP zero-copy on kernel 4.18 and later. Note that XDP is blocked for frame sizes larger than 3KB. Identifying Your Adapter ======================== The driver is compatible with devices based on the following: * Intel(R) Ethernet Controller E810-C * Intel(R) Ethernet Controller E810-XXV For information on how to identify your adapter, and for the latest Intel network drivers, refer to the Intel Support website: http://www.intel.com/support Important Notes =============== Packet drops may occur under receive stress ------------------------------------------- Devices based on the Intel(R) Ethernet Controller 800 Series are designed to tolerate a limited amount of system latency during PCIe and DMA transactions. If these transactions take longer than the tolerated latency, it can impact the length of time the packets are buffered in the device and associated memory, which may result in dropped packets. These packets drops typically do not have a noticeable impact on throughput and performance under standard workloads. If these packet drops appear to affect your workload, the following may improve the situation: 1) Make sure that your system's physical memory is in a high-performance configuration, as recommended by the platform vendor. A common recommendation is for all channels to be populated with a single DIMM module. 2) In your system's BIOS/UEFI settings, select the "Performance" profile. 3) On RHEL 7.x/8.x, use the tuned power management tool to set the "latency-performance" profile. 4) In other operating systems and environments, use the equivalent tool to set the equivalent profile. Configuring SR-IOV for improved network security ------------------------------------------------ In a virtualized environment, on Intel(R) Ethernet Network Adapters that support SR-IOV, the virtual function (VF) may be subject to malicious behavior. Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE 802.1Qbb (priority based flow-control), and others of this type, are not expected and can throttle traffic between the host and the virtual switch, reducing performance. To resolve this issue, and to ensure isolation from unintended traffic streams, configure all SR-IOV enabled ports for VLAN tagging from the administrative interface on the PF. This configuration allows unexpected, and potentially malicious, frames to be dropped. See "Configuring VLAN Tagging on SR-IOV Enabled Adapter Ports" later in this README for configuration instructions. Do not unload port driver if VF with active VM is bound to it ------------------------------------------------------------- Do not unload a port's driver if a Virtual Function (VF) with an active Virtual Machine (VM) is bound to it. Doing so will cause the port to appear to hang. Once the VM shuts down, or otherwise releases the VF, the command will complete. Firmware Recovery Mode ---------------------- A device will enter Firmware Recovery mode if it detects a problem that requires the firmware to be reprogrammed. When a device is in Firmware Recovery mode it will not pass traffic or allow any configuration; you can only attempt to recover the device's firmware. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on Firmware Recovery Mode and how to recover from it. Important notes for SR-IOV, RDMA, and Link Aggregation ------------------------------------------------------ Link Aggregation is mutually exclusive with SR-IOV and RDMA. - If Link Aggregation is active, RDMA peers will not be able to register with the PF, and SR-IOV VFs cannot be created on the PF. - If either RDMA or SR-IOV is active, you cannot set up Link Aggregation on the interface. Bridging and MACVLAN are also affected by this. If you wish to use bridging or MACVLAN with RDMA/SR-IOV, you must set up bridging or MACVLAN before enabling RDMA or SR-IOV. If you are using bridging or MACVLAN in conjunction with SR-IOV and/or RDMA, and you want to remove the interface from the bridge or MACVLAN, you must follow these steps: 1. Remove RDMA if it is active 2. Destroy SR-IOV VFs if they exist 3. Remove the interface from the bridge or MACVLAN 4. Reactivate RDMA and recreate SRIOV VFs as needed Building and Installation ========================= The ice driver requires the Dynamic Device Personalization (DDP) package file to enable advanced features (such as dynamic tunneling, Flow Director, RSS, and ADQ, or others). The driver installation process installs the default DDP package file and creates a soft link ice.pkg to the physical package ice-x.x.x.x.pkg in the firmware root directory (typically /lib/firmware/ or /lib/firmware/updates/). The driver install process also puts both the driver module and the DDP file in the initramfs/initrd image. NOTE: When the driver loads, it looks for intel/ice/ddp/ice.pkg in the firmware root. If this file exists, the driver will download it into the device. If not, the driver will go into Safe Mode where it will use the configuration contained in the device's NVM. This is NOT a supported configuration and many advanced features will not be functional. See "Dynamic Device Personalization" later for more information. To build a binary RPM package of this driver -------------------------------------------- Note: RPM functionality has only been tested in Red Hat distributions. 1. Run the following command, where is the version number for the driver tar file. # rpmbuild -tb ice-.tar.gz NOTE: For the build to work properly, the currently running kernel MUST match the version and configuration of the installed kernel sources. If you have just recompiled the kernel, reboot the system before building. 2. After building the RPM, the last few lines of the tool output contain the location of the RPM file that was built. Install the RPM with one of the following commands, where is the location of the RPM file: # rpm -Uvh or # dnf/yum localinstall NOTES: - To compile the driver on some kernel/arch combinations, you may need to install a package with the development version of libelf (e.g. libelf-dev, libelf-devel, elfutilsl-libelf-devel). - When compiling an out-of-tree driver, details will vary by distribution. However, you will usually need a kernel-devel RPM or some RPM that provides the kernel headers at a minimum. The RPM kernel-devel will usually fill in the link at /lib/modules/'uname -r'/build. To manually build the driver ---------------------------- 1. Move the base driver tar file to the directory of your choice. For example, use '/home/username/ice' or '/usr/local/src/ice'. 2. Untar/unzip the archive, where is the version number for the driver tar file: # tar zxf ice-.tar.gz 3. Change to the driver src directory, where is the version number for the driver tar: # cd ice-/src/ 4. Compile the driver module: # make install The binary will be installed as: /lib/modules//updates/drivers/net/ethernet/intel/ice/ice.ko The install location listed above is the default location. This may differ for various Linux distributions. NOTE: To build the driver using the schema for unified ethtool statistics defined in https://sourceforge.net/p/e1000/wiki/Home/, use the following command: # make CFLAGS_EXTRA='-DUNIFIED_STATS' install NOTE: To compile the driver with ADQ (Application Device Queues) flags set, use the following command, where is the number of logical cores: # make -j CFLAGS_EXTRA='-DADQ_PERF -DADQ_PERF_COUNTERS' install (This will also apply the above 'make install' command.) 5. Load the module using the modprobe command. To check the version of the driver and then load it: # modinfo ice # modprobe ice Alternately, make sure that any older ice drivers are removed from the kernel before loading the new module: # rmmod ice; modprobe ice NOTE: To enable verbose debug messages in the kernel log, use the dynamic debug feature (dyndbg). See "Dynamic Debug" later in this README for more information. 6. Assign an IP address to the interface by entering the following, where is the interface name that was shown in dmesg after modprobe: # ip address add / dev 7. Verify that the interface works. Enter the following, where IP_address is the IP address for another machine on the same subnet as the interface that is being tested: # ping Command Line Parameters ======================= The only command line parameter the ice driver supports is the debug parameter that can control the default logging verbosity of the driver. (Note: dyndbg also provides dynamic debug information.) In general, use ethtool and other OS-specific commands to configure user-changeable parameters after the driver is loaded. Additional Features and Configurations ====================================== ethtool ------- The driver utilizes the ethtool interface for driver configuration and diagnostics, as well as displaying statistical information. The latest ethtool version is required for this functionality. Download it at: https://kernel.org/pub/software/network/ethtool/ NOTE: The rx_bytes value of ethtool does not match the rx_bytes value of Netdev, due to the 4-byte CRC being stripped by the device. The difference between the two rx_bytes values will be 4 x the number of Rx packets. For example, if Rx packets are 10 and Netdev (software statistics) displays rx_bytes as "X", then ethtool (hardware statistics) will display rx_bytes as "X+40" (4 bytes CRC x 10 packets). Viewing Link Messages --------------------- Link messages will not be displayed to the console if the distribution is restricting system messages. In order to see network driver link messages on your console, set dmesg to eight by entering the following: # dmesg -n 8 NOTE: This setting is not saved across reboots. Dynamic Device Personalization ------------------------------ Dynamic Device Personalization (DDP) allows you to change the packet processing pipeline of a device by applying a profile package to the device at runtime. Profiles can be used to, for example, add support for new protocols, change existing protocols, or change default settings. DDP profiles can also be rolled back without rebooting the system. The ice driver automatically installs the default DDP package file during driver installation. NOTE: It's important to do 'make install' during initial ice driver installation so that the driver loads the DDP package automatically. The DDP package loads during device initialization. The driver looks for intel/ice/ddp/ice.pkg in your firmware root (typically /lib/firmware/ or /lib/firmware/updates/) and checks that it contains a valid DDP package file. If the driver is unable to load the DDP package, the device will enter Safe Mode. Safe Mode disables advanced and performance features and supports only basic traffic and minimal functionality, such as updating the NVM or downloading a new driver or DDP package. Safe Mode only applies to the affected physical function and does not impact any other PFs. See the "Intel(R) Ethernet Adapters and Devices User Guide" for more details on DDP and Safe Mode. NOTES: - If you encounter issues with the DDP package file, you may need to download an updated driver or DDP package file. See the log messages for more information. - The ice.pkg file is a symbolic link to the default DDP package file installed by the Linux-firmware software package or the ice out-of-tree driver installation. - You cannot update the DDP package if any PF drivers are already loaded. To overwrite a package, unload all PFs and then reload the driver with the new package. - Only the first loaded PF per device can download a package for that device. You can install specific DDP package files for different physical devices in the same system. To install a specific DDP package file: 1. Download the DDP package file you want for your device. 2. Rename the file ice-xxxxxxxxxxxxxxxx.pkg, where 'xxxxxxxxxxxxxxxx' is the unique 64-bit PCI Express device serial number (in hex) of the device you want the package downloaded on. The filename must include the complete serial number (including leading zeros) and be all lowercase. For example, if the 64-bit serial number is b887a3ffffca0568, then the file name would be ice-b887a3ffffca0568.pkg. To find the serial number from the PCI bus address, you can use the following command: # lspci -vv -s af:00.0 | grep -i Serial Capabilities: [150 v1] Device Serial Number b8-87-a3-ff-ff-ca-05-68 You can use the following command to format the serial number without the dashes: # lspci -vv -s af:00.0 | grep -i Serial | awk '{print $7}' | sed s/-//g b887a3ffffca0568 3. Copy the renamed DDP package file to /lib/firmware/updates/intel/ice/ddp/. If the directory does not yet exist, create it before copying the file. 4. Unload all of the PFs on the device. 5. Reload the driver with the new package. NOTE: The presence of a device-specific DDP package file overrides the loading of the default DDP package file (ice.pkg). RDMA (Remote Direct Memory Access) ---------------------------------- Remote Direct Memory Access, or RDMA, allows a network device to transfer data directly to and from application memory on another system, increasing throughput and lowering latency in certain networking environments. The ice driver supports the following RDMA protocols: - iWARP (Internet Wide Area RDMA Protocol) - RoCEv2 (RDMA over Converged Ethernet) The major difference is that iWARP performs RDMA over TCP, while RoCEv2 uses UDP. For detailed installation and configuration information, see the README file in the RDMA driver tarball. Notes: - Devices based on the Intel(R) Ethernet Controller 800 Series do not support RDMA when operating in multiport mode with more than 4 ports. - You cannot use RDMA or SR-IOV when link aggregation (LAG)/bonding is active, and vice versa. To enforce this, on kernels 4.5 and above, the driver checks for this mutual exclusion. On kernels older than 4.5, the driver cannot check for this exclusion and is unaware of bonding events. Application Device Queues (ADQ) ------------------------------- Application Device Queues (ADQ) allow you to dedicate one or more queues to a specific application. This can reduce latency for the specified application, and allow Tx traffic to be rate limited per application. The ADQ information contained here is specific to the ice driver. For more details, contact your Intel Corp. representative to obtain the E810 ADQ Configuration Guide. Requirements: - Kernel version 4.19.58 or later - Operating system: Red Hat* Enterprise Linux* 7.5+ or SUSE* Linux Enterprise Server* 12+ - The sch_mqprio, act_mirred and cls_flower modules must be loaded. For example: # modprobe sch_mqprio # modprobe act_mirred # modprove cls_flower - The latest version of iproute2 # cd iproute2 # ./configure # make DESTDIR=/opt/iproute2 install - The latest ice driver and NVM image (Note: You must compile the ice driver with the ADQ flag as shown in the "Building and Installation" section.) When ADQ is enabled: - You cannot change RSS parameters, the number of queues, or the MAC address in the PF or VF. Delete the ADQ configuration before changing these settings. - The driver supports subnet masks for IP addresses in the PF and VF. When you add a subnet mask filter, the driver forwards packets to the ADQ VSI instead of the main VSI. - When the PF adds or deletes a port VLAN filter for the VF, it will extend to all the VSIs within that VF. Known issues: - The kernel driver does not support ADQ. You must use the latest out-of-tree driver to use ADQ. - If the application stalls, the application-specific queues may stall for up to two seconds. Configuring only one application per Traffic Class (TC) channel may resolve the issue. - DCB and ADQ cannot coexist. A switch with DCB enabled might remove the ADQ configuration from the device. To resolve the issue, do not enable DCB on the switch ports being used for ADQ. You must disable LLDP on the interface and stop the firmware LLDP agent using the following command: # ethtool --set-priv-flags fw-lldp-agent off - MACVLAN offloads and ADQ are mutually exclusive. System instability may occur if you enable l2-fwd-offload and then set up ADQ, or if you set up ADQ and then enable l2-fwd-offload. - Commands such as 'tc qdisc add' and 'ethtool -L' will cause the driver to close the associated RDMA interface and reopen it. This will disrupt RDMA traffic for 3-5 seconds until the RDMA interface is available again for traffic. - Commands such as 'tc qdisc add' and 'ethtool -L' will clear other tuning settings such as interrupt affinity. These tuning settings will need to be reapplied. When the number of queues are increased using 'ethtool -L', the new queues will have the same interrupt moderation settings as queue 0 (i.e., Tx queue 0 for new Tx queues and Rx queue 0 for new Rx queues). You can change this using the ethtool per-queue coalesce commands. - TC filters may not get offloaded in hardware if you apply them immediately after issuing the 'tc qdisc add' command. We recommend you wait 5 seconds after issuing 'tc qdisc add' before adding TC filters. Dmesg will report the error if TC filters fail to add properly. - Each TC filter bound to a device based on the Intel(R) Ethernet Controller 800 Series consumes a certain number of hardware resources that are shared at the device level. Once resources are assigned, deleting individual filters does not completely free those hardware resources. To free them completely, you must unload and reload the driver. See the ADQ Configuration Guide for more details. To set up the adapter for ADQ, where is the interface in use: 1. Reload the ice driver to remove any previous TC configuration: # modprobe -r ice # modprobe ice 2. Enable hardware TC offload on the interface: # ethtool -K hw-tc-offload on 3. Disable LLDP on the interface, if it isn't already: # ethtool --set-priv-flags fw-lldp-agent off 4. Verify settings: # ethtool -k | grep "hw-tc" # ethtool --show-priv-flags Example output: Private flags for p1p1: link-down-on-close : off fw-lldp-agent : off channel-inline-flow-director : off channel-pkt-inspect-optimize : on To create traffic classes (TCs) on the interface: NOTE: Run all TC commands from the ../iproute2/tc/ directory. 1. Use the tc command to create traffic classes. You can create a maximum of 16 TCs per interface. # tc qdisc add dev root mqprio num_tc map queues hw 1 mode channel shaper bw_rlimit min_rate max_rate Where: num_tc : The number of TCs to use. map : The map of priorities to TCs. You can map up to 16 priorities to TCs. queues : For each TC, @. The max total number of queues for all TCs is the number of cores. hw 1 mode channel: 'channel' with 'hw' set to 1 is a new hardware offload mode in mqprio that makes full use of the mqprio options, the TCs, the queue configurations, and the QoS parameters. shaper bw_rlimit: For each TC, sets the minimum and maximum bandwidth rates. The totals must be equal to or less than the port speed. This parameter is optional and is required only to set up the Tx rates. min_rate : Sets the minimum bandwidth rate limit for each TC. max_rate : Sets the maximum bandwidth rate limit for each TC. You can set a min and max rate together. NOTE: See the mqprio man page and the examples below for more information. 2. Verify the bandwidth limit using network monitoring tools such as ifstat or sar -n DEV [interval] [number of samples] NOTE: Setting up channels via ethtool (ethtool -L) is not supported when the TCs are configured using mqprio. 3. Enable hardware TC offload on the interface: # ethtool -K hw-tc-offload on 4. Apply TCs to ingress (Rx) flow of the interface: # tc qdisc add dev ingress EXAMPLES: See the tc and tc-flower man pages for more information on traffic control and TC flower filters. - To set up two TCs (tc0 and tc1), with 16 queues each, priorities 0-3 for tc0 and 4-7 for tc1, and max Tx rate set to 1Gbit for tc0 and 3Gbit for tc1: # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues 16@0 16@16 hw 1 mode channel shaper bw_rlimit max_rate 1Gbit 3Gbit Where: map 0 0 0 0 1 1 1 1: Sets priorities 0-3 to use tc0 and 4-7 to use tc1 queues 16@0 16@16: Assigns 16 queues to tc0 at offset 0 and 16 queues to tc1 at offset 16 - To set a minimum rate for a TC: # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues 4@0 8@4 hw 1 mode channel shaper bw_rlimit min_rate 25Gbit 50Gbit - To set a maximum data rate for a TC: # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues 4@0 8@4 hw 1 mode channel shaper bw_rlimit max_rate 25Gbit 50Gbit - To set both minimum and maximum data rates together: # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues 4@0 8@4 hw 1 mode channel shaper bw_rlimit min_rate 10Gbit 20Gbit max_rate 25Gbit 50Gbit - To configure TCP TC filters, where: protocol: Encapsulation protocol (valid options are IP and 802.1Q). prio: Priority. flower: Flow-based traffic control filter. dst_ip: IP address of the device. ip_proto: IP protocol to use (TCP or UDP). dst_port: Destination port. src_port: Source port. skip_sw: Flag to add the rule only in hardware. hw_tc : Route incoming traffic flow to this hardware TC. The TC count starts at 0. For example, hw_tc 1 indicates that the filter is on the second TC. - TCP: Destination IP + L4 Destination Port To route TCP traffic with a matching destination IP address and destination port to the given TC: # tc filter add dev protocol ip parent ffff: prio 1 flower dst_ip ip_proto tcp dst_port skip_sw hw_tc 1 - TCP: Destination IP + L4 Source Port To route TCP traffic with a matching destination IP address and source port to the given TC: # tc filter add dev protocol ip parent ffff: prio 1 flower dst_ip ip_proto tcp src_port skip_sw hw_tc 1 - To verify successful TC creation and traffic filtering, after filters are created: # tc qdisc show dev - To view all filters: # tc filter show dev parent ffff: Multiple filters can be added to the device, using the same recipe (and requires no additional recipe resources), either on the same interface or on different interfaces. Each filter uses the same fields for matching, but can have different match values. tc filter add dev protocol ip ingress prio 1 flower ip_proto tcp dst_port $app_port skip_sw hw_tc 1 For example: # tc filter add dev protocol ip ingress prio 1 flower ip_proto tcp dst_port 5555 skip_sw hw_tc 1 Intel(R) Ethernet Flow Director ------------------------------- The Intel Ethernet Flow Director performs the following tasks: - Directs receive packets according to their flows to different queues - Enables tight control on routing a flow in the platform - Matches flows and CPU cores for flow affinity NOTE: An included script (set_irq_affinity) automates setting the IRQ to CPU affinity. NOTE: This driver supports the following flow types: - IPv4 - TCPv4 - UDPv4 - SCTPv4 - IPv6 - TCPv6 - UDPv6 - SCTPv6 Each flow type supports valid combinations of IP addresses (source or destination) and UDP/TCP/SCTP ports (source and destination). You can supply only a source IP address, a source IP address and a destination port, or any combination of one or more of these four parameters. NOTE: This driver allows you to filter traffic based on a user-defined flexible two-byte pattern and offset by using the ethtool user-def and mask fields. Only L3 and L4 flow types are supported for user-defined flexible filters. For a given flow type, you must clear all Intel Ethernet Flow Director filters before changing the input set (for that flow type). NOTE: Flow Director filters impact only LAN traffic. RDMA filtering occurs before Flow Director, so Flow Director filters will not impact RDMA. The following table summarizes supported Intel Ethernet Flow Director features across Intel(R) Ethernet controllers. --------------------------------------------------------------------------- Feature 500 Series 700 Series 800 Series =========================================================================== VF FLOW DIRECTOR Supported Routing to VF Not supported not supported --------------------------------------------------------------------------- IP ADDRESS RANGE Supported Not supported Field masking FILTER --------------------------------------------------------------------------- IPv6 SUPPORT Supported Supported Supported --------------------------------------------------------------------------- CONFIGURABLE Configured Configured Configured INPUT SET per port globally per port --------------------------------------------------------------------------- ATR Supported Supported Not supported --------------------------------------------------------------------------- FLEX BYTE FILTER Starts at Starts at Starts at beginning beginning of beginning of packet payload of packet --------------------------------------------------------------------------- TUNNELED PACKETS Filter matches Filter matches Filter matches outer header inner header inner header --------------------------------------------------------------------------- Flow Director Filters --------------------- Flow Director filters are used to direct traffic that matches specified characteristics. They are enabled through ethtool's ntuple interface. To enable or disable the Intel Ethernet Flow Director and these filters: # ethtool -K ntuple NOTE: When you disable ntuple filters, all the user programmed filters are flushed from the driver cache and hardware. All needed filters must be re-added when ntuple is re-enabled. To display all of the active filters: # ethtool -u To add a new filter: # ethtool -U flow-type src-ip [m ] dst-ip [m ] src-port [m ] dst-port [m ] action Where: - the Ethernet device to program - can be ip4, tcp4, udp4, sctp4, ip6, tcp6, udp6, sctp6 - the IP address to match on - the IPv4 address to mask on NOTE: These filters use inverted masks. - the port number to match on - the 16-bit integer for masking NOTE: These filters use inverted masks. - the queue to direct traffic toward (-1 discards the matched traffic) To delete a filter: # ethtool -U delete Where is the filter ID displayed when printing all the active filters, and may also have been specified using "loc " when adding the filter. EXAMPLES: To add a filter that directs packet to queue 2: # ethtool -U flow-type tcp4 src-ip 192.168.10.1 dst-ip \ 192.168.10.2 src-port 2000 dst-port 2001 action 2 [loc 1] To set a filter using only the source and destination IP address: # ethtool -U flow-type tcp4 src-ip 192.168.10.1 dst-ip \ 192.168.10.2 action 2 [loc 1] To set a filter based on a user-defined pattern and offset: # ethtool -U flow-type tcp4 src-ip 192.168.10.1 dst-ip \ 192.168.10.2 user-def 0x4FFFF action 2 [loc 1] where the value of the user-def field contains the offset (4 bytes) and the pattern (0xffff). To match TCP traffic sent from 192.168.0.1, port 5300, directed to 192.168.0.5, port 80, and then send it to queue 7: # ethtool -U enp130s0 flow-type tcp4 src-ip 192.168.0.1 dst-ip 192.168.0.5 src-port 5300 dst-port 80 action 7 To add a TCPv4 filter with a partial mask for a source IP subnet: # ethtool -U flow-type tcp4 src-ip 192.168.0.0 m 0.255.255.255 dst-ip 192.168.5.12 src-port 12600 dst-port 31 action 12 NOTES: For each flow-type, the programmed filters must all have the same matching input set. For example, issuing the following two commands is acceptable: # ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7 # ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.5 src-port 55 action 10 Issuing the next two commands, however, is not acceptable, since the first specifies src-ip and the second specifies dst-ip: # ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7 # ethtool -U enp130s0 flow-type ip4 dst-ip 192.168.0.5 src-port 55 action 10 The second command will fail with an error. You may program multiple filters with the same fields, using different values, but, on one device, you may not program two tcp4 filters with different matching fields. The ice driver does not support matching on a subportion of a field, thus partial mask fields are not supported. Flex Byte Flow Director Filters ------------------------------- The driver also supports matching user-defined data within the packet payload. This flexible data is specified using the "user-def" field of the ethtool command in the following way: +----------------------------+--------------------------+ | 31 28 24 20 16 | 15 12 8 4 0 | +----------------------------+--------------------------+ | offset into packet payload | 2 bytes of flexible data | +----------------------------+--------------------------+ For example, ... user-def 0x4FFFF ... tells the filter to look 4 bytes into the payload and match that value against 0xFFFF. The offset is based on the beginning of the payload, and not the beginning of the packet. Thus flow-type tcp4 ... user-def 0x8BEAF ... would match TCP/IPv4 packets which have the value 0xBEAF 8 bytes into the TCP/IPv4 payload. Note that ICMP headers are parsed as 4 bytes of header and 4 bytes of payload. Thus to match the first byte of the payload, you must actually add 4 bytes to the offset. Also note that ip4 filters match both ICMP frames as well as raw (unknown) ip4 frames, where the payload will be the L3 payload of the IP4 frame. The maximum offset is 64. The hardware will only read up to 64 bytes of data from the payload. The offset must be even because the flexible data is 2 bytes long and must be aligned to byte 0 of the packet payload. The user-defined flexible offset is also considered part of the input set and cannot be programmed separately for multiple filters of the same type. However, the flexible data is not part of the input set and multiple filters may use the same offset but match against different data. RSS Hash Flow ------------- Allows you to set the hash bytes per flow type and any combination of one or more options for Receive Side Scaling (RSS) hash byte configuration. # ethtool -N rx-flow-hash