ixl FreeBSD* Base Driver for the Intel(R) Ethernet 700 Series of Adapters ================================================================ August 18, 2021 Contents ======== - Overview - Identifying Your Adapter - The VF Driver - Building and Installation - Configuration and Tuning - Known Issues/Troubleshooting Important Notes =============== Firmware Recovery Mode ---------------------- A device will enter Firmware Recovery mode if it detects a problem that requires the firmware to be reprogrammed. When a device is in Firmware Recovery mode it will not pass traffic or allow any configuration; you can only attempt to recover the device's firmware. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on Firmware Recovery Mode and how to recover from it. Overview ======== This file describes the FreeBSD* driver for Intel(R) Ethernet. This driver has been developed for use with all community-supported versions of FreeBSD. For questions related to hardware requirements, refer to the documentation supplied with your Intel Ethernet Adapter. All hardware requirements listed apply to use with FreeBSD. NOTE: 1 Gb devices based on the Intel(R) Ethernet Controller X722 do not support the following features: * SR-IOV * Energy Efficient Ethernet (EEE) * Auto-media detect The associated Virtual Function (VF) driver for this driver is iavf. The associated RDMA driver for this driver is iw_ixl. Identifying Your Adapter ======================== This driver is compatible with devices based on the following: * Intel(R) Ethernet Controller I710 * Intel(R) Ethernet Controller X710 * Intel(R) Ethernet Controller XL710 * Intel(R) Ethernet Network Connection X722 * Intel(R) Ethernet Controller XXV710 * Intel(R) Ethernet Controller V710 For the best performance, make sure the latest NVM/FW is installed on your device and that you are using the newest drivers. For information on how to identify your adapter, and for the latest NVM/FW images and Intel network drivers, refer to the Intel Support website: http://www.intel.com/support SFP+ and QSFP+ Devices: ----------------------- For information about supported media, refer to this document: http://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710- ethernet-controller-feature-matrix.pdf NOTE: Some adapters based on the Intel(R) Ethernet 700 Series only support Intel Ethernet Optics modules. On these adapters, other modules are not supported and will not function. NOTE: For connections based on Intel(R) Ethernet 700 Series, support is dependent on your system board. Please see your vendor for details. NOTE: In all cases Intel recommends using Intel Ethernet Optics; other modules may function but are not validated by Intel. Contact Intel for supported media types. NOTE: In systems that do not have adequate airflow to cool the adapter and optical modules, you must use high temperature optical modules. The VF Driver ============= The VF driver is normally used in a virtualized environment where a host driver manages SRIOV, and provides a VF device to the guest. In the FreeBSD guest, the iavf driver would be loaded and will function using the VF device assigned to it. The VF driver provides most of the same functionality as the core driver, but is actually a subordinate to the host. Access to many controls is accomplished by a request to the host via what is called the "Admin queue." These are startup and initialization events, however; once in operation, the device is self-contained and should achieve near native performance. Some notable limitations of the VF environment: * The PF can configure the VF to allow promiscuous mode, when using iovctl. * Media info is not available from the PF, so it will always appear as auto. Adaptive Virtual Function ------------------------- Adaptive Virtual Function (AVF) allows the virtual function driver, or VF, to adapt to changing feature sets of the physical function driver (PF) with which it is associated. This allows system administrators to update a PF without having to update all the VFs associated with it. All AVFs have a single common device ID and branding string. AVFs have a minimum set of features known as "base mode," but may provide additional features depending on what features are available in the PF with which the AVF is associated. The following are base mode features: - 4 Queue Pairs (QP) and associated Configuration Status Registers (CSRs) for Tx/Rx - iavf descriptors and ring format - Descriptor write-back completion - 1 control queue, with iavf descriptors, CSRs and ring format - 5 MSI-X interrupt vectors and corresponding iavf CSRs - 1 Interrupt Throttle Rate (ITR) index - 1 Virtual Station Interface (VSI) per VF - 1 Traffic Class (TC), TC0 - Receive Side Scaling (RSS) with 64 entry indirection table and key, configured through the PF - 1 unicast MAC address reserved per VF - 16 MAC address filters for each VF - Stateless offloads - non-tunneled checksums - AVF device ID - HW mailbox is used for VF to PF communications (including on Windows) Building and Installation ========================= NOTE: This driver package is to be used only as a standalone archive and the user should not attempt to incorporate it into the kernel source tree. In the instructions below, x.x.x is the driver version as indicated in the name of the driver tar file. 1. Move the base driver tar file to the directory of your choice. For example, use /home/username/ixl or /usr/local/src/ixl. 2. Untar/unzip the archive: # tar xzf ixl-x.x.x.tar.gz This will create the ixl-x.x.x directory. 3. To install man page: # cd ixl-x.x.x # gzip -c ixl.4 > /usr/share/man/man4/ixl.4.gz 4. To load the driver onto a running system: # cd ixl-x.x.x/src # make # kldload ./if_ixl.ko 5. To assign an IP address to the interface, enter the following, where X is the interface number for the device: # ifconfig ixlX 6. Verify that the interface works. Enter the following, where is the IP address for another machine on the same subnet as the interface that is being tested: # ping 7. If you want the driver to load automatically when the system is booted: # cd ixl-x.x.x/src # make # make install Edit /boot/loader.conf, and add the following line: if_ixl_load="YES" Edit /etc/rc.conf, and create the appropriate ifconfig_ixlX entry: ifconfig_ixlX="" Example usage: ifconfig_ixl0="inet 192.168.10.1 netmask 255.255.255.0" NOTE: For assistance, see the ifconfig man page. Configuration and Tuning ======================== Important System Configuration Changes -------------------------------------- - Change the file /etc/sysctl.conf, and add the line: hw.intr_storm_threshold: 0 (the default is 1000) - Best throughput results are seen with a large MTU; use 9706 if possible. The default number of descriptors per ring is 1024. Increasing this may improve performance, depending on your use case. - If you have a choice, run on a 64-bit OS rather than a 32-bit OS. Jumbo Frames ------------ Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU) to a value larger than the default value of 1500. Use the ifconfig command to increase the MTU size. For example, enter the following where X is the interface number: # ifconfig ixlX mtu 9000 To confirm an interface's MTU value, use the ifconfig command. To confirm the MTU used between two specific devices, use: # route get NOTE: The maximum MTU setting for jumbo frames is 9706. This corresponds to the maximum jumbo frame size of 9728 bytes. NOTE: This driver will attempt to use multiple page sized buffers to receive each jumbo packet. This should help to avoid buffer starvation issues when allocating receive packets. NOTE: Packet loss may have a greater impact on throughput when you use jumbo frames. If you observe a drop in performance after enabling jumbo frames, enabling flow control may mitigate the issue. VLANS ----- To create a new VLAN interface: # ifconfig create To associate the VLAN interface with a physical interface and assign a VLAN ID, IP address, and netmask: # ifconfig netmask vlan vlandev Example: # ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixl0 In this example, all packets will be marked on egress with 802.1Q VLAN tags, specifying a VLAN ID of 10. To remove a VLAN interface: # ifconfig destroy NOTE: When the VLAN_HWFILTER flag is enabled on an interface and there aren't enough hardware resources available to filter all the configured VLANs, the driver will disable hardware filtering and allow the operating system or other software to filter VLAN tags. Checksum Offload ---------------- Checksum offloading supports both TCP and UDP packets and is supported for both transmit and receive. Checksum offloading can be enabled or disabled using ifconfig. Both transmit and receive offloading will be either enabled or disabled together. You cannot enable/disable one without the other. To enable checksum offloading: # ifconfig ixlX rxcsum rxcsum6 # ifconfig ixlX txcsum txcsum6 To disable checksum offloading: # ifconfig ixlX -rxcsum -rxcsum6 # ifconfig ixlX -txcsum -txcsum6 To confirm the current setting: # ifconfig ixlX Look for the presence or absence of the following line: options=3 See the ifconfig man page for further information. TSO --- TSO (TCP Segmentation Offload) supports both IPv4 and IPv6. TSO can be disabled and enabled using the ifconfig utility or sysctl. NOTE: TSO requires Tx checksum, if Tx checksum is disabled, TSO will also be disabled. To enable/disable TSO in the stack: # sysctl net.inet.tcp.tso=0 (or 1 to enable it) Doing this disables/enables TSO in the stack and affects all installed adapters. To disable BOTH TSO IPv4 and IPv6, where X is the number of the interface in use: # ifconfig ixlX -tso To enable BOTH TSO IPv4 and IPv6: # ifconfig ixlX tso You can also enable/disable IPv4 TSO or IPv6 TSO individually. Simply replace tso|-tso in the above command with tso4 or tso6. For example, to disable TSO IPv4: # ifconfig ixlX -tso4 To disable TSO IPv6: # ifconfig ixlX -tso6 LRO --- LRO (Large Receive Offload) may provide Rx performance improvement. However, it is incompatible with packet-forwarding workloads. You should carefully evaluate the environment and enable LRO when possible. To enable: # ifconfig ixlX lro It can be disabled by using: # ifconfig ixlX -lro VxLAN Offloads -------------- In FreeBSD 13.0 and later, the ixl driver supports the following offloads in VxLAN packets: - Tx Checksum Offload (tunnel IPv4, transport IPv4, transport TCP/UDP) - Rx Checksum Offload (tunnel IPv4, transport IPv4, transport TCP/UDP) - TCP Segmentation Offload - RSS To enable VxLAN offloads: # ifconfig ixlX vxlanhwcsum vxlanhwtso To disable VxLAN offloads: # ifconfig ixlX -vxlanhwcsum -vxlanhwtso Rx and Tx Descriptor Rings -------------------------- Allows you to set the Rx and Tx descriptor rings independently. The tunables are: hw.ixl.rx_ring_size hw.ixl.tx_ring_size The valid range is 32-4096 in increments of 32. Use kenv to configure the descriptor rings. Changes will take effect on the next driver reload. For example: # kenv hw.ixl.rx_ring_size=1024 # kenv hw.ixl.rx_ring_size=1280 You can verify the descriptor ring size by using the following sysctls: # sysctl dev.ixl..rx_ring_size # sysctl dev.ixl..tx_ring_size RDMA (Remote Direct Memory Access) ---------------------------------- Remote Direct Memory Access, or RDMA, allows a network device to transfer data directly to and from application memory on another system, increasing throughput and lowering latency in certain networking environments. The ixl driver supports the following RDMA protocols: - iWARP (Internet Wide Area RDMA Protocol) For detailed installation and configuration information for RDMA, see the README file in the iw_ixl driver tarball. Link-Level Flow Control (LFC) ----------------------------- Ethernet Flow Control (IEEE 802.3x) can be configured with sysctl to enable receiving and transmitting pause frames for ixl. When transmit is enabled, pause frames are generated when the receive packet buffer crosses a predefined threshold. When receive is enabled, the transmit unit will halt for the time delay specified when a pause frame is received. NOTE: You must have a flow control capable link partner. Flow Control is disabled by default. The ixl driver also supports a "hw.ixl.flow_control" load-time parameter to set the initial flow control configuration on all supported interfaces in a system. You can set the tunable using the kenv command and then reload the driver, or by adding a line to /boot/loader.conf and rebooting the system. Use sysctl to change the flow control settings for a single interface without reloading the driver. The available values for flow control are: 0 = Disable flow control 1 = Enable Rx pause 2 = Enable Tx pause 3 = Enable Rx and Tx pause NOTE: -1 is the expected value for the hw.ixl.flow_control sysctl when the tunable is not set. This tells the driver to use the settings configured by the NVM. Do not use -1 as a configurable setting. Examples: - To set the tunable in the config file, add the following line to /boot/loader.conf and then reboot the system: hw.ixl.flow_control=3 - To set the tunable using kenv, use the following command and then reload the driver: # kenv hw.ixl.flow_control=3 - To remove the tunable and return the system to its default settings: # kenv -u hw.ixl.flow_control - To enable a flow control setting with sysctl: # sysctl dev.ixl..fc=3 - To disable flow control using sysctl: # sysctl dev.ixl..fc=0 NOTE: - The ixl driver requires flow control on both the port and link partner. If flow control is disabled on one of the sides, the port may appear to hang on heavy traffic. - For more information on priority flow control, refer to the "Data Center Bridging (DCB)" section in this README. - The VF driver does not have access to flow control. It must be managed from the host side. Forward Error Correction (FEC) ------------------------------ Allows you to set the Forward Error Correction (FEC) mode. FEC improves link stability, but increases latency. Many high quality optics, direct attach cables, and backplane channels provide a stable link without FEC. NOTE: For devices to benefit from this feature, link partners must have FEC enabled. Use sysctl to configure FEC. (0 to disable, 1 to enable) # sysctl dev.ixl.#.fec.fc_ability # sysctl dev.ixl.#.fec.rs_ability # sysctl dev.ixl.#.fec.fc_requested # sysctl dev.ixl.#.fec.rs_requested # sysctl dev.ixl.#.fec.auto_fec_enabled FW-LLDP (Firmware Link Layer Discovery Protocol) ------------------------------------------------ Use sysctl to change FW-LLDP settings. The FW-LLDP setting is per port and persists across boots. To enable LLDP: # sysctl dev.ixl..fw_lldp=1 To disable LLDP: # sysctl dev.ixl..fw_lldp=0 To check the current LLDP setting: # sysctl dev.ixl..fw_lldp or # sysctl -a|grep lldp NOTE: You must enable the UEFI HII "LLDP Agent" attribute for this setting to take effect. If "LLDP AGENT" is set to disabled, you cannot enable it from the OS. EEE (Energy Efficient Ethernet) ------------------------------- Valid Range: 0-1 0 = Disables EEE 1 = Enables EEE A link between two EEE-compliant devices will result in periodic bursts of data followed by periods where the link is in an idle state. This Low Power Idle (LPI) state is supported at 2.5 Gbps and 5 Gbps link speeds. NOTES: - EEE support requires auto-negotiation. - Both link partners must support EEE. - EEE is not supported on all Intel(R) Ethernet Network devices or at all link speeds. Example: # sysctl dev.ixl.#.eee.enable=[0|1] Disabling physical link when the interface is brought down ---------------------------------------------------------- When the link_active_on_if_down sysctl flag is set to "0", the port's link will go down when the interface is brought down using the 'ifconfig ixlX down' command. To disable link when the interface is down: # sysctl dev.ixl..link_active_on_if_down=0 Speed and Duplex Configuration ------------------------------ You cannot set speed, duplex, or autonegotiation settings. To have your device advertise supported speeds, use the following: # sysctl dev.ixl..advertise_speed= Supported speeds will vary by device. Depending on the speeds your device supports, available speed masks could include: 0x1 - 100 Mbps 0x2 - 1 Gbps 0x4 - 10 Gbps 0x8 - 20 Gbps 0x10 - 25 Gbps 0x20 - 40 Gbps 0x40 - 2.5 Gbps 0x80 - 5 Gbps Known Issues/Troubleshooting ============================ Kernel crash during boot when enable_iwarp is set ------------------------------------------------- If you set the enable_iwarp tunable, the driver may fail to reserve enough MSIX vectors during system boot, which can result in a kernel crash. To avoid this, set the limit_iwarp_msix tunable to the appropriate value, which will limit the number of MSIX vectors reserved for iWARP driver. The default value of this tunable is 64 on FreeBSD 11.0 and later. Set the value of this tunable to 64 if you do not plan to have the driver auto-load on boot. If you experience a kernel crash during the boot, follow these steps to boot into the system: 1. At the boot screen choose option '3. Escape to loader prompt'. 2. Enter one of the following commands: set hw.ixl.enable_iwarp=0 or set hw.ixl.limit_iwarp_msix=1 3. Resume booting process by entering the 'boot' command. 4. Once the system boots, set an appropriate value for the hw.ixl.limit_iwarp_msix tunable in the file /boot/loader.conf. Driver Buffer Overflow Fix -------------------------- The fix to resolve CVE-2016-8105, referenced in Intel SA-00069 , is included in this and future versions of the driver. Network Memory Buffer Allocation -------------------------------- FreeBSD may have a low number of network memory buffers (mbufs) by default. If your mbuf value is too low, it may cause the driver to fail to initialize and/or cause the system to become unresponsive. You can check to see if the system is mbuf-starved by running 'netstat -m'. Increase the number of mbufs by editing the lines below in /etc/sysctl.conf: kern.ipc.nmbclusters kern.ipc.nmbjumbop kern.ipc.nmbjumbo9 kern.ipc.nmbjumbo16 kern.ipc.nmbufs The amount of memory that you allocate is system specific, and may require some trial and error. Also, increasing the following in /etc/sysctl.conf could help increase network performance: kern.ipc.maxsockbuf net.inet.tcp.sendspace net.inet.tcp.recvspace net.inet.udp.maxdgram net.inet.udp.recvspace UDP Stress Test Dropped Packet Issue ------------------------------------ Under small packet UDP stress with the ixl driver, the system may drop UDP packets due to socket buffers being full. Setting the driver Intel Ethernet Flow Control variables to the minimum may resolve the issue. You may also try increasing the kernel's default buffer sizes by changing the values in /proc/sys/net/core/rmem_default and rmem_max Disable LRO when routing/bridging --------------------------------- LRO must be turned off when forwarding traffic. Lower than expected performance ------------------------------- Some PCIe x8 slots are actually configured as x4 slots. These slots have insufficient bandwidth for full line rate with dual port and quad port devices. In addition, if you put a PCIe v4.0 or v3.0-capable adapter into a PCIe v2.x slot, you cannot get full bandwidth. The driver detects this situation and writes one of the following messages in the system log: "PCI-Express bandwidth available for this card is not sufficient for optimal performance. For optimal performance a x8 PCI-Express slot is required." or "PCI-Express bandwidth available for this device may be insufficient for optimal performance. Please move the device to a different PCI-e link with more lanes and/or higher transfer rate." If this error occurs, moving your adapter to a true PCIe v3.0 x8 slot will resolve the issue. Fiber optics and auto-negotiation --------------------------------- Modules based on 40GBASE-SR4, 25GBASE-SR, active optical cable (AOC), and active copper cable (ACC) do not support auto-negotiation per the IEEE specification. To obtain link with these modules, you must turn off auto-negotiation on the link partner's switch ports. Support ======= For general information, go to the Intel support website at: http://www.intel.com/support/ If an issue is identified with the released source code on a supported kernel with a supported adapter, email the specific information related to the issue to freebsd@intel.com Copyright(c) 2014 - 2021 Intel Corporation. Trademarks ========== Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and/or other countries. * Other names and brands may be claimed as the property of others.