ixl iWARP FreeBSD* driver for Intel(R) Ethernet Connection X722 ================================================================ July 9, 2019 Contents ======== - Prerequisites - Building and Installation - Testing - Configuration - Interoperability - Known Issues Prerequisites ============= - FreeBSD version 11.2 - Kernel configuration: Please add the following kernel configuration options: include GENERIC options COMPAT_LINUXKPI options IPOIB_CM options IXL_IW nodevice ixl nodevice iavf Note: IXL_IW is required for FreeBSD-CURRENT branch. - For the iw_ixl driver to work, an if_ixl driver with iwarp interface is required. The interface is available in if_ixl version 1.7.12 or later. It should be enabled prior to usage, as the setting is switched off by default. To enable iwarp compatibility, add hw.ixl.enable_iwarp=1 to /boot/loader.conf The lan driver can be downloaded from https://downloadcenter.intel.com/download/25160/Ethernet-Intel-Network-Adapter-D river-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD Or search on downloadcenter.intel.com using '40 Gigabit Ethernet Network Connection under FreeBSD'. Newer OS releases contain the if_ixl driver in the ixl driver version 1.7.12-k or later. There are some known issues with the interface on if_ixl-1.7.12. Please use version 1.7.13 or later. - fastreg memory mode in krping needs a patch applied to krping. Refer to the 'Testing' and 'Known Issues' sections for details. Building and Installation ========================= 1. Untar ixl-.tar.gz and iw_ixl-.tar.gz # tar -xf ixl-.tar.gz # tar -xf iw_ixl-.tar.gz 2. Install the if_ixl driver: # cd ixl-/src directory # make # make install 3. Install the iw_ixl driver: # cd iw_ixl-/src # make clean # make IXL_DIR=$PATH_TO_IXL/ixl-/src # make install 4. Install the man page for the iw_ixl driver by copying the iw_ixl.4.gz file to the directory where manual pages are held on your system. For instance: # cp iw_ixl-/doc/iw_ixl.4.gz /usr/share/man/man4/ For in-tree driver if_ixl-1.7.12-k or later, it is sufficient to follow the instruction from point 3 but ensure the correct path to if_ixl source folder is supplied. For instance: IXL_DIR=/usr/src/sys/dev/ixl/ Testing ------- 1. To load the iw_ixl driver, call: # kldload iw_ixl If if_ixl is not already loaded, the system will load it on its own. Please remember to add hw.ixl.enable_iwarp=1 to /boot/loader.conf file prior to if_ixl loading, to ensure the ixl driver has the iwarp interface enabled. 2. To validate the load of the driver, check: # sysctl -a | grep infiniband A number of sys.class.infiniband should appear, provided at least one port of the X722 is up. 3. The source code for krping software is provided with the kernel in /usr/src/sys/contrib/rdma/krping/. To compile the software, change directory to /usr/src/sys/modules/rdma/krping/ and invoke the following: # make clean # make # make install 4. Start krping server on one machine: # echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping 5. Connect client from another machine: # echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping Configuration ============= The following sysctl options are visible: - hw.iw_ixl.max_ceq determines the maximum number of msix vectors available to the driver for CEQ usage. - hw.iw_ixl.debug defines level of debug messages. - hw.iw_ixl.mpa_version shows the current MPA version used. The max_ceq setting may be changed by adding: hw.iw_ixl.max_ceq=$value to /boot/loader.conf file. The final number of CEQ is evaluated depending on the available msix vectors, number of cpu cores, and hardware limits. If max_ceq=0, the value is ignored. The debug setting may be changed either by adding: hw.iw_ixl.debug=$value to the /boot/loader.conf file or by calling sysctl hw.iw_ixl.debug=$value The mpa_version may be changed by adding: hw.iw_ixl.mpa_version=$value to the /boot/loader.conf file. Interoperability ================ To interoperate with Chelsio iWARP devices: 1. Load the ixl driver with parameter mpa_version set to 1. Add the line: hw.iw_ixl.mpa_version=1 to /boot/loader.conf 2. Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameter dack_mode set to 0. Known Issues ============ - Loopback is not supported. - MTU changes are not supported. - IPv6 is not supported. - MW memory mode is not supported. - MR memory mode supports only single buffer. - The function ib_cq_resize is not supported. - The max number of registered cq, qp, pd or mr reported by the device may differ from the actual number of registrations achievable. - A kernel crash may occur when trying to run krping without ensuring that the two machines are able to ping each other. - A kernel crash may occur when trying to load the iw_ixl driver when hw.ixl.enable_iwarp=0 (fixed with if_ixl 1.7.13). - A kernel crash may occur when loading the iw_ixl driver on a card that is supported by if_ixl driver, but does not have iWARP capability (fixed with if_ixl 1.7.13). - Krping with fastreg memory mode will not work unless some changes are made to krping. To work around the issue, modify the krping_rdma_rkey function such that, in the case of FASTREG memory mode, the ib_post_send function with &cd->invalidate_wr parameter is not called during the first run of the function. Support ======= For general information, go to the Intel support website at: http://www.intel.com/support/ If an issue is identified with the released source code on a supported kernel with a supported adapter, email the specific information related to the issue to e1000-rdma@lists.sourceforge.net Copyright(c) 2017-2019 Intel Corporation. Trademarks ========== Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and/or other countries. * Other names and brands may be claimed as the property of others.