ixl iWARP FreeBSD* driver for Intel(R) Ethernet Connection X722 ================================================================ July 31, 2017 Contents ======== - Prerequisites - Building and Installation - Testing - Configuration - Interoperability - Known Issues ================================================================================ Prerequisites ------------- - FreeBSD version 10.3 or 11.0, or later. - Kernel configuration: Please add the following kernel configuration options: include GENERIC options OFED options OFED_DEBUG_INIT options COMPAT_LINUXKPI options SDP options IPOIB_CM options IXL_IW nodevice ixl nodevice ixlv Note: IXL_IW is required for FreeBSD-CURRENT branch. - For the iw_ixl driver to work, an if_ixl driver with iwarp interface is required. The interface is available in if_ixl version 1.7.12 or later. It should be enabled prior to usage, as the setting is switched off by default. To enable iwarp compatibility, add hw.ixl.enable_iwarp=1 to /boot/loader.conf The lan driver can be downloaded from https://downloadcenter.intel.com/download/25160/Ethernet-Intel-Network-Adapter-D river-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD Or search on downloadcenter.intel.com using '40 Gigabit Ethernet Network Connection under FreeBSD'. Newer OS releases contain the if_ixl driver in the ixl driver version 1.7.12-k or later. There are some known issues with the interface on if_ixl-1.7.12. Please use version 1.7.13 or later. - fastreg memory mode in krping needs a patch applied to krping. Refer to the 'Testing' and 'Known Issues' sections for details. - fr test in krping in FreeBSD 10.3 needs a patch applied to krping. Refer to 'Testing' and 'Known Issues' section for details. Building and Installation ------------------------- 1. Untar ixl-.tar.gz and iw_ixl-.tar.gz tar -xf ixl-.tar.gz tar -xf iw_ixl-.tar.gz 2. Install the if_ixl driver: cd ixl-/src directory make make install 3. Install the iw_ixl driver: cd iw_ixl-/src make clean make IXL_DIR=$PATH_TO_IXL/ixl-/src make install 4. Install the man page for the iw_ixl driver by copying the iw_ixl.4.gz file to the directory where manual pages are held on your system, for instance: cp iw_ixl-/doc/iw_ixl.4.gz /usr/share/man/man4/ For in-tree driver if_ixl-1.7.12-k or later, it is sufficient to follow the instruction from point 3 but ensure the correct path to if_ixl source folder is supplied, for instance: IXL_DIR=/usr/src/sys/dev/ixl/ Testing ------- 1. To load iw_ixl driver call: kldload iw_ixl If if_ixl is not already loaded, the system will load it on its own. Please remember to add hw.ixl.enable_iwarp=1 to /boot/loader.conf file prior to if_ixl loading, to ensure the ixl driver has the iwarp interface enabled. 2. To validate the load of the driver check: sysctl -a | grep infiniband a number of sys.class.infiniband should appear provided at least one port of the X722 is up. 3. The source code for krping software is provided with the kernel in /usr/src/sys/contrib/rdma/krping/. To compile the software, change directory to /usr/src/sys/modules/rdma/krping/ and invoke: make clean make make install Note: In FreeBSD 10.3, some changes need to be applied to the krping.c file in order to enable fastreg memory mode and/or fr test. For details please refer to Known Issues section. 4. Start krping server on one machine: echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping 5. Connect client from another machine: echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping ================================================================================ Configuration ------------- The following sysctl options are visible: - hw.iw_ixl.max_ceq determines the maximum number of msix vectors available to the driver for CEQ usage. - hw.iw_ixl.debug defines level of debug messages. - hw.iw_ixl.mpa_version shows the current MPA version used. The max_ceq setting may be changed by adding: hw.iw_ixl.max_ceq=$value to /boot/loader.conf file. The final number of CEQ is evaluated depending on the available msix vectors, number of cpu cores, and hardware limits. If max_ceq=0, the value is ignored. The debug setting may be changed either by adding: hw.iw_ixl.debug=$value to the /boot/loader.conf file or by calling sysctl hw.iw_ixl.debug=$value The mpa_version may be changed by adding: hw.iw_ixl.mpa_version=$value to the /boot/loader.conf file. Interoperability ---------------- To interoperate with Chelsio iWARP devices: 1. Load the ixl driver with parameter mpa_version set to 1. Add the line: hw.iw_ixl.mpa_version=1 to /boot/loader.conf 2. Load Chelsio T4/T5 RDMA driver (iw_cxgb4) with parameters c4iw_max_read_depth set to 63 and dack_mode set to 0. ================================================================================ Known Issues ------------ - Loopback is not supported. - MTU changes are not supported. - IPv6 is not supported. - MW memory mode is not supported. - MR memory mode supports only single buffer. - The function ib_cq_resize is not supported. - The max number of registered cq, qp, pd or mr reported by the device may differ from the actual number of registrations achievable. - A kernel crash may occur when trying to run krping without ensuring that the two machines are able to ping each other. - A kernel crash may occur when trying to load the iw_ixl driver when hw.ixl.enable_iwarp=0 (fixed with if_ixl 1.7.13). - A kernel crash may occur when loading the iw_ixl driver on a card that is supported by if_ixl driver, but does not have iWARP capability (fixed with if_ixl 1.7.13). - On FreeBSD 10.3 you can run krping traffic with a Chelsio T5 card if mpa_version on the X722 card is set to '1'. To set the MPA version, add: hw.iw_ixl.mpa_version=1 to /boot/loader.conf. Setting MPA version on both the X722 and the Chelsio T5 card to '2' may result in a kernel crash. On FreeBSD 11.0, krping traffic is possible with the default settings. - Krping with fastreg memory mode will not work unless some changes are made to krping. To workaround the issue, modify the krping_rdma_rkey function such that, in the case of FASTREG memory mode, the ib_post_send function with &cd->invalidate_wr parameter is not called during the first run of the function - In FreeBSD 10.3, krping fr test is incorrectly implemented. Please consult the krping.c file provided with the FreeBSD 11.0 kernel for the required fixes. Needed changes include, but are not limited to: - The fr test requires the server side (unlike what is written in the comments). function: krping_doit function: krping_fr_test_server function: krping_run_server - The fr test require MR memory mode. function: krping_doit - Wrong device is queried on the client side when checking fastreg support. function: fastreg_supported - Note: the fr itself is designed to work endlessly, however lack of wait_event_interruptible_timeout function support causes the test to do nothing after the first iteration. function: krping_fr_test ------------------- Changes since 0.1.2 ------------------- - Fix for kernel crash on FreeBSD 11.0 when setting ixl3 down. - Fix for krping test with fastreg and server_inv options enabled. -------------------- Changes since 0.0.24 -------------------- - Added patch for 1.7.12 to fix a number of bugs (These changes are available in the ixl driver, version 1.7.13 and later): o kernel crash on loading iw_ixl on XL710 cards. o kernel crash on loading iw_ixl with enable_iwarp tunable set to 0. - MSIX vector utilization has been reworked. - CPU affinity of the CEQ has been implemented. - Added sysctl controls. -------------------- Changes since 0.0.18 -------------------- - No patch for OFED is necessary for FreeBSD 11.0. -------------------- Changes since 0.0.11 -------------------- - Support for FreeBSD 11.0 connection is added. - No patch for OFED is necessary for FreeBSD 10.3. ================================================================================ Support ------- For general information, go to the Intel support website at: http://www.intel.com/support/ or the Intel Wired Networking project hosted by Sourceforge at: http://sourceforge.net/projects/e1000 If an issue is identified with the released source code on a supported kernel with a supported adapter, email the specific information related to the issue to e1000-rdma@lists.sourceforge.net ================================================================================ License ------- This software is available to you under a choice of one of two licenses. You may choose to be licensed under the terms of the GNU General Public License (GPL) Version 2, available from the file COPYING in the main directory of this source tree, or the OpenFabrics.org BSD license below: Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================================================ Trademarks ---------- Intel and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and/or other countries. * Other names and brands may be claimed as the property of others.