irdma RDMA FreeBSD* driver for Intel(R) Ethernet Controller E810 ================================================================ December 21, 2020 Contents ======== - Prerequisites - Building and Installation - Testing - Configuration - Interoperability - Known Issues - Support ================================================================================ Prerequisites ------------- - FreeBSD version 11.4 or 12.1, or later. - Kernel configuration: Please add the following kernel configuration options: include GENERIC options OFED options OFED_DEBUG_INIT options COMPAT_LINUXKPI options SDP options IPOIB_CM nodevice ice - For the irdma driver to work, an if_ice module with RDMA interface is required. The interface is available in if_ice version 0.28.2 or later. The RDMA interface may be turned on or off by using tunable of if_ice module: hw.ice.irdma It may be modified by putting: hw.ice.irdma=1 to /boot/loader.conf file. Reboot is needed for the change to take effect. The RDMA interface is turned on by default (the value is 1). Building and Installation ------------------------- 1. Untar ice-.tar.gz and irdma-.tar.gz tar -xf ice-.tar.gz tar -xf irdma-.tar.gz 2. Install the if_ice driver: cd ice-/ directory make make install 3. Install the irdma driver: cd irdma-/src/ make clean make ICE_DIR=$PATH_TO_ICE/ice-/ make install Testing ------- 1. To load the irdma driver, call: kldload irdma If if_ice is not already loaded, the system will load it on its own. Please check whether the value of sysctl hw.ice.irdma is 1, in case of irdma driver not loading. To change the value put: hw.ice.irdma=1 to /boot/loader.conf file and reboot. 2. To validate the load of the driver, check: sysctl -a | grep infiniband A number of sys.class.infiniband should appear, provided at least one port of the E810 is up. 3. Each interface of the card may work in either iWARP or RoCEv2 mode. The iWARP mode is the default. To enable RoCEv2 compatibility, add: dev.irdma.roce_enable=1 where is a desired ice interface number on which RoCEv2 protocol needs to be enabled to file: /boot/loader.conf for instance: dev.irdma0.roce_enable=0 dev.irdma1.roce_enable=1 will keep iWARP mode on ice0, and enable RoCEv2 mode on interface ice1. To check irdma roce status execute command: sysctl dev.irdma.roce_enable for instance: sysctl dev.irdma2.roce_enable with returned value of '0' indicate the iWARP mode, and the value of '1' indicate the RoCEv2 mode. Note: An interface configured in one mode will not be able to connect to a node configured in another mode. 4. Enable flow control in the ice driver: sysctl dev.ice..fc=3 Enable flow control on the switch your system is connected to. See your switch documentation for details. 5. The source code for krping software is provided with the kernel in /usr/src/sys/contrib/rdma/krping/. To compile the software, change directory to /usr/src/sys/modules/rdma/krping/ and invoke the following: make clean make make install 6. Start krping server on one machine: echo size=64,count=1,port=6601,addr=100.0.0.189,server > /dev/krping 7. Connect client from another machine: echo size=64,count=1,port=6601,addr=100.0.0.189,client > /dev/krping ============================================================================== Configuration ------------- The following sysctl options are visible: - hw.irdma.debug defines level of debug messages. Typical value: 1 for errors only, 0x7fffffff for full debug. - dev.irdma.roce_enable enables RoCEv2 protocol usage on interface. By default iWARP protocol is used. Interoperability ---------------- Known Issues ------------ Support ------- For general information, go to the Intel support website at: www.intel.com/support/ or the Intel Wired Networking project hosted by Sourceforge at: http://sourceforge.net/projects/e1000 If an issue is identified with the released source code on a supported kernel with a supported adapter, email the specific information related to the issue to e1000-rdma@lists.sourceforge.net ================================================================================ License ------- This software is available to you under a choice of one of two licenses. You may choose to be licensed under the terms of the GNU General Public License (GPL) Version 2, available from the file COPYING in the main directory of this source tree, or the OpenFabrics.org BSD license below: Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================================================ Trademarks ---------- Intel and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and/or other countries. * Other names and brands may be claimed as the property of others.