Q-Logic IB0056101-00 G.02 manual

Draft IB0056101-00 G .02 Preliminary QLogic HCA and QLogic OFED Sof tware Inst all Guide QLogic OFED V ersion 1.4[...]

Draft Page ii Preliminary IB0056101-00 G .02 S QLogic HCA and QLogic OFED Software Inst all Guide QLogic OFED V ersion 1.4 Information fu rnished in this manual is belie ved to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patent s or other right s of third part ies which m[...]

Draft QLogic HCA and QLogic OFED Sof tware Inst all Guide QLogic OFED V ersion 1.4 IB0056101-00 G .02 Preliminary Page iii S Replace EM64T with 64-bit Intel Xeon. “Supported Linux Distributions” on p age 5-1 Added more inform ation on VNIC inter face. “OpenSM” on page 6 -3 S plit the Install section into three separate sec- tions. “Softwa[...]

Draft Page iv Preliminary IB0056101-00 G .02 S QLogic HCA and QLogic OFED Software Inst all Guide QLogic OFED V ersion 1.4 Sections rearranged, renam ed, and expanded due to multiple install meth ods. Previous sections: Installing the InfiniPath and Open fabrics RPMs Downloading and Unp acking the InfiniPath and OpenFabrics Softwar e “Choose the [...]

Draft QLogic HCA and QLogic OFED Sof tware Inst all Guide QLogic OFED V ersion 1.4 IB0056101-00 G .02 Preliminary Page v S New Appendix for Write Combining. MTRR set- tings information moved her e. “Write Co mbining” on pag e B-1 Minor wording change to introductio n section. “Software Inst allation” on p age 5-1 Added new issue in Troubles[...]

Draft Page vi Preliminary IB0056101-00 G .02 S QLogic HCA and QLogic OFED Software Inst all Guide QLogic OFED V ersion 1.4 Changed title of T roubleshooting issue. W as “ ipath_ether Configuration Setup Gener- ates Error” “Outdated ipath_ether Configuration Setup Generates Error” on page A-5 Change main config uration file from /etc/sys- co[...]

Draft QLogic HCA and QLogic OFED Sof tware Inst all Guide QLogic OFED V ersion 1.4 IB0056101-00 G .02 Preliminary Page vii S Adapter Setting s section name ch anged. Also added new bullet point “Check the PCIe bus wid th” “Adapter and Other Settin gs” on page 7-1 Updated tables of RPMs. Moved a ll documentation RPMs together . New table for[...]

Draft Page viii Preliminary IB0056101-00 G .02 S QLogic HCA and QLogic OFED Software Inst all Guide QLogic OFED V ersion 1.4 Notes[...]

Draft IB0056101-00 G .02 Preliminary ix T able of Content s 1 Introduction Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .[...]

Draft x Preliminary IB0056101-00 G .02 QLogic HCA and QLogic OFED Software Inst all Guide QLogic OFED V ersion 1.4 S Hardware Installation for QLE724 0, QLE7280, or QLE7 140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 Dual Adapter Installatio n. . . . . . . . . . . . . . . . . . . .[...]

Draft IB0056101-00 G .02 Preliminary xi QLogic HCA and QLogic OFED Sof tware Inst all Guide QLogic OFED V ersion 1.4 A Uninstalling So ftware with Rocks o r Platform OCS . . . . . . . . . . . . . . . 5-25 Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25 6 Configuring Dri vers and Services InfiniPath[...]

Draft xii Preliminary IB0056101-00 G .02 QLogic HCA and QLogic OFED Software Inst all Guide QLogic OFED V ersion 1.4 S ibsrpdm Command Hangs When T wo HCAs are Installed but Only Unit 1 is Connected to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5 Outdated ipath_et her Configuration Setup Generates Error . . . .[...]

Draft IB0056101-00 G .02 Preliminary xiii QLogic HCA and QLogic OFED Sof tware Inst all Guide QLogic OFED V ersion 1.4 A List of Figures Figure Page 4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-2 QLogic QLE7140 Card with Riser , T op View . . . . . . . . . . . . . . . . . . . . . .[...]

Draft xiv Preliminary IB0056101-00 G .02 QLogic HCA and QLogic OFED Software Inst all Guide QLogic OFED V ersion 1.4 S Notes[...]

Draft IB0056101-00 G .02 Preliminary 1-1 1 Introduction This chapter describes the content s, intended au dience, and organization of the QLogic HCA and QLogic OFED Software In stall Guide . The QLogic HCA and QLogic OF ED Sof tware Install Guid e contains instructions for installing t he QLogic Host Channel A dapte rs (HCAs) and the QLogic InfiniP[...]

Draft Introduction Overview 1-2 Preliminary IB0056101-00 G .02 S  Section 3, S tep-by-S tep Inst allation Checklist , provides a high-level overview of the hardware and sof tware installation procedures.  Section 4, Hardware Inst allation , includes instructio ns for insta lling the QLogic QLE7140, QLE7240, QLE7280, QHT7040, an d QHT7140 HCAs[...]

Draft Introduction Interoperability IB0056101-00 G .02 Preliminary 1-3 A QLogic OFED OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) runn ing compatible Ope nFabrics releases. There are several options fo r subnet management in your cluster:  Use the embedded Subnet Manager (SM) in one or more [...]

Draft Introduction Documenta tion 1-4 Preliminary IB0056101-00 G .02 S Document ation The product document ation includes:  The QLogic HCA and QLogic OF ED Soft ware Install Guid e  The QLogic HCA and QLogic OF ED Sof tware Users Guide  The QLogic InfiniBand So ftware Inst all Guide  The QLogic ULP and T ools Reference Guide (OFED+ User[...]

Draft Introduction Contact Information IB0056101-00 G .02 Preliminary 1-5 A Cont act Information Support Headquart ers QLogic Corporation 4601 Dean Lakes Blvd Shakopee, MN 55379 USA QLogic Web Site www .qlogic.com T echnical Support Web Site support.qlogic.com T echnical Support Email support@qlogic.com T e chnical Training Email tech.training@qlog[...]

Draft Introduction Contact Information 1-6 Preliminary IB0056101-00 G .02 S Notes[...]

Draft IB0056101-00 G .02 Preliminary 2-1 2 Feature Overview This section contain s the features for this release, t he supported QLogic adapter models, supported distributions and kern els, and a list of the sof tware component s. Features The QLogic OFED 1.4 sof tware release cont ains the complete OFED 1.4, plus additional QLogic improvement s, i[...]

Draft Feature Overview Features 2-2 Preliminary IB0056101-00 G .02 S  MV APICH and OpenMPI compiled with new er versions of each of four different co mpilers (GCC, PGI, Intel and PathScale) are available.  The QLogic InfiniBand Fabric Suite (IFS) is available sep arately for purchase. It includes FastFabric, the QLogic Host Sub net Manager (H[...]

Draft Feature Overview Features IB0056101-00 G .02 Preliminary 2-3 A This version of the QLogic OFED software provides support for all of the QLogic HCAs in Ta b l e 2 - 1 . Additional up-to-date information c an be f ound on the QLogic web site, specifically:  The high performance computing p age at www .qlogic.com/Products/HPC_product s_landin[...]

Draft Feature Overview Software Comp onents 2-4 Preliminary IB0056101-00 G .02 S Sof tware Component s This release includes all o f OFED 1.4 with enhancement s (QLogic OFED 1.4), including a new version of the VNIC tools and driver , and support for the QHT7xxx and QLE7xxx adapters. The sof tware inclu des the QLogic InfiniPath HCA driver , librar[...]

Draft IB0056101-00 G .02 Preliminary 3-1 3 S tep-by-S tep Inst allation Checklist This section provides an overview of the hardware and softwar e installa tion procedures. Det ailed steps are foun d in Section 4 “ Hardware Inst allation ” and Section 5 “ Software Inst allation ”. Hardware Inst allation The following step s summarize the bas[...]

Draft S tep-by-S tep Installation Checklist Software Inst allation 3-2 Preliminary IB0056101-00 G .02 S Sof tware Inst allation The following step s summarize the basic QLogic OFED 1.4 software inst allation and startup . These steps must be perfor med on each node in the cluster: 1. Make sure that the HCA hardware in stallation has been completed [...]

Draft S tep-by-S tep Installation Checklist Software Inst allation IB0056101-00 G .02 Preliminary 3-3 A 1 1. Y ou can optimize your system and adapter for the best performance. See “Adapter and Other Settings” o n page 7-1 . Also see the Performance Settings and Management T ips section in the QLogic HCA and QLogic OFED Software Users Guide. 12[...]

Draft S tep-by-S tep Installation Checklist Software Inst allation 3-4 Preliminary IB0056101-00 G .02 S Notes[...]

Draft IB0056101-00 G .02 Preliminary 4-1 4 Hardware Inst allation This section lists the req uirements and pro vides instructions for inst alling the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE724 0 and QLE7280; the QLogic PCIe adapter and PCIe riser card, QLE7140; and the QHT7[...]

Draft Hardware Installation Hardware Installa tion Requ irement s 4-2 Preliminary IB0056101-00 G .02 S Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7 140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical motherboard. See Figure 4-7 for an illustratio[...]

Draft Hardware In stallation Hardware Installa tion Requirements IB0056101-00 G .02 Preliminary 4-3 A Cabling and Switches The cable inst allation uses a st andard InfiniBand (IB) 4X cab le. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the long est passive copper IB cable that QLog ic has currently qualified is 2[...]

Draft Hardware Installation Hardware Installa tion Requ irement s 4-4 Preliminary IB0056101-00 G .02 S Optical Fibre Option The QLogic adapter also support s connection to the switch b y means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. For more informatio n on[...]

Draft Hardware In stallation Safety with Electricity IB0056101-00 G .02 Preliminary 4-5 A Safety with Electricity Observe these guidelines and sa fety prec autions when working around computer hardware and electrical equipment:  Locate the power source shutof f for the computer room or lab where you are working. This is where you will turn OFF t[...]

Draft Hardware Installation Unpacking Information 4-6 Preliminary IB0056101-00 G .02 S The package conten ts for the QLE72 80 adapter are:  QLogic QLE7280  Additional short bracket  Quick S tart Guide S tandard PCIe risers can be used, typically supplied by your system or motherboard vendor . The package conten ts for the QLE71 40 adapter [...]

Draft Hardware In stallation Unpacking Information IB0056101-00 G .02 Preliminary 4-7 A Figure 4-1. QLogic QLE7280 with IBA7220 ASIC Figure 4-2. QLogic QLE7140 Ca rd with Riser , T op V iew InfiniBand Connector Face Plate IBA7220 ASIC PCI Express Edge Connectors PCI Express Edge Connectors InfiniBand Connector Face Plate IBA6120 ASIC PCI Express Ri[...]

Draft Hardware Installation Unpacking Information 4-8 Preliminary IB0056101-00 G .02 S Figure 4-3. QLogic QHT7 040/QHT7140 Full and Low Profile Ca rds with Riser , Top V iew Unp a cking the QLogic Adapter Follow these steps whe n unpacking the QLogic adapter: 1. When unpacking, grou nd yourself before removing the QLogic adapter from the anti-stati[...]

Draft Hardware In stallation Hardware In stallation IB0056101-00 G .02 Preliminary 4-9 A Hardware Inst allation This section cont ains hardware inst allation instructions for the QLE72 40, QLE7280, QLE7140, QHT7040, and QHT7140. Hardware Inst allation for QL E7240, QLE7280, or QLE7140 with PCI Express Riser Installation for the QLE7240, QLE7280, an[...]

Draft Hardware Installation Hardware Installation 4-10 Preliminary IB0056101-00 G .02 S 4. Remove the cover screws and cover plate to e xpose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system. 5. Locate the PCIe slot on your moth erboard. Note that the PCIe slot has[...]

Draft Hardware In stallation Hardware In stallation IB0056101-00 G .02 Preliminary 4-1 1 A 9. Connect the QLogic adapter and PCIe riser card together , forming the assembly that you will insert into your motherboard. First, visually line up th e adapter slot connector edge with the edge conne ctor of the PCIe riser card (see Figure 4-5 ). . Figure [...]

Draft Hardware Installation Hardware Installation 4-12 Preliminary IB0056101-00 G .02 S 13. Insert the riser assembly into the motherboard’ s PCIe slot, ensuring good contact. The QLo gic adapter should now be parallel to th e motherboard and about one inch above it (see Figu re 4-6 ). Figure 4-6. Assembled PCIe HCA with Riser 14. Secure the face[...]

Draft Hardware In stallation Hardware In stallation IB0056101-00 G .02 Preliminary 4-13 A T o install the QLogic adapter with an HTX riser card: 1. The BIOS should be already be conf igured properly by the moth erboard manufacturer . However , if any additional BIOS con figuration is require d, it will usually need to be done before inst alling the[...]

Draft Hardware Installation Hardware Installation 4-14 Preliminary IB0056101-00 G .02 S 7. Remove the QLogic QHT7140 from the anti-st atic bag. 8. Locate the face plate on the connector edge of the card. 9. Connect the QLogic adap ter and HTX rise r card together , forming the assembly that you will insert into your motherboard. First, visually lin[...]

Draft Hardware In stallation Hardware In stallation IB0056101-00 G .02 Preliminary 4-15 A 13. Insert the HT riser assembly into the motherboard’ s HTX slot, ensuring good contact. The QLo gic adapter should now be parallel to th e motherboard and about one inch above it, as shown in Figure 4 -9 . Figure 4-9. Assembled QHT7140 with Riser 14. Secur[...]

Draft Hardware Installation Hardware Installation 4-16 Preliminary IB0056101-00 G .02 S Hardware Inst allation for the QHT7140 Without an HTX Riser Installing the QL ogic QHT7140 without an HTX riser card requires a 3U or larger chassis. The card slot connectors on the QHT7140 fit in to the HTX slot in a vertical installation. T o install the QLogi[...]

Draft Hardware In stallation Switch Configuration and Moni toring IB0056101-00 G .02 Preliminary 4-17 A 8. Insert the card by pressing firmly a nd evenly on the top of the horizont al bracket and the top rear corner of the card simult aneously . The card should insert evenly into the slot. Be careful not t o push, grab, or put pressure on any other[...]

Draft Hardware Installation Completing the Installation 4-18 Preliminary IB0056101-00 G .02 S The QLE7240, QLE7280, QLE7 140, QHT7040, and QHT7140 adapters are all cabled the same way . T o install the InfiniBand cables: 1. Check that you have removed the protect or plugs from the cable connector ends. 2. Different vendor cables might have di ffere[...]

Draft IB0056101-00 G .02 Preliminary 5-1 5 Sof tware Inst allation This section provides instructions for in stalling QLogic OFED 1.4, which includes QLogic InfiniPath and the OpenFabrics so ftware. The sof tware includes d rivers, protocol libraries, QLogic’ s implement at ion of the MPI message passing st andard, associated utilities, and examp[...]

Draft Software Inst allation Cluster Setup 5-2 Preliminary IB0056101-00 G .02 S Distribution Identifiers Distribution identifiers for this relea se are listed in the ta ble below . They are used in file naming conventions. T able 5-1. InfiniPath/ OpenFabrics Supported Distributions and Kernels Distribution InfiniPath/OpenFabrics Supported Kernels R[...]

Draft Software Inst allation Compiler Support IB0056101-00 G .02 Preliminary 5-3 A Compiler Support QLogic MPI support s use of a number of compilers. These include:  GNU gcc 3.3. x , 3.4. x , 4.0, 4.1, 4.2.x, and 4.3.x comp iler suites  PathScale Compiler Suite 3.0, 3.1 and 3.2  PGI 5.2, 6.0. 7.1, 7.2-4, and 8.0-3  Intel 9.x, 10.1, and[...]

Draft Software Inst allation Choose th e Appropri ate Download Files 5-4 Preliminary IB0056101-00 G .02 S Some operating system packages are required for OpenFabrics; they are listed in T able 5-3 . There are also OS p ackage requirement s for some specific comp onents in QLogic OFED. Choose the Appropriate Download Files This section assumes that [...]

Draft Software Inst allation Choose the Appropriate Download Files IB0056101-00 G .02 Preliminary 5-5 A The components of th e QLogic OFED 1.4 release are available pa ckaged as separate d ownloads:  QLogicIB-Basic with the T ext User Interface Installer  QLogic OFED 1.4 RPM Set  QLogic OFED 1.4 User-level RPM Set  Rocks Rolls  Platf[...]

Draft Software Inst allation Choose th e Appropri ate Download Files 5-6 Preliminary IB0056101-00 G .02 S QLogic OFED 1.4 RPM Set Includes:  QLogic OFED 1.4  InfiniPath HCA driver  Optimized stack for MPI (PSM)  QLogic MPI  Other MPIs (MV APICH and Open MPI compiled with GC C, PathScale, PGI, and Intel com- pilers)  User tools Ins[...]

Draft Software Inst allation Install QLogicIB-Basic with the Installer T ool IB0056101-00 G .02 Preliminary 5-7 A Inst all QLogicIB-Basic with the Inst aller T ool The Installer to ol has a T ext User Interfac e (TUI) that af fords easy inst allation of the software. Use this method if you have downloaded the QLogic-IB-Basic package. This me thod i[...]

Draft Software Inst allation Install QLogicIB-Basic with the Installer T ool 5-8 Preliminary IB0056101-00 G .02 S 2. After unp acking the .tgz file, change directory to: cd <QLogicIB-Basic>.<version> 3. Become root, then type: # ./INSTALL (Note that if you need 32-bit supp ort on 64 bit OSs, invoke the inst aller with ./INSTALL --32bit [...]

Draft Software Inst allation Install QLogicIB-Basic with the Installer T ool IB0056101-00 G .02 Preliminary 5-9 A 4. T ype 1 , which will display the scree n for software inst allation. The next screen shows the packag es to select for inst allation: QLogic Inc. IB Install (4.4.1.0.8 release) Menu Please Select Install Action (screen 1 of 3): 0) OF[...]

Draft Software Inst allation Install QLogicIB-Basic with the Installer T ool 5-10 Preliminary IB0056101-00 G .02 S The next screen contain s these choices: QLogic Inc. IB Install (4.4.1.0.8 release) Menu Please Select Install Action (screen 2 of 3): 0) MVAPICH/PSM (gcc) [ Install ][Available] 1.4.0.1.5 1) MVAPICH/PSM (PGI) [ Install ][Available] 1.[...]

Draft Software Inst allation Install QLogicIB-Basic with the Installer T ool IB0056101-00 G .02 Preliminary 5-1 1 A Open SM ( 1 ) should only be inst alled on one node in the cluster where it will be used. If desired, T ype 1 for Ope n SM. 7. Finally , type p to start the installation. The installer will uninst all older OFED RPMs, and then will a [...]

Draft Software Inst allation Install QLogicIB-Basic with the Installer T ool 5-12 Preliminary IB0056101-00 G .02 S ./INSTALL -C list s all the available comp onents, which include: ib_stack, truescale, oftools, ib_stack_dev, fastfabric, qlgc_srp, qlgc_vnic, ofed_ipoib, ofed_sdp, ofed_udapl, qlgc_fm, mvapich, mvapich2, openmpi, mvapich_gcc_qlc, mvap[...]

Draft Software Inst allation About rpm Inst allation IB0056101-00 G .02 Preliminary 5-13 A About rpm Inst allation Linux distributions of QLogic OFED (Inf iniPath and OpenFa brics) software can be installed from binary RPMs. RPM is a Li nux p ackaging and installa tion tool used by Red Hat, SUSE, and CentOS. Instructions given below are for a singl[...]

Draft Software Inst allation Using rpm to Inst all Infini Path and OpenFabr ics 5-14 Preliminary IB0056101-00 G .02 S RPMs contain config files. Y our current config files will not be o verwritten when new RPMs are inst alled. New config files will cont ain the suf fix .rpmnew and can be found in /etc/sysconfig and /etc/infiniband . Che ck the new [...]

Draft Software Inst allation Using rpm to Install InfiniPath and OpenFabrics IB0056101-00 G .02 Preliminary 5-15 A Become root, then: # cd QLogicOFED<version>-<distribution-x86_64 (Another way is to put the RPMs in a directory that is accessible (e.g., via Network File System (NFS)) to every node.) 3. T o install InfiniPath, QLogic MPI, an[...]

Draft Software Inst allation Using rpm to Inst all Infini Path and OpenFabr ics 5-16 Preliminary IB0056101-00 G .02 S Note that you need to inst all the InfiniPath-MPI/32bit/mpi-f rontend-*.rpm even if you do not plan to use any other 32-bit RPMs. Howeve r , you can add the other 32bit subdirectories to the rpm command, if you need 32-bit support. [...]

Draft Software Inst allation Install QLogic OFED User-l evel Software with the rpm Command IB0056101-00 G .02 Preliminary 5-17 A OpenFabrics/32bit/ OpenFabrics-Static/ OpenFabrics-Static/32bit/ OpenFabrics-Devel/ OpenFabrics-Devel/32bit/ OpenFabrics-Devel-Static/ OpenFabrics-Debuginfo/ OpenFabrics-Debuginfo/32bit/ OpenSM/ OpenSM/32bit/ OpenSM-Debug[...]

Draft Software Inst allation Install QLogic OFED User-l evel Software with the rpm Command 5-18 Preliminary IB0056101-00 G .02 S The tar command creates a directory based on the tar file name and places the RPMs and other files in this directo ry . 2. The RPMs need to be available on each node on which they will be used. Y ou can copy the RPMs to a[...]

Draft Software Inst allation Rebuilding or Reinstalling the kernel-ib Driver with rpm Af ter a Ke rnel Upgrade IB0056101-00 G .02 Preliminary 5-19 A The complete RPM directories for this down load are organized as follows: InfiniPath_license.txt,LEGAL.txt (top level) Documentation/ InfiniPath/ InfiniPath-Devel/ InfiniPath-MPI/ OtherMPIs/ Rebuilding[...]

Draft Software Inst allation Install QLogic OFED Using Rocks 5-20 Preliminary IB0056101-00 G .02 S Inst all QLogic OFED Using Rocks Rocks is a distribution designed f or managing clusters from the San Dieg o Supercomputer center (SDSC). Rocks is a way to manage the kickst art automated inst allation method created by Red Hat. By using the Rocks con[...]

Draft Software Inst allation Install QLogic OFED Using Ro cks IB0056101-00 G .02 Preliminary 5-21 A Boot, Core, OS Ro ll DVD Note that you may also need updates; look fo r the latest files with the service-pack prefix. Make sure you downloaded the .iso images correctly; verify by checking the md 5 checksum from the web site. b. Burn the .iso image([...]

Draft Software Inst allation Install QLogic OFED Usin g a Platform OCS Kit 5-22 Preliminary IB0056101-00 G .02 S If you have a burned a CD version of InfiniPath roll from the .iso image: # mount /mnt/cdrom # rocks-dist --install copyroll # umount /mnt/cdrom # cd /home/install # rocks-dist dist Y ou can use this method if you download the .iso imag [...]

Draft Software Inst allation Install Additional Sof tware IB0056101-00 G .02 Preliminary 5-23 A See also the current version of the F ast Fabric Users Guide, available with the software or from the QLogic download p age. Inst all Additional Sof tware This section cont ains information about a dditional third-party sof tware inst allation. Inst alli[...]

Draft Software Inst allation Removing Software Packages 5-24 Preliminary IB0056101-00 G .02 S Documentation is fo und in: /usr/share/man /usr/share/doc/infinipath /usr/share/doc/mpich-infinipath Note that license information only is found in usr/share/doc/infinipath. InfiniPath user document ation can be found on the QLogic web site on the software[...]

Draft Software Inst allation Removing Sof twa re Packages IB0056101-00 G .02 Preliminary 5-25 A For both InfiniPath and OpenFabrics, QLogic recommends that you remove all the package s at the same time. 1. Use the script ofed_uninstal l.sh to uninstall the OFED software. See the OFED Installa tion release notes that are part of the OFED documentati[...]

Draft Software Inst allation Removing Software Packages 5-26 Preliminary IB0056101-00 G .02 S Notes[...]

Draft IB0056101-00 G .02 Preliminary 6-1 6 Configuring Drivers and Services This section provides instructions for configuring and using the drivers and services available with QLogic OFED 1. 4. InfiniPath and OpenFa brics Driver Overview The InfiniPath ib_ipath module provide s low level QLogic hardware support, and is the base driver for both MPI[...]

Draft Configuring Drivers and Services OpenFabrics Drivers and Servic es Configuration and S tartup 6-2 Preliminary IB0056101-00 G .02 S Configuring the IPoIB Network Interface The following instructions show you how to manually configur e your OpenFabrics IPoIB network interface. This example assume s that you are using sh or bash as your shell, a[...]

Draft Configuring Drivers and Services OpenFabrics Drivers an d Services Configuration an d S tartup IB0056101-00 G .02 Preliminary 6-3 A The output of the ping command will be similar to the following, with a line for each host already configured a nd connected: WARNING: pinging broadcast address PING 10.1.17.255 (10.1.17.255) 517(84) bytes of dat[...]

Draft Configuring Drivers and Services OpenFabrics Drivers and Servic es Configuration and S tartup 6-4 Preliminary IB0056101-00 G .02 S Y ou can stop opensmd again like this: # /etc/init.d/opensmd stop If you want to pass any argu ments to t he OpenSM program, modify the following file, and add the argument s to the OPTIONS variable: /etc/init.d/o[...]

Draft Configuring Drivers and Services OpenFabrics Drivers an d Services Configuration an d S tartup IB0056101-00 G .02 Preliminary 6-5 A In the output, look for lines similar to the se: GUID: 0002c90200402c04 ID: LSI Storage Systems SRP Driver 200400a0b8114527 service entries: 1 service[ 0]: 200400a0b8114527 / SRP.T10:200400A0B8114527 GUID: 0002c9[...]

Draft Configuring Drivers and Services OpenFabrics Drivers and Servic es Configuration and S tartup 6-6 Preliminary IB0056101-00 G .02 S 5. Y ou can look for the newly created devices in the /proc/parti tions file. The file will look similar to this example (the p artition names may vary): # cat /proc/partitions major minor #blocks name 8 64 142325[...]

Draft Configuring Drivers and Services OpenFabrics Drivers an d Services Configuration an d S tartup IB0056101-00 G .02 Preliminary 6-7 A their IOCGUID, IOCSTRING , etc. Use the ib_qlgc_vnic_query tool to get this information. When ib_qlgc_vnic_query is executed without a ny options, it displays detailed information abou t all the EVIC/VEx IOCs pre[...]

Draft Configuring Drivers and Services OpenFabrics Drivers and Servic es Configuration and S tartup 6-8 Preliminary IB0056101-00 G .02 S HCA No = 0, HCA = mlx4_0, Po rt = 2, Port GUID = 0x0002c903000010fa, State = Active IO Unit Info: port LID: 0009 port GID: fe800000000 0000000066a11de000070 change ID: 0003 max controllers: 0x02 controller[ 1] GUI[...]

Draft Configuring Drivers and Services OpenFabrics Drivers an d Services Configuration an d S tartup IB0056101-00 G .02 Preliminary 6-9 A 2. Create the VNIC interfaces using the configuration file: /etc/infiniband/qlgc_vnic.cfg. Look at the qlgc_vnic. cfg.sample file to see how VNIC configuration files are written. It can be found with the OFED do [...]

Draft Configuring Drivers and Services OpenFabrics Drivers and Servic es Configuration and S tartup 6-10 Preliminary IB0056101-00 G .02 S b. Format 2: Defining an IOC using the IOCST RING . Defining the IOC using the IOCSTRING allows VIO hardwa re to be hot-swapped in and out of a specific slot. Th e host attempt s to connect t o the specified IOC [...]

Draft Configuring Drivers and Services OpenFabrics Drivers an d Services Configuration an d S tartup IB0056101-00 G .02 Preliminary 6-1 1 A c. Format 3: S tarting VNIC using DGID. Following is an example of a DGID and IOCGUID VNIC configurati on. This configuration allows for the quickest start up of VNIC service: {CREATE; NAME="eioc1"; D[...]

Draft Configuring Drivers and Services OpenFabrics Drivers and Servic es Configuration and S tartup 6-12 Preliminary IB0056101-00 G .02 S 5. S tart the QLogic VNIC driver and the QLo gic VNIC interfaces. Once you have created a configuration file, you can st art the VNIC driver and create the VNIC interfaces specified in t he configuration file by [...]

Draft Configuring Drivers and Services OpenFabrics Drivers an d Services Configuration an d S tartup IB0056101-00 G .02 Preliminary 6-13 A If you want to restart the QLogic VNIC interfaces, run the following command: # /etc/init.d/qlgc_vnic restart Y ou can get information about the QLo gic VNIC interfaces by using the following script (as root): #[...]

Draft Configuring Drivers and Services Other Configuration: Changing th e MTU Size 6-14 Preliminary IB0056101-00 G .02 S 2. Check that you have a /etc/dat.c onf file. It should be inst alled by the dapl- RPM. The file dat.conf cont ains a list of inte rface adapters supported by uDAPL service providers. In p articular , it must contain mapping entr[...]

Draft Configuring Drivers and Services Managing the InfiniPath Driver IB0056101-00 G .02 Preliminary 6-15 A Managing the InfiniPath Driver The st artup script for ib_ipa th is inst alled automatically a s p art of the sof tware inst allation, and normally does not need to be changed. It runs as a system service. The primary configuration file for t[...]

Draft Configuring Drivers and Services Managing the Infini Path Driver 6-16 Preliminary IB0056101-00 G .02 S The device files are : /dev/ipath /dev/ipath0, /dev/ipath1, ... The numbered device files allow access to a specific InfiniPath unit. See the ib_ipath man p age for more det ails. Configure InfiniPath Driver S t ate Use the following command[...]

Draft Configuring Drivers and Services Further Inform ation on Configuring and Load ing Drivers IB0056101-00 G .02 Preliminary 6-17 A The sequence of commands to rest art the driver are as follows. # /etc/init.d/opensmd stop # /etc/init.d/ openibd stop ... # /etc/init.d/ openibd start # /etc/init.d/opensmd start The ... represents wh atever activit[...]

Draft Configuring Drivers and Services Further Information on Configurin g and Loading Drivers 6-18 Preliminary IB0056101-00 G .02 S Notes[...]

Draft IB0056101-00 G .02 Preliminary 7-1 7 Inst allation V erification and Additional Settings This section provides instructions for verifying that the software has been properly installed and that the drivers are load ed, and the fabric is active and ready to use. Information on adapter performan ce tuning is also provided. LED Link and Dat a Ind[...]

Draft Installati on V erification and Additional Settings Customer Acceptance Utility 7-2 Preliminary IB0056101-00 G .02 S  Make sure that write combining is en abled. The x86 Page Attribute T able (P A T) mechanism that allocates write-combining (WC) mapping s for the PIO buffers has been added and is now the default. I f P A T is unavailable o[...]

Draft Installation V erification and Ad ditional Settings Customer Acceptance Utility IB0056101-00 G .02 Preliminary 7-3 A 3. Gathers and analyzes system c onfiguration from the nodes. 4. Gathers and analyzes RPMs insta lled on the nodes. Missing RPMs can be found this way . 5. V erifies QLogic hardware and software st atus and con figuration. Incl[...]

Draft Installati on V erification and Additional Settings Customer Acceptance Utility 7-4 Preliminary IB0056101-00 G .02 S In most cases of failure, the script suggest s recommended a ctions. Please see the ipath_checkout man pag e for more information and updates. Also refer to the T roubleshooting appe ndix in the QLogic HCA and QLogic OFED Softw[...]

Draft IB0056101-00 G .02 Preliminary A-1 A Inst allation T roubleshooting The following sections cont ain informat ion about issues that may occur during installation. Some of this material is repeated in the T roubleshooting appendix of the QLogic HCA and QLogic OF ED Sof tware User Guide . Many programs and files are available that gather informa[...]

Draft Installati on T roubleshooting BIOS Settings A-2 Preliminary IB0056101-00 G .02 S BIOS Settings This section covers issues related to BIOS settings. Y ou can check and adjust BIOS settings using the BIOS Setup utility . For specific instructions, follow the hardware documentation that came with your system. Enable Advanced Configuration and P[...]

Draft Installation T roubleshooting Software Inst allation Issues IB0056101-00 G .02 Preliminary A-3 A If these RPMs have not been installed, you will see error messages like this when installing InfiniPath: Building and installing InfiniPath modules for 2.6.16_sles10 2.6.16.21-0.8-debug kernel *** ERROR: /lib/modules/2.6.16.21-0.8-debug/build/.con[...]

Draft Installati on T roubleshooting Software Inst a llation Issues A-4 Preliminary IB0056101-00 G .02 S Resolving Conflict s Occasionally , conflicts may arise when trying to inst all "on top of" an existing set of files that may come from a differen t set of RPMs. For example, if you inst all the QLogic MPI RPMs after having previously [...]

Draft Installation T roubleshooting Configuration Issues IB0056101-00 G .02 Preliminary A-5 A In older distributions, such as RHEL4, the 32-bit glibc is cont ained in the libgcc RPM. The RPM name will b e similar to this: libgcc-<version>.EL4.i386.rpm In newer distributions, glibc is an RPM name. The 32-b it glibc is na med similarly to: glib[...]

Draft Installati on T roubleshooting Configuration Issues A-6 Preliminary IB0056101-00 G .02 S When ifconfig or ifup are run, the error will look similar to this (assuming ipath_ether was used for eth2 ): eth2: error fetching interface information: Device not found T o prevent the error message, remove the following files (assuming ipath_ether was [...]

Draft IB0056101-00 G .02 Preliminary B-1 B W rite Combining Introduction Write comb ining improves write bandwidth to the QLogic chip by writing mu ltiple words in a single bus transaction (t ypically 64 bytes). W rite combining applies only to x86_64 systems. The x86 Page Attribute T able (P A T) mechanism that allocate s write-combining (WC) mapp[...]

Draft Write Combining P A T and Write Combining B-2 Preliminary IB0056101-00 G .02 S . . If you do not see any of these messages on your console, but suspect this problem, check the /var/log/messages file. Some systems suppress driver load messages but still output them to the log file. Methods for enabling and disabling the two write combin ing me[...]

Draft Write Combining MTRR Mapping and Write Combining IB0056101-00 G .02 Preliminary B-3 A For systems with very large amount s of memory (32GB or mor e), it may also be necessary to adjust the BIOS setting for the PCI ho le granularity to 2GB. This setting allows the memory to be mapped with fewer MTRRs, so t hat there will be one or more unused [...]

Draft Write Combining MTRR Mapping and Write Combining B-4 Preliminary IB0056101-00 G .02 S Notes[...]

Draft IB0056101-00 G .02 Preliminary C-1 C Configuration Files T able C-1 contains descriptions of the conf iguration and configuratio n template files used by the InfiniPath and OpenFabrics sof tware. T able C-1. Co nfiguration Files Configuration File Na me Description /etc/infiniband/qlgc_vnic.cfg V irtualNIC configuration f ile. Create this fil[...]

Draft Configuration Files C-2 Preliminary IB0056101-00 G .02 S /etc/sysconfig/network/ifcfg- <NAME> Network configuration file for network inter- faces When used for VNIC configuration, <NAME> is in the form eioc X, where X is the device number . There will be one interface configuration file for ea ch inter- face define d in /etc/infin[...]

Draft IB0056101-00 G .02 Preliminary D-1 D Package Descriptions The following sections cont ain detailed descriptions of the p ackages for the InfiniPath and OpenFabrics sof tware. In this release, sof tware may be installed in several ways. With the QLogicIB-Basic download, which u ses the Installer tool, the package gro upings are a bit dif feren[...]

Draft Package Descriptions InfiniPath RPM V ersion Numbers and Identi fiers D-2 Preliminary IB0056101-00 G .02 S Although QLogic recommen ds installing all RPMs on all nodes, not all InfiniPath software is required on all nodes. See T able D-2 , Ta b l e D - 3 , or Ta b l e D - 4 for information on inst allation of sof tware RPMs on specif ic types[...]

Draft Package Descr iptions Documentation RPMs IB0056101-00 G .02 Preliminary D-3 A Document ation RPMs InfiniPath RPMs T able D-1. Doc umentation/RPMs RPM Name Front End Compute Development infinipath-doc- xxx _yyy .noarch.rpm InfiniPath man p ages Optional Optional Optional mpi-doc- xxx_yyy .no arch.rpm Man page s for MPI functions and othe r MPI[...]

Draft Package Descriptions OpenFabrics RPMs D-4 Preliminary IB0056101-00 G .02 S OpenFabrics RPMs For ease of inst allation, QLogic reco mm ends that all of the OpenFabrics RPMs listed in T able D-5 be installed on all nodes. V ersion numbers are indicated by xxx . T able D-3. InfiniPath-Dev el/RPMs RPM Name Front End Compute Development infini- pa[...]

Draft Package Descr iptions OpenFabrics RPMs IB0056101-00 G .02 Preliminary D-5 A dapl- xxx .x86_64.rpm uDAPL 2.0.15 support Optional for OpenFabrics dapl-utils- xxx .x86_64.rpm uDAPL support Optional for OpenFabrics ibsim- xxx .x86_64.rpm InfiniBand Fabric Simulator Optional for OpenFabrics ibutils- xxx .x86_64.rpm ibutils provides InfiniBand (IB)[...]

Draft Package Descriptions OpenFabrics RPMs D-6 Preliminary IB0056101-00 G .02 S libibverbs- xxx .x86_64.rpm Library that allows usersp ace processes to use In finiBand verbs as described in the InfiniBan d Architecture Specification . This library includes direct hardware access for fast path o perations. For this library to be useful, a device-sp[...]

Draft Package Descr iptions OpenFabrics RPMs IB0056101-00 G .02 Preliminary D-7 A The development RPMs in T able D-6 are only needed on the nodes where OFED programs are compiled. qlgc_vnic_daemon- xxx .x86_64.rpm Used with VNIC ULP service Optional for OpenFabrics qlvnictools- xxx .x86_64.rpm S tartup script, sample config file, and utilities Opti[...]

Draft Package Descriptions OpenFabrics RPMs D-8 Preliminary IB0056101-00 G .02 S The opensm pa ckage in Ta b l e D - 7 should be installed only on the node that will be used as a subnet manage r . libibcm-devel- xxx .x86_64.rpm Development f iles for the libibcm library Optional for OpenFabric libibcommon-devel- xxx .x86_64.rpm Development f iles f[...]

Draft Package Descr iptions Other HCAs IB0056101-00 G .02 Preliminary D-9 A Other HCAs The packages in T able D-9 should be in stalled only if other HCAs are used. The development RPMs in Ta b l e D - 1 0 are only needed on the nodes where programs are compiled T able D-8. OpenSM-Devel/RPM RPM Name Comment s opensm-devel- xxx .x86_64.rpm Developmen[...]

Draft Package Descriptions Other MPIs D-10 Preliminary IB0056101-00 G .02 S Other MPIs The packages in T able D-1 1 should be installed if ot her MPI implement ations are desired.RPMs that in clude yyy in the name are supplied by QLogic; others a re from OFED. libmlx4-devel- xxx .x86_64.rpm Development file s for Mella nox ® ConnectX ™ InfiniBan[...]

Draft Package Descr iptions Other MPIs IB0056101-00 G .02 Preliminary D-1 1 A openmpi_gcc- xxx .x86_64.rpm Open MPI compiled with GNU Optional openmpi_gcc_qlc- xxx . yyy. x86_64.rpm Open MPI compiled with GNU for PSM support Optional openmpi_intel_qlc- xxx . yyy .x86_64.rpm a Open MPI compiled with Intel for PSM support Optional openmpi_pathscale_q[...]

Draft Package Descriptions Other MPIs D-12 Preliminary IB0056101-00 G .02 S Notes[...]

Draft IB0056101-00 G .02 Preliminary Index-1 Index A ACPI 4-4, A-2 Adapter , see HCA B BIOS configuring 4-4 settings A-2 settings to fix MTRR issues B-2 C -c 7-3 Cables supported 4-3 Compiler support 5-3 Configuration files C-1 ib_ipath 6-15 OpenSM 6-3 VNIC 6-6 --continue 7-3 CPUs, HTX motherboards may require two or more CPUs A-1 D -d 7-3 --debug [...]

Draft QLogic HCA and QLogic OFED Software Install Guide QLogic OFED V ersion 1.4 Index-2 Preliminary IB0056101-00 G .02 S Hardware requirements 4-1 HCA form factors 4-2 other HCAs-Devel/RPMs D-9 model numbers 2-1 not recognized A-1 other HCAs/RPMs D-9 QHT7140 installation with HT X riser 4-12 QHT7140 installation without an HTX riser 4-16 QLE7140/7[...]

Draft IB0056101-00 G .02 Preliminary Index-3 QLogic HCA and QLogic OFED Sof tware Inst all Guide QLogic OFED V ersion 1.4 A Lockable memory error A-5 Lustre, installing 5-23 M Model numbers for HCAs 2-1 MPI, other MPIs/RPMs D-10 MPI over uDAPL 6-13 mpirun , installatio n requires 32-bit support A-4 mpi-selector 5-15, 5-18 MTRR editing BIOS settings[...]

Draft QLogic HCA and QLogic OFED Software Install Guide QLogic OFED V ersion 1.4 Index-4 Preliminary IB0056101-00 G .02 S using to inst all InfiniPath and OpenFabrics 5-14 version numbers and identifiers D-2 --run=LIST 7-3 S Safety precautions 4-5 --skip=LIST 7-3 Software files to download 5-4 InfiniPath 2-4 installati on overvi ew 3-2 package requ[...]

Draft[...]

Draft Preliminary © 2005-2009 QLogic Corporation. S pecifications are subject to change without notic e. All rights reserved worldwide . QLogic and the QLogic logo are registered trademarks of QLogic Corporatio n. QLA, SANsurf er, InfiniPath, and SilverS torm are trademarks or registered trade marks of QLogic Corporation. AMD Opteron is a trademar[...]