English dropdown-ico

Q-Logic IB0056101-00 G manual

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

Go to page of

Similar user manuals

A good user manual

The rules should oblige the seller to give the purchaser an operating instrucion of Q-Logic IB0056101-00 G, along with an item. The lack of an instruction or false information given to customer shall constitute grounds to apply for a complaint because of nonconformity of goods with the contract. In accordance with the law, a customer can receive an instruction in non-paper form; lately graphic and electronic forms of the manuals, as well as instructional videos have been majorly used. A necessary precondition for this is the unmistakable, legible character of an instruction.

What is an instruction?

The term originates from the Latin word „instructio”, which means organizing. Therefore, in an instruction of Q-Logic IB0056101-00 G one could find a process description. An instruction's purpose is to teach, to ease the start-up and an item's use or performance of certain activities. An instruction is a compilation of information about an item/a service, it is a clue.

Unfortunately, only a few customers devote their time to read an instruction of Q-Logic IB0056101-00 G. A good user manual introduces us to a number of additional functionalities of the purchased item, and also helps us to avoid the formation of most of the defects.

What should a perfect user manual contain?

First and foremost, an user manual of Q-Logic IB0056101-00 G should contain:
- informations concerning technical data of Q-Logic IB0056101-00 G
- name of the manufacturer and a year of construction of the Q-Logic IB0056101-00 G item
- rules of operation, control and maintenance of the Q-Logic IB0056101-00 G item
- safety signs and mark certificates which confirm compatibility with appropriate standards

Why don't we read the manuals?

Usually it results from the lack of time and certainty about functionalities of purchased items. Unfortunately, networking and start-up of Q-Logic IB0056101-00 G alone are not enough. An instruction contains a number of clues concerning respective functionalities, safety rules, maintenance methods (what means should be used), eventual defects of Q-Logic IB0056101-00 G, and methods of problem resolution. Eventually, when one still can't find the answer to his problems, he will be directed to the Q-Logic service. Lately animated manuals and instructional videos are quite popular among customers. These kinds of user manuals are effective; they assure that a customer will familiarize himself with the whole material, and won't skip complicated, technical information of Q-Logic IB0056101-00 G.

Why one should read the manuals?

It is mostly in the manuals where we will find the details concerning construction and possibility of the Q-Logic IB0056101-00 G item, and its use of respective accessory, as well as information concerning all the functions and facilities.

After a successful purchase of an item one should find a moment and get to know with every part of an instruction. Currently the manuals are carefully prearranged and translated, so they could be fully understood by its users. The manuals will serve as an informational aid.

Table of contents for the manual

  • Page 1

    IB0056101-00 G QLogic HCA and InfiniPath ® Sof tware Inst all Guide V ersion 2.2[...]

  • Page 2

    Page ii IB0056101-00 G S QLogic HCA and InfiniPath ® Software Install Guide V er sion 2.2 Information fu rnished in this manual is belie ved to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patent s or other right s of third part ies which may result from it s use. QLogic [...]

  • Page 3

    IB0056101-00 G Page iii T able of Content s 1 Introduction Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .[...]

  • Page 4

    Page iv IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide V er sion 2.2 S Hardware Installation for QLE724 0, QLE7280, or QLE7 140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 Dual Adapter Installatio n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9[...]

  • Page 5

    IB0056101-00 G Page v QLogic HCA and InfiniPat h ® Software Install Guide V er sion 2.2 A S tarting and S topping the InfiniPath Software . . . . . . . . . . . . . . . . . . . . . . . 5-26 Rebuilding or Reinst alling Drivers Af ter a Kernel Upgr ade . . . . . . . . . . . . . 5-28 Rebuilding or Reinst alling Drivers if a Dif ferent Kernel is In sta[...]

  • Page 6

    Page vi IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide V er sion 2.2 S B Configuration Files C RPM Descriptions InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1 Different Nodes May Use Dif ferent RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . C-1 RPM V ersion Numbers a[...]

  • Page 7

    IB0056101-00 G Page vii QLogic HCA and InfiniPat h ® Software Install Guide V er sion 2.2 A List of Figures Figure Page 4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-2 QLogic QLE7140 Card with Riser , T op View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-3 QLo[...]

  • Page 8

    Page viii IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide V er sion 2.2 S Notes[...]

  • Page 9

    IB0056101-00 G 1-1 1 Introduction This chapter describes the content s, intended au dience, and organization of the QLogic HCA and InfiniPath Sof tware Install Guide. The QLogic HCA and InfiniPath Sof tware Inst all Guide cont ains instructions for installing the QLogic Host Channel Adapters (HCAs) and the In finiPath and OpenFabrics software. The [...]

  • Page 10

    1 – Introduction Overview 1-2 IB0056101-00 G S  Section 3 , S tep-by-S tep Installat ion Che cklist, provides a high-level overview of the hardware and sof tware inst allation procedures.  Section 4 , Hardware Installatio n, includes instructions for inst alling the QLogic QLE7140, QLE7240, QLE7280, QHT7040, an d QHT7140 HCAs.  Section 5[...]

  • Page 11

    1 – Introduction Interoperability IB0056101-00 G 1-3 A InfiniPath OpenFabrics sof tware is interoperable with other vendors’ InfiniBa nd Host Channel Adapters (HCAs) running comp atible Op enFabrics releases. There are several options for sub net management in your cluster:  Use the embedded Subnet Manager (SM) in one or more managed switche[...]

  • Page 12

    1 – Introduction Documenta tion 1-4 IB0056101-00 G S Document ation The product document ation includes:  The QLogic HCA and InfiniPath So f tware Install Guide  The QLogic HCA and InfiniPath So f tware User Guide  The QLogic FastFabric Users Gu ide (for informat ion on QLogic InfiniServ T ools)  The OFED+ Users Guid e (for informatio[...]

  • Page 13

    1 – Introduction Contact Information IB0056101-00 G 1-5 A Cont act Information Support Headquarters QLogic Corporation 4601 Dean Lakes Blvd Shakopee, MN 55379 USA QLogic Web Site www .qlogic.com T e chnical Support W eb Site support.qlogic.com T echnical Support Email support@qlogic.com T echnical T raining Email tech.training@qlogic.com North Am[...]

  • Page 14

    1 – Introduction Contact Information 1-6 IB0056101-00 G S Notes[...]

  • Page 15

    IB0056101-00 G 2-1 2 Feature Overview This section contain s the features for this release, t he supported QLogic adapter models, supported distributions and kern els, and a list of the sof tware components. What’ s New in this Release This release adds support for the QLE724 0 and QLE7280 InfiniBand DDR Host Channel Adapters (HCAs), which of fer[...]

  • Page 16

    2 – Feature Overview New Featur es 2-2 IB0056101-00 G S New Features The following features are new to the 2.2 re lease:  Expanded MPI scalability enhancements for PCI Express have been a dded. On the QLE7240 and QLE7280, up to 16 dedicated hardware context s per node are available. The QHT7140 has eig ht per node. The QLE7140 has four per nod[...]

  • Page 17

    2 – Feature Overview Supported Distributions and Kern els IB0056101-00 G 2-3 A  A subset of the QLogic InfiniBand F abric Suite, the enablement tools, are offered with this release.  T wo separate SCSI RDMA Protocol (SRP) modules are provided: the standard OpenFabrics (OFED) SRP , and QLogic SRP .  QLogic MPI support s running exclusivel[...]

  • Page 18

    2 – Feature Overview Compiler Support 2-4 IB0056101-00 G S Compiler Support QLogic MPI support s a number of compilers. These include:  PathScale Compiler Suite 3.0 and 3.1  PGI 5.2 , 6.0, and 7.1  Intel 9.x and 10.1  GNU gcc 3.3. x , 3.4. x , 4.0 a nd 4.1 compiler suites  gfortran The PathScale Compiler Suite V ersion 3. x is now [...]

  • Page 19

    2 – Feature Overview Software Comp onents IB0056101-00 G 2-5 A Included component s are:  InfiniPath driver  InfiniPath Ethernet emulation ( ipat h_ether )  InfiniPath libraries  InfiniPath utilities, configuration, and support to ols, including ipath_checkout , ipath_control , ipat h_pkt_test , and ipathstats  QLogic MPI  QLogi[...]

  • Page 20

    2 – Feature Overview Software Comp onents 2-6 IB0056101-00 G S More details about the hardware and sof tware can be found in Section 4 and Section 5 .[...]

  • Page 21

    IB0056101-00 G 3-1 3 S tep-by-S tep Inst allation Checklist This section provides an overview of the hardware and softwar e installa tion procedures. Det ailed steps ar e found in Section 4 “ Hardware Inst allation ” and Section 5 “ Software Inst allation ”. Hardware Inst allation The following step s summarize the bas ic hardware installat[...]

  • Page 22

    3 – S tep-by-S t ep Installation Checklist Software Inst allation 3-2 IB0056101-00 G S Sof tware Inst allation The following step s summarize the basic Infin iPath and OpenFabrics sof tware installation and st artup. These steps must be performed on each node in the cluster: 1. Make sure that the HCA hardware in stallation has been completed acco[...]

  • Page 23

    3 – S tep-by-S t ep Inst allation Checklist Software Inst allation IB0056101-00 G 3-3 A 10. Optimize your adapter for the best perfo rmance. See “Adapter Settings” on page 5-30 . Also see the Performance and Management T ips section in the QLogic HCA and InfiniPath Sof tware User Guide. 1 1. Perform the recommended health checks. See “Custo[...]

  • Page 24

    3 – S tep-by-S t ep Installation Checklist Software Inst allation 3-4 IB0056101-00 G S Notes[...]

  • Page 25

    IB0056101-00 G 4-1 4 Hardware Inst allation This section lists the req uirements a nd provides instructions for inst alling the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic InfiniPath PCIe adapter and PCIe riser card, QLE7140; and the QHT7 040 or QHT7[...]

  • Page 26

    4 – Hardware Installation Hardware Installa tion Requ irement s 4-2 IB0056101-00 G S Installation of the QLE7240, QLE7280, QLE7140, QHT7 040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figur e 4-4 for an illustration of a PCI Express (PCIe) slot in a typical motherboard. See Figure 4-7 for an illustration of an HTX slo[...]

  • Page 27

    4 – Hardware Installation Hardware Installa tion Requirements IB0056101-00 G 4-3 A Cabling and Switches The cable inst allation uses a st andard InfiniBand (IB) 4X cab le. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cable that QLog ic has currently qualified is 20 meters. For DDR-[...]

  • Page 28

    4 – Hardware Installation Hardware Installa tion Requ irement s 4-4 IB0056101-00 G S Optical Fibre Option The QLogic adapter also support s connection to the switch b y means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. For more informatio n on the EMCORE conv[...]

  • Page 29

    4 – Hardware Installation Safety with Electricity IB0056101-00 G 4-5 A Safety with Electricity Observe these guidelines and safety prec autions when working around computer hardware and electrical equipment:  Locate the power source shutof f for the computer room or lab where you are working. This is where you will turn OFF the power in the ev[...]

  • Page 30

    4 – Hardware Installation Unpacking Information 4-6 IB0056101-00 G S The package conten ts for the QLE72 80 adapter are:  QLogic QLE7280  Additional short bracket  Quick S tart Guide S tandard PCIe risers can be used , typically supplied by your system or motherboard vendor . The package conten ts for the QLE71 40 adapter are:  QLogic[...]

  • Page 31

    4 – Hardware Installation Unpacking Information IB0056101-00 G 4-7 A Figure 4-1. QLogic QLE7280 with IBA7220 ASIC Figure 4-2. QLogic QLE7140 Card with Riser , T op V iew InfiniBand Connector Face Plate IBA7220 ASIC PCI Express Edge Connectors PCI Express Edge Connectors InfiniBand Connector Face Plate IBA6120 ASIC PCI Express Riser Card. Not supp[...]

  • Page 32

    4 – Hardware Installation Unpacking Information 4-8 IB0056101-00 G S Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profil e Cards with Riser , Top Vi e w Unp acking the QLogic Adapter Follow these steps whe n unpackin g the QLogic adapter: 1. When unpacking, grou nd yourself before removing the QLogic adapter from the anti-static bag. 2. Graspi[...]

  • Page 33

    4 – Hardware Installation Hardware In stallation IB0056101-00 G 4-9 A Hardware Inst allation This section contain s hardware inst allation instructions for the QLE72 40, QLE7280, QLE7140, QHT7040, and QHT7140. Hardware Inst allation for QL E7240, QLE7280, or QLE7140 with PCI Express Riser Installation for the QLE7240, QLE7280, and QLE7140 is simi[...]

  • Page 34

    4 – Hardware Installation Hardware Installation 4-10 IB0056101-00 G S 4. Remove the cover screws and cover plate to e xpose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system. 5. Locate the PCIe slot on your moth erboard. Note that the PCIe slot has two separate se[...]

  • Page 35

    4 – Hardware Installation Hardware In stallation IB0056101-00 G 4-1 1 A 9. Connect the QLogic adapter and PCIe riser card together , forming the assembly that you will insert into your motherboard. First, visually line up th e adapter slot connector edge with the edge conne ctor of the PCIe riser card (see Figure 4-5 ). . Figure 4-5. QLogic PCIe [...]

  • Page 36

    4 – Hardware Installation Hardware Installation 4-12 IB0056101-00 G S 13. Insert the riser assembly into the motherboard’ s PCIe slot, ensuring good contact. The QLo gic adapter should now be parallel to th e motherboard and about one inch above it (see Figu re 4-6 ). Figure 4-6. Assembled PCIe HCA with Riser 14. Secure the face plate to the ch[...]

  • Page 37

    4 – Hardware Installation Hardware In stallation IB0056101-00 G 4-13 A T o install the QLogic adapter with an HTX riser card: 1. The BIOS should be already be conf igured properly by the motherboard manufacturer . However , if any additional BIOS configuration is required, it will usually need to be done before inst alling the QLogic adap ter . S[...]

  • Page 38

    4 – Hardware Installation Hardware Installation 4-14 IB0056101-00 G S 7. Remove the QLogic QHT7140 from the anti-st atic bag. 8. Locate the face plate on the connector edge of the card. 9. Connect the QLogic adapter and HTX riser card together , forming the assembly that you will insert into your motherboard. First, visually line up th e adapter [...]

  • Page 39

    4 – Hardware Installation Hardware In stallation IB0056101-00 G 4-15 A 13. Insert the HT riser assembly into the motherboard’ s HTX slot, ensuring good contact. The QLo gic adapter should now be parallel to th e motherboard and about one inch above it, as shown in Figure 4 -9 . Figure 4-9. Assembled QHT7140 with Riser 14. Secure the face plate [...]

  • Page 40

    4 – Hardware Installation Hardware Installation 4-16 IB0056101-00 G S Hardware Inst allation for the QHT7140 Without an HTX Riser Installing the QL ogic QHT7140 without an HTX riser card requires a 3U or larger chassis. The card slot connectors on the QHT7140 fit in to the HTX slot in a vertical installation. T o install the QLogic adapter withou[...]

  • Page 41

    4 – Hardware Installation Switch Configuration and Moni toring IB0056101-00 G 4-17 A 8. Insert the card by pressing firmly a nd evenly on the top of the horizont al bracket and the top rear corner of the card simult aneously . The card should insert evenly into the slot. Be careful not t o push, grab, or put pressure on any other p art of the car[...]

  • Page 42

    4 – Hardware Installation Completing the Installation 4-18 IB0056101-00 G S The QLE7240, QLE7280, QLE7 140, QHT7040, and QHT7140 adapters a re all cabled the same way . T o install the InfiniBand cables: 1. Check that you have removed the protect or plugs from the cable connector ends. 2. Different vendor cables might have di f ferent latch mecha[...]

  • Page 43

    IB0056101-00 G 5-1 5 Sof tware Inst allation This section provides instructions for installing th e InfiniPath and OpenFabrics software. The InfiniPath softwa re includes drivers, pro tocol libraries, QLogic’ s implementation of the MPI messa ge pa ssing standard, and example programs, including benchmarks. A complet e list of the provided sof tw[...]

  • Page 44

    5 – Software Installation Cluster Setup 5-2 IB0056101-00 G S Supported Linux Distributions The currently supported distributions and a ssociated Linux kernel versions for InfiniPath and OpenFabrics a re listed in T able 5-1 . The kernels are the ones that shipped with the distributions. Setting Up Y our Environment Keep the following in mind when[...]

  • Page 45

    5 – Software Inst allation Downloading and Unpacking the Infi niPath an d OpenFabrics Sof tware IB0056101-00 G 5-3 A  Make sure that all previously ex ist ing (stock) OpenFabrics RPMs are uninst alled. See “Removing Sof tware Packages” on page 5-33 for more information on uninst alling. If you are using RHEL 5, make sure that opensm-* is m[...]

  • Page 46

    5 – Software Installation Downloading and Unp acking the InfiniPath and OpenFab rics Software 5-4 IB0056101-00 G S Follow the links for your distribution to the desired d ownload, then follow the instructions on the web p age for downloading t he files to a convenient directory . Instructions for installing the In finiPath and OpenFabrics sof twa[...]

  • Page 47

    5 – Software Inst allation Downloading and Unpacking the Infi niPath an d OpenFabrics Sof tware IB0056101-00 G 5-5 A The RPMs are organized as follows: InfiniPath_license.txt,LEGAL.txt (top level) Documentation/ InfiniPath/ InfiniPath-Devel/ InfiniPath-MPI/ OpenFabrics/ OpenFabrics-Devel/ OpenSM/ OtherHCAs/ OtherHCAs-Devel/ OtherMPIs/ A complete [...]

  • Page 48

    5 – Software Installation Installing the InfiniPath and OpenFabrics RPMs 5-6 IB0056101-00 G S Inst alling the InfiniPath and OpenFabrics RPMs Linux distributions of InfiniPath and O pen Fabrics software are inst alled from binary RPMs. RPM is a Linux p ackaging and installa tion tool used by Red Hat, SUSE, and CentOS. There are multiple interdepe[...]

  • Page 49

    5 – Software Inst allation Installing the InfiniPa th and OpenFabrics RPMs IB0056101-00 G 5-7 A Choosing the RPMs to Inst all Although QLogic recommen ds that all RPMs are inst alled on all nodes, some are optional depending on which t ype of node is being use d. T o see which RPMs are required or optional for each type of node, according to its [...]

  • Page 50

    5 – Software Installation Installing the InfiniPath and OpenFabrics RPMs 5-8 IB0056101-00 G S Using rpm to Inst all InfiniPath and OpenFabrics The RPMs need to be available on each no de on which they will be used. One way to do this is to copy the RPMs to a directory on each node that will ne ed them. Another way is to put the RPMs in a director[...]

  • Page 51

    5 – Software Inst allation Installed La yout IB0056101-00 G 5-9 A Inst alled Layout The default inst alled layout for the InfiniPa th sof tware is described in th e following paragraphs. The InfiniPath shared libraries are inst alled in: /usr/lib for 32-bit applications /usr/lib64 for 64-bit applications MPI include files are in: /usr/include MPI[...]

  • Page 52

    5 – Software Installation S t arting the InfiniPath Service 5-10 IB0056101-00 G S Other OFED-installed modules may also be in this directory; these are also renamed if found during th e install process. S t arting the InfiniPath Service If this is the initial in stallation of InfiniPath (see “Lockable Memory Error on Initial Installation of Inf[...]

  • Page 53

    5 – Software Inst allation InfiniPath and Op enFabrics Driver Overview IB0056101-00 G 5-1 1 A InfiniPath and OpenFa brics Driver Overview The InfiniPath ib_ipath module provide s low level QLogic hardware su pport, and is the base driver for both the Infin iPath and OpenFabrics software components. The ib_ipath module does hardware initialization[...]

  • Page 54

    5 – Software Installation Configuring the InfiniPath Drive rs 5-12 IB0056101-00 G S Configuring the ib_ipath Driver The ib_ipath module provides both low-level In finiPath support and management functions for OpenFabrics protoco ls. The ib_ipath driver has several configuration variables th at set reserved buf fers for the sof tware, define event[...]

  • Page 55

    5 – Software Inst allation Configuring the InfiniPath Drivers IB0056101-00 G 5-13 A Servers typically have two Ethernet devices present, n umbered as 0 ( eth0 ) and 1( eth1 ). This example creates a third device, eth2 . 1. Check for the numbe r of Ethernet driv ers you currently have by typing either one of the following two comman ds: $ ifconfig[...]

  • Page 56

    5 – Software Installation Configuring the InfiniPath Drive rs 5-14 IB0056101-00 G S 4. Check whether the Ethernet driver has been loaded with: $ lsmod | grep ipath_ether 5. V erify that the driver is up with: $ ifconfig -a ipath_ether Configuration on SLES The following procedure causes the ipath_ ether network interfaces to be automatically con [...]

  • Page 57

    5 – Software Inst allation Configuring the InfiniPath Drivers IB0056101-00 G 5-15 A The Globally Unique IDentifer (GUID) can also be returned by running: # ipath_control -i $Id: QLogic Release2.2 $ $Date: 2007-09-05-04:16 $ 00: Version: ChipABI 2.0, InfiniPath_QHT7140, InfiniPath1 3.2, PCI 2, SW Compat 2 00: Status: 0xe1 Initted Present IB_link_u[...]

  • Page 58

    5 – Software Installation OpenFabrics Drivers and Servic es Configuration and S t artup 5-16 IB0056101-00 G S If you are using Dynamic Host Config uration Protocol (DHCP), add these lines to the file: STARTMODE=onboot BOOTPROTO=dhcp NAME=’InfiniPath Network Card’ _nm_name=eth-id-$MAC Proceed to St e p 6 . If you are you are using st atic IP a[...]

  • Page 59

    5 – Software Inst allation OpenFabrics Drivers an d Services Configuration and S tartup IB0056101-00 G 5-17 A Configuring the IPoIB Network Interface The following instructions show you how to manually configure your OpenFabrics IPoIB network interface. This example assume s that you are using sh or bash as your shell, all required InfiniPath and[...]

  • Page 60

    5 – Software Installation OpenFabrics Drivers and Servic es Configuration and S t artup 5-18 IB0056101-00 G S Configuring and Administering the VNIC Interface The Virtu alNIC (VNIC) Upper Layer Prot ocol (UL P) works in conjunction with firmware running on V irtual Input/Output (VIO) hardware such as the SilverS torm Ethernet V irtual I/O Control[...]

  • Page 61

    5 – Software Inst allation OpenFabrics Drivers an d Services Configuration and S tartup IB0056101-00 G 5-19 A service[ 1]: 1000066a00000101 InfiniNIC.InfiniConSys.Data:01 . . . 2. Create the VNIC interfaces using the configuration file /etc/infiniband/qlgc_vnic.cfg . Look at the qlgcvnictools/qlgc _vnic.cfg sample to see how VNIC configuration fi[...]

  • Page 62

    5 – Software Installation OpenFabrics Drivers and Servic es Configuration and S t artup 5-20 IB0056101-00 G S a. Format 1: Defining an IOC using the IOCGUID. Use t he following format to allow the host to connect to a specific VIO hardware card, regardless of which cha ssis and/or slot the VIO hardware card resides: {CREATE; NAME="eioc1"[...]

  • Page 63

    5 – Software Inst allation OpenFabrics Drivers an d Services Configuration and S tartup IB0056101-00 G 5-21 A Each CREA TE block must specify a unique NAME. The NAME represents the Ethernet interface name that will be registered wit h the Linux operating system. c. Format 3: S tarting VNIC using DGID. Following is an example of a DGID and IOCGUID[...]

  • Page 64

    5 – Software Installation OpenFabrics Drivers and Servic es Configuration and S t artup 5-22 IB0056101-00 G S 5. S tart the QLogic VNIC driver and the QLogic VNIC interfaces. Once you have created a configuration file, you can st art the VNIC drive r and create the VNIC interfaces specified in t he configuration file by running the following comm[...]

  • Page 65

    5 – Software Inst allation OpenFabrics Drivers an d Services Configuration and S tartup IB0056101-00 G 5-23 A If you want to restart the QLogic VNIC interfaces, run the following command: # /etc/init.d/qlgc_vnic restart Y ou can get information about the QLo gic VNIC interfaces by using the follo wing script: # ib_qlgc_vnic_info This information [...]

  • Page 66

    5 – Software Installation OpenFabrics Drivers and Servic es Configuration and S t artup 5-24 IB0056101-00 G S For example: # Use the UPDN algorithm instead of the Min Hop algorithm. OPTIONS="-R updn" SRP SRP stands for SCSI RDMA Protocol. It was originally intended to allow the SCSI protocol to run over InfiniBand for S torage Area Netw[...]

  • Page 67

    5 – Software Inst allation OpenFabrics Drivers an d Services Configuration and S tartup IB0056101-00 G 5-25 A ID: Data Direct Networks SRP Target System service entries: 1 service[ 0]: f60b04ff01000021 / SRP.T10:21000001ff040bf6 Note that not all the output is shown here; ke y elements are e xpected to show the match in St e p 3 . 3. Choose the d[...]

  • Page 68

    5 – Software Installation Other Configuration: Changing th e MTU Size 5-26 IB0056101-00 G S MPI over uDAPL Some MPI implementation s, such as Intel MPI and HP-MPI, can be run over uDAPL. uDAPL is the user mode version of the Direct Access Pro vider Library (DAPL). Examples of these types of MPI implemen tations are Intel MPI and one option on Ope[...]

  • Page 69

    5 – Software Inst allation S t arting and S topping the InfiniPath Sof tware IB0056101-00 G 5-27 A T o check the configuration state, use the command: $ chkconfig --list infinipath T o enable the driver , use the command (as root): # chkconfig infinipath on 2345 T o disable the driver on the next system boot, use the command (as root): # chkconfi[...]

  • Page 70

    5 – Software Installation Rebuilding or Reinstalling Dr ivers Af ter a Kernel Upgrade 5-28 IB0056101-00 G S An equivalent way to rest art infinipath this is to use same sequence as above, except use the restart command instead of start and stop : # /etc/init.d/opensmd stop # ifdown eth2 # /etc/init.d/infinipath restart # ifup eth2 # /etc/init.d/o[...]

  • Page 71

    5 – Software Inst allation Rebuilding or Reinstallin g Drivers if a Different Kernel is Installed IB0056101-00 G 5-29 A Rebuilding or Reinst allin g Drivers if a Different Kernel is Inst alled Installation of the InfiniPath driver RPM ( infinipath-kernel-2.2-xxx-yyy ) builds kernel modules for the curren tly running kernel version. T hese InfiniP[...]

  • Page 72

    5 – Software Installation Adapter Settings 5-30 IB0056101-00 G S Adapter Settings The following adapter settings can be adjusted for better performance.  Use taskset to tune CPU affinity on Opteron syste ms with the QLE7240, QLE7280, and QLE7140. Latency will be slightly lower for the Opteron socket that is closest to the PCI Express bridge. O[...]

  • Page 73

    5 – Software Inst allation Customer Acceptance Utility IB0056101-00 G 5-31 A  Use a PCIe MaxReadRequest size of at least 512 bytes with the QLE7240 and QLE7280. QLE7240 and QLE7280 adapters can supp ort sizes from 128 bytes to 4096 b yte in powers of two. This value is typically set by the BIOS.  Use the largest available PCIe MaxPayload si[...]

  • Page 74

    5 – Software Installation Customer Acceptance Utility 5-32 IB0056101-00 G S 6. V erifies the ability to mpirun jobs on the nodes. 7. Runs a bandwidth an d latency test on every p air of nodes and analyzes the results. The options available with ipath_checkout are shown in T able 5-6 . In most cases of failure, the script suggest s recommended a c[...]

  • Page 75

    5 – Software Inst allation Removing Sof tware Packages IB0056101-00 G 5-33 A Removing Sof tware Packages This section provides instructions for uni nstalling or downgrading the InfiniPa th and OpenFabrics sof tware. Uninst alling InfiniPath and OpenFabrics RPMs T o uninstall the InfiniPath soft ware p ackages on any node, type the following comma[...]

  • Page 76

    5 – Software Installation Additional Installation Instructions 5-34 IB0056101-00 G S Inst alling Lustre This InfiniPath release support s Lustre. Lustre is a fa st, scalable Linux cluster file system that interoperates with InfiniBand . T o use Lustre, you need:  A Linux kernel that is one of the supported kernels for this release, p atched wi[...]

  • Page 77

    5 – Software Inst allation Additional Installation Instr uctions IB0056101-00 G 5-35 A For example, insta ll all RPMs that relate to QLogic MPI in /usr/mpi/ qlogic . Leave all remaining Infin iPath libraries and tools in their de fault installation location ( /usr) . This approach leaves Infini Path libr aries (such as libpsm_infinipath.so and li[...]

  • Page 78

    5 – Software Installation Additional Installation Instructions 5-36 IB0056101-00 G S The following example inst allation is for a Red Hat Enterprise Linux 4 Up date 4 compatible kernel, where the /etc/redhat-release file indicates another distribution. If you are a bash or sh use r , type: # export IPATH_DISTRO=2.6.9_U4 Follow this with your norm[...]

  • Page 79

    5 – Software Inst allation Additional Installation Instr uctions IB0056101-00 G 5-37 A Rocks is a way to manage the kickst art automated inst allation method created by Red Hat. By using the Rocks conventions, the inst allation process can be automated for clusters of any size. A Roll is an extension to t he Rocks base distribution that support s[...]

  • Page 80

    5 – Software Installation Additional Installation Instructions 5-38 IB0056101-00 G S Use the following content s: <?xml version="1.0" standalone="no"?> <kickstart> <description> A skeleton XML node file. This file is only a template and is intended as an example of how to customize your Rocks cluster and use [...]

  • Page 81

    5 – Software Inst allation Additional Installation Instr uctions IB0056101-00 G 5-39 A <package>-libipathverbs-devel</package> <package>-libipathverbs</package> <package>-libmthca</package> <package>-librdmacm</package> <package>-libsdp</package> <package>-mstflint</package> [...]

  • Page 82

    5 – Software Installation Additional Installation Instructions 5-40 IB0056101-00 G S mode="create" perms="a+rx"> #!/bin/sh cd /home/install/contrib/4.2.1/x86_64/RPMS rpm -Uvh --force infinipath*.rpm ‘ls mpi*rpm | grep -v openmpi‘ # If and Only IF OpenSM is needed and then please enable OpenSM # only on one node. rpm -Uv[...]

  • Page 83

    5 – Software Inst allation Additional Installation Instr uctions IB0056101-00 G 5-41 A 6. The completion of the inst allation is done using the extend-compute .xml file. More specific instructions for completing the inst all process can be found under the Document ation lin k on the Rocks web site: http://www .rocksclusters.org/ Further Informati[...]

  • Page 84

    5 – Software Installation Additional Installation Instructions 5-42 IB0056101-00 G S Notes[...]

  • Page 85

    IB0056101-00 G A-1 A Inst allation T roubleshooting The following sections cont ain informat ion about issues that may occur during installation. Some of this material is repeated in the T roubleshooting appendix of the QLogic HCA and InfiniPath So f tware User Guide . Many programs and files are available that ga ther information about the cluster[...]

  • Page 86

    A – Installation T roubleshooting BIOS Settings A-2 IB0056101-00 G S MTRR Mapping and W rite Combining MTRR is used by the InfiniPath drive r to enable write combining to the QLo gic on-chip transmit buf fers. W rite co mbinin g improves write bandwid th to the QLogic chip by writing multiple words in a single bus transaction (typically 64 bytes)[...]

  • Page 87

    A – Installation T roubleshootin g Software Inst allation Issues IB0056101-00 G A-3 A Some BIOS’ do not have the MTRR mapping option. It may h ave a dif ferent name, depending on the chipset, vendor , BIOS, or other factors. For example, it is sometimes referred to as 32 bit memory hole . This setting must be enabled. If there is no setting for[...]

  • Page 88

    A – Installation T roubleshooting V ersion Number C onflict with opensm-* on RHEL5 Systems A-4 IB0056101-00 G S V ersion Number Conflict with opensm-* on RHEL5 Systems The older opensm-* p ackages that come with the RHEL 5 distribution have a version number (3) that is greater the InfiniPath version n umber (2.2). This prevents the newer InfiniPa[...]

  • Page 89

    A – Installation T roubleshootin g V ersion Number Conflict with opensm-* on R HEL5 Systems IB0056101-00 G A-5 A Missing Kernel RPM Errors Install the kernel-so urce , kernel-devel , and, if using an older release, kernel-smp-devel RPMs for your distribution before inst alling the InfiniPath RPMs, as there are dependencies. Use uname -a to find o[...]

  • Page 90

    A – Installation T roubleshooting V ersion Number C onflict with opensm-* on RHEL5 Systems A-6 IB0056101-00 G S Resolving Conflict s Occasionally , conflicts may arise when trying to inst all "on top of" an existing set of files that may come from a differen t set of RPMs. For example, if you inst all the QLogic MPI RPMs after having pr[...]

  • Page 91

    A – Installation T roubleshootin g V ersion Number Conflict with opensm-* on R HEL5 Systems IB0056101-00 G A-7 A In newer distributions, glibc is an RPM name. The 32-b it glibc is na med similarly to: glibc-2.3.4-2.i686.rpm OR glibc-2.3.4-2.i386.rpm Check your distribution for the exact RPM name. ifup on ipath_ether on SLES 10 Report s "unkn[...]

  • Page 92

    A – Installation T roubleshooting V ersion Number C onflict with opensm-* on RHEL5 Systems A-8 IB0056101-00 G S Notes[...]

  • Page 93

    IB0056101-00 G B-1 B Configuration Files T able B-1 contains descriptions of the configura tion and configuration template files used by the InfiniPath and OpenFabrics sof tware. T able B-1. Co nfiguration Files Configuration File Na me Description /etc/infiniband/qlogic_vnic.cfg V irtualNIC configuration file /etc/modprobe.conf S pecifies options [...]

  • Page 94

    B – Configuration Files B-2 IB0056101-00 G S /etc/sysconfig/net- work-scripts/ifcfg-<NAME> Network configuration file for network inter- faces. When used for ipath_ether , < NAME > is in the form eth X, where X is the number of the device, typically , 2, 3, etc. When used for VNIC configuration, the name is in the form eioc X, where X[...]

  • Page 95

    IB0056101-00 G C-1 C RPM Descriptions The following sections contain detailed des criptions of the RPMs for InfiniPath and OpenFabrics software. InfiniPath and OpenFabrics RPMs For ease of inst allation, QLogic recommends that a ll RPMs are installed o n all nodes. However , some RPMs are optional . Since cluster nodes can be used for different f u[...]

  • Page 96

    C – RPM Descriptions RPM Organization C-2 IB0056101-00 G S Non-InfiniPath component s may al so have their own version number: mvapich_gcc-2.2-33597.832.1_0_0.sles10_qlc.x86_64.rpm 1_0_0 is the 1.0.0 build for mvap ich . In all of the t ables in this appendix, t he build identifier is xxx and the distribution identifier is yyy . Using this conven[...]

  • Page 97

    C – RPM Descriptions Document ation and InfiniPa th RPMs IB0056101-00 G C-3 A The InfiniPath/RPMs are listed in Ta b l e C - 2 . The InfiniPath-Devel/RPMs are listed in Ta b l e C - 3 . T able C-2. InfiniPath/RPMs RPM Name Front End Compute Development infinipath-2.2- xxx _yyy .x86_64.rpm Utilities and source code InfiniPath configuration files C[...]

  • Page 98

    C – RPM Descriptions OpenFabrics RPMs C-4 IB0056101-00 G S The InfiniPath-MPI/RPMs are listed in T able C-4 . OpenFabrics RPMs OpenFabrics and OpenSM are optional componen ts. For ease o f installa tion, QLogic recommends that all of the OpenFabrics RPMs liste d in T able C-5 and T able C-6 be installed on all nodes. The development RPMs in T abl[...]

  • Page 99

    C – RPM Descriptions OpenFabrics RPMs IB0056101-00 G C-5 A dapl-utils-2.2- xxx .2_0_7. yyy .x86_64.rpm uDAPL support Optional for OpenFabrics ib-bonding-2.2- xxx .0_9_0. yyy .x86_64.rpm Utilities to manage and co ntrol the driver operation Optional for OpenFabrics ibsim-2.2- xxx .0_4. yyy .x86_64.rpm V oltaire InfiniBand Fabric Simulator Optional[...]

  • Page 100

    C – RPM Descriptions OpenFabrics RPMs C-6 IB0056101-00 G S libibverbs-2.2- xxx .1_1_1. yyy .x86_64.rpm Library that allows usersp ace processes to use In finiBand verbs as described in the InfiniBan d Architecture Specification . This library includes direct hardware access for fast p ath operations. For this library to be useful, a device-specif[...]

  • Page 101

    C – RPM Descriptions OpenFabrics RPMs IB0056101-00 G C-7 A qlgc_vnic_daemon-2.2- xxx .0_0_1. yyy .x86_64.rpm Used with VNIC ULP service Optional for OpenFabrics qlvnictools-2.2- xxx .0_0_1. yyy .x86_64.rpm S t artup script, sample config file, and utilities Optional for OpenFabrics qperf-2.2- xxx .0_4_0. yyy .x86_64.rpm IB performance test s Opti[...]

  • Page 102

    C – RPM Descriptions OpenFabrics RPMs C-8 IB0056101-00 G S libibcommon-devel-2.2- xxx .1_0_8. yyy .x86_64.rpm Development f iles for the libibcommon libra ry Optional for OpenFabrics libibmad-devel-2.2- xxx .1_1_6. yyy .x86_64.rpm Development f iles for the libibmad library Optional for OpenFabrics libibumad-devel-2.2- xxx .1_1_7. yyy .x86_64.rpm[...]

  • Page 103

    C – RPM Descriptions OpenFabrics RPMs IB0056101-00 G C-9 A libmlx4-2.2- xxx .1_0. yyy .x86_64.rpm Userspace driver for Mellanox ® ConnectX ™ Infin iBand HCAs Optional for OpenFabrics libmthca-2.2- xxx .1_0_4. yyy .x86_64.rpm Provides a device-specific usersp ace driver for Mellanox HCAs for u se with the libibverbs library Optional for OpenFab[...]

  • Page 104

    C – RPM Descriptions OpenFabrics RPMs C-10 IB0056101-00 G S mpitests_mvapich_pathscale-2.2- xxx .3_0. yyy .x86_64.rpm MV A PICH MPI tests compiled with PathScale Optional Optiona l Optional mpitests_mvapich_pgi-2.2- xxx .3_0. y yy .x86_64.rpm MV A PICH MPI tests compiled with PGI Optional Optiona l Optional mpitests_openmpi_gcc-2.2- xxx. 3.0 .y y[...]

  • Page 105

    C – RPM Descriptions OpenFabrics RPMs IB0056101-00 G C-1 1 A openmpi_intel-2.2- xxx .1_2_5. yyy .x8 6_64.rpm a Open MPI compiled with Intel Optional Optiona l Optional openmpi_pathscale-2.2- xxx .1_2_5. yy y .x86_64.rpm Open MPI compiled with PathScale Optional Optiona l Optional openmpi_pgi-2.2- xxx .1_2_5. yyy .x86_ 64.rpm Open MPI compiled wit[...]

  • Page 106

    C – RPM Descriptions OpenFabrics RPMs C-12 IB0056101-00 G S Notes[...]

  • Page 107

    IB0056101-00 G Index-1 Index A ACPI 4-4, A-1 Adapter , see HCA B BIOS configuring 4-4 settings A-1 settings to fix MTRR issues A-2 C -c 5-32 Cables supported 4-3 Compiler support 2-4 Configuration files B-1 ib_ipath 5-12 ipath_ether on Fedora, RHEL4, RHEL5 5-12 ipath_ether on SLES 5-14 OpenSM 5-23 VNIC 5-18 --continue 5-32 CPUs, HTX motherboards ma[...]

  • Page 108

    QLogic HCA and InfiniPath ® Software Install Guide V er sion 2.2 Index-2 IB0056101-00 G S H -h 5-32 Hardware installation o verview 3-1 Hardware requirements 4-1 HCA form factors 4-2 other HCAs-Devel/RPMs C-9 model numbers 2-1 not recognized A-1 other HCAs/RPMs C-8 QHT7140 installation with HT X riser 4-12 QHT7140 installation without an HTX riser[...]

  • Page 109

    IB0056101-00 G Index- 3 QLogic HCA and InfiniPat h ® Software Install Guide V er sion 2.2 A K -k 5-32 --keep 5-32 Kernel supported 2-3 Kernel, missing kernel RPM errors A-5 Kernels supported 2-3 L LEDs, blink patte rns 5-29 Linux, supported distributions 5-2 Lockable memory error A-7 Lustre, installing 5-34 M Model numbers for HCAs 2-1 MPI, other [...]

  • Page 110

    QLogic HCA and InfiniPath ® Software Install Guide V er sion 2.2 Index-4 IB0056101-00 G S QLE7280 4-7 QLogic MPI, installing in an alternate locatio n 5-34 R RD 2-5 RHEL4 and RHEL5, OpenFabrics library dependencies A-4 RHEL5 opensm-* packages, version n umber conflict A-4 Rocks, managing and inst alling sof tware with 5-36 RPMs choosing which to i[...]

  • Page 111

    IB0056101-00 G Index- 5 QLogic HCA and InfiniPat h ® Software Install Guide V er sion 2.2 A V -v 5-32 --verbose 5-32 VNIC, configuration 5-18 -vv 5-32 --vverbose 5-32 -vvv 5-32 --vvverbose 5-32 W --workdir=DIR 5-32[...]

  • Page 112

    QLogic HCA and InfiniPath ® Software Install Guide V er sion 2.2 Index-6 IB0056101-00 G S Notes[...]

  • Page 113

    [...]

  • Page 114

    © 2006–2008 QLogic Cor poration. S pecifications are subj ect to change without no tice. All right s reserv ed worldwid e. QLA, QLog ic , SANsurfer , the QLogic logo , InfiniPath, SilverS torm, and EKOPath are trademarks or registered trademarks of Q Logic Corpor ation. AMD Opteron i s a trademark of Advanced Microdevices Inc. BladeCenter and IB[...]