Sun Microsystems PCI manual

Sun Microsystems, Inc. www .sun.com Submit comments about this document at: http://www.sun.com/hwdocs/feedback Sun™ Dual P or t 4x IB Host Channel Adapter PCI-X Installation and User’ s Guide P ar t No . 819-1280-10 September 2005, Revision A[...]

Copyright 2005 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved. Sun Microsystems, Inc. has intellectual pr operty rights relating to technology that is described in this document. In particular , and without limitation, these intellectual property rights may include one or mor e of the U.S. pat[...]

iii Contents Regulatory Compliance Statements xi Declaration of Conformity xv Preface xvii How This Book Is Organized xvii Using UNIX Commands xviii Shell Prompts xviii Typographic Conventions xix Related Documentation xix Documentation, Support, and Training xx Third-Party Web Sites xx Contacting Sun Technical Support xx Sun Welcomes Your Comments[...]

iv Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 LED Displays 3 Hardware, Software, and Cable Requirements 5 2. Adapter Installation 7 IB-HCA Bus and Slot Selection 7 ▼ Installing the Adapter 7 ▼ V erifying the Installation 9 3. Conf iguring IP Over Inf iniBand 11 IPoIB Configuration 11 ▼ Co[...]

Contents v Other Useful Utilities 26 cfgadm 26 snoop 27 netstat 27 kstat 27 Index 29[...]

vi Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

vii Figur es FIGURE 1-1 Sun Dual Port 4x IB Host Channel Adapter PCI-X Card 2 FIGURE 1-2 LED Layout 4 FIGURE 2-1 Typical Adapter-to-Switch Configuration 9 FIGURE 4-1 InfiniBand Software Stack 16 FIGURE A-1 Sun 4x-to-12x InfiniBand Cable Pin Assignments 21[...]

viii Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

ix T ables TABLE 1-1 IB-HCA LEDs 3 TABLE 1-2 Hardware, Software, and Cable Requirements 5 TABLE B-1 SunVTS Documentation 24[...]

x Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

xi Regulatory Compliance Statements Y our Sun product is marked to indicate its compliance class: • Federal Communications Commission (FCC) — USA • Industry Canada Equipment Standard for Digital Equipment (ICES-003) — Canada • V oluntary Control Council for Interference (VCCI) — Japan • Bureau of Standards Metrology and Inspection (BS[...]

xii Sun Dual P or t 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 ICES-003 Class B Notice - A vis NMB-003, Classe B This Class B digital apparatus complies with Canadian ICES-003. Cet appareil numérique de la classe B est conforme à la norme NMB-003 du Canada. BSMI Class A Notice The following statement is ap[...]

Regulatory Compliance Statements xiii CCC Class A Notice The following statement is applicable to products shipped to China and marked with “Class A” on the pr oduct’s compliance label. GOST -R Certification Mark[...]

xiv Sun Dual P or t 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

xv Declaration of Conformity EMC USA—FCC Class A This equipment complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: 1. This equipment may not cause harmful interference. 2. This equipment must accept any interference that may cause undesir ed operation. European Union This equipment complies with the foll[...]

xvi Sun Dual P or t 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

xvii Pr eface The Sun™ Dual Port 4x IB Host Channel Adapter PCI-X Installation and User ’s Guide provides installation instructions for the Sun Dual Port 4x IB Host Channel Adapter PCI-X card. This manual includes an overview of the adapter , supported platforms, cable information, and related documentation. These instructions are designed for [...]

xviii Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 Using UNIX Commands This document might not contain information about basic UNIX® commands and procedur es such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information: ■ Software d[...]

Preface xix T ypographic Conventions Related Documentation The following documents provide additional information regar ding the IB-HCA and related pr oducts. T ypeface 1 1 The settings on your browser might differ from these settings. Meaning Examples AaBbCc123 The names of commands, files, and directories; on-scr een computer output Edit your .lo[...]

xx Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 Documentation, Support, and T raining Thir d-Party W eb Sites Sun is not responsible for the availability of third-party web sites mentioned in this document. Sun does not endorse and is not responsible or liable for any content, advertising, produc[...]

Preface xxi Sun W elcomes Y our Comments Sun is interested in impr oving its documentation and welcomes your comments and suggestions. Y ou can submit your comments by going to: http://www.sun.com/hwdocs/feedback Please include the title and part number of your document with your feedback: Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation[...]

xxii Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

1 CHAPTER 1 Pr oduct Overview This chapter provides an overview of the Sun Dual Port 4x IB Host Channel Adapter PCI-X card and includes the following sections: ■ “Product Features” on page 1 ■ “Hardwar e Description” on page 1 ■ “I/O Interfaces” on page 3 ■ “LED Displays” on page 3 ■ “Hardwar e, Software, and Cable Requi[...]

2 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 The board characteristics ar e: ■ Power: approx. 12 watts total ■ Low prof ile PCI form-factor card (appr ox. 2.5 in. by approx. 6.7 in.) ■ Requires 4x Inf iniBand cables ■ 3.3V PCI-X or PCI slot (Note that the IB-HCA can not be plugged into [...]

Chapter 1 Product Overview 3 I/O Interfaces The IB-HCA includes the following interfaces: ■ T wo 4x InfiniBand copper connectors ■ PCI-X 64-bit 133 MHz edge connector InfiniBand Interface The IB-HCA provides two 4x Inf iniBand v1.1 connector (or ports) for external copper cables and is IBT A specification 1.1-compliant. PCI-X Interface The PCI-[...]

4 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 FIGURE 1-2 LED Layout Logical link (Amber LED) Ph ysical link (Green LED)[...]

Chapter 1 Product Overview 5 Har dware, Softwar e, and Cable Requir ements The hardwar e, software, and cable requirements for the Sun Dual Port 4x IB Host Channel Adapter PCI-X card are listed in T ABLE 1-2 . T ABLE 1-2 Hardwar e, Software, and Cable Requirements Operating System Servers Solaris 10 SP ARC Sun Fire™ V240, V440, V490, V890, V1280,[...]

6 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

7 CHAPTER 2 Adapter Installation This chapter describes how to install the adapter in your system and verify that it has been installed correctly . IB-HCA Bus and Slot Selection For optimal performance, the IB-HCA card should be installed in the fastest available bus, preferably a 133 MHz PCI-X bus. But the IB-HCA can operate on the slower PCI-X an[...]

8 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 2. Remove the cover from the unit to access the card slots and connectors. 3. Select an available 3.3 V olt PCI-X or PCI slot and remove the blank front panel. If you are r eplacing an existing card, r emove the card. 4. Install the IB-HCA card into [...]

Chapter 2 Adapter Installation 9 This completes the hardwar e installation. Proceed to the verif ication instructions in “V erifying the Installation” on page 9 . FIGURE 2-1 T ypical Adapter-to-Switch Configuration ▼ V erifying the Installation Before you can verify the installation, you must install the adapter in the chassis, power the serv[...]

10 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 2. Check that the amber LED is illuminated for each port that is connected to the switch. If the amber LED is not on, refer to Appendix B , “Diagnostic Software and T roubleshooting” on page 23 for more information.[...]

11 CHAPTER 3 Conf iguring IP Over Inf iniBand This chapter contains the following sections: ■ “IPoIB Configuration” on page 1 1 ■ “Managing IB Devices for IPoIB” on page 13 IPoIB Conf iguration IP Over InfiniBand (IPoIB) is automatically configur ed when the system is booted. The following procedur e can be used when making changes to y[...]

12 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 For example: Refer to dmesg (1M) for more information. 2. Create the Inf iniBand devices using the following command: A /dev/ibd n device is created for each IB-HCA port in the system where n is the port number . Refer to devfsadm (1M) for more info[...]

Chapter 3 Configuring IP Over InfiniBand 13 5. V erify that the devices are plumbed. T o do this, type the command: The output includes information on the IB devices. 6. Check that data can be send over the Inf iniBand link. Use the ping command to send ICMP (ICMP6) ECHO_REQUEST packets and receive ICMP (ICMP6) ECHO_RESPONSE packets between Inf ini[...]

14 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

15 CHAPTER 4 Inf iniBand Software Overview This chapter includes the following topics: ■ “InfiniBand Software” on page 15 ■ “Related Softwar e Documentation” on page 18 Inf iniBand Software The InfiniBand softwar e is included in the Solaris 10 Operating System. FIGURE 4-1 shows a block diagram of the InfiniBand software stack which inc[...]

16 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 FIGURE 4-1 Inf iniBand Software Stack User Dir ect Access Pr ogramming Library This library supports service providers written according to the uDAPL 1.2 specification. Sun pr ovides default service provider modules to support the T avor HCA driver [...]

Chapter 4 InfiniBand Software Ov erview 17 Inf iniBand T ransport Framework IBTF is the Solaris implementation of the InfiniBand T ransport Layer (IBTL). IBTF consists of InfiniBand Device Manager (IBDM), Inf iniBand Communications Manager (IBCM) and an InfiniBand Nexus driver . The Nexus driver along with IBDM creates and manages Inf iniBand devic[...]

18 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 Related Softwar e Documentation Manuals ■ System Administration Guide: Devices and Files System (817-5093) For information on InfiniBand devices, see the System Administration Guide: Devices and Files System (817-5093) ■ System Administration Gu[...]

19 APPENDIX A Inf iniBand Cable Pin Assignments FIGURE A-1 shows the InfiniBand cable connectors and pin assignments.[...]

20 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 FIGURE A-1 Sun 4x-to-12x InfiniBand Cable Pin Assignments S1 G1 G25 S47 S2 S48 S16 G9 S2 S15 S1 G1 P1 (12X) P2 (4X) G S1 S2 G S3 S4 G S5 S6 G S7 S8 G S41 S42 G S43 S44 G S45 S46 G S47 S48 G SHELL G1 S16 S15 G2 S14 S13 G3 S12 S11 G4 S10 S9 G5 S8 S7 G[...]

23 APPENDIX B Diagnostic Softwar e and T r oubleshooting This appendix provides an overview of the SunVTS™ diagnostic application and troubleshooting tips for checking the adapter . There is also a section outlining some common troubleshooting issues. This appendix contains the following sections: ■ “SunVTS Diagnostic T esting” on page 23 ?[...]

24 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 Using the SunVTS ibhcatest The ibhcatest diagnostic test checks the functionality of Sun Dual Port 4x IB Host Channel Adapter PCI-X card. This test can be run fr om the SunVTS user interface, or it can be run from the command line. See the SunVTS 6.[...]

Appendix B Diagnostic Software and T roub leshooting 25 ▼ Using the ibhcatest 1. Ensure that the SunVTS software and the T avor driver are installed on your system. T o do this, type the following command: If a SunVTS software package is not installed, refer to the SunVTS User’ s Guide for installation instruction. If the SUNWtavor package is n[...]

26 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 If an InfiniBand softwar e package is not installed, check your Solaris Operating System documentation for software package information. ■ Check the system log or console for T avor driver error messages. See tavor (7D) for error messages and desc[...]

Appendix B Diagnostic Software and T roub leshooting 27 snoop The snoop program captur es and inspects network packets. See the snoop (1M) man page for details. For example: netstat netstat shows network status. See the netstat (1M) man page for details. For example: kstat kstat displays kernel statistics. See the kstat (1M) man page for details. F[...]

28 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005[...]

29 Index A adapter description, 1 features, 1 installation, 7 layout, 2 C cable requir ements, 5 cables connectors, 19 part numbers, 5 pin assignments, 19 D diagnostics, ibhcatest, 24 documentation man pages, 18 related, xix, 18 SunVTS, 23 I I/O interfaces, 3 IB-HCA installation, 8 Sun Dual Port 4x IB Host Channel Adapter PCI-X card, 1 ibhcatest di[...]

30 Sun Dual P or t 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005 versions, 5 T T avor driver, 17 troubleshooting IB-HCA, 25 tasks, 25 utilities, 26 U uDAPL, 15, 1 6 utilities, 26 cfgadm, 26 kstat, 27 netstat, 27 snoop, 27[...]