Mellanox Technologies MNPH29D-XTR manual

www .mellanox.com ConnectX ® -2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, MNPH29C-XTR, MNPH29D-XSR, MNPH29D-XTR Rev 1.6[...]

Mellanox T echnologies 350 Oakmead Parkwa y Suite 100 Sunnyvale, CA 94085 U.S.A. www .mellanox.com T el: (408) 970-3400 Fax: (408) 970-34 03 Mellanox T echnologies, Ltd. PO Box 586 Hermon Building Y okneam 20692 Israel T el: +972-4-909-7200 Fax: +972-4-959-3245 © Copyright 201 1. Mellanox T echno logies. All rights reserved. Mellanox®, BridgeX®,[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 3 T able of Content s Table of Contents 3 List of Figures 5 List of Tables 6 Revision History 7 About this Manual 8 Intended Audience 8 Related Docume ntation 8 Online Resources 8 Document Conventions 8 Technical Support 8 Firmware and Soft ware Updates 9 Chapter 1 Overview 10 1.1 A[...]

Rev 1.6 Mellanox Technologies 4 Appendix E Sicherheitshinweise (Warnings in German) 46 Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 47[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 5 List of Figures Figure 1: SFP+ NIC Component Side and Bracket Views 12 Figure 2: Card Product Label 14 Figure 3: Port Numbering 16 Figure 4: LED - Port Association 17 Figure 5: Physical and Log ical Link Indications 17 Figure 6: I2C Connecto r 18 Figure 7: Flash Jumper 19 Figure 8[...]

Rev 1.6 Mellanox Technologies 6 List of T ables Table 1: Revision History Table 7 Table 2: Documents List 8 Table 3: Network Interface Cards 11 Table 4: Mellanox Cards Part Numbering Key 13 Table 5: LEDs 17 Table 6: Jumper Configuration 19 Table 7: VPD Layout for MNPH29D-X[ST]R 19 Table 8: VPD Layout for MNPH29C-X[ST]R 20 Table 9: Hardware and Soft[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 7 Revision History This document was printed on 2/15 /1 1. T able 1 - Revision History T able Date Rev Comments/Changes Jan. 201 1 1.6 Changed Air flow requirement Added to the Set up section Dec. 19th, 2010 1.5 Minor formatting changes Nov . 22nd, 2010 1.4 Added Generation D ca rd [...]

Rev 1.6 Mellanox Technologies 8 About this Manual This User Manual describes Mellanox T echnologies ConnectX ® -2 Dual Port 10Gigabit Ethernet PCI Express x8 adapter cards. It provides details as to the interface s of the board, specifications, required software and firmware for opera ting the board , and relevant documentation. Intended Audience [...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 9 • T el: +1.408.916.0055 Customers who purchased Mellanox M-1 Global Su pport Services, please see your contract for details regarding T echnical Support. Customers who purchased Mellanox prod ucts thro ugh a Mellanox approved reseller should first seek assistance through their r[...]

Overview Rev 1.6 Mellanox Technologies 10 1 Overview This document is a User Manu al for Mellanox T echnologies cards based on the ConnectX ® -2 ENintegrated circuit device. The cards described in this manu al have the following main features: • IEEE S td 802.3 compli ant • 10 Gb/s Ethernet • Dual SFP+ ports for conn ection Ethernet traf fic[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 11 1.1 Adapter Cards Covered in this Manual T able 3 lists the Ethernet adapte r cards described in this manual. T able 3 - Network Interface Cards Ordering Part Number (OPN) PCI Express SERDES Speed Data T ransmission Rate/ # of ports Short / T all Bracket RoHS Adapter IC Part Numb[...]

Overview Rev 1.6 Mellanox Technologies 12 Figure 1: SFP+ NIC Component Side and Bracke t V iews J1 U1 MNPH29C J1 U1 MNPH29D[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 13 1.2 Mellanox Part Numbering Legend T able 4 describes the Mellanox T e chnologies adapter cards part numbering legend. For example, the part number MNPH29C-X TR describes Mellanox T echnologies’ ConnectX EN adapter card with two SFP+ ports, a PCIe2.0 x8 5.0GT/s interface, no on[...]

Overview Rev 1.6 Mellanox Technologies 14 1.3 Finding the MAC and Serial Nu mber on the Adapter Cards All Mellanox adapter cards have a label on the printed side of th e adapter card that has the card serial number , and the card MAC . Figure 2: Card Product Label 1.4 Safety W arnings For safety warnings in French se e “ A vertissements de sécur[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 15 3. During Lightning - Electrical Hazard 4. Copper Cable Connecting/Disconnecting 5. Equipment Installation 6. Equipment Disposal 7. Local and National Electrical Codes During periods of lightning activi ty , do not work on the equipment or connect or dis- connect cables. Some cop[...]

Adapter Card Interfaces Rev 1.6 Mellanox Technologies 16 2 Adapter Card Interfaces 2.1 I/O Interfaces Each card includes the following interfaces: •S F P + p o r t s • I/O panel LEDs •I 2 C compatible connector (for debug) For dual port cards, p ort 1 connects to connecto r 1 of the device, while port 2 co nnects to connec- tor 2 of the devic[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 17 2.1.3 LED Assignm ent The board has I/O LEDs located on the I/O panel. The green LED, when lit, indicates that the driver is running and a valid ph ysical connection between nodes ex ists. If the green LED is blink- ing, it indicates a problem with the physical link. The yellow L[...]

Adapter Card Interfaces Rev 1.6 Mellanox Technologies 18 2.1.4 I 2 C Comp atible Interface A three-pin header on the card is provided as the I 2 C compatible interface. See Appendix A,“Specifications,” on page 34 for the locati on on the board. Figure 6: I 2 C Connector 2.2 Power All adapter c ards receive 12V and 3.3 V power from the PCI Expre[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 19 2.3.3 EEPROM Each board incorporates an EEPROM that is accessible through the I2C. The EEPROM is used for storing the V ital Product Data (V PD). The EEPROM capacity is 4Kb. 2.4 VPD Layout The PCI VPD (V ital Product Data) layout, for each of the d escribed Mellanox T echnologies[...]

Adapter Card Interfaces Rev 1.6 Mellanox Technologies 20 76 VPD Keywor d V0 STR Misc Informa tion 78 Length 0x10 79 Data PCIe Gen2 x8 STR_SPC 95 VPD Keyword R V STR 97 Length 0x1 98 Data 0,97 %CS0 99 Large Resource T ype VP D-W T ag (0x11) 0x91 100 Length [7:0] LSB 0x99 101 Length [15:8] MSB 0x00 102 VPD Keyword V1 STR EFI Driver version 104 Length[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 21 76 VPD Keyword V0 STR Misc Information 78 Length 0x10 79 Data PCIe Gen2 x8 STR_SPC 95 VPD Keyword R V STR 97 Length 0x1 98 Data 0,97 %CS0 99 Large Resou rce T ype VPD-W T ag (0x11) 0x91 100 Length [7:0] LSB 0x99 101 Length [15:8] MSB 0x00 102 VPD Keyword V1 STR EFI Driver version[...]

Driver Software and Firmware Rev 1.6 Mellanox Technologies 22 3 Driver Sof tware and Firmware 3.1 Driver Sof tware 3.1.1 Linux For Linux, download and install th e latest MLNX_ EN driver software package available via the Mellanox W eb site at: http://www .mellanox.com => Downloads => Ethernet SW/Drive rs. Follow the installation instruc- tio[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 23 3.3 Up d ating Card Firmware Each card is shipped with the latest version of qualified firmware at th e time of manufacturing. Firmware is updated occasionally , and the mo st recent firmware can be obtained from: http://www .mellanox.com =>Suppo rt > Downlo ad Center . Che[...]

Driver Software and Firmware Rev 1.6 Mellanox Technologies 24 Figure 8: Suppor t Download Assist ant 3.4 ConnectX EN PXE The Mellanox ConnectX EN PXE soluti on enables the booting of servers from an IP based LAN environment. T o enable or disable PXE use the following procedure: The user must use the MFT tools: T o disable one must delete the PXE R[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 25 4 Ethernet Adapter Card Inst allation 4.1 Hardware and Sof tware Requirement s Before installing the adapter card, please make su re that the system meets the hardware and soft- ware requirements listed in T able 9. Refer to Chapter 3,“Driver Software and Firmware” on page 22[...]

Ethernet Adapter Card Installation Rev 1.6 Mellanox Technologies 26 Any PCI slot with the proper configuration is acce pt able for connection. If the card is installed in a PCI slot with less lanes than the card requires then the adapter card will not provide the optimum data transfer . 4.3 Set Up The basic steps to embed Mellanox cards in your com[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 27 Figure 9: Hard wa re Devices 3. Select a PCI Device / InfiniBand Controller entry . 4. Right-click. 5. Select “ Pr operties to display the PCI Devic e Properties ” window . 6. Click the Detail s tab and select Device In stance Id (W indows 2003) or Hardware Ids (W indows 2008[...]

Ethernet Adapter Card Installation Rev 1.6 Mellanox Technologies 28 Figure 10: PCI De vice 7. In the V a lue display box, check the fields VE N and DEV (fields are sepa rated by ‘&’). In the display exam ple above, notice the sub-strin g “PCIVEN_15B3&DEV_6368”: VEN is equal to 0x15B3 – this is the V endor ID of Mellanox T ech nol[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 29 4.3.1.2 Linux Get the device location on the PCI bus by runnin g lspci and locating lines with the string “Mel- lanox T echnologies”: > lspci | grep Mellanox 2:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0 ) Make sur[...]

Ethernet Adapter Card Installation Rev 1.6 Mellanox Technologies 30 Rate: 10 Base lid: 6 LMC: 0 SM lid: 3 Capability mask: 0x0 251086a Port GUID: 0x0 002c903000c871 1 Link layer: IB Check the OFED version T o get the version of the running Mellanox OFED/BXOFED, run th e fo llowing co mmand: Host# ofed_info | head -1 BXOFED-1.5.1-1.3.7-rc19: T roubl[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 31 The “Usage: openibd {start|stop|re start|status}” co mmand to modify this f ile and thereby control the drivers. Ethernet Driver Usage and Configuration T o assign an IP address to the interface run: #> ifconfig eth<n> <ip> where 'n' is the OS assigne[...]

Ethernet Adapter Card Installation Rev 1.6 Mellanox Technologies 32 The adapter cards are shipped without optical modules. Me llanox 10GBASE-SR (MFM1T02A- SR) and 10GBASE-LR (MFM1T02A-LR) optical mo dules ar e recommen ded. The fig ure below shows the Mellanox SFP+ module. Inserting the Optica l T ransceiver Module T o insert the module into the ca[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 33 Care should be taken not to impede the air ex haust flow through the ventilation holes. Cable lengths should be used wh ich allow for routing horizontally around to the side of the chassis before bending upward or do wnward in the rack. T o remove a c able, disengage the locks an[...]

Rev 1.6 Mellanox Technologies 34 Appendix A: S pecifications A.1 MNPH29D-X[ST]R Specifications T able 10 - Specifications for MNPH29D-X[ ST]R Physical Power and Envir onmental Size: Air Flow: 10Gb/s Connector: 2.71in x5.60in (68.90mm x 142.25mm) 200LFM a SFP+ a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 35 A.2 MNPH29C-X[ST]R Specifications A.3 Board Mechanical Drawing and Dimensions All of the cards covered in this User Manual have the same mechani cal drawing and share the same dimensions as depicted in Figure 1 3 or Figure 14. T able 1 1 - Specificati ons for MNPH29C-X[ST]R Physi[...]

Rev 1.6 Mellanox Technologies 36 Figure 13: Schematic of the Connect X-2 Small Form Factor MNPH29D Adapter Ca rd All dimensions are i n millimeters. All the mechanical tolerances are +/-0.1mm J4 J3 U1 J1 U3 Q1 L8 U4 U14 45.01 3.65 1.90 15.00 45.96 142.25 56. 15 12.75 64.40 Q2 J7 J8[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 37 Figure 14: Schematic of th e ConnectX-2 Card MNPH29C Adapter Card A.4 EMC Certification St atements T able 12 lists the approved certification stat us per in different reg ions of the world. T able 12 - Adapter Cards Certificat ion S tatus Adapter Card P/N FCC VCCI EN ICES CE CB [...]

Rev 1.6 Mellanox Technologies 38 A.4.1 FCC St atement s (USA) Class A S tatements: § 15.19(a)(4) This device complies with P art 15 of the FCC Rules. Operation is subject to th e following two conditions: 1. This device may not cause harmful interference, and 2. This device must accept any interference re ceived, including interference that may ca[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 39 A.4.4 VCCI St atement s (Japan) Class A S tatement: (T ranslation - "This is a Class A product based on the standard of the V oluntary Control Council for Interference by Information T echnology Equi pment (VCCI). If this equipment is used in a domestic environment, radio in[...]

Rev 1.6 Mellanox Technologies 40 Appendix B: Interface Connectors Pinout B.1 I 2 C-Comp atible Connector Pinout Figure 15: Compatible Connector Plug and Pinout B.2 PCI Express x8Connector Pinout These cards use a standard PCI Express x8 edge conne ctor and the PCI Express x8 standard pinout according to the PCI Express 2.0 s pecification. 5 1 2 3 4[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 41 B.3 PCI Express Connector Pinout B.4 SFP+ Connector Pinout Figure 16: Rear V iew of Module With Pin Placement 13.70 8.50 Top VeeT VeeT VeeT TX_Fault TX_Disable SDA SCL MOD_ABS RS0 RX_LO S RS1 VeeR VeeR VeeR VccT VccR TD+ TD- RD- RD+ 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 0 1 1 1 1 1[...]

Rev 1.6 Mellanox Technologies 42 T able 14 - SFP+ Connector Pinout Pin Symbol Name Description 1 V eeT T ransmitter Ground (Comm on with Receiver Ground) a a. Circuit ground is internally isolated from chassis ground. 2 T X _ F a u l t T ransmitter Fault. b b. T FA U LT is an open collector/drain out put, which should be p ulled up with a 4.7k – [...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 43 Appendix C: Replacing a T all Bracket With a Short Bracket This section provides instructions on how to remove the tall bracket of a standard Me llanox T ec h- nologies adapter card and replace it with a sh ort one. It includes the following sections: • Removing a brac ket • [...]

Rev 1.6 Mellanox Technologies 44 Figure 18: Putting on the Bracket 6. Make sure that the LEDs a re a ligned onto the bracket holes. 7. Use a torque driver to apply up to 2 lbs-in torque on the screws.[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 45 Appendix D: A vertissements de sécurité d’inst allation (W arnings in French) 1. Instructions d’installation 2. T empérature excessive 3. Orages – dangers électriques 4. Branchement/débranchement des câbles InfiniBand en cuivre 5. Installation du matériel 6. Eliminat[...]

Rev 1.6 Mellanox Technologies 46 Appendix E: Sicherheit shinweise (W arnings in German) 1. Installationsanleitungen 2. Übertemperatur 3. Bei Gewitter - Elektrische Gefahr 4. Anschließen/T rennen von InfiniBand-Kupferkabel 5. Geräteinstallation 6. Geräteentsorgung 7. Regionale und nationale elektrische Bestimmungen Lesen Sie alle Installatio nsa[...]

ConnectX-2 Ethernet Card User Manual Rev 1.6 Mellanox Technologies 47 Appendix F: Advertencias de seguridad p ara la inst alación (W arnings in S p anish) 1. Instrucciones de instalación 2. Sobreca lentamiento 3. Cuando hay rayos: pelig ro de descar ga eléctrica 4. Conexión y desconexión del cable Copp er InfiniBand 5. Instalación de equipos [...]

Rev 1.6 Mellanox Technologies 48 Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacio- nales.[...]