Cisco Systems SFS 3012 manual

Part N u m b e r : OL-7722-01 Cisco SFS 3012 Multifabric Server Switch Hardware Guide ( Topspin 360 Hardware Guide ) Web Release v2.1.0 Product ID: 140262[...]

Copyright © 2 004 - 2005 T opspin Communi cations, Inc. All rights reserved. The T opspin Server Switch, T opspin Host Channel Adapter, T opspin Element Manager , and collateral software programs and documentation are subject to and made available only pursuant to the license agreem ent signed by you and T opspin, Com munications, Inc. (or if no s[...]

iii Contents Regulatory Notices ........................................ vii FCC Statement .................................................................................................................. ........................ vii Safety Inform ation ....... .......................................................................................[...]

iv Introduction ................................................................................................................... ............................... 5 Controller Cards ............................................................................................................... ............................ 6 Install a Controller Car[...]

v Installing a Power Supply ...................................................................................................... ....... 44 Removing a Power Supply ........................................................................................................ ... 45 Replacing Blower Trays .................................................[...]

vi[...]

vii Regulatory Notices FCC S t atement This equipment has been tested and found to comply wit h the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. Th ese limits are designed to provide reasonabl e protection against harmful interference when the equi pment is operated in a commerci al environment. This equipment generate[...]

viii CAUTION: Grounding is supplied by the gro und-prong on the 3-prong power cable. Do not attach a separate ground cable. Do not use adapter plugs. Do not remov e the ground prong from the cable. Ensure the ground connection on the power supply is correct and functioning before applying power to the T ops pin 360. CAUTION: Always ground yourself [...]

ix CAUTION: Use only T opspin-approved gateway expansion modul es, controller cards, fans, and power supply modules. CAUTION: Never place your hand inside an empty card or module bay . Y ou should never have cause to place a hand anywhere inside the T opspin chassis. Unused card and module bays should always have a T opspin co ver over the bay to e[...]

x CAUTION: Only open, modify , or service laser devi ces as specified in the manufacturer documentation. CAUTION: Use of control s or adjustments or performanc e of procedures other than those specified herein may result in hazardo us laser radiation exposure. Laser S pecs Cont act Information T able iii-1: T opspin P/N Applicat ion Optical Paramet[...]

xi Product Registration THANK YOU FOR PURCHASING YOUR T OPSPIN® PRODUCT! T opspin Comm unications is pleased that you have selected T opspin for your computing infrastructure. T opspin provides co mprehensive customer s upport resources via telephon e and W eb access. T opspin’ s Customer Support W eb site has the latest T opspinOS (IB Switch an[...]

xii • Phone: (800) 499 -1473 • Online Help Desk: htt p://support.topspi n.com[...]

xiii S t andard Product Agreement PLEASE READ THESE TERMS AND CONDITI ONS CAREFULL Y BEFORE USING THE PRODUCTS THA T ARE PROVIDED WITH THIS AGREEMENT . BY USING ANY PRODUCTS, YOU ACKNOWLEDGE THA T YO U HA VE READ AND UNDERST OOD ALL THESE TERMS AND CONDITIONS AND YOU WI LL BE CONSENTING TO BE BOUND BY THEM. IF YOU DO NOT ACCE PT THESE TERMS AND CON[...]

xiv 3. Payment/Financing . T opspin will determine Customer's cred it terms on a per-order basis and such terms are subject to a credit revi ew and approval by T opspin in its sole discretion. The amount of credit may be changed or credit w ithdrawn by T opspin at anytime in its sole discretion. All payments shall be made in U.S. Dollars net t[...]

xv 6. Intellectual Property Rights . T itle to and ownership of th e Software and any associated documentation, and any im proved, updated, modified or additi onal parts thereof, and al l copyright, patent, trade secret, trademark a nd other intellectual pro perty rights embodied in the Products, shall at all times remain the property of T opspin o[...]

xvi HA VE OTHER RIGHTS, W HICH V AR Y FROM ST A TE/JURISDICTION TO ST A TE/JURISDI CTION. 10. Confidentiality . “Confidential Information” means any oral , written, graphic or machine-readable information disclosed by T opspin that is designated in writin g to be confidential or proprietary or should be reasonably understood to be confidenti al[...]

xvii acquisition or sale of all or substantiall y all of assets relating to this Agreement. T opspin or any successor may assign all of part of the right to pa yments under this Agreem ent. Any assignment or transfer of this Agreement mad e in contravention o f the terms hereof shall be null and void. Subject to the foregoing, the terms and co ndit[...]

xviii[...]

1 1 T op spin 360 Overview The following sections ap pear in this chapter: • “Introduction” on page 1 • “Features” on page 2 Introduction The T opspin 360 Server Swi tch provides data center managers with a high-performance, low-latency interconnect that runs Fibre Channel and Ethernet traffic transparently ov er an InfiniBand (IB) back[...]

2 Figure 1-2 displays the back of the T opspin 360 Server Switch and numbers the slots on the chass is. Figure 1-2: T opspin 360 Chassis - Back V iew (Service End) with Slot Numbers The vertical slots at the top of the service end of the T opspin 360 are numbered 1 to 14 from left to right. The horizontal slots at the bottom of the service end of t[...]

3 Redundancy and High-A vailability The T opspin 360 is arch itected for full redundancy within a single box. Every component in the data path can be dual-connected from end to end. For example, a se rver with a 2-port HCA can be dual-attached to the two switch blades on the T opspin 360. Each gateway is also dual atta ched to the dual switch blade[...]

4[...]

5 2 T op spin 360 Component s The following sections ap pear in this chapter: • “Introduction” on page 5 • “Controller Cards” on page 6 • “Ethernet and Fibre Channel Gateways” on page 9 • “InfiniBand Switch Cards” on page 10 • “Power Supply Modules” on page 13 • “Blower T rays” on page 15 Introduction The compone[...]

6 CAUTION: Never place your hand inside an empty card or module bay . Y ou should never have cause to place a hand anywhere inside th e T opspin 360 chassis. Unused car d and module bays should always have a T opspin 360 blanking panel over the bay to ensure proper safety , vent ilation, and cooling. Controller Cards Controller cards manage the T o[...]

7 3. Remove the controller card from any packagin g. 4. Extend the ejector lever on the card completely . 5. Insert the controller card module into slot 1 or sl ot 14 such that the ejector lever sits on the bottom-left-hand side o f the module. 6. Push the controller card fi rmly into the slot. The eject or lever begins to close. Figure 2-2: Insert[...]

8 7. Push the ejector lever up until it clicks into place. Figure 2-3: Closing the Ejecto r Lever[...]

9 Ethernet and Fibre Channel Gateways T opspin Ethernet and Fibre Channel gateways connect the T opspin 360 , and IB-connected hosts, to IP and FC networks. All gateways are dual-connected to the redundant InfiniBand b ackplane of the T op spin 360. Figure 2-4 displays an Ethernet gateway . Figure 2-5 displays a Fibre Channel gateway . Figure 2-4: [...]

10 Inst all an Ethernet or Fibre Channel Gateway T o inst all a gateway , perform the following steps: 1. Ground yourself appropriat ely . 2. Remove the blanking panel if one resides in the slot. 3. Remove the gateway module from an y packaging. 4. Extend the ejector lever on t he gateway completely . 5. Insert the gateway into an expansion mo dule[...]

11 Inst all InfiniBand Switch Cards T o inst all a switch card, pe rform the follo wing steps: 1. Ground yourself appropriat ely . 2. Remove the blanking panel if one resides in the slot with a #1 Phillips-head screwdriver . 3. Remove the switch card from any packaging. 4. Extend the ejector levers on the left- and right-hand sides of the switch ca[...]

12 7. Close both fastener-levers simultaneously and completely . Figure 2-8: Closing Switch Ejector Levers[...]

13 8. T ighten the screw s on either side of th e switch card to secure it to the chassis. Figure 2-9: Securin g Switch Card Remove InfiniBand Switch Cards T o remov e an InfiniBand switch card from the T o pspin 360 chassis, perform the following steps: 1. Make sure the T opspin 360 chassis is secure. 2. If you are removing the InfiniBand card in [...]

14 power supply , you can add a second power supply while the chassis runs. If you have two power supplies installed, you can remove one of them without remov ing power from the chassis. Figure 2-10 displays the location of the power suppl ies in the T opspin 360. Figure 2-10: Power Supply Modules Inst all Power Supply Modules T o in sert a power s[...]

15 NOTE: The power socket on the back (service end) of the T opspin 360 corresponds to the po wer supply module on the same si de of the Server Switch. Removing a Power Supply T o remove the power supply , perform the foll owing steps: 1. Remove the power cord from the power supply . 2. Disengage the power supply by pullin g it toward you. 3. Slide[...]

16 3. Insert the blower tray in to an open slot until it is fully seated, such that the captive screw sits at the top of the tray . 4. T ighten the capt ive screw on the front pa nel to secure the tray to the chassis. Removing Blower T rays T o remove a blower tray , perform the following steps: 1. Ground yourself appropriat ely . 2. Unfasten the c[...]

17 4. Insert a replacement blower tr ay into the empty chassis bay to ensure appropriate cooling. NOTE: Each blower module contains two fans for a to tal of four fans. The system can tolerate one out of four blowers failing. Wh en you remove a blower module, you have three minutes to replace it with a fully-functionin g blower module.[...]

18[...]

19 3 Inst all and Boot the T op spin 360 The following sections ap pear in this chapter: • “Introduction” on page 1 9 • “Prepare the Site for the T opspin 360” on page 1 9 • “Install T opspin 360 Components” on page 20 • “Rack the T opspin 36 0” on page 20 • “Attach a Serial Console Cable to a PC or T erminal” on page [...]

20 Read the Cautionary S t atement s Refer to the “FCC Statement” on page vii and the “Safety Information” on page vii befo re you install your T o pspin 360. Prep are the Physical Environment for the T op spin 360 • Ground yourself using an approved ground wrist-strap. • Make sure you have the right cables and sufficient ventilation. ?[...]

21 7. Attach the side flanges to the sides of th e chassis with the screws provided, as shown Figure 3-1 and Figure 3-2 . Figure 3-1: Positioning the Side Flanges Figure 3-2: Fastening the Side Flanges 8. If necessary for your rack, add fast eners with appropriate threads to your rack to prepare your rack for the rails.[...]

22 9. Position the chassis brackets to the rack rails, as shown in Figure 3-3 . Figure 3-3: Rail Positioning[...]

23 10. Screw the top and bottom of the front of the rails to the rack but do not fully tig hten the scr ews . Figure 3-4 displays the rail loo sely connected to the rack. NOTE: When you connect the rails, do not ful ly ti ghten the screws. When you slide the chassis into the rails, you may need t o reposition the rail s to align the flang e screws [...]

24 1 1. Screw the top and bottom of the back of the rails to the rack, as shown in Figure 3-5 . Figure 3-5: Att aching Rails from the Back[...]

25 12. Slide the T o pspin 360 onto the rails as shown in Figure 3-6 and Figure 3-7 . Figure 3-6: Positioning the Chassis on the Rails Figure 3-7: Sliding the Chassis onto the Rails[...]

26 13. Adjust the front of the rails (that you left loose in step 10 ) so that the threaded hole in the center of the front of the rail aligns with the screw in the flange. 14. Screw the flange screws tightly into the rails, as shown in Figu re 3-8 . Figure 3-8: Securin g the Chassis to the Rails 15. Firmly tighten the front rail screws. For the VF[...]

27 • Stop Bits: 1 • Flow control: None For the VFrame Evaluation T est Plan, continue to Ste p b of “Hardware Configuratio n Steps” on page 14 in the “VFrame Evaluation T est Plan” . Boot your T op spin 360 and Configure Basic Connectivity T o configure basic conn ectivity , you must power-on the Server Switch and configure it with the [...]

28 3. Enter the interface mgmt-ethernet command. Example 4. Enter the IP address of the management port fo llowed by the net mask. Example 5. Set the default gateway of the managem ent port. Example 6. Enable the management port with th e no shutdown command. Y ou can now telnet to your T o pspin 360, or enter the IP address of your T opspin 360 in[...]

29 5. Enable the management port with th e no shutdown command. Y ou can now telnet to your T o pspin 360, or enter the IP address of your T opspin 360 in a W eb browser to launch Chassis Manager . Example 6. Save the configuration to preserve it between reboots. Example For the VFrame Evaluation T est Plan, continue to Ste p c of “Hardware Confi[...]

30 Use InfiniBand cables to connect the HCA in your hos t to the InfiniBand switch card of your T opspin 360. T o plug in an InfiniBand cab le, push the connector into the interface until you hear/feel a click. Figure 3-9: Fully Inst alled IB Cable with Pinch Co nnector Figure 3-10: Fully Inst alled IB C able with Pull Connector NOTE: If your host [...]

31 T o remove a cable with a pinch co nnector , pinch both sides of the back of the connector and pull the connector away from the port. Figure 3-1 1: Removing a Pinch Connector[...]

32 T o remove a cab le with a pull connector , grasp the connector with one hand and push it towar d the port, then pull the latch away from the port with your other hand and gently wiggle the con nector away from the port. Figure 3-12: Removing a Pull Connector Manage the T op spin 360 Y ou can manage the T opspin 360 with the follo wing methods: [...]

33 V iewing and Up dating the Sof tware Image on the Chassis V iew the Sof tware Image V ersion with the CLI 1. T o view the primary software image vers ion for all cards in the chassis, use the show card-inventory command. Look for the primary-image-source field for each slot. (The slots are described in Figure 1-2 .) 2. Note that it is possible t[...]

34 An installed image file will have a slash (/) in the name. Example V iew the Sof tware Image V ersion with the GUI Use the Element Manager GUI to view th e software version(s) on the chassis: 1. Launch Element Manager . 2. Select Maintenance > System Info to view the version of software installed on the controller . Figure 3-13: Sof tware Ima[...]

35 3. V iew the software version on individual cards in the chassis: a. Double-click on any card in the main GUI. b. Click the Inventory tab. The properties wi ndow appears for that card. Figure 3-14: V iew Current Image of an Individual Ca rd c. V iew the Current Image Sour ce field. If you find a card that has a different image than the others, y[...]

36 NOTE: If you do not have enough available memory to in stall an additional image, delete an old image. “Delete an Image with the CLI” on page 38 . 3. Specify the new boot image to be used. a. Find the name of your new syst em image by entering the di r image command. Copy the name of the new image. b. Enter boot-config primary-image-sourc e [...]

37 3. Click the Import button. The Import File window appears. Figure 3-15: Copy an Imag e File to the Switch a. Select Image from the File T ype drop-down menu. b. Click the Remote FTP radio button from the Copy Fr om section. c. Enter the name or IP address of the FTP se rver from which to co py the file in the Server Name or IP Addr ess field. d[...]

38 • All cards must have an oper-sta tus of “ Up ” a. Click on the newly copied. img file in the File Management window . b. Click the Install button. c. Click Ye s to the warning asking if you wish to proceed. NOTE: If you do not have enough available memory to in stall an additional image, delete an old image. “Delete an Image with the GU[...]

39 The following example deletes an image file from slot 1 (slots 1 and 14 can contain controller cards). Example Delete an Image with the GUI Y ou can delete an .img file through the GUI or the CLI. A .img file is an image file that is not installed. Do not delete an installed image fi le. An installed image will have a slash (/) in the name. 1. S[...]

40[...]

41 4 T op spin 360 Field Replaceable Unit s The following sections ap pear in this chapter: • “Controller Cards” on page 41 • “Expansion Modules” on page 41 • “InfiniBand Switch Cards” on page 42 • “Power Modules” on page 44 • “Replacing Blower T rays” on page 46 Controller Cards The T opspin 360 has two removable con [...]

42 InfiniBand Switch Cards Each InfiniBand switch card is held in place by two sc rews, one on the left side and one on the right side of the card. Inst alling an InfiniBand Switch Card NOTE: Slot 16 must always have an Infiniband switch card insert ed in order to maintain connectivity . T o inst all the InfiniBand switch card in the T opspin 360 c[...]

43 • The keyword all (The CLI defaults to “all” if you enter just “show card”). Example 2. Update the switch card firmware. En ter the install command: Example NOTE: Only cards with an oper-status of “up” are upda ted. If a card is down when you run instal l, or a card is added after running install, bring up the card and run install [...]

44 3. Reboot the switch card. Example Removing an InfiniBand Switch Card T o remov e an InfiniBand switch card from the T o pspin 360 chassis: 1. Make sure the T opspin 360 chassis is secure. 2. Properly ground yourself to avoid potential elect rostatic discharge and damage to the card. 3. Facing the back of the T op spin 360 chassis, unfasten both[...]

45 NOTE: The power supplies should be inserted in opposite direction s of each other . See Figure 4-2 . Figure 4-2: Power Supp ly Module Positioning 3. Press the power supply firm ly into the ch assis. Removing a Power Supply T o remove the power supply , perform the foll owing steps: 1. Remove the power cord from the power supply . 2. Disengage th[...]

46 Replacing Blower T rays The T opspin 360 fan tray is hot-swappable. Y ou do not need to turn off power to the chassis to replace the fan tray . Access the fan tray from the front of the T opspin 360 chassis. Inst alling a Blower T ray T o insert a blower tray , perf orm the following steps: 1. If a blanking panel is already attached to the modul[...]

47 3. T ighten the tw o captive panel screws on the front panel to secure the card to the chassis. Removing a Blower T ray T o remove a blower tray , perform the following steps: 1. Unfasten the fan tray screws. 2. Pull the fan toward you until it completely diseng ages from the chassis. 3. Fasten a blanking panel to the empty chassis bay to ensure[...]

48[...]

49 5 T op spin 360 LEDs The following sections ap pear in this chapter: • “Introduction” on page 4 9 • “T opsp in 360 System LED” on page 50 • “Power Supply LEDs” on page 5 1 • “Blower Tray LEDs” on page 52 • “Controller Card LEDs” on page 53 • “IB Switch Card LEDs” on page 54 • “IB Port LEDs” on page 54 ?[...]

50 • Ethernet cards • T opspin 360 bezel side The LEDs have special significance depending on the chassis component. This section describes the LED indicators of each component T op spin 360 System LED The T opspin 360 system LED appears on the top-left-han d corner of the front (bezel end) of the Server Switch. Y ou can view the system LED wit[...]

51 Power Supply LEDs Power supply statu s LEDs are located on either side of the power supply module pu ll handle. Controller cards mirror Power Supply LEDs on the service end of the chassis. Figure 5-1 displays power supply LEDs. Figure 5-1: Power Supply LEDs Ta b l e 5 - 2 displays AC LED indications. Ta b l e 5 - 3 displays DC LED indications. T[...]

52 Blower T ray LEDs Blower tray LEDs appear immediat ely to the left of the System LED on the bezel end of the T opspin 360. Each blower LED is a dual-co lor green/yellow LED. Controller Cards mirror the blower LEDs. Figure 5-2 displays the blower LEDs. Figure 5-2: Blower T ray LEDs Ta b l e 5 - 4 explains blower tray LED indications. T able 5-4: [...]

53 Controller Card LEDs Controller cards include card st atus LEDs as well as Management Ethernet port LEDs. The top-left -most LED represents the controller card in slot 1, and the top-right-most LED represents the controller card in port 14. Figure 5-3 displays the LEDs on the controller card. Figure 5-3: Controller Card LEDs NOTE: The LEDs immed[...]

54 IB Switch Card LEDs Each InfiniBand switch card includes two LEDs (one green and one yellow) at one side of the interface-end of the card. Figure 5-4 displays the IB switch card LEDs. Figure 5-4: InfiniBand Switch Card LEDs Ta b l e 5 - 7 explains IB switch card LED indication s. IB Port LEDs T wo port LEDs (both green) appear next to each IB po[...]

55 Fibre Channel Gateway LEDs The FC gateway provides one yellow (amber) LED and one green LED. Ta b l e 5 - 9 lists and describes the LED indications. FC Port LEDs Fibre Channel Port LEDs are adjacent to each FC port. Th ey indicate port status and activity . Each port has a one bi-color, green/yellow , LED to indicate port status. When you enable[...]

56 T able 5-11 explains the FC port activity LED. Ethernet Gateway LEDs The Ethernet gateway provides one yellow (amber) LED and one green LED. Ta b l e 5 - 1 2 lists and describes the LED indications. Ethernet Port LEDs Ethernet port LEDs appear on the Ethernet ports. Figure 5-7 displays the Ethernet interface LEDs. Figure 5-7: Ethernet Inter face[...]

57 6 S pecifications and Compliance This chapter details the T opspin 360 sp ecifications and compliance certifications. General S pecifications T able 6-1: General S pecifications Spec ification Det ail Operating T emperature Range 0 to 35C Non-Operating T emperature Rang e -40 to 65 degrees Celsius Operating T emperature Gradie nt 20C maximum per[...]

58 Electrical S pecifications Certifications/Regulatory Laser S pecs Non-Operating V ibration, Sinuso idal 3 axis with the following parameters • “X and Y axis: 0.0004 G 2 / H z flat from 5-500Hz (0.472 GR MS) • Z-axis: 0.0008 G 2 / Hz flat from 5-500Hz (0.629GR MS) W eight < 95 pounds, fully config ured Management Inter faces • Console [...]

59 *Standards used: CDRH - 21CFR Chapter I§ 1040.10 ; IEC - IEC60825-1:1994 T able 6-4: Laser S pecifications T opspin P/N Applicat ion Optical Parameters[...]

60[...]

Index A architecture .............................................................. 2 B basic connectivity .................................................27 C configure basic connectivity .................................27 connectivity, basic ................................................27 controller, installing & removing ...............[...]