HP (Hewlett-Packard) zl manual

HP 24-Port 4x Fabric Copper Switch User Guide November 2004 (First Edition) Part Number 377710-001[...]

© Copyright 2004 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Docume ntation, and Technical Data for Commercial Items are license d to the U.S. Government under vendo r?[...]

i Contents Intended Audi ence .............................................................................................................. ....................... ix Typographical C onventions ...................................................................................................... ................. ix Contact Information ...........[...]

ii Change Default User Name and Password ................................................................................... 18 3: Understanding the Management Options ................ 19 About the CLI .................................................................................................................. .......................... 19 Und[...]

iii Setting the System Clock ....................................................................................................... ....................41 Setting Time ................................................................................................................... ............... 42 Synchronize the Clock to an NTP Server .......[...]

iv Change the Response Time out of a SM ........................................................................................ 65 Managing Synchronization Between SMs ........................................................................................... ......66 Enable/Disable Database Synchronization ........................................[...]

v TopspinOS U pgrades ............................................................................................................. ....... 93 About the System Image ......................................................................................................... ................... 93 What is a System Image? ...............................[...]

vi Display Entire Log ............................................................................................................. ......... 113 Show Most Recent Log Entries .................................................................................................. 113 Show Details of a Sp ecific Log ........................................[...]

vii Adding an SNMP Trap Receivers .......... .................................................................................... 147 Editing a Current SNMP Tr ap Receiver ..................................................................................... 148 14: Monitoring Through the CLI . ................... ............. 149 About InfiniBand [...]

viii[...]

ix Preface This document is a guide to the HP 24-Port 4x Fabric Copper Switch. Intended Audience The intended audience is the administrator responsi ble for installing, confi guring, and managing your equipment. This admin istrator should have experi ence administering similar netw orking or storag e equipment. T ypographical Conventions The follow[...]

x NOTE: Indicates an important point or aspect that you need to consider before continuing. Cont act Information T able 2-1: Customer Cont act Information For the name of your nearest authorized HP reseller: In the United States, call 1-800-345-1518. In Canada, call 1-800-263-5868. For HP technical support: In the United States and Canada, call 1-8[...]

1 1 Introducing the InfiniBand System This chapter gives an overview of the following: • “About the HP 24-Port 4x Fabric Copp er Switch User Guide” on page 1 • “Maximize Server Networks” on page 1 • “What is InfiniBand?” on page 2 • “How Does InfiniBand W ork?” on page 2 About the HP 24-Port 4x Fa bric Copper Switch User Gui[...]

2 What is InfiniBand? InfiniBand (IB) is a high speed, h igh density seri al interconnect that incr eases CPU utilization, decreases latency , and eases the management pain of data centers . The term “InfiniBand” refers to the entire hard ware, communication , and management infrastructure. Use of this technology increases the communicati on sp[...]

3 devices. This protocol allows InfiniBan d hosts to natively send SCSI commands as if the storage was direct attached. The SRP protocol is designed to operate using an RDMA com munication service. An RDMA communication service provides communication between pairs of consumers; it uses messages for control information and RDMA operat ions for data [...]

4 The Queue Pair (QP) is one of the primary architectural elements of InfiniBand. In InfiniBand, communication occurs between Queue Pairs, instead of between ports. A Queue Pair (QP) in an addressable entity , and consists of two W ork Queues: 1). Send W ork Queue and a 2). Receive W o rk Queue. (A work queue prov ides a verb consumer with the abil[...]

5 • Local Identifiers (LIDs) • Global Unique Identifier (GUIDs) • Partition Key (P_Keys) • Configur es switch forwardi ng tables. • Receives traps from Subnet Management Agents (SMAs ). • Sweeps the subnet, discovering topolo gy change s and managing changes as nodes are added and deleted. Understanding the Subnet Management Agent s (SM[...]

6 • Routing internally within a sw itch (hops between switch chips) • Routing between whole switch es (hops between nodes). This is also referred to as routing between “switch elements.” Internal switch routing can be configu red to provide the highest perform ance in passing traffic, and to minimize the threat of congestion within the swit[...]

7 Determine the First Path that will be Discovered The SM Routing Algorithm selects th e first best path that it finds. If multiple paths with the same properties are available then the first of these paths found is the one that is selected. Therefore, it is possible to setup the cabling between switch elements to force the algo rithm to prio ritiz[...]

8[...]

9 2 Getting S t arted The information in this chapter focuses on the softwa re and firmware aspect s of the initial set-up, and assumes that you have additional documen tation for the hardware. This chapter provides the following informati on: • Overview of entire system installation on page 9 , wi th references to more detailed info rmation. •[...]

10 Inst all HCAs and Drivers in Host s Refer to the HP Dual-port 4x Fabric Ad apter Quick Setup Installation card and the HP Dual-port 4x Fabric Adapter User Guide . Inst all and Power on the InfiniBand Chassis Refer to the HP 24-Port 4x Fabric Copper Switch Hardwar e Quick Setup Installation card and the HP 24-Port 4x Fabric Copper Switch Hardwar [...]

11 6. Set the IP address and netmask. Th e following addre ss is an example. 7. Set the default gateway address. This address is an example. 8. Enable the management port 9. T est IP connectivit y by pinging the managemen t station. 10. Y ou must save th e configuration persisten tly by using the copy command. Y ou are now ready to power do wn the [...]

12 T o login vi a SSH: 1. Use an SSH client (e.g. P utty) to port 22. Configuring an InfiniBand Management IP Address T o configure an In -band InfiniBand management IP address: 1. At the Login: prompt, enter the username and password. The default is super and super . 2. At the CLI prompt, enter enable . This enters the privi leged-ex ecute mode, a[...]

13 T o assign a hostn ame name to the management port: 1. Start a CLI session. 2. Enter the privileged-user mode. 3. Enter the global-configuration mode. 4. Enter the hostname command with the name you wish to assign. The hostname command assigns a convenient name to the system that shows up at the CLI prompt. NOTE: This command also changes the CL[...]

14 V erify Switch to Host Communication T o verify the InfiniBand device can reach a host on the network, enter either the user -execute or privileged-execute mode on the InfiniBand device, th en enter the ping command. This is an In-band procedure only . Inst all the GUI (Element Manager) HP 24-Port 4x Fabric Co pper Switches can be ma na ged visu[...]

15 T o ru n the CLI remotely: 1. Open a terminal or terminal emulator window . For example: from the comma nd line, enter the telnet command with the IP addre ss, or network name, of the Management-Ethernet port. The CLI login prompt ( Login: ) is displayed. 2. Enter a CLI user name. The CLI password prompt ( Password: ) is displayed. 3. Enter the [...]

16 Create a Partition ID (P_Key) A default partition is configured autom atically . The me mbers of a default partit ion include all connected ports, and provide full membership. Ho wever , to creat e separation between traffic, you must configure specific partitions. 1. Launch Element Manager , if you have not already done so. 2. Select Infini Ban[...]

17 S pecify Partition Members and the Membership T ype b. Click on the new Partition in the left window . The available partition members ap pear in the right-side window . Note that the “A vailable Members” refers only to members that are known to the Subnet Manager . This includes HCAs and switches that are alread y plugged into the fabric as[...]

18 • Fibre Channel gateways. f. If you have multiple InfiniBan d switches in your fabric: • Exchange the partition configurati on between switches by enablin g database synchronization, if you have not already don e so. Refer to “En able/Disable Database Synchronization” on page 84 . Set User Levels and Passwords Change Default User Name an[...]

19 3 Underst anding the Management Options This chapter gives an overview of the following system Management options: The CLI • “About the CLI” on page 19 • “Using the CLI” on page 21 The Java GUI • “About Element Manager” on page 24 • “Using Element Manag er” on page 27 The W eb GUI • Refer to the HP 24-Port Fabric Copp e[...]

20 • Multiple Comm and Modes Example Underst anding the Command Modes The CLI has four command modes • user-execute mode (read-only) • privileged-execute mode • global-configuration mode • sub-command mode The commands you can enter depend upon the current command mode and who you log in as. Y ou may enter a question mark (?) at the CLI p[...]

21 Sub-Command Mode The final mode is sub-command mod e. Anything to do with InfiniBand, Ethernet, and Fibre Channel interface cards, ports, and gateways is done in this mode, including the Management-Ethernet ports. This mode is used to assign IP addresses to interface gateway ports, set connection speeds, set connection types, etc. Using the CLI [...]

22 For example: In the preceding example, srp- is short for srp-global , gate is short for gateway-portmask- pol , and res is short for restricted . Note: Command completio n only works for commands; it is not effective for keywords. Displaying Command History The T opspin sy stem “remembers” the last 40 comm ands you enter . Display the comman[...]

23 T o set the number of lin es displayed on the terminal screen to 67 lines at a time and raise the tim e-out limit to 60 minutes 1. Enter the user -execute or privileged-execute mode. 2. Enter the terminal command with the length parameter and the number of lines to display p er page. 3. Enter the terminal command with the time-out parameter and [...]

24 Quick Help Y ou can ente r the question mark (?) at the CLI prom pt to display on e of three types of user information. 1. Enter a question mark ( ? ) at the CLI prompt at any tim e to display the commands you can execute. Only the commands appropriate to the cu rrent mode and user login are displayed. 2. Y ou may also enter part of a comm and s[...]

25 The Chassis Window Once you log into the Element Manager, the chassis window is displayed. This is the primary window in which you work. It graphically depicts the curren t configuration of the attached InfiniBand system chassis. Sample HP 24-Port 4x Fabric Copper Switch Chassis V iew The T ool Bar The tool bar contains a set of icons for common[...]

26 About Selecting Items Interaction with the T opspin system is performed u sing a combination of pull-down menus, icons in the tool bar , and pop-up menus. The windows these open contain a combination of text fields, radio buttons, and toggles with which to configu r e selected cards an d/or ports. Depending upon the function you wish to execute,[...]

27 For example, if you <Control > left-click multiple Ethernet port s and then attempted to select a Fibre Channel port: The Ethernet ports are de-sel ected and the Fibre Channel por t is the only thing selected. Using Element Manager Inst alling the Element Manager Program For informatio n regarding upgr ading the Element Manager, refer to ?[...]

28 Linux, the default is / home /TopspinEM , where home is the home directory of the person installing the software. If the folder does not exist, you will be prompted to create it. 7. Click the Next button. On W indows, the Choose Shortcut Folder window op ens. Figure 3-1: Element Ma nager Installation, Ch oose Shortcut Folder Window On Linux, the[...]

29 The Pre-Installation Summary wi ndow is opened. This window lists the installati on choices you have made thus far . 10. If you are satisfied with your installation choi ces, click Install. The Installing Element Manager window op ens to indicate installation status. If you are not satisfied with the configuration, cl ick Previous to return to t[...]

30 The Element Mana ger is now ready to configure the InfiniBand network, as well as Ethernet or Fibre Channel expansion module(s). The Element Manager is dynamically up dated to sh ow changes to the configuration. As cards and ports are configured, the corresponding run lights and port frames reflect the changes by turning green. Depending upon yo[...]

31 receiver . Y ou can have a maximum of six trap receive rs. Entries in this list can be configured from the Health --> Trap R eceivers menu. • Community S trings - Y ou can set communi ty strings as a simple method of authentication between the system and the remote Network Manage r . One unique community string is associated with each usern[...]

32[...]

33 4 Performing Admin T asks Through the GUI This chapter describes the following admini strative procedures that can be performed through the Element Manager GUI: • “Configuring the IB Interface Speed” on page 33 • “Setting the System Clo ck” on page 36 • “Rebooting the System” on page 37 Configuring the IB Interface S peed Expli[...]

34 The IB Port window appears. 4. Uncheck the Enable Auto-Negotiate box. Leave auto-negotiate check ed if you want the speed of the transmit port and the receiving port to automatically negotiate the highest possible speed. 5. Select the 2500M or 10G speed. The 30G speed is not yet supp orted. 6. Click the Apply button. Set IB Interface S peed to A[...]

35 The IB Port window appears. 3. Check the Enable Auto-Negotiate box. 4. Click the Apply button. V i ew the IB Interface S peed V iew the current speed of the InfiniBand interfaces: 1. Launch Element Manager , if it is not already open. A graphic representation of your Infin iBand switch appears. 2. Double-click an InfiniBand port. The IB Port win[...]

36 4. Click the Apply button. Setting the System Clock Maintaining accurate time is import ant for statistics and auditing. The InfiniBand chassis provides an on-board system clock to save the time settings acro ss reboots. Ti me is maintained in one of two ways: • An on-board system clock • External NTP servers T ime is set at the factory , an[...]

37 4. Click Apply . 5. Click Close . Rebooting the System Reboot a System with a Single Controller Card 1. Launch the Element Manager . 2. Choose the reboot opt ion: a. Right-click in any unused part of the Elem ent Manager disp lay , includin g unused interface slots. A pop-up window opens that al lows you to select Reboot . b. or select Maintenan[...]

38[...]

39 5 Performing Admin T asks Through the CLI This chapter describes the following administ rative procedures that can be perform ed through the CLI. • “Setting the IB Interface Speed” on page 39 • “Notifying Users” on p age 40 • “Setting the System Clo ck” on page 41 • “Rebooting the System” on page 43 Setting the IB Interfa[...]

40 3. Enter the following command: config interface ib [{ IB switch car d / IB switch port | all } speed 1x | 4x ] Example Set IB Interface S peed to Auto-Negotiate Set the InfiniBand interface speed to auto-negotiate if you want the sp eed of the transmit port and the receiving port to automatically negotiate the highest possible speed. IB speed c[...]

41 Broadcasting Messages to all Users Message broadcasting i s an important feature to fore warn all CLI users that some major event is about to take place, such as bringing down a network for ad ministration. A broadcast message is sent to every active CLI session on the InfiniBand system chassis. Enclose multi-word messages wi thin double-quotes.[...]

42 T ime is set at the factory , and can be manually set. T o ensure accurate synchron ization, it is recommended that you use an extern al NTP server . This enab les logs to be synchronized with other management sy stems. Setting T ime Note: If you have an NTP server configured, it is recomm ended that you use the process described in: “Synchron[...]

43 Rebooting the System Reboot a System with a Single Controller Enter the CLI rel oa d command in p rivileged EXEC mode. The system prompts for yo u to verify the reload. If you had not already saved configuration changes, and the system detects the changes, it prompts you to save. If you enter yes, the new configuration is stored in startup-confi[...]

44[...]

45 6 Setting Access and Security This chapter describes the follow ing Access and Security features: • “Understanding Access and Accounts” on page 45 • “Managing Access and Accounts” on page 47 • “About Partitions” o n page 53 • “Create Partitions (CLI)” on page 56 or • “Create Partitions (GU I)” on page 57 • “Abou[...]

46 In order for users to initiate an administration session, the User has to supply login cred entials. The credentials supplied depend upon the interface being us ed. Element s of the Access System The T opspin access system associat es the following key elements: • Username - Creates a unique username in the system. • Password • Community S[...]

47 Understanding Permission Level s The following table displays the dif ferent access-levels. Managing Access and Account s Setting or Changing a Password 1. Log in to the CLI as a super user . Use the defau lt username and password if they have not already been changed (refer to page 14 ). 2. Enter the privileged-user mode. 3. Enter the global-co[...]

48 7. Use the show user command to verify changes. Displaying User Information T o disp lay the information of configured users: 1. Log in to the CLI as a super user . Only a user with unrestricted pr ivileges may view user info rmation. 2. Enter the show user all command to display curr ent user information. Topspin-360# sho w user ===============[...]

49 Adding New Users T o add a new user account for bot h C LI and Element Manager access: 1. Log in as the unrestricted user . Only a user with unrestricted pe rmissions may add new user accounts. 2. Enter the privileged-user mode. 3. Enter the global-configuration mode. 4. Create the user account and user password. The user name and password are a[...]

50 9. V iew the new user account information. Deleting a User Account 1. Log in as the unrestricted user . Only a user with unrestricted permissi ons may create and modify user accounts. 2. Enter the privileged-user mode. 3. Enter the global-configuration mode. 4. Enter the username command with the name of the user , and the no keyword. For exampl[...]

51 For rapid access, the Element Manager saves th e IP address and comm unity string of recent administrators. These address/commun ity string pairs are displayed in a scroll-list the nex t time you want to open an Element Manager session. Note that whoever logs on aft er you can reuse this connection information and, if you connected with superuse[...]

52 3. Enter the global-configuration mode. 4. Enter the username command with the name of the user , the community-string keyword, and the new comm unity string to assig n this user . In the example above, ib-admin is the name of the user , community-string is a mandatory keyword. ib-commStr is the SNMP community string the user will have to use to[...]

53 In the example above, ib-admin is the name of the user account, privilege is a mandator y keyword, and ib-rw , ip-ethernet-ro , and fc-rw are access privileges. NOTE: When you change a user ’ s privileges, all the old privileges are removed and replaced with the new privilege(s). If the user h ad multiple privilege s, include the other privile[...]

54 InfiniBand pa rtitions ar e comparable to hardware-enforced s ecurity features of conventional I/O networking technologi es, such as Ethe rnet VLANs and Fibre-Channel zones. Partition Members W itho ut members, a Partition doesn’t have meaning to the system. Ports are added to the Partition, and become members of that Partition. Each port may [...]

55 Hex to Binary Conversions The following table is provided to assist in the creation of P_keys. When creating the Partition p_key , enter a hexidecimal value that is the equivalent of 16 -bits in binary . For example, enter 80:00 (hex) to be 10000000000000 000 (binary). The default Partition (which cannot be altered) is 7f:ff. T able 6-5: Binary [...]

56 Examples of V alid P_Key V alues Y ou can choose your own p_key values, or you can s imply choose your values from th e list in the following ta ble. Underst anding how P_Keys are Saved Partition inform ation is saved pe rsistently by the master subnet manager . P_key inform ation can be synchronized between the master subnet manager an d a slav[...]

57 Create a Partition ID (P_Key) Default partitions are configured automa tically . The members of a defau lt partit ion include all connected ports, and provide full m embership. Howeve r , to create separation bet ween traffic, you must configure specific partitions. Create a partition using the following steps: a. Enter the following items at th[...]

58 3. Click open the Subnet Manager folders in the left window . The Partitions folder appears. 4. Click on the Partitions folder in the left window . The Partition s Summary window appears. 5. Click the Add button. The Add Partition dialog box appears. Enter a Partition key (P_Key) to identify the new partition. For information regarding selecting[...]

59 The available partition members ap pear in the right-side window . Note that the “A vailable Members” refers only to members that are known to the Subnet Manager . This includes HCAs and Switches that are alread y plugged into the fabric as well as manually configured entries. If you know the GUID and port count of an HCA that has not yet be[...]

60[...]

61 7 Using the Subnet Manager Through the GUI This chapter provides the following informati on: • “The Subnet Manager (SM)” on page 61 • “V iewing the Subnet Manager C onfigurations” on page 62 • “Changing the Subnet Manager Configurations” on page 64 • “Managing Synchronization Between SMs” on page 66 • “Adding a Subnet[...]

62 S t andby Subnet Manager A standby subnet manager is a subnet man ager (SM) that is curren tly quiescent, and not in the role of a master SM. Standby SMs are dormant managers, and can take over in case of failure of the master subnet manag er . V iewing the Subnet Manager Configurations V i ew a Summary of Subnet Management 1. Launch the Element[...]

63 Information specific to that subnet ma nager appears. • V iew the subnet-prefix of the subnet manag er . • V iew the Global Unique Identifier (GUID) of the subnet manager . •V i e w t h e St a t u s of the subnet manager . This is the operational status, as determined by self-detection. The values are notActive, discovering, or Master . As[...]

64 Changing the Subnet Manager Configurations Change the Priority of a SM The priority number of a subnet manager tells the subnet manag er how to interact with other subnet managers; the hi ghest priority subnet manager becomes the master . The integer must be between 0 and 15, with the default being 0. 1. Select Infini Band > Subnet Management[...]

65 2. Highlight the subnet manager that you want to config ure from the left-navigation bar . 3. Click into the Sweep Interval field. 4. Enter a value that indicates the number of seconds between sweeps of the network. The default is 10 seconds. The Apply and Reset buttons become active if a change is made. 5. Click the Apply button to save the cha[...]

66 2. Highlight the subnet manager that you want to config ure from the left-navigation bar . 3. Click into the Response Timeout field. 4. Enter a value that indicates the number of mi croseconds allowed between the port reception of a subnet management p acket and the transmission of the associated response. The Apply and Reset buttons become acti[...]

67 3. Click the Database Sync tab. 4. Click the Enable box to enable database synchronizatio n between the active and backup subnet managers. Set Configurations for the Master SM 1. Select Infini Band > Subnet Management . The Subnet Management window appears. 2. Highlight the subnet manager that you want to config ure from the left-navigation b[...]

68 The Subnet Management General tab appears. 3. Click into the Master Poll Interval field to change the interval (in seconds) at which the m aster SM polls an active slave SM to verify synchronization. 4. Click into the Master Poll Retries field to specify the number of unanswered po lls that cause the slave to identify the master as dead. 5. Clic[...]

69 3. Click the Database Sync tab. 4. Click into the Max Backup SMs field to enter the maximum number of backup subnet managers with which the master subnet manager will synchr onize. A b ackup subnet manager is aut omatically added whenever a new InfiniBand (IB) sw itch is connected to the IB fabric. The default is 1. 5. Click into the Session Tim[...]

70 12. Click the Apply button to save the changes. Adding a Subnet Manager Adding additional sub net managers (in addition to the one that is provided by default on the InfiniBand system) should only be done by experien ced users. In the event th at additional sw itch is added to an InfiniBand fabric, an ad ditional subnet manager is added by defau[...]

71 6. Click the Add button. V iewing Partitions For complete pa rtition in formation, ref er to “About Partitions” on page 53 . The partitions that are currently being managed by the subnet manager can be vi ewed by performing the following steps. 1. Launch the Element Manager, if you have not already done so. 2. Select Infini Band > Subnet [...]

72 The A vailable Members window appears. About InfiniBand Multicast Group s An InfiniBand Multicast Group is a collection of Host Channel Adapter (HCA) ports that receive multicast packets sent to a single address. The configuration and members of a multicast group can be viewed through the Element Ma nager , but cannot be modified th rough these [...]

73 The Multicast Groups Su mmary page appears. 5. V iew the Multicast Global ID ( MGID ), which is the 64-bit multicast GID address for the multicast group. 6. V iew the Queue Key ( Q_Key ), which is the 16-bit Q-Key of this multicast group. The queue key is a construct that is used to validat e a remote sender’ s right to access. 7. V iew the Ma[...]

74 5. Click on a specific multicast gro up from the left-n avigation tree. 6. V iew the General tab, which i s displa yed by de fault. • V iew the Q_Key for this multicast group. The Queue Key ( QKey ) is a 16-bit construct that is used to validate a remote sender ’ s right to access. • V iew the Local Identifier ( MLID ) for this multicast g[...]

75 8. Click the Members tab in the Multicast Group wind ow . 9. V iew the Members tab: • V iew the Port Global Identifier ( Port GID ) of the multicast group member . •V i e w t h e Join S tate of the multicast group member . The join state m ay be one or more of the following values: Full Memb er , No n-Member, and Send Only Member . •V i e [...]

76 4. Click on Services from the left-navigation tree. The Services window appears. 5. V iew a Summary of the selected subnet managers services: •V i e w t h e Name of the Service. • V iew the 64-bit ID of the service. • V iew the 128-bit Global ID of the service. • V iew the partition keys af filiated with the service in the PKey field. Re[...]

77 6. V iew the Details of the subnet managed services: •V i e w t h e Service Name of the subnet m anaged service. • V iew the 64-bit Service ID of th e subnet managed service in the Service ID field. • V iew the 128-bit Service Global ID of the subnet managed service in the Service GID field. • V iew the partitions associated wi th this s[...]

78 Configure the LID Mask Control (LMC) The Subnet Manager (SM) allows an administrator to define the LMC (Local Identifier Mask Control) value per subnet. Once the LMC value has been assi gned, the SM routes dif ferent paths for each LID (an address assigned to a port) that is associated with the same host port. 1. Launch Element Manager , if you [...]

79 The Routes window appears. 6. Enter the Source LID and the Destination LID into the fields in the Filter Route section. 7. Click the Show Rout e button.[...]

80[...]

81 8 Using the Subnet Manager Through the CLI This chapter provides the following informati on: • “The Subnet Manager (SM)” on page 81 • “V iewing the Subnet Manager C onfigurations” on page 82 • “Changing the Subnet Manager Configurations” on page 83 • “Managing Synchronization Between SMs” on page 84 • “Adding a Subnet[...]

82 St andby Subnet Manager • A subnet manager (SM) that is currently quiescent, and not in the role of a master SM. Standby SMs are dormant managers, and can take over in cas e of failure of the master subnet manager . V iewing the Subnet Manager Configurations V i ew a Summary of Subnet Management 1. Enter the show ib sm configurat ion subnet-pr[...]

83 or Example • V iew the subnet-prefix of the subnet manager . • V iew the Global Unique Identifier (guid) of th e subnet manager . • V iew the priority for the subnet manage r . The priori ty numb er of a subnet manager tells the subnet manager how to interact wi th other subnet manage rs; th e highest priority subnet m anager becomes the m[...]

84 can run on the system and other SMs may run in your IB network, the priority attribute iden tifies the master SM. The integer must be between 0 and 15, with the default being 0. 1. Enter the ib sm subnet-pr efix pr efix priority sm-priority command. Example Change the Sweep Interval of a SM The sweep interval indicate s the rate (in seconds) at [...]

85 Set Configurations for the Master SM 1. Enter the config ib sm subnet-p refix < pr efix > master-poll-interval < 1..60 > command to change the interval (in seconds) at which the m aster SM polls an active slave SM to verify sy nchronization. The default is 3 seconds. Example 2. Enter the config ib sm subnet-p refix < pr efix > [...]

86 3. Enter the ib sm db-sync subnet-prefix < pr efi x > poll-interval < 1..30 > command to change the interval at which the master subnet mana ger will send a synchr onization sess ion status request packet to an active session. The default is 3 seconds and th e possible entries are 1...30. Example 4. Enter the ib sm db-sync su bnet-pr[...]

87 The default is 3600 and the possible entries are 1...86400. Example Adding a Subnet Manager Adding additional sub net managers (in addition to the one that is provided by default on the InfiniBand system) should only be done by experien ced users. In the event that an addit ional switch is added to an InfiniBand fabric, an additional subnet mana[...]

88 V iewing Multicast Group s V i ew a Multicast Group Summary 1. Enter the show ib sm multicas t summary command. Example • V iew the multicast group subnet-prefix. • V iew the Multicast Glob al ID ( MGID ), which is the 64-bit multicast GID address for the mult icast group. • V iew information regardin g the multicast group mem bers: • V [...]

89 V i ew Multicast Group Det ails 1. Enter the show ib sm multicast { subnet-prefix pr efix | all } command. Example • V iew the subnet-prefix of the multicast gro up. • V iew the Multicast Glob al ID ( MGID ), which is the 64-bit multicast GID address for the multicast group. • V iew the Queue Key ( QKey ), which is the 16-bit Q-Key of this[...]

90 • V iew the Service Level ( SL ) of the multi cast group. The Service Level value is located in the Local Route Header of a packet. It iden tifies the appropriate virtual lane for a packet, which enables the ability to have multiple servi ces on one physical lane. • V iew information regardin g the multicast group mem bers: • V iew the Por[...]

91 Configure the LID Mask Control (LMC) The Subnet Manager (SM) allows an administrator to define the LMC (Local Identifier Mask Control) value per subnet. Once the LMC value has been assi gned, the SM routes dif ferent paths for each LID (an address assigned to a port) that is associated with the same host port. The default LMC is 0. V i ew Infini[...]

92[...]

93 9 Using Image Files This chapter describes the follow ing Access and Security features: • “T ypes of Image Upgrades” on page 93 • “Upgrade Procedure Overview” on page 95 • Upgrade Procedure Steps (begins page 95 ) T ypes of Image Upgrades There are two different types of upgrades that may need to be downloaded to the T opspin syste[...]

94 When an image file is installed, th e image file is expanded i nto a “system image.” The system image is what the user will refer to in order to specify wh at the system should use to boot-up each card in the system. What is an Image File? Image files are stored in flash memory as a single complete file with a “ .img ” extension. Each im[...]

95 Card S t atus Requirement s Only cards with an oper-status of “up” ar e updated. If a card is down when you run install , or a card is added after running insta ll : 1. Bring up the card 2. Run install again. Specify the same image file. If the image is alr eady installed on a card, that card is skipped. 3. Be sure to specify the boot-config[...]

96 V erify the Inst alled Image V ersion Check the Image V ersion Through the GUI 1. In the Element Manager , select Maintenance -> Fi le Management… . The File Management W indow appears. 2. Note the version number of the installed image by lo oking in the File Name column. The installed system image has a .img extension. 3. Image files that [...]

97 • “Copy/Download an Image Th rough the CLI” on page 98 Copy/Download the Image Through the GUI Images must be installed using the CLI; however , images can be copi ed from a remote or local location through the GUI. Copy an Image from a Remote Location This section describes using the Elemen t Manager to copy an image file from a remote lo[...]

98 Copy an Image from a Local Location This section describes using the Element Manager to copy an image file fr om a locally available location before installing it as the active image. 1. In the Element Manager , select Maintenance -> Fil e Management … . Note the uninstalled image files(.img) and installed system -images. Do not modify or d[...]

99 Directory management is automatically perform ed on the InfiniBand system, so do not include path information for files on the chassis. 1. (Optional) If you are using In-B and Management, configure the et hernet management interface (if you have not already done so). 2. V erify that you have a working ftp server connection, and that you have a u[...]

100 Directory management is automatically perform ed on the InfiniBand system, so do not include path information for files on the chassis. 1. (Optional) If you are using In -Band Management, configure the et hernet management interface (if you have not already done so). 2. In the privileg ed-execute mode, enter the copy command with the remo te sy[...]

101 The name of the im age file will change after it is installed. 2. Repeat the install procedure on an y cards that did not have an oper-status of “up” at the time of image installation. V iew a card’ s status by using the show card command. S pecify a New Boot Image After a system-image is installed on the system chassis, specify the syste[...]

102 S pecify a New Boot Image Through the CLI 1. Find the name of your new system image by listing all the image files on the T opspin chassis. Enter the dir image command. 2. Copy the displayed system-image (this wil l be pasted into the boot-config command). 3. Specify the new boot image. Enter boot-config primary-image-source , then paste the th[...]

103 Deleting Image Files Up to two image files may be sav ed on the system. Older versions of image files will have to be removed before you can add more. • “Deleting Images Through the GUI” on page 103 • “Deleting Images Through the CLI” on page 103 Deleting Images Through the GUI For information regarding deletin g images through the [...]

104[...]

105 10 Using Configuration Files This chapter describes: • “Understanding Confi guration Files” on p age 105 • “Listing Configuration Files” on page 106 • “Export a Configuration File” on page 106 • “Import a Configuration File” on page 108 Underst anding Configuration Files A configuration file is a text file that stores a [...]

106 Listing Configuration Files List Config Files Through the CLI T o li st the configuration files currently stored on the InfiniBand system, enter the dir command with the config keyword. For example: List Config Files Through the GUI T o show a list of current configurat ion files through the Element Manager: 1. Launch the Element Manager, if yo[...]

107 Export a Config File Through the CLI T o copy a configurat ion file to a remote FTP server through the CLI: 1. Enter the copy command, as well as the necessa ry ftp information in one string: • ftp file system • name of the config file that you want to copy . • ftp keyword • your ftp username and password • FTP server domain name or I[...]

108 6. (Optional) If you are copying th e config file to an ftp server: a. V erify that you have a connectio n and privileges to an FTP serv er . b. Click the Remote FTP Server radio button i n the Copy T o section. c. Enter the name or IP addre ss of the FTP server in the Server Name or IP Addr ess field. d. Enter your User Name for the FTP server[...]

109 • @ FTP server domain name or IP address. • directory path on the host from which you want to copy the config fil e. • name of the config file that you want to copy . • directory path on the host to which you want to co py the config file. • name of the file where you want to store the copied config file. Download a Config File From a[...]

110[...]

111 11 Using Log Files This chapter describes: • “Understanding Log Fil es” on page 11 1 • “Listing Current Log File Names” on page 112 • “V iewing a Log File Through th e CLI” on page 1 13 • “V iewing a Log File Through th e GUI” on page 1 14 • “Configuring Remote Logging” on page 1 1 7 Underst anding Log Files Log fi[...]

112 • CONF - configuration changes. No user action is required • INFO - general information. No user action is required • WA R N - abnormal condition. User interventi on may be required • ERROR - abnormal condition. User intervention is required • FATA L - abnormal condition. User must reboot • TRACE - Refer to “About T racing” on p[...]

113 V iewing a Log File Through the CLI Display Entire Log 1. Set terminal pa rameters, if you have n ot already done so. Refer to “Setting T erminal Parameters” on page 22 . 2. Enter the show logging command. Show Most Recent Log Entries T o disp lay the tail-end of the active log-fi le and display new log entries as they occur: 1. Set termina[...]

114 Show Det ails of a S pecific Log The more command displays the contents of a specifi ed log file, includin g the active log-file. 1. Enter mor e syslog: file name V iewing a Log File Through the GUI 1. Launch Element Manager , if you have not already done so. 2. Select Health --> Log Vi ewer . Topspin-360# more syslog: ts_log Nov 19 22:08:49[...]

115 The Log V iewer window appears. Filtering Logs 1. Launch Element Manager , if you have not already done so. 2. Select Health --> Log Vi ewer . The Log V iewer window appears. 3. Click the filter button at the top of the window . The Log Filter window appears. 4. Determine the type of filter that you want to use: • “Filter Logs by Time”[...]

116 • “Filter Logs by Category” on page 1 1 7 • “Filter Logs by T ext String” on page 1 17 Filter Logs by Time 1. Follow the steps in “Filtering Log s” on pa ge 1 15 . 2. Click the From date radio butto n. 3. Select the Month and Day from the drop-down menus. 4. Enter the Hour , Minute, and second in the fields that follow the date.[...]

117 Filter Logs by Category 1. Follow the steps in “Filtering Log s” on page 115 . 2. Click on the technology for whi ch you want to capture logs from the Category column. 3. Click the Apply button. Filter Logs by T ext S tring 1. Follow the steps in “Filtering Log s” on pa ge 1 15 . 2. Enter the text string that you want to use as your log[...]

118[...]

119 12 V iewing the IB Network Through the GUI This chapter provides the following informati on: • “About the Device Manager (DM)” on page 119 • “Display the Device Manager” on page 1 19 • “About the T opology V iew” on page 121 • “Display the Infin iBand T opology” on p age 122 • “V iew the Internal Chassis T opology”[...]

120 The Device Manager window opens. 3. V iew the cumulative number of changes to the Controller list since th e device last booted in the Change ID field. 4. V iew the maximum number of control lers that your device can support in the Max Controllers field. 5. V iew the Diag Device ID field. The number 1 is displayed if diagnostics can provide IOC[...]

121 14. V iew the maximum depth of th e pe r- channel RDMA Read Queue in the RDMA Read Queue Depth field. 15. V iew the maximum size, in bytes, of send messages in the Send Message Size field. 16. V iew the maximum size, in bytes, of the outbound RDMA transfers that the IOC initiates in the RDMA T ransfer Size field. 17. V iew the integer value (fr[...]

122 Display the InfiniBand T opology V i ew the T opology 1. Launch Element Manager , if you have not already done so. 2. Select Infini Band --> T opology . The Specify Devices dialog box opens, if this is y our first time viewin g the T opology . T o add a device, refer to “Ad d an Attached Device to the T o pology V iew” on page 126 . 3. C[...]

123 V i ew the Name of an HCA T o easily view the name of a Host Channel Adapter , use the T opology view . 1. Open the T opo logy view , if you have not already done so. Refer to “V iew the T opology” on page 122 . 2. V iew the name of t he HCA that i s displayed with the HCA icon s. HCAs[...]

124 V i ew the GUID of an HCA T o easily associate a Host Channel Adapter with the Global Unique Identifier (GUID), use the T opology view . 1. Open the T opo logy view , if you have not already done so. Refer to “V iew the T opology” on page 122 . 2. V iew the GUID inform ation that is di splayed with th e HCA icons. Determine Which HCA Port i[...]

125 • The slot/port number of the IB switch is labelled close to the IB switch icon. Port 1 of the the HCA is connected to Port 16/4 of the IB sw itch[...]

126 V i ew the GUID of an IB Switch T o easily associate an InfiniBand (IB) switch with the Global U nique Identifier (GUID), use the T opology view . 1. Open the T opo logy view , if you have not already done so. Refer to “V iew the T opology” on page 122 . 2. V iew the switch GUID inform ation that is displayed with the In finiBand switch ico[...]

127 The Specify Devices dialog box appears. 3. Click the Add button to add another connected In finiBand device to the T opology view . The Add dialog box appears. 4. Enter the DNS name or t he Management-Ethernet por t IP address of the InfiniBand device in the Device Name or IP Address field. 5. Enter the SNMP Community strin g for the InfiniBand[...]

128 The external T opolo gy view appears. 3. Double-click the InfiniBand switch icon. The icon will appear different, depending on the ty pe of IB switch you are using in your network. TS360 Icon[...]

129 The Internal InfiniBand T o pology appears. 4. V iew the connections between InfiniBand nodes and Ethern et or Fibre Channel Gateways. 5. V iew the descriptions and node GUIDs of the InfiniBand nodes and any gateways. V iew Subnet Manager Det ails In addition to initializing and maintaining the Infin iBand fabric, the subnet manager also commun[...]

130 4. The InfiniBand Subnet Details window appears. The Node tab is selected by default. 5. V iew the Subnet Prefix of the node. The Subnet Pref ix field identifies th e InfiniBand subnet to which the node belongs. 6. V iew the Global Unique Identifier ( GUID ) of the node. 7. V iew a text string that describes the node is the Description field. 8[...]

131 An additional level of node info rmation is added to the windo w . 3. V iew the Device vendor ID in the DeviceID field. The value will be the same for all ports on the node. 4. V iew the supported base managem ent datagram (MAD) version in the BaseV ersion field. This field indicates that the channel adapter , switch, or router supports up to a[...]

132 4. The InfiniBand Subnet Details window appears. The Node tab is selected by default. 5. Click the Ports tab. 6. V iew the Subnet Prefix of the nod e to which the port belongs. The SubnetPrefix field identifies the InfiniBand subnet to which th e node belongs. 7. V iew the Global Unique Identifier ( GUID ) of the node to which the port belongs.[...]

133 10. Determine whether or not the no des can actually communicate, and view the state transition that has occurred in the St a te field. A T ransition is a port change from down to initiali ze, from initialize to down, from armed to down, or from active to down as a re su lt of link stat e machine logic. 1 1. V iew the Active l ink width in the [...]

134 8. V iew the initial value of the lease-period timer in seconds in the MKeyLeasePeriod field. Refer to section 14.2.4, Mana gement Key , InfiniBand® Ar chitectur e , V ol. 1, Release 1.0, for more information. 9. V iew the enabled link width in the LinkWidthEnabled field. The value is an integer that indicates the enabled link-width set s for [...]

135 18. V iew the active maximum transmission uni t enabled on this port for transmit in the NeighborMTU field. Check the MTUCap value at both ends of every link and use the lesser speed. 19. V iew the administrativ e service level required fo r this port t o send a non-subnet management packet (SMP) message to the subnet manager in the MasterSmSL [...]

136 used by requestors to determine the interval to wa it for a response to a request before taki ng other action. Duration is calculated as (4.096 ms * 2^SubnetTimeout ). 36. V iew the maximum time allowed between the port reception of a subnet management packet and the transmission of the associated response in the RespT ime field. 37. V iew the [...]

137 13 Monitoring and Reporting Through the GUI This chapter gives an overview of the following: • “About Analyzing Network Data” on page 137 • “About T a bular Formats” on page 138 • “About Graph Formats” on page 138 • “Creating a Data Analysis T able” on page 140 • “Creating a Data Analysis Graph” on page 143 • “[...]

138 Dat a Captured The data displayed depends upon the selected cards and ports. General interface activity , such as the number of octets, packets, and transmission errors, is provided for all port types. About T abular Format s Network data may be viewed in a tabular or graph format. The tabular format is ideal for numbers cru nching when you wan[...]

139 •B a r •A r e a Figure 13-2: In-Octet s S tatistics for a Single Infin iBand Port The appropriate graph icons become enabled when a set of graph data is selected. Y ou can click a graph type and modifier to indicate how to di splay that data. T able 13-1: Graph T ypes and Descriptions Graph T ype Associated Icon Purpose Pie This graph repre[...]

140 V isual Modifiers Modifiers may be used singul arly , or in conjunction w ith other mo difiers, to change the orientation and scale of the graph displayed. W ith the use of display modifiers, the data displayed can be: • Horizontal/V ertical • Stacked/s ide-by-side • Standard/Logarithmic Creating a Dat a Analysis T able Create a Dat a T a[...]

141 4. If multiple ports (or cards) are se lected, then select the type of counter data you want to display from the counter scroll-list on the botto m of the graph window . 5. Select the statistical data to be monitored. a. Hold down the <Cntrl> key and click multiple fiel ds to select specific counters or b. Left-click and drag the mouse to[...]

142 6. Click the Export table icon. 7. Navigate to the location where you want to store th e data table. 8. Enter a file name for the data table. 9. Save the file. Print a Dat a T able Once you have created a data table, yo u can print the table for reference. 1. Create a data table. Refer to “Create a Data T able” on page 140 .[...]

143 2. Click the print icon from th e data table. Creating a Dat a Analysis Graph Statistical data is displayed in a table by default. Ho wever, you can use the data in the tabular format to create a variety of graphs. 1. Create a table of network data. Refer to “Creating a Data Analysis T able” on page 140[...]

144 Once you have selected the data to be monitored, the Graph icon s become active. 2. Select the icon that represents th e type of graph you want to create. Refer to “T ypes of Graphs” on page 138 The graph appears in a new window .[...]

145 Modify a Graph All graphs (excep t for pie graphs) ha ve the o ption of using vi sual modi fiers to alter th em. For example, a graph can be displaye d horizontally or ver tically . 1. Click the icon that represents the visual mod ifier that you want to apply to the graph. Refer to “V isual Modifiers” on page 140 . In the following example,[...]

146 Print a Graph Once you have created a data graph, you can print the graph for reference. 1. Create a data table. Refer to “Create a Data T able” on page 140 . 2. Click the print icon above th e graph. About SNMP T rap s Event s Sent to T rap Receivers Event messages are generated whenever a significan t event occurs. Significan t events sen[...]

147 • InfiniBand Multicast group is de leted • InfiniBand Partition is added • InfiniBand Partition is deleted • InfiniBand Multicast member is added • InfiniBand Multicast member is deleted • InfiniBand Node is added to a partition • InfiniBand Node is deleted from a partition • InfiniBand Node is added to subnet • InfiniBand Nod[...]

148 The T rap Receivers, Insert T rap... window appears. 4. Enter the IP address of the desired T rap Receiver in the Addr ess field. 5. Enter the SNMP community string in the Community field. 6. Check the Receive Events box if you want to enable the T rap Receiver to receive events. Y ou can also add the Trap Recei ver , then enable it later . 7. [...]

149 14 Monitoring Through the CLI This chapter gives an overview of the following: • “About InfiniBand Events” on page 149 • “About T racing” on page 149 • “About SNMP Traps” on page 151 • “Configuring SNMP Settings” on page 152 About InfiniBand Event s Event messages are generated whenever a signifi cant event occurs in th [...]

150 The trace feature has different levels of c ontrol to allow you to dynamically select the amount and/or type of tracing information to be generated. T ypes of T races The following types of traces are available: • Application • Module Application "Application" is used to specify the application for whi ch tracing should be configu[...]

151 Flow "Flow" is used to specify a particular part of the code. The way that flows are specified varies depending on the module. • 0x1 : This flow shows all the Subnet Man agement (SM) discovery , assignment routi ng and routing processing. • 0x2: This flow shows all the Subn et Ma nagement Agent (SA) related processing. • 0x4 : T[...]

152 • InfiniBand Node is added to a partition • InfiniBand Node is deleted from a partition • InfiniBand Node is added to subnet • InfiniBand Node is deleted from subnet Configuring SNMP Settings V i ewing Current SNMP T rap Receivers 1. Enter the show sn mp command in the CLI. The current T rap Receivers ar e displayed in the output. Add a[...]

Index A acceptable p_key values ........................................56 access changing access levels .................................... 52 community strings ........................................... 51 access levels ..........................................................47 adding new users ................................................[...]

master poll retries ............................................ 68 default link down ................................................134 default partitions ...................................................54 default passwords ..................................................46 deleting a user account ............................................[...]

IPoIB .......................................................................2 L LID .............................................................. 132 , 133 destination .......................................................78 source .............................................................. 78 link speed active ...............................[...]

PKey ................................................................74 , 89 poll interval synchronization ...............................................69 poll-interval ...........................................................86 port transition ......................................................133 PortGUID fiel d ...........................[...]

super ...................................................................... 46 switch element display route ....................................................91 switch elements sm routing .........................................................6 synchronization cold sync .....................................................5 , 69 poll interval [...]