HP (Hewlett-Packard) XC manual

HP X C S y st ems w ith HP Serv e r Blades and Enc losur es Ho wT o V er si on 3 . 1 or V er si on 3 .2 P ublished: Apr il 200 7 E dition: 9[...]

© Copyright 2006, 2007 Hewlett-P ackard Development Company , L.P . Confidential computer softw are. V alid license from HP required for possession, use or copying. Consistent with F AR 12.211 and 12.212, Commercial Computer Softw are, Computer Software Documentation, and T echnical Data for Commercial Items are licensed to the U.S. Gov ernment un[...]

T able o f C ont ents 1 Ov ervi e w ......................................................................................................................... 9 1 .1 M in i m um R e qu i re m en t s ...................................................................................................................9 1 .2 R ea d t he D o cu m en t at i[...]

6 Making Node -S pec if i c Settings .................................................................................. 33 6 .1 M ak i n g Se t ti n gs o n N on - Bl a d e Se r v e rs ...........................................................................................3 3 6 .2 M a ki n g Se t ti n gs o n H P Pr o Li a nt S e r v e r Bl a de s[...]

L is t of F igur es 3 -1 A dm i ni s tr a ti o n Ne t w o rk C o nn e c ti o ns ..........................................................................................1 6 3 -2 C on s ol e N et w o r k Co n ne c ti o ns ......................................................................................................1 7 3 -3 G ig a bi t E th [...]

6[...]

L is t of T a bles 1 -1 M in i mu m R eq u ir e me n t s.................................................................................................................9 2 -1 I ns t al l at i on a n d Co n fi g u ra t io n C he c kl i s t......................................................................................1 3 4 -1 H ea d N od e I [...]

8[...]

1 Ov erv i e w HP Server Blade c-Class servers (hereafter called serv er blades) are perfectly suited to form HP XC systems. Physical characteristics make it possible to have many tightly interconnected nodes while at the same time reducing cabling requirements. T ypically , serv er blades are used as compute nodes but they can also function as the[...]

If you do not hav e the required documentation in y our possession, see the following sources: • The most current documentation for HP Serv er Blades, enclosures, and other serv er blade components is av ailable at the following W eb site: http://www .hp.com/go/bladesystem/documentation • The most current edition of the V ersion 3.1 or V ersion[...]

• HP ProLiant BL685c (full-height) — Up to four dual core Opteron processors — Four built-in NICs — T wo hot plug driv es — Three mezzanine slots • HP ProLiant BL860c (full-height) 1 .4.2 Enc losur e s and Onboar d Adminis tr ator s HP Server Blade c7 000 Enclosure The HP Server Blade c7000 Enclosure is the enclosure model supported for[...]

Onboard Administrator . On server blades, iLO2 adv anced features are enabled by default and include the following: • Full remote graphics console access including full keyboard, video, mouse (KVM) access through a W eb browser • Support for remote virtual media which enables you to mount a local CD or diskette and serve it to the serv er blade[...]

2 T ask Summar y and Chec klis t This chapter contains a summary of the steps required to configure HP server blades in an HP XC cluster . 2 . 1 Best Pr acti ce fo r S ys tem C onf i gur ati on In order to function properly as an HP XC system, each component must be configured according to HP XC guidelines. T o make configuration settings on certai[...]

T able 2 - 1 Installation and Configuration Checklist (continued) Where Documented T ask Description T ask Category Section 5.4 Run the discover --enclosurebased --network command on the head node to discover the switches Section 5.5 Set the Onboard Administrator password, which must match the passwords on the ProCurv e switches and console managem[...]

3 C abling The following topics are addressed in this chapter: • “Network Ov erview ” (page 15) • “Cabling for the Administration Network” (page 15) • “Cabling for the Console Network” (page 16) • “Cabling for the Interconnect Network” (page 17) • “Cabling for the External Network” (page 19) 3 . 1 Net w ork Ov er v ie [...]

Figur e 3- 1 Administration Network Connections NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 iLO2 Interconn ect bay 1 Interconn ect bay 2 Interconn ect bay 3 Interconn ect bay 4 Interconn ect bay 5 Interconn ect bay 6 Interconn ect bay 7 Interconn ect bay 8 ONBO ARD ADMINIS TRA TOR MG T NIC PCI SLOT PCI SLOT NIC Non-B[...]

Figur e 3- 2 Console N etwor k Connections NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 iLO2 Interconn ect bay 1 Interconn ect bay 2 Interconn ect bay 3 Interconn ect bay 4 Interconn ect bay 5 Interconn ect bay 6 Interconn ect bay 7 Interconn ect bay 8 ONBO ARD ADMINIS TRA TO R MG T NIC PCI SLOT PCI SLOT NIC Non-Blade[...]

Figur e 3-3 Gigabit Eth er net Interconnect Connec tions NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 iLO2 Interconn ect bay 1 Interconn ect bay 2 Interconn ect bay 3 Interconn ect bay 4 Interconn ect bay 5 Interconn ect bay 6 Interconn ect bay 7 Interconn ect bay 8 ONBO ARD ADMINIS TRA TOR MG T NIC PCI SLOT PCI SLOT [...]

3 .4. 3 C onf i gur ing the Inter connect Netw or k Ov er the Adminis tr ati on Netw ork In cases where an additional Gigabit Ethernet port or switch may not be available, the HP XC System Softw are enables you to configure the interconnect on the administration netw ork. When the interconnect is configured on the administration netw ork, only a si[...]

server blades do not have three NICs, and therefore, half-height serv er blades are not included in this example Because NIC1 and NIC3 on a full-height server blade are connected to interconnect bay 1, y ou must use VLANs on the switch in that bay to separate the external network from the administration network. Also, in this example, PCI Ethernet [...]

Figur e 3-6 External Network Connections: Half-Height Ser v er Blades and NI C1 and NIC2 in Use NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 iLO2 Interconn ect bay 1 Interconn ect bay 2 Interconn ect bay 3 Interconn ect bay 4 Interconn ect bay 5 Interconn ect bay 6 Interconn ect bay 7 Interconn ect bay 8 ONBO ARD ADMI[...]

Figur e 3- 7 External Net w ork Connections: Half and Full-Height Server Blades and NI C1 in Use NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 iLO2 Interconn ect bay 1 Interconn ect bay 2 Interconn ect bay 3 Interconn ect bay 4 Interconn ect bay 5 Interconn ect bay 6 Interconn ect bay 7 Interconn ect bay 8 ONBO ARD ADM[...]

4 Inst alling HP X C S y st em So ft w ar e On the Head Node The information in this chapter parallels the information in Chapter 2 in the HP XC System Software Installation Guide . At some points, y ou might be instructed to refer to that document. T able 4-1 lists where to find head node installation instructions. The instructions differ if the h[...]

1. T o provide IP addresses for the Onboard Administrator and all iLO2 devices in the enclosure, disconnect the Onboard Administrator associated with the head node from the administration network, and plug the Onboard Administrator into an activ e netw ork with a DHCP server . 2. Obtain the IP address of the Onboard Administrator from the Insight D[...]

1. Insert the HP XC System Softw are V ersion 3.1 or V ersion 3.2 DVD into the DVD drive of the local laptop or PC. 2. Do the following from the Onboard Administrator management W eb page: a. In the left frame, click the plus sign (+) to open the Device Bays menu. b. Click the plus sign (+) next to the node that represents the head node. c. Click o[...]

8. See T able 2-5 in the HP XC System Software Installation Guide , which describes each installation prompt and provides information to help you with your answ ers. 9. When the HP XC softw are installation process is complete, log in as the root user . 10. Follow the instructions in Sections 2.4 and 2.5 in the HP XC System Software Installation Gu[...]

5 Disco v er ing the H ar d w ar e Com ponen ts This chapter describes the following tasks, which y ou must complete in the order shown: • “T ask 1: Prepare for the System Configuration” (page 27) • “T ask 2: Change the Default IP Address Base (Optional)” (page 28) • “T ask 3: Use the cluster_prep Command to Prepare the System” (p[...]

5 .2 T ask 2 : Change the De fa ult IP Addr e ss Bas e (Optio nal) Y ou may need to change the default IP address base of the HP XC priv ate networks if the default v alues conflict with another IP address base at your site. The IP address base is defined in the base_addrV2.ini file. Follow this optional procedure to change the default IP address b[...]

on the head node, to start the MySQL service, and initialize the configuration and management database: 1. Begin this procedure as the root user on the head node. 2. Change to the following directory: # cd /opt/hptc/config/sbin 3. Start the system preparation process: # ./cluster_prep --enclosurebased 4. See Chapter 3, T ask 3 in the HP XC System S[...]

Enter the MAC address of the switch the head node plugs into in the format xx:xx:xx:xx:xx:xx : your_MAC_address Please enter the Procurve switch Administrator password : your_password Please re-enter password: your_password Discovering Switches... Restarting dhcpd Waiting for IP addresses to stabilize .................... done discoverSwitches: IP:[...]

1. Change to the following directory: # cd /opt/hptc/config/sbin 2. Discover all enclosures: # ./discover --enclosurebased --enclosures Discovery - XC Cluster version HP XC V3.1 RC2 20061022 Enter the common user name for all console port management devices: your_username Please enter the password for Administrator: your_password Please re-enter pa[...]

Starting CMF for discover... Stopping cmfd: [FAILED] 1 Starting cmfd: [ OK ] Waiting for CMF to establish console connections .......... done uploading database Restarting dhcpd Opening /etc/hosts Opening /etc/hosts.new.XC Opening /etc/powerd.conf Building /etc/powerd.conf ... done Attempting to start hpls power daemon ... done Waiting for power da[...]

6 Making Node - Spec ifi c Se ttings Running the discov er process has assigned IP addresses to all hardw are components. Y ou can now use the active administration netw ork to access the v arious hardw are components to make node-specific BIOS settings that are required for HP XC. IMPORT ANT: Making the required settings on each node is a key elem[...]

6. In the left frame, click the plus sign (+) next to Device Bays to display the list of nodes contained in the enclosure. 7. Click the link to the first hardw are model in the list. W ait a few seconds until the frame to the right is populated with node-specific information. 8. Click the Boot Options tab. a. Select a boot device, and use the up an[...]

e. Click the Apply button to sav e the settings. 12. Configure disks into the smart array from the remote graphics console. All server blades hav e smart array cards, you must add the disk or disks to the smart array before attempting to image the node. T o set up the smart array device, click the Remote Console tab on the virtual console page of t[...]

16. Close the iLO2 utility W eb page. 17. Repeat this procedure from every active Onboard Administrator and make the same settings for each server blade in each enclosure. 6 . 3 Making Se ttings on HP Integr it y Serv er Blades Use the management processor (MP) on each HP Integrity serv er blade to make the following settings: • Clear all event l[...]

9. Enter UC (user configuration) and use the menu options to remov e the default administrator and operator accounts. Then, for security purposes, create your own unique user login ID and password. Assign all rights (privileges) to this new user . The user login ID must have a minimum of 6 characters, and the passw ord must have exactly 8 character[...]

15. T urn off power to the node: a. Press Ctrl+b to exit the console mode. b. Return to the Command Menu : MP> CM c. T urn off power to the node: MP:CM> pc -off -nc 16. Use the RB command to reset the BMC. 17. Press Ctrl+b to exit the console mode and press the x key to exit. 38 Making Node-Specific Settings[...]

7 C onf i gur ing the HP X C S y s tem The information in this chapter parallels the information in Chapter 3 in the HP XC System Software Installation Guide . T o configure your system, y ou use a combination of the instructions in this HowT o and the instructions in the HP XC System Software Installation Guide . P erform the configuration tasks i[...]

# startsys --image_only --flame_sync_wait=480 --power_control_wait=90 --image_timeout=90 2. Boot all nodes when the imaging process is complete: # startsys --power_control_wait=90 --boot_group_delay=45 --max_at_once=50 For more information about startsys options, see startsys (8). 7 . 3 T ask 3: V er if y Su ccess Go to Chapter 4 in the HP XC S[...]

8 T r ouble shooting This chapter contains troubleshooting information to help y ou resolve problems y ou might encounter . 8. 1 One or Mor e P orts Do not C ommuni cate Pr operl y on a G i gabit Ether net S w itc h If your hardw are configuration contains Gigabit Ethernet switches, verify the following if one of more ports do not communicate prope[...]

42[...]

A C onf i gur ation Ex amples This appendix contains illustrations and descriptions of fully cabled HP XC systems based on interconnect type and server blade height. The connections are color-coded, so consider viewing the PDF file online or printing this appendix on a color printer to take advantage of the color coding. A. 1 Gi gab it Ether net In[...]

Figur e A- 2 InfiniBand Interconnect With Full-Height Server Blades NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 iLO2 Interconn ect bay 1 Interconn ect bay 2 Interconn ect bay 3 Interconn ect bay 4 Interconn ect bays Interconn ect bay 7 Interconn ect bay 8 ONBO ARD ADMINIS TRA TOR MG T NIC PCI SLOT PCI SLOT NIC Non-Bl[...]

Figur e A-3 InfiniBand Interconnect With Mixed Height Server Blades NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 iLO2 Interconn ect bay 1 Interconn ect bay 2 Interconn ect bay 3 Interconn ect bay 4 Interconn ect bays Interconn ect bay 7 Interconn ect bay 8 ONBO ARD ADMINIS TRA TO R MG T NIC PCI SLOT PCI SLOT NIC Non-B[...]

46[...]

Inde x A administration network activ ating, 29 as interconnect, 19 defined, 15 B base_addrV2.ini file, 28 boot order , 34 C checklist, 13 cluster_config utility , 39 cluster_prep command, 28 communication problems with GigE switch ports, 41 console network defined, 16 D discover command enclosures, 30 nodes, 30 switches, 29 discovery process, 27 d[...]

O onboard administrator connecting to, 23 defined, 11 IP address, 33 putting on an active netw ork, 23 setting the password, 30 static IP address for , 24 OVP command, 40 P password iLO2, 34 MP , 37 onboard administrator , 30 public network ( see external netw ork) R real enclosure defined, 27 S server blade boot order , 34 defined, 9 preparing HP [...]