IBM Partner Pavilion DS4100 manual

IB M T o ta lSt o ra ge DS4 1 00 Storage Ser v er In sta ll a ti o n , Us e r ’ s , an d M a in t e n an ce Gu i d e GC26-7712-00 [...]

[...]

IB M T o talS to rag e DS4 1 00 St o rage Server In sta ll a ti o n , Us e r ’ s , an d M ain t e n a n c e Gu i de GC26-7712-00 [...]

Note: Before using this information and the product it supports, be sure to read the general information in “Notices” on page 127. First Edition (November 2004) © Copyright International Business Machines Corporation 2004. All rights reserved. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedu[...]

Safety The caution and danger statements that this document contains can be referenced in the multilingual IBM ® Safety Information document that is provided with your IBM T otalStorage ® DS4100 storage server . Each caution and danger statement is numbered for easy reference to the corresponding statements in the translated document. v Danger: T[...]

Statement 1: DANGER Electrical current from power , telephone, and communication cables is hazardous. To avoid a shock hazard: v Do not connect or disconnect any cables or perform installation, maintenance, or reconfiguration of this product during an electrical storm. v Connect all power cords to a properly wired and grounded electrical outlet. v [...]

Statement 3: CAUTION: When laser products (such as CD-ROMs, DVD drives, fiber optic devices, or transmitters) are installed, note the following: v Do not remove the covers. Removing the covers of the laser product could result in exposure to hazardous laser radiation. There are no serviceable parts inside the device. v Use of controls or adjustment[...]

Statement 4: ≥ 18 kg (39.7 lb) ≥ 32 kg (70.5 lb) ≥ 55 kg (121.2 lb) CAUTION: Use safe practices when lifting. Statement 5: CAUTION: The power control button on the device and the power switch on the power supply do not turn off the electrical current supplied to the device. The device also might have more than one power cord. To remove all el[...]

Statement 8: CAUTION: Never remove the cover on a power supply or any part that has the following label attached. Hazardous voltage, current, and energy levels are present inside any component that has this label attached. There are no serviceable parts inside these components. If you suspect a problem with one of these parts, contact a service tec[...]

viii IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Contents Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii T ables . . . . . . . . . . . . . . . . . . . . . . . . . . . .x v About this document . . . . . . . . . . . . . . . . . . . . . xvii F AStT product renaming . . . . . . . . . . . . . . . . . . . . . xvii Wh[...]

Preparing the site . . . . . . . . . . . . . . . . . . . . . . .1 9 Preparing the rack cabinet . . . . . . . . . . . . . . . . . . . .1 9 Installing the DS4100 . . . . . . . . . . . . . . . . . . . . . .2 0 Rack mounting template . . . . . . . . . . . . . . . . . . . .2 0 Installing the support rails . . . . . . . . . . . . . . . . . . . .2 2 Remov[...]

T urning off the storage server . . . . . . . . . . . . . . . . . .6 7 Restoring power after an unexpected shutdown . . . . . . . . . . . . .6 8 Performing an emergency shutdown . . . . . . . . . . . . . . . .6 9 Restoring power after an emergency shutdown . . . . . . . . . . . .6 9 Restoring power after an over-temperature shutdown . . . . . . . .[...]

European Union EMC Directive conformance statement . . . . . . . . 129 T aiwan Class A warning statement . . . . . . . . . . . . . . . . 130 Japanese V oluntary Control Council for Interference (VCCI) statement 130 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . 131 131 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 [...]

Figures 1. Installation process flow by current publications . . . . . . . . . . . . . . . . . . . xix 2. DS4100 hot-swap drive bays . . . . . . . . . . . . . . . . . . . . . . . . . . .8 3. Front controls and indicators . . . . . . . . . . . . . . . . . . . . . . . . . . .9 4. Back view of DS4100 base storage server . . . . . . . . . . . . . . . .[...]

51. Location of host cables (single-controller storage server) . . . . . . . . . . . . . . . .5 6 52. Ethernet and serial port locations . . . . . . . . . . . . . . . . . . . . . . . . .5 7 53. Adding an expansion unit . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 9 54. DS4000 EXP100 redundant loop configuration . . . . . . . . . . . . .[...]

T ables 1. Mapping of F AStT names to DS4000 Series names . . . . . . . . . . . . . . . . . . xvii 2. T otalStorage DS4500 storage server document titles by user tasks . . . . . . . . . . . .x x 3. T otalStorage DS4400 storage server document titles by user tasks . . . . . . . . . . . . xxii 4. T otalStorage DS4300 storage server document titles by[...]

xvi IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

About this document This document provides instructions for installing and customizing the configuration of your IBM T otalStorage DS4100 storage server . It also provides hardware maintenance procedures and troubleshooting information. F AStT product renaming IBM is in the process of renaming some F AStT family products. T able 1 identifies each n[...]

How this document is organized Chapter 1, “Introduction,” on page 1 describes the IBM T otalStorage DS4100 storage server . This chapter includes an inventory checklist and an overview of the storage server features, operating specifications, and components. Chapter 2, “Installing the storage server,” on page 17 contains instructions for in[...]

v Danger: These statements indicate situations that can be potentially lethal or extremely hazardous to you. A danger statement is placed just before the description of a potentially lethal or extremely hazardous procedure step or situation. Figures used in this document The figures used in this document are for illustrative purposes only . In some[...]

DS4000 Storage Server publications The following tables present an overview of the DS4500, DS4400, DS4300 Fibre Channel, and DS4100 S ATA Storage Server product libraries, as well as other related documents. Each table lists documents that are included in the libraries and what common tasks they address. Click on active links in the tables to acces[...]

T able 2. T otalStorage DS4500 storage server document titles by user tasks (continued) Title User T asks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM F AStT Management Suite Java User’s Guide , 32P0081 U U IBM T otalStorage DS4000 Hardware Maintenance Manual, GC26-7[...]

DS4400 storage server library T able 3 associates each document in the DS4400 (previously F AStT700) storage server library with its related common user tasks. T able 3. T otalStorage DS4400 storage server document titles by user tasks Title User T asks Planning Hardware Installation Software Installation Configuration Operation and Administration [...]

DS4300 storage server library T able 4 associates each document in the DS4300 (previously F AStT600) storage server library with its related common user tasks. T able 4. T otalStorage DS4300 storage server document titles by user tasks Title User T asks Planning Hardware Installation Software Installation Configuration Operation and Administration [...]

T able 5. T otalStorage DS4100 storage server document titles by user tasks Title User T asks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM T otalStorage DS4100 Installation, User’s, and Maintenance Guide , GC26-7712 U U U U IBM T otalStorage DS4100 Fibre Channel Cabl[...]

DS4000-related hardware publications T able 6 associates each of the following documents related to DS4000 (previously F AStT) operations with its related common user tasks. T able 6. T otalStorage DS4000-related document titles by user tasks Title User T asks Planning Hardware Installation Software Installation Configuration Operation and Administ[...]

T able 6. T otalStorage DS4000-related document titles by user tasks (continued) Title User T asks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM F AStT Host Adapter Installation and User’s Guide , 59P5712 U U RS/6000 Eserv er pSeries Fibre Channel Planning and Integra[...]

Getting information, help, and service If you need help, service, or technical assistance or just want more information about IBM products, you will find a wide variety of sources available from IBM to assist you. This section contains information about where to go for additional information about IBM and IBM products, what to do if you experience [...]

v Yo u can order publications through the IBM Publications Ordering System at the following web site: www .elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi/ v For the latest information about IBM xSeries products, services, and support, go to the following Web site: www .ibm.com/eserver/xseries/ v For the latest information abo[...]

T able 9 on page 14 lists the environmental specifications for the DS4100. How to send your comments Y our feedback is important in helping us to provide the most accurate and high-quality information. If you have comments or suggestions for improving this publication, you can send us comments electronically by using these addresses: v Internet: st[...]

xxx IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Chapter 1. Introduction This chapter describes the operating specifications, features, and components for the IBM T otalStorage DS4100 storage server (hereafter referred to as DS4100 or storage server ). This chapter also includes a list of hardware that comes with the storage server . Overview IBM DS4000 solutions support the large and growing dat[...]

connections for expanding the DS4100 storage capacity to S ATA hard drives in external DS4000 EXP100 storage expansion units. Currently , the only DS4000 storage expansion unit model that can be attached to the DS4100 base storage server is the DS4000 EXP100. Do not connect other DS4000 storage expansion units (such as the DS4000 EXP700 or DS4000 E[...]

v The DS4100 single-controller storage server does not support automatic firmware updates. All firmware upgrades and downgrades for the DS4100 single-controller storage server must be performed manually , as described in “Manually upgrading the firmware for the DS4100 single-controller storage server” on page 103. v All DS4100 single-controller[...]

Go to the Personalized Support section of the web page and click My Support . On the next page, go to the We use IBM Registration section. To register to use this site, click Register . Perform the following steps to receive product updates: 1. Once you have registered, type your user ID and password to log into the site. The Welcome page opens. 2.[...]

Clustering support Clustering is a means of sharing array groups among controllers to provide redundancy of controllers and servers. This redundancy is important if a hardware component fails. If a hardware component failure occurs in a cluster , another server takes ownership of the array group. Clustering requires software specific to your operat[...]

Y our DS4100 storage server has 14 hard disk drive slots; any drive slots not containing hard disk drives will contain blank drive trays. Each of the 14 drive slots must always contain either a blank tray or a hard disk drive. – Box ID labels (used to label the storage server IDs on the front of the DS4100) – Wrap plug and coupler kit Attention[...]

the DS4000 storage expansion units attached to the DS4100 storage server should be plugged into the same two independent external power circuits as the DS4100. This ensures that the DS4100 storage server and all its attached storage expansion units will have power if only one power circuit is available. In addition, having all the right or all the [...]

Attention: The DS4100 single-controller storage server does not support hot-swap RAID controllers or power supplies. Yo u will lose access to data if you attempt to remove or replace a functioning (non-failed) RAID controller or power supply in a DS4100 single-controller storage server without first powering down the storage server . Hot-swap drive[...]

The DS4100 has blank trays in the unused drive bays. To begin installing new drives, you must first remove the blank trays and save them. Each of the 14 bays must always contain either a blank tray or a drive CRU. Yo u can install up to 14 hot-swap drive CRUs in the storage server . v Activity LED: Each drive CRU has an associated Activity LED on t[...]

Figure 5 shows the components at the back of the DS4100 single-controller storage server . The DS4100 single-controller storage server has a controller blank instead of a second RAID controller and a power supply blank instead of a second power supply . RAID controller Each RAID controller contains three ports for Small Form-Factor Pluggable (SFP) [...]

fan CRU continues to operate. Yo u must install both fan CRUs to maintain proper cooling within the storage server , even if one fan CRU is not operational. Hot-swap power supplies The DS4100 base storage server comes with two hot-swap power supplies. Yo u must install both power supplies to maintain proper cooling. The DS4100 single-controller sto[...]

RAID controller Each RAID controller contains several connectors and LEDs. Each controller has two host ports and one expansion port for connecting the storage server to hosts or expansion units. Yo u first insert SFPs into the ports and then connect the fibre channel cables. Host ports The host ports (labelled Host 1 and Host 2 ) are used to conne[...]

Attention: When connecting the DS4100 to DS4000 EXP100 storage expansion units, DO NOT use the tens digit (x10) setting. Use only the ones digit (x1) setting to set unique server IDs or enclosure IDs. For more information, see “Server ID (enclosure ID) settings” on page 33. 1 Gbps/2 Gbps switch The default switch setting is 2 Gbps. This is the [...]

DS4100 specifications T able 9 lists the physical specifications for the DS4100. T able 9. IBM T otalStorage DS4100 specifications Size v Width : 48.18 cm (18.97 in.) v Height : 13.23 cm (5.21 in.) v Depth : 59.74 cm (23.52 in.) Weight v Drive-ready weight (without disk drive modules installed): – Model 100: 26.64 kg (58.9 lb) – Model 1S x : 23[...]

T able 10. IBM T otalStorage DS4100 AC power requirements Low Range High Range Nominal V oltage 90 to 136 VA C 180 to 264 VA C Frequency 50 to 60 Hz 50 to 60 Hz Idle Current 2.50 A a 1.33 A b Maximum Operating Current 3.01 A a 1.57 A b a. T ypical voltage: 11 5 V AC, 60 Hz at 77% power-supply efficiency and 0.96 power factor b. T ypical voltage: 23[...]

Cold aisle Air conditioner Hot aisle back front back back front Airflow front 2440 mm between center lines of hot and cold aisle P erforated tiles or gratings 1220 mm cold aisle width T42 rac ks T42 rac ks T42 rac ks f10ug001 Figure 8. Example of cold aisle/hot aisle rack cabinet configuration 16 IBM T otalStorage DS4100 Storage Server: Installatio[...]

Chapter 2. Installing the storage server This chapter provides the information that you need to install the DS4100 into a rack cabinet. This chapter also contains information about cabling, setting interface options, and installing optional cables. Installation overview The following steps provide an overview of the installation process you will pe[...]

10. Install the DS4000 Storage Manager 9.10 host software. Refer to the IBM T otalStorage DS4000 Storage Manager 9.10 Installation and Support Guide for the appropriate operating system for instructions on how to install the DS4000 Storage Manager 9.10 host software. 11 . T urn on the DS4100. See “T urning the storage server on and off” on page[...]

CAUTION: Use safe practices when lifting. 3. Remove the DS4100 from its shipping container and check the contents (see “Inventory checklist” on page 5). If any items are missing, contact your IBM reseller before proceeding. 4. Assemble the tools and equipment that you will need for installation. These might include: v The power cords that are s[...]

v Always connect the storage server to a properly grounded outlet. v Always connect the rack power distribution units (PDUs) to at least two different power circuits or sources. Complete the following steps to prepare the rack cabinet before you install the DS4100: 1. Move, unpack, and level the rack cabinet at the installation site (if needed). 2.[...]

Rac k Mounting T emplate FR ONT Rail assembly () Front Right Rail assembly () Front Left Bezel () Front Left Bezel () Front Right DS4100 0 U 1 U 3 U 2 U 3 U 2 U 1 U 0 U f10ug058 Figure 9. Front rack mounting template Chapter 2. Installing the storage server 21[...]

Installing the support rails Before installing the DS4100 in a rack cabinet, you must install the rails and rack-mounting hardware that come with your storage server . The DS4100 requires an Electronic Industries Association (EIA) 310-D T ype A 19-inch rack cabinet. The distance between EIA rails, from the front to the rear of the rack cabinet, is [...]

Attention: Both support rails are shipped with three screws and spacers, called alignment pins , already installed at the front of each rail. They have been inserted in the front of each support rail as guides to indicate where not to use M6 screws when you attach the front of the support rail to the rack cabinet. See Figure 11 . Depending on which[...]

5. Starting with the left rail, remove and save the 6/32 in. screw ( 5 ). Loosen the four rail adjustment screws ( 2 ). Note: The support rails are not marked as Left or Right. However , each rail can be correctly mounted on only one side of the rack cabinet. The rails should be mounted with the alignment pins at the front of the rack c[...]

Removing the CRUs Attention: This section describes how to remove the CRUs in order to minimize the weight of the DS4100 before you install it in the rack cabinet. However , if you have three or more people available to lift and install the DS4100 in a rack cabinet, you might not find it necessary to remove the CRUs before you install the DS4100. I[...]

1. Push down on the latch (centered above the RAID controller). Pull both levers at the same time out of the locked position. 2. Grasp the pull-rings; then pull on the levers and remove the RAID controller . 3. Place the RAID controller on a level surface. 4. For the DS4100 base storage server , repeat step 1 through step 3 for the second RAID cont[...]

3. Place the power supply on a level surface. 4. For the DS4100 base storage server , repeat step 1 through step 3 for the second power supply . Note: In the DS4100 single-controller storage server , you do not need to remove the power supply blank. Removing a fan Complete the following steps to remove a fan: 1. Unlock the latch by moving the latch[...]

Removing a hard disk drive Complete the following steps to remove a hot-swap hard disk drive: Note: The hard disk drive comes installed in a drive tray . Do not attempt to detach the hard disk drive from the drive tray . 1. Use T able 19 on page 11 9 to record the location and identify your DS4000 S ATA hard disk drives. Yo u must record this infor[...]

2. The bezel is secured to the front of the DS4100 either by plastic wingnuts or by adhesive tape. Remove and discard the wingnuts or the adhesive tape. The bezel should remain attached to the DS4100 chassis. Attention: The bottom cross member of the bezel has a tab that must fit beneath the DS4100 chassis and snap into place. The top cross member [...]

Replacing a RAID controller Complete the following steps to replace the RAID controller that you removed before the installation: 1. Slide the RAID controller all the way into the empty slot. 2. Close both levers until the latch locks into place. Attention: Make sure that the levers lock into place in the storage server chassis. 3. For the DS4100 b[...]

2. Close the lever until the pull-ring latch locks in place. Make sure that the lever locks into place in the storage server chassis. Then gently push the front of the power supply to ensure that it is fully seated. 3. For the DS4100 base storage server , repeat step 1 and step 2 to replace the second power supply . Replacing a fan Complete the fol[...]

3. If the latch does not automatically lock when you have successfully inserted the fan unit into the bay , pull back slightly on the fan and then push it in again until the latch snaps into place. 4. Repeat step 1 through step 3 to replace the second fan. Replacing a hard disk drive Complete the following steps to replace the hot-swap hard disk dr[...]

Fibre channel loop and ID settings When you install a hard disk drive in the expansion unit, the drive tray plugs into a printed circuit board called the midplane . The midplane sets the fibre channel loop ID automatically , based on the enclosure ID switch setting and the physical location (bay) of the hard disk drive. Server ID (enclosure ID) set[...]

valid for the drive loop. The amber Conflict LED (located on the right-side of the switch) will be lit if another unit in the drive loop has the same ID setting as the server ID. Attention: When connecting the DS4100 to DS4000 EXP100 storage expansion units, DO NOT use the tens digit (x10) setting. Use only the ones digit (x1) setting to set unique[...]

Ethernet Host-agent software Client software Controller Controller Controller Controller Fibre Channel I/O path Management station (one or more) Note: The client software is installed on one or more management systems, or on the host computer . f10ug004 Direct (out-of-band) management method This method uses Ethernet connections from a management s[...]

Ethernet Host computer Controller Controller Controller Controller Fibre Channel I/O path Management station Note: The client software is installed on one or more management systems, or on the host computer . Storage subsystems f10ug008 Fibre channel connections The storage server fibre channel connection consists of up to four host fibre-channel l[...]

connect more than two hosts to the fibre channel storage server through switches. The illustrations in the following sections show common host system configurations. Note: The default configuration is one partition. Four-partition, eight-partition, and sixteen-partition configuration upgrades are available as upgrade options. Yo u must perform part[...]

In Figure 26 on page 39, the fibre channel switches are connected together via the inter-switch link (ISL) to form a single SAN fabric. Host system with two host adapters Host system with two host adapters Host system with two host adapters DS4100 Switch Switch Note: Node lev el redundancy with cluster software. Host system with two host adapters H[...]

In Figure 27, the fibre channel switches are not connected together . Each switch forms its own SAN fabric. Host system with two host adapters Host system with two host adapters Switch Switch Interswitch link f10ug043 DS4100 Figure 26. Example of a single SAN fabric configuration (base storage server) Host system with two host adapters Host system [...]

DS4100 single-controller storage server configurations: This section provides the following examples of DS4100 single-controller storage server host fibre channel configurations. v Single server configuration with one adapter , as shown in Figure 29 v Single server configuration with two adapters, as shown in Figure 30 on page 41 v Dual server conf[...]

Host Ser v er DS4100 Single Controller f11ug002 Figure 30. Example of a single server configuration with two adapters (single-controller storage server) Host Ser v er Host Ser v er DS4100 Single Controller f11ug003 Figure 31. Example of a dual server configuration with one adapter on each server (single-controller storage server) Chapter 2. Install[...]

Installing the storage subsystem configuration Using the information gathered in the previous sections, install the host systems and host adapters. Notes: 1. See the documentation provided with your host adapters for installation requirements and procedures. Host Ser v er FC Switch Host Ser v er DS4100 Single Controller f11ug004 Figure 32. Example [...]

2. Use the correct host adapter driver . For the latest supported host adapters and drivers, go to the following Web site: www .ibm.com/servers/storage/disk/ Attach fiber-optic interface cables to each host adapter . Yo u will connect the other end of the cables to the controller later in the installation process. For more information about handlin[...]

44 IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Chapter 3. Cabling the storage server This chapter provides fibre channel and power cabling information for the storage server . After you attach the storage server power cables, use the instructions that are provided in “T urning the storage server on and of f” on page 65 for the initial startup of the storage server . W orking with SFPs and f[...]

v Do not overtighten the cable straps or bend the cables to a radius of less than 38 mm (1.5 in.). v Do not put excess weight on the cable at the connection point. Be sure that the cable is well supported. Installing SFP modules The DS4100 host ports require SFP modules. SFP modules are used to convert electrical signals to optical signals that are[...]

DANGER Some laser products contain an embedded Class 3A or Class 3B laser diode. Note the following. Laser radiation when open. Do not stare into the beam, do not view directly with optical instruments, and avoid direct exposure to the beam. Attention: When you handle static-sensitive devices, take precautions to avoid damage from static electricit[...]

Removing SFP modules To remove the SFP module from the host port, perform the following steps. Attention: To avoid damage to the cable or to the SFP module, make sure you unplug the LC-LC fibre channel cable before you remove the SFP module. 1. Remove the LC-LC fibre channel cable from the SFP module. For more information, see “Handling fiber-opt[...]

1. Remove the protective cap from the fiber-optic cable. See Figure 38. 2. Connect the fiber-optic cable to the installed SFP , as shown in Figure 39. 3. Check the LEDs on the RAID controller . When the RAID controller is operating properly , the amber Fault LED is off. For information about the status of the RAID controller LEDs, see T able 13 on [...]

For more information about cabling these devices, see the documentation that comes with the LC-LC fibre-channel cable. Note: If you are connecting the DS4100 to a 1 Gbps device (such as a 1 Gbps fibre channel switch), you must also use an LC-SC fibre-channel cable adapter . For more information about using an LC-SC fibre-channel cable adapter , see[...]

2. If necessary , remove the protective cap from the SFP module, as shown in Figure 34 on page 47. Save the protective cap for future use. 3. Remove the two protective caps from one end of the LC-LC cable, as shown in Figure 41. Save the protective caps for future use. 4. Carefully insert this end of the LC-LC cable into an SFP module that is insta[...]

1. On the end of the LC-LC cable that connects into the SFP module or host bus adapter , press down and hold the lever to release the latches, as shown in Figure 43. Lev er Latches f10ug018 2. Carefully pull on the connector to remove the cable from the SFP module, as shown in Figure 44. 3. Replace the protective caps on the cable ends. 4. Replace [...]

The following sections provide the procedures for properly connecting and removing an LC-SC fibre-channel cable. Connecting an LC-SC cable adapter to a device Complete the following steps to connect an LC-SC fibre-channel cable adapter to a device: Statement 3: CAUTION: When laser products (such as CD-ROMs, DVD drives, fiber optic devices, or trans[...]

3. Remove the two protective caps from the LC connector end of the LC-SC cable adapter as shown in Figure 46. Save the protective caps for future use. 4. Carefully insert the other end of the LC-LC cable into the LC connector end of the LC-SC cable adapter as shown in Figure 47. Push in the connector until it clicks into place. 5. If you are connec[...]

1. On the end of the cable that connects into the LC connector end of the LC-SC cable adapter , press down and hold the lever to release the latches. Figure 48 shows the location of the lever and latches. Lev er Latches f10ug018 2. Carefully pull on the connector to remove it. Make sure you grasp the connector and not the cable when removing the LC[...]

For examples of redundant, partially-redundant, and non-redundant host and drive loop configurations, see “Fibre channel loop configurations” on page 36. Connecting hosts to a DS4100 single-controller storage server To connect a host adapter to the DS4100 single-controller storage server RAID controllers, perform the following steps: 1. Install[...]

Important: 1. The serial interface ports are intended to be used by service technicians to perform diagnostic operations on the storage server . Incorrect use of the serial port can result in loss of data access and, in some cases, in loss of data. 2. Do not connect the DS4100 in a public LAN or public subnet. Use a local private network for the DS[...]

Attention: In the initial installation of the DS4100, you can add only new expansion units to the DS4100 storage server . This means that there must be no existing configuration information on the expansion unit that you want to install. If the expansion unit that you want to install currently contains logical drives or configured hot spares, and y[...]

3. Remove the blank filler trays and insert drive trays. Connect power to the drive expansion unit first and then connect power to the DS4100. After the storage server is powered on, the storage server locates the new drives. 4. Use the DS4000 Storage Manager client to check the status of the new drives, correct any errors found, and configure the [...]

To create the redundant loop shown in Figure 54, you would perform the following steps: 1. Make two fibre-channel connections from the First expansion unit to the Second expansion unit (connections 1 and 2). 2. Make two fibre-channel connections from the Second expansion unit to the Last expansion unit (connections 3 and 4). Currently , you can con[...]

Power cabling The DS4100 base storage server uses two standard power cords, and the single-controller storage server uses only one. Yo u can connect the power cords to a primary power unit inside the rack cabinet, such as a properly grounded ac distribution unit, or to an external source, such as a properly grounded electrical outlet. The DS4100 st[...]

one-half of the current load. If there is only one power supply unit, it will handle all of the current load. To ensure maximum protection against power loss due to power supply unit failure or a power circuit tripping, both power supply units must be connected and powered on. See T able 10 on page 15 for the DS4100 AC power requirements. See Figur[...]

system documentation to make the new logical drives accessible to the operating system. Do not proceed with the configuration set-up you have completed the DS4000 Storage Manager installation. Assemble any additional items in preparation for software installation. These items might include: v HBA drivers v Controller firmware version 05.42.xx.xx v [...]

64 IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Chapter 4. Operating the storage server To ensure optimal operation of your system, always follow these best practices guidelines: v Back up the data on your storage drives periodically . v DS4100 base storage server only: To maintain power redundancy , plug the storage server’s right and the left power supplies into two independent external powe[...]

T urning on the storage server Important: DS4100 base storage server only: Yo u must turn on the storage expansion units and verify that the storage expansion unit fibre channel connections are optimal by checking the indicator lights before you turn on the DS4100. The controllers might not recognize the correct configuration if the hard disk drive[...]

green on the front of the storage server . If they are not, use the DS4000 Storage Manager client to diagnose the problem (see “Monitoring status through software” on page 70). Note: The green drive active LED and amber drive fault LED below the drive CRUs might flash intermittently as the drives spin-up. W ait until the storage server is finis[...]

3. Stop all I/O activity to the storage server and attached expansion units. Make sure that: v All of the green Drive active LEDs on the front of the storage server (and on all attached expansion units) are not flashing. v The green Cache active LEDs on the back of the storage server are off. Attention: To turn off all power to the storage server ,[...]

Attention: To avoid damage to the hardware, take special care when you restart the system after an unexpected shutdown. If the storage server shuts down unexpectedly , go to “Restoring power after an over-temperature shutdown.” Otherwise, go to “Restoring power after an emergency shutdown.” Note: Unattended restoration of power is supported[...]

4. Complete step 2 on page 66 to determine the proper power-on sequence for your system. 5. When the internal temperature of the storage server is below 35° C (95° F), complete steps 3 and 4 on page 66 to turn on power to the devices in your system and to check the status of the storage server . Monitoring status through software Attention To ens[...]

v Locate the failed component v Determine the recovery procedures to repair the failure Although the storage server has fault LEDs, these lights do not necessarily indicate which component has failed or needs to be replaced, or which type of recovery procedure you must perform. In some cases (such as loss of redundancy in various components), the f[...]

T able 12. Storage server LEDs (front) LED Color Operating states 1 Drive active Green v On - Normal operation. v Flashing - The drive is reading or writing data. v Flashing every 5 seconds - The drive has not spun up or the drive is bad. v Off - One of the following situations has occurred: – The storage server has no power . – The drive is no[...]

T able 13. RAID controller LEDs Icon LED Color Operating states 1 Fault Amber v Off - Normal operation. v On - One of the following situations has occurred: – The RAID controller has failed. – The RAID controller was placed offline – The controller battery had failed (in conjunction with the battery LED in off state). Host loop Green v On - N[...]

T able 13. RAID controller LEDs (continued) Icon LED Color Operating states 1 Expansion port bypass Amber v Off - Normal operation. v On - One of the following situations has occurred: – An SFP module is inserted in the drive loop port and the fibre-channel cable is not attached to it. – The fibre-channel cable is not attached to an expansion u[...]

T able 14. Fan LED (continued) LED Color Operating states 1 1 Always use the DS4000 Storage Manager client to identify the failure. T able 15. Power supply LEDs LED Color Operating states 1 Fault Amber v Off - Normal operation. v On - One of the following situations has occurred: – The power supply has failed. – An over-temperature condition ha[...]

Note: Always use the DS4000 Storage Manager client to check your cache memory settings before assuming a hardware failure. Figure 60 shows the location of the Cache active LED on the front of the RAID controller . RAID controller cache battery Each RAID controller contains a sealed, rechargeable 4-volt lead-acid battery . This battery provides cach[...]

Battery LED f10ug037 Figure 61. Battery LED Chapter 4. Operating the storage server 77[...]

78 IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Chapter 5. Installing and replacing components This chapter provides instructions to help you install or remove customer replaceable units (CRUs), such as hot-swap drives, fans, RAID controllers, and power supplies. This chapter also provides instructions to help you add new DS4000 EXP100 expansion units to existing DS4100 configuration. For more C[...]

Attention: After you remove a drive CRU, wait at least 70 seconds before replacing or reseating the drive CRU to allow the drive to properly spin down. Failure to do so may cause undesired events. Before you install or remove drive CRUs, review the following information: v Blank trays: A storage server without a full set of drives (14) contains bla[...]

1. Read the documentation that comes with the hard disk drive. 2. Check for Fault LEDs shown in Figure 62. If any amber LEDs are lit, see Chapter 6, “Hardware maintenance,” on page 111 . 3. Determine the bay into which you want to install the drive. 4. Remove the blank tray: Note: To locate blank drives, look at the Activity LEDs on the front o[...]

b. Pull the handle on the tray out so that it is in the open position, as shown in Figure 63. T ra y handle T ra y latch f10ug049 c. Slide the drive CRU into the empty bay until the hinge of the tray handle latches beneath the storage server bezel. d. Push the tray handle down until it latches into place. Replacing hot-swap hard disk drives Drive p[...]

a. Release the latch on the drive CRU by pinching together the blue line on the tray latch and the finger hole on the tray handle, as shown by the arrow in Figure 63 on page 82. b. Pull the tray handle out into the open position. c. Lift the drive CRU partially out of the bay . d. To avoid possible damage to the drive, wait at least 70 seconds befo[...]

to use the storage server (or any expansion units attached to the storage server) until you finish the procedure. Yo u must use this method on RAID 0 logical drives. v Replacing the drives one at a time In this method, you manually fail each drive, replace it, and wait for the system to restore data to the new drive before installing the next drive[...]

2. Use the DS4000 Storage Manager client to check the status of the storage server . Correct any problems that are reported. 3. Perform a complete backup of the drives that you are replacing. Yo u need the backup to restore data on the drives later in this procedure. Attention: When you handle static-sensitive devices, take precautions to avoid dam[...]

b. Yo u must turn on the expansion units before the storage server . The controllers might not recognize the correct configuration if the drives are powered up after the storage server . For instructions on powering up the expansion units, see the expansion-unit documentation. c. T urn on the power to the storage server; then restart or turn on the[...]

Attention: When you handle static-sensitive devices, take precautions to avoid damage from static electricity . For details about handling static-sensitive devices, see “Handling static-sensitive devices” on page 79. 4. Unpack the new drives. Set the drives on a dry , level surface away from magnetic fields. Save the packing material and docume[...]

8. Use the DS4000 Storage Manager client to monitor the status of the new drive and the progress of the data reconstruction. W ait for data reconstruction to finish (the Drive activity LED stops flashing). Note: The Drive activity LED will continue to flash after reconstruction is finished if there is I/O activity to that drive. In that case, use t[...]

Both fan units must always be in place, even if one is not functioning properly , to maintain proper cooling. Use the following procedure to replace a hot-swap fan: 1. Check the LEDs on the back of the storage server . 2. If the amber Fault LED is on, remove the fan CRU that has failed. a. Slide the latch to unlock the fan CRU. b. Use the handle (b[...]

to “T urning off the storage server” on page 67 and follow the procedure described there before replacing the power supply on a DS4100 single-controller storage server . Each power supply CRU has a built-in sensor that detects the following conditions: v Over-voltage v Over-current v Overheated power supply If any of these conditions occurs, on[...]

AC power connectors This connection is for the ac power cord. Strain-relief clamp Use this clamp to provide strain relief on the power cord. Secure the clamp tightly with the power-supply clamp-nut. Removing a power supply Statement 8: CAUTION: Never remove the cover on a power supply or any part that has the following label attached. Hazardous vol[...]

3. Unplug the power cord from the electrical outlet. 4. Disconnect the power cord from the ac power connector on the power supply . 5. Remove the power-supply nut and power-supply cable strain-relief clamp from the rear of the power supply . 6. Remove the power supply from the storage server , as follows: a. Grasp the pull-ring on the power-supply [...]

Statement 8: CAUTION: Never remove the cover on a power supply or any part that has the following label attached. Hazardous voltage, current, and energy levels are present inside any component that has this label attached. There are no serviceable parts inside these components. If you suspect a problem with one of these parts, contact a service tec[...]

power to the storage server . Go to “T urning on the storage server” on page 66 and follow the procedure described there before proceeding with the following step. 9. Make sure that the green Power LED on the new power-supply CRU is on and the amber Fault LED is off. v If the Power LED is off, the power-supply CRU might not be installed correct[...]

inserted into DS4100 controller slots. Unpredictable results can occur when nonmatching controller CRUs are present in the same DS4100 chassis. v After the new controller in a DS4100 single-controller storage server completes the boot process, its WWNN will change. If you have any applications that rely on the previous WWNNs, such as fibre channel [...]

Replacing a RAID controller When instructed to do so by the DS4000 Storage Manager client, replace a RAID controller that has failed. The procedure for replacing a failed controller is different for DS4100 single-controller and base (dual-controller) storage servers. If a controller in a DS4100 base storage server fails, you can replace the failed [...]

Note: Avoid operating the DS4100 with controllers of dif ferent cache sizes. Operating the storage server with controllers of different cache size will disable caching and might adversely affect performance. Always verify that you replace a failed controller with a controller that has the same FRU part number . To replace a RAID controller , perfor[...]

b. Unlock the SFP module latch: v For SFP modules that contain plastic tabs, unlock the SFP module latch by pulling the plastic tab outward 10°, as shown in Figure 72. v For SFP modules that contain wire tabs, unlock the SFP module latch by pulling the wire latch outward 90°, as shown in Figure 73. c. With the SFP latch in the unlocked position, [...]

b. Grasp the pull-rings; then pull on the levers and remove the RAID controller . 7. To access the cache battery , place the RAID controller CRU bottom up on a flat surface. Using a #2 Phillips head screwdriver , remove the six screws (three on each side of the RAID controller) that hold the RAID controller bottom panel to the sides of the RAID con[...]

10. Remove the screw that holds the battery access panel. Carefully slide the access panel toward the front to unlatch it from the three pins mounted on the RAID controller bottom panel. 11 . Remove the battery unit and place it on a dry , level surface. If this is a battery replacement, then replace the old battery with the new one and go to step [...]

v Replacement Date – Copy the expiration date from the battery label on the old controller . Note: The preferred method of checking for the battery operating age is through the storage server management software. 19. Position the battery inside the replacement RAID controller . 20. Reassemble the battery access panel to the RAID controller bottom[...]

The startup process should complete within five minutes. After the startup process completes, the new RAID controller can accept I/Os from the host server and is ready to be managed by the DS4000 Storage Manager client. Note: If the new RAID controller is not recognized by the DS4000 Storage Manager client, use the Enterprise Management Window to r[...]

Go to “Manually upgrading the firmware for the DS4100 single-controller storage server.” 28. For the DS4100 base storage server: See the DS4000 Storage Manager client online help for instructions on bringing the controller online. The RAID controller replacement procedure for the DS4100 base storage server is completed. 29. For the DS4100 singl[...]

All factory-delivered DS4100 RAID controllers are shipped with a recent version of controller firmware loaded on the RAID controller , but that firmware version might not match precisely the firmware version of the failed RAID controller you have replaced. When you replace the RAID controller in the DS4100 single-controller storage server , if the [...]

10. Use the SMW to download firmware to the new controller to match the version number shown in your storage subsystem profile. (Refer to your printout.) After 15-20 minutes, the SMW reports a download failure. Click OK to ignore the error warning and instructions and proceed with the following step. 11 . Use the EMW to remove the RAID controller f[...]

might involve clearing cache memory on the RAID controller that contains the battery that needs to be replaced. This is done by either placing the controller offline before removing it from the DS4100 server or turning of f caching in the DS4100 server , using the DS4000 Storage Manager client. Follow the steps given in the software procedure befor[...]

v Installed date – Write today’ s date v Replacement Date – Write the date three years from today’ s date Note: The preferred method of checking for the battery operating age is through the storage server management software. CONT AINS SEALED LEAD BA TTER Y . BA TTER Y MUST BE RECYCLED . Pb CAUTION: This product contains a sealed lead acid [...]

sealed lead acid batteries outside the United States, go to http:// www .ibm.com/ibm/environment/products/batteryrecycle.shtml or contact your local waste disposal facility . 20. Install the RAID controller , as shown in Figure 77. a. Slide the RAID controller all the way into the empty slot. b. Close both levers until the latch locks into place. 2[...]

Installing SFPs and fiber-optic cables Each RAID controller unit has two host ports and one expansion unit port. SFPs are inserted into the ports, and then fibre-channel cables are connected to the SFPs. For information on installing SFPs and fiber-optic cables, see “Working with SFPs and fiber-optic cables” on page 45. Adding a DS4000 EXP100 t[...]

In Figure 78, the existing DS4100 configuration already includes two DS4000 EXP100s. The dashed lines show you how to add a third DS4000 EXP100 to the configuration. To add the new DS4000 EXP100 shown in Figure 78, you would perform the following steps: 1. V erify that the existing configuration is powered on and is in optimal state. Use the DS4000[...]

Chapter 6. Hardware maintenance This chapter contains information to help you solve some of the simpler problems that you might have with your storage server . It contains the problem indicators and error messages along with suggested actions to take to resolve the problem. For instructions on how to obtain service and technical assistance for your[...]

T able 17. Symptom-to-FRU index Problem indicator Component Possible cause Possible solutions Amber LED on Drive CRU Drive failure Replace the drive that has failed. Fan CRU Fan failure Replace the fan that has failed. RAID controller Fault LED RAID controller failure If the RAID controller Fault LED is on, replace the RAID controller . If the RAID[...]

T able 17. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions All green LEDs off All CRUs Subsystem power is off Check that all storage-server power cords are plugged in and the power switches are on. If applicable, check that the main circuit breakers for the rack cabinet are turned on. AC power failure [...]

T able 17. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions One or more green LEDs off (continued) Expansion loop Drives are improperly installed or not installed Ensure that the drives are properly installed. RAID controller has no power or has failed Ensure that the unit is powered on. Replace the RAI[...]

T able 17. Symptom-to-FRU index (continued) Problem indicator Component Possible cause Possible solutions The DS4000 Storage Manager client Enterprise Management window sees each storage server controller as an independent storage subsystem or a partially managed device. Controller No drives are present in the storage server . Ensure that at least [...]

T able 18. Parts listing (T otalStorage DS4100 storage server) Index DS4100 storage server FRU P/N 1 Rail kit 01K6670 2 Fan ASM CRU 19K1293 3 Power supply CRU, 400W 19K1289 4 DS4100 S ATA RAID controller 25R0099 5 S ATA frame, midplane (base storage server) 25R021 1 5 S ATA frame, midplane (single-controller storage server) 25R0321 6 DS4100 bezel A[...]

T able 18. Parts listing (T otalStorage DS4100 storage server) (continued) Index DS4100 storage server FRU P/N Cable, FRU-1M 19K1265 Cable, FRU-5M 19K1266 Cable, FRU-25M 19K1267 Short wave SFP 19K1280 Long wave SFP 19K1281 Power cord, 2.8M 6952300 Power cord, jumper 36L8886 Battery 24P8062 Wrap plug (loopback adapter) 17P6918 LC-SC fibre channel ca[...]

11 8 IBM T otalStorage DS4100 Storage Server: Installation, User ’s, and Maintenance Guide[...]

Appendix A. Records Whenever you add options to your DS4100, be sure to update the information in this appendix. Accurate, up-to-date records make it easier to add other options and provide needed data whenever you contact your IBM technical support representative. Identification numbers Record and retain the following information. Product name: IB[...]

120 IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Appendix B. Rack mounting template This appendix provides duplicate copies of the rack mounting templates. If you want to tear out the templates from this document for easier use, use these copies rather than those provided in “Rack mounting template” on page 20. Use the following templates (Figure 80 on page 122 and Figure 81 on page 123) to i[...]

Rac k Mounting T emplate FR ONT Rail assembly () Front Right Rail assembly () Front Left Bezel () Front Left Bezel () Front Right DS4100 0 U 1 U 3 U 2 U 3 U 2 U 1 U 0 U f10ug058 Figure 80. Front rack mounting template 122 IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Rac k Mounting T emplate REAR Printed in the U.S .A. ® Rail assembly () Rear Left Rail assembly () Rear Right DS4100 3 U 2 U 1 U 0 U 3 U 2 U 1 U 0 U f10ug059 Figure 81. Rear rack mounting template Appendix B. Rack mounting template 123[...]

124 IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Appendix C. Power cords For your safety , IBM provides a power cord with a grounded attachment plug to use with this IBM product. To avoid electrical shock, always use the power cord and plug with a properly grounded outlet. IBM power cords used in the United States and Canada are listed by Underwriter’s Laboratories (UL) and certified by the Can[...]

IBM power cord part number Cord specifics Used in these countries or regions 14F0015 250V/10A 2.8M Bangladesh, Lesotho, Maceo, Maldives, Namibia, Nepal, Pakistan, Samoa, South Africa, Sri Lanka, Swaziland, Uganda 14F0033 250V/10A 2.8M Abu Dhabi, Bahrain, Botswana, Brunei Darussalam, Channel Islands, Cyprus, Dominica, Gambia, Ghana, Grenada, Guyana,[...]

Notices This publication was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or [...]

Intel is a trademark of Intel Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company , product, or service names may be trademarks or service marks of others. Important not[...]

communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits. IBM is not responsible for any radio or television [...]

This product has been tested and found to comply with the limits for Class A Information T echnology Equipment according to CISPR 22/European Standard EN 55022. The limits for Class A equipment were derived for commercial and industrial environments to provide reasonable protection against interference with licensed communication equipment. Attenti[...]

Glossary This glossary provides definitions for the terminology and abbreviations used in IBM T otalStorage DS4000 publications. If you do not find the term you are looking for , see the IBM Glossary of Computing T erms located at the following Web site: www .ibm.com/ibm/terminology This glossary also includes terms and definitions from: v Informat[...]

auto-volume transfer/auto-disk transfer (A VT/ADT). A function that provides automatic failover in case of controller failure on a storage subsystem. A VT/ADT . See auto-volume transfer/auto-disk transfer . AW T . See Abstract Windowing T oolkit . basic input/output system (BIOS). The personal computer code that controls basic hardware operations, [...]

Dynamic Host Configuration Protocol (DHCP). A protocol defined by the Internet Engineering T ask Force that is used for dynamically assigning Internet Protocol (IP) addresses to computers in a network. dynamic random access memory (DRAM). A storage in which the cells require repetitive application of control signals to retain stored data. ECC. See [...]

GBIC. See gigabit interface converter gigabit interface converter (GBIC). A transceiver that performs serial, optical-to-electrical, and electrical-to-optical signal conversions for high-speed networking. A GBIC can be hot swapped. See also small form-factor pluggable . Global Copy . Refers to a remote logical drive mirror pair that is set up using[...]

Internet Protocol (IP). A protocol that routes data through a network or interconnected networks. IP acts as an intermediary between the higher protocol layers and the physical network. Internet Protocol (IP) address. The unique 32-bit address that specifies the location of each device or workstation on the Internet. For example, 9.67.97.103 is an [...]

micro channel architecture (MCA). Hardware that is used for PS/2 Model 50 computers and above to provide better growth potential and performance characteristics when compared with the original personal computer design. Microsoft Cluster Server (MSCS). MSCS, a feature of Windows NT Server (Enterprise Edition), supports the connection of two servers [...]

random-access memory (RAM). A temporary storage location in which the central processing unit (CPU) stores and executes its processes. Contrast with DASD . RDAC. See redundant disk array controller . read-only memory (ROM). Memory in which stored data cannot be changed by the user except under special conditions. recoverable virtual shared disk (RV[...]

small computer system interface (SCSI). A standard hardware interface that enables a variety of peripheral devices to communicate with one another . small form-factor pluggable (SFP). An optical transceiver that is used to convert signals between optical fiber cables and switches. An SFP is smaller than a gigabit interface converter (GBIC). See als[...]

T ransmission Control Protocol (TCP). A communication protocol used in the Internet and in any network that follows the Internet Engineering T ask Force (IETF) standards for internetwork protocol. TCP provides a reliable host-to-host protocol between hosts in packed-switched communication networks and in interconnected systems of such networks. It [...]

140 IBM T otalStorage DS4100 Storage Server: Installation, User ’ s, and Maintenance Guide[...]

Index A about this document xvii AC power requirements 14 acoustical noise emission values 14 Activity LED 9 address label, hardware Ethernet 94 airflow 15 B battery LED 76 life of 76 recycling properly 107 replacing 76, 105 voltage of 76 bays, hot-swap drive 8 best practices 6 blank tray 9 C cabling an IBM DS4000 EXP100 49 an IBM DS4100 49 drive l[...]

Electronic emission notices (continued) European Union EMC Directive conformance statement 129 Federal Communications Commission (FCC) statement 128 Industry Canada Class A emission compliance statement 129 Japanese V oluntary Control Council for Interference (VCCI) statement 130 T aiwanese Class A warning statement 130 United Kingdom telecommunica[...]

IBM T otalStorage DS4100 (continued) site preparation 19 size of 14 speed settings, setting the 34 tray handle 9 unpacking the 19 weight 14 in-band management method 34 indicators 8 install and verify SM SW on host and workstation xix install network hardware xix install storage expansion unit xix install storage server/RAID enclosures in rack xix [...]

recycling the cache battery 107 redundant host and drive loops 37 removing components fiber-optic cable 49 GBIC 49 power supply 91 replacing a hot-swap hard disk drive 82 replacing components cache battery 105 fan 89 RAID controller 96 replacing CRUs fan unit 31 hat-swap hard disk drives 32 power supply 30 RAID controller 30 restoring power after e[...]

Readers’ Comments — W e’d Like to Hear from Yo u IBM T otalStorage DS4100 Storage Server Installation, User ’ s, and Maintenance Guide Publication No. GC26-7712-00 Overall, how satisfied are you with the information in this book? V ery Satisfied Satisfied Neutral Dissatisfied V ery Dissatisfied Overall satisfaction hhhhh How satisfied are y[...]

Readers’ Comments — We’d Like to Hear from Yo u GC26-7712-00 GC26-7712-00  Cut or Fold Along Line Cut or Fold Along Line Fold and T ape Please do not staple Fold and T ape Fold and T ape Please do not staple Fold and T ape NO POST AGE NECESSARY IF MAILED IN THE UNITED ST A TES BUSINESS REPL Y MAIL FIRST -CLASS MAIL PERMIT NO. 40 A[...]

[...]

 Part Number: 25R0314 Printed in USA GC26-7712-00 (1P) P/N: 25R0314[...]