IBM SC34-7012-01 manual

CIC S T ransa ct i o n Serv er f o r z /OS V er si on 4 R elease 1 R e co v e ry a n d R esta rt G ui d e SC34-7012-01 [...]

[...]

CIC S T ransa ct i o n Serv er f o r z /OS V er si on 4 R elease 1 R e co v e ry a n d R esta rt G ui d e SC34-7012-01 [...]

Note Before using this information and the pr oduct it supports, read the information in “Notices” on page 243. This edition applies to V ersion 4 Release 1 of CICS T ransaction Server for z/OS (product number 5655-S97) and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright IBM Corporation 1982,[...]

Contents Preface .............. v i i What this book is about .......... v i i Who should read this book ......... v i i What you need to know to understand this book vii How to use this book ........... v i i Changes in CICS T ransaction Server for z/OS, V ersion 4 Release 1 ....... i x Part 1. CICS recovery and restart concepts .............. 1 C[...]

Journal names and journal models ...... 5 8 T erminal control r esources ........ 5 8 Distributed transaction resour ces ...... 5 9 URIMAP definitions and virtual hosts .... 5 9 Chapter 6. CICS emergency restart . . 61 Recovering after a CICS failure ........ 6 1 Recovering information from the system log . . 61 Driving backout processing for in-fl[...]

Input extrapartition data sets ....... 1 3 4 Output extrapartition data sets ...... 1 3 5 Using post-initialization (PL TPI) programs . . 135 Recovery for temporary storage ....... 1 3 5 Backward r ecovery .......... 1 3 5 Forward r ecovery ........... 1 3 6 Recovery for W eb services ......... 1 3 6 Configuring CICS to support persistent messages [...]

Forward r ecovery logging ........ 2 1 5 Forward r ecovery ........... 2 1 6 Recovering VSAM spheres with AIXs .... 2 1 7 An assembler program that calls DFSMS callable services ............... 2 1 8 Chapter 19. Disaster recovery .... 2 2 3 Why have a disaster recovery plan? ...... 2 2 3 Disaster recovery testing ......... 2 2 4 Six tiers of soluti[...]

Preface What this book is about This book contains guidance about determining your CICS ® recovery and r estart needs, deciding which CICS facilities are most appr opriate, and implementing your design in a CICS region. The information in this book is generally restricted to a single CICS r egion. For information about interconnected CICS r egions[...]

viii CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Changes in CICS T ransaction Server for z/OS, V ersion 4 Release 1 For information about changes that have been made in this release, please r efer to What's New in the information center , or the following publications: v CICS T ransaction Server for z/OS What's New v CICS T ransaction Server for z/OS Upgrading from CICS TS V ersion 3.2 [...]

x CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Part 1. CICS recovery and restart concepts It is very important that a transaction processing system such as CICS can r estart and recover following a failur e. This section describes some of the basic concepts of the recovery and r estart facilities provided by CICS. © Copyright IBM Corp. 1982, 2010 1[...]

2 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 1. Recovery and restart facilities Problems that occur in a data pr ocessing system could be failures with communication protocols, data sets, pr ograms, or hardware. These pr oblems are potentially more sever e in online systems than in batch systems, because the data is processed in an unpr edictable sequence from many differ ent sources.[...]

In general, forward r ecovery is applicable to data set failures, or failures in similar data resour ces, which cause data to become unusable because it has been corrupted or because the physical storage medium has been damaged. Minimizing the effect of failures An online system should limit the effect of any failur e. Where possible, a failure tha[...]

Another way is to shut down CICS with an immediate shutdown and perform the forward r ecovery , after which a CICS emer gency restart performs the backward recovery . Recoverable resources In CICS, a recoverable r esource is any resour ce with recor ded recovery information that can be recover ed by backout. The following resour ces can be made rec[...]

v In the event of an emergency r estart, when CICS backs out all those transactions that were in-flight at the time of the CICS failur e (emergency restart backout). Although these occur in differ ent situations, CICS uses the same backout process in each case. CICS does not distinguish between dynamic backout and emergency restart backout. See Cha[...]

The recovery manager also drives: v The backout processing for any units of work that wer e in a backout-failed state at the time of the CICS failure v The commit processing for any units of work that had not finished commit processing at the time of failur e (for example, for resource definitions that wer e being installed when CICS failed) v The [...]

Forward r ecovery journal names are of the form DFHJ nn where nn is a number in the range 1–99 and is obtained from the forwar d recovery log id (FWDRECOVLOG) in the FILE resour ce definition. In this case, CICS creates a journal entry for the forwar d recovery log, which can be mapped by a JOURNALMODEL resour ce definition. Although this method [...]

2. If the failure occurs during the execution of a CICS syncpoint, wher e the conversation is with another resour ce manager (perhaps in another CICS region), CICS handles the r esynchronization. This is described in the CICS Intercommunication Guide . If the link fails and is later reestablished, CICS and its partners use the SNA set-and-test-sequ[...]

When the operator replies to IXC402D, the CICS interr egion communication program, DFHIRP , is notified and the suspended tasks are abended, and MRO connections closed. Until the reply is issued to IXC402D, an INQUIRE CONNECTION command continues to show connections to regions in the failed MVS as in service and normal. When the failed MVS image an[...]

The CICS recovery manager then uses the information r etrieved from the system log to: v Back out recoverable r esources. v Recover changes to terminal resour ce definitions. (All resource definitions installed at the time of the CICS failure ar e initially restored fr om the CICS global catalog.) A special case of CICS processing following a syste[...]

12 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 2. Resource recovery in CICS Before you begin to plan and implement r esource recovery in CICS, you should understand the concepts involved, including units of work, logging and journaling. Units of work When resour ces are being changed, there comes a point when the changes ar e complete and do not need backout if a failure occurs later . [...]

v W orking storage v Any LU6.2 sessions v Any LU6.1 links v Any MRO links The resour ces CICS retains include: v Locks on recoverable data. If the unit of work is shunted indoubt, all locks ar e retained. If it is shunted because of a commit- or backout-failur e, only the locks on the failed resour ces are retained. v System log recor ds, which inc[...]

When a lock is first acquired, it is an active lock. It r emains an active lock until successful completion of the unit of work, when it is released, or is converted into a retained lock if the unit of work fails, or for a CICS or SMSVSAM failur e: v If a unit of work fails, RLS VSAM or the CICS enqueue domain continues to hold the recor d locks th[...]

– EXEC CICS CREATE CONNECTION COMPLETE – EXEC CICS DISCARD CONNECTION – EXEC CICS DISCARD TERMINAL A UOW that does not change a recoverable resour ce has no meaningful effect for the CICS recovery mechanisms. Nonr ecoverable resources ar e never backed out. A unit of work can also be ended by backout, which causes a syncpoint in one of the fo[...]

CICS recovery manager The recovery manager ensur es the integrity and consistency of resources (such as files and databases) both within a single CICS region and distributed over interconnected systems in a network. Figure 3 on page 18 shows the resour ce managers and their resources with which the CICS recovery manager works. The main functions of[...]

v Managing the state, and controlling the execution, of each UOW v Coordinating UOW -related changes during syncpoint pr ocessing for recoverable resour ces v Coordinating UOW -related changes during r estart processing for recoverable resour ces v Coordinating r ecoverable conversations to remote nodes v T emporarily suspending completion ( shunti[...]

v Notification that the resour ce is not available, requiring temporary suspension (shunting) of the UOW v Notification that the resour ce is available, enabling retry of shunted UOWs v Notification that a connection is reestablished, and can deliver a commit or rollback (backout) decision v Syncpoint rollback v Normal termination of the UOW The id[...]

others. This can happen, for example, if two data sets are updated and the UOW has to be backed out, and the following happens: v One resour ce backs out successfully v While committing this successful backout, the commit fails v The other resour ce fails to back out These events leave one data set commit-failed, and the other backout-failed. In th[...]

Resynchronization after system or connection failure Units of work that fail while in an indoubt state remain shunted until the indoubt state can be resolved following successful r esynchronization with the coordinator . Resynchronization takes place automatically when communications ar e next established between subordinate and coor dinator . Any [...]

CICS also writes “backout-failed” recor ds to the system log if a failure occurs in backout processing of a VSAM data set during dynamic backout or emer gency restart backout. Records on the system log ar e used for cold, warm, and emergency restarts of a CICS region. The only type of start for which the system log r ecords are not used is an i[...]

v User journaling is entirely under your application pr ograms’ control. Y ou write recor ds for your own purpose using EXEC CICS WRITE JOURNALNAME commands. See “Flushing journal buffers” on page 28 for information about CICS shutdown considerations. v Automatic journaling means that CICS automatically writes recor ds to a log stream, r efer[...]

24 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 3. Shutdown and restart recovery CICS can shut down normally or abnormally and this affects the way that CICS restarts after it shuts down. CICS can stop executing as a result of: v A normal (warm) shutdown initiated by a CEMT , or EXEC CICS, PERFORM SHUT command v An immediate shutdown initiated by a CEMT , or EXEC CICS, PERFORM SHUT IMMED[...]

v The DFHCESD program started by the CICS-supplied transaction, CESD, attempts to purge and back out long-r unning tasks using increasingly stronger methods (see “The shutdown assist transaction” on page 30). v T asks that ar e automatically initiated are run—if they start before the second quiesce stage. v Any programs listed in the first pa[...]

this indicator to determine the type of startup it is to perform. See “How the state of the CICS region is r econstructed” on page 34. v CICS writes warm keypoint recor ds to: – The global catalog for terminal control and pr ofiles – The CICS system log for all other resour ces. See “W arm keypoints.” v CICS deletes all completed units [...]

Flushing journal buffers During a successful normal shutdown, CICS calls the log manager domain to flush all journal buffers, ensuring that all journal r ecords are written to their corresponding MVS system logger log str eams. During an immediate shutdown, the call to the log manager domain is bypassed and journal recor ds are not flushed. This al[...]

2. If the default shutdown assist transaction CESD is run, it allows as many tasks as possible to commit or back out cleanly , but within a shorter time than that allowed on a normal shutdown. See “The shutdown assist transaction” on page 30 for more information about CESD, which r uns the CICS-supplied program DFHCESD. 3. None of the programs [...]

The next initialization of CICS must be an emergency r estart, in order to preserve data integrity . An emer gency restart is ensur ed if the next initialization of CICS specifies ST AR T=AUTO. This is because the recovery manager ’s type-of-restart indicator is set to “emergency-r estart-needed” during initialization, and is not reset in the[...]

Y ou ar e recommended always to use the CESD shutdown-assist transaction when shutting down your CICS regions. Y ou can use the DFHCESD pr ogram “as is”, or use the supplied source code as the basis for your own customized version (CICS supplies versions in assembler , COBOL, and PL/I). For more information about the operation of the CICS-suppl[...]

- File control r ecovery blocks (only if a SHCDS NONRLSUPDA TEPERMITTED command has been used). – T ransient data queue definitions – Dump table information – Interval control elements and automatic initiate descriptors at shutdown – APPC connection information so that relevant values can be r estored during a persistent sessions restart ?[...]

If you ever need to redefine and r einitialize the CICS local catalog, you should also reinitialize the global catalog. After reinitializing both catalog data sets, you must perform an initial start. Shutdown initiated by CICS log manager The CICS log manager initiates a shutdown of the region if it encounters an err or in the system log that indic[...]

and therefor e recovery of the most recent units of work cannot be carried out. However , data might be missing from any part of the system log and CICS cannot identify what is missing. CICS cannot examine the log and determine exactly what data is missing, because the log data might appear consistent in itself even when CICS has detected that some[...]

Overriding the type of start indicator The operation of the recovery manager's contr ol record can be modified by running the r ecovery manager utility program, DFHRMUTL. About this task This can set an autostart recor d that determines the type of start CICS is to perform, effectively overriding the type of start indicator in the contr ol rec[...]

performs the recovery pr ocess for work that was in-flight when the previous run of CICS was abnormally terminated. Recovery of data during an emergency restart During the final stage of emergency r estart, the recovery manager uses the system log data to drive backout processing for any units of work that wer e in-flight at the time of the failure[...]

Y ou can do this by specifying ST AR T=INITIAL as a system initialization parameter , or by running the r ecovery manager's utility program (DFHRMUTL) to override the type of start indicator to force an initial start. See the CICS Operations and Utilities Guide for information about the DFHRMUTL utility program. Dynamic RLS restart If a CICS r[...]

Recovery with VT AM persistent sessions W ith VT AM persistent sessions support, if CICS fails or undergoes immediate shutdown (by means of a PERFORM SHUTDOWN IMMEDIATE command), VT AM holds the CICS LU-LU sessions in recovery-pending state, and they can be r ecovered during startup by a newly starting CICS region. W ith multinode persistent sessio[...]

During an emergency r estart of CICS, CICS restores those sessions pending recovery fr om the CICS global catalog and the CICS system log to an in-session state. This process of persistent sessions r ecovery takes place when CICS opens its VT AM ACB. W ith multinode persistent sessions support, if VT AM or z/OS fails, sessions are r estored when CI[...]

v If CICS determines that it cannot recover the session without unbinding and rebinding it. The result in each case is as if CICS has r estarted following a failure without VT AM persistent sessions support. In some other situations APPC sessions are unbound. For example, if a bind was in progr ess at the time of the failure, sessions are unbound. [...]

Y ou can then start further CICS r egions with or without persistent sessions support as appropriate, pr ovided that you do not exceed the limit for the number of regions that do have persistent sessions support. If you specify NOPS (no persistent session support) for the PSTYPE system initialization parameter , a zero value is r equired for the PS[...]

42 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Part 2. Recovery and restart processes Y ou can add your own pr ocessing to the CICS recovery and restart pr ocesses. This part contains the following sections: v Chapter 4, “CICS cold start,” on page 45 v Chapter 5, “CICS warm restart,” on page 53 v Chapter 6, “CICS emergency restart,” on page 61 v Chapter 7, “Automatic restart manag[...]

44 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 4. CICS cold start This section describes the CICS startup processing specific to a cold start. It covers the two forms of cold start: v “Starting CICS with the ST AR T=COLD parameter” v “Starting CICS with the ST AR T=INITIAL parameter” on page 50 Starting CICS with the ST ART=COLD parameter ST AR T=COLD performs a dual type of sta[...]

– CICS requests the SMSVSAM server , if connected, to r elease all RLS retained locks. – CICS does not rebuild the non-RLS r etained locks. v CICS requests the SMSVSAM server to clear the RLS sharing contr ol status for the region. v CICS does not restor e the dump table, which may contain entries controlling system and transaction dumps. v CIC[...]

specify on the GRPLIST system initialization parameter . The CSD file definition is built and installed from the CSD xxxx system initialization parameters. Data tables As for VSAM file definitions. BDAM File definitions are installed fr om file control table entries, specified by the FCT system initialization parameter . Attention: If you use the S[...]

T ransient data r esource definitions are installed fr om Resource groups defined in the CSD, as specified in the CSD group list (named on the GRPLIST system initialization parameter). Any extrapartition TD queues that requir e opening are opened; that is, any that specify OPEN(INITIAL). All the newly-installed TD queue definitions are written to t[...]

If you define new resour ce definitions and install them dynamically , ensur e the group containing the r esources is added to the appropriate gr oup list. Monitoring and statistics The initial status of CICS monitoring is determined by the monitoring system initialization parameters (MN and MN xxxx ). The initial recor ding status for CICS statist[...]

Installable set install The following VT AM terminal control r esources are committed in installable sets: v Connections and their associated sessions v Pipeline terminals—all the terminal definitions sharing the same POOL name If one definition in an installable set fails, the set fails. However , each installable set is treated independently wi[...]

information saved in the system log from a pr evious run. The primary and secondary system log streams ar e purged and CICS begins writing a new system log. v Because CICS is starting a new catalog, it uses a new logname token in the “exchange lognames” process when connecting to partner systems. Thus, r emote systems are notified that CICS has[...]

52 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 5. CICS warm restart This section describes the CICS startup processing specific to a warm r estart. If you specify ST AR T=AUTO, which is the recommended method, CICS determines which type of start to perform using information retrieved fr om the recovery manager's contr ol record in the global catalog. If the type-of-r estart indicat[...]

Files File control information fr om the previous run is r ecovered from information recor ded in the CICS catalog only . File resour ce definitions for VSAM and BDAM files, data tables, and LSR pools are installed from the global catalog, including any definitions that wer e added dynamically during the previous r un. The information recovered and[...]

T emporary storage Auxiliary temporary storage queue information (for both recoverable and non-recoverable queues) is r etrieved from the warm keypoint. Note that TS READ pointers are r ecovered on a warm restart (which is not the case on an emer gency restart). CICS opens the auxiliary temporary storage data set for update. T emporary storage data[...]

v All intrapartition TD queues are initialized empty . v The queue resour ce definitions are installed from the global catalog, but they ar e not updated by any log recor ds or keypoint data. They are always installed enabled. This option is intended for use when initiating remote site r ecovery (see Chapter 6, “CICS emergency r estart,” on pag[...]

Autoinstall for programs If program autoinstall is enabled (PGAIPGM=ACTIVE), pr ogram, mapset, and partitionset resour ce definitions are installed from the CSD only if they wer e cataloged; otherwise they are installed at first r eference by the autoinstall process. All definitions installed from the CSD ar e updated with information from the warm[...]

Journal names and journal models The CICS log manager restor es the journal name and journal model definitions from the global catalog. Journal name entries contain the names of the log str eams used in the previous r un, and the log manager reconnects to these during the warm restart. T erminal control resources T erminal contr ol information is i[...]

v Different TCT from last run . CICS installs the TCT only , and does not apply the warm keypoint information, effectively making this a cold start for these devices. Note: CICS TS for z/OS, V ersion 4.1 supports only remote TCAM terminals—that is, the only TCAM terminals you can define are those attached to a r emote, pre-CICS TS 3.1, terminal-o[...]

60 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 6. CICS emergency restart This section describes the CICS startup processing specific to an emer gency restart. If you specify ST AR T=AUTO, CICS determines what type of start to perform using information retrieved fr om the recovery manager ’s control r ecord in the global catalog. If the type-of-restart indicator in the contr ol record [...]

Any non-RLS locks associated with in-flight (and other failed) transactions are acquired as active locks for the tasks attached to perform the backouts. This means that, if any new transaction attempts to access non-RLS data that is locked by a backout task, it waits normally rather than receiving the LOCKED condition. Retained RLS locks are held b[...]

Reconnecting to SMSVSAM for RLS access As on a warm restart, CICS connects to the SMSVSAM server . In addition to notifying CICS about lost locks, VSAM also informs CICS of the units of work belonging to the CICS region for which it holds r etained locks. See “Lost locks recovery” on page 89 for information about the lost locks recovery pr oces[...]

Start requests In general, start requests ar e recovered only when they ar e associated with recoverable data or ar e protected and the issuing unit of work is indoubt. However , recovery can be further limited by the use of the specific COLD option on the system initialization parameter for TS, ICP , or BMS. If you suppress start requests by means[...]

is successful, but CICS abnormally terminates before the catalog can be updated, CICS recovers the information fr om the forward recovery r ecords on the system log. If the installation or deletion of installable sets or individual resour ces is unsuccessful, or has not reached commit point when CICS abnormally terminates, CICS does not recover the[...]

66 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 7. Automatic restart management CICS uses the automatic restart manager (ARM) component of MVS to incr ease the availability of your systems. MVS automatic restart management is a sysplex-wide integrated automatic r estart mechanism that performs the following tasks: v Restarts an MVS subsystem in place if it abends (or if a monitor program[...]

If CICS is restarted by ARM with the same persistent JCL, CICS forces ST AR T=AUTO to ensure data integrity . Registering with ARM T o r egister with ARM, you must implement automatic r estart management on the MVS images that the CICS workload is to run on. Y ou must also ensure that the CICS startup JCL used to restart a CICS r egion is suitable [...]

CANCEL, CICS de-registers fr om ARM befor e terminating, because if CICS remained r egistered, an automatic restart would pr obably encounter the same error condition. For other error situations, CICS does not de-r egister , and automatic restarts follow . T o contr ol the number of restarts, specify in your ARM policy the number of times ARM is to[...]

CICS ST ART options Y ou ar e recommended to specify ST AR T=AUTO, which causes a warm start after a normal shutdown and an emergency r estart after failure. Y ou ar e also recommended always to use the same JCL, even if it specifies ST AR T=COLD or ST AR T=INITIAL, to ensure that CICS r estarts correctly when restarted by the MVS automatic r estar[...]

The COVR transaction T o ensur e that CICS reconnects to VT AM in the event of a VT AM abend, CICS keeps retrying the OPEN VT AM ACB using a time-delay mechanism via the non-terminal transaction COVR. After CICS has completed clean-up following the VT AM failure, it invokes the CICS open VT AM retry (COVR) transaction. The COVR transaction invokes [...]

Y ou can also r estart a server explicitly using either the server command CANCEL RESTART=YES , or the MVS command CANCEL jobname ,ARMRESTART By default, the server uses an ARM element type of SYSCICSS, and an ARM element identifier of the form DFH xxnn _ poolname where xx is the server type (XQ, CF or NC) and nn is the one- or two-character &S[...]

Chapter 8. Unit of work recovery and abend processing A number of different events can cause the abnormal termination of transactions in CICS. These events include: v A transaction ABEND r equest issued by a CICS management module. v A program check or operating system abend (this is trapped by CICS and converted into an ASRA or ASRB transaction ab[...]

See “Commit-failed recovery” on page 83. Backout-failed A unit of work fails while backing out updates to file control recoverable resour ces. (The concept of backout-failed applies in principle to any resour ce that performs backout recovery , but CICS file control is the only resour ce manager to provide backout failure support.) A partial co[...]

terminating transaction takes place immediately . Therefore, it does not cause any active locks to be converted into retained locks. In the case of a CICS r egion abend, in-flight tasks have to wait to be backed out when CICS is restarted, during which time the locks are r etained to protect uncommitted resour ces. T o r estore the resour ces to th[...]

Intrapartition transient data Intrapartition destinations specified as logically recoverable are r estored by transaction backout. Read and write pointers are r estored to what they were befor e the transaction failure occurr ed. Physically recoverable queues are r ecovered on warm and emergency r estarts. T ransient data does not pr ovide any supp[...]

intended for the started task, but does not back out the ST AR T request itself. Thus the new task will start at its specified time, but the data will not be available to the started task, to which CICS will return a NOTFND condition in response to the RETRIEVE command. START with recoverable data (PROTECT) T ransaction backout of the task issuing [...]

T able 1. Effect of REST ART option on started transactions (continued) Description of non-terminal ST AR T command Events Effect of REST AR T(YES) Effect of REST AR T(NO) Specifies nonrecoverable data Started task abends without retrieving its data T ransaction is restarted with its data still available, up to n ¹ times. ST ART r equest and its d[...]

Backout-failed recovery Backout failure support is curr ently provided only by CICS file control. If backout to a VSAM data set fails for any reason, CICS performs the following processing: v Invokes the backout failure global user exit pr ogram at XFCBF AIL, if this exit is enabled. If the user exit program chooses to bypass backout failur e proce[...]

T ransient data All updates to logically recoverable intrapartition queues ar e managed in main storage until syncpoint, or until a buffer must be flushed because all buf fers are in use. TD always commits forwards; ther efore, TD can never suffer a backout failur e on DFHINTRA. Retrying backout-failed units of work Backout retry for a backout-fail[...]

It might be worth initially deciding to leave a data set online for some time after a backout failure, to evaluate the level of impact the failur es have on users. T o r ecover from a media failure, r e-create the data set by applying forward recovery logs to the latest backup. The steps you take depend on whether the data set is opened in RLS or n[...]

and issues a console message. If the failure has alr eady been detected by some other (earlier) request, CICS has alr eady started to close the SMSVSAM control ACB when the backout request fails. The backout is normally retried automatically when the SMSVSAM server becomes available. (See “Dynamic RLS restart” on page 37.) Ther e is no need to [...]

This situation can be resolved only by deleting the rival r ecord with the duplicate key value. Lock structure full error The backout requir ed VSAM to acquire a lock for internal processing, but it was unable to do so because the RLS lock structur e was full. This error can occur only for VSAM data sets opened in RLS access mode. T o r esolve the [...]

distinguishes between a commit failure wher e recoverable work was performed, and one for which only repeatable r ead locks were held. Indoubt failure recovery The CICS recovery manager is r esponsible for maintaining the state of each unit of work in a CICS region. For example, typical events that cause a change in the state of a unit of work are [...]

reads against VSAM data sets and has made no updates to other r esources, it is safe to force the unit of work using the SET DSNAME or SET UOW commands. CICS saves enough information about the unit of work to allow it to be either committed or backed out when the indoubt unit of work is unshunted when the coordinator pr ovides the resolution (or wh[...]

T o r etrieve information about a unit of work (UOW), you can use either the CEMT , or EXEC CICS, INQUIRE UOW command. For the purposes of this illustration, the CEMT method is used. Y ou can filter the command to show only UOWs that are associated with a particular transaction. For example, Figure 4 shows one UOW (AC0CD65E5D990800) associated with[...]

When a UOW has been shunted indoubt, CICS retains locks on the r ecoverable resour ces that the UOW has updated. This prevents further tasks from changing the resour ce updates while they are indoubt. T o display CICS locks held by a UOW that has been shunted indoubt, use the CEMT INQUIRE UOWENQ command. Y ou can filter the command to show only loc[...]

Recovery from failures associated with the coupling facility This topic deals with recovery fr om failures arising from the use of the coupling facility , and which affect CICS units of work. It covers: v SMSVSAM cache structur e failures v SMSVSAM lock structur e failures (lost locks) v Connection failure to a coupling facility cache str ucture v [...]

CICS recovers after a cache failur e automatically . Ther e is no need for manual intervention (other than the prer equisite action of resolving the underlying cause of the cache failure). Lost locks recovery The failure of a coupling facility lock str ucture that cannot be rebuilt by VSAM creates the lost locks condition. The lost locks condition [...]

region that was not sharing the data set at the time the lost locks condition occurred, and on RLS access r equests issued by any new units of work in CICS regions that wer e sharing the data set. Performing lost locks recovery for failed units of work Lost locks recovery r equires that any units of work that had been updating the data set at the t[...]

simultaneously all data sets in use when the lock structur e fails, each data set can be restor ed to service individually as soon as all its sharing CICS regions have completed lost locks recovery . Connection failure to a coupling facility cache structure If connection to a coupling facility cache structur e is lost, DFSMS ™ attempts to rebuild[...]

Recovery from the failur e of a sysplex is just the equivalent of multiple MVS failure r ecoveries. T ransaction abend processing If, during transaction abend processing, another abend occurs and CICS continues, there is a risk of a transaction abend loop and further pr ocessing of a resource that has lost integrity , because of uncompleted recover[...]

The exit code then executes as an extension of the abending task, and runs at the same level as the program that issued the HANDLE ABEND command that activated the exit. After any program-level abend exit code has been executed, the next action depends on how the exit code ends: v If the exit code ends with an ABEND command, CICS gives control to t[...]

1. CICS invokes DFHREST only when REST AR T(YES) is specified in a transaction’s resour ce definition. 2. Ensure that r esources used by restartable transactions, such as files, temporary storage, and intrapartition transient data queues, are defined as r ecoverable. 3. When transaction restart occurs, a new task is attached that invokes the init[...]

v CICS remains operational, but the task curr ently in control terminates. v CICS terminates (see “Shutdown requested by the operating system” on page 29). If a program check occurs when a user task is pr ocessing, the task abends with an abend code of ASRA. If a program check occurs when a CICS system task is processing, CICS terminates. If an[...]

96 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 9. Communication error processing The types of communication error that can occur include terminal err or processing and intersystem communication failures. T erminal error processing There ar e two main CICS programs that participate in terminal error pr ocessing. These are the node err or program, DFHZNEP , and the terminal error program,[...]

The TEP is enter ed once for each terminal error , and therefor e should be designed to process only one err or for each invocation. Intersystem communication failures An intersystem communication failure can be caused by the failur e of a CICS region, or the r emote system to which it is connected. A network failure can also cause the loss of the [...]

Part 3. Implementing recovery and restart This part describes the way you implement recovery and r estart for CICS regions. © Copyright IBM Corp. 1982, 2010 99[...]

100 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 10. Planning aspects of recovery When you are planning aspects of r ecovery , you must consider your applications, system definitions, internal documentation, and test plans. Application design considerations Think about recoverability as early as possible during the application design stages. This topic covers a number of aspects of design[...]

Question 5: If a data set becomes unusable, should all applications be terminated while recovery is performed? If degraded service to any application must be pr eserved while recovery of the data set takes place, you will need to include pr ocedures to do this. Question 6: Which of the files to be updated are to be r egarded as vital? Identify any [...]

Before any design or pr ogramming work begins, all interested parties should agree on the statement—including: v Those responsible for business management v Those responsible for data management v Those who are to use the application—including the end users, and those responsible for computer and online system operation Designing the end user ?[...]

v If a user ’s printer becomes unusable (because of hardwar e or communication problems), consider the use of alternatives, such as the computer center ’s printer , as a standby . Security Decide the security procedur es for an emergency restart or a br eak in communications. For example, when confidential data is at risk, specify that the user[...]

and general log data to log streams defined to the MVS system logger . For more information, see Chapter 1 1, “Defining system and general log streams,” on page 107. Files For VSAM files defined to be accessed in RLS mode, define the recovery attributes in the ICF catalog, using IDCAMS. For VSAM files defined to be accessed in non-RLS mode, you[...]

normal conditions. They should, nevertheless, be tested as far as possible, to ensure that they handle the functions for which they are designed. CICS facilities, such as the execution diagnostic facility (CEDF) and command interpreter (CECI), can help to cr eate exception conditions and to interpret program and system r eactions to those condition[...]

Chapter 1 1. Defining system and general log streams All CICS system logging and journaling is controlled by the CICS log manager , which uses MVS system logger log streams to stor e its output. About this task CICS logging and journaling can be divided into four broad types of activity: System logging CICS maintains a system log to support transac[...]

System log streams These are used by the CICS log manager and the CICS r ecovery manager exclusively for unit of work recovery purposes. Each system log is unique to a CICS region, and must not be mer ged with any other system log. General log streams These are used by the CICS log manager for all other types of logging and journaling. Y ou can mer[...]

CICS log manager connects to its log stream automatically during system initialization, unless it is defined as TYPE(DUMMY) in a CICS JOURNALMODEL resour ce definition. Although the CICS system log is logically a single logical log stream, it is written to two physical log streams—a primary and a secondary . In general, it is not necessary to dis[...]

Model log streams for CICS system logs If CICS fails to connect to its system log streams because they have not been defined, CICS attempts to have them created dynamically using model log str eams. T o cr eate a log stream dynamically , CICS must specify to the MVS system logger all the log stream attributes needed for a new log str eam. T o deter[...]

However , using model log streams defined with the CICS default name ar e always assigned to the same structur e within an MVS image. This may not give you the best allocation in terms of recovery considerations if you ar e using structures defined across two or mor e coupling facilities. For example, consider a two-way sysplex that uses two coupli[...]

V arying the model log stream name: T o balance log str eams across log structures, using model log str eams means customizing the model log stream names. Y ou cannot achieve the distribution of log streams shown in this scenario using the CICS default model name. About this task Y ou can use an XLGSTRM global user exit pr ogram to vary , in a numb[...]

work. W ith this information, CICS continues reading backwards, but this time reading only the r ecords for units of work that are identified in the activity keypoint. Reading continues until CICS has read all the r ecords for the units of work identified by the activity keypoint. This process means that completed units of work, including shunted b[...]

v If a system log stream exceeds the primary storage space allocated, it spills onto secondary storage. (For a definition of primary and secondary storage, see the CICS T ransaction Server for z/OS Installation Guide .) The resulting I/O can adversely affect system performance. v If the interval between activity keypoints is long, the volume of dat[...]

Writing user-recovery data About this task Y ou should write only r ecovery-related recor ds to the system log stream. Y ou can do this using the commands provided by the application pr ogramming interface (API) or the exit programming interfaces (XPI). This is important because user recovery r ecords are pr esented to a global user exit program en[...]

About this task The dddd value specifies the minimum number of days for which data is to be retained on the log. Y ou ar e strongly recommended not to use the system log for r ecords that need to be kept. Any log and journal data that needs to be preserved should be written to a general log stream. See the CICS System Definition Guide for advice on[...]

2. Define a general log stream for forwar d recovery data. If you do not define a general log stream, CICS attempts to cr eate a log stream dynamically . See “Model log streams for CICS general logs” for details. 3. Decide how you want to merge forwar d recovery data from dif ferent CICS regions into one or mor e log streams. See “Merging dat[...]

Merging data on shared general log streams Unlike system log streams, which ar e unique to one CICS region, general log streams can be shar ed between many CICS regions. This means that you can merge forwar d recovery data from a number of CICS r egions onto the same forward r ecovery log stream. About this task v Y ou can use the same forwar d rec[...]

About this task The CICS-supplied group, DFHLGMOD, includes a JOURNALMODEL for the log of logs, called DFHLGLOG, which has a log stream name of &USERID..CICSVR.DFHLGLOG. Note that &USERID resolves to the CICS r egion userid, and if your CICS regions r un under different RACF user IDs, the DFHLGLOG definition resolves to a unique log of logs[...]

v In a format compatible with utility programs written for versions of CICS that use the log manager for logging and journaling. See the CICS Operations and Utilities Guide for more information about using the LOGR SSI to access log stream data, and for sample JCL. If you plan to write your own utility program to r ead log stream data, see the CICS[...]

Operating a recovery pr ocess that is independent of time-stamps in the system log data ensures that CICS can r estart successfully after an abnormal termination, even if the failure occurs shortly after local time has been put back. Offline utility program, DFHJUP Changing the local time forward has no ef fect on the processing of system log strea[...]

122 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 12. Defining recoverability for CICS-managed resources This section describes what to do to ensure that you can r ecover the resources controlled by CICS on behalf of your application pr ograms. About this task It covers the recoverability aspects of the various r esources as follows: v “Recovery for transactions” v “Recovery for file[...]

SPURGE({NO|YES}) This option indicates whether the transaction is initially system-purgeable; that is, whether CICS can purge the transaction as a r esult of: v Expiry of a deadlock timeout (DTIMOUT) delay interval v A CEMT , or EXEC CICS, SET T ASK(id) PURGE|FORCEPURGE command. The default is SPURGE(NO). TPURGE({NO|YES}) This option indicates whet[...]

Recovery for files A CICS file is a logical view of a physical data set, defined to CICS in a file resour ce definition with an 8-character file name. A CICS file is associated with a VSAM or BDAM data set by one of the following: v By dynamic allocation, where the data set name is pr edefined on the DSNAME parameter in the file definition (or set [...]

Forward recovery For VSAM files, you can use a forward r ecovery utility , such as CICSVR, when online backout processing has failed as a r esult of some physical damage to the data set. For forward r ecovery: v Create backup copies of data sets. v Record after -images of file changes in a forward recovery str eam. CICS does this for you automatica[...]

uses the ICF catalog entry recovery attributes instead of the FILE r esource. T o force CICS to use the FILE r esource attributes instead of the catalog, set the NONRLSRECOV system initialization parameter to FILEDEF . v Y ou define the r ecovery attributes for BDAM files in file entries in the file control table (FCT). VSAM files accessed in non-R[...]

VSAM files accessed in RLS mode If you specify file definitions that open a data set in RLS mode, specify the recovery options in the ICF catalog. The recovery options on the CICS file r esource definitions (RECOVER Y , FWDRECOVLOG, and BACKUPTYPE) are ignor ed if the file definition specifies RLS access. The VSAM parameters LOG and LOGSTREAMID, on[...]

INQUIRE DSNAME command returns values fr om the VSAM base cluster block (BCB). However , because base cluster block (BCB) recovery values ar e not set until the first open, if you issue an INQUIRE DSNAME command before the first file is opened, CICS returns NOT APPLIC for RECOVST A TUS. BDAM files Y ou can specify CICS support for backwar d recover[...]

About this task If you use XFCNREC to suppress open failur es that are a result of inconsistencies in the backout settings, CICS issues a message to warn you that the integrity of the data set can no longer be guaranteed. Any INQUIRE DSNAME RECOVSTATUS command that is issued from this point onwar d will return NOTRECOV ABLE, regardless of the r eco[...]

- File is defined with RECOVER Y(ALL): the open fails. – Base cluster has RECOVER Y(ALL): - File is defined with RECOVER Y(NONE): the open fails. - File is defined with RECOVER Y(BACKOUTONL Y): the open fails. - File is defined with RECOVER Y(ALL): the open proceeds unless FWDRECOVLOG specifies a differ ent journal id from the base cluster , in w[...]

For more information about allocation and space r equirements, see the CICS System Definition Guide .) For extrapartition transient data considerations, see “Recovery for extrapartition transient data” on page 134. Y ou must specify the name of every intrapartition transient data queue that you want to be recoverable in the queue definition. Th[...]

Making intrapartition TD physically recoverable can be useful in the case of some CICS queues. For example, after a CICS failure, you might choose to r estart CICS as quickly as possible, and then look for the cause of the failure. By specifying queues such as CSMT as intrapartition and physically recoverable, the messages produced just befor e the[...]

Recovery for extrapartition transient data CICS does not recover extrapartition data sets. If you depend on extrapartition data, you will need to develop procedur es to recover data for continued execution on restart following either a contr olled or an uncontrolled shutdown of CICS. There ar e two areas to consider in recovering extrapartition dat[...]

Output extrapartition data sets The recovery of output extrapartition data sets is somewhat dif ferent from the recovery of input data sets. For a tape output data set, use a new output tape on restart. Y ou can then use the previous output tape if you need to r ecover information recorded befor e termination. T o avoid losing data in tape output b[...]

Define temporary storage queues as recoverable using temporary storage model resour ce definitions as shown in the following example define statements: CEDA DEFINE DESCRIPTION(Recoverable TS queues for START requests) TSMODEL(RECOV1) GROUP(TSRECOV) PREFIX(DF) LOCATION(AUXILIARY) RECOVERY(YES) CEDA DEFINE DESCRIPTION(Recoverable TS queues for BMS) T[...]

About this task CICS uses Business T ransaction Services (BTS) to ensur e that persistent messages are r ecovered in the event of a CICS system failure. For this to work corr ectly , follows these steps: Procedure 1. Use IDCAMS to define the local request queue and r epository file to MVS. Y ou must specify a suitable value for STRINGS for the file[...]

2. For each local request queue, define a QLOCAL object. Use the following command: DEFINE QLOCAL(' queuename ') DESCR(' description ') PROCESS( processname ) INITQ(' initiation_queue ') TRIGGER TRIGTYPE(FIRST) TRIGDATA(' default_target_service ') BOTHRESH( nnn ) BOQNAME(' requeuename ') where: v qu[...]

not usable, message DFHPI01 17 is issued, and CICS continues without BTS, using the existing channel-based container mechanism. If a CICS failure occurs befor e the W eb service starts or completes processing, BTS recovery ensur es that the process is rescheduled when CICS is r estarted. If the W eb service abends and backs out, the BTS pr ocess is[...]

140 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 13. Programming for recovery When you are designing your application pr ograms, you can include recovery facilities that are pr ovided by CICS; for example, you can use global user exits for backout recovery . This section covers the following topics: v “Designing applications for recovery” v “Program design” on page 143 v “Managi[...]

v Progr ess transaction , to check on progress thr ough the application. Such a function could be used after a transaction failure or after an emer gency restart, as well as at any time during normal operation. For example, it could be designed to find the correct r estart point at which the terminal user should recommence the interr upted work. Th[...]

SAA-compatible applications The resour ce recovery element of the Systems Application Architectur e ® (SAA) common programming interface (CPI) pr ovides an alternative to the standard CICS application program interface (API) if you need to implement SAA-compatible applications. The resour ce recovery facilities provided by the CICS implementation [...]

committed in one unit of work, but the transaction is to continue with one or more units of work for further pr ocessing. 3. Where file or database updates must be kept in step, make sur e that your application does them in the same unit of work. This approach ensur es that those updates will all be committed together or , in the event of the unit [...]

back out only the updates made during that individual step; the application is responsible for r estarting at the appropriate point in the conversation, which might involve recr eating a screen format. However , other tasks might try to update the database between the time when update information is accepted and the time when it is applied to the d[...]

v Data tables (user-maintained) v Coupling facility data tables CICS can return all these r esources to their status at the beginning of an in-flight unit of work if a task ends abnormally . T emporary storage (auxiliary) Y ou can use a temporary storage item to communicate between transactions. (For this purpose, the temporary storage item needs t[...]

Procedure v Arrange for all transactions to access files in a sequence agreed in advance. This could be a suitable subject for installation standards. Be extra car eful if you allow updates through multiple paths. v Enforce explicit installation enqueueing standar ds so that all applications do the following: 1. Enqueue by the same character string[...]

The abend processing should analyze the cause of failur e as far as possible, and restart the task if appr opriate. Ensure that either the user or the master terminal operator can take appropriate action to r epeat the updates. Y ou could, for example, allow the user to reinitiate the task. An alternative solution is for the started transaction to [...]

About this task Such queuing can be done on a transient data queue associated with a terminal. A special transaction, triggered when the terminal is available, can then format and present the data. For recovery and r estart purposes: v The transient data queue should be specified as logically recoverable. v If the transaction that presents the data[...]

For example, if file input and output errors occur (wher e the default action is merely to abend the task), you might want to inform the master terminal operator , who can decide to terminate CICS, especially if one of the files is critical to the application. Y our installation might have standar ds relating to the use of RESP options or HANDLE CO[...]

v DTB takes place only after program level abend exits (if any) have attempted cleanup or logical recovery . T ransaction restart after DTB For each transaction where DTB is specified, consider also specifying automatic transaction restart. For example, for transactions that access DL/I databases (and are subject to pr ogram isolation deadlock), au[...]

v Send a message to the terminal operator if, for example, you believe that the abend is due to bad input data. Information that is available to a program-level exit r outine or program includes the following: Command Information provided ADDRESS TW A The address of the TW A ASSIGN ABCODE The curr ent CICS abend code ASSIGN ABPROGRAM The name of th[...]

ST ART TRANSID commands In a transaction that uses the START TRANSID command to start other transactions, you must maintain logical data integrity . Y ou can maintain data integrity by following these guidelines: 1. Always use the PROTECT option of the ST AR T TRANSID command. This ensures that if the ST AR T -issuing task is backed out, the new ta[...]

About this task There ar e two forms of locking: 1. The implicit locking functions performed by CICS (or the access method) whenever your transactions issue a request to change data. These ar e described under: v “Implicit locking for files” v “Implicit enqueuing on logically recoverable TD destinations” on page 157 v “Implicit enqueuing [...]

Recoverable files For VSAM or BDAM files designated as recoverable, the duration of the locking action is extended. For VSAM files, the extended locking is on the updated recor d only , not the whole control interval. READ WRITE UPDATE ====== Locking ===== during update (See Note below) Task A SOT READ WRITE SP UPDATE ===Wait=== ===== Locking ==== [...]

The extended period of locking is needed to avoid an update committed by one task being backed out by another . (Consider what could happen if the nonextended locking action shown in Figure 13 on page 155 was used when updating a recoverable file. If task A abends just after task B has reached syncpoint and has thus committed its changes, the subse[...]

The backout fails because a duplicate key is detected in the AIX indicated by message DFHFC4701, with a failure code of X'F0'. Ther e is no locking on the AIX ® key to prevent the second task taking the key befor e the end of the first task’s unit of work. If there is an application r equirement for this sort of operation, you should u[...]

enqueuing on temporary storage queues where concurr ently executing tasks can read and change queue(s) with the same temporary storage identifier . (See “Explicit enqueuing (by the application programmer).”) T emporary storage contr ol commands that invoke implicit enqueuing are: v WRITEQ TS v DELETEQ TS Implicit enqueuing on DL/I databases wit[...]

After a task has issued an ENQ RESOURCE( data-area ) command, any other task that issues an ENQ RESOURCE command with the same data-area parameter is suspended until the task issues a matching DEQ RESOURCE( data-area ) command, or until the unit of work ends. Note: Enqueueing on more than one r esource concurrently might cr eate a deadlock between [...]

v If both deadlocked resour ces are CICS resour ces (but not both VSAM resour ces), or one is CICS and the other DL/I, CICS abends the task whose DTIMOUT period elapses first. It is possible for both tasks to time out simultaneously . If neither task has a DTIMOUT period specified, they both remain suspended indefinitely , unless one of them is can[...]

Procedure v Enable them in PL T programs in the first part of PL T processing. v Specify them on the system initialization parameter , TBEXITS . This takes the form TBEXITS=( name1,name2,name3,name4,name5,name6 ), where name1, name2, name3, name4, name5 , and name6 are the names of your global user exit pr ograms for the XRCINIT , XRCINPT , XFCBF A[...]

XFCLDEL global user exit XFCLDEL is invoked when backing out a unit of work that performed a write operation to a VSAM ESDS, or a BDAM data set. XFCBOVER global user exit XFCBOVER is invoked whenever CICS is about to decide not to backout an uncommitted update, because the recor d could have been updated by a non-RLS batch program. This situation c[...]

Chapter 14. Using a program error program (PEP) The program err or program (PEP) gains control after all pr ogram-level ABEND exit code has executed and after dynamic transaction backout has been performed. About this task There is only one pr ogram error program for the whole r egion. Procedure 1. Decide whether you want to use the CICS-supplied p[...]

7. The CICS transaction failure pr ogram, DFHTFP , links to DFHPEP befor e transaction backout is performed. This means resour ces used by the abending transaction may not have been released. DFHPEP needs to be aware of this, and might need logic to handle resour ces that are still locked. 8. Do not use the restart function for distributed transact[...]

When you have corrected the err or , you can re-enable the relevant installed transaction definition to allow terminals to use it. Y ou can also disable transaction identifiers when transactions are not to be accepted for application-dependent reasons, and can enable them again later . The CICS Resour ce Definition Guide tells you more about the ma[...]

166 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 15. Resolving retained locks on recoverable resources This section describes how you can locate and resolve r etained locks that are preventing access to r esources, either by CICS transactions or by batch jobs. About this task Although the main emphasis in this section is on how you can switch from RLS to non-RLS access mode in preparation[...]

The RLS quiesce and unquiesce functions The RLS quiesce and unquiesce functions are initiated by a CICS command in one region, and pr opagated by the VSAM RLS quiesce interface to other CICS regions in the sysplex. When these functions are complete, the ICF catalog shows the quiesce state of the target data set. The state of the quiesce flag in the[...]

Note: 1. A suitably-authorized user application program (AOR1 in the diagram) issues an EXEC CICS SET DSNAME(...) QUIESCED command (or a terminal operator issues the equivalent CEMT command). If the command specifies the BUSY(NOW AIT) option, all phases of the quiesce operation are asynchr onous, and the user application program continues processin[...]

(4a) SMSVSAM uses the coupling facility to propagate the r equest to the other SMSVSAM servers in the sysplex. 5. The CICS RLS quiesce exit program schedules a CICS r egion task (CFQR) to perform asynchronously the r equired quiesce actions in that CICS region. 6. When CICS has closed all open RLS ACBs for the data set, CICS issues the “quiesce c[...]

W ith the new RLS quiesce mechanism, you do not have to close a data set to take a non-BWO backup. However , because this causes new transactions to be abended, you may prefer to quiesce your data sets befor e taking a non-BWO backup. Non-BWO data set backup end A quiesce interface function initiated by DFSMSdss at the end of non-BWO backup process[...]

Lost locks recovery complete A quiesce interface function initiated by VSAM. VSAM takes action associated with a sphere having completed lost locks r ecovery on all CICS regions that wer e sharing the data set. SMSVSAM invokes the CICS RLS quiesce exit program in each r egion that is register ed with an SMSVSAM control ACB. Until lost locks recover[...]

Note: If your file definitions specify an LSR pool id that is built dynamically by CICS, consider using the RLSTOLSR system initialization parameter . v Open the files non-RLS read-only mode in CICS. v Concurrently , run batch non-RLS. v When batch work is finished: – Close the read-only non-RLS mode files in CICS. – Re-define the files as RLS [...]

The remainder of this topic on switching to non-RLS access mode describes the options that are available if you need to switch to non-RLS mode and ar e prevented fr om doing so by retained locks. Resolving retained locks before opening data sets in non-RLS mode VSAM sets an ‘RLS-in-use’ indicator in the ICF catalog cluster entry when a data set[...]

About this task However , it does know about the uncommitted changes that are pr otected by such locks, and why the changes have not yet been committed successfully . CICS uses this information to help you resolve any r etained locks that are preventing you from switching to non-RLS access mode. INQUIRE DSNAME Y ou can use the INQUIRE DSNAME comman[...]

v Commit failure , where a unit of work has failed during the commit action. The commit action may be either to commit the changes made by a completed unit of work, or to commit the successful backout of a unit of work. This failure is caused by a failure of the SMSVSAM server , which is r eturned as RLSSER VER on the CAUSE option and COMMITF AIL o[...]

4. If a unit of work has been shunted with a differ ent CAUSE and REASON, review the descriptions of these values in the INQUIRE UOWDSNFAIL command to determine what action to take to allow the shunted unit of work to complete. Choosing data availability over data integrity There may be times when you cannot r esolve all the retained locks correctl[...]

Diagnostic messages DFHFC3003 and DFHFC3010 are issued for each log r ecord. If a data set has both indoubt-failed and other (backout- or commit-) failed units of work, deal with the indoubt UOWs first, using SET DSNAME UOW ACTION, because this might result in other failur es which can then be cleared by the SET DSNAME RESETLOCKS command. The batch[...]

CEMT INQUIRE UOWDSNFAIL DSN(’RLS.ACCOUNTS.ESDS.DBASE1’) STATUS: RESULTS Dsn(RLS.ACCOUNTS.ESDS.DBASE1 ) Dat Del Uow(AA6DB080C40CEE01) Rls Dsn(RLS.ACCOUNTS.ESDS.DBASE1 ) Dat Ind Uow(AA6DB08AC66B4000) Rls The display shows a REASON code of DELEXITERROR (Del) for one unit of work, and INDEXRECFULL (Ind) for the other . 2. A CEMT SET DSNAME(...) RET[...]

X’AA6DB08AC66B4000’ and file ACCNT1 . Update was a write-add made by transaction WKLY at terminal T583 under task number 00027. Key length 4, data length 7, base ESDS RBA X’00000DDF’, record key X’00000DDF’ A special case: lost locks If a lost locks condition occurs, any affected data set r emains in a lost locks state until all CICS re[...]

v Do not use DENYNONRLSUPDA TE if you run non-RLS work after specifying PERMITNONRLSUPDA TE. The permit status is automatically reset by the CICS regions that hold r etained locks when they open the data set in RLS mode. Post-batch processing After a non-RLS program has been permitted to override r etained locks, the uncommitted changes that were p[...]

Coupling facility data table retained locks Recoverable coupling facility data table recor ds can be the subject of retained locks, like any other recoverable CICS r esource that is updated in a unit of work that subsequently fails. A recoverable CFDT supports indoubt and backout failures. If a unit of work fails when backing out an update to a CFD[...]

Chapter 16. Moving recoverable data sets that have retained locks There may be times when you need to r e-define a VSAM data set by creating a new data set and moving the data from the old data set to the new data set. About this task For example, you might need to do this to make a data set larger . In this case, note that special action is needed[...]

The following access method services examples assume that CICS.DATASET.A needs to be redefined and the data moved to a data set named CICS.DATASET.B , which is then renamed: DEFINE CLUSTER (NAME(CICS.DATASET.B) ... REPRO INDATASET(CICS.DATASET.A) OUTDATASET(CICS.DATASET.B) DELETE CICS.DATASET.A ALTER CICS.DATASET.B NEWNAME(CICS.DATASET.A) If the re[...]

This makes the data set unavailable while the move from old to new is in progr ess, and also allows the following unbind operation to succeed. 4. Issue the SHCDS FRUNBIND subcommand to unbind any retained locks against the old data set. For example: SHCDS FRUNBIND(CICS.DATASET.A) This enables SMSVSAM to preserve the locks r eady for rebinding later[...]

v Create a new empty data set into which the copy is to be r estored, and use IMPOR T to copy the data from the exported version of the data set to the new empty data set. v Use SHCDS FRSETRR to mark the original data set as being under maintenance. v Use SHCDS FRUNBIND to unbind the locks from the original data set. v Use SHCDS FRSETRR to mark the[...]

Chapter 17. Forward recovery procedures If a data set that is being used by CICS fails, perhaps because of physical damage to a disk, you can recover the data by performing forwar d recovery of the data set. About this task Y our forwar d recovery procedur es can be based either on your own, or an ISV -supplied utility program for pr ocessing the r[...]

Recovery of data set with volume still available The procedur e described here is necessary to preserve any r etained locks that are held by SMSVSAM against the data in the old data set. Unless you follow all the steps of this procedur e, the locks will not be valid for the new data set, with potential loss of data integrity . The following steps o[...]

9. Alter the new data set name Use access method services to rename the new data set to the name of the old data set. ALTER CICS.DATASETB NEWNAME(CICS.DATASETA) Y ou must give the r estored data set the name of the old data set to enable the following bind operation to succeed. 10. Issue the FRBIND subcommand Use this access method services SHCDS s[...]

There ar e several methods you can use to recover data sets after the loss of a volume. Whichever method you use (whether a volume restor e, a logical data set recovery , or a combination of both), you need to ensure SMSVSAM puts data sets into a lost locks state to protect data integrity . This means that, after you have carried out the initial st[...]

This is because CICS cannot run the lost locks r ecovery process until the data sets are available, and the data sets ar e made available only after the CICS VR recovery jobs are finished. If you physically restor e the volume, however , the data sets that need to be forward r ecovered are immediately available for backout. In this case you must us[...]

This clears the SMSVSAM CFVOL-QUIESCED state and allows SMSVSAM RLS access to the volume. CICS ensures that access is not allowed to the data sets that will eventually be forward r ecovered, but the volume is available for other data sets. 6. Run data set forward r ecovery jobs. The following two examples show forward r ecovery after the loss of a [...]

PIDS/565501800 LVLS/510 MS/DFHFC0152 RIDS/DFHFCCA PTFS/UN92873 REGS/GR15 VALU/00000008 PCSS/IDARETLK PRCS/000000A9 +DFHFC0312 ADSWA03A Message DFHFC0152 data set RLSADSW.VF04D.DATAENDB W e used the CEMT command INQUIRE UOWDSNF AIL IOERROR to display the UOWS that were shunted as a r esult of the I/O errors. For example, on the CICS region ADSW A01D[...]

effect in CICS r egion ADSW A03C was shown by the following response to an INQUIRE UOWDSNF AIL command for data set RLSADSW .VF01D.BANKACCT : INQUIRE UOWDSNFAIL DSN(RLSADSW.VF01D.BANKACCT) STATUS: RESULTS Dsn(RLSADSW.VF01D.BANKACCT ) Dat Ope Uow(ADD19B8166268E02) Rls Dsn(RLSADSW.VF01D.BANKACCT ) Rls Com Uow(ADD19B9D93DE1200) Rls After the SMSVSAM s[...]

work. Assuming that all CICS regions ar e active, and there are no indoubt UOWs, lost locks processing, for all data sets except the ones on the failed volume, should complete quickly . 9. In this example, CEMT INQUIRE UOWDSNF AIL on CICS region ADSW A01D showed UOW failures only for the RLSADSW .VF04D.TELLCTRL and RLSADSW .VF04D.DA T AENDB data se[...]

waits for indoubt resolution befor e allowing general access to the data set. In such a situation you can still release the locks immediately , using the SET DSNAME command, although in most cases you will lose data integrity . See “Lost locks recovery” on page 89 for mor e information about resolving indoubt UOWs following lost locks processin[...]

ROUTE *ALL,VARY SMS,SMSVSAM,TERMINATESERVER 8. When all SMSVSAM servers were down, we deleted the IGWLOCK00 lock structur e with the MVS command: VARY SMS,SMSVSAM,FORCEDELETELOCKSTRUCTURE 9. W e r estarted the SMSVSAM servers with the MVS command: ROUTE *ALL,VARY SMS,SMSVSAM,ACTIVE CICS was informed during dynamic RLS restart about the data sets fo[...]

that before r unning SHCDS CFREP AIR, the restored user catalog must be import connected to the master catalog on all systems (see the “Recovering Shared Catalogs” topic in DFSMS/MVS Managing Catalogs ). Forward recovery of data sets accessed in non-RLS mode For data sets accessed in non-RLS mode, use the following forward r ecovery procedur e:[...]

In these cases, you can resolve the cause of the failur e and try the whole process again. This topic describes what to do when the failure in forwar d recovery cannot be resolved. In this case, wher e you are unsuccessful in applying all the forward recovery log data to a r estored backup, you are for ced to abandon the forward recovery , and reve[...]

1) Force shunted indoubt units of work using SET DSNAME(...) UOW ACTION(COMMIT | BACKOUT | FORCE). Before issuing the next command, wait until the SET DSNAME(...) UOW ACTION has completed against all shunted indoubt units of work. If the UOW ACTION command for an indoubt unit of work r esults in backout, this will fail for the data set that is bein[...]

Procedure for failed non-RLS mode forward recovery operation If you are not successful in applying all the forwar d recovery log data to a restor ed backup, you are for ced to abandon the forward recovery , and revert to your most recent full backup. However , during its recovery pr ocessing, CICS assumes that it is operating on a data set that has[...]

202 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 18. Backup-while-open (BWO) The BWO facility , together with other system facilities and products, allows you to take a backup copy of a VSAM data set while it remains open for update. Many CICS applications depend on their data sets being open for update over a long period of time. Normally , you cannot take a backup of the data set while [...]

forward-r ecovery logs. Long-running transactions, automated teller machines, and continuously available applications requir e the database to be up and running when the backup is being taken. The concurrent copy function used along with BWO by DFSMSdss allows backups to be taken with integrity even when control-ar ea and control-interval splits an[...]

Hardware requirements The concurrent copy function is supported by the IBM ® 3990 Model 3 with the extended platform and the IBM 3990 Model 6 control units. Which data sets are eligible for BWO Y ou can use BWO only for: v Data sets that are on SMS-managed storage and that have an integrated catalog facility (ICF) catalog. v VSAM data sets accesse[...]

How you request BWO Y ou can define files as eligible for BWO in one of two ways. Procedure Decide which method you want to use for data sets: v If your data set is accessed in RLS mode, you must define the BWO option in the ICF catalog. Defining BWO in the ICF catalog requir es DFSMS 1.3. v If your data set is accessed only in non-RLS mode, you ca[...]

v But if you specify BWO(TYPECICS), and the PTF has not been applied, and you have not specified LOG(ALL) and a forward r ecovery log stream name, BWO processing for RLS r emains disabled for such files. T o achieve BWO for the file, you must either: – apply the PTF , – or specify LOG(ALL) and a forward r ecovery log stream name (if those actio[...]

Removing BWO attributes If you want to remove BWO attributes fr om your data sets, you must follow the correct pr ocedure to avoid problems when taking subsequent back ups. Procedure 1. Close the VSAM data set either by shutting down CICS normally or issuing the command CEMT SET FILE CLOSED . Do not perform an immediate shutdown, as CICS does not c[...]

After an uncontrolled or immediate shutdown, further BWO backups might be taken by DFSMShsm, because the BWO status in the ICF catalog is not reset. These backups should be discarded; only the non-BWO backups taken at the end of the batch window should be used during forward r ecovery , together with the CICS forward r ecovery logs. Data set securi[...]

Each of these operations is discussed in the following sections. File opening Differ ent processing is done for each of the three cases when a file is opened for an update. The following processing takes place: v First file opened for update against a cluster v Subsequent file opened for update against a cluster while the previous file is still ope[...]

v If the file was defined with BACKUPTYPE(ST A TIC) and the ICF catalog indicates that the data set is already ineligible for BWO, CICS sets the BACKUPTYPE attribute in the DSNB to indicate ineligibility for BWO. However , if the ICF catalog indicates that the data set is currently eligible for BWO, IGW ABWO makes it ineligible for BWO and sets the[...]

v If the file was defined with BACKUPTYPE(ST A TIC) and the ICF catalog indicates that the data set is already ineligible for BWO, the ICF catalog is not updated. However , if the ICF catalog indicates that the data set is currently eligible for BWO, IGW ABWO makes it ineligible for BWO and sets the r ecovery point to the current time. CICS issues [...]

Shutdown and restart The way CICS closes files is determined by whether the shutdown is controlled, immediate, or uncontrolled. Controlled shutdown During a controlled shutdown, CICS closes all open files defined in the FCT . This ensures that, for files that ar e open for update and eligible for BWO, the BWO attributes in the ICF catalog are set t[...]

When you use DFSMShsm, you still use DFSMSdss as the data mover . Y ou can specify this using the DFSMShsm SETSYS command: SETSYS DATAMOVER(DSS) The DFSMS processing at the start of backup is dependent on the DFSMS r elease level. For releases befor e DFSMS 1.2, DFSMSdss first checks the BWO attributes in the ICF catalog to see if the data set is e[...]

DFSMSdfp must now disallow the pending change to ‘BWO enabled’ (and DFSMSdss must fail the backup) because, if the split did not finish before the end of the backup, the invalid backup would not be discarded. v From ‘BWO disabled and VSAM split occurr ed’ to ‘BWO enabled’. This state change could be attempted if: 1. At the start of data[...]

each CICS allows all units of work with updates for the data set to complete, and then they write the tie-up recor ds to the forward recovery log and the log of logs, and replies to DFSMSdss. For BWO backups, it is usually not necessary for the forward r ecovery utility to process a log fr om file-open time. Therefore, the tie-up r ecords for all o[...]

The forward r ecovery utility should ALLOCA TE, with DISP=OLD, the data set that is to be recover ed. This prevents other jobs accessing a back level data set and ensures that data managers such as CICS ar e not still using the data set. Before the data set is opened, the forwar d recovery utility should set the BWO attribute flags to the ‘Forwar[...]

An assembler program that calls DFSMS callable services *ASM XOPTS(CICS,NOEPILOG,SP) * * A program that can be run as a CICS transaction to Read and Set * the BWO Indicators and BWO Recovery Point via DFSMS Callable * Services (IGWABWO). * * Invoke the program via a CICS transaction as follows: * * Rxxx ’data_set_name’ * Sxxx 100 ’data_set_na[...]

DS 30C DATEVAL DS 8C Date value from BWO recovery point SUCMSG1 DS 8C Message text TIMEVAL DS 8C Time value from BWO recovery point SUCMSG2 DS C Message text READMSG DS 0CL11 If function = READ put out BWO flags DS 7C Message text BWOVAL1 DS C BWO indicator 1 BWOVAL2 DS C BWO indicator 2 BWOVAL3 DS C BWO indicator 3 DS C Message text * DATETIME DS [...]

MVC BWOFLAGS(12),ZEROES LA R4,1(0) CLI BWOC1,C’0’ BE PRGBIT2 ST R4,BWOF1 Set BWO indicator 1 if required PRGBIT2 DS 0H CLI BWOC2,C’0’ BE PRGBIT3 ST R4,BWOF2 Set BWO indicator 2 if required PRGBIT3 DS 0H CLI BWOC3,C’0’ BE PRGCONT ST R4,BWOF3 Set BWO indicator 3 if required B PRGCONT PRGREAD DS 0H CLI TRANFUNC,C’R’ BNE PRGABORT If tra[...]

MVC SUCMSG1(8),SUCTXT1 MVC SUCMSG2(1),SUCTXT2 UNPK KEYWORK(9),BWOTIME(5) Make date printable TR KEYWORK(8),HEXTAB-C’0’ MVC DATEVAL(8),KEYWORK UNPK KEYWORK(9),BWOTIME+4(5) Make time printable TR KEYWORK(8),HEXTAB-C’0’ MVC TIMEVAL(8),KEYWORK CLI TRANFUNC,C’S’ If READ then print BWO flags BNE PRGREADO * * Got all the info we need, so put i[...]

222 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Chapter 19. Disaster recovery If your CICS system is normally available about 99 percent of the time, it would be wise to look at your disaster recovery plan. The same pr essure that drives high availability drives the need for timely and current disaster r ecovery . Y ou must plan what level of disaster r ecovery you require for your CICS environm[...]

acceptable. If you are located in an ar ea prone to hurricanes or earthquakes, for example, a disaster recovery site next door would be pointless. When you are planning for disaster r ecovery , consider the cost of being unable to operate your business for a period of time. Y ou have to consider the number of lost transactions, and the future loss [...]

v How critical and sensitive your business processes ar e: the more critical they are, the more fr equently testing may be required. Six tiers of solutions for off-site recovery One blueprint for recovery planning describes a scheme consisting of six tiers of off-site r ecoverability (tiers 1-6), with a seventh tier (tier 0) that relies on local re[...]

Y our disaster r ecovery plan has to include information to guide the staff responsible for r ecovering your system, from hardwar e requir ements to day-to-day operations. The backups requir ed for off-site storage must be created periodically . After a disaster , your data can only be as up-to-date as the last backup—daily , weekly , monthly , o[...]

Tier 1 T ier 1 provides a very basic level of disaster recovery . Y ou will lose data in the disaster , perhaps a considerable amount. However , tier 1 allows you to recover and provide some form of service at low cost. Y ou must assess whether the loss of data and the time taken to restor e a service will prevent your company from continuing in bu[...]

The advantage of tier 3 is that you should be able to provide a service to your users quite rapidly . Y ou must assess whether the loss of data will pr event your company from continuing in business. Figure 20 summarizes the tier 3 solution. T ier 3 is similar to tier 2. The difference is that data is electr onically transmitted to the hot site. Th[...]

The advantage of these methods is their low cost. The disadvantages of these methods are: v Recovery is slow , and it can take days or weeks to r ecover . v Any recovery is incomplete, because any updates made after the point-in-time backup are lost. v Disaster recovery is risky , and difficulties in testing your disaster recovery plan could lead t[...]

T ier 4 closes the gap between the point-in-time backups and current online processing r ecovery . Under a tier 4 r ecovery plan, site one acts as a backup to site two, and site two acts as a backup to site one. T ier 4 duplicates the vital data of each system at the other's site. Y ou must transmit image copies of data to the alternate site o[...]

v Cost is higher than for the tier 1 to 3 solutions, because you need dedicated hardwar e, software, and communication links. T ier 5 - two-site, two-phase commit A tier 5 solution is appropriate for a custom-designed recovery plan with special applications. Because these applications must be designed to use this solution, it cannot be implemented [...]

Figure 24 summarizes the tier 6 solution. T ier 6, minimal to zero data loss, is the ultimate level of disaster recovery . There ar e two tier 6 solutions, one hardware-based and the other softwar e-based. For details of the hardwar e and software available for these solutions, see “Peer-to-peer r emote copy (PPRC) and extended remote copy (XRC)?[...]

support the XRC DFSMS/MVS host, and one for the recovery 3990, this allows a total of 86 km (53.4 miles) between the 3990s. If you use channel extenders with XRC, there is no limit on the distance between your primary and r emote site. For RRDF there is no limit to the distance between the primary and secondary sites. T ier 4–6 solutions This sum[...]

Disaster recovery and high availability This topic describes the tier 6 solutions for high availability and data currency when recovering fr om a disaster . Peer-to-peer remote copy (PPRC) and extended remote copy (XRC) PPRC and XRC are both 3990-6 har dware solutions that provide data curr ency to secondary , remote volumes. Updates made to second[...]

v IMS write-ahead data set (W ADS) and IMS online log data set (OLDS) v ACBLIB for IMS v Boot-strap data set (BSDS), the catalog and the directory for DB2 v DB2 logs v Any essential non-database volumes CICS applications can use non-DASD storage for processing data. If your application depends on this type of data, be aware that PPRC and XRC do not[...]

where ther e is a high volume of transactions, but each transaction is typically less than 200 dollars in value. Other benefits of PPRC and XRC PPRC or XRC may eliminate the need for disaster recovery backups to be taken at the primary site, or to be taken at all. PPRC allows you to temporarily suspend the copying of updates to the secondary site. [...]

between the primary and secondary sites is interrupted. Remote logging is only as effective as the curr ency of the data that is sent off-site. RRDF transports log stream data to a r emote location in real-time, within seconds of the log operation at the primary site. When the RRDF address space at the r emote site receives the log data, it formats[...]

Y ou should ensur e that a senior manager is designated as the disaster recovery manager . The recovery manager must make the final decision whether to switch to a remote site, or to try to r ebuild the local system (this is especially true if you have adopted a solution that does not have a warm or hot standby site). Y ou must decide who will run [...]

CICS VSAM Recovery QSAM copy CICS VSAM Recovery (CICS VR) provides a QSAM copy function that can copy MVS log streams to a QSAM data set. Copies of the QSAM data can be sent either electronically or physically to the remote site. On arrival at the r emote site, you can use the MVS system logger import services to put the log recor ds into an MVS sy[...]

If a disaster occurs at the primary site, your disaster recovery pr ocedures should include recovery of VSAM data sets at the designated r emote recovery site. Y ou can then emergency r estart the CICS regions at the remote site so that they can backout any uncommitted data. Special support is needed for RLS because recor d locks, which were pr ote[...]

Part 4. Appendixes © Copyright IBM Corp. 1982, 2010 241[...]

242 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Notices This information was developed for products and services of fered in the U.S.A. IBM may not offer the pr oducts, services, or features discussed in this document in other countries. Consult your local IBM repr esentative for information on the products and services curr ently available in your area. Any refer ence to an IBM product, pr ogra[...]

Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The licensed program described in this document and all licensed material available for it are pr ovided by IBM under terms of the IBM Customer Agr eement, IBM International Programming License Agreement, or any equivalent agre[...]

Bibliography CICS books for CICS T ransaction Server for z/OS General CICS T ransaction Server for z/OS Program Directory , GI13-0536 CICS T ransaction Server for z/OS What's New , GC34-6994 CICS T ransaction Server for z/OS Upgrading from CICS TS V ersion 2.3 , GC34-6996 CICS T ransaction Server for z/OS Upgrading from CICS TS V ersion 3.1 , [...]

CICS Shared Data T ables Guide , SC34-7017 CICSPlex SM books for CICS T ransaction Server for z/OS General CICSPlex SM Concepts and Planning , SC34-7044 CICSPlex SM Web User Interface Guide , SC34-7045 Administration and Management CICSPlex SM Administration , SC34-7005 CICSPlex SM Operations V iews Reference , SC34-7006 CICSPlex SM Monitor V iews [...]

Accessibility Accessibility features help a user who has a physical disability , such as restricted mobility or limited vision, to use software pr oducts successfully . Y ou can perform most tasks r equired to set up, run, and maintain your CICS system in one of these ways: v using a 3270 emulator logged on to CICS v using a 3270 emulator logged on[...]

248 CICS TS for z/OS 4.1: Recovery and Restart Guide[...]

Index A abend handling 95, 151 ACID properties, of a transaction 20 activity keypoints description 22 ADCD abend 159 AFCF abend 159 AFCW abend 159 AIRDELA Y 39 AIX (alternate index) 130, 147 alternate index (AIX) 130, 147 alternate indexes preserving locks over a r ebuild 186 application processing unit designing 141 applications division into unit[...]

DL/I (continued) implicit enqueuing upon 158 intertransaction communication 146 scheduling program isolation scheduling 158 documenting recovery and r estart programs 105 DSNBs, data set name blocks recovery 54 DTIMOUT option (DEFINE TRANSACTION) 123 dump table 50 dynamic RLS restart 37 dynamic transaction backout 6 basic mapping support 78 decisio[...]

locking (continued) implicit locking on recoverable files 156 in application programs 154 locks 14 log of logs failures 1 19 logical levels, application program 92 logical recovery 132 lost locks recovery fr om 89 M managing UOW state 18 MNPS 38, 39, 40 moving data sets using EXPOR T and IMPORT commands 185 using REPRO command 183 moving data sets [...]

system log stream basic definition 104 system logs log-tail deletion 1 14 system or abend exit creation 95 system recovery table (SR T) definition of 104 user extensions to 95 system warm keypoints 27 systems administration for BWO 208 T tables for recovery 104 task termination, abnormal 94 DFHPEP execution 94 DFHREST execution 93 task termination,[...]

Readers’ Comments — We'd Like to Hear from Y ou CICS T ransaction Server for z/OS V ersion 4 Release 1 Recovery and Restart Guide Publication No. SC34-7012-01 W e appreciate your comments about this publication. Please comment on specific err ors or omissions, accuracy , organization, subject matter , or completeness of this book. The comm[...]

Readers’ Comments — We'd Like to Hear from Y ou SC34-7012-01 SC34-7012-01  Cut or Fold Along Line Cut or Fold Along Line Fold and T ape Please do not staple Fold and T ape Fold and T ape Please do not staple Fold and T ape PLACE POST AGE ST AMP HERE IBM United Kingdom Limited User T echnologies Department (MP095) Hursley Park W[...]

[...]

 SC34-7012-01[...]