p590 Microcode Upgrade procedures Scheduled for 06/03/2006

The following document describes the procedure to upgrade the Microcode on the DDC IBM p590 systems scheduled to occur on June 3, 2006. Upgrading the microcode requires the each frame in the pair to be powered-down, causing all systems running on the frame to be shutdown. Since production databases are running on each frame, a microcode upgrade requires the coordination of the AIX system administration team, DBA's, and application administrators.

Summary of Activities

The first phase of this process is to upgrade the HMC associated with the p590 frames.

Then failover all production systems to the alternate frame. Non-production systems will not be failed over, they will simply be shutdown.

Next, power off the p590 frame and perform the Microcode upgrade.

After microcode upgrade is complete and verified, power up the p590 and restart all systems.

Failback all production systems and test.

Repeat this process for the second p590 frame.

Materials Checklist

Latest Microcode DVD

Downloaded by: Dana French
Download Date: 05/31/2006
Version......: SF_230_156_BP230_155

Latest HMC Upgrade DVD's

Downloaded by: Dana French
Download Date: 05/31/2006
Version......: HMC Recovery V5R2.1

Latest HMC fixes and patches

Downloaded by:
Download Date:
Version......: None

Special Instructions:

None

Phase 1

- Upgrade the HMC associated with the DDC p590 frames.

Phase 1 Start Time:
Phase 1 Duration:

Shutdown the HMC and boot from latest DVD's, Install HMC Upgrade
ddclphmc01:
Start Time:
Duration:

Follow these steps to determine the current BIOS revision level on the HMC.
1. Shut down the HMC.
2. Power on the HMC, and then press the F1 key to display the Setup utility.
3. In the System Information menu, the BIOS Build Level should be T2E134AUS and the Machine Type/Model should be 7315-CR2 or 7310-CR2. Note: If the Machine Type/Model is NOT 7315-CR2 or 7310-CR2, then do not install this BIOS update.
4. If the BIOS Build Level is NOT T2E134AUS, then use the following Download and install procedure to acquire and install the BIOS update.
Follow these steps to download and install the BIOS update.
1. Click the following link to download the EXE file for the BIOS update:
  ftp://ftp.software.ibm.com/eserver/pseries/hmc/fixes/73xxT12f.exe
2. Copy the 73xxT12f.exe file to a PC running WindowsXX.
3. To create the bootable diskette required to install the update, put a blank, formatted 1.44MB diskette in the PC's floppy drive.
4. On the Start menu, select Run, browse to the 73xxT12f.exe file, and then click OK.
5. Follow the instructions on the screen to create the diskette.
6. To install the update, follow the installation instructions in the Readme First file. It is a good idea to print the Readme First file before you begin installing the BIOS update.
ddclphmc01:
Start Time:
Duration:

Install latest required HMC patches and fixes
ddclphmc01:
Start Time:
Duration:

Restart HMC and verify upgrade, patches, fixes
ddclphmc01
Start Time:
Duration:

Verify system information, network connectivity, DNS resolution
ddclphmc01:
Start Time:
Duration:

Phase 2

- System Failovers

Failover all production systems to the alternate frame. Non-production systems will not be failed over, they will simply be shutdown.

Phase 2 Start Time:
Phase 2 Duration:

Contact p590 DBA's and prior to outage start shutdown databases as follows:
1. Shutdown any AIX databases "under construction" for project SUMO on DDC lpars
2. Shutdown Clarity (NIKPRE) preprod database on ddcaaora03
3. Shutdown Production Month (ADWPRE) preprod database on ddcaaora03
4. Shutdown all EDNA databases on DDCAPEDN01/DDCAPEDN02
5. Shutdown all eGate preprod databases on DDCAAEGA01 and DDCAAEGA02
6. Shutdown all ATLAS 9i (dev/preprod) databases on DDCAACAL01/02 and DDCADCAL01/02 (these database are NOT in use by anyone)
ddcap*:
Start Time:
Duration:

Clear Oracle database off frame 9119-590-SN51A972E
1. Failover CRIP (CRPP, crppdtu0) from ddcapora01 to ddcapora02
2. Failover Shadow Settlements and ERCOT Extracts ( EEXTPRD, ETSSPRD, etapdtu0) from ddcapora03 to ddcapora04
ddcapora01/02:
Start Time:
Duration:

Shutdown all non-VIO LPARs
!(ddcapvio)*:
Start Time:
Duration:

Shutdown VIO Servers
ddcapvio01:
Start Time:
Duration:
ddcapvio02:
Start Time:
Duration:

Verify application and database integrity and operation on frame 9119-590-SN51A973E. Obtain OK to proceed from management, DO NOT POWER OFF YET.
9119-590-SN51A973E:
Start Time:
Duration:

Phase 3

- Perform Microcode upgrade on frame 9119-590-SN51A972E

Phase 3 Start Time:
Phase 3 Duration:

Prepare HMC and Frame 9119-590-SN51A972E for microcode upgrade
ddclphmc01:
Start Time:
Duration:
Perform Microcode upgrade of p590 frame 9119-590-SN51A972E
9119-590-SN51A972E:
Start Time:
Duration:

Verify Microcode upgrade of p590 frame 9119-590-SN51A972E
9119-590-SN51A972E:
Start Time:
Duration:

From HMC, modify VIO server settings, change VIO's to dedicated CPUs.
ddcapvio01/ddcapvio02
Start Time:
Duration:

Check HMC Settings for all LPAR's on frame 9119-590-SN51A972E.

ddclphmc01

Start Time:
Duration:

Minimum Memory: XXX
Desired Memory: XXX
Maximum Memory: XXX
Minimum Processing Units: 0.2
Desired Processing Units: 0.2
Maximum Processing Units: XXX
Minimum Virtual Processors: 2
Desired Virtual Processors: 2 
Maximum Virtual Processors: 9
Cap Mode: uncapped
Weight: 128

Phase 4

- Power on p590 Frame 9119-590-SN51A972E and activate systems

Phase 4 Start Time:
Phase 4 Duration:

Power-on p590 frame 9119-590-SN51A972E to "standby" state
9119-590-SN51A972E:
Start Time:
Duration:

Activate VIO Servers
ddcapvio01:
Start Time:
Duration:
ddcapvio02:
Start Time:
Duration:

Activate all other production LPARs on the frame
ddcapedn01:
ddcapedn03:
ddcapedn05:
ddcapedn07:
ddcapora01:
ddcapora03:
Start Time:
Duration:

Activate all Non-production LPARs on the frame
ddcaacal01:
ddcaacal02:
ddcaaega01:
ddcaaora01:
ddcaaora03:
ddcaasee01:
Start Time:
Duration:

Phase 5

- Application Failback

Failback the production databases and production application servers.

Phase 5 Start Time:
Phase 5 Duration:

Clear Oracle databases off FRAME2
1. Shutdown NMarket (NKTPRD) production database on ddcapora04
2. Failover RiskDataMart (RDMPRD, znmpdtu0) production database from ddcapora04 to ddcapora03
3. Failover CRIP (CRPP, crppdtu0) from ddcapora02 to ddcapora01
4. Failover Shadow Settlements and ERCOT Extracts (EEXTPRD, ETSSPRD, etapdtu0) from ddcapora04 to ddcapora03
ddcapora*:
Start Time:
Duration:

Shutdown all non-VIO LPARs on Frame 9119-590-SN51A973E
!(ddcapvio)*:
Start Time:
Duration:

Shutdown VIO Servers on Frame 9119-590-SN51A973E
ddcapvio03:
Start Time:
Duration:
ddcapvio04:
Start Time:
Duration:

Verify application and database integrity and operation on frame 9119-590-SN51A972E. Obtain OK to proceed from management, DO NOT POWER OFF YET.
9119-590-SN51A972E:
Start Time:
Duration:

Prepare HMC/Frame for microcode upgrade
ddclphmc01:
Start Time:
Duration:
Perform Microcode upgrade of p590 frame 9119-590-SN51A973E
9119-590-SN51A973E:
Start Time:
Duration:

Verify Microcode upgrade of p590 frame 9119-590-SN51A973E
9119-590-SN51A973E:
Start Time:
Duration:

From HMC, modify VIO server settings, change VIO's to dedicated CPUs.
ddcapvio03/ddcapvio04
Start Time:
Duration:

Check HMC Settings for all LPAR's on frame 9119-590-SN51A973E.

ddclphmc01

Start Time:
Duration:

Minimum Memory: XXX
Desired Memory: XXX
Maximum Memory: XXX
Minimum Processing Units: 0.2
Desired Processing Units: 0.2
Maximum Processing Units: XXX
Minimum Virtual Processors: 2
Desired Virtual Processors: 2 
Maximum Virtual Processors: 9
Cap Mode: uncapped
Weight: 128

Phase 6

- Power on p590 Frame 9119-590-SN51A973E and activate systems

Phase 4 Start Time:
Phase 4 Duration:

Power-on p590 Frame 9119-590-SN51A973E to "standby" state
9119-590-SN51A973E:
Start Time:
Duration:

Activate VIO Servers
ddcapvio03:
Start Time:
Duration:
ddcapvio04:
Start Time:
Duration:

Activate all other production LPARs on the frame
ddcapedn02:
ddcapedn04:
ddcapedn06:
ddcapedn08:
ddcapora02:
ddcapora04:
ddcapsee01:
Start Time:
Duration:

Activate all Non-production LPARs on the frame
ddcaaega02:
ddcaaora02:
ddcaaora04:
ddcadcal01:
ddcadcal02:
Start Time:
Duration:

Phase 7

- Application Failback

Failback the production databases and production application servers to the second frame.

Phase 7 Start Time:
Phase 7 Duration:

Failback databases to FRAME2 (for those database's that have primary homes on Frame2)
1. Failover RiskDataMart (RDMPRD) production database from ddcapora03 to ddcapora04
2. Startup NMarket (NKTPRD) production database on ddcapora04
ddcapora03/04:
Start Time:
Duration:

Startup all non-production and "under construction" databases
1. Same list as #1 just perform startup
ddcap*:
Start Time:
Duration:

Verify PLM communication with all LPAR's on frame 9119-590-SN51A972E:
ddcapnim01:
Start Time:
Duration: tail -f /var/opt/plm/51A972E.log

Verify PLM communication with all LPAR's on frame 9119-590-SN51A973E:
ddcapnim01:
Start Time:
Duration: tail -f /var/opt/plm/51A973E.log

Phase 8

- Project Completion

Notify all interested parties of project completion.

Phase 8 Start Time:
Phase 8 Duration:

Contact p590 DBA's and application admins.
Application Admins:
Start Time:
Duration:
DBA's:
Start Time:
Duration:
Operations:
Start Time:
Duration:
Management:
Start Time:
Duration:

Update change request to completed state.
CHMN00002331386:
Start Time:
Duration: