Mt Xia: Technical Consulting Group

Business Continuity / Disaster Recovery / High Availability
Data Center Automation / Audit Response / Audit Compliance
-
Current Location
-

css
  GlobalSysAdmin
    Power5

-

digg Digg this page
del.icio.us Post to del.icio.us
Slashdot Slashdot it!

Business Web Site Hosting
$3.99 / month includes Tools,
Shopping Cart, Site Builder
www.siteox.com

FREE Domain Registration
included with Web Site Hosting
Tools, Social Networking, Blog
www.siteox.com

p590 Microcode Upgrade procedures Scheduled for 06/03/2006

The following document describes the procedure to upgrade the Microcode on the DDC IBM p590 systems scheduled to occur on June 3, 2006. Upgrading the microcode requires the each frame in the pair to be powered-down, causing all systems running on the frame to be shutdown. Since production databases are running on each frame, a microcode upgrade requires the coordination of the AIX system administration team, DBA's, and application administrators.

    Summary of Activities

  1. The first phase of this process is to upgrade the HMC associated with the p590 frames.

  2. Then failover all production systems to the alternate frame. Non-production systems will not be failed over, they will simply be shutdown.

  3. Next, power off the p590 frame and perform the Microcode upgrade.

  4. After microcode upgrade is complete and verified, power up the p590 and restart all systems.

  5. Failback all production systems and test.

  6. Repeat this process for the second p590 frame.

    Materials Checklist

  • Latest Microcode DVD
    • Downloaded by: Dana French
    • Download Date: 05/31/2006
    • Version......: SF_230_156_BP230_155

  • Latest HMC Upgrade DVD's
    • Downloaded by: Dana French
    • Download Date: 05/31/2006
    • Version......: HMC Recovery V5R2.1

  • Latest HMC fixes and patches

    • Downloaded by:
    • Download Date:
    • Version......: None

    Special Instructions:
  • None

Phase 1

- Upgrade the HMC associated with the DDC p590 frames.


Phase 1 Start Time:
Phase 1 Duration: 







    


    


  •  Shutdown the HMC and boot from latest DVD's, Install HMC Upgrade


    ddclphmc01:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Follow these steps to determine the current BIOS revision level on
the HMC.

  
      

  
    1.  Shut down the HMC.
      2. 

  
    2.  Power on the HMC, and then press the F1 key to display the Setup
   utility.
      3. 

  
    3.  In the System Information menu, the BIOS Build Level should be
   T2E134AUS and the Machine Type/Model should be 7315-CR2 or 7310-CR2.
   Note: If the Machine Type/Model is NOT 7315-CR2 or 7310-CR2, then do 
   not install this BIOS update.
      4. 

  
    4.  If the BIOS Build Level is NOT T2E134AUS, then use the following   
   Download and install procedure to acquire and install the BIOS update.
      5. 

  
    

    • 


  •  Follow these steps to download and install the BIOS update.

  
      

  
    1.  Click the following link to download the EXE file for the BIOS
   update:
      
   ftp://ftp.software.ibm.com/eserver/pseries/hmc/fixes/73xxT12f.exe
      2. 

  
    2.  Copy the 73xxT12f.exe file to a PC running WindowsXX.
      3. 

  
    3.  To create the bootable diskette required to install the update, put a
   blank, formatted 1.44MB diskette in the PC's floppy drive.
      4. 

  
    4.  On the Start menu, select Run, browse to the 73xxT12f.exe file, and
   then click OK.
      5. 
   
  
    5.  Follow the instructions on the screen to create the diskette.
      6. 

  
    6.  To install the update, follow the installation instructions in the
   Readme First file. It is a good idea to print the Readme First file
   before you begin installing the BIOS update.
      7. 

  
    


    ddclphmc01:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Install latest required HMC patches and fixes


    ddclphmc01:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Restart HMC and verify upgrade, patches, fixes


    ddclphmc01

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Verify system information, network connectivity, DNS resolution


    ddclphmc01:

    

    Start Time: 

    Duration: 

    

    • 















Phase 2
 - System Failovers



Failover all production systems to the alternate frame. 
Non-production systems will not be failed over, they will simply be
shutdown.







Phase 2 Start Time: 

Phase 2 Duration: 









    




    


  •  Contact p590 DBA's and  prior to outage start shutdown 
databases as follows:

  
      

  
    1.  Shutdown any AIX databases "under construction" for project SUMO on DDC lpars 
      2. 

  
    2.  Shutdown Clarity (NIKPRE) preprod database on ddcaaora03
      3. 

  
    3.  Shutdown Production Month (ADWPRE) preprod database on ddcaaora03
      4. 

  
    4.  Shutdown all EDNA databases on DDCAPEDN01/DDCAPEDN02
      5. 

  
    5.  Shutdown all eGate preprod databases on DDCAAEGA01 and DDCAAEGA02 
      6. 

  
    6.  Shutdown all ATLAS 9i (dev/preprod) databases on DDCAACAL01/02 and DDCADCAL01/02 (these database are NOT in use by anyone)
      7. 

  
    


    ddcap*: 

    

    Start Time: 

    Duration: 

    

    • 


    

    


  •  Clear Oracle database off frame 9119-590-SN51A972E

  
      

  
    1.  Failover CRIP (CRPP, crppdtu0) from ddcapora01 to ddcapora02
      2. 

  
    2.  Failover Shadow Settlements and ERCOT Extracts ( EEXTPRD, ETSSPRD, etapdtu0) from ddcapora03 to ddcapora04
      3. 

  
    


    ddcapora01/02: 

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Shutdown all non-VIO LPARs


    !(ddcapvio)*: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Shutdown VIO Servers


    ddcapvio01: 

    

    Start Time: 

    Duration: 

    


    

    ddcapvio02: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Verify application and database integrity and operation on frame 
9119-590-SN51A973E.  Obtain OK to proceed from management, DO NOT POWER
OFF YET.


    9119-590-SN51A973E:

    

    Start Time: 

    Duration: 

    

    • 















Phase 3
 - Perform Microcode upgrade on frame
9119-590-SN51A972E







Phase 3 Start Time: 

Phase 3 Duration: 









    




    


  •  Prepare HMC and Frame 9119-590-SN51A972E for microcode upgrade


    ddclphmc01:

    

    Start Time: 

    Duration: 

    

    • 




  • 

 Perform Microcode upgrade of p590 frame 9119-590-SN51A972E


    9119-590-SN51A972E:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Verify Microcode upgrade of p590 frame 9119-590-SN51A972E


    9119-590-SN51A972E:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  From HMC, modify VIO server settings, change VIO's to dedicated CPUs.


    ddcapvio01/ddcapvio02

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Check HMC Settings for all LPAR's on frame 9119-590-SN51A972E.


    ddclphmc01

    

    Start Time: 

    Duration: 

    Minimum Memory: XXX
Desired Memory: XXX
Maximum Memory: XXX
Minimum Processing Units: 0.2
Desired Processing Units: 0.2
Maximum Processing Units: XXX
Minimum Virtual Processors: 2
Desired Virtual Processors: 2 
Maximum Virtual Processors: 9
Cap Mode: uncapped
Weight: 128

    


    • 















Phase 4
 - Power on p590 Frame 9119-590-SN51A972E and
activate systems







Phase 4 Start Time: 

Phase 4 Duration: 









    




    


  •  Power-on p590 frame 9119-590-SN51A972E to "standby" state


    9119-590-SN51A972E:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Activate VIO Servers


    ddcapvio01: 

    

    Start Time: 

    Duration: 

    


    

    ddcapvio02: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Activate all other production LPARs on the frame


    ddcapedn01: 

    ddcapedn03: 

    ddcapedn05: 

    ddcapedn07: 

    ddcapora01: 

    ddcapora03: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Activate all Non-production LPARs on the frame


    ddcaacal01: 

    ddcaacal02: 

    ddcaaega01: 

    ddcaaora01: 

    ddcaaora03: 

    ddcaasee01: 

    

    Start Time: 

    Duration: 

    


    • 















Phase 5
 - Application Failback



Failback the production databases and production application
servers.







Phase 5 Start Time: 

Phase 5 Duration: 









    




    


  •  Clear Oracle databases off FRAME2

  
      

  
    1.  Shutdown NMarket (NKTPRD) production database on ddcapora04 
      2. 

  
    2.  Failover RiskDataMart (RDMPRD, znmpdtu0) production database from ddcapora04 to ddcapora03
      3. 

  
    3.  Failover CRIP (CRPP, crppdtu0) from ddcapora02 to ddcapora01
      4. 

  
    4.  Failover Shadow Settlements and ERCOT Extracts (EEXTPRD, ETSSPRD, etapdtu0) from ddcapora04 to ddcapora03
      5. 

  
    


    ddcapora*: 

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Shutdown all non-VIO LPARs on Frame 9119-590-SN51A973E


    !(ddcapvio)*: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Shutdown VIO Servers on Frame 9119-590-SN51A973E


    ddcapvio03: 

    

    Start Time: 

    Duration: 

    


    

    ddcapvio04: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Verify application and database integrity and operation on frame 
9119-590-SN51A972E. Obtain OK to proceed from management, DO NOT POWER
OFF YET.


    9119-590-SN51A972E:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Prepare HMC/Frame for microcode upgrade


    ddclphmc01:

    

    Start Time: 

    Duration: 

    

    • 




  • 

 Perform Microcode upgrade of p590 frame 9119-590-SN51A973E


    9119-590-SN51A973E:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Verify Microcode upgrade of p590 frame 9119-590-SN51A973E


    9119-590-SN51A973E:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  From HMC, modify VIO server settings, change VIO's to dedicated CPUs.


    ddcapvio03/ddcapvio04

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Check HMC Settings for all LPAR's on frame 9119-590-SN51A973E.


    ddclphmc01

    

    Start Time: 

    Duration: 

    Minimum Memory: XXX
Desired Memory: XXX
Maximum Memory: XXX
Minimum Processing Units: 0.2
Desired Processing Units: 0.2
Maximum Processing Units: XXX
Minimum Virtual Processors: 2
Desired Virtual Processors: 2 
Maximum Virtual Processors: 9
Cap Mode: uncapped
Weight: 128

    


    • 















Phase 6
 - Power on p590 Frame 9119-590-SN51A973E and
activate systems







Phase 4 Start Time: 

Phase 4 Duration: 









    




    


  •  Power-on p590 Frame 9119-590-SN51A973E to "standby" state


    9119-590-SN51A973E:

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Activate VIO Servers


    ddcapvio03: 

    

    Start Time: 

    Duration: 

    


    

    ddcapvio04: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Activate all other production LPARs on the frame


    ddcapedn02: 

    ddcapedn04: 

    ddcapedn06: 

    ddcapedn08: 

    ddcapora02: 

    ddcapora04: 

    ddcapsee01: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Activate all Non-production LPARs on the frame


    ddcaaega02: 

    ddcaaora02: 

    ddcaaora04: 

    ddcadcal01: 

    ddcadcal02: 

    

    Start Time: 

    Duration: 

    


    • 















Phase 7
 - Application Failback



Failback the production databases and production application
servers to the second frame.







Phase 7 Start Time: 

Phase 7 Duration: 









    




    


  •  Failback databases to FRAME2 (for those database's that have
primary homes on Frame2) 

  
      

  
    1.  Failover RiskDataMart (RDMPRD) production database from ddcapora03 to ddcapora04
      2. 

  
    2.  Startup NMarket (NKTPRD) production database on ddcapora04 
      3. 

  
    


    ddcapora03/04: 

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Startup all non-production and "under construction" databases 

  
      

  
    1.  Same list as #1 just perform startup
      2. 

  
    


    ddcap*: 

    

    Start Time: 

    Duration: 

    

    • 




    


  •  Verify PLM communication with all LPAR's on frame 9119-590-SN51A972E:


    ddcapnim01: 

    

    Start Time: 

    Duration: 

    tail -f /var/opt/plm/51A972E.log

    


    • 




    


  •  Verify PLM communication with all LPAR's on frame 9119-590-SN51A973E:


    ddcapnim01: 

    

    Start Time: 

    Duration: 

    tail -f /var/opt/plm/51A973E.log

    


    • 















Phase 8
 - Project Completion



Notify all interested parties of project completion.







Phase 8 Start Time: 

Phase 8 Duration: 









    




    


  •  Contact p590 DBA's and application admins.


    Application Admins: 

    

    Start Time: 

    Duration: 

    


    

    DBA's: 

    

    Start Time: 

    Duration: 

    


    

    Operations: 

    

    Start Time: 

    Duration: 

    


    

    Management: 

    

    Start Time: 

    Duration: 

    


    • 




    


  •  Update change request to completed state.


    CHMN00002331386: 

    

    Start Time: 

    Duration: 

    


    • 

























   

















-


DDC p590 MC Upgrade


-




 



      










FREE Domain Registration

included with Web Site Hosting

Tools, Social Networking, Blog

www.siteox.com




Business Web Site Hosting

$3.99 / month includes Tools,

Shopping Cart, Site Builder

www.siteox.com