The following document describes the procedure to upgrade the Microcode on the IBM p590 systems. Upgrading the microcode requires the each frame in the pair to be powered-down, causing all systems running on the frame to be shutdown. Since production databases are running on each frame, a microcode upgrade requires the coordination of the AIX system administration team, DBA's, and application administrators.

1. The first phase of this process is to upgrade the HMC associated with the p590 frames.
2. Then failover all production systems to the alternate frame. Non-production systems will not be failed over, they will simply be shutdown.
3. Next, power off the p590 frame and perform the Microcode upgrade.
4. After microcode upgrade is complete and verified, power up the p590 and restart all systems.
5. Failback all production systems and test.
6. Repeat this process for the second p590 frame.

Materials Checklist
• Latest Microcode DVD
• Downloaded by:
• Download Date:
• Version......:
• Latest HMC Upgrade DVD's
• Downloaded by:
• Download Date:
• Version......:
• Latest HMC fixes and patches

• Downloaded by:
• Download Date:
• Version......:
• Downloaded by:
• Download Date:
• Version......:
• Downloaded by:
• Download Date:
• Version......:
• Downloaded by:
• Download Date:
• Version......:

Special Instructions:
• Perform simultaneous microcode upgrade of mdctxuapp81. See separate document for mdctxuapp81 procedure.

Phase 1

Phase 1 Start Time:
Phase 1 Duration:

---

• Shutdown the HMC and boot from latest DVD's, Install HMC Upgrade

  mdclphmc01:

  
  Start Time:
  Duration:

---

• Install latest required HMC patches and fixes

  mdclphmc01:

  
  Start Time:
  Duration:

---

• Restart HMC and verify upgrade, patches, fixes

  mdclphmc01

  
  Start Time:
  Duration:

---

• Verify system information, network connectivity, DNS resolution

  mdclphmc01:

  
  Start Time:
  Duration:

Phase 2

Failover all production systems to the alternate frame. Non-production systems will not be failed over, they will simply be shutdown.

Phase 2 Start Time:
Phase 2 Duration:

---

• Contact p590 DBA's and application admins, failover Oracle databases from mdcapega01 to mdcapega02.

  mdcapega01:

  
  Start Time:
  Duration:

---

• Shutdown p590 Oracle database server mdcapega01

  mdcapega01:

  
  Start Time:
  Duration:

---

• Other Production System failovers - HACMP shutdown w/ takeover

  mdcapora01:

  
  Start Time:
  Duration:

  ---

  mdcapora03:

  
  Start Time:
  Duration:

  ---

  mdcapsee01:

  
  Start Time:
  Duration:

  ---

  mdcapcal01:

  
  Start Time:
  Duration:

  ---

  mdcapcal03:

  
  Start Time:
  Duration:

  ---

  mdcapcal05:

  
  Start Time:
  Duration:

---

• Non-production system shutdown - No failover

  mdcaaora01:

  
  Start Time:
  Duration:

  ---

  mdcaaora03:

  
  Start Time:
  Duration:

  ---

  mdcadedn01:

  
  Start Time:
  Duration:

  ---

  mdcadedn03:

  
  Start Time:
  Duration:

  ---

  mdcadedn05:

  
  Start Time:
  Duration:

  ---

  mdcadedn07:

  
  Start Time:
  Duration:

  ---

  mdceperf:

  
  Start Time:
  Duration:

  ---

  mdclpldap1:

  
  Start Time:
  Duration:

---

• Shutdown VIO Servers

  mdcapvio01:

  
  Start Time:
  Duration:

  ---

  mdcapvio02:

  
  Start Time:
  Duration:

---

• Verify application and database integrity and operation on frame 2. Obtain OK to proceed from management, DO NOT POWER OFF YET.

  9119-590-SN51A971E:

  
  Start Time:
  Duration:

Phase 3

Phase 3 Start Time:
Phase 3 Duration:

---

• Prepare HMC/Frame for microcode upgrade

  mdclphmc01:

  
  Start Time:
  Duration:

• ---

  Perform Microcode upgrade of p590 frame

  9119-590-SN51A971E:

  
  Start Time:
  Duration:

---

• Verify Microcode upgrade of p590 frame 9119-590-SN51A971E

  9119-590-SN51A971E:

  
  Start Time:
  Duration:

---

• Reconfigure power cable to alternate PDU

  9119-590-SN51A971E:

  
  Start Time:
  Duration:

---

• Check HMC Settings for all LPAR's on frame 9119-590-SN51A971E.

  mdclphmc01

  
  Start Time:
  Duration:

  Minimum Memory
  Desired Memory
  Maximum Memory
  Minimum Processing Units
  Desired Processing Units
  Maximum Processing Units
  Minimum Virtual Processors
  Desired Virtual Processors
  Maximum Virtual Processors
  Cap Mode
  Weight
  

Phase 4

Phase 4 Start Time:
Phase 4 Duration:

---

• Power-on p590 frame 1 to "standby" state

  9119-590-SN51A971E:

  
  Start Time:
  Duration:

---

• Activate the EGATE database server.

  mdcapega01:

  
  Start Time:
  Duration:

---

• Activate VIO Servers

  mdcapvio01:

  
  Start Time:
  Duration:

  ---

  mdcapvio02:

  
  Start Time:
  Duration:

---

• Activate other Production Systems

  mdcapora01:

  
  Start Time:
  Duration:

  ---

  mdcapora03:

  
  Start Time:
  Duration:

  ---

  mdcapsee01:

  
  Start Time:
  Duration:

  ---

  mdcapcal01:

  
  Start Time:
  Duration:

  ---

  mdcapcal03:

  
  Start Time:
  Duration:

  ---

  mdcapcal05:

  
  Start Time:
  Duration:

---

• Activate Non-production systems

  mdcaaora01:

  
  Start Time:
  Duration:

  ---

  mdcaaora03:

  
  Start Time:
  Duration:

  ---

  mdcadedn01:

  
  Start Time:
  Duration:

  ---

  mdcadedn03:

  
  Start Time:
  Duration:

  ---

  mdcadedn05:

  
  Start Time:
  Duration:

  ---

  mdcadedn07:

  
  Start Time:
  Duration:

  ---

  mdceperf:

  
  Start Time:
  Duration:

  ---

  mdclpldap1:

  
  Start Time:
  Duration:

Phase 5

Failback the production databases, production application servers, and restart non-production systems.

Phase 5 Start Time:
Phase 5 Duration:

---

• Contact p590 DBA's and application admins, failback EGATE databases from mdcapega02 to mdcapega01.

  mdcapega01:

  
  Start Time:
  Duration:

---

• Other production system application failbacks

  mdcapora01:

  
  Start Time:
  Duration:

  ---

  mdcapora03:

  
  Start Time:
  Duration:

  ---

  mdcapsee01:

  
  Start Time:
  Duration:

  ---

  mdcapcal01:

  
  Start Time:
  Duration:

  ---

  mdcapcal03:

  
  Start Time:
  Duration:

  ---

  mdcapcal05:

  
  Start Time:
  Duration:

---

• Non-production system application startup and cluster verify

  mdcaaora01:

  
  Start Time:
  Duration:

  ---

  mdcaaora03:

  
  Start Time:
  Duration:

  ---

  mdcadedn01:

  
  Start Time:
  Duration:

  ---

  mdcadedn03:

  
  Start Time:
  Duration:

  ---

  mdcadedn05:

  
  Start Time:
  Duration:

  ---

  mdcadedn07:

  
  Start Time:
  Duration:

  ---

  mdceperf:

  
  Start Time:
  Duration:

  ---

  mdclpldap1:

  
  Start Time:
  Duration:

---

• Verify PLM communication with all LPAR's on frame 9119-590-SN51A971E:

  mdcapnim01:

  
  Start Time:
  Duration:

  tail -f /var/opt/plm/51A971E.log
  

Phase 6

The failover process will now be repeated on the second frame in preparation for performing the microcode upgrade on that frame.

Phase 6 Start Time:
Phase 6 Duration:

---

• Contact p590 DBA's and application admins, failover Oracle databases from mdcapega02 to mdcapega01.

  mdcapega02:

  
  Start Time:
  Duration:

• Shutdown p590 Oracle database server mdcapega02

  mdcapega02:

  
  Start Time:
  Duration:

• Other Production System failovers - HACMP Shutdown w/ Takeover

  mdcapora02:

  
  Start Time:
  Duration:

  ---

  mdcapora04:

  
  Start Time:
  Duration:

  ---

  mdcaasee01: Shutdown - no failover

  
  Start Time:
  Duration:

  ---

  mdcapcal02:

  
  Start Time:
  Duration:

  ---

  mdcapcal04:

  
  Start Time:
  Duration:

  ---

  mdcapcal06:

  
  Start Time:
  Duration:

• Non-production system shutdown - No Failover

  mdcaaora02:

  
  Start Time:
  Duration:

  ---

  mdcaaora04:

  
  Start Time:
  Duration:

  ---

  mdcadedn02:

  
  Start Time:
  Duration:

  ---

  mdcadedn04:

  
  Start Time:
  Duration:

  ---

  mdcadedn06:

  
  Start Time:
  Duration:

  ---

  mdcadedn08:

  
  Start Time:
  Duration:

  ---

  mdclpadm02:

  
  Start Time:
  Duration:

  ---

  mdclpldap2:

  
  Start Time:
  Duration:

• Shutdown VIO Servers

  mdcapvio03:

  
  Start Time:
  Duration:

  ---

  mdcapvio04:

  
  Start Time:
  Duration:

• Verify application and database integrity and operation on frame 1. Obtain OK to preceed from management, DO NOT POWER OFF YET.

  9119-590-SN51A974E:

  
  Start Time:
  Duration:

Phase 7

Phase 7 Start Time:
Phase 7 Duration:

• Prepare HMC/Frame for microcode upgrade of p590 51A974E

  mdclphmc01:

  
  Start Time:
  Duration:

• Perform Microcode upgrade of p590 frame 2

  9119-590-SN51A974E:

  
  Start Time:
  Duration:

• Verify Microcode upgrade of p590 frame 2

  9119-590-SN51A974E:

  
  Start Time:
  Duration:

• Reconfigure power cable to alternate PDU

  9119-590-SN51A974E:

  
  Start Time:
  Duration:

• Check HMC Settings for all LPAR's on frame 9119-590-SN51A974E.

  mdclphmc01

  
  Start Time:
  Duration:

  Minimum Memory
  Desired Memory
  Maximum Memory
  Minimum Processing Units
  Desired Processing Units
  Maximum Processing Units
  Minimum Virtual Processors
  Desired Virtual Processors
  Maximum Virtual Processors
  Cap Mode
  Weight
  

Phase 8

Phase 8 Start Time:
Phase 8 Duration:

• Power-on p590 frame 2 to "standby" state

  9119-590-SN51A974E:

  
  Start Time:
  Duration:

• Activate the EGATE database server.

  mdcapega02:

  
  Start Time:
  Duration:

• Activate VIO Servers

  mdcapvio03:

  
  Start Time:
  Duration:

  ---

  mdcapvio04:

  
  Start Time:
  Duration:

• Activate other Production Systems

  mdcapora02:

  
  Start Time:
  Duration:

  ---

  mdcapora04:

  
  Start Time:
  Duration:

  ---

  mdcaasee01:

  
  Start Time:
  Duration:

  ---

  mdcapcal02:

  
  Start Time:
  Duration:

  ---

  mdcapcal04:

  
  Start Time:
  Duration:

  ---

  mdcapcal06:

  
  Start Time:
  Duration:

• Activate Non-production systems

  mdcaaora02:

  
  Start Time:
  Duration:

  ---

  mdcaaora04:

  
  Start Time:
  Duration:

  ---

  mdcadedn02:

  
  Start Time:
  Duration:

  ---

  mdcadedn04:

  
  Start Time:
  Duration:

  ---

  mdcadedn06:

  
  Start Time:
  Duration:

  ---

  mdcadedn08:

  
  Start Time:
  Duration:

  ---

  mdclpadm02:

  
  Start Time:
  Duration:

  ---

  mdclpldap2:

  
  Start Time:
  Duration:

Phase 9

Failback the production databases, production application servers, and restart non-production systems.

Phase 9 Start Time:
Phase 9 Duration:

• Contact p590 DBA's and application admins, failback EGATE databases from mdcapega01 to mdcapega02.

  mdcapega02:

  
  Start Time:
  Duration:

• Other production system application failbacks

  mdcapora02:

  
  Start Time:
  Duration:

  ---

  mdcapora04:

  
  Start Time:
  Duration:

  ---

  mdcaasee01:

  
  Start Time:
  Duration:

  ---

  mdcapcal02:

  
  Start Time:
  Duration:

  ---

  mdcapcal04:

  
  Start Time:
  Duration:

  ---

  mdcapcal06:

  
  Start Time:
  Duration:

• Non-production system application startup and cluster verify

  mdcaaora02:

  
  Start Time:
  Duration:

  ---

  mdcaaora04:

  
  Start Time:
  Duration:

  ---

  mdcadedn02:

  
  Start Time:
  Duration:

  ---

  mdcadedn04:

  
  Start Time:
  Duration:

  ---

  mdcadedn06:

  
  Start Time:
  Duration:

  ---

  mdcadedn08:

  
  Start Time:
  Duration:

• Verify PLM communication with all LPAR's on frame 9119-590-SN51A974E:

  mdcapnim01:

  
  Start Time:
  Duration:

  tail -f /var/opt/plm/51A974E.log
  

Phase 10

Perform microcode upgrade of NIM Server to latest.

Phase 10 Start Time:
Phase 10 Duration:

• Shutdown NIM Server

  mdcapnim01:

  
  Start Time:
  Duration:

  shutdown -Fh
  

• Perform microcode update from HMC of frame 9111-520-SN105401E

  mdclphmc01:

  
  Start Time:
  Duration:

• Restart NIM Server

  mdcapnim01:

  
  Start Time:
  Duration:

• Verify PLM communication with all LPAR's on frames 9119-590-SN51A971E and 9119-590-SN51A974E

  mdcapnim01:

  
  Start Time:
  Duration:

  tail -f /var/opt/plm/51A971E.log
  tail -f /var/opt/plm/51A974E.log
  

Phase 11

Phase 11 Start Time:
Phase 11 Duration:

• Contact p590 DBA's and application admins, failback EGATE databases from mdcapega01 to mdcapega02.

  Application Admins:

  
  Start Time:
  Duration:

  ---

  DBA's:

  
  Start Time:
  Duration:

  ---

  Operations:

  
  Start Time:
  Duration:

  ---

  Management:

  
  Start Time:
  Duration:

• Update change request to completed state.

  CHMN00002331386:

  
  Start Time:
  Duration: