The following document describes the procedure used to upgrade the EGATE application Servers in the Mt Xia Franklin (MXF) data center.

  1. The first phase of this process is to verify the current cluster configuration and ensure stability of the cluster before proceeding with the upgrade.

  2. The second phase of this process is to manually fail-over each EGATE application server to the hot-standby node, one at a time, upgrading each unoccupied system as we go.

  • Ray Meachum will open bridge line at 05:00.

    • Bridge Line:

  • Notify DSAM that work is beginning on the EGATE production application servers. Contact EGATE LOS team and also notify Ray Meachum.

    Notification List:

  • Verify cluster configuration and ensure stability

    • mxfmtxapp83:


    Start Time:
    Duration: 
    clstat
smitty clverify

  • Do a "graceful" shutdown of HACMP on mxfmtxapp81

    • mxfmtxapp80:


    Start Time:
    Duration: 
    smitty clstop

  • Run the following importvg commands on mxfmtxapp83 to fix 80s VGs

    • mxfmtxapp83:


    Start Time:
    Duration: 
    # run loop on mxfmtxapp80 to generate list of commands to run on 83

for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mxfmtxapp83 to fix 80s VGs
if PV=$( lspv | grep "^vpath.*000cadmxd26ccf3f" | head -1 | awk '{ print $1 }' )
then    exportvg egate103vg
        importvg -V 203 -n -y egate103vg ${PV}
        varyonvg egate103vg
        chvg -a n egate103vg
        varyoffvg egate103vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c2cd6" | head -1 | awk '{ print $1 }' )
then    exportvg egate104vg
        importvg -V 204 -n -y egate104vg ${PV}
        varyonvg egate104vg
        chvg -a n egate104vg
        varyoffvg egate104vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c3b01" | head -1 | awk '{ print $1 }' )
then    exportvg egate105vg
        importvg -V 205 -n -y egate105vg ${PV}
        varyonvg egate105vg
        chvg -a n egate105vg
        varyoffvg egate105vg
fi
if PV=$( lspv | grep "^vpath.*000c644ddf34e037" | head -1 | awk '{ print $1 }' )
then    exportvg egate14vg
        importvg -V 114 -n -y egate14vg ${PV}
        varyonvg egate14vg
        chvg -a n egate14vg
        varyoffvg egate14vg
fi
if PV=$( lspv | grep "^vpath.*000c644d227a4992" | head -1 | awk '{ print $1 }' )
then    exportvg egate19vg
        importvg -V 119 -n -y egate19vg ${PV}
        varyonvg egate19vg
        chvg -a n egate19vg
        varyoffvg egate19vg
fi
if PV=$( lspv | grep "^vpath.*000c644d22206319" | head -1 | awk '{ print $1 }' )
then    exportvg egate1vg 
        importvg -V 101 -n -y egate1vg ${PV}
        varyonvg egate1vg
        chvg -a n egate1vg
        varyoffvg egate1vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c4204" | head -1 | awk '{ print $1 }' )
then    exportvg egate2vg
        importvg -V 102 -n -y egate2vg ${PV}
        varyonvg egate2vg   
        chvg -a n egate2vg  
        varyoffvg egate2vg
fi
if PV=$( lspv | grep "^vpath.*000c644db8cf9057" | head -1 | awk '{ print $1 }' )
then    exportvg egate6vg
        importvg -V 106 -n -y egate6vg ${PV}
        varyonvg egate6vg   
        chvg -a n egate6vg  
        varyoffvg egate6vg
fi

  • Run the "dpovgfix" command on the hot standby node before bringing up the "app80rg1 resource group.

    • mxfmtxapp83:


    Start Time:
    Duration: 
    for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

  • Verify cluster configuration and ensure stability

    • mxfmtxapp83:


    Start Time:
    Duration: 
    clstat
smitty clverify

  • Bring the resource group "app80rg1" up on mxfmtxapp83

    • mxfmtxapp83:


    Start Time:
    Duration: 
    smitty cl_resgrp

  • Perform time zone variable changes on mxfmtxapp80:

    • mxfmtxapp80:


    Start Time:
    Duration: 
    cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

  • Perform the firmware upgrade on mxfmtxapp80:

    • mxfmtxapp80:


    Start Time:
    Duration: 
    rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

  • Verify firmware upgrade on mxfmtxapp80:

    • mxfmtxapp80:


    Start Time:
    Duration: 
    lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

  • Perform Microcode Discovery Survey and check for recommended updates

    • mxfmtxapp80:


    Start Time:
    Duration: 
    invscout

# On local workstation:
scp root@mxfmtxapp80:/var/adm/invscout/mxfmtxapp80.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mxfmtxapp80:/tmp/microcode

  • Upgrade microcode as necessary:

    • mxfmtxapp80:


    Start Time:
    Duration: 
    cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

  • Restart HACMP on mxfmtxapp80 and add it back into the cluster:

    • mxfmtxapp80:


    Start Time:
    Duration: 
    smitty clstart

  • Verify cluster configuration and ensure stability

    • mxfmtxapp80:


    Start Time:
    Duration: 
    clstat
smitty clverify

  • Notification of node completion and preparation for next node:

    • mxfmtxapp80:


    Start Time:
    Duration: 
    


    

    

    






    

    


  •  Verify cluster configuration and ensure stability

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    clstat
smitty clverify

    




    


  •  Do a "graceful" shutdown of HACMP on mxfmtxapp81

    • 


    mxfmtxapp81: 

    

    Start Time: 




    Duration: 

    smitty clstop

    




    


  •  Run the following importvg commands on mxfmtxapp83 to fix 81s VGs

    • 


    mxfmtxapp83: 

    

    Start Time: 

    Duration: 

    # run this loop on mxfmtxapp81 to generate list of command to run on 83
for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mxfmtxapp83 to fix 81s VGs
if PV=$( lspv | grep "^vpath.*000c643d5cdb878f" | head -1 | awk '{ print $1 }' )
then    exportvg egate106vg
        importvg -V 206 -n -y egate106vg ${PV}
        varyonvg egate106vg
        chvg -a n egate106vg
        varyoffvg egate106vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdc5482" | head -1 | awk '{ print $1 }' )
then    exportvg egate107vg
        importvg -V 207 -n -y egate107vg ${PV}
        varyonvg egate107vg
        chvg -a n egate107vg
        varyoffvg egate107vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdcbe3c" | head -1 | awk '{ print $1 }' )
then    exportvg egate108vg
        importvg -V 208 -n -y egate108vg ${PV}
        varyonvg egate108vg
        chvg -a n egate108vg
        varyoffvg egate108vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdd1665" | head -1 | awk '{ print $1 }' )
then    exportvg egate109vg
        importvg -V 209 -n -y egate109vg ${PV}
        varyonvg egate109vg
        chvg -a n egate109vg
        varyoffvg egate109vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdded53" | head -1 | awk '{ print $1 }' )
then    exportvg egate110vg
        importvg -V 210 -n -y egate110vg ${PV}
        varyonvg egate110vg
        chvg -a n egate110vg
fi
if PV=$( lspv | grep "^vpath.*000c643d4993bafa" | head -1 | awk '{ print $1 }' )
then    exportvg egate13vg 
        importvg -V 113 -n -y egate13vg ${PV}
        varyonvg egate13vg  
        chvg -a n egate13vg
        varyoffvg egate13vg
fi
if PV=$( lspv | grep "^vpath.*000c643d1ba56485" | head -1 | awk '{ print $1 }' )
then    exportvg egate15vg 
        importvg -V 215 -n -y egate15vg ${PV}
        varyonvg egate15vg  
        chvg -a n egate15vg
        varyoffvg egate15vg
fi
if PV=$( lspv | grep "^vpath.*000c644de4d56fe0" | head -1 | awk '{ print $1 }' )
then    exportvg egate17vg 
        importvg -V 117 -n -y egate17vg ${PV}
        varyonvg egate17vg  
        chvg -a n egate17vg
        varyoffvg egate17vg
fi
if PV=$( lspv | grep "^vpath.*000c644d373f3be3" | head -1 | awk '{ print $1 }' )
then    exportvg egate18vg 
        importvg -V 118 -n -y egate18vg ${PV}
        varyonvg egate18vg  
        chvg -a n egate18vg
        varyoffvg egate18vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c8d2a" | head -1 | awk '{ print $1 }' )
then    exportvg egate22vg
        importvg -V 122 -n -y egate22vg ${PV}
        varyonvg egate22vg  
        chvg -a n egate22vg 
        varyoffvg egate22vg
fi
if PV=$( lspv | grep "^vpath.*000c643d2698e454" | head -1 | awk '{ print $1 }' )
then    exportvg egate8vg
        importvg -V 108 -n -y egate8vg ${PV}
        varyonvg egate8vg   
        chvg -a n egate8vg  
        varyoffvg egate8vg
fi

    




    


  •  Run the "dpovgfix" command on the hot standby node before
bringing up the "app81rg1 resource group.

    • 


    mxfmtxapp80: 

    

    Start Time: 

    Duration: 

    for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

    




    


  •  Verify cluster configuration and ensure stability

    • 


    mxfmtxapp80: 

    

    Start Time: 

    Duration: 

    clstat
smitty clverify

    




    


  •  Bring the resource group "app81rg1" up on mxfmtxapp80

    • 


    mxfmtxapp80: 

    

    Start Time: 

    Duration: 

    smitty cl_resgrp

    




    


  •  Perform time zone variable changes on mxfmtxapp81:

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

    




    


  •  Perform the firmware upgrade on mxfmtxapp81:

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

    




    


  •  Verify firmware upgrade on mxfmtxapp81:

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

    




    


  •  Perform Microcode Discovery Survey and check for recommended updates

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    invscout

# On local workstation:
scp root@mxfmtxapp81:/var/adm/invscout/mxfmtxapp81.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mxfmtxapp81:/tmp/microcode

    




    


  •  Upgrade microcode as necessary:

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

    




    


  •  Restart HACMP on mxfmtxapp81 and add it back into the cluster:

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    smitty clstart

    




    


  •  Verify cluster configuration and ensure stability

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    clstat
smitty clverify

    




    


  •  Notification of node completion and preparation for next node:

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    


    

    

    






    

    


  •  Verify cluster configuration and ensure stability

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    clstat
smitty clverify

    




    


  •  Do a "graceful" shutdown of HACMP on mxfmtxapp82

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    smitty clstop

    




    


  •  Run the following importvg commands on mxfmtxapp83 to fix 82s VGs

    • 


    mxfmtxapp83: 

    

    Start Time: 

    Duration: 

    # run this loop on mxfmtxapp82 to generate commands to run on 83
for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mxfmtxapp83 to fix 82s VGs
if PV=$( lspv | grep "^vpath.*000caddd11b64709" | head -1 | awk '{ print $1 }' )
then    exportvg egate100vg
        importvg -V 47 -n -y egate100vg ${PV}
        varyonvg egate100vg
        chvg -a n egate100vg
        varyoffvg egate100vg
fi
if PV=$( lspv | grep "^vpath.*000caddd11b88d31" | head -1 | awk '{ print $1 }' )
then    exportvg egate101vg
        importvg -V 45 -n -y egate101vg ${PV}
        varyonvg egate101vg
        chvg -a n egate101vg
        varyoffvg egate101vg
fi
if PV=$( lspv | grep "^vpath.*000caddd1ccaf18b" | head -1 | awk '{ print $1 }' )
then    exportvg egate16vg
        importvg -V 44 -n -y egate16vg ${PV}
        varyonvg egate16vg
        chvg -a n egate16vg
        varyoffvg egate16vg
fi
if PV=$( lspv | grep "^vpath.*000cadddb8d8f150" | head -1 | awk '{ print $1 }' )
then    exportvg egate21vg
        importvg -V 48 -n -y egate21vg ${PV}
        varyonvg egate21vg
        chvg -a n egate21vg
        varyoffvg egate21vg
fi
if PV=$( lspv | grep "^vpath.*000caddd1da96046" | head -1 | awk '{ print $1 }' )
then    exportvg egate7vg
        importvg -V 78 -n -y egate7vg ${PV}
        varyonvg egate7vg
        chvg -a n egate7vg
        varyoffvg egate7vg
fi

    




    


  •  Run the "dpovgfix" command on the hot standby node before
bringing up the "app82rg1 resource group.

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

    




    


  •  Verify cluster configuration and ensure stability

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    clstat
smitty clverify

    




    


  •  Bring the resource group "app82rg1" up on mxfmtxapp81

    • 


    mxfmtxapp81: 

    

    Start Time: 

    Duration: 

    smitty cl_resgrp

    




    


  •  Perform time zone variable changes on mxfmtxapp82:

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

    




    


  •  Perform the firmware upgrade on mxfmtxapp82:

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

    




    


  •  Verify firmware upgrade on mxfmtxapp82:

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

    




    


  •  Perform Microcode Discovery Survey and check for recommended updates

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    invscout

# On local workstation:
scp root@mxfmtxapp82:/var/adm/invscout/mxfmtxapp82.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mxfmtxapp82:/tmp/microcode

    




    


  •  Upgrade microcode as necessary:

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

    




    


  •  Restart HACMP on mxfmtxapp82 and add it back into the cluster:

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    smitty clstart

    




    


  •  Run the "dpovgfix" command on the hot standby node before
bringing up the "app82rg1 resource group.

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

    




    


  •  Verify cluster configuration and ensure stability

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    clstat
smitty clverify

    




    


  •  Notification of node completion and outage completion:

    • 


    mxfmtxapp82: 

    

    Start Time: 

    Duration: 

    


    

    

    






    

    


  •  Perform Microcode Discovery Survey on all systems

    • 


    mxfmtxapp80 / 81 / 82 / 83: 

    

    Start Time: 

    Duration: 

    invscout

    




    


  •  Upload invscout survey files to IBM MDS

    • 


    local workstation: 

    

    Start Time: 

    Duration: 

    cd /tmp
scp root@mxfmtxapp80:/var/adm/invscout/mxfmtxapp80.mup .
scp root@mxfmtxapp81:/var/adm/invscout/mxfmtxapp81.mup .
scp root@mxfmtxapp82:/var/adm/invscout/mxfmtxapp82.mup .
scp root@mxfmtxapp83:/var/adm/invscout/mxfmtxapp83.mup .

# Open a browser window and go to the MDS upload URL:

http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the report to your local disk.
# Then copy each MDS survey report to the MXF NIM server as follows:

scp MDS_mxfmtxapp80_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp80.content.shtml
scp MDS_mxfmtxapp81_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp81.content.shtml
scp MDS_mxfmtxapp82_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp82.content.shtml
scp MDS_mxfmtxapp83_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp83.content.shtml


    




    


  •  Execute AMMS script to upload MDS reports to mtxdoc web sites

    • 


    mxfapnim01: 

    

    Start Time: 

    Duration: 

    print "mxfmtxapp80|mxfmtxapp80_man
mxfmtxapp81|mxfmtxapp81_man
mxfmtxapp82|mxfmtxapp82_man
mxfmtxapp83|mxfmtxapp83_man" > tmp.list

/usr/local/scripts/amms_k93 -vMgrt -W mtxdoc -f ./tmp.list
/usr/local/scripts/amms_k93 -vMgrt -W mxdmtxdoc -f ./tmp.list
/usr/local/scripts/amms_k93 -vMgrt -W mxdmtxdoc -x /srv/www/htdocs/js/invscout -f ./tmp.list


    






    

    


  •  Perform DST TZ Survey on all systems

    • 


    mxfapnim01: 

    

    Start Time: 

    Duration: 

    cd /home/s45x
./dstnew.433.ksh mxf

scp dstmxftz*.content.shtml root@mxdmtxdoc:/srv/www/htdocs/js/GlobalSysAdmin/Power5

    




  •  
Close Bridge Line



    •