The following document describes the procedure used to upgrade the EGATE application Servers in the MDC data center.

The first phase of this process is to verify the current cluster configuration and ensure stability of the cluster before proceeding with the upgrade.

The second phase of this process is to manually fail-over each EGATE application server to the hot-standby node, one at a time, upgrading each unoccupied system as we go.

Ray Meachum will open bridge line at 05:00.

Bridge Line:

PhoneNbr: 866.844.3871
Passcode: 675840

Notify DSAM that work is beginning on the EGATE production application servers. Contact EGATE LOS team and also notify Ray Meachum.

Notification List:

DSAM
EGATE LOS
Ray Meachum

Verify cluster configuration and ensure stability

mdctxuapp83:

Start Time:
Duration: clstat smitty clverify

Do a "graceful" shutdown of HACMP on mdctxuapp81

mdctxuapp80:

Start Time:
Duration: smitty clstop

Run the following importvg commands on mdctxuapp83 to fix 80s VGs

mdctxuapp83:

Start Time:
Duration:

# run loop on mdctxuapp80 to generate list of commands to run on 83

for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mdctxuapp83 to fix 80s VGs
if PV=$( lspv | grep "^vpath.*000cadddc26ccf3f" | head -1 | awk '{ print $1 }' )
then    exportvg egate103vg
        importvg -V 203 -n -y egate103vg ${PV}
        varyonvg egate103vg
        chvg -a n egate103vg
        varyoffvg egate103vg
fi
if PV=$( lspv | grep "^vpath.*000cadddc26c2cd6" | head -1 | awk '{ print $1 }' )
then    exportvg egate104vg
        importvg -V 204 -n -y egate104vg ${PV}
        varyonvg egate104vg
        chvg -a n egate104vg
        varyoffvg egate104vg
fi
if PV=$( lspv | grep "^vpath.*000cadddc26c3b01" | head -1 | awk '{ print $1 }' )
then    exportvg egate105vg
        importvg -V 205 -n -y egate105vg ${PV}
        varyonvg egate105vg
        chvg -a n egate105vg
        varyoffvg egate105vg
fi
if PV=$( lspv | grep "^vpath.*000c644ddf34e037" | head -1 | awk '{ print $1 }' )
then    exportvg egate14vg
        importvg -V 114 -n -y egate14vg ${PV}
        varyonvg egate14vg
        chvg -a n egate14vg
        varyoffvg egate14vg
fi
if PV=$( lspv | grep "^vpath.*000c644d227a4992" | head -1 | awk '{ print $1 }' )
then    exportvg egate19vg
        importvg -V 119 -n -y egate19vg ${PV}
        varyonvg egate19vg
        chvg -a n egate19vg
        varyoffvg egate19vg
fi
if PV=$( lspv | grep "^vpath.*000c644d22206319" | head -1 | awk '{ print $1 }' )
then    exportvg egate1vg 
        importvg -V 101 -n -y egate1vg ${PV}
        varyonvg egate1vg
        chvg -a n egate1vg
        varyoffvg egate1vg
fi
if PV=$( lspv | grep "^vpath.*000cadddc26c4204" | head -1 | awk '{ print $1 }' )
then    exportvg egate2vg
        importvg -V 102 -n -y egate2vg ${PV}
        varyonvg egate2vg   
        chvg -a n egate2vg  
        varyoffvg egate2vg
fi
if PV=$( lspv | grep "^vpath.*000c644db8cf9057" | head -1 | awk '{ print $1 }' )
then    exportvg egate6vg
        importvg -V 106 -n -y egate6vg ${PV}
        varyonvg egate6vg   
        chvg -a n egate6vg  
        varyoffvg egate6vg
fi

Run the "dpovgfix" command on the hot standby node before bringing up the "app80rg1 resource group.

mdctxuapp83:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mdctxuapp83:

Start Time:
Duration: clstat smitty clverify

Bring the resource group "app80rg1" up on mdctxuapp83

mdctxuapp83:

Start Time:
Duration: smitty cl_resgrp

Perform time zone variable changes on mdctxuapp80:

mdctxuapp80:

Start Time:
Duration:

cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

Perform the firmware upgrade on mdctxuapp80:

mdctxuapp80:

Start Time:
Duration:

rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

Verify firmware upgrade on mdctxuapp80:

mdctxuapp80:

Start Time:
Duration:

lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

Perform Microcode Discovery Survey and check for recommended updates

mdctxuapp80:

Start Time:
Duration:

invscout

# On local workstation:
scp root@mdctxuapp80:/var/adm/invscout/mdctxuapp80.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mdctxuapp80:/tmp/microcode

Upgrade microcode as necessary:

mdctxuapp80:

Start Time:
Duration:

cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

Restart HACMP on mdctxuapp80 and add it back into the cluster:

mdctxuapp80:

Start Time:
Duration: smitty clstart

Verify cluster configuration and ensure stability

mdctxuapp80:

Start Time:
Duration: clstat smitty clverify

Notification of node completion and preparation for next node:

mdctxuapp80:

Start Time:
Duration:

DSAM
EGATE LOS
Ray Meachum

Verify cluster configuration and ensure stability

mdctxuapp81:

Start Time:
Duration: clstat smitty clverify

Do a "graceful" shutdown of HACMP on mdctxuapp81

mdctxuapp81:

Start Time:
Duration: smitty clstop

Run the following importvg commands on mdctxuapp83 to fix 81s VGs

mdctxuapp83:

Start Time:
Duration:

# run this loop on mdctxuapp81 to generate list of command to run on 83
for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mdctxuapp83 to fix 81s VGs
if PV=$( lspv | grep "^vpath.*000c643d5cdb878f" | head -1 | awk '{ print $1 }' )
then    exportvg egate106vg
        importvg -V 206 -n -y egate106vg ${PV}
        varyonvg egate106vg
        chvg -a n egate106vg
        varyoffvg egate106vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdc5482" | head -1 | awk '{ print $1 }' )
then    exportvg egate107vg
        importvg -V 207 -n -y egate107vg ${PV}
        varyonvg egate107vg
        chvg -a n egate107vg
        varyoffvg egate107vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdcbe3c" | head -1 | awk '{ print $1 }' )
then    exportvg egate108vg
        importvg -V 208 -n -y egate108vg ${PV}
        varyonvg egate108vg
        chvg -a n egate108vg
        varyoffvg egate108vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdd1665" | head -1 | awk '{ print $1 }' )
then    exportvg egate109vg
        importvg -V 209 -n -y egate109vg ${PV}
        varyonvg egate109vg
        chvg -a n egate109vg
        varyoffvg egate109vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdded53" | head -1 | awk '{ print $1 }' )
then    exportvg egate110vg
        importvg -V 210 -n -y egate110vg ${PV}
        varyonvg egate110vg
        chvg -a n egate110vg
fi
if PV=$( lspv | grep "^vpath.*000c643d4993bafa" | head -1 | awk '{ print $1 }' )
then    exportvg egate13vg 
        importvg -V 113 -n -y egate13vg ${PV}
        varyonvg egate13vg  
        chvg -a n egate13vg
        varyoffvg egate13vg
fi
if PV=$( lspv | grep "^vpath.*000c643d1ba56485" | head -1 | awk '{ print $1 }' )
then    exportvg egate15vg 
        importvg -V 215 -n -y egate15vg ${PV}
        varyonvg egate15vg  
        chvg -a n egate15vg
        varyoffvg egate15vg
fi
if PV=$( lspv | grep "^vpath.*000c644de4d56fe0" | head -1 | awk '{ print $1 }' )
then    exportvg egate17vg 
        importvg -V 117 -n -y egate17vg ${PV}
        varyonvg egate17vg  
        chvg -a n egate17vg
        varyoffvg egate17vg
fi
if PV=$( lspv | grep "^vpath.*000c644d373f3be3" | head -1 | awk '{ print $1 }' )
then    exportvg egate18vg 
        importvg -V 118 -n -y egate18vg ${PV}
        varyonvg egate18vg  
        chvg -a n egate18vg
        varyoffvg egate18vg
fi
if PV=$( lspv | grep "^vpath.*000cadddc26c8d2a" | head -1 | awk '{ print $1 }' )
then    exportvg egate22vg
        importvg -V 122 -n -y egate22vg ${PV}
        varyonvg egate22vg  
        chvg -a n egate22vg 
        varyoffvg egate22vg
fi
if PV=$( lspv | grep "^vpath.*000c643d2698e454" | head -1 | awk '{ print $1 }' )
then    exportvg egate8vg
        importvg -V 108 -n -y egate8vg ${PV}
        varyonvg egate8vg   
        chvg -a n egate8vg  
        varyoffvg egate8vg
fi

Run the "dpovgfix" command on the hot standby node before bringing up the "app81rg1 resource group.

mdctxuapp80:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mdctxuapp80:

Start Time:
Duration: clstat smitty clverify

Bring the resource group "app81rg1" up on mdctxuapp80

mdctxuapp80:

Start Time:
Duration: smitty cl_resgrp

Perform time zone variable changes on mdctxuapp81:

mdctxuapp81:

Start Time:
Duration:

cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

Perform the firmware upgrade on mdctxuapp81:

mdctxuapp81:

Start Time:
Duration:

rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

Verify firmware upgrade on mdctxuapp81:

mdctxuapp81:

Start Time:
Duration:

lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

Perform Microcode Discovery Survey and check for recommended updates

mdctxuapp81:

Start Time:
Duration:

invscout

# On local workstation:
scp root@mdctxuapp81:/var/adm/invscout/mdctxuapp81.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mdctxuapp81:/tmp/microcode

Upgrade microcode as necessary:

mdctxuapp81:

Start Time:
Duration:

cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

Restart HACMP on mdctxuapp81 and add it back into the cluster:

mdctxuapp81:

Start Time:
Duration: smitty clstart

Verify cluster configuration and ensure stability

mdctxuapp81:

Start Time:
Duration: clstat smitty clverify

Notification of node completion and preparation for next node:

mdctxuapp81:

Start Time:
Duration:

DSAM
EGATE LOS
Ray Meachum

Verify cluster configuration and ensure stability

mdctxuapp82:

Start Time:
Duration: clstat smitty clverify

Do a "graceful" shutdown of HACMP on mdctxuapp82

mdctxuapp82:

Start Time:
Duration: smitty clstop

Run the following importvg commands on mdctxuapp83 to fix 82s VGs

mdctxuapp83:

Start Time:
Duration:

# run this loop on mdctxuapp82 to generate commands to run on 83
for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mdctxuapp83 to fix 82s VGs
if PV=$( lspv | grep "^vpath.*000caddd11b64709" | head -1 | awk '{ print $1 }' )
then    exportvg egate100vg
        importvg -V 47 -n -y egate100vg ${PV}
        varyonvg egate100vg
        chvg -a n egate100vg
        varyoffvg egate100vg
fi
if PV=$( lspv | grep "^vpath.*000caddd11b88d31" | head -1 | awk '{ print $1 }' )
then    exportvg egate101vg
        importvg -V 45 -n -y egate101vg ${PV}
        varyonvg egate101vg
        chvg -a n egate101vg
        varyoffvg egate101vg
fi
if PV=$( lspv | grep "^vpath.*000caddd1ccaf18b" | head -1 | awk '{ print $1 }' )
then    exportvg egate16vg
        importvg -V 44 -n -y egate16vg ${PV}
        varyonvg egate16vg
        chvg -a n egate16vg
        varyoffvg egate16vg
fi
if PV=$( lspv | grep "^vpath.*000cadddb8d8f150" | head -1 | awk '{ print $1 }' )
then    exportvg egate21vg
        importvg -V 48 -n -y egate21vg ${PV}
        varyonvg egate21vg
        chvg -a n egate21vg
        varyoffvg egate21vg
fi
if PV=$( lspv | grep "^vpath.*000caddd1da96046" | head -1 | awk '{ print $1 }' )
then    exportvg egate7vg
        importvg -V 78 -n -y egate7vg ${PV}
        varyonvg egate7vg
        chvg -a n egate7vg
        varyoffvg egate7vg
fi

Run the "dpovgfix" command on the hot standby node before bringing up the "app82rg1 resource group.

mdctxuapp81:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mdctxuapp81:

Start Time:
Duration: clstat smitty clverify

Bring the resource group "app82rg1" up on mdctxuapp81

mdctxuapp81:

Start Time:
Duration: smitty cl_resgrp

Perform time zone variable changes on mdctxuapp82:

mdctxuapp82:

Start Time:
Duration:

cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

Perform the firmware upgrade on mdctxuapp82:

mdctxuapp82:

Start Time:
Duration:

rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

Verify firmware upgrade on mdctxuapp82:

mdctxuapp82:

Start Time:
Duration:

lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

Perform Microcode Discovery Survey and check for recommended updates

mdctxuapp82:

Start Time:
Duration:

invscout

# On local workstation:
scp root@mdctxuapp82:/var/adm/invscout/mdctxuapp82.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mdctxuapp82:/tmp/microcode

Upgrade microcode as necessary:

mdctxuapp82:

Start Time:
Duration:

cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

Restart HACMP on mdctxuapp82 and add it back into the cluster:

mdctxuapp82:

Start Time:
Duration: smitty clstart

Run the "dpovgfix" command on the hot standby node before bringing up the "app82rg1 resource group.

mdctxuapp82:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mdctxuapp82:

Start Time:
Duration: clstat smitty clverify

Notification of node completion and outage completion:

mdctxuapp82:

Start Time:
Duration:

DSAM
EGATE LOS
Ray Meachum

Perform Microcode Discovery Survey on all systems

mdctxuapp80 / 81 / 82 / 83:

Start Time:
Duration: invscout

Upload invscout survey files to IBM MDS

local workstation:

Start Time:
Duration:

cd /tmp
scp root@mdctxuapp80:/var/adm/invscout/mdctxuapp80.mup .
scp root@mdctxuapp81:/var/adm/invscout/mdctxuapp81.mup .
scp root@mdctxuapp82:/var/adm/invscout/mdctxuapp82.mup .
scp root@mdctxuapp83:/var/adm/invscout/mdctxuapp83.mup .

# Open a browser window and go to the MDS upload URL:

http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the report to your local disk.
# Then copy each MDS survey report to the MDC NIM server as follows:

scp MDS_mdctxuapp80_808.html root@mdcapnim01:/tmp/invscout.mup/MDS_mdctxuapp80.content.shtml
scp MDS_mdctxuapp81_808.html root@mdcapnim01:/tmp/invscout.mup/MDS_mdctxuapp81.content.shtml
scp MDS_mdctxuapp82_808.html root@mdcapnim01:/tmp/invscout.mup/MDS_mdctxuapp82.content.shtml
scp MDS_mdctxuapp83_808.html root@mdcapnim01:/tmp/invscout.mup/MDS_mdctxuapp83.content.shtml

Execute AMMS script to upload MDS reports to eperf web sites

mdcapnim01:

Start Time:
Duration:

print "mdctxuapp80|mdctxuapp80_man
mdctxuapp81|mdctxuapp81_man
mdctxuapp82|mdctxuapp82_man
mdctxuapp83|mdctxuapp83_man" > tmp.list

/usr/local/scripts/amms_k93 -vMgrt -W eperf -f ./tmp.list
/usr/local/scripts/amms_k93 -vMgrt -W ddceperf -f ./tmp.list
/usr/local/scripts/amms_k93 -vMgrt -W ddceperf -x /srv/www/htdocs/css/invscout -f ./tmp.list

Perform DST TZ Survey on all systems

mdcapnim01:

Start Time:
Duration:

cd /home/s45x
./dstnew.433.ksh mdc

scp dstmdctz*.content.shtml root@ddceperf:/srv/www/htdocs/css/GlobalSysAdmin/Power5

Close Bridge Line