DST EGATE Upgrade - Mt Xia: Global System Administration: AIX

Current Location

js
GlobalSysAdmin
AIX

AIX Admin Methodology
Global Consolidation Project
All AIX admins should join
www.aixexpert.com

Join our LinkedIn Group
AIX Advanced Technical Experts
Contract Opportunities
www.LinkedIn.com

	Digg this page
	Post to del.icio.us
	Slashdot it!

LPAR Leasing
Lease an AIX / i5 LPAR
Reduce your costs
www.mtxia.com

Server Leasing
Lease a Server off-site
Reduce your costs
www.mtxia.com

Data Center Automation
Business Continuity and DR
Virtualization/Consolidation
www.mtxia.com

HMC Service
Hardware Management Console
Manage Remote AIX / i5 LPARs
www.siteox.com

Business Web Site Hosting
$3.99 / month includes Tools,
Shopping Cart, Site Builder
www.siteox.com

Disaster Recovery
Small Business Oriented
Off-Site Facilities
www.mtxia.com

IBM pSeries / iSeries
Reduce your Costs
Off-Site Server Hosting
www.mtxia.com

The following document describes the procedure used to upgrade the EGATE application Servers in the Mt Xia Franklin (MXF) data center.

The first phase of this process is to verify the current cluster configuration and ensure stability of the cluster before proceeding with the upgrade.

The second phase of this process is to manually fail-over each EGATE application server to the hot-standby node, one at a time, upgrading each unoccupied system as we go.

Ray Meachum will open bridge line at 05:00.

Bridge Line:

PhoneNbr: 866.844.3871
Passcode: 675840

Notify DSAM that work is beginning on the EGATE production application servers. Contact EGATE LOS team and also notify Ray Meachum.

Notification List:

DSAM
EGATE LOS
Ray Meachum

Verify cluster configuration and ensure stability

mxfmtxapp83:

Start Time:
Duration: clstat smitty clverify

Do a "graceful" shutdown of HACMP on mxfmtxapp81

mxfmtxapp80:

Start Time:
Duration: smitty clstop

Run the following importvg commands on mxfmtxapp83 to fix 80s VGs

mxfmtxapp83:

Start Time:
Duration:

# run loop on mxfmtxapp80 to generate list of commands to run on 83

for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mxfmtxapp83 to fix 80s VGs
if PV=$( lspv | grep "^vpath.*000cadmxd26ccf3f" | head -1 | awk '{ print $1 }' )
then    exportvg egate103vg
        importvg -V 203 -n -y egate103vg ${PV}
        varyonvg egate103vg
        chvg -a n egate103vg
        varyoffvg egate103vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c2cd6" | head -1 | awk '{ print $1 }' )
then    exportvg egate104vg
        importvg -V 204 -n -y egate104vg ${PV}
        varyonvg egate104vg
        chvg -a n egate104vg
        varyoffvg egate104vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c3b01" | head -1 | awk '{ print $1 }' )
then    exportvg egate105vg
        importvg -V 205 -n -y egate105vg ${PV}
        varyonvg egate105vg
        chvg -a n egate105vg
        varyoffvg egate105vg
fi
if PV=$( lspv | grep "^vpath.*000c644ddf34e037" | head -1 | awk '{ print $1 }' )
then    exportvg egate14vg
        importvg -V 114 -n -y egate14vg ${PV}
        varyonvg egate14vg
        chvg -a n egate14vg
        varyoffvg egate14vg
fi
if PV=$( lspv | grep "^vpath.*000c644d227a4992" | head -1 | awk '{ print $1 }' )
then    exportvg egate19vg
        importvg -V 119 -n -y egate19vg ${PV}
        varyonvg egate19vg
        chvg -a n egate19vg
        varyoffvg egate19vg
fi
if PV=$( lspv | grep "^vpath.*000c644d22206319" | head -1 | awk '{ print $1 }' )
then    exportvg egate1vg 
        importvg -V 101 -n -y egate1vg ${PV}
        varyonvg egate1vg
        chvg -a n egate1vg
        varyoffvg egate1vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c4204" | head -1 | awk '{ print $1 }' )
then    exportvg egate2vg
        importvg -V 102 -n -y egate2vg ${PV}
        varyonvg egate2vg   
        chvg -a n egate2vg  
        varyoffvg egate2vg
fi
if PV=$( lspv | grep "^vpath.*000c644db8cf9057" | head -1 | awk '{ print $1 }' )
then    exportvg egate6vg
        importvg -V 106 -n -y egate6vg ${PV}
        varyonvg egate6vg   
        chvg -a n egate6vg  
        varyoffvg egate6vg
fi

Run the "dpovgfix" command on the hot standby node before bringing up the "app80rg1 resource group.

mxfmtxapp83:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mxfmtxapp83:

Start Time:
Duration: clstat smitty clverify

Bring the resource group "app80rg1" up on mxfmtxapp83

mxfmtxapp83:

Start Time:
Duration: smitty cl_resgrp

Perform time zone variable changes on mxfmtxapp80:

mxfmtxapp80:

Start Time:
Duration:

cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

Perform the firmware upgrade on mxfmtxapp80:

mxfmtxapp80:

Start Time:
Duration:

rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

Verify firmware upgrade on mxfmtxapp80:

mxfmtxapp80:

Start Time:
Duration:

lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

Perform Microcode Discovery Survey and check for recommended updates

mxfmtxapp80:

Start Time:
Duration:

invscout

# On local workstation:
scp root@mxfmtxapp80:/var/adm/invscout/mxfmtxapp80.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mxfmtxapp80:/tmp/microcode

Upgrade microcode as necessary:

mxfmtxapp80:

Start Time:
Duration:

cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

Restart HACMP on mxfmtxapp80 and add it back into the cluster:

mxfmtxapp80:

Start Time:
Duration: smitty clstart

Verify cluster configuration and ensure stability

mxfmtxapp80:

Start Time:
Duration: clstat smitty clverify

Notification of node completion and preparation for next node:

mxfmtxapp80:

Start Time:
Duration:

DSAM
EGATE LOS
Ray Meachum

Verify cluster configuration and ensure stability

mxfmtxapp81:

Start Time:
Duration: clstat smitty clverify

Do a "graceful" shutdown of HACMP on mxfmtxapp81

mxfmtxapp81:

Start Time:
Duration: smitty clstop

Run the following importvg commands on mxfmtxapp83 to fix 81s VGs

mxfmtxapp83:

Start Time:
Duration:

# run this loop on mxfmtxapp81 to generate list of command to run on 83
for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mxfmtxapp83 to fix 81s VGs
if PV=$( lspv | grep "^vpath.*000c643d5cdb878f" | head -1 | awk '{ print $1 }' )
then    exportvg egate106vg
        importvg -V 206 -n -y egate106vg ${PV}
        varyonvg egate106vg
        chvg -a n egate106vg
        varyoffvg egate106vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdc5482" | head -1 | awk '{ print $1 }' )
then    exportvg egate107vg
        importvg -V 207 -n -y egate107vg ${PV}
        varyonvg egate107vg
        chvg -a n egate107vg
        varyoffvg egate107vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdcbe3c" | head -1 | awk '{ print $1 }' )
then    exportvg egate108vg
        importvg -V 208 -n -y egate108vg ${PV}
        varyonvg egate108vg
        chvg -a n egate108vg
        varyoffvg egate108vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdd1665" | head -1 | awk '{ print $1 }' )
then    exportvg egate109vg
        importvg -V 209 -n -y egate109vg ${PV}
        varyonvg egate109vg
        chvg -a n egate109vg
        varyoffvg egate109vg
fi
if PV=$( lspv | grep "^vpath.*000c643d5cdded53" | head -1 | awk '{ print $1 }' )
then    exportvg egate110vg
        importvg -V 210 -n -y egate110vg ${PV}
        varyonvg egate110vg
        chvg -a n egate110vg
fi
if PV=$( lspv | grep "^vpath.*000c643d4993bafa" | head -1 | awk '{ print $1 }' )
then    exportvg egate13vg 
        importvg -V 113 -n -y egate13vg ${PV}
        varyonvg egate13vg  
        chvg -a n egate13vg
        varyoffvg egate13vg
fi
if PV=$( lspv | grep "^vpath.*000c643d1ba56485" | head -1 | awk '{ print $1 }' )
then    exportvg egate15vg 
        importvg -V 215 -n -y egate15vg ${PV}
        varyonvg egate15vg  
        chvg -a n egate15vg
        varyoffvg egate15vg
fi
if PV=$( lspv | grep "^vpath.*000c644de4d56fe0" | head -1 | awk '{ print $1 }' )
then    exportvg egate17vg 
        importvg -V 117 -n -y egate17vg ${PV}
        varyonvg egate17vg  
        chvg -a n egate17vg
        varyoffvg egate17vg
fi
if PV=$( lspv | grep "^vpath.*000c644d373f3be3" | head -1 | awk '{ print $1 }' )
then    exportvg egate18vg 
        importvg -V 118 -n -y egate18vg ${PV}
        varyonvg egate18vg  
        chvg -a n egate18vg
        varyoffvg egate18vg
fi
if PV=$( lspv | grep "^vpath.*000cadmxd26c8d2a" | head -1 | awk '{ print $1 }' )
then    exportvg egate22vg
        importvg -V 122 -n -y egate22vg ${PV}
        varyonvg egate22vg  
        chvg -a n egate22vg 
        varyoffvg egate22vg
fi
if PV=$( lspv | grep "^vpath.*000c643d2698e454" | head -1 | awk '{ print $1 }' )
then    exportvg egate8vg
        importvg -V 108 -n -y egate8vg ${PV}
        varyonvg egate8vg   
        chvg -a n egate8vg  
        varyoffvg egate8vg
fi

Run the "dpovgfix" command on the hot standby node before bringing up the "app81rg1 resource group.

mxfmtxapp80:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mxfmtxapp80:

Start Time:
Duration: clstat smitty clverify

Bring the resource group "app81rg1" up on mxfmtxapp80

mxfmtxapp80:

Start Time:
Duration: smitty cl_resgrp

Perform time zone variable changes on mxfmtxapp81:

mxfmtxapp81:

Start Time:
Duration:

cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

Perform the firmware upgrade on mxfmtxapp81:

mxfmtxapp81:

Start Time:
Duration:

rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

Verify firmware upgrade on mxfmtxapp81:

mxfmtxapp81:

Start Time:
Duration:

lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

Perform Microcode Discovery Survey and check for recommended updates

mxfmtxapp81:

Start Time:
Duration:

invscout

# On local workstation:
scp root@mxfmtxapp81:/var/adm/invscout/mxfmtxapp81.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mxfmtxapp81:/tmp/microcode

Upgrade microcode as necessary:

mxfmtxapp81:

Start Time:
Duration:

cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

Restart HACMP on mxfmtxapp81 and add it back into the cluster:

mxfmtxapp81:

Start Time:
Duration: smitty clstart

Verify cluster configuration and ensure stability

mxfmtxapp81:

Start Time:
Duration: clstat smitty clverify

Notification of node completion and preparation for next node:

mxfmtxapp81:

Start Time:
Duration:

DSAM
EGATE LOS
Ray Meachum

Verify cluster configuration and ensure stability

mxfmtxapp82:

Start Time:
Duration: clstat smitty clverify

Do a "graceful" shutdown of HACMP on mxfmtxapp82

mxfmtxapp82:

Start Time:
Duration: smitty clstop

Run the following importvg commands on mxfmtxapp83 to fix 82s VGs

mxfmtxapp83:

Start Time:
Duration:

# run this loop on mxfmtxapp82 to generate commands to run on 83
for vg in $( lsvg -o | sort | grep -v rootvg)
do
  pv=$( lspv | grep ${vg} | grep '^vpath' | head -1 | awk '{ print $1 }' )
  PVID=$( lspv | grep "^${pv} " | head -1 | awk '{ print $2 }' )
  MJNBR=$( ls -l /dev/${vg} | awk '{ print $5 }' )
  MJNBR="${MJNBR%,}"
  print -- "if PV=\$( lspv | grep \"^vpath.*${PVID}\" | head -1 | awk '{ print \$1 }' )"
  print -- "then    exportvg ${vg}"
  print -- "        importvg -V ${MJNBR} -n -y ${vg} \${PV}"
  print -- "        varyonvg ${vg}"
  print -- "        chvg -a n ${vg}"
  print -- "        varyoffvg ${vg}"
  print -- "fi"
done

# run the following importvg commands on mxfmtxapp83 to fix 82s VGs
if PV=$( lspv | grep "^vpath.*000caddd11b64709" | head -1 | awk '{ print $1 }' )
then    exportvg egate100vg
        importvg -V 47 -n -y egate100vg ${PV}
        varyonvg egate100vg
        chvg -a n egate100vg
        varyoffvg egate100vg
fi
if PV=$( lspv | grep "^vpath.*000caddd11b88d31" | head -1 | awk '{ print $1 }' )
then    exportvg egate101vg
        importvg -V 45 -n -y egate101vg ${PV}
        varyonvg egate101vg
        chvg -a n egate101vg
        varyoffvg egate101vg
fi
if PV=$( lspv | grep "^vpath.*000caddd1ccaf18b" | head -1 | awk '{ print $1 }' )
then    exportvg egate16vg
        importvg -V 44 -n -y egate16vg ${PV}
        varyonvg egate16vg
        chvg -a n egate16vg
        varyoffvg egate16vg
fi
if PV=$( lspv | grep "^vpath.*000cadddb8d8f150" | head -1 | awk '{ print $1 }' )
then    exportvg egate21vg
        importvg -V 48 -n -y egate21vg ${PV}
        varyonvg egate21vg
        chvg -a n egate21vg
        varyoffvg egate21vg
fi
if PV=$( lspv | grep "^vpath.*000caddd1da96046" | head -1 | awk '{ print $1 }' )
then    exportvg egate7vg
        importvg -V 78 -n -y egate7vg ${PV}
        varyonvg egate7vg
        chvg -a n egate7vg
        varyoffvg egate7vg
fi

Run the "dpovgfix" command on the hot standby node before bringing up the "app82rg1 resource group.

mxfmtxapp81:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mxfmtxapp81:

Start Time:
Duration: clstat smitty clverify

Bring the resource group "app82rg1" up on mxfmtxapp81

mxfmtxapp81:

Start Time:
Duration: smitty cl_resgrp

Perform time zone variable changes on mxfmtxapp82:

mxfmtxapp82:

Start Time:
Duration:

cd /etc
/tmp/tz.ksh
grep "^TZ=" /etc/environment
ls -l /etc/environment*

# grep output should be "TZ=CST6CDT,M3.2.0,M11.1.0"
# ls output should produce multiple versions of /etc/environment

Perform the firmware upgrade on mxfmtxapp82:

mxfmtxapp82:

Start Time:
Duration:

rm -rf /tmp/fwupdate
mkdir /tmp/fwupdate
cd /tmp/fwupdate
chmod 755 /tmp/microcode/7017S8XF.BIN
/tmp/microcode/7017S8XF.BIN

sum S8xU9.img
# sum output should be 37427 3284

cd /usr/lpp/diagnostics/bin
./update_flash -f /tmp/fwupdate/S8xU9.img
# enter 1 to continue and reboot the system

Verify firmware upgrade on mxfmtxapp82:

mxfmtxapp82:

Start Time:
Duration:

lscfg -vp | grep -i alterable

# output should show system firmware level 20040716 (B)
# output should show service processor firmware level 20040628 (B)

Perform Microcode Discovery Survey and check for recommended updates

mxfmtxapp82:

Start Time:
Duration:

invscout

# On local workstation:
scp root@mxfmtxapp82:/var/adm/invscout/mxfmtxapp82.mup .

# Open a browser window and go to the MDS upload URL:
http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the MDS report to your local disk.
# Upload each recommended update ".rpm" file to mxfmtxapp82:/tmp/microcode

Upgrade microcode as necessary:

mxfmtxapp82:

Start Time:
Duration:

cd /tmp/microcode
rpm --ignoreos -ihv *.rpm

diag

Restart HACMP on mxfmtxapp82 and add it back into the cluster:

mxfmtxapp82:

Start Time:
Duration: smitty clstart

Run the "dpovgfix" command on the hot standby node before bringing up the "app82rg1 resource group.

mxfmtxapp82:

Start Time:
Duration:

for i in $( lsvg | grep -v rootvg )
do
    dpovgfix ${i}
done
lsvpcfg

Verify cluster configuration and ensure stability

mxfmtxapp82:

Start Time:
Duration: clstat smitty clverify

Notification of node completion and outage completion:

mxfmtxapp82:

Start Time:
Duration:

DSAM
EGATE LOS
Ray Meachum

Perform Microcode Discovery Survey on all systems

mxfmtxapp80 / 81 / 82 / 83:

Start Time:
Duration: invscout

Upload invscout survey files to IBM MDS

local workstation:

Start Time:
Duration:

cd /tmp
scp root@mxfmtxapp80:/var/adm/invscout/mxfmtxapp80.mup .
scp root@mxfmtxapp81:/var/adm/invscout/mxfmtxapp81.mup .
scp root@mxfmtxapp82:/var/adm/invscout/mxfmtxapp82.mup .
scp root@mxfmtxapp83:/var/adm/invscout/mxfmtxapp83.mup .

# Open a browser window and go to the MDS upload URL:

http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

# Upload each ".mup" file and save the report to your local disk.
# Then copy each MDS survey report to the MXF NIM server as follows:

scp MDS_mxfmtxapp80_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp80.content.shtml
scp MDS_mxfmtxapp81_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp81.content.shtml
scp MDS_mxfmtxapp82_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp82.content.shtml
scp MDS_mxfmtxapp83_808.html root@mxfapnim01:/tmp/invscout.mup/MDS_mxfmtxapp83.content.shtml

Execute AMMS script to upload MDS reports to mtxdoc web sites

mxfapnim01:

Start Time:
Duration:

print "mxfmtxapp80|mxfmtxapp80_man
mxfmtxapp81|mxfmtxapp81_man
mxfmtxapp82|mxfmtxapp82_man
mxfmtxapp83|mxfmtxapp83_man" > tmp.list

/usr/local/scripts/amms_k93 -vMgrt -W mtxdoc -f ./tmp.list
/usr/local/scripts/amms_k93 -vMgrt -W mxdmtxdoc -f ./tmp.list
/usr/local/scripts/amms_k93 -vMgrt -W mxdmtxdoc -x /srv/www/htdocs/js/invscout -f ./tmp.list

Perform DST TZ Survey on all systems

mxfapnim01:

Start Time:
Duration:

cd /home/s45x
./dstnew.433.ksh mxf

scp dstmxftz*.content.shtml root@mxdmtxdoc:/srv/www/htdocs/js/GlobalSysAdmin/Power5

Close Bridge Line

DST EGATE Upgrade

LPAR Leasing
Lease an AIX / i5 LPAR
Reduce your costs
www.mtxia.com

Server Leasing
Lease a Server off-site
Reduce your costs
www.mtxia.com

Data Center Automation
Business Continuity and DR
Virtualization/Consolidation
www.mtxia.com

HMC Service
Hardware Management Console
Manage Remote AIX / i5 LPARs
www.siteox.com

Business Web Site Hosting
$3.99 / month includes Tools,
Shopping Cart, Site Builder
www.siteox.com

FREE Domain Registration
included with Web Site Hosting
Tools, Social Networking, Blog
www.siteox.com

Disaster Recovery
Small Business Oriented
Off-Site Facilities
www.mtxia.com

IBM pSeries / iSeries
Reduce your Costs
Off-Site Server Hosting
www.mtxia.com