The following document describes the procedure used to move the EGATE
Pre-Production database environment from the temporary Proof-of-Concept
architecture running on the p590, back to the original 6M2 platforms,
then permanently back to the p590 with the Production Disaster Recovery
provisions in place.
- The first phase of this process is to shutdown the EGATE
database environment on the POC platform. The purpose of this phase is
to quantify the time required to perform each step of this process. The
timing information is needed for planning the movement of the EGATE
Production environment to the new Power 5 architecture.
- Restart the EGATE Pre-Prod environment on the Power 4
architecture and validate. Again the purpose is to quantifiy the time
required to perform this phase.
- Shutdown the EGATE Pre-Prod environment on the Power 4
architecture, modify the Power 5 systems to EXACTLY replicate the Power
4 systems.
- Restart the Pre-prod environment on the Power 5 systems.
- Perform Disaster Recovery test of Production environment onto
the new Power 5 Architecture.
- Deconfigure production environment on Power 5 architecture and
restart the pre-prod environment
Phase 1
Phase 1 Start Time: 12:00
Phase 1 Duration: 30m
- Generate HDLM SAN Disk identification document on current pre-prod
power 5 nodes. Save this information to a file and copy the file to an
accessible location.
ddcaaega01:
Start Time: 12:00
Duration: 1m
cd /usr/DynamicLinkManager/bin
lspv | grep dlmfdrv | grep -v dlmfdrvio |
while read -- HDISK PVID VGNAME STATE
do
MAJOR=""
LOC=""
ls -l /dev/${VGNAME} 2>/dev/null |
IFS="${IFS}," read -- PERMS LINKS OWNER GROUP MAJOR MINOR REMAIN
print -- "\n${HDISK} ${PVID} ${VGNAME} ${STATE} ${MAJOR:+VGmajor#:${MAJOR}}"
/usr/DynamicLinkManager/bin/dlnkmgr view -drv |
grep " ${HDISK} " |
while read -r -- LINE
do
print -r -- "${LINE}"
print -r -- "${LINE}" | read -r -- DNBR DLMDISK DDISK LDEV
LOC="${LOC}$( lscfg -l ${DDISK} | awk '{ print $2, $1 }' )\\n"
done
print -- "${LOC}"
done
ddcaaega02:
Start Time: 12:00
Duration: 1m
cd /usr/DynamicLinkManager/bin
lspv | grep dlmfdrv | grep -v dlmfdrvio |
while read -- HDISK PVID VGNAME STATE
do
MAJOR=""
LOC=""
ls -l /dev/${VGNAME} 2>/dev/null |
IFS="${IFS}," read -- PERMS LINKS OWNER GROUP MAJOR MINOR REMAIN
print -- "\n${HDISK} ${PVID} ${VGNAME} ${STATE} ${MAJOR:+VGmajor#:${MAJOR}}"
/usr/DynamicLinkManager/bin/dlnkmgr view -drv |
grep " ${HDISK} " |
while read -r -- LINE
do
print -r -- "${LINE}"
print -r -- "${LINE}" | read -r -- DNBR DLMDISK DDISK LDEV
LOC="${LOC}$( lscfg -l ${DDISK} | awk '{ print $2, $1 }' )\\n"
done
print -- "${LOC}"
done
- Wait for database shutdown
ddcaaega01:
Start Time: 12:01
Duration: 20m
ddcaaega02:
Start Time: 12:01
Duration: 20m
- Shutdown HACMP on EGATE database POC power 5 machines and varyoff
all non-"rootvg" volume groups
ddcaaega01:
Start Time: 12:20
Duration: 2m
smitty clstop
umount all
for VG in $( lsvg -o )
do
[[ ${VG} = "rootvg" ]] && continue
varyoffvg ${VG}
done
ddcaaega02:
Start Time: 12:20
Duration: 2m
smitty clstop
umount all
for VG in $( lsvg -o )
do
[[ ${VG} = "rootvg" ]] && continue
varyoffvg ${VG}
done
- Shutdown EGATE database POC power 5 machines
ddcaaega01:
Start Time: 12:25
Duration: 5m
shutdown -Fh
ddcaaega02:
Start Time: 12:25
Duration: 5m
shutdown -Fh
- Contact Doug Lemons to make DNS changes
Start Time: 12:26
Duration: 10m
CNAME: point ddcaaega01 to the power 4 architecture machine
CNAME: point ddcaaega02 to the power 4 architecture machine
- Start Power 4 systems
ddcaaega01:
Start Time: 12:28
Duration: 5m
shutdown -Fh
ddcaaega02:
Start Time: 12:30
Duration: 5m
shutdown -Fh
Phase 2
- Restart the EGATE database pre-production environment on the
Power 4 architecture.
Phase 2 Start Time: 12:30
Phase 2 Duration: 55m
- Clear Hitachi persistant reservations
ddcaaega01:
Start Time: 12:35
Duration: 15m
cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
print y | ./dlmpr -c ${i}
done
ddcaaega02: 12:35
Start Time: 12:35
Duration: 15m
cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
print y | ./dlmpr -c ${i}
done
- All disks should already be visible and no configuration changes
should be necessary. Restart HACMP on each power 4 node, one node at a
time.
ddcaaega01:
Start Time: 12:52
Duration: 5m
smitty clstart
ddcaaega02:
Start Time: 12:57
Duration: 5m
smitty clstart
- Turnover pre-production power 4 systems to database team to
complete the startup process
ddcaaega01:
Start Time: 13:03
Duration: 25m
ddcaaega02:
Start Time: 13:03
Duration: 25m
Phase 3
- Shutdown the EGATE Pre-Prod environment on the Power 4
architecture, modify the Power 5 systems to EXACTLY replicate the Power
4 systems.
Phase 3 Start Time: 13:25
Phase 3 Duration: 170m (2h 50m)
-
ddcaaega01:
Start Time: 13:07
Duration: 5m
ddcaaega02:
Start Time: 13:07
Duration: 5m
- Shutdown EGATE pre-prod database on power 4 systems
ddcaaega01:
Start Time: 13:25
Duration: 7m
ddcaaega02:
Start Time: 13:25
Duration: 7m
- Shutdown HACMP on EGATE pre-prod power 4 systems
ddcaaega01:
Start Time: 13:32
Duration: 5m
smitty clstop
ddcaaega02:
Start Time: 13:37
Duration: 5m
smitty clstop
- Varyoff any remaining non-rootvg volume groups on the Power 4 systems
ddcaaega01:
Start Time: 13:38
Duration: 1m
ddcaaega02:
Start Time: 13:38
Duration: 1m
- Shutdown EGATE database POC power 4 machines
ddcaaega01:
Start Time: 13:39
Duration: 5m
shutdown -Fh
ddcaaega02:
Start Time: 13:39
Duration: 5m
shutdown -Fh
- Change IP addresses of the Power 5 systems to be the original IP's from
the Power 4 system.
ddcpocega01:
Start Time: 13:12
Duration: 48m
Service: 146.61.64.29
Boot: 146.61.64.30
Standby: 192.168.30.1
man: 146.61.68.23
ddcpocega02:
Start Time: 13:12
Duration: 48m
Service: 146.61.64.32
Boot: 146.61.64.44
Standby: 192.168.30.2
man: 146.61.68.41
- Change the host names of the Power 5 systems to be the original
names from the Power 4 systems.
ddcpocega01:
Start Time: 14:01
Duration: 1m
chdev -l inet0 -a hostname=ddcaaega01
ddcpocega02:
Start Time: 14:02
Duration: 1m
chdev -l inet0 -a hostname=ddcaaega02
-
- Restore the /etc/hosts and /.rhosts files from the original Power 4
system from the DR files
ddcaaega01:
Start Time: 14:04
Duration: 1m
/usr/lpp/dr/ddcaaega01_man/uueConfig/!etc!hosts.uue
/usr/lpp/dr/ddcaaega01_man/uueConfig/!.rhosts.uue
ddcaaega02:
Start Time: 14:05
Duration: 1m
/usr/lpp/dr/ddcaaega02_man/uueConfig/!etc!hosts.uue
/usr/lpp/dr/ddcaaega02_man/uueConfig/!.rhosts.uue
- Run configuration manager to pull in all SAN disks
ddcaaega01:
Start Time: 14:21
Duration: 3m
cfgmgr
ddcaaega02:
Start Time: 14:24
Duration: 3m
cfgmgr
- Clear persistant reserves on all SAN disks
ddcaaega01:
Start Time: 14:24
Duration: 9m
cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
print y | ./dlmpr -c ${i}
done
ddcaaega02:
Start Time: 14:26
Duration: 9m
cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
print y | ./dlmpr -c ${i}
done
- Remove all SAN disks and rediscover to obtain PVID's
ddcaaega01:
Start Time: 14:33
Duration: 5m
for i in $( lspv | awk '{ print $1 }' )
do
rmdev -Rdl ${i}
done
cfgmgr
ddcaaega02:
Start Time: 14:35
Duration: 5m
for i in $( lspv | awk '{ print $1 }' )
do
rmdev -Rdl ${i}
done
cfgmgr
- Check SAN disks against list of PVID's then import VG's using OCIDR.
ddcaaega01:
Start Time: 14:40
Duration: 15m
cd /usr/lpp/dr/ddcaaega01
./OCIDR -I config
ddcaaega02:
Start Time: 14:40
Duration: 15m
cd /usr/lpp/dr/ddcaaega02
./OCIDR -I config
- Reconfigure HACMP to use the original node names, resource group
names, and application server names. Then verify and synchronize.
ddcaaega01:
Start Time: 14:41
Duration: 55m
Node Name: ddcaaega01
Node Name: ddcaaega02
Resource Group: orcon1
Resource Group: ddcaaega01_ip
Resource Group: ddcaaega02_ip
Application Server: dpcheck_01
Application Start Script: /usr/local/scripts/start_dpcheck_01.ksh
Application Start Script: /usr/local/scripts/stop_dpcheck_01.ksh
Application Server: dpcheck_02
Application Start Script: /usr/local/scripts/start_dpcheck_02.ksh
Application Start Script: /usr/local/scripts/stop_dpcheck_02.ksh
Application Server: ddcaaega01_tsm
Application Start Script: /usr/local/hascripts/start_tsm.ksh
Application Start Script: /usr/local/hascripts/stop_tsm.ksh
Application Server: ddcaaega02_tsm
Application Start Script: /usr/local/hascripts/start_tsm.ksh
Application Start Script: /usr/local/hascripts/stop_tsm.ksh
- Varyon the non-concurrent volume groups and set to automatically varyon
at boot time.
ddcaaega01:
Start Time: 15:31
Duration: 1m
varyonvg oradb01vg01
chvg -a y oradb01vg01
varyonvg oradb01vg02
chvg -a y oradb01vg02
ddcaaega02:
Start Time: 15:33
Duration: 1m
varyonvg oradb02vg01
chvg -a y oradb02vg01
varyonvg oradb02vg02
chvg -a y oradb02vg02
- Set the database filesystems to automatically mount at boot time.
ddcaaega01:
Start Time: 15:33
Duration: 1m
chfs -A y /u01001
chfs -A y /u01002
chfs -A y /backup
ddcaaega02:
Start Time: 15:34
Duration: 1m
chfs -A y /u01001
chfs -A y /u01002
chfs -A y /backup
- Modify the /etc/hosts file to reflect the new hostnames
ddcaaega01:
Start Time: 15:35
Duration: 1m
vi /etc/hosts
ddcaaega02:
Start Time: 15:36
Duration: 1m
vi /etc/hosts
- Sync and verify, then start the HACMP configuration.
ddcaaega01:
Start Time: 15:30
Duration: 5m
smitty clstart
ddcaaega02:
Start Time: 15:35
Duration: 5m
smitty clstart
- Change ownership of oracle LV's
ddcaaega01:
Start Time: 15:40
Duration: 1m
chown oracle:dba /dev/*orarp*
ddcaaega02:
Start Time: 15:41
Duration: 1m
chown oracle:dba /dev/*orarp*
- Change the number of processes per user to 3000
ddcaaega01:
Start Time: 15:42
Duration: 1m
chdev -l sys0 -a maxuproc=3000
ddcaaega02:
Start Time: 15:43
Duration: 1m
chdev -l sys0 -a maxuproc=3000
- Mount the first Oracle CD from the NIM server and run the
"rootpre.sh" script to set permissions and such.
mdcapnim01:
Start Time: 15:44
Duration: 1m
exportfs -i -o 'root=*' /export/cdimages/Oracle8.1.7_Disk1
ddcaaega01:
Start Time: 15:45
Duration: 1m
mount mdcapnim01:/export/cdimages/Oracle8.1.7_Disk1 /mnt
cd /mnt
./rootpre.sh
cd /tmp
umount /mnt
ddcaaega02:
Start Time: 15:46
Duration: 1m
mount mdcapnim01:/export/cdimages/Oracle8.1.7_Disk1 /mnt
cd /mnt
./rootpre.sh
cd /tmp
umount /mnt
mdcapnim01:
Start Time: 15:47
Duration: 1m
exportfs -u /export/cdimages/Oracle8.1.7_Disk1
- Notify database and application group the machines are ready
Start Time: 15:48
Duration: 25m
- Contact Doug Lemons to make DNS changes
Start Time: 15:48
Duration: 25m
CNAME: point ddcaaega01 to the power 5 architecture machine
CNAME: point ddcaaega02 to the power 5 architecture machine
Phase 4
- Perform Disaster Recovery test of the production
EGATE database servers
Phase 4 Start Time: 16:15 - 18:55
Phase 4 Duration: 160m (2h 40m)
- Login to ddcaaega01 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines. The HMC client can be
downloaded from the following url:
http://ddclahmc01.tu.com/remote_client.html
- Notify Users the DR is about to occur
$ wall
A disaster has occurred and the pre-production environment is
being reconfigured as production NOW. Please LOG OFF.
- Change to the DR scripts directory for the target server:
$ cd /usr/lpp/dr/mdctxudb85
- Execute the deconfiguration phase of the DR. The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).
Start Time: 16:18
Duration: 77m
$ ./OCIDR -hkemndv deconfig
- Check to see that HACMP is down, user processes were killed,
all non-rootvg file systems were unmounted,
all non-static volume groups were varied off and exported. The
remaining volume groups may include one or more of the following:
rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
- Execute the configuration phase of the DR and import the volume
groups. The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).
Start Time: 17:35
Duration: 1m
$ ./OCIDR -IGmNraoduv config
- Check to see that all volume groups were imported and varied on,
all file systems were mounted,
network addresses were changed,
IQ directories were created,
configuration scripts were installed,
and background daemons were refreshed.
- The Oracle parallel server requires the disks and logical volumes
be owned by the oracle user. Execute the following script to change the
disk ownerships:
Start Time: 17:36
Duration: 1m
$ /usr/lpp/dr/helpers/fixForOracle.sh
- After HACMP startup is complete, initialize the Oracle environment
by running the following script:
Start Time: 17:36
Duration: 1m
$ mount ddcapnim01:/export/cdimages /mnt
$ cd /mnt/Oracle8i/CD1
# DO NOT SKIP
$ ./rootpre.sh
- After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:
Start Time: 17:44
Duration: 30m
$ smitty hacmp -> Cluster Services -> Start Cluster
Services
- Login to ddcaaega02 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines. The HMC client can be
downloaded from the following url:
http://ddclahmc01.tu.com/remote_client.html
- Notify Users the DR is about to occur
$ wall
A disaster has occurred and the pre-production environment is
being reconfigured as production NOW. Please LOG OFF.
- Change to the DR scripts directory for the target server:
$ cd /usr/lpp/dr/mdctxudb86
- Execute the deconfiguration phase of the DR. The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).
Start Time: 16:18
Duration: 61m
$ ./OCIDR -hkemndv deconfig
- Check to see that HACMP is down, user processes were killed,
all non-rootvg file systems were unmounted,
all non-static volume groups were varied off and exported. The
remaining volume groups may include one or more of the following:
rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
- Execute the configuration phase of the DR and import the volume
groups. The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).
$ ./OCIDR -IGmNraoduv config
- Check to see that all volume groups were imported and varied on,
all file systems were mounted,
network addresses were changed,
IQ directories were created,
configuration scripts were installed,
and background daemons were refreshed.
- The Oracle parallel server requires the disks and logical volumes
be owned by the oracle user. Execute the following script to change the
disk ownerships:
Start Time: 17:19
Duration: 2m
$ /usr/lpp/dr/helpers/fixForOracle.sh
- After HACMP startup is complete, initialize the Oracle environment
by running the following script:
Start Time: 17:21
Duration: 2m
$ mount ddcapnim01:/export/cdimages /mnt
$ cd /mnt/Oracle8i/CD1
# DO NOT SKIP
$ ./rootpre.sh
- After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:
Start Time: 17:47
Duration: 30m
$ smitty hacmp -> Cluster Services -> Start Cluster
Services
- Notify the Oracle database team the server is ready.
Start Time: 18:17
Duration: 40m
Phase 5
- Deconfigure the Production EGATE database
environment from the Power 5 systems and restore the Pre-Production
EGATE database environment.
Phase 5 Start Time: 18:55
Phase 5 Duration: 30m
- Login to ddcaaega01 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines. The HMC client can be
downloaded from the following url:
http://ddclahmc01.tu.com/remote_client.html
- Change to the DR scripts directory for the target server:
$ cd /usr/lpp/dr/ddcaaega01
- Execute the deconfiguration phase of the DR. The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).
Start Time: 18:57
Duration: 5m
$ ./OCIDR -hkemndv deconfig
- Check to see that HACMP is down, user processes were killed,
all non-rootvg file systems were unmounted,
all non-static volume groups were varied off and exported. The
remaining volume groups may include one or more of the following:
rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
- Execute the configuration phase of the DR and import the volume
groups. The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).
Start Time: 19:02
Duration: 1m
$ ./OCIDR -IGmNraoduv config
- Check to see that all volume groups were imported and varied on,
all file systems were mounted,
network addresses were changed,
IQ directories were created,
configuration scripts were installed,
and background daemons were refreshed.
- The Oracle parallel server requires the disks and logical volumes
be owned by the oracle user. Execute the following script to change the
disk ownerships:
Start Time: 19:03
Duration: 1m
$ /usr/lpp/dr/helpers/fixForOracle.sh
- After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:
Start Time: 19:04
Duration: 4m
$ smitty hacmp -> Cluster Services -> Start Cluster
Services
- After HACMP startup is complete, initialize the Oracle environment
by running the following script:
Start Time: This step not executed
Duration: 0m
$ mount ddcapnim01:/export/cdimages /mnt
$ cd /mnt/Oracle8i/CD1
# DO NOT SKIP
$ ./rootpre.sh
- Login to ddcaaega01 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines. The HMC client can be
downloaded from the following url:
http://ddclahmc01.tu.com/remote_client.html
- Change to the DR scripts directory for the target server:
$ cd /usr/lpp/dr/ddcaaega02
- Execute the deconfiguration phase of the DR. The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).
Start Time: 18:57
Duration: 3m
$ ./OCIDR -hkemndv deconfig
- Check to see that HACMP is down, user processes were killed,
all non-rootvg file systems were unmounted,
all non-static volume groups were varied off and exported. The
remaining volume groups may include one or more of the following:
rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
- Execute the configuration phase of the DR and import the volume
groups. The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).
Start Time: 19:00
Duration: 2m
$ ./OCIDR -IGmNraoduv config
- Check to see that all volume groups were imported and varied on,
all file systems were mounted,
network addresses were changed,
IQ directories were created,
configuration scripts were installed,
and background daemons were refreshed.
- The Oracle parallel server requires the disks and logical volumes
be owned by the oracle user. Execute the following script to change the
disk ownerships:
Start Time: 19:02
Duration: 1m
$ /usr/lpp/dr/helpers/fixForOracle.sh
- After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:
Start Time: 19:08
Duration: 4m
$ smitty hacmp -> Cluster Services -> Start Cluster Services
- After HACMP startup is complete, initialize the Oracle environment
by running the following script:
Start Time: This step not executed
Duration: 0m
$ mount ddcapnim01:/export/cdimages /mnt
$ cd /mnt/Oracle8i/CD1
# DO NOT SKIP
$ ./rootpre.sh