The following document describes the procedure used to move the EGATE Production database environment from the MDC data center to the Disaster Recovery provisions at DDC. The DR provisions at DDC normally run the Egate pre-production environment.

1. The first phase of this process is to shutdown the pre-production EGATE database and start the production EGATE database environment in the DR provision.
2. The second phase of this process is to shutdown the production EGATE database in the DR provision and restart the pre-production EGATE database environment.

Phase 1

Phase 1 Start Time:
Phase 1 Duration:

1. Contact SAN group to turn off the Tru-Copy mirroring of the production EGATE database drives.

   True Copy Procedure:

   • Login to Trucopy
   • Run the following command:

     pairsplit -S -g tc_dbXX
     

     ( XX - database to split with read/write access )
   • Notify Unix Team when done

   mdcapega01:

   
   Start Time:
   Duration:

   Mike Wilson Cell Phone: 972-249-8494
   

   mdcapega02:

   
   Start Time:
   Duration:

   Mike Wilson Cell Phone: 972-249-8494
   

2. Login to ddcaaega01 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
   

   http://ddclahmc01.tu.com/remote_client.html

3. Notify Users the DR is about to occur
   
   wall
A disaster has occurred and the pre-production environment is
being reconfigured as production NOW. Please LOG OFF.



4. Change to the DR scripts directory for the target server:
   
   cd /usr/lpp/dr/mdcapega01
   
   
5. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).

   
   Start Time:
   Duration:

   ./OCIDR -hkemndv deconfig
   
   
6. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
7. Clear persistant reserves on all SAN disks

   ddcaaega01:

   
   Start Time:
   Duration:

   cd /usr/DynamicLinkManager/bin
   for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
   do
       print y | ./dlmpr -c ${i}
   done
   

   ddcaaega02:

   
   Start Time:
   Duration:

   cd /usr/DynamicLinkManager/bin
   for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
   do
       print y | ./dlmpr -c ${i}
   done
   

8. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).

   
   Start Time:
   Duration:

   ./OCIDR -IGmNraoduv config
   
   for i in $( lsvg )
   do
       varyonvg -c ${i}
   done
   

   
   
   
9. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
10. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:

    
    Start Time:
    Duration:

    /usr/lpp/dr/helpers/fixForOracle.sh
    
    
11. After HACMP startup is complete, initialize the Oracle environment by running the following script:

    
    Start Time:
    Duration:

    mount ddcapnim01:/export/cdimages/Oracle8i/CD1 /mnt
cd /mnt # DO NOT SKIP
    ./rootpre.sh
    
    
12. Restore the /etc/hosts and /.rhosts files from the EGATE production environment

    ddcaaega01:

    
    Start Time:
    Duration:

    /usr/lpp/dr/ddcaaega01/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/ddcaaega01/uueConfig/!.rhosts.uue
    
    

    ddcaaega02:

    
    Start Time:
    Duration:

    /usr/lpp/dr/ddcaaega02/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/ddcaaega02/uueConfig/!.rhosts.uue
    
    

13. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:

    
    Start Time:
    Duration:

    smitty hacmp -> Cluster Services -> Start Cluster Services
    
    
14. Login to ddcaaega02 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
    

    http://ddclahmc01.tu.com/remote_client.html

15. Notify Users the DR is about to occur
    
    wall
A disaster has occurred and the pre-production environment is
being reconfigured as production NOW. Please LOG OFF.



16. Change to the DR scripts directory for the target server:
    
    cd /usr/lpp/dr/mdcapega01
    
    
17. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).

    
    Start Time:
    Duration:

    ./OCIDR -hkemndv deconfig
    
    
18. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
19. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).
    
    

    ./OCIDR -IGmNraoduv config
    
    for i in $( lsvg )
    do
        varyonvg -c ${i}
    done
    

20. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
21. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:

    
    Start Time:
    Duration:

    /usr/lpp/dr/helpers/fixForOracle.sh
    
    
22. After HACMP startup is complete, initialize the Oracle environment by running the following script:

    
    Start Time:
    Duration:

    mount ddcapnim01:/export/cdimages/Oracle8i/CD1 /mnt
cd /mnt # DO NOT SKIP
    ./rootpre.sh
    
    
23. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:

    
    Start Time:
    Duration:

    smitty hacmp -> Cluster Services -> Start Cluster Services
    
    
24. Notify the Oracle database team the server is ready.

    
    Start Time:
    Duration:

    
    
    

Phase 2

Phase 2 Start Time:
Phase 2 Duration:

1. Login to ddcaaega01 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
   

   http://ddclahmc01.tu.com/remote_client.html

2. Change to the DR scripts directory for the target server:
   
   cd /usr/lpp/dr/ddcaaega01
   
   
3. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).

   
   Start Time:
   Duration:

   ./OCIDR -hkemndv deconfig
   
   
4. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
5. Clear persistant reserves on all SAN disks

   ddcaaega01:

   
   Start Time:
   Duration:

   cd /usr/DynamicLinkManager/bin
   for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
   do
       print y | ./dlmpr -c ${i}
   done
   

   ddcaaega02:

   
   Start Time:
   Duration:

   cd /usr/DynamicLinkManager/bin
   for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
   do
       print y | ./dlmpr -c ${i}
   done
   

6. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).

   
   Start Time:
   Duration:

   ./OCIDR -IGmNraoduv config
   
   
7. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
8. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:

   
   Start Time:
   Duration:

   /usr/lpp/dr/helpers/fixForOracle.sh
   
   
9. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:

   
   Start Time:
   Duration:

   smitty hacmp -> Cluster Services -> Start Cluster Services
   
   
10. After HACMP startup is complete, initialize the Oracle environment by running the following script:

    
    Start Time: This step not executed
    Duration:

    mount ddcapnim01:/export/cdimages/Oracle8i/CD1 /mnt
cd /mnt # DO NOT SKIP
    ./rootpre.sh
    
    
11. Login to ddcaaega01 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
    

    http://ddclahmc01.tu.com/remote_client.html

12. Change to the DR scripts directory for the target server:
    
    cd /usr/lpp/dr/ddcaaega02
    
    
13. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).

    
    Start Time:
    Duration:

    ./OCIDR -hkemndv deconfig
    
    
14. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
15. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).

    
    Start Time:
    Duration:

    ./OCIDR -IGmNraoduv config
    
    
16. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
17. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:

    
    Start Time:
    Duration:

    /usr/lpp/dr/helpers/fixForOracle.sh
    
    
18. Restore the /etc/hosts and /.rhosts files from the EGATE pre-production environment

    ddcaaega01:

    
    Start Time:
    Duration:

    /usr/lpp/dr/ddcaaega01/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/ddcaaega01/uueConfig/!.rhosts.uue
    
    

    ddcaaega02:

    
    Start Time:
    Duration:

    /usr/lpp/dr/ddcaaega02/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/ddcaaega02/uueConfig/!.rhosts.uue
    
    

19. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:

    
    Start Time:
    Duration:

    smitty hacmp -> Cluster Services -> Start Cluster Services
    
    
20. After HACMP startup is complete, initialize the Oracle environment by running the following script:

    
    Start Time: This step not executed
    Duration: 0m

    mount ddcapnim01:/export/cdimages/Oracle8i/CD1 /mnt
cd /mnt # DO NOT SKIP
    ./rootpre.sh
    
    
21. Contact SAN group to restart the Tru-Copy mirroring of the production EGATE database drives.

    True Copy Procedure:

    • Login to Trucopy server
    • Run the following command:

      paircreate -g tc_dbXX
      

      ( XX - database to sync with read/write access )
    • Notify Unix Team when done

    mdcapega01:

    
    Start Time:
    Duration:

    Mike Wilson Cell Phone: 972-249-8494
    

    mdcapega02:

    
    Start Time:
    Duration:

    Mike Wilson Cell Phone: 972-249-8494