The following document describes the procedure used to move the EGATE Production database environment from the Mt Xia Franklin (MXF)data center to the Disaster Recovery provisions at Mt Xia Dallas (MXD) Data Center. The DR provisions at MXD normally run the Egate pre-production environment.

  1. The first phase of this process is to shutdown the pre-production EGATE database and start the production EGATE database environment in the DR provision.
  2. The second phase of this process is to shutdown the production EGATE database in the DR provision and restart the pre-production EGATE database environment.


Phase 1

- Shutdown the pre-production EGATE database and start the production EGATE database environment in the DR provision.


Phase 1 Start Time:
Phase 1 Duration:


  1. Contact SAN group to turn off the Tru-Copy mirroring of the production EGATE database drives.

    True Copy Procedure:

    mxfapega01:


    Start Time:
    Duration:
    Mike Wilson Cell Phone: 972-249-8494
    

    mxfapega02:


    Start Time:
    Duration:
    Mike Wilson Cell Phone: 972-249-8494
    

  2. Login to mxdaaega01 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
    http://mxdlahmc01.tu.com/remote_client.html
  3. Notify Users the DR is about to occur

    wall
    A disaster has occurred and the pre-production environment is
    being reconfigured as production NOW. Please LOG OFF.



  4. Change to the DR scripts directory for the target server:

    cd /usr/lpp/dr/mxfapega01

  5. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).


    Start Time:
    Duration:

    ./OCIDR -hkemndv deconfig

  6. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
  7. Clear persistant reserves on all SAN disks

    mxdaaega01:


    Start Time:
    Duration:
    cd /usr/DynamicLinkManager/bin
    for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
    do
        print y | ./dlmpr -c ${i}
    done
    

    mxdaaega02:


    Start Time:
    Duration:
    cd /usr/DynamicLinkManager/bin
    for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
    do
        print y | ./dlmpr -c ${i}
    done
    

  8. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).


    Start Time:
    Duration:
    ./OCIDR -IGmNraoduv config
    
    for i in $( lsvg )
    do
        varyonvg -c ${i}
    done
    



  9. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
  10. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:


    Start Time:
    Duration:

    /usr/lpp/dr/helpers/fixForOracle.sh

  11. After HACMP startup is complete, initialize the Oracle environment by running the following script:


    Start Time:
    Duration:

    mount mxdapnim01:/export/cdimages/Oracle8i/CD1 /mnt
    cd /mnt # DO NOT SKIP
    ./rootpre.sh

  12. Restore the /etc/hosts and /.rhosts files from the EGATE production environment

    mxdaaega01:


    Start Time:
    Duration:
    /usr/lpp/dr/mxdaaega01/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/mxdaaega01/uueConfig/!.rhosts.uue
    
    

    mxdaaega02:


    Start Time:
    Duration:
    /usr/lpp/dr/mxdaaega02/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/mxdaaega02/uueConfig/!.rhosts.uue
    
    

  13. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:


    Start Time:
    Duration:

    smitty hacmp -> Cluster Services -> Start Cluster Services

  14. Login to mxdaaega02 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
    http://mxdlahmc01.tu.com/remote_client.html
  15. Notify Users the DR is about to occur

    wall
    A disaster has occurred and the pre-production environment is
    being reconfigured as production NOW. Please LOG OFF.



  16. Change to the DR scripts directory for the target server:

    cd /usr/lpp/dr/mxfapega01

  17. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).


    Start Time:
    Duration:

    ./OCIDR -hkemndv deconfig

  18. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
  19. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).

    ./OCIDR -IGmNraoduv config
    
    for i in $( lsvg )
    do
        varyonvg -c ${i}
    done
    

  20. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
  21. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:


    Start Time:
    Duration:

    /usr/lpp/dr/helpers/fixForOracle.sh

  22. After HACMP startup is complete, initialize the Oracle environment by running the following script:


    Start Time:
    Duration:

    mount mxdapnim01:/export/cdimages/Oracle8i/CD1 /mnt
    cd /mnt # DO NOT SKIP
    ./rootpre.sh

  23. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:


    Start Time:
    Duration:

    smitty hacmp -> Cluster Services -> Start Cluster Services

  24. Notify the Oracle database team the server is ready.


    Start Time:
    Duration:






Phase 2

- Shutdown the production EGATE database in the DR provision and restart the pre-production EGATE database environment.


Phase 2 Start Time:
Phase 2 Duration:


  1. Login to mxdaaega01 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
    http://mxdlahmc01.tu.com/remote_client.html
  2. Change to the DR scripts directory for the target server:

    cd /usr/lpp/dr/mxdaaega01

  3. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).


    Start Time:
    Duration:

    ./OCIDR -hkemndv deconfig

  4. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
  5. Clear persistant reserves on all SAN disks

    mxdaaega01:


    Start Time:
    Duration:
    cd /usr/DynamicLinkManager/bin
    for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
    do
        print y | ./dlmpr -c ${i}
    done
    

    mxdaaega02:


    Start Time:
    Duration:
    cd /usr/DynamicLinkManager/bin
    for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
    do
        print y | ./dlmpr -c ${i}
    done
    

  6. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).


    Start Time:
    Duration:

    ./OCIDR -IGmNraoduv config

  7. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
  8. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:


    Start Time:
    Duration:

    /usr/lpp/dr/helpers/fixForOracle.sh

  9. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:


    Start Time:
    Duration:

    smitty hacmp -> Cluster Services -> Start Cluster Services

  10. After HACMP startup is complete, initialize the Oracle environment by running the following script:


    Start Time: This step not executed
    Duration:

    mount mxdapnim01:/export/cdimages/Oracle8i/CD1 /mnt
    cd /mnt # DO NOT SKIP
    ./rootpre.sh

  11. Login to mxdaaega01 on the console as root. You will need to install and run the Hardware Management Console (HMC) client to be able to access the console of these machines. The HMC client can be downloaded from the following url:
    http://mxdlahmc01.tu.com/remote_client.html
  12. Change to the DR scripts directory for the target server:

    cd /usr/lpp/dr/mxdaaega02

  13. Execute the deconfiguration phase of the DR. The options shown indicate the deconfiguration should kill all processes (k), varyoff and export VGs (e), unmount filesystems (m), deconfigure the network adapters (n), delete any existing IQ directories (d), and to do it all in verbose mode (v).


    Start Time:
    Duration:

    ./OCIDR -hkemndv deconfig

  14. Check to see that HACMP is down, user processes were killed, all non-rootvg file systems were unmounted, all non-static volume groups were varied off and exported. The remaining volume groups may include one or more of the following: rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
  15. Execute the configuration phase of the DR and import the volume groups. The options shown indicate the configuration should import the VGs, LVs, and FSs (i), mount the filesystems (m), configure the network adapters in HACMP mode (N), install the configuration files (r), start the applications (a), create the home directories (o), create the IQ directories (d), change the hostname (u), and to do it all in verbose mode (v).


    Start Time:
    Duration:

    ./OCIDR -IGmNraoduv config

  16. Check to see that all volume groups were imported and varied on, all file systems were mounted, network addresses were changed, IQ directories were created, configuration scripts were installed, and background daemons were refreshed.
  17. The Oracle parallel server requires the disks and logical volumes be owned by the oracle user. Execute the following script to change the disk ownerships:


    Start Time:
    Duration:

    /usr/lpp/dr/helpers/fixForOracle.sh

  18. Restore the /etc/hosts and /.rhosts files from the EGATE pre-production environment

    mxdaaega01:


    Start Time:
    Duration:
    /usr/lpp/dr/mxdaaega01/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/mxdaaega01/uueConfig/!.rhosts.uue
    
    

    mxdaaega02:


    Start Time:
    Duration:
    /usr/lpp/dr/mxdaaega02/uueConfig/!etc!hosts.uue
    /usr/lpp/dr/mxdaaega02/uueConfig/!.rhosts.uue
    
    

  19. After the OCIDR script has completed on ALL Oracle Parallel Server nodes, start HACMP on this node:


    Start Time:
    Duration:

    smitty hacmp -> Cluster Services -> Start Cluster Services

  20. After HACMP startup is complete, initialize the Oracle environment by running the following script:


    Start Time: This step not executed
    Duration: 0m

    mount mxdapnim01:/export/cdimages/Oracle8i/CD1 /mnt
    cd /mnt # DO NOT SKIP
    ./rootpre.sh

  21. Contact SAN group to restart the Tru-Copy mirroring of the production EGATE database drives.

    True Copy Procedure:

    mxfapega01:


    Start Time:
    Duration:
    Mike Wilson Cell Phone: 972-249-8494
    

    mxfapega02:


    Start Time:
    Duration:
    Mike Wilson Cell Phone: 972-249-8494