The following document describes the procedure used to move the EGATE Production database environment from the MDC data center to the Disaster Recovery provisions at DDC. The DR provisions at DDC normally run the Egate pre-production environment.

  1. The first phase of this process is to shutdown the pre-production EGATE database and start the production EGATE database environment in the DR provision.

  2. The second phase of this process is to shutdown the production EGATE database in the DR provision and restart the pre-production EGATE database environment.

Phase 1

- Shutdown the pre-production EGATE database and start the production EGATE database environment in the DR provision.


Phase 1 Start Time:
Phase 1 Duration: 








    




  1.   Contact SAN group to turn off the
Tru-Copy mirroring of the production EGATE database drives.



    True Copy Procedure:

    

    


    mdcapega01:

    

    Start Time: 

    Duration: 

    Mike Wilson Cell Phone: 972-249-8494

    


    mdcapega02:

    

    Start Time: 

    Duration: 

    Mike Wilson Cell Phone: 972-249-8494

    


    2. 

    




  2.  Login to ddcaaega01 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines.  The HMC client can be
downloaded from the following url:
    

    
http://ddclahmc01.tu.com/remote_client.html

    3. 


  3.  Notify Users the DR is about to occur

    

     wall
    
A disaster has occurred and the pre-production environment is
    
being reconfigured as production NOW. Please LOG OFF.


    
    

    4. 


  4.  Change to the DR scripts directory for the target server:

    

     cd /usr/lpp/dr/mdcapega01
     

    5. 


  5.  Execute the deconfiguration phase of the DR.  The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).

    

    

    Start Time: 

    Duration: 

    
     ./OCIDR -hkemndv deconfig
     

    6. 

  6.  Check to see that HACMP is down, user processes were killed,
     all non-rootvg file systems were unmounted,
     all non-static volume groups were varied off and exported.  The 
     remaining volume groups may include one or more of the following: 
     rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
     
    7. 





  7.  Clear persistant reserves on all SAN disks


    ddcaaega01: 

    

    Start Time: 

    Duration: 

    cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
    print y | ./dlmpr -c ${i}
done

    


    ddcaaega02: 

    

    Start Time: 

    Duration: 

    cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
    print y | ./dlmpr -c ${i}
done

    


    8. 




  8.  Execute the configuration phase of the DR and import the volume
groups.  The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).

    

    

    Start Time: 

    Duration: 

    ./OCIDR -IGmNraoduv config

for i in $( lsvg )
do
    varyonvg -c ${i}
done

    
     

    9. 

  9.  Check to see that all volume groups were imported and varied on,
     all file systems were mounted,
     network addresses were changed,
     IQ directories were created,
     configuration scripts were installed,
     and background daemons were refreshed.
     
    10. 


  10.  The Oracle parallel server requires the disks and logical volumes 
be owned by the oracle user.  Execute the following script to change the
disk ownerships:

    

    

    Start Time: 

    Duration: 

    
     /usr/lpp/dr/helpers/fixForOracle.sh
     

    11. 


  11.  After HACMP startup is complete, initialize the Oracle environment
     by running the following script:

    

    

    Start Time: 

    Duration: 

    

     mount ddcapnim01:/export/cdimages/Oracle8i/CD1 /mnt
    
     cd /mnt # DO NOT SKIP
    
     ./rootpre.sh
     

    12. 




  12.  Restore the /etc/hosts and /.rhosts files from the EGATE production
environment


    ddcaaega01: 

    

    Start Time: 

    Duration: 

    /usr/lpp/dr/ddcaaega01/uueConfig/!etc!hosts.uue
/usr/lpp/dr/ddcaaega01/uueConfig/!.rhosts.uue


    



    ddcaaega02: 

    

    Start Time: 

    Duration: 

    /usr/lpp/dr/ddcaaega02/uueConfig/!etc!hosts.uue
/usr/lpp/dr/ddcaaega02/uueConfig/!.rhosts.uue


    


    13. 




  13.  After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:


    

    

    Start Time: 

    Duration: 

    

     smitty hacmp -> Cluster Services -> Start Cluster
Services
     

    14. 


  14.  Login to ddcaaega02 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines.  The HMC client can be
downloaded from the following url:
    


    
http://ddclahmc01.tu.com/remote_client.html

    15. 


  15.  Notify Users the DR is about to occur

    
     wall
    
A disaster has occurred and the pre-production environment is
    
being reconfigured as production NOW. Please LOG OFF.


    
    


    16. 


  16.  Change to the DR scripts directory for the target server:

    
     cd /usr/lpp/dr/mdcapega01
     

    17. 


  17.  Execute the deconfiguration phase of the DR.  The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).


    

    

    Start Time: 

    Duration: 

    

     ./OCIDR -hkemndv deconfig

     

    18. 

  18.  Check to see that HACMP is down, user processes were killed,
     all non-rootvg file systems were unmounted,
     all non-static volume groups were varied off and exported.  The 
     remaining volume groups may include one or more of the following: 
     rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
     
    19. 


  19.  Execute the configuration phase of the DR and import the volume
groups.  The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).


    

    

    

    ./OCIDR -IGmNraoduv config

for i in $( lsvg )
do
    varyonvg -c ${i}
done

    

    20. 


  20.  Check to see that all volume groups were imported and varied on,
     all file systems were mounted,
     network addresses were changed,
     IQ directories were created,
     configuration scripts were installed,
     and background daemons were refreshed.
     
    21. 


  21.  The Oracle parallel server requires the disks and logical volumes 
be owned by the oracle user.  Execute the following script to change the
disk ownerships:


    

    

    Start Time: 

    Duration: 

    

     /usr/lpp/dr/helpers/fixForOracle.sh
     

    22. 


  22.  After HACMP startup is complete, initialize the Oracle environment
     by running the following script:


    

    

    Start Time: 

    Duration: 

    

     mount ddcapnim01:/export/cdimages/Oracle8i/CD1  /mnt
    
     cd /mnt # DO NOT SKIP
    
     ./rootpre.sh

     

    23. 


  23.  After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:


    

    

    Start Time: 

    Duration: 

    

     smitty hacmp -> Cluster Services -> Start Cluster
Services

     

    24. 


  24.  Notify the Oracle database team the server is ready.


    

    

    Start Time: 

    Duration: 

    
     

    25. 
















Phase 2
 - Shutdown the production EGATE database in the DR
provision and restart the pre-production EGATE database environment.







Phase 2 Start Time: 

Phase 2 Duration: 









    




  1.  Login to ddcaaega01 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines.  The HMC client can be
downloaded from the following url:
    

    
http://ddclahmc01.tu.com/remote_client.html

    2. 


  2.  Change to the DR scripts directory for the target server:

    

     cd /usr/lpp/dr/ddcaaega01
     

    3. 


  3.  Execute the deconfiguration phase of the DR.  The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).


    

    

    Start Time: 

    Duration: 

    

     ./OCIDR -hkemndv deconfig
     

    4. 

  4.  Check to see that HACMP is down, user processes were killed,
     all non-rootvg file systems were unmounted,
     all non-static volume groups were varied off and exported.  The 
     remaining volume groups may include one or more of the following: 
     rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
     
    5. 





  5.  Clear persistant reserves on all SAN disks


    ddcaaega01: 

    

    Start Time: 

    Duration: 

    cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
    print y | ./dlmpr -c ${i}
done

    


    ddcaaega02: 

    

    Start Time: 

    Duration: 

    cd /usr/DynamicLinkManager/bin
for i in $( lspv | awk '{ print $1 }' | grep "^hdisk" )
do
    print y | ./dlmpr -c ${i}
done

    


    6. 




  6.  Execute the configuration phase of the DR and import the volume
groups.  The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).


    

    

    Start Time: 

    Duration: 

    

     ./OCIDR -IGmNraoduv config
     

    7. 

  7.  Check to see that all volume groups were imported and varied on,
     all file systems were mounted,
     network addresses were changed,
     IQ directories were created,
     configuration scripts were installed,
     and background daemons were refreshed.
     
    8. 


  8.  The Oracle parallel server requires the disks and logical volumes 
be owned by the oracle user.  Execute the following script to change the
disk ownerships:


    

    

    Start Time: 

    Duration: 

    

     /usr/lpp/dr/helpers/fixForOracle.sh
     

    9. 


  9.  After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:


    

    

    Start Time: 

    Duration: 

    

     smitty hacmp -> Cluster Services -> Start Cluster
Services
     

    10. 


  10.  After HACMP startup is complete, initialize the Oracle environment
     by running the following script:


    

    

    Start Time: This step not executed

    Duration: 

    

     mount ddcapnim01:/export/cdimages/Oracle8i/CD1  /mnt
    
     cd /mnt # DO NOT SKIP
    
     ./rootpre.sh
     

    11. 





  11.  Login to ddcaaega01 on the console as root. You will need to
install and run the Hardware Management Console (HMC) client to be able
to access the console of these machines.  The HMC client can be
downloaded from the following url:
    

    
http://ddclahmc01.tu.com/remote_client.html

    12. 


  12.  Change to the DR scripts directory for the target server:

    

     cd /usr/lpp/dr/ddcaaega02
     

    13. 


  13.  Execute the deconfiguration phase of the DR.  The options shown
indicate the deconfiguration should kill all processes (k), varyoff and
export VGs (e), unmount filesystems (m), deconfigure the network
adapters (n), delete any existing IQ directories (d), and to do it all
in verbose mode (v).


    

    

    Start Time: 

    Duration: 

    

     ./OCIDR -hkemndv deconfig
     

    14. 

  14.  Check to see that HACMP is down, user processes were killed,
     all non-rootvg file systems were unmounted,
     all non-static volume groups were varied off and exported.  The 
     remaining volume groups may include one or more of the following: 
     rootvg, altinst_rootvg, egate1[0-9][0-9]vg_dr.
     
    15. 


  15.  Execute the configuration phase of the DR and import the volume
groups.  The options shown indicate the configuration should import the
VGs, LVs, and FSs (i), mount the filesystems (m), configure the network
adapters in HACMP mode (N), install the configuration files (r), start
the applications (a), create the home directories (o), create the IQ
directories (d), change the hostname (u), and to do it all in verbose
mode (v).


    

    

    Start Time: 

    Duration: 

    

     ./OCIDR -IGmNraoduv config
     

    16. 

  16.  Check to see that all volume groups were imported and varied on,
     all file systems were mounted,
     network addresses were changed,
     IQ directories were created,
     configuration scripts were installed,
     and background daemons were refreshed.
     
    17. 


  17.  The Oracle parallel server requires the disks and logical volumes 
be owned by the oracle user.  Execute the following script to change the
disk ownerships:


    

    

    Start Time: 

    Duration: 

    

     /usr/lpp/dr/helpers/fixForOracle.sh
     

    18. 




  18.  Restore the /etc/hosts and /.rhosts files from the EGATE pre-production
environment


    ddcaaega01: 

    

    Start Time: 

    Duration: 

    /usr/lpp/dr/ddcaaega01/uueConfig/!etc!hosts.uue
/usr/lpp/dr/ddcaaega01/uueConfig/!.rhosts.uue


    



    ddcaaega02: 

    

    Start Time: 

    Duration: 

    /usr/lpp/dr/ddcaaega02/uueConfig/!etc!hosts.uue
/usr/lpp/dr/ddcaaega02/uueConfig/!.rhosts.uue


    


    19. 




  19.  After the OCIDR script has completed on ALL Oracle Parallel Server
nodes, start HACMP on this node:


    

    

    Start Time: 

    Duration: 

    

     smitty hacmp -> Cluster Services -> Start Cluster Services
     

    20. 


  20.  After HACMP startup is complete, initialize the Oracle environment
     by running the following script:


    

    

    Start Time:  This step not executed

    Duration: 0m

    

     mount ddcapnim01:/export/cdimages/Oracle8i/CD1  /mnt
    
     cd /mnt # DO NOT SKIP
    
     ./rootpre.sh
     

    21. 




  21.   Contact SAN group to restart the
Tru-Copy mirroring of the production EGATE database drives.



    True Copy Procedure:

    

    


    mdcapega01:

    

    Start Time: 

    Duration: 

    Mike Wilson Cell Phone: 972-249-8494

    


    mdcapega02:

    

    Start Time: 

    Duration: 

    Mike Wilson Cell Phone: 972-249-8494