EDNA Systems Design

The EDNA project provides IT infrastructure to host the eMeter PiPE application and database. The database is Oracle 9i running in a pair of HACMP cluster nodes, and the application is Java based running on several (currently 3) AIX 5.3 LPAR's. The availability of each application server is protected by an HACMP hot-standby node. In the event of a primary system or network failure, the application is failed over to it's hot-standby failover node. There are current four clusters in the production environment and an additional four clusters in the pre-production environment, for a total of 8 clusters. Each cluster consists of two nodes, a primary server and a hot-standby.

In each environment, production and pre-production, there exists a cluster dedicated to hosting the Oracle database, and three clusters hosting separate instances of the eMeter PiPE application.

Each system is built as an LPAR in the p590 AIX environment using virtualized I/O for access to the network and SAN storage. All disks, including those dedicated to the operating system are virtualized from dual VIO servers on each p590 frame. Each disk communicates to the SAN via multiple communication paths using virtualized fiber channel adapters and Multi-path IO.

Each system has two virtual ethernet adapters (one from each VIO) providing access to the standard p590 network. These adapters are bundled together in an etherchannel configured with one primary adapter and one backup adapter.

Each system has two virtual ethernet adapters (one from each VIO) providing access to the p590 management network. These adapters are bundled together in an etherchannel configured with one primary adapter and one backup adapter.

In addition to the virtualized ethernet adapters, each PRODUCTION EDNA system has one single physical gigabit ethernet adapter configured into the LPAR. This adapter is dedicated for communication between the AIX EDNA systems and the Windows EDNA systems, and is on a separate subnet. This is intended to segment and isolate a large amount of broadcast traffic expected between these machines. This additional gigabit ethernet adapter does NOT exist in the pre-production environment and is unique to the production EDNA environment. This additional physical ethernet adapter does not exist in any other LPAR configuration.

The CPU and memory for all EDNA systems is managed by the Partition Load Manager (PLM) and can be dynamically tuned on an as needed basis.


Dallas Data Center: EDNA Production


Resource Groups

All volume groups, logical volumes, and filesystems have been designed and implemented using enterprise wide unique identifiers for the purpose of business continuity and disaster recovery. This enterprise wide unique identifier is based on the "resource group" name.

Resource Groups configured on the Production EDNA Servers

Resource Group Service IP Application Server Volume Group
ednpdtu1 mx1apedn01-rg01 ednpdtu1_oradb01 ednpdtu101vg
ednpdtu2 mx1apedn02-rg01 ednpdtu2_oradb01 ednpdtu201vg
ednpdtu3 mx1apedn03-rg01 ednpdtu3_oradb01 ednpdtu301vg
ednpdtu4 mx1apedn04-rg01 ednpdtu4_oradb01 ednpdtu401vg
ednpdtu5 mx1apedn05-rg01 ednpdtu5_oradb01 ednpdtu501vg
ednpdtu6 mx1apedn06-rg01 ednpdtu6_oradb01 ednpdtu601vg
ednpdtu7 mx1apedn07-rg01 ednpdtu7_oradb01 ednpdtu701vg
ednpdtu8 mx1apedn08-rg01 ednpdtu8_oradb01 ednpdtu801vg


Cluster Heartbeat

A cluster heartbeat is configured on the first disk of the shared concurrent volume group for the purpose of determining cluster health.


Volume Groups, Logical Volumes, and Filesystems

A single logical volume and JFS2 filesystem is configured for each volume group on each system to provide application storage.

Volume Group Logical Volume Log LV Filesystem
ednpdtu101vg ednpdtu1ora1lv ednpdtu1jfs2lv /ednpdtu1/u01
ednpdtu201vg ednpdtu2ora1lv ednpdtu2jfs2lv /ednpdtu2/u01
ednpdtu301vg ednpdtu3ora1lv ednpdtu3jfs2lv /ednpdtu3/u01
ednpdtu401vg ednpdtu4ora1lv ednpdtu4jfs2lv /ednpdtu4/u01
ednpdtu501vg ednpdtu5ora1lv ednpdtu5jfs2lv /ednpdtu5/u01
ednpdtu601vg ednpdtu6ora1lv ednpdtu6jfs2lv /ednpdtu6/u01
ednpdtu701vg ednpdtu7ora1lv ednpdtu7jfs2lv /ednpdtu7/u01
ednpdtu801vg ednpdtu8ora1lv ednpdtu8jfs2lv /ednpdtu8/u01


Ethernet Network Configuration

Network access to all systems by users shall be through using the DNS alias name associated with the service name for each resource group. Each system currently has a single resource group defined that may float between any node of the cluster. Currently all clusters are configured with two nodes, a primary and a secondary. The resource group service name and address is configured on the physical ethernet adapter in each system (en6). Again, this physical adapter configured on the EDNA production servers is unique to this environment and is not configured on any other LPAR.

Server name for user access to production EDNA servers

Server Name for User Access RG Service Name RG IP Address
ednpdtu1 mx1apedn01-rg01 146.61.61.31
ednpdtu2 mx1apedn02-rg01 146.61.61.32
ednpdtu3 mx1apedn03-rg01 146.61.61.33
ednpdtu4 mx1apedn04-rg01 146.61.61.34
ednpdtu5 mx1apedn05-rg01 146.61.61.35
ednpdtu6 mx1apedn06-rg01 146.61.61.36
ednpdtu7 mx1apedn07-rg01 146.61.61.37
ednpdtu8 mx1apedn08-rg01 146.61.61.38


An additional resource group is configured on each server to provide access to the RG service name associated with the node name. The name of this resource group is the same as the node name. This name and address is configured to exist on the standard p590 virtualized etherchannel adapter (en4)

Node Name Resource Group Name RG Service Name RG IP Address
mx1apedn01 mx1apedn01 mx1apedn01 146.61.171.56
mx1apedn02 mx1apedn02 mx1apedn02 146.61.171.58
mx1apedn03 mx1apedn03 mx1apedn03 146.61.171.60
mx1apedn04 mx1apedn04 mx1apedn04 146.61.171.62
mx1apedn05 mx1apedn05 mx1apedn05 146.61.171.64
mx1apedn06 mx1apedn06 mx1apedn06 146.61.171.66
mx1apedn07 mx1apedn07 mx1apedn07 146.61.171.68
mx1apedn08 mx1apedn08 mx1apedn08 146.61.171.70


In order to insure access to each system, a persistent address is configured and exists on the standard p590 virtualized ethernchannel adapter (en4).

Node Name Persistent Name Persistent IP Boot Address
mx1apedn01 mx1apedn01-pers 146.61.171.57 192.168.171.56
mx1apedn02 mx1apedn02-pers 146.61.171.59 192.168.171.58
mx1apedn03 mx1apedn03-pers 146.61.171.61 192.168.171.60
mx1apedn04 mx1apedn04-pers 146.61.171.63 192.168.171.62
mx1apedn05 mx1apedn05-pers 146.61.171.65 192.168.171.64
mx1apedn06 mx1apedn06-pers 146.61.171.67 192.168.171.66
mx1apedn07 mx1apedn07-pers 146.61.171.69 192.168.171.68
mx1apedn08 mx1apedn08-pers 146.61.171.71 192.168.171.70


The virtualized management etherchannel adapter (en5) in each system is configured with a single IP address on the management network.

Node Name Management Name Management IP Address
mx1apedn01 mx1apedn01-man 146.61.172.42
mx1apedn02 mx1apedn02-man 146.61.172.43
mx1apedn03 mx1apedn03-man 146.61.172.44
mx1apedn04 mx1apedn04-man 146.61.172.45
mx1apedn05 mx1apedn05-man 146.61.172.46
mx1apedn06 mx1apedn06-man 146.61.172.47
mx1apedn07 mx1apedn07-man 146.61.172.48
mx1apedn08 mx1apedn08-man 146.61.172.49


The resource group service names configured on each system are only available when HACMP is running. When the system first boots, HACMP is not automatically started, therefore the service names do not yet exist on the virtualized etherchannel adapter or physical adapter. At boot time these adapters are configured with non-routed boot addresses.

Virt EthChan Name Virt EthChan IP Phys GigE Name Phys Gige IP
mx1apedn01-boot 192.168.171.56 mx1apedn01-bt01 192.168.61.31
mx1apedn02-boot 192.168.171.58 mx1apedn01-bt01 192.168.61.32
mx1apedn03-boot 192.168.171.60 mx1apedn01-bt01 192.168.61.33
mx1apedn04-boot 192.168.171.62 mx1apedn01-bt01 192.168.61.34
mx1apedn05-boot 192.168.171.64 mx1apedn01-bt01 192.168.61.35
mx1apedn06-boot 192.168.171.66 mx1apedn01-bt01 192.168.61.36
mx1apedn07-boot 192.168.171.68 mx1apedn01-bt01 192.168.61.37
mx1apedn08-boot 192.168.171.70 mx1apedn01-bt01 192.168.61.38


From this information, it can be derived that during nominal operation, any production EDNA machine will have three active network interfaces. Each interfaces will have one or more IP addresses configured on it:

The adapter "en4" will have 1 boot address on the "192.168.171.0" network, 1 persistent address on the "146.61.171.0" network, and 1 resource group service address on the "146.61.171.0" network.

The adapter "en5 will have 1 management address on the "146.61.172.0" network.

The adapter "en6" will have 1 boot address on the "192.168.61.0" network, and 1 resource group service address on the "146.61.61.0" network.

If a machine is running in "failover" mode, the adapters "en4" and "en6" may have additional service addresses configured.


User Names and UID Numbers

Two service user names were added to the Oracle database servers for managing Oracle, one for each instance of Oracle running on each cluster node. These service user names were created for the purpose of providing the DBA's with access to the system to manage Oracle. The UID number is a calculated value based on the BSD checksum of each user name.

The application servers provide two service user names: "pipe" and "pipetran". These user names are used for managing the eMeter PiPE application and for transfering files between systems. Direct login access to these service user names shall be turned off, thus requiring the user to first login using their individual login ID. Multiple individual login ID's are configured in the form of RACF ID's. Several eMeter personnel have access to these system for maintenance and support. The UID for all user names shall be a calculated value based on the BSD checksum of each user name.


Group Names and GID Numbers

One group name was added to the Oracle database servers for association with the Oracle service user names. The GID number is a calculated value based on the BSD checksum of the group name.

 

Mesquite Data Center: EDNA Test development


Resource Groups

All volume groups, logical volumes, and filesystems have been designed and implemented using enterprise wide unique identifiers for the purpose of business continuity and disaster recovery. This enterprise wide unique identifier is based on the "resource group" name.

Resource Groups configured on the Pre-Production EDNA Servers

Resource Group Service IP Application Server Volume Group
ednadtu1 mx0adedn01-rg01 ednadtu1_oradb01 ednadtu101vg
ednadtu2 mx0adedn02-rg01 ednadtu2_oradb01 ednadtu201vg
ednadtu3 mx0adedn03-rg01 ednadtu3_oradb01 ednadtu301vg
ednadtu5 mx0adedn05-rg01 ednadtu5_oradb01 ednadtu501vg
ednadtu7 mx0adedn07-rg01 ednadtu7_oradb01 ednadtu701vg


Cluster Heartbeat

A cluster heartbeat is configured on the first disk of the shared concurrent volume group for the purpose of determining cluster health.


Volume Groups, Logical Volumes, and Filesystems

A single logical volume and JFS2 filesystem is configured for each volume group on each system to provide application storage.

Volume Group Logical Volume Log LV Filesystem
ednadtu101vg ednadtu1ora1lv ednadtu1jfs2lv /ednadtu1
ednadtu201vg ednadtu2ora1lv ednadtu2jfs2lv /ednadtu2
ednadtu301vg ednadtu3ora1lv ednadtu3jfs2lv /ednadtu3
ednadtu501vg ednadtu5ora1lv ednadtu5jfs2lv /ednadtu5
ednadtu701vg ednadtu7ora1lv ednadtu7jfs2lv /ednadtu7


Ethernet Network Configuration

Network access to all systems by users shall be through using the DNS alias name associated with the service name for each resource group. Each system currently has a single resource group defined that may float between any node of the cluster. Currently all clusters are configured with two nodes, a primary and a secondary. The resource group service name and address is configured on the virtual etherchannel adapter in each system (en4).

User Access to Pre-Production EDNA Servers

Server Name for User Access RG Service Name RG IP Address
ednadtu1 mx0adedn01-rg01 146.61.122.206
ednadtu2 mx0adedn02-rg01 146.61.122.219
ednadtu3 mx0adedn03-rg01 146.61.122.207
ednadtu5 mx0adedn05-rg01 146.61.122.208
ednadtu7 mx0adedn07-rg01 146.61.122.209


An additional resource group is configured on each server to provide access to the RG service name associated with the node name. The name of this resource group is the same as the node name. This name and address is configured to exist on the standard p590 virtualized etherchannel adapter (en4)

Resource Group Name RG Service Name RG IP Address
mx0adedn01 mx0adedn01 146.61.122.190
mx0adedn02 mx0adedn02 146.61.122.192
mx0adedn03 mx0adedn03 146.61.122.194
mx0adedn04 mx0adedn04 146.61.122.196
mx0adedn05 mx0adedn05 146.61.122.198
mx0adedn06 mx0adedn06 146.61.122.200
mx0adedn07 mx0adedn07 146.61.122.202
mx0adedn08 mx0adedn08 146.61.122.204


In order to insure access to each system, a persistent address is configured and exists on the standard p590 virtualized ethernchannel adapter (en4).

Persistent Name Persistent IP Address Boot Address
mx0adedn01-pers 146.61.122.191 192.168.30.60
mx0adedn02-pers 146.61.122.193 192.168.30.61
mx0adedn03-pers 146.61.122.195 192.168.30.62
mx0adedn04-pers 146.61.122.197 192.168.30.63
mx0adedn05-pers 146.61.122.199 192.168.30.64
mx0adedn06-pers 146.61.122.201 192.168.30.65
mx0adedn07-pers 146.61.122.203 192.168.30.66
mx0adedn08-pers 146.61.122.205 192.168.30.67


The virtualized management etherchannel adapter (en5) in each system is configured with a single IP address on the management network.

Management Name Management IP Address
mx0adedn01-man 146.61.124.50
mx0adedn02-man 146.61.124.51
mx0adedn03-man 146.61.124.52
mx0adedn04-man 146.61.124.53
mx0adedn05-man 146.61.124.54
mx0adedn06-man 146.61.124.55
mx0adedn07-man 146.61.124.56
mx0adedn08-man 146.61.124.57


The resource group service names configured on each system are only available when HACMP is running. When the system first boots, HACMP is not automatically started, therefore the service names do not yet exist on the virtualized etherchannel adapter. At boot time this adapter is configured with non-routed boot address.

Virt EthChan Name Virt EthChan IP
mx0adedn01-boot 192.168.30.60
mx0adedn02-boot 192.168.30.61
mx0adedn03-boot 192.168.30.62
mx0adedn04-boot 192.168.30.63
mx0adedn05-boot 192.168.30.64
mx0adedn06-boot 192.168.30.65
mx0adedn07-boot 192.168.30.66
mx0adedn08-boot 192.168.30.67


From this information, it can be derived that during nominal operation, any pre-production EDNA machine will have two active network interfaces. Each interface will have one or more IP addresses configured on it:

The adapter "en4" will have 1 boot address on the "192.168.30.0" network, 1 persistent address on the "146.61.122.0" network, and 2 resource group service addresses on the "146.61.122.0" network.

The adapter "en5 will have 1 management address on the "146.61.124.0" network.

If a machine is running in "failover" mode, the adapter "en4" may have additional service addresses configured.


User Names and UID Numbers

Two service user names were added to the Oracle database servers for managing Oracle, one for each instance of Oracle running on each cluster node. These service user names were created for the purpose of providing the DBA's with access to the system to manage Oracle. The UID number is a calculated value based on the BSD checksum of each user name.

The application servers provide two service user names: "pipe" and "pipetran". These user names are used for managing the eMeter PiPE application and for transfering files between systems. Direct login access to these service user names shall be turned off, thus requiring the user to first login using their individual login ID. Multiple individual login ID's are configured in the form of RACF ID's. Several eMeter personnel have access to these system for maintenance and support. The UID for all user names shall be a calculated value based on the BSD checksum of each user name.


Group Names and GID Numbers

One group name was added to the Oracle database servers for association with the Oracle service user names. The GID number is a calculated value based on the BSD checksum of the group name.