The EDNA project provides IT infrastructure to host the eMeter PiPE application and database. The database is Oracle 9i running in a pair of HACMP cluster nodes, and the application is Java based running on several (currently 3) AIX 5.3 LPAR's. The availability of each application server is protected by an HACMP hot-standby node. In the event of a primary system or network failure, the application is failed over to it's hot-standby failover node. There are current four clusters in the production environment and an additional four clusters in the pre-production environment, for a total of 8 clusters. Each cluster consists of two nodes, a primary server and a hot-standby.
In each environment, production and pre-production, there exists a cluster dedicated to hosting the Oracle database, and three clusters hosting separate instances of the eMeter PiPE application.
Each system is built as an LPAR in the p590 AIX environment using virtualized I/O for access to the network and SAN storage. All disks, including those dedicated to the operating system are virtualized from dual VIO servers on each p590 frame. Each disk communicates to the SAN via multiple communication paths using virtualized fiber channel adapters and Multi-path IO.
Each system has two virtual ethernet adapters (one from each VIO) providing access to the standard p590 network. These adapters are bundled together in an etherchannel configured with one primary adapter and one backup adapter.
Each system has two virtual ethernet adapters (one from each VIO) providing access to the p590 management network. These adapters are bundled together in an etherchannel configured with one primary adapter and one backup adapter.
In addition to the virtualized ethernet adapters, each PRODUCTION EDNA system has one single physical gigabit ethernet adapter configured into the LPAR. This adapter is dedicated for communication between the AIX EDNA systems and the Windows EDNA systems, and is on a separate subnet. This is intended to segment and isolate a large amount of broadcast traffic expected between these machines. This additional gigabit ethernet adapter does NOT exist in the pre-production environment and is unique to the production EDNA environment. This additional physical ethernet adapter does not exist in any other LPAR configuration.
The CPU and memory for all EDNA systems is managed by the Partition Load Manager (PLM) and can be dynamically tuned on an as needed basis.
All volume groups, logical volumes, and filesystems have been designed and implemented using enterprise wide unique identifiers for the purpose of business continuity and disaster recovery. This enterprise wide unique identifier is based on the "resource group" name.
Resource Groups configured on the Production EDNA Servers
| Resource Group | Service IP | Application Server | Volume Group |
|---|---|---|---|
| ednpdtu1 | mx1apedn01-rg01 | ednpdtu1_oradb01 | ednpdtu101vg |
| ednpdtu2 | mx1apedn02-rg01 | ednpdtu2_oradb01 | ednpdtu201vg |
| ednpdtu3 | mx1apedn03-rg01 | ednpdtu3_oradb01 | ednpdtu301vg |
| ednpdtu4 | mx1apedn04-rg01 | ednpdtu4_oradb01 | ednpdtu401vg |
| ednpdtu5 | mx1apedn05-rg01 | ednpdtu5_oradb01 | ednpdtu501vg |
| ednpdtu6 | mx1apedn06-rg01 | ednpdtu6_oradb01 | ednpdtu601vg |
| ednpdtu7 | mx1apedn07-rg01 | ednpdtu7_oradb01 | ednpdtu701vg |
| ednpdtu8 | mx1apedn08-rg01 | ednpdtu8_oradb01 | ednpdtu801vg |
A cluster heartbeat is configured on the first disk of the shared concurrent volume group for the purpose of determining cluster health.
A single logical volume and JFS2 filesystem is configured for each volume group on each system to provide application storage.
| Volume Group | Logical Volume | Log LV | Filesystem |
|---|---|---|---|
| ednpdtu101vg | ednpdtu1ora1lv | ednpdtu1jfs2lv | /ednpdtu1/u01 |
| ednpdtu201vg | ednpdtu2ora1lv | ednpdtu2jfs2lv | /ednpdtu2/u01 |
| ednpdtu301vg | ednpdtu3ora1lv | ednpdtu3jfs2lv | /ednpdtu3/u01 |
| ednpdtu401vg | ednpdtu4ora1lv | ednpdtu4jfs2lv | /ednpdtu4/u01 |
| ednpdtu501vg | ednpdtu5ora1lv | ednpdtu5jfs2lv | /ednpdtu5/u01 |
| ednpdtu601vg | ednpdtu6ora1lv | ednpdtu6jfs2lv | /ednpdtu6/u01 |
| ednpdtu701vg | ednpdtu7ora1lv | ednpdtu7jfs2lv | /ednpdtu7/u01 |
| ednpdtu801vg | ednpdtu8ora1lv | ednpdtu8jfs2lv | /ednpdtu8/u01 |
Network access to all systems by users shall be through using the DNS alias name associated with the service name for each resource group. Each system currently has a single resource group defined that may float between any node of the cluster. Currently all clusters are configured with two nodes, a primary and a secondary. The resource group service name and address is configured on the physical ethernet adapter in each system (en6). Again, this physical adapter configured on the EDNA production servers is unique to this environment and is not configured on any other LPAR.
Server name for user access to production EDNA servers
| Server Name for User Access | RG Service Name | RG IP Address |
|---|---|---|
| ednpdtu1 | mx1apedn01-rg01 | 146.61.61.31 |
| ednpdtu2 | mx1apedn02-rg01 | 146.61.61.32 |
| ednpdtu3 | mx1apedn03-rg01 | 146.61.61.33 |
| ednpdtu4 | mx1apedn04-rg01 | 146.61.61.34 |
| ednpdtu5 | mx1apedn05-rg01 | 146.61.61.35 |
| ednpdtu6 | mx1apedn06-rg01 | 146.61.61.36 |
| ednpdtu7 | mx1apedn07-rg01 | 146.61.61.37 |
| ednpdtu8 | mx1apedn08-rg01 | 146.61.61.38 |
| Node Name | Resource Group Name | RG Service Name | RG IP Address |
|---|---|---|---|
| mx1apedn01 | mx1apedn01 | mx1apedn01 | 146.61.171.56 |
| mx1apedn02 | mx1apedn02 | mx1apedn02 | 146.61.171.58 |
| mx1apedn03 | mx1apedn03 | mx1apedn03 | 146.61.171.60 |
| mx1apedn04 | mx1apedn04 | mx1apedn04 | 146.61.171.62 |
| mx1apedn05 | mx1apedn05 | mx1apedn05 | 146.61.171.64 |
| mx1apedn06 | mx1apedn06 | mx1apedn06 | 146.61.171.66 |
| mx1apedn07 | mx1apedn07 | mx1apedn07 | 146.61.171.68 |
| mx1apedn08 | mx1apedn08 | mx1apedn08 | 146.61.171.70 |
| Node Name | Persistent Name | Persistent IP | Boot Address |
|---|---|---|---|
| mx1apedn01 | mx1apedn01-pers | 146.61.171.57 | 192.168.171.56 |
| mx1apedn02 | mx1apedn02-pers | 146.61.171.59 | 192.168.171.58 |
| mx1apedn03 | mx1apedn03-pers | 146.61.171.61 | 192.168.171.60 |
| mx1apedn04 | mx1apedn04-pers | 146.61.171.63 | 192.168.171.62 |
| mx1apedn05 | mx1apedn05-pers | 146.61.171.65 | 192.168.171.64 |
| mx1apedn06 | mx1apedn06-pers | 146.61.171.67 | 192.168.171.66 |
| mx1apedn07 | mx1apedn07-pers | 146.61.171.69 | 192.168.171.68 |
| mx1apedn08 | mx1apedn08-pers | 146.61.171.71 | 192.168.171.70 |
| Node Name | Management Name | Management IP Address |
|---|---|---|
| mx1apedn01 | mx1apedn01-man | 146.61.172.42 |
| mx1apedn02 | mx1apedn02-man | 146.61.172.43 |
| mx1apedn03 | mx1apedn03-man | 146.61.172.44 |
| mx1apedn04 | mx1apedn04-man | 146.61.172.45 |
| mx1apedn05 | mx1apedn05-man | 146.61.172.46 |
| mx1apedn06 | mx1apedn06-man | 146.61.172.47 |
| mx1apedn07 | mx1apedn07-man | 146.61.172.48 |
| mx1apedn08 | mx1apedn08-man | 146.61.172.49 |
| Virt EthChan Name | Virt EthChan IP | Phys GigE Name | Phys Gige IP |
|---|---|---|---|
| mx1apedn01-boot | 192.168.171.56 | mx1apedn01-bt01 | 192.168.61.31 |
| mx1apedn02-boot | 192.168.171.58 | mx1apedn01-bt01 | 192.168.61.32 |
| mx1apedn03-boot | 192.168.171.60 | mx1apedn01-bt01 | 192.168.61.33 |
| mx1apedn04-boot | 192.168.171.62 | mx1apedn01-bt01 | 192.168.61.34 |
| mx1apedn05-boot | 192.168.171.64 | mx1apedn01-bt01 | 192.168.61.35 |
| mx1apedn06-boot | 192.168.171.66 | mx1apedn01-bt01 | 192.168.61.36 |
| mx1apedn07-boot | 192.168.171.68 | mx1apedn01-bt01 | 192.168.61.37 |
| mx1apedn08-boot | 192.168.171.70 | mx1apedn01-bt01 | 192.168.61.38 |
The adapter "en4" will have 1 boot address on the "192.168.171.0" network, 1 persistent address on the "146.61.171.0" network, and 1 resource group service address on the "146.61.171.0" network.
The adapter "en5 will have 1 management address on the "146.61.172.0" network.
The adapter "en6" will have 1 boot address on the "192.168.61.0" network, and 1 resource group service address on the "146.61.61.0" network.
If a machine is running in "failover" mode, the adapters "en4" and "en6" may have additional service addresses configured.
Two service user names were added to the Oracle database servers for managing Oracle, one for each instance of Oracle running on each cluster node. These service user names were created for the purpose of providing the DBA's with access to the system to manage Oracle. The UID number is a calculated value based on the BSD checksum of each user name.
The application servers provide two service user names: "pipe" and "pipetran". These user names are used for managing the eMeter PiPE application and for transfering files between systems. Direct login access to these service user names shall be turned off, thus requiring the user to first login using their individual login ID. Multiple individual login ID's are configured in the form of RACF ID's. Several eMeter personnel have access to these system for maintenance and support. The UID for all user names shall be a calculated value based on the BSD checksum of each user name.
One group name was added to the Oracle database servers for association with the Oracle service user names. The GID number is a calculated value based on the BSD checksum of the group name.
All volume groups, logical volumes, and filesystems have been designed and implemented using enterprise wide unique identifiers for the purpose of business continuity and disaster recovery. This enterprise wide unique identifier is based on the "resource group" name.
Resource Groups configured on the Pre-Production EDNA Servers
| Resource Group | Service IP | Application Server | Volume Group |
|---|---|---|---|
| ednadtu1 | mx0adedn01-rg01 | ednadtu1_oradb01 | ednadtu101vg |
| ednadtu2 | mx0adedn02-rg01 | ednadtu2_oradb01 | ednadtu201vg |
| ednadtu3 | mx0adedn03-rg01 | ednadtu3_oradb01 | ednadtu301vg |
| ednadtu5 | mx0adedn05-rg01 | ednadtu5_oradb01 | ednadtu501vg |
| ednadtu7 | mx0adedn07-rg01 | ednadtu7_oradb01 | ednadtu701vg |
A cluster heartbeat is configured on the first disk of the shared concurrent volume group for the purpose of determining cluster health.
A single logical volume and JFS2 filesystem is configured for each volume group on each system to provide application storage.
| Volume Group | Logical Volume | Log LV | Filesystem |
|---|---|---|---|
| ednadtu101vg | ednadtu1ora1lv | ednadtu1jfs2lv | /ednadtu1 |
| ednadtu201vg | ednadtu2ora1lv | ednadtu2jfs2lv | /ednadtu2 |
| ednadtu301vg | ednadtu3ora1lv | ednadtu3jfs2lv | /ednadtu3 |
| ednadtu501vg | ednadtu5ora1lv | ednadtu5jfs2lv | /ednadtu5 |
| ednadtu701vg | ednadtu7ora1lv | ednadtu7jfs2lv | /ednadtu7 |
Network access to all systems by users shall be through using the DNS alias name associated with the service name for each resource group. Each system currently has a single resource group defined that may float between any node of the cluster. Currently all clusters are configured with two nodes, a primary and a secondary. The resource group service name and address is configured on the virtual etherchannel adapter in each system (en4).
User Access to Pre-Production EDNA Servers
| Server Name for User Access | RG Service Name | RG IP Address |
|---|---|---|
| ednadtu1 | mx0adedn01-rg01 | 146.61.122.206 |
| ednadtu2 | mx0adedn02-rg01 | 146.61.122.219 |
| ednadtu3 | mx0adedn03-rg01 | 146.61.122.207 |
| ednadtu5 | mx0adedn05-rg01 | 146.61.122.208 |
| ednadtu7 | mx0adedn07-rg01 | 146.61.122.209 |
| Resource Group Name | RG Service Name | RG IP Address |
|---|---|---|
| mx0adedn01 | mx0adedn01 | 146.61.122.190 |
| mx0adedn02 | mx0adedn02 | 146.61.122.192 |
| mx0adedn03 | mx0adedn03 | 146.61.122.194 |
| mx0adedn04 | mx0adedn04 | 146.61.122.196 |
| mx0adedn05 | mx0adedn05 | 146.61.122.198 |
| mx0adedn06 | mx0adedn06 | 146.61.122.200 |
| mx0adedn07 | mx0adedn07 | 146.61.122.202 |
| mx0adedn08 | mx0adedn08 | 146.61.122.204 |
| Persistent Name | Persistent IP Address | Boot Address |
|---|---|---|
| mx0adedn01-pers | 146.61.122.191 | 192.168.30.60 |
| mx0adedn02-pers | 146.61.122.193 | 192.168.30.61 |
| mx0adedn03-pers | 146.61.122.195 | 192.168.30.62 |
| mx0adedn04-pers | 146.61.122.197 | 192.168.30.63 |
| mx0adedn05-pers | 146.61.122.199 | 192.168.30.64 |
| mx0adedn06-pers | 146.61.122.201 | 192.168.30.65 |
| mx0adedn07-pers | 146.61.122.203 | 192.168.30.66 |
| mx0adedn08-pers | 146.61.122.205 | 192.168.30.67 |
| Management Name | Management IP Address |
|---|---|
| mx0adedn01-man | 146.61.124.50 |
| mx0adedn02-man | 146.61.124.51 |
| mx0adedn03-man | 146.61.124.52 |
| mx0adedn04-man | 146.61.124.53 |
| mx0adedn05-man | 146.61.124.54 |
| mx0adedn06-man | 146.61.124.55 |
| mx0adedn07-man | 146.61.124.56 |
| mx0adedn08-man | 146.61.124.57 |
| Virt EthChan Name | Virt EthChan IP |
|---|---|
| mx0adedn01-boot | 192.168.30.60 |
| mx0adedn02-boot | 192.168.30.61 |
| mx0adedn03-boot | 192.168.30.62 |
| mx0adedn04-boot | 192.168.30.63 |
| mx0adedn05-boot | 192.168.30.64 |
| mx0adedn06-boot | 192.168.30.65 |
| mx0adedn07-boot | 192.168.30.66 |
| mx0adedn08-boot | 192.168.30.67 |
The adapter "en4" will have 1 boot address on the "192.168.30.0" network, 1 persistent address on the "146.61.122.0" network, and 2 resource group service addresses on the "146.61.122.0" network.
The adapter "en5 will have 1 management address on the "146.61.124.0" network.
If a machine is running in "failover" mode, the adapter "en4" may have additional service addresses configured.
Two service user names were added to the Oracle database servers for managing Oracle, one for each instance of Oracle running on each cluster node. These service user names were created for the purpose of providing the DBA's with access to the system to manage Oracle. The UID number is a calculated value based on the BSD checksum of each user name.
The application servers provide two service user names: "pipe" and "pipetran". These user names are used for managing the eMeter PiPE application and for transfering files between systems. Direct login access to these service user names shall be turned off, thus requiring the user to first login using their individual login ID. Multiple individual login ID's are configured in the form of RACF ID's. Several eMeter personnel have access to these system for maintenance and support. The UID for all user names shall be a calculated value based on the BSD checksum of each user name.
One group name was added to the Oracle database servers for association with the Oracle service user names. The GID number is a calculated value based on the BSD checksum of the group name.