|
EDNA Systems Design
The EDNA project provides IT infrastructure to host the eMeter PiPE
application and database. The database is Oracle 9i running in a pair
of HACMP cluster nodes, and the application is Java based running on
several (currently 3) AIX 5.3 LPAR's. The availability of each
application server is protected by an HACMP hot-standby node. In the
event of a primary system or network failure, the application is failed
over to it's hot-standby failover node. There are current four clusters
in the production environment and an additional four clusters in the
pre-production environment, for a total of 8 clusters. Each cluster
consists of two nodes, a primary server and a hot-standby.
In each environment, production and pre-production, there exists a
cluster dedicated to hosting the Oracle database, and three clusters
hosting separate instances of the eMeter PiPE application.
Each system is built as an LPAR in the p590 AIX environment using
virtualized I/O for access to the network and SAN storage. All disks,
including those dedicated to the operating system are virtualized from
dual VIO servers on each p590 frame. Each disk communicates to the SAN
via multiple communication paths using virtualized fiber channel
adapters and Multi-path IO.
Each system has two virtual ethernet adapters (one from each VIO)
providing access to the standard p590 network. These adapters are
bundled together in an etherchannel configured with one primary adapter
and one backup adapter.
Each system has two virtual ethernet adapters (one from each VIO)
providing access to the p590 management network. These adapters are
bundled together in an etherchannel configured with one primary adapter
and one backup adapter.
In addition to the virtualized ethernet adapters, each PRODUCTION EDNA
system has one single physical gigabit ethernet adapter configured into
the LPAR. This adapter is dedicated for communication between the AIX
EDNA systems and the Windows EDNA systems, and is on a separate subnet.
This is intended to segment and isolate a large amount of broadcast
traffic expected between these machines. This additional gigabit
ethernet adapter does NOT exist in the pre-production environment and is
unique to the production EDNA environment. This additional physical
ethernet adapter does not exist in any other LPAR configuration.
The CPU and memory for all EDNA systems is managed by the Partition Load
Manager (PLM) and can be dynamically tuned on an as needed basis.
Dallas Data Center: EDNA Production
- Server-9119-590-SN51A972E
- ddcapedn01 - Oracle Database Server
- ddcapedn03 - PiPe application Server - Primary node
- ddcapedn05 - PiPe application Server - Primary node
- ddcapedn07 - PiPe application Server - Primary node
- Server-9119-590-SN51A973E
- ddcapedn02 - Oracle Database Server
- ddcapedn04 - PiPe application Server - Secondary Node
- ddcapedn06 - PiPe application Server - Secondary Node
- ddcapedn08 - PiPe application Server - Secondary Node
- Cluster Pairs - EDNA Production - Primary - Secondary
- ddcapedn01 - ddcapedn02
- ddcapedn03 - ddcapedn04
- ddcapedn05 - ddcapedn06
- ddcapedn07 - ddcapedn08
Resource Groups
All volume groups, logical volumes, and filesystems have been designed
and implemented using enterprise wide unique identifiers for the purpose
of business continuity and disaster recovery. This enterprise wide
unique identifier is based on the "resource group" name.
Resource Groups configured on the Production EDNA Servers
| Resource Group | Service IP | Application Server | Volume Group |
|---|
| ednpdtu1 | ddcapedn01-rg01 | ednpdtu1_oradb01 | ednpdtu101vg |
| ednpdtu2 | ddcapedn02-rg01 | ednpdtu2_oradb01 | ednpdtu201vg |
| ednpdtu3 | ddcapedn03-rg01 | ednpdtu3_oradb01 | ednpdtu301vg |
| ednpdtu4 | ddcapedn04-rg01 | ednpdtu4_oradb01 | ednpdtu401vg |
| ednpdtu5 | ddcapedn05-rg01 | ednpdtu5_oradb01 | ednpdtu501vg |
| ednpdtu6 | ddcapedn06-rg01 | ednpdtu6_oradb01 | ednpdtu601vg |
| ednpdtu7 | ddcapedn07-rg01 | ednpdtu7_oradb01 | ednpdtu701vg |
| ednpdtu8 | ddcapedn08-rg01 | ednpdtu8_oradb01 | ednpdtu801vg |
Cluster Heartbeat
A cluster heartbeat is configured on the first disk of the shared
concurrent volume group for the purpose of determining cluster health.
Volume Groups, Logical Volumes, and Filesystems
A single logical volume and JFS2 filesystem is configured for each
volume group on each system to provide application storage.
| Volume Group | Logical Volume | Log LV | Filesystem |
|---|
| ednpdtu101vg | ednpdtu1ora1lv | ednpdtu1jfs2lv | /ednpdtu1/u01 |
| ednpdtu201vg | ednpdtu2ora1lv | ednpdtu2jfs2lv | /ednpdtu2/u01 |
| ednpdtu301vg | ednpdtu3ora1lv | ednpdtu3jfs2lv | /ednpdtu3/u01 |
| ednpdtu401vg | ednpdtu4ora1lv | ednpdtu4jfs2lv | /ednpdtu4/u01 |
| ednpdtu501vg | ednpdtu5ora1lv | ednpdtu5jfs2lv | /ednpdtu5/u01 |
| ednpdtu601vg | ednpdtu6ora1lv | ednpdtu6jfs2lv | /ednpdtu6/u01 |
| ednpdtu701vg | ednpdtu7ora1lv | ednpdtu7jfs2lv | /ednpdtu7/u01 |
| ednpdtu801vg | ednpdtu8ora1lv | ednpdtu8jfs2lv | /ednpdtu8/u01 |
Ethernet Network Configuration
Network access to all systems by users shall be through using the DNS
alias name associated with the service name for each resource group.
Each system currently has a single resource group defined that may float
between any node of the cluster. Currently all clusters are configured
with two nodes, a primary and a secondary. The resource group service
name and address is configured on the physical ethernet adapter in each
system (en6). Again, this physical adapter configured on the EDNA
production servers is unique to this environment and is not configured
on any other LPAR.
Server name for user access to production EDNA servers
| Server Name for User Access | RG Service Name | RG IP Address |
|---|
| ednpdtu1 | ddcapedn01-rg01 | 146.61.61.31 |
| ednpdtu2 | ddcapedn02-rg01 | 146.61.61.32 |
| ednpdtu3 | ddcapedn03-rg01 | 146.61.61.33 |
| ednpdtu4 | ddcapedn04-rg01 | 146.61.61.34 |
| ednpdtu5 | ddcapedn05-rg01 | 146.61.61.35 |
| ednpdtu6 | ddcapedn06-rg01 | 146.61.61.36 |
| ednpdtu7 | ddcapedn07-rg01 | 146.61.61.37 |
| ednpdtu8 | ddcapedn08-rg01 | 146.61.61.38 |
An additional resource group is configured on each server to provide
access to the RG service name associated with the node name. The name
of this resource group is the same as the node name. This name and
address is configured to exist on the standard p590 virtualized
etherchannel adapter (en4)
| Node Name | Resource Group Name | RG Service Name | RG IP Address |
|---|
| ddcapedn01 | ddcapedn01 | ddcapedn01 | 146.61.171.56 |
| ddcapedn02 | ddcapedn02 | ddcapedn02 | 146.61.171.58 |
| ddcapedn03 | ddcapedn03 | ddcapedn03 | 146.61.171.60 |
| ddcapedn04 | ddcapedn04 | ddcapedn04 | 146.61.171.62 |
| ddcapedn05 | ddcapedn05 | ddcapedn05 | 146.61.171.64 |
| ddcapedn06 | ddcapedn06 | ddcapedn06 | 146.61.171.66 |
| ddcapedn07 | ddcapedn07 | ddcapedn07 | 146.61.171.68 |
| ddcapedn08 | ddcapedn08 | ddcapedn08 | 146.61.171.70 |
In order to insure access to each system, a persistent address is
configured and exists on the standard p590 virtualized ethernchannel
adapter (en4).
| Node Name | Persistent Name | Persistent IP | Boot Address |
|---|
| ddcapedn01 | ddcapedn01-pers | 146.61.171.57 | 192.168.171.56 |
| ddcapedn02 | ddcapedn02-pers | 146.61.171.59 | 192.168.171.58 |
| ddcapedn03 | ddcapedn03-pers | 146.61.171.61 | 192.168.171.60 |
| ddcapedn04 | ddcapedn04-pers | 146.61.171.63 | 192.168.171.62 |
| ddcapedn05 | ddcapedn05-pers | 146.61.171.65 | 192.168.171.64 |
| ddcapedn06 | ddcapedn06-pers | 146.61.171.67 | 192.168.171.66 |
| ddcapedn07 | ddcapedn07-pers | 146.61.171.69 | 192.168.171.68 |
| ddcapedn08 | ddcapedn08-pers | 146.61.171.71 | 192.168.171.70 |
The virtualized management etherchannel adapter (en5) in each system is
configured with a single IP address on the management network.
| Node Name | Management Name | Management IP Address |
|---|
| ddcapedn01 | ddcapedn01-man | 146.61.172.42 |
| ddcapedn02 | ddcapedn02-man | 146.61.172.43 |
| ddcapedn03 | ddcapedn03-man | 146.61.172.44 |
| ddcapedn04 | ddcapedn04-man | 146.61.172.45 |
| ddcapedn05 | ddcapedn05-man | 146.61.172.46 |
| ddcapedn06 | ddcapedn06-man | 146.61.172.47 |
| ddcapedn07 | ddcapedn07-man | 146.61.172.48 |
| ddcapedn08 | ddcapedn08-man | 146.61.172.49 |
The resource group service names configured on each system are only
available when HACMP is running. When the system first boots, HACMP is
not automatically started, therefore the service names do not yet exist
on the virtualized etherchannel adapter or physical adapter. At boot
time these adapters are configured with non-routed boot addresses.
| Virt EthChan Name | Virt EthChan IP | Phys GigE Name | Phys Gige IP |
|---|
| ddcapedn01-boot | 192.168.171.56 | ddcapedn01-bt01 | 192.168.61.31 |
| ddcapedn02-boot | 192.168.171.58 | ddcapedn01-bt01 | 192.168.61.32 |
| ddcapedn03-boot | 192.168.171.60 | ddcapedn01-bt01 | 192.168.61.33 |
| ddcapedn04-boot | 192.168.171.62 | ddcapedn01-bt01 | 192.168.61.34 |
| ddcapedn05-boot | 192.168.171.64 | ddcapedn01-bt01 | 192.168.61.35 |
| ddcapedn06-boot | 192.168.171.66 | ddcapedn01-bt01 | 192.168.61.36 |
| ddcapedn07-boot | 192.168.171.68 | ddcapedn01-bt01 | 192.168.61.37 |
| ddcapedn08-boot | 192.168.171.70 | ddcapedn01-bt01 | 192.168.61.38 |
From this information, it can be derived that during nominal operation,
any production EDNA machine will have three active network interfaces.
Each interfaces will have one or more IP addresses configured on it:
The adapter "en4" will have 1 boot address on the "192.168.171.0"
network, 1 persistent address on the "146.61.171.0" network, and 1
resource group service address on the "146.61.171.0" network.
The adapter "en5 will have 1 management address on the "146.61.172.0"
network.
The adapter "en6" will have 1 boot address on the "192.168.61.0"
network, and 1 resource group service address on the "146.61.61.0"
network.
If a machine is running in "failover" mode, the adapters "en4" and "en6"
may have additional service addresses configured.
User Names and UID Numbers
Two service user names were added to the Oracle database servers for
managing Oracle, one for each instance of Oracle running on each cluster
node. These service user names were created for the purpose of
providing the DBA's with access to the system to manage Oracle. The UID
number is a calculated value based on the BSD checksum of each user
name.
The application servers provide two service user names: "pipe" and
"pipetran". These user names are used for managing the eMeter PiPE
application and for transfering files between systems. Direct login
access to these service user names shall be turned off, thus requiring
the user to first login using their individual login ID. Multiple
individual login ID's are configured in the form of RACF ID's. Several
eMeter personnel have access to these system for maintenance and
support. The UID for all user names shall be a calculated value based
on the BSD checksum of each user name.
Group Names and GID Numbers
One group name was added to the Oracle database servers for association
with the Oracle service user names. The GID number is a calculated
value based on the BSD checksum of the group name.
Mesquite Data Center: EDNA Test development
- Server-9119-590-SN51A971E
- mdcadedn01 - Oracle Database Server
- mdcadedn03 - PiPe application Server - Primary node
- mdcadedn05 - PiPe application Server - Primary node
- mdcadedn07 - PiPe application Server - Primary node
- Server-9119-590-SN51A974E
- mdcadedn02 - Oracle Database Server
- mdcadedn04 - PiPe application Server - Secondary Node
- mdcadedn06 - PiPe application Server - Secondary Node
- mdcadedn08 - PiPe application Server - Secondary Node
- Cluster Pairs - EDNA Test Development - Primary - Secondary
- mdcadedn01 - mdcadedn02
- mdcadedn03 - mdcadedn04
- mdcadedn05 - mdcadedn06
- mdcadedn07 - mdcadedn08
Resource Groups
All volume groups, logical volumes, and filesystems have been designed
and implemented using enterprise wide unique identifiers for the purpose
of business continuity and disaster recovery. This enterprise wide
unique identifier is based on the "resource group" name.
Resource Groups configured on the Pre-Production EDNA Servers
| Resource Group | Service IP | Application Server | Volume Group |
|---|
| ednadtu1 | mdcadedn01-rg01 | ednadtu1_oradb01 | ednadtu101vg |
| ednadtu2 | mdcadedn02-rg01 | ednadtu2_oradb01 | ednadtu201vg |
| ednadtu3 | mdcadedn03-rg01 | ednadtu3_oradb01 | ednadtu301vg |
| ednadtu5 | mdcadedn05-rg01 | ednadtu5_oradb01 | ednadtu501vg |
| ednadtu7 | mdcadedn07-rg01 | ednadtu7_oradb01 | ednadtu701vg |
Cluster Heartbeat
A cluster heartbeat is configured on the first disk of the shared
concurrent volume group for the purpose of determining cluster health.
Volume Groups, Logical Volumes, and Filesystems
A single logical volume and JFS2 filesystem is configured for each
volume group on each system to provide application storage.
| Volume Group | Logical Volume | Log LV | Filesystem |
|---|
| ednadtu101vg | ednadtu1ora1lv | ednadtu1jfs2lv | /ednadtu1 |
| ednadtu201vg | ednadtu2ora1lv | ednadtu2jfs2lv | /ednadtu2 |
| ednadtu301vg | ednadtu3ora1lv | ednadtu3jfs2lv | /ednadtu3 |
| ednadtu501vg | ednadtu5ora1lv | ednadtu5jfs2lv | /ednadtu5 |
| ednadtu701vg | ednadtu7ora1lv | ednadtu7jfs2lv | /ednadtu7 |
Ethernet Network Configuration
Network access to all systems by users shall be through using the DNS
alias name associated with the service name for each resource group.
Each system currently has a single resource group defined that may float
between any node of the cluster. Currently all clusters are configured
with two nodes, a primary and a secondary. The resource group service
name and address is configured on the virtual etherchannel adapter in each
system (en4).
User Access to Pre-Production EDNA Servers
| Server Name for User Access | RG Service Name | RG IP Address |
|---|
| ednadtu1 | mdcadedn01-rg01 | 146.61.122.206 |
| ednadtu2 | mdcadedn02-rg01 | 146.61.122.219 |
| ednadtu3 | mdcadedn03-rg01 | 146.61.122.207 |
| ednadtu5 | mdcadedn05-rg01 | 146.61.122.208 |
| ednadtu7 | mdcadedn07-rg01 | 146.61.122.209 |
An additional resource group is configured on each server to provide
access to the RG service name associated with the node name. The name
of this resource group is the same as the node name. This name and
address is configured to exist on the standard p590 virtualized
etherchannel adapter (en4)
| Resource Group Name | RG Service Name | RG IP Address |
|---|
| mdcadedn01 | mdcadedn01 | 146.61.122.190 |
| mdcadedn02 | mdcadedn02 | 146.61.122.192 |
| mdcadedn03 | mdcadedn03 | 146.61.122.194 |
| mdcadedn04 | mdcadedn04 | 146.61.122.196 |
| mdcadedn05 | mdcadedn05 | 146.61.122.198 |
| mdcadedn06 | mdcadedn06 | 146.61.122.200 |
| mdcadedn07 | mdcadedn07 | 146.61.122.202 |
| mdcadedn08 | mdcadedn08 | 146.61.122.204 |
In order to insure access to each system, a persistent address is
configured and exists on the standard p590 virtualized ethernchannel
adapter (en4).
| Persistent Name | Persistent IP Address | Boot Address |
|---|
| mdcadedn01-pers | 146.61.122.191 | 192.168.30.60 |
| mdcadedn02-pers | 146.61.122.193 | 192.168.30.61 |
| mdcadedn03-pers | 146.61.122.195 | 192.168.30.62 |
| mdcadedn04-pers | 146.61.122.197 | 192.168.30.63 |
| mdcadedn05-pers | 146.61.122.199 | 192.168.30.64 |
| mdcadedn06-pers | 146.61.122.201 | 192.168.30.65 |
| mdcadedn07-pers | 146.61.122.203 | 192.168.30.66 |
| mdcadedn08-pers | 146.61.122.205 | 192.168.30.67 |
The virtualized management etherchannel adapter (en5) in each system is
configured with a single IP address on the management network.
| Management Name | Management IP Address |
|---|
| mdcadedn01-man | 146.61.124.50 |
| mdcadedn02-man | 146.61.124.51 |
| mdcadedn03-man | 146.61.124.52 |
| mdcadedn04-man | 146.61.124.53 |
| mdcadedn05-man | 146.61.124.54 |
| mdcadedn06-man | 146.61.124.55 |
| mdcadedn07-man | 146.61.124.56 |
| mdcadedn08-man | 146.61.124.57 |
The resource group service names configured on each system are only
available when HACMP is running. When the system first boots, HACMP is
not automatically started, therefore the service names do not yet exist
on the virtualized etherchannel adapter. At boot time this adapter is
configured with non-routed boot address.
| Virt EthChan Name | Virt EthChan IP |
|---|
| mdcadedn01-boot | 192.168.30.60 |
| mdcadedn02-boot | 192.168.30.61 |
| mdcadedn03-boot | 192.168.30.62 |
| mdcadedn04-boot | 192.168.30.63 |
| mdcadedn05-boot | 192.168.30.64 |
| mdcadedn06-boot | 192.168.30.65 |
| mdcadedn07-boot | 192.168.30.66 |
| mdcadedn08-boot | 192.168.30.67 |
From this information, it can be derived that during nominal operation,
any pre-production EDNA machine will have two active network interfaces.
Each interface will have one or more IP addresses configured on it:
The adapter "en4" will have 1 boot address on the "192.168.30.0"
network, 1 persistent address on the "146.61.122.0" network, and 2
resource group service addresses on the "146.61.122.0" network.
The adapter "en5 will have 1 management address on the "146.61.124.0"
network.
If a machine is running in "failover" mode, the adapter "en4"
may have additional service addresses configured.
User Names and UID Numbers
Two service user names were added to the Oracle database servers for
managing Oracle, one for each instance of Oracle running on each cluster
node. These service user names were created for the purpose of
providing the DBA's with access to the system to manage Oracle. The UID
number is a calculated value based on the BSD checksum of each user
name.
The application servers provide two service user names: "pipe" and
"pipetran". These user names are used for managing the eMeter PiPE
application and for transfering files between systems. Direct login
access to these service user names shall be turned off, thus requiring
the user to first login using their individual login ID. Multiple
individual login ID's are configured in the form of RACF ID's. Several
eMeter personnel have access to these system for maintenance and
support. The UID for all user names shall be a calculated value based
on the BSD checksum of each user name.
Group Names and GID Numbers
One group name was added to the Oracle database servers for association
with the Oracle service user names. The GID number is a calculated
value based on the BSD checksum of the group name.
|
