PROJECT INFORMATION Enterprise Template Architecture ESB Regional... · format and sends these...
Transcript of PROJECT INFORMATION Enterprise Template Architecture ESB Regional... · format and sends these...
Page 1 of 61
PROJECT INFORMATION Template
Enterprise Architecture
Title:
Enterprise Integration Platform Regional Design
Pre-Transfer
Unique Identifier: 206-1313
Document Type: ETE
Revision: 3.1
Total pages: 61
Revision date: September 2017
Classification:
PUBLIC/CONFIDENTIAL/ CONTROLLED DISCLOSURE
Page 2 of 61
Table of Contents 1 Project Information ..................................................... Error! Bookmark not defined.
1.1 Relevant Business Units ...................................... Error! Bookmark not defined.1.2 List of Attached Supporting Documents ............... Error! Bookmark not defined.
2 Introduction ................................................................................................................ 33 Background ................................................................................................................ 34 Scope ......................................................................................................................... 55 Design Approach ........................................................................................................ 66 EIP Architecture Design ............................................................................................. 8
6.1 Business Architecture ........................................................................................... 86.1.1 Business Strategies and Plans ...................................................................... 86.1.2 Business Processes and Policies .................................................................. 96.1.3 Business Organisation Design ....................................................................... 9
6.2 Business Information Architecture ........................................................................ 96.2.1 Business Information Flow/Inventory ............................................................. 96.2.2 Information Integrity and Custodianship ........................................................ 96.2.3 Information Access and Confidentiality ........................................................ 106.2.4 Information Related Business Continuity ..................................................... 10
6.3 Data Architecture ................................................................................................ 116.3.1 Data Models ................................................................................................. 11
6.4 Application Architecture ...................................................................................... 116.4.1 EIP Regional Platform Application Architecture Deployment ...................... 116.4.2 EIP Application Functional Decomposition .................................................. 126.4.3 Application Strategy ..................................................................................... 226.4.4 Application Development ............................................................................. 22
6.5 Integration Architecture ...................................................................................... 226.5.1 Integration Interface Map ............................................................................. 226.5.2 Integration Services Directory ...................................................................... 236.5.3 Data Migration ............................................................................................. 23
6.6 Technical Architecture ........................................................................................ 246.6.1 Basic Infrastructure ...................................................................................... 24
6.7 Network Architecture .......................................................................................... 421.1.1 IT & DMZ Restrictions .................................................................................. 421.1.2 ODA Placement ........................................................................................... 441.1.8 Firewall Access ............................................................................................ 48
6.8 Security Architecture .......................................................................................... 517 High Availability and Disaster Recovery .................................................................. 608 Operational Support Structure ................................................................................. 60
Page 3 of 61
1 Introduction
The Enterprise Integration Programme (EIP) was launched to replace the legacy integration systems also known as the Enterprise Service Bus (ESB). For more than 10 years, Eskom has utilised SUN Java CAPS and SeeBeyond 4.5.3 as the platform for intra-application communication, across multiple divisions and business areas. In 2010 Oracle acquired SUN and a directive was sent that stipulates that the SUN Java CAPS tooling would be discontinued. In November 2014 Eskom acquired Oracle’s Fusion Middleware (OFMW) suites as a replacement for its Seebeyond and JavaCaps platforms. The 1st phase of this program was to design and implement OFMW platform in central environment and this was completed in December 2015. The program is currently in the migrating the centrally hosted integration paths that are deployed on the legacy platform onto OFMW. In addition to the central environment some regional Operating Units (OU) also has legacy ESB implementations; these are extensions of the central legacy integration platforms. The primary objective of these implementations is to allow the central corporate environment to integrate with the regional operational technology environment (OT) environment. The purpose of this document is to describe the detailed EIP platform infrastructure architecture that is going to replace the above mentioned implementations with OFMW at the various regional OU sites. The document delves further into why there is a need for an integration platform at the regions followed by the scope of the design. Thereafter the EIP design pertaining to the various architecture domains including security will be detailed.
2 Background An Enterprise Service Bus (ESB) as shown in Figure 1: ESB Concept is an open standard, message-based, distributed integration infrastructure that provides routeing, invocation and mediation services that facilitate interactions of disparate distributed applications and services in a secure reliable manner. The ESB is usually implemented through service containers distributed across a networked environment. These containers host integration services like routers, transformers, application adapters and provide them with a range of communication facilities.
Page 4 of 61
Figure 1: ESB Concept
The current ESB platform deployed at the regions allows for information integration between the applications deployed on the corporate networks, also known as the IT network, and applications deployed on regional OT networks at the various OUs. The following solutions leverage the existing regional ESB platform:
1. The SmallWorld system is responsible for managing the geographical data for network devices and substations. It assists in maintaining the data on SCADA by means of a Network Change Set message.
2. The Engineering Network Schematics (ENS) system resides between SmallWorld GDC and SENS and generates schematic representations of the Geographic Data within SmallWorld.
3. SENS (SCADA ENS) system adapts the ENS schematics into the SCADA
format and sends these diagrams to SCADA by means of Network Diagram Change-set message. SENS also requires Telemetry information from SCADA and this is sent in the form of SCADA Point Data Set, otherwise known as List of Values, messages from SCADA.
4. The Fault Management System (FMS) is responsible for fault management and
enables automated logging of network events. It performs network tracing and stores the trace data locally for future reference.
5. The Business Events Notifier (BEN) system is a web based real time notification system that creates alerts based on the Alarm Event messages received from SCADA. These alerts are sent out in the form of an email or SMS.
Page 5 of 61
The approach the EIP project is taking with regards to the implementation is that the legacy platform will be maintained and operational until all the regional integration solutions are migrated over to the new EIP platform in a phased approach. Thus the interim architecture for a duration will have two platforms co-existing hosting various integration solutions. The architecture roadmap for this transition illustrates how the final integration architecture will manifest itself.
Figure 2 : Architecture Roadmap
3 Scope The scope of this design is the implementation of the regional ESB platforms which will be leveraged in the migration of existing or future integration solutions that require an ESB capability in the regions. It excludes actual migration of the solutions that are currently deployed in the regions. This will be initiated as a separate project whilst the integration platform implementation is being executed. Details of the current integration solutions are outlined in the Background . The design will encompass the following environments:
• Production for each region • Disaster Recovery (Swap out strategy) • Quality Assurance • Development
Page 6 of 61
4 Constraints The following constrains were imposed on the EIP program
1) The design of the platform should not impact existing business processes. 2) The platform should perform in a functional like-for-like manner to the legacy
ESB as not to impact the implementation of end systems. 3) The legacy ESBs will only be decommissioned once all integration solutions have
been migrated to the OFMW.
5 Design Approach The preliminary phase of the design involved analyzing the existing deployment topology of the ESB platforms at the various regions. One fundamental finding was that the IT and OT environments were segregated using a demilitarized zone (DMZ) implementation. The purpose of this segregation is:
1. Decoupling – To shield either side from changes in implementation 2. Security – To prevent direct access to the OT network via the IT network
This segregation manifested itself via two fundamental network topology patterns depending on the site in question. For the rest of this document these patterns will be referred to as the Type A and Type B pattern. The Type A deployment pattern shown in Figure 3 : Type A Regional ESB Deploymentconsists of:
1. An ESB platform connected to the IT network deployed at an OU. The purpose of this platform is to extend the central integration IT footprint to the region.
2. Dual firewall DMZ, the purpose of this DMZ is to segregate the IT and OT network.
3. An ESB platform implementation deployed in the above mentioned DMZ , the purpose of this platform is extend the local IT integration platform footprint into the OT environment allowing for an indirect communication channel between the central and regional OT environment
Page 7 of 61
Figure 3 : Type A Regional ESB Deployment The table below details the sites that conform to the Type A pattern Table 1: Type A Sites
SiteName TypeSimmerpan ABellville ABloemfontein A–ITOnlyMkondeni AEastLondon A The Type B pattern shown in Figure 4: consists of:
1. Two IT ESB platforms that are geographically isolated by being deployed at separate regional OUs. The purpose of these platforms is the same as the Type A IT deployment.
2. Dual firewall DMZ, the purpose of this DMZ is to segregate the IT and OT network. This DMZ is deployed at a 3rd regional OU separate that is separated to the above mentioned point 1 deployment.
3. An ESB platform deployed in the above mentioned DMZ. The purpose of this
platform is to allow for indirect connectivity between the single OT network and the two IT implementations mentioned in point 1
Page 8 of 61
Figure 4: Type B Regional ESB Deployment
The table below details the sites that conform to the Type B design
Table 2: Type B Sites SiteName Type
Duvha B–DMZOnlyPolokwane B–ITonlyWitbank B–ITonly
Due to the constraints mentioned in section 4 we had to follow the current infrastructure deployment of Type A and Type B patterns when designing the new EIP ESB platform.
6 EIP Architecture Design The following sections detail the architecture design of the ESB platform that will be deployed per regional site and its functionality relative to the central ESB implementation. This is achieved by describing each of the TOGAF 9.1 standard architecture domains Business Information (Application) Data Technology (BIDAT) and the security architecture that applies to the EIP regional ESB platform.
6.1 Business Architecture
6.1.1 Business Strategies and Plans The Enterprise Integration Programme seeks to replace the legacy integration systems. The 2012 Group IT Business Plan - Enterprise Integration Programme initiative is unique and crucial to business success as it underpins the transport of key information in Eskom’s business, including the safety and revenue systems/application The current short –term objective of the EIP platform is to migrate all existing SeeBeyond and Java Caps integration interfaces to the new OFMW platform. The in
Page 9 of 61
addition to this any new integration requirements will be developed on the EIP platform where applicable.
6.1.2 Business Processes and Policies The purpose of the platform as previously mentioned will impact the major EHPUM processes and policies due to the fact that it is the primary platform for transporting data between all major systems within the Enterprise.
6.1.3 Business Organisation Design The functional responsibility of the Integration Centre of Excellence (ICoE) is the deliver and maintain the IT systems integration within it the organisation . Hence the business owner of the EIP platform would be the ICoE. The figure below illustrates where the ICoE is placed relative to the rest of the enterprise. The support of the platform will be shared across multiple entities within Group IT in partnership with Oracle and MSA provider..
Figure 5 :ICoE Business Context
6.2 Business Information Architecture
6.2.1 Business Information Flow/Inventory TBC
6.2.2 Information Integrity and Custodianship The Table 3: Oracle Fusion Middleware Users, Groups and Roles define how user access can be restricted on the platform. Please note that these are not application level groups and roles but rather specifies how individuals can access platform from an operational perspective.
Page 10 of 61
Table 3: Oracle Fusion Middleware Users, Groups and Roles Group Membership
Administrators By default, this group contains the user information entered as part of the installation process (that is, the Configuration Wizard), and the system user if the WebLogic Server instance is running Compatibility security. Any user assigned to the Administrators group is granted the Admin security role by default.
Deployers By default, this group is empty. Any user assigned to the Deployers group is granted the Deployer security role by default.
Operators By default, this group is empty. Any user assigned to the Operators group is granted the Operator security role by default.
Monitors By default, this group is empty. Any user assigned to the Monitors group is granted the Monitor security role by default.
AppTesters By default, this group is empty. Any user assigned to the AppTesters group is granted the AppTester security role by default.
CrossDomainConnectors By default, this group is empty. Any user assigned to the CrossDomainConnectors group is granted the CrossDomainConnector security role by default.
AdminChannelUsers By default, this group is empty. Any user assigned to the AdminChannelUsers group is granted the AdminChannelUser security role by default.
OracleSystemGroup By default, this group contains the user OracleSystemUser and is granted the OracleSystemRole role by default.
6.2.3 Information Access and Confidentiality The platform itself as previous mentioned is a means to transport data from one enterprise system to therefore minimal enterprise information that will be persisted on it. The only instances where information that would be persisted is,
1) To maintain state of an integration path 2) Error and audit logs
In both instances access to these data stores is restricted by the various out of the box OFM security measures. Data is either persisted in the file system or in data bases and both have restricted access by virtue of user name and password depending on the context of the data store. The section on the Error! Reference source not found. of the platform will detail this further
6.2.4 Information Related Business Continuity The EIP platform is classified as a safety and revenue critical system therefore,
1) It is a 24x7 systems 2) Recovery point objective of data being - TBC 3) Recovery time objective of – TBC
Page 11 of 61
6.3 Data Architecture
6.3.1 Data Models Not applicable
6.4 Application Architecture
6.4.1 EIP Regional Platform Application Architecture Deployment All of Eskom’s IT integration requirements and development is managed by a single body, the ICoE (Integration Centre of Excellence), and as result of this that all integration requirements will follow the same governance and architecture standards. Taking this into account the EIP regional ESB deployment design follows a centralized federated integration topology as shown in Figure 6. Centralized Federation Topology is a specific setup in which the ESBs are deployed at the regions using a well-defined hierarchical relationship.
Figure 6: Centralized Federation Topology
The deployment consists of :
1) Centrally hosted ESB which acts as the master which has already been deployed
2) Two types of regional deployment patterns
Page 12 of 61
a. Two Scaled down instance of the master bus region deployed at each of the regions IT and DMZ environments as slaves consistent with the current deployment pattern as in Figure 3 : Type A Regional ESB Deployment.
b. A scaled down instance of the master bus deployed in the DMZ and two scaled downs instances deployed in the IT environment consistent with the current Figure 4: Type B Regional ESB Deployment.
This topology requires all the regional or federated Service Buses to route the requests to one another via the central backbone Master Service Bus. One key assumption is that the regional integration instances currently do not have a need to communicate directly with each other. The rationale for this design is,
1) Central service bus acts as the integration mediator and orchestrator between the end systems deployed in central environment and the regional OT systems.
2) The regional services buses act as integration service intermediaries to the regional OT systems. This means that communication to the OT systems will be routed via these regional instances.
3) The regional service bus composition is further decomposed into two instance per region because
a. Direct access to the OT environment is not allowed from a security perspective
b. Integration buffering capability on the IT and OT environments. The capability will buffer traffic in the event that either the IT or OT network is disrupted any reason.
The master Service Bus has already been deployed as part of the 1st phase of the EIP project. This system is currently hosted in MWP on the Oracle Engineered Systems hardware. The 2nd phase of the project which is the focus of this design is the deployment architecture the regional slave service bus implementations. The following section will detail the fictional capability provided by each instance of the regional ESBs’
6.4.2 EIP Application Functional Decomposition Each instance of the regional service bus will be implemented using the Eskom standard OFMW .The OFMW suite is a purpose built ESB platform that provides industry standard ESB functionality to the enterprise. OFMW was chosen as the ESB standard in Eskom in March 2015 and was implemented in the central environment. In keeping with the central integration standard the regional integration platforms is also implemented using OFMW.
Page 13 of 61
This suite gives the enterprise the capability to develop different types of middleware applications. This achieved by leveraging components such as,
• Business Process Management • BPEL process manager • Proxy • Technology and vendor adapters (connectors) • Enterprise Service Repository • Web Services Manager • Business Activity Monitoring • Could Management Console
The platform is built on top of the Oracle J2EE application server Weblogic and uses this as its runtime environment. In addition to this Weblogic provides the following functionality,
• Clustering • Performance Monitoring • Diagnostics • Java Messaging Services (asynchronous integration) • Diagnostics • Caching Frameworks
The Enterprise embarked on an unlimited user licences (ULA) agreement with Oracle which allows the organisation to have an unlimited number of instances of a particular application contained in the Oracle ULA. The Figure 7: OFM Suite Decomposition below defines the middleware applications that are covered by the ULA and are deployed by the EIP.
Figure 7: OFM Suite Decomposition
Page 14 of 61
The key solution building blocks that define the define middleware application architecture for the regions is defined below,
1. Oracle SOA Suite
o ORACLE SERVICE BUS (OSB)
Oracle Service Bus is an enterprise service bus (ESB) that provides a virtualization layer required for any integration architecture. Using Service Bus, organizations can decouple service consumers from changes that might occur in the backend. They can also hide from developers the often intricate and complex details of underlying implementations of back-end applications, such as legacy protocols
o BUSINESS PROCESS EXECUTION LANGUAGE (BPEL) PROCESS
MANGER – (BPEL-PM)
Oracle BPEL provides a set of discrete services into an end-to-end process flow, reducing the cost and complexity of process integration. It executes standard BPEL processes and provides a “dehydration” capability so that the state of long-running flows is automatically maintained in a database, enabling clustering for both fail-over and scalability. The built-in human workflow capabilities of Oracle SOA Suite allow for people to be included in these processes for approvals and reviews
o CONNECTORS
Oracle SOA Suite provides a connectivity layer, enabling connectivity to a data source inside as well as outside the enterprise. Oracle Adapters (Database, JMS, file etc.) are available for on-premise applications and technology. In addition, B2B & Managed File Transfer capabilities are included to extend processes to external business partners.
2. Oracle Web Service Manager (OWSM) Oracle Web Service Manager provides the first line security for client agent and last line security via server agents. Whether services are accessed within the enterprise or externally they may require authentication and authorization in accordance with the organization’s security policy or regulatory compliance. OWSM is centralized, declarative, externalized and consistent.
3. Weblogic Server
Oracle WebLogic Server is a Java Platform, Enterprise Edition (Java EE) application server. The WebLogic Server infrastructure supports Oracle Fusion Middleware and other applications and is a foundation for building SOA based applications.
4. Oracle Enterprise Manager (OEM) Cloud Control
Page 15 of 61
Oracle Enterprise Manager provides visibility into your application servers and their resident applications. Oracle Enterprise Manager and the associated SOA Management Pack plug-in provide these capabilities in an easy to use web console. Using Oracle Enterprise Manager, you can monitor your running servers, applications and service engines: to facilitate trouble-shooting at runtime within your enterprise SOA environment. The SOA Management Packs for Oracle Enterprise Manager 12c introduce the Java VM Diagnostics as a Service capability, which allows applications and middleware administrators to provide Java VM diagnostics capabilities directly to developers and QA engineers on an as needed basis. Users are provisioned automatically and receive their own self-service portal for accessing diagnostics capabilities. Oracle Enterprise Manager does more than provide visibility into your enterprise SOA environment: it also works with Oracle Web Services Manager to allow you to define security policies for your services and components and to apply those security policies as needed. This separates security management from application development, a best practice in the security world. This allows you to evolve and implement your security strategy outside of application development, providing you with greater agility and flexibility
5. SOA Management Pack
SOA Management Pack with Oracle Enterprise Manager Cloud Control facilitates monitoring of Oracle SOA Suite and OSB. It provides administrators of the SOA environment with a consolidated browser-based view of the entire enterprise. This enables administrators to monitor and manage all of their components from a central location. It stores the collected metric and configuration data in a central repository, thereby enabling administrators to analyse metrics through various historical views and facilitate strategic trend analysis and reporting.
6. WebLogic Management Pack
WebLogic Server Management Pack Enterprise Edition greatly improves server as well as application performance by providing unique functionality to automatically detect performance bottlenecks; quickly diagnose these performance problems, and identify their root cause.
WebLogic Server Management Pack Enterprise Edition provides common administration operations - traditionally available from the Oracle Enterprise Manager Fusion Middleware Control console or the WebLogic Server Administration Console – directly from the Cloud Control console. Consequently, a single console can be used to centrally administer multiple domains.
The for each regional service bus instance the recommended deployment model was to expose a single central Enterprise Service Bus (ESB) with multiple SOA Domains. These logical domains will be used to containerize integration components that require some form of state to be maintained, and it will be split on non-functional requirements such as volumes, payload sizes and transaction times, as shown in Figure 8: SOA Federation
Page 16 of 61
Figure 8: SOA Federation The platform is architecturally comprised of 4 tiers,
1. Application tier – Systems that leverage that integration platform for intra-application communication
2. Middle tier – The integration components are deployed on this tier 3. Monitoring tier- Oracle Enterprise Manager will act as a central monitoring and
management tool for all the databases and middleware components deployed 4. Data tier - Oracle Fusion Middleware is reliant on the database as it uses it to
manage state The above mentioned is depicted in Figure 9 : EIP Platform Tiring
Page 17 of 61
Figure 9 : EIP Platform Tiring
6.4.2.1 Regional Middleware Tier Domains and Clustering Structure
A domain is an interrelated set of WebLogic Server resources that are managed as a unit. A domain also contains the application components deployed in the domain, and the resources and services required by those application components and the server instances in the domain. In each domain, one WebLogic Server instance acts as the Administration Server - the server instance which configures, manages, and monitors all other server instances and resources in the domain. Each Administration Server manages one domain only
WebLogic Server clusters provide scalability and reliability by distributing the work load among multiple instances of WebLogic Server. Incoming requests can be routed to a WebLogic Server instance in the cluster based on the volume of work being processed. In case of hardware or other failures, session state is available to other cluster nodes that can resume the work of the failed node. Domains can contain both managed servers and clusters.
The server instances that constitute a cluster can run on the same machine, or be located on different machines. You can increase a cluster's capacity by adding additional server instances to the cluster on an existing machine, or you can add machines to the cluster to host the incremental server instances.
Page 18 of 61
All server instances in a cluster must reside in the same domain. Clustered WebLogic Server instances behave similarly to non-clustered instances, except that they provide failover and load balancing. The process and tools used to configure clustered WebLogic Server instances are the same as those used to configure non-clustered instances.
Based on the Federated SOA Domain architecture, two Traffic Director and one Service Bus will be deployed with multiple SOA Domains, each optimized for specific non-functional requirements. These domains will be supplemented by a JMS domain to host all queues and topics. Each of the middleware tier components are segregated into Weblogic domains. Each of these domains are specialized for its intended purpose of the products deployed in it. The regional ESB domains shown in Figure 10 are:
• OTD – The OTD domain is specialized for the purposes of Oracle Traffic Director. This will be used for load balancing within the ODA system as well as fronting incoming communications.
• OSB – The OSB domain is specialized for Oracle Service Bus. Oracle Service Bus is used for simple mediation, routing and transformation that requires no orchestration.
• SOA – The SOA domains are specialized for SOA Suite. SOA Suite is BPEL-based and hence process-aware. The SOA Suite functionality has been subdivided into two domains optimized for different purposes:
o SOA 1 - Optimized for small payload, high volume and low latency interfaces.
o SOA 2 - Optimized for large payloads where streaming will enhance performance and increase stability.
• JMS – The JMS domain will be optimized for supporting JMS servers.
Page 19 of 61
ODA
IT DMZ
OTD
OSB
SOA1 SOA2
JMS
SOADB JMSDB CSMDB
ODA
OTD
OSB
SOA1 SOA2
JMS
SOADB JMSDB
Figure 10: EIP Domain Structure for the IT and DMZ instance at each site
The Table 4 : Summary of EIP Domains summarizes the domains that will be deployed, and the intended use for each domain
Table 4 : Summary of EIP Domains
Domain Components Deployed Usage SOA Suite 1 • SOA Suite
• Web Services Manager • Vanilla WebLogic
Cluster for JEE
Domain will be optimised for small payload, high volume and low latency interfaces
SOA Suite 2 • SOA Suite • Web Services Manager • Vanilla WebLogic
Cluster for JEE
Domain will be optimised for large payloads where streaming will enhance performance and increase stability
OSB • OSB • Web Services Manager
Mediation, Routing and Transformation
JMS • JMS Asynchronous messaging OTD • OTD Used for load balancing
within the ODA system.
6.4.2.1.1 OTD Domain An instance of Oracle Traffic Director will be deployed to handle internal communication and inbound load balancing requirements. The instance will be clustered across two compute nodes for HA purposes
6.4.2.1.2 OSB Domain
Page 20 of 61
OSB will be the entry point for all synchronous integrations within each environment, and as such it will be clustered across four vServers to start off with. Capacity can easily be increased by scaling the domain horizontally should the need arise. Oracle Web Services Manager (OWSM / WSM) will be deployed as part of the domain in order to enable the service bus to apply security policies to inbound and outbound services. OWSM will be deployed on separate managed servers within a separate cluster as per the recommended approach. This reduces the load on the OSB servers when security has to be applied. The admin server can also be failed over to each of the vServers due to the u01 shared storage that will be hosting the domain and admin server. The following sections will detail the clusters and managed servers will be configured for the OSB domain.
6.4.2.1.3 Oracle SOA Suite Domains As detailed in the overview of the Application Architecture, multiple SOA domains will be deployed split on non-functional requirements. This allows for specific SOA-INFRA and JVM optimizations to ensure solutions run optimally within the SOA domains.
The following table defines the decision making Table 5 : SOA Domain decision matrix should be used as input when deciding which domain will host a specific solution
Table 5 : SOA Domain decision matrix
Domain Payload Size > 5120 KB
Throughput > 10ms/sec
Default
prdsoad100 X X prdsoad200 X
SOA Domain 2 will be extended with Enterprise Scheduling Services and Managed File Transfer components in order to future proof the domain architecture. Apart from the above mentioned components, the domains will have the same managed server and component configurations. Ports will differ between the two domains to simplify the OTD load balancing algorithms. Again the admin server can be failed over between the vServers for a domain due to the domain and admin server running from /u01 shared storage. The following section will detail the domains, clusters and managed servers deployed for each SOA domains.
6.4.2.1.4 JMS Domain The JMS domain will be responsible for all asynchronous messaging within the integration platform. Two JMS clusters will be deployed within the domain, one dedicated to functional, business impacting messages and the other to non-functional, less critical messages like “error & audit” messages. From a JMS persistence perspective, data tier will be used.
Page 21 of 61
Domain level JMS configuration is the highest level of configuration in terms of a JMS with regards to WebLogic. The domain level configuration are broader arching and are categorized as either environmental-related or application-related. Environmental configuration examples are definitions and identification of the JMS servers and data sources, persistent stores and network addresses, JMS Modules, Bridging and Store-And-Forward. These JMS configurations are stored as modules defined in XML similar to standard J2EE and can be deployed as J2EE managed modules or standalone modules within a domain. These specific configurations usually differ between domains.
JMS Servers are targeted at a specific managed server, and all managed servers within a cluster share availability of managed resources, in turn making them available to all deployments within that cluster.
Different JMS Servers are targeted at different Managed Servers. These are also managed from the WebLogic administrator console. JMS Servers are targeted at specific clusters, making them available to all deployments within that cluster.
JMS Modules are the configuration hub for lower level JMS items such as Connection Factories, Queues and Topics and their JNDI contexts. The JMS Modules are all responsible for their security at a module level, which means each module can have specific roles and security policies depending on requirements.
There are different types of Queues and Topics configurations available on WebLogic, such as standard Queues (specific to a module) or a Distributed Queue (which is uniform across a cluster)
Queues and Topics have the lowest level of configuration in the hierarchy, but the most specific. You can configure things such as thresholds and quotas, overrides in terms of delivery, security and control.
6.4.2.2 Data Tier Domain and Cluster Structure
Database tier currently has two functions
1. Maintain state for the OFM components 2. Non-functional integration components such as Error and Audit
The database version Oracle 12.1.0.2.0 will be installed for Eskom. The Table 6: EIP Databases below define the databases that will be deployed per environment. The following Fusion Middleware components interact with the database layer:
• OWSM connect to the database to access the OWSM Repository. The repository stores OWSM metadata, such as policies, policy sets, assertions templates, and policy usage data.
• Managed File Transfer (MFT) stores configuration data in an Oracle Metadata Repository. You can edit, back up, and restore this configuration data
• Oracle Business Process Management (BPM) stores process instance data in a database schema called "SOAINFRA"
• SOA Suite is using the database to store state information.
Page 22 of 61
• For Oracle Service Bus (OSB) most of the internal data-structures that are required by OSB are stored in-memory. Only reporting functionality of OSB requires DB tables to be accessible.
• Weblogic Java Messaging Server (JMS) uses persistent storage (JDBC-accessible database) for storing persistent message data
Table 6: EIP Databases Database Name Database
Description Service Name State
SOA Database will host all the Fusion Middleware Schemas
OSB_Service Active SOA_Service
JMS Used as JMS persistence store.
JMS_Service Active
CSM Application database Change Set Management
CSM_Service Active
6.4.3 Application Strategy Please refer to Business Strategies and Plans for the migration strategy
6.4.4 Application Development Oracle J Developer will be used as the platform to build OFM components that are going to be deployed in the EIP environments.
6.5 Integration Architecture
6.5.1 Integration Interface Map
Page 23 of 61
Figure 11 : System Integration Map The following Enterprise Systems have been integrated with EIP to deliver the required platform functionality
1) F5 – Load balancing traffic into the OFM environment 2) OEM – Monitoring of the fusion environment 3) netIQ LDAP – Users authentication 4) DNS – Hostname resolution 5) EIP – Is the master service bus already implemented in the central IT
environment 6) Regional EIP Instances – IT/OT ESB instances deployed at the various sites
6.5.2 Integration Services Directory The current 250 candidate interfaces of EIP will be migrated
6.5.3 Data Migration No data will be migrated from the any on the existing databases
Page 24 of 61
6.6 Technical Architecture
6.6.1 Basic Infrastructure In following with the Eskom Hardware standard for EIP the infrastructure that was chosen to host OFMW was the Oracle Data Appliance (ODA). Essentially this appliance is a scaled down version of the engineered system deployed in the central environment. It is a self-contained appliance that encompasses both the application and storage tier. The environments that are in the scope of this implementation are:
1) Production 2) QA 3) Dev 4) Swap out DR
• Production
The following deployment pattern will be applied for the production environment in the regions:
• Type A region
1) One (1) ODA in the regional IT network 2) One (1) ODA in the DMZ
• Type B region
1) One (1) ODA in site 1 IT network 2) One (1) ODA in site 2 IT network 3) Two (2) ODAs in site 2 DMZ – the second ODA is to accommodate for the
eventual OT fragmentation of this site
• Non-Production
The following deployment pattern will be applied for the non-prod environment in the regions:
1) One (1) ODA in141 to simulate the IT regional environment 2) One (1) ODA in EAL to simulate the OT regional environment
• Disaster Recovery
The DR strategy is to have standby ODAs that will be used to swap any regional units that experiences a DR scenario. The rationale behind this is that currently the network work infrastructure between the sites and the central DR environment was deemed not sufficient to implement a centralised DR solution.
Page 25 of 61
6.6.1.1 Hardware Virtualisation on EIP
• ODA Virtualization
Multiple layers will be deployed while provisioning Oracle Fusion Middleware on the Oracle engineered systems. From a deployment architecture perspective, Oracle Fusion Middleware will be deployed on the top layer (Virtual machine or vServer) of the hardware virtualization layer. The Figure 12 : EIP Hardware Virtualization Structure details the relationship between the different layers on the ODA machines
Figure 12 : EIP Hardware Virtualization Structure
Each ODA comprises multiple compute nodes, similar to blade servers, each with CPU’s, Memory and Network interfaces. OVM will be deployed on each of the compute nodes to act as the hypervisor layer where the vServers or Virtual Machines, running Oracle Enterprise Linux or OEL, will be hosted on. Hardware is the only limiting factor when determining how many vServers can be deployed, with the resources assigned to all the As previously mentioned both the application and database tier will be deployed on a single ODA instance per site. This is achieved by dividing the ODA platform into multiple virtualization domains (this is not the same as Weblogic domains) as shown in the Figure 13. Each of the 2 nodes consists of the following:
• DOM0 (Domain 0) • ODA-BASE (Domain 1) • Multiple DOMU’s (Domain U or User Domain)
DOM0 is the management domain or also known as the host domain. DOM0 is started by the hypervisor on boot. This is a privileged domain that starts and manages all other domains. DOM0 only has access to the local storage (350 GB free space) on each node.
Page 26 of 61
The ODA_BASE is a privileged virtual machine domain, specifically for hosting the Oracle Grid Infrastructure and Oracle Databases. The ODA_BASE domain contains Oracle Appliance Manager, which includes commands and utilities to manage virtual machines, virtual disk, and templates. The ODA_BASE domain is a hardware virtualized, with par virtualized drivers (PVHVM) domain running an unmodified Unbreakable Enterprise Kernel (UEK). DOMU is for Virtual machines which are provisioned to host non-database workloads such as applications and middleware.
Figure 13: ODA Virtualization
6.6.1.2 EIP Central Hardware Specifications Table 7: ODA Hardware Specification and Utilization
Environment Production CPUs 64 cores, 89% used
Storage 64 TB (double-mirrored), 10% used MemoryOFM 208 GB, 41% used
Environment Disaster Recovery
CPUs 64 cores, 89% used Storage 64 TB (double-mirrored), 10% used
MemoryOFM 208 GB, 41% used
Environment Non-Production CPUs 48 cores, 67% used
Storage 64 TB (double-mirrored), 10% used MemoryOFM 224 GB, 44% used
Page 27 of 61
6.6.1.3 Physical Deployment Architecture
6.6.1.3.1 Production The following section will detail the Oracle Fusion Middleware deployment architecture for Production. From a domain architecture perspective, the same Oracle Fusion Middleware deployment template will be applied to both the IT and DMZ environment.The
ODA
ComputeNode ComputeNode
DOM1 VM
VM VM
VM
VM
VM VM
DOM1VM
VM VM
VM
VM
VM VM
OracleTrafficDirector
OracleServiceBus
OracleSOASuite
OracleSOASuite
JMS
DOM0 DOM0DatabaseRAC
Figure 14: High-Level Production Middle Tier Domain Deployment illustrates how the domains are configured on the production regional ODAs
Page 28 of 61
ODA
ComputeNode ComputeNode
DOM1 VM
VM VM
VM
VM
VM VM
DOM1VM
VM VM
VM
VM
VM VM
OracleTrafficDirector
OracleServiceBus
OracleSOASuite
OracleSOASuite
JMS
DOM0 DOM0DatabaseRAC
Figure 14: High-Level Production Middle Tier Domain Deployment
The diagram below depicts the database architecture deployed for the Database 12c Production environment. The ODA in each region on the OT and IT network is 2 Nodes RAC.
Page 29 of 61
Figure 15: EIP Data Tier Domain Deployment
From a deployment perspective, three different mount points will be created;
• /u01 – To host all installation binaries as well as the domain and admin server configuration files. This mount point will be shared across all the vServers for a specific domain.
• /u02 – This will host all the managed server structures. This is separated from the domain structure in /u01 to facilitate library isolation and to create a split between admin and managed servers.
• /u03 – Dedicated to logging. This enables a central location for all managed server log files on a vServer and reduces the operational overhead of having to navigate to each managed server to view log files.
The following table displays a sample file system structure for a SOA domain. Table 8: Middle tier mount points Mount Point Sample Path Comments /u01 /u01/oracle/products/….. Binary installation path /u01 /u01/oracle/config/domains/prdsoad100/… Domain and admin
server location /u02 /u02/oracle/config/domains/prdsoad100/… Domain location for
managed servers /u03 /u03/oracle/logs Central logging location
Page 30 of 61
for all managed servers
• OTD Deployment
ODA
ComputeNode1 ComputeNode2
vServer vServer
OTD
OTD1
/u02 /u03
OTD2
/u02 /u03
/u01
AdminNode AdminNode
Figure 16 : OTD Production Deployment
Table 9 : OTD Specifications for managed servers and clusters Environment vServer #CPU’s Memory u01SharedStorage u02 u03Production <reg>odavm<num> 4 16GB
250GB50GB 100GB
<reg>odavm<num> 4 16GB 50GB 100GB
• OSB Deployment
Page 31 of 61
ODA
ComputeNode1 ComputeNode2
vServer vServer vServer vServer
OSBDomain
AdminServer(7001)
WLS_OSB1
WLS_WSM1
/u02 /u03
AdminServer(7001)
WLS_OSB2
WLS_WSM2
/u02 /u03
/u01
AdminServer(7001)
WLS_OSB3
WLS_WSM3
/u02 /u03
AdminServer(7001)
WLS_OSB4
WLS_WSM4
/u02 /u03
Figure 17: OSB Production Deployment Table 10: OSB Specifications for managed servers and clusters Environment Domain Cluster Managed
ServersMemory
Production prdosbd100 prdosbcl100 prdosbms101 4GBprdosbms102 4GBprdosbms103 4GBprdosbms104 4GB
prdwsmcl100 prdwsmms101 4GBprdwsmms102 4GBprdwsmms103 4GBprdwsmms104 4GB
Environment vServer #CPU’s Memory u01SharedStorage u02 u03Production <reg>odavm<num> 4 12GB
50GB50GB 250GB
<reg>odavm<num> 4 12GB 50GB 250GB<reg>odavm<num> 4 12GB 50GB 250GB<reg>odavm<num> 4 12GB 50GB 250GB
Page 32 of 61
• SOA Suite Deployment
ODA
ComputeNode1 ComputeNode2
vServervServer
SOADomain1
AdminServer(7001)
WLS_SOA1
WLS_WSM1
/u02 /u03
AdminServer(7001)
WLS_SOA2
WLS_WSM2
/u02 /u03
WLS_EE1 WLS_EE2
/u01
vServervServer
SOADomain2
AdminServer(7001)
WLS_SOA1
WLS_WSM1
/u02 /u03
AdminServer(7001)
WLS_SOA2
WLS_WSM2
/u02 /u03
WLS_EE1 WLS_EE2
/u01
WLS_ESS1
WLS_MFT1
WLS_ESS2
WLS_MFT2
Figure 18: SOA Suite Production Deployment
Table 11 :SOA Suite Specifications for managed servers and clusters Environment Domain Cluster
NameManagedServers
Memory
Production prdsoad100 prdsoacl100 prdsoams101 4GBprdsoams102 4GB
prdwsmcl100 prdwsmms101 4GBprdwsmms102 4GB
prdeecl100 prdeems101 4GBprdeems102 4GB
prdsoad200 prdsoacl200 prdsoams201 4GBprdsoams202 4GB
prdwsmcl200 prdwsmms201 4GB
Page 33 of 61
prdwsmms202 4GBprdeecl200 prdeems201 4GB
prdeems202 4GB Table 12: Coherence Specifications for managed servers and clusters Environment Domain ClusterName Clustering
ModeTransport
prdsoad100 prdcoherencecl200 Unicast UDP
prdsoad200 prdcoherencecl200 Unicast UDP
Environment vServer #CPU’s Memory u01SharedStorage u02 u03 <reg>odavm<num> 6 16GB 50GB
50GB 250GB
<reg>odavm<num> 6 16GB 50GB 250GB<reg>odavm<num> 6 16GB 50GB
50GB 250GB
<reg>odavm<num> 6 16GB 50GB 250GB
• JMS deployment
ComputeNode1 ComputeNode2
vServer vServer
AdminServer(7001)
WLS_JMS1
/u02 /u03
AdminServer(7001)
WLS_JMS2
/u02 /u03
ODA
ComputeNode1 ComputeNode2
vServer vServer vServer vServer
JMSDomain
AdminServer(7001)
WLS_JMS1
/u02 /u03
AdminServer(7001)
WLS_JMS2
/u02 /u03
/u01
AdminServer(7001)
WLS_JMS3
/u02 /u03
AdminServer(7001)
WLS_JMS4
/u02 /u03
Figure 19: JMS Production Deployment
Page 34 of 61
Table 13: JMS Specifications for managed servers and clusters Environment Domain Cluster
NameManagedServers
Memory
Production prdjmsd100 prdjmscl100 prdjmsms101 4GBprdjmsms102 4GB
prdjmscl200 prdjmsms103 4GBprdjmsms104 4GB
Environment vServer #CPU’s Memory u01SharedStorage u02 u03Production <reg>odavm<num> 2 8GB
50GB
50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB
6.6.1.3.2 Non-Production The following section will detail the Oracle Fusion Middleware deployment architecture for non-production environments, where an ODA will be deployed for each of the IT & DMZ environments respectively. From a domain architecture perspective, Development and QA will be clustered and identical from a deployment perspective, with only the virtual machine resources differing for each of the environments as shown in Figure 20 .
ODA
ComputeNode ComputeNode
ODA
ComputeNode ComputeNode
DEV
QA
DEV
QA
Figure 20 :High-Level Non Production Middle Tier Domain Deployment
Page 35 of 61
Figure 21 : Non-Production Database Architecture In Figure 21 : Non-Production Database Architecture the clients will connect to the databases using services, the services will be running on the Node with the green color and it will be standby on the node with the orange color; This approach to maintain high availability and load balancing among all the DB nodes in the Oracle RAC. In Non-Production Exadata we have 3 environments Preprod, Development and QA We will have a separate database for each environment for the database PIEP and PINT and one database for SOA, OSB, JMS for each environment Database PRESOA for SOA, OSB and JMS for preprod environment Database DEVSOA for SOA, OSB and JMS for development environment Database QASOA for SOA, OSB and JMS for QA environment The following sections detail the Non-Production physical deployment and specifications of the domains covered in Regional Middleware Tier Domains and Clustering Structure
• OTD Deployment
Page 36 of 61
ODA
ComputeNode1 ComputeNode2
vServer vServer
OTD
OTD1
/u02 /u03
OTD2
/u02 /u03
/u01
Figure 22 :Oracle Traffic Director – Non-Production
An instance of Oracle Traffic Director will be deployed to handle internal communication and inbound load balancing requirements. The instance will be clustered across two compute nodes for HA purposes in the Development and QA environments. The following table details the resource allocation.
Table 14 : OTD Non-prod Environment Specifications Environment vServer #CPU’s Memory u01SharedStorage u02 u03QA <reg>odavm<num> 2 8GB
250GB50GB 100GB
<reg>odavm<num> 2 8GB 50GB 100GBDevelopment <reg>odavm<num> 2 8GB
250GB50GB 100GB
<reg>odavm<num> 2 8GB 50GB 100GB
• OSB Deployment
Page 37 of 61
ODA
ComputeNode1 ComputeNode2
vServer vServer
OSBDomain
AdminServer(7001)
WLS_OSB1
WLS_WSM1
/u02 /u03
AdminServer(7001)
WLS_OSB2
WLS_WSM2
/u02 /u03
/u01
Figure 23: Oracle Service Bus – Non-Production
Table 15 : OSB Managed Servers and Clusters Environment Domain Cluster Managed
ServersMemory
QA qaosbd100 qaosbcl100 qaosbms101 4GBqaosbms102 4GB
qawsmcl100 qawsmms101 4GBqawsmms102 4GB
Development devosbd100 devosbcl100 devosbms101 4GBdevosbms102 4GB
devwsmcl100 devwsmms101 4GBdevwsmms102 4GB
Environment vServer #CPU’s Memory u01SharedStorage u02 u03
Page 38 of 61
QA <reg>odavm<num> 2 8GB 50GB
50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB
Development <reg>odavm<num> 2 8GB 50GB 50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB
• SOA Suite Deployment
ODA
ComputeNode1 ComputeNode2
vServervServer
SOADomain1
AdminServer(7001)
WLS_SOA1
WLS_WSM1
/u02 /u03
AdminServer(7001)
WLS_SOA2
WLS_WSM2
/u02 /u03
WLS_EE1 WLS_EE2
/u01
vServervServer
SOADomain2
AdminServer(7001)
WLS_SOA1
WLS_WSM1
/u02 /u03
AdminServer(7001)
WLS_SOA2
WLS_WSM2
/u02 /u03
WLS_EE1 WLS_EE2
/u01
WLS_ESS1
WLS_MFT1
WLS_ESS2
WLS_MFT2
Figure 24 : SOA Suite Deployment
Table 16: SOA Suite Managed Servers and Clusters Specification
Page 39 of 61
Environment Domain ClusterName
ManagedServers
Memory
QA qasoad100 qasoacl100 qasoams101 4GBqasoams102 4GB
qawsmcl100 qawsmms101 4GBqawsmms102 4GB
qaeecl100 qaeems101 4GBqaeems102 4GB
qasoad200 qasoacl200 qasoams201 4GBqasoams202 4GB
qawsmcl200 qawsmms201 4GBqawsmms202 4GB
qaeecl200 qaeems201 4GBqaeems202 4GB
Development devsoad100 devsoacl100 devsoams101 4GBdevsoams102 4GB
devwsmcl100 devwsmms101 4GBdevwsmms102 4GB
deveecl100 deveems101 4GBdeveems102 4GB
devsoad200 devsoacl200 devsoams201 4GBdevsoams202 4GB
devwsmcl200 devwsmms201 4GBdevwsmms202 4GB
deveecl200 deveems201 4GBdeveems202 4GB
Environment Domain ClusterName Members Clustering
ModeTransport
QA qasoad100 qacoherencecl200 qasoacl201qawsmcl201qaeecl201
Unicast UDP
qasoad200 qacoherencecl200 qasoacl201qawsmcl201qaeecl201
Unicast UDP
Development devsoad100 devcoherencecl200 devsoacl201devwsmcl201deveecl201
Unicast UDP
devsoad200 devcoherencecl200 devsoacl201devwsmcl201deveecl201
Unicast UDP
Table 17 :Coherence Managed servers and Cluster Specification
Page 40 of 61
Environment Domain ClusterName Members ClusteringMode
Transport
QA qasoad100 qacoherencecl200 qasoacl201qawsmcl201qaeecl201
Unicast UDP
qasoad200 qacoherencecl200 qasoacl201qawsmcl201qaeecl201
Unicast UDP
Development devsoad100 devcoherencecl200 devsoacl201devwsmcl201deveecl201
Unicast UDP
devsoad200 devcoherencecl200 devsoacl201devwsmcl201deveecl201
Unicast UDP
Environment vServer #CPU’s Memory u01SharedStorage u02 u03QA <reg>odavm<num> 4 16GB 50GB
50GB 250GB
<reg>odavm<num> 4 16GB 50GB 250GB<reg>odavm<num> 4 16GB 50GB
50GB 250GB
<reg>odavm<num> 4 16GB 50GB 250GBDevelopment <reg>odavm<num> 2 16GB 50GB
50GB 250GB
<reg>odavm<num> 2 16GB 50GB 250GB<reg>odavm<num> 2 16GB 50GB
50GB 250GB
<reg>odavm<num> 2 16GB 50GB 250GB
• JMS Domain
Page 41 of 61
ComputeNode1
vServer
AdminServer(7001)
WLS_JMS1
/u02 /u03
ODA
ComputeNode1 ComputeNode2
vServer vServer
JMSDomain
AdminServer(7001)
WLS_JMS1
/u02 /u03
AdminServer(7001)
WLS_JMS2
/u02 /u03
/u01
Figure 25: JMS Domain
Table 18: JMS managed servers and clusters specification Environment Domain Cluster
NameManagedServers
Memory
QA qajmsd100 qajmscl100 qajmsms101 4GBqajmscl200 qajmsms102 4GB
Development devjmsd100 devjmscl100 devjmsms101 4GBdevjmscl200 devjmsms102 4GBdevjmscl200
Environment Domain Cluster
NameManagedServers
Memory
QA qajmsd100 qajmscl100 qajmsms101 4GBqajmscl200 qajmsms102 4GB
Page 42 of 61
Development devjmsd100 devjmscl100 devjmsms101 4GBdevjmscl200 devjmsms102 4GBdevjmscl200
6.7 Network Architecture This section provides an overview of the:
1) Firewall restrictions between the OT, DMZ and IT networks
2) Identifies Application network requirements
3) IP Addresses
4) Firewall Access
6.7.1 IT & DMZ Restrictions The diagram below depicts the firewall design which is the pattern for all regional DMS sites:
• ThereareanumberofVirtualLocalAreaNetworks(VLANs):o ESKOMBUSLAN(EskomIT/CorporateNetwork)o DMZ_MGMT_LANVLAN40(DMZManagementNetwork)o DMZ_SERVICESVLANXX(DMZNetwork)o DMZLAN(DMZNetwork)
• Thesenetworksareseparated/isolatedbyafirewall(CHECK_PRI)
Figure 26: Regional Network/Firewall Design (pattern for all DMS sites)
To access a system/platform/service in another network, a rule must be added on the firewall. This rule must clearly indicate the address of the source and target system as well as the specific ports and network protocol which will be used
Page 43 of 61
As noted above, a core Security requirement is that a physical server cannot be connected to both IT and DMZ networks. For this reason, there will be two Oracle Database Appliances (ODAs) deployed to a region – one allocated to the Corporate/IT network and one which will reside in the DMZ network.
Page 44 of 61
6.7.2 ODA Placement
IT Network Figure 27 below depicts the network connectivity for the Oracle Database Appliance deployed to the regional IT/Corporate network. Both the management interfaces
Figure 27: ODA Placement in IT Network
DMZ Network
Figure 28 below depicts the network connectivity for the Oracle Database Appliance deployed to the regional DMZ network. The dedicated management network ports on the ODA will be connected to the management network to allow remote administration and management of the machine.
Figure 28: ODA Placement in DMZ Network
Page 45 of 61
Type A Connectivity The diagram below illustrates the network connectivity between the Central and Regional Integration Platforms for a “Type A” site. As the reader will note:
• Connectivity between central and the regional site is achieved via the Eskom Wide Area Network (WAN)
• Connectivity between IT, DMZ and OT platforms is routed via a firewall (please refer to Section 6.7.1)
Figure 29: Central - Regional Network Connectivity (Type A)
Type B Connectivity The diagram below illustrates the network connectivity between the Central and Regional Integration Platforms for a “Type A” site. As the reader will note:
• Connectivity between central and the regional sites is achieved via the Eskom Wide Area Network (WAN)
• Connectivity between regional IT and regional OT sites is also achieved via the Eskom WAN
• Connectivity between IT, DMZ and OT platforms is routed via a firewall (please refer to Section 6.7.1)
Figure 30: Central - Regional Network Connectivity (Type B)
Page 46 of 61
6.7.3 Regional Network Capability The table below provides a summary of the regional IT and DMZ/OT network capability:
IT DMZ/OT
Fiber Copper Fiber Copper Site 1Gb 10Gb 1Gb 10Gb Simmerpan 8x 10Gb LC available Yes None No UTP in
place None
Bellvile 10g available Yes None No UTP in place
None
Bloemfontein 10g avaialble Yes None No UTP in place
None
Mkondeni 8x 10Gb LC available Yes None No UTP in place
None
Polokwane To be confirmed Yes None No UTP in place
None
Witbank 8x 10Gb LC available Yes None No UTP in place
None
East London 8x 10Gb LC available Yes None No UTP in place
None
EAL No 10 Gig available only 1 gig utp/fiber
Yes None No UTP in place
None
Duvha TBC TBC TBC TBC TBC TBC 141 TBC TBC TBC TBC TBC TBC
6.7.4 Load Balancing OTD acts as the load balancer between client systems and application instances on the ODA. Data traffic between application instances will be performed on the internal private application network only. The backend network will be used for administrative purposes (OS configurations, management of applications etc.) Advantages of this design:
• Inter application traffic will stay within the ODA system directly on the Infiniband network. This traffic will benefit from lower latencies and higher throughput.
• Reduced complexity thru reduced number of network interfaces in application VMs.
• Security • Segregation of network traffic
6.7.4.1 OTD Oracle Traffic Director will be used to load-balance all HTTP or HTTPS traffic internally between all the fusion middleware components and domains as well as to front incoming traffic. One OTD instance, configured for high availability with failover between instance
Page 47 of 61
nodes, will be deployed to handle inbound communication and to perform SSL offloading. It will also be responsible for internal communication between the various domains and components as shown in Figure 31: OTD Deployment.
From a security architecture perspective, OTD will only expose routing to OSB on the external network. All other routing will be done by binding to internal InfiniBand network addresses.
ODA
ExternalNetwork InternalNetwork
OSBSOA JMS
OTD
ExternalClient
Figure 31: OTD Deployment
Load Balancer Configurations
Load balancer Target Host Protocol External Application OTD HTTPS OTD OSB HTTP OTD SOA HTTP OTD SOA HTTP OTD EE HTTP OTD EE HTTP
As described in the previous section, OTD is also capable of load balancing other protocols. These can be configured as and when the need arises. Virtual IP Configuration As part of the network architecture we deployed Virtual IP’s to enable Admin server failover for all domains. Weblogic managed servers bind and connect to the Admin server on a pre-configured address, so in order for failover to work, the Admin address cannot change when moving between different hosts. This is achieved by binding the host that is hosting the Admin server to the Virtual IP assigned to the Admin server. External Virtual IP Environment Domain Servers Production PRDOSBD100 Admin PRDSOAD100 Admin PRDSOAD200 Admin PRDJMSD100 Admin QA QAOSBD100 Admin
Page 48 of 61
QASOAD100 Admin QASOAD200 Admin QAJMSD100 Admin Development DEVOSBD100 Admin DEVSOAD100 Admin DEVSOAD200 Admin DEVJMSD100 Admin
6.7.5 Firewall Access OT à DMZ
Source Destination Port Business Reason (per port) ABB STCMS server on
DMZ ODA 18017 Write messages to STCMS queue. As noted
above, the STCMS is a legacy SeeBeyond JMS which must be maintained to receive messages form systems/platforms in the OT network.
FMS STCMS server on DMZ ODA
18017 Write messages to STCMS queue. As noted above, the STCMS is a legacy SeeBeyond JMS which must be maintained to receive messages form systems/platforms in the OT network.
DMZ à OT Source Port Business Reason (per port)
All virtual machines on the DMZ ODA
123 Time synchronization with the network time server
Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA
1521 Database access to SCADA/Engineering database
Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA
1521 Database access to Universal Data Warehouse database
DMZ Outbound Source Business Reason (per port)
All virtual machines on the DMZ ODA
Communication to the NetIQ IAM LDAP services to manage and authenticate user accounts for the OS and Weblogic
All virtual machines on the DMZ ODA
SSH, SFTP access to the jump server
Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA
SSH, SFTP access to the IT ODAs from the Oracle Fusion Middleware Environment of the DMZ ODA.
All virtual machines on the DMZ ODA
Allow email access
All virtual machines on the DMZ ODA
Send SNMP alerts to Eskom SNMP server – for notifications regarding component outage.
All virtual machines on the DMZ ODA
Send syslog events to central logging server
All OFM Administration servers
Authentication of Oracle Fusion Middleware service and named users
Page 49 of 61
on the DMZ ODA against netIQ IAM. Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA
Access to web services deployed in IT domain.
Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA
Access to JMS resources in IT domain.
Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA
Access to databases in IT domain.
All virtual machines on DMZ ODA
Allows the OEM agents deployed to send metrics to central OEM server.
Simmerpan Source Business Reason (per port)
All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
Bellville Source Business Reason (per port)
All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
Bloemfontein Source Business Reason (per port)
All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
Mkondeni Source Business Reason (per port)
All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
Polokwane Source Business Reason (per port)
All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
Witbank Source Business Reason (per port)
All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
Page 50 of 61
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
East London Source Business Reason (per port)
All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
Page 51 of 61
Eskom Academy of Learning (Non-Production)
Source Business Reason (per port) All virtual machines on the DMZ ODA
Send logs to Splunk regional collector
All virtual machines on the DMZ ODA
Netbackup/Infrastructure requirements
DMZ Inbound
Source Business Reason (per port) Communication to the NetIQ IAM LDAP
services to manage and authenticate against user accounts for the OS and Weblogic
SSH, SFTP access to the DMZ ODAs from the Jump Server.
Oracle Fusion Middleware servers (OSB, SOA, WL) on IT ODA
SSH, SFTP access to the DMZ ODAs from the Oracle Fusion Middleware Environment of the IT ODA.
Oracle Fusion Middleware servers (OSB, SOA, WL) on IT ODA
Access to web services deployed in DMZ domain.
Oracle Fusion Middleware servers (OSB, SOA, WL) on IT ODA
Access to JMS resources in DMZ domain.
Oracle Fusion Middleware servers (OSB, SOA, WL) on IT ODA
Access to databases in DMZ domain.
All virtual machines on DMZ ODA
Allows the OEM agents deployed to send metrics to central OEM server.
Eskom Management Network
Oracle Integrated Lights Out Management: CD, Mouse & Keyboard, Diskette, Encryption, Authentication, Servicetag Daemon, Video, Serial
Jump Server Access to e-manger in the DMZ to be able to create queue
9 SCADA application server zone1 Eccvssaz0203 on Corporate LAN accessing STCMS zone in DMZ
SCADA application server zone1 Eccvssaz0204 on Corporate LAN accessing STCMS zone in DMZ
SCADA application server zone2 on Corporate LAN pushing messages onto message broker on DMZ LAN
6.8 Security Architecture The following section will detail security elements around the deployment architecture. The following diagram depicts a high level security architecture and all the paths that will be secured.
Page 52 of 61
ODA
vServer
OTD
EnterpriseApplications
(1)
vServer vServer vServer
FMW FMW FMW
(2) (2)
(3)OperationalPerson
(3a)
(3b)
DB DB DB
InfinibandNetwork
(4)
(5)
LDAP
(3b)
EnterpriseApplications
(6)
NetIQ
Figure 32: Security Architecture
• (1) – When applications outside of the integration platform needs to communicate
with the integration layer, communication will happen via OTD and all communication will be secured using SSL. The application will also provide HTTP basic credentials that will be passed to the Oracle Fusion Middleware layer, where the WebLogic embedded LDAP store will be used to store these credentials.
• (2) – OTD will load balance both inbound and internal component traffic. The internal traffic will not be secured using SSL as it happens over a secured internal connection and is local to the Oracle Database Appliance. The unsecured load balancing rules will not be accessible from outside the ODA as the listening address will be bound to the same internal secured connection.
• (3a) – Operational resources need to login to the virtual machines via SSH. They will have to authenticate using username and password. Please refer to Section 0 User Management Strategy.
Page 53 of 61
• (3b) – When operational resources logs into the WebLogic console, they will provide their Active Directory credentials. WebLogic will be configured to authenticate against Active Directory for non-system accounts.
• (4) – Connections to the integration databases will happen via the internal and thus no additional security is required.
• (5) – Credentials to connect to the database will be stored within the Weblogic JDBC configuration files. All these credentials will be encrypted using a unique encryption key per domain.
• (6) – Outbound communication to backend systems, like Maximo, will be handled by OSB. If communication to the backend system needs to be secured, the certificates will be stored within the OSB Weblogic keystore. Because the OSB instances are listening for incoming communications on the local network, this approach does not pose a security thread as OSB will not be reachable from outside the ODA.
The following table can be used as a security matrix when trying to determine if security will be applied and how. Connection Method Secured Technologies Inbound Application connectivity
Y • OTD SSL Offloading • HTTP Basic Authentication • WebLogic embedded LDAP
SSH to vServers Y • Username & Password • Active Directory
WebLogic console login Y • Username & Password • WebLogic embedded LDAP • Active Directory
OTD Internal Routing Y • Infiniband Secured Network Database Connectivity Y • Encrypted Username &
Password • Infiniband Secured Network
OSB Outbound Connectivity
Y • SSL • Certificates imported to keystore
OTD
OTD guarantees high performance through SSL/TLS offloading, content caching and effective HTTP compression. Flexible configuration of routing and load control on back-end servers through a selection or combination of Request-based routing, Content-based routing, Request rate acceleration and Connection limiting. The request load and quality of service are also tunable through request rate limiting and quality of service tuning. OTD is also capable of TCP load balancing if required.
SSL As part of the network architecture, OTD will be deployed and utilized within the environment. Each of these components have the ability to do SSL offloading and onboarding depending on the requirements. From a security perspective all communication into the integration landscape needs to be secured using SSL. OTD will do SSL offloading and pass unencrypted communication to the OSB instances. All inter-
Page 54 of 61
component communication within the Oracle Database Appliance does not need to be secured as the internal network is considered to be secure and no packets can be sniffed from outside the ODA.
Keystores As part of the deployment process, custom keystores will be deployed to store server and client certificates. Weblogic will need a trust store if outbound communication to a backend system needs to be secured with SSL.
WebLogic Groups & Roles The following table details the default Oracle Fusion Middleware Users, Groups and Roles that defines how user access can be restricted on a WebLogic side. Please note that these are not application level groups and roles but rather specifies how individuals can access WebLogic from an operational perspective. The following table details the default groups. Group Membership Administrators By default, this group contains the user information entered
as part of the installation process (that is, the Configuration Wizard), and the system user if the WebLogic Server instance is running Compatibility security. Any user assigned to the Administrators group is granted the Admin security role by default.
Deployers By default, this group is empty. Any user assigned to the Deployers group is granted the Deployer security role by default.
Operators By default, this group is empty. Any user assigned to the Operators group is granted the Operator security role by default.
Monitors By default, this group is empty. Any user assigned to the Monitors group is granted the Monitor security role by default.
AppTesters By default, this group is empty. Any user assigned to the AppTesters group is granted the AppTester security role by default.
CrossDomainConnectors By default, this group is empty. Any user assigned to the CrossDomainConnectors group is granted the CrossDomainConnector security role by default.
AdminChannelUsers By default, this group is empty. Any user assigned to the AdminChannelUsers group is granted the AdminChannelUser security role by default.
OracleSystemGroup By default, this group contains the user OracleSystemUser and is granted the OracleSystemRole role by default.
Page 55 of 61
The following table details the global roles that WebLogic Server defines in the security realm that it installs. The table also summarizes the access that the default security policies grant to each role and indicates which groups are in each role by default. Global Role Default Policies Grant
Access To… Default Conditions Include This Group…
Admin • View the server configuration, including the encrypted value of some encrypted attributes.
• Modify the entire server configuration.
• Deploy Enterprise Applications and Web application, EJB, Java EE Connector, and Web Service modules.
• Start, resume, and stop servers.
Administrators
AdminChannelUser Access the administrative channel, AdminChannel
AdminChannelUsers, Administrators, Deployers, Operators, Monitors, and AppTesters
Deployer • View the server configuration, including some encrypted attributes related to deployment activities.
• Change startup and shutdown classes, Web applications, JDBC data pool connections, EJB, Java EE Connector, Web Service, and WebLogic Tuxedo Connector components. If applicable, edit deployment descriptors.
• Access deployment operations in the Java EE Deployment Implementation (JSR-88)
Deployers
Operator • View the server configuration, except for encrypted attributes.
Operators
Page 56 of 61
• Start, resume, and stop servers.
Monitor View the server configuration, except for encrypted attributes. This security role effectively provides read-only access to the WebLogic Server Administration Console, WLST, and MBean APIs.
Monitors
AppTester Access applications for testing purposes that are running in Administration mode.
AppTesters
CrossDomainConnector Make inter-domain calls from foreign domains.
CrossDomainConnectors
OracleSystemRole Assert identity on behalf of users whose WS-Security tokens have been authenticated. Note: This global role is provided for use by Oracle Web Services Manager.
OracleSystemGroup
Page 57 of 61
File System Security
The Oracles Enterprise deployment guides suggests to create 3 mount points. As a result the oracle user owns all three mount points. In order to manage this effectively without the need to use the oracle user the following groups need to be created and assigned to each mount point to be able to administrate the different mount points. A default user “oracle” belongs to default group “oinstall”. This user owns all installed technology and mount points /u01, /u02 and /u03. This user should not be used to login and SSH. This “oracle” user is a local user to each virtual machine. Note: A key requirement is the installation of the NetIQ fan-driver before any additional OS users are created. This will ensure that the UID and GID of users does not change. Groups Mount Comments Group 1 /u03 and /u02 Used to administrate the Managed Servers and logging. Group 2 /u03 Used to administrate logging. oinstall /u01,/u02 and /u03 Default created group. Must only be assigned to administrator with full
access.
Operating System Groups
The following groups are required for the Operating System: • root • DBA • logging • osadmin • oinstallstaff
This section considers security from the following perspectives:
• User Management Strategy • Application Access • Server Access • Firewall Access
Page 58 of 61
User Management Strategy
The table below provides a summary of the user management strategy: System Account Service Account User Account Description Built-In Account Software
Installation Service Access
Named User
Location Local Eskom IAM Eskom IAM Usage These are the
accounts which are provided “out-of-the-box” with the hardware/software elements.
These are the accounts which are used to install and configure the application software, for example: Oracle database, Oracle Fusion Middleware. These are also the accounts which are used for “programmatic” access, for example: web service or API to solutions deployed on the application stack.
These are operational, human users which access the infrastructure/application/solution for day-to-day Operation and Administration.
Sample of users to commission
root Primary OS account
oraagent OEM account
lukhans Shaun Lukhan
weblogic Primary OFM account
orarom Platinum services account
bhoolar Rakesh Bhoola
Implementation Requirements
None The Novell® Identity Manager Fan-Out Driver (detailed in Section 0) LDAP Authentication
The Novell® Identity Manager Fan-Out Driver (detailed in Section 0) LDAP Authentication
Page 59 of 61
NetIQ Identify & Access Manager
As NetIQ has been established as the Identity & Access Management (IAM) standard in the Eskom environment, this will be installed on all Virtual Machines immediately after OS installation and configuration. The Novell® Identity Manager 4.0.2 Fan-Out Driver is an identity provisioning solution, based on Novell eDirectory™, Identity Manager, and related technology. With Identity Manager, you can manage the full user life cycle, delivering first-day access to essential resources, providing single login, and modifying or revoking access rights. Identity Manager also provides self-service features that enable users to maintain their own passwords and profile information. By adding the Fan-Out Driver, you can use Identity Manager to fan out identity provisioning to hundreds of systems with minimal effort. You can centrally manage user accounts and have them automatically created, configured, maintained, and removed when appropriate. It uses extensible scripts to manage account rights, home directories, and other resources as well as the user definitions themselves. Meanwhile, system administrators for the individual platforms in your enterprise can retain control over their areas.
Installation Scope The NetIQ Fan-Out Driver can be installed on the ODA base domain and any VM. However, the agent cannot be installed on the DOM0 or DOMU domain. DOM0 is running Oracle VM Server for x86, which provides a minimal Oracle Linux with virtualization technology included.
Application Access The table below provides a summary of the anticipated OFM Application access requirements for the Regional EIP environment from IT to DMZ and vice versa: Source Target Requirement Security Mechanism IT DMZ Web Services HTTPS, Basic Auth,
SSL JMS T3 over IIP, Basic Auth JDBC Username/Password
SSH/SFTP SSL, Certs, Basic Auth DMZ IT Web Services HTTPS, Basic Auth,
SSL JMS T3 over IIP, Basic Auth JDBC Username/Password
SSH/SFTP SSL, Certs, Basic Auth OT DMZ STCMS Username/Password DMZ OT JDBC Username/Password
• Table 19: Application Access Summary
Page 60 of 61
7 High Availability From a physical resource mapping perspective, all domains will be deployed across two different compute nodes. This approach ensures that we achieve a highly-available architecture in conjunction with WebLogic clustering. If we lose one virtual machine or even a complete compute node, part of the cluster will continue running on the other compute node. The appliance is also designed with mission-critical requirements in mind, with hot-swappable and redundant components.
2 ODA X5-2 machines will be allocated to each region, deployed as follows:
Machine #1, Virtualized & Clustered Platform Optimized for Database and Applications – IT-ODA;
Machine #2, Virtualized & Clustered Platform Optimized for Database and Applications – DMZ-ODA;
The environment will be configured in a 2 node RAC (Real Application Cluster) for the databases to promote high availability in the regions.
The identified Disaster Recovery solution is the replacement of the physical infrastructure element that has failed. In the event of a whole server (ODA) failure, a replacement server (ODA) will be shipped to the affected site and will be configured and restored as necessary. In the event of a component failure, the failed component will be replaced. This strategy was selected to minimize infrastructure costs as well as due to the network infrastructure between regional sites and the Central platform.
The servers allocated for Disaster Recovery will be pre-built with the necessary software elements – such as virtualization, database, Operating System and Oracle Fusion Middleware Platform. Upon commissioning at the affected site, additional configuration can be applied to restore the platform. Database will then be restored from RMAN backups.
8 Disaster Recovery
There is 3 major failure points at each site : 1) Catastrophic failure on the regional OT system environment
2) Catastrophic failure regional ODA DMZ environment
3) Catastrophic failure regional IT ODA environment
Page 61 of 61
• The DR for the 3 major failure points at each site is :
1. Catastrophic failure on the regional OT system environment
• Do nothing – the OT does not currently have DR
2. Catastrophic failure regional ODA DMZ environment
• Assuming the DMZ network is still up
• Swap out the DMZ ODA with a templatzied ODA
3. Catastrophic failure regional IT ODA environment
• Assuming the regional IT network is still up
• Re-route the traffic to a racked and stacked ODA connected to the IT 141