WebSphere Extreme Scale (WXS Components) – Operational readiness Considerations. By Nitin Gaur Introduction: IBM WebSphere eXtreme Scale is specifically designed to help applications lower the load they place on back-end servers, increase availability and scalability, and provide the facilities for extreme transaction processing. A software layer between an application and its back-end resources, WebSphere eXtreme Scale accelerates applications by caching objects obtained from the back-end, providing a customizable, pluggable data fabric framework that allows applications to share data across a wide range of application scenarios. This paper attempts to serve as an operational readiness checklist and is designed for customers and IBM software services community, who are interested in implementing a highly available and scalable e-business infrastructure using the IBM WebSphere eXtreme Scale. Through WebSphere eXtreme Scale, customers can postpone or virtually eliminate costs associated with upgrading more expensive, heavily loaded back-end database and transactional systems, while meeting the high availability and scalability requirements for today’s environments. While not an exhaustive list, this paper includes primarily the infrastructure planning requirements of WXS environment. Understanding WXS Components: From a broad environment perspective WXS has Three management entities:

1. Grid Servers: These are JVM that will contain grid container. Much like a typical description of a container in a J2EE™ context, grid containers essentially provide the grid application services such as security, transaction support, JNDI lookup service, remote connectivity, and so forth. The grid containers house shard distribution and placement, and enable easy manageability of the grid infrastructure. Much like other containers (Web and EJB™ container, for example), a grid container can also take advantage of the configuration service provided by the WebSphere Application Server infrastructure in a managed environment.

2. Grid Clients: Clients connect to a grid and are attached to the whole grid. Clients need

to examine the key of application data to determine to which partition to route the request. Any entity that is attached to the grid with any kind of request becomes a client. The client directly accesses the partition that holds the data. The location information regarding the partition is provided by the catalog servers.

3. Catalog Servers: The catalog server is the engine that drives the grid operations. The

catalog server maintains the healthy operation of grid servers and containers. The catalog server becomes the central nervous system of the grid operation by providing the following essential operation services:

a. Location service to all the clients b. Health management of the grid itself c. Shard distribution and re-distribution d. Policy and rule enforcement e. High availability and group service

The client will begin its access to the grid by obtaining a routing table from the

catalog servers, which enables the client to locate the primary partition shard and object. In the event of a JVM failure, or re-distribution of partitions due to a change in grid membership, the client is provided with an up-to-date routing table by one of the grid servers. This mechanism is maintained by an epoch time and version maintained with the routing table held by the client.

Understanding the WXS component relationship:

The above Data Model describes the relationship between the components in an eXtreme Scale environment. 1. The JVM can be either an application server or a stand-alone JVM and can host many grid containers. A JVM contains a runtime and a number of containers, usually one. 2. A grid container that resides in a JVM can host many ObjectGrid instances, or many ObjectGrid instances spread across many grid containers. 3. A MapSet is a collection of maps that are typically used together. Many MapSets can exist in one ObjectGrid instance. 4. An ObjectGrid consists of a number of partitions. Each partition has a primary shard and N replica shards. 5. One partition can host many BackingMaps. 6. An ObjectGrid consists of a number of maps, called a mapSet. A mapSet is partitioned using a key. Each map in the mapSet is defined by a BackingMap. 7. An ObjectGrid can host a set of grid containers. A grid container can only host shards from one ObjectGrid. This means that multiple ObjectGrids can be started on a single JVM. Each grid started has its own container within the JVM. Those grid containers will host shards from the grid as determined by the catalog server.

WXS Infrastructure Planning Considerations:

1. Grid Server Hosting. a. Grid Server on WebSphere managed node: WebSphere eXtreme Scale fits

perfectly into an environment managed by WebSphere Application Server Network Deployment, allowing all the services, such as transaction management, security and monitoring, to be used by grid components. The decision to host Grid Servers in a managed environment, while providing features like ease of use and administration, may have licensing cost considerations.

b. Grid Server on standalone Supported JVM: WebSphere eXtreme Scale has

several flexible deployment options. One of these deployment options is installing WebSphere eXtreme Scale into a stand-alone Java2 Standard Edition environment (J2SE). The stand-alone deployment option differs from the WebSphere Network Deployment-managed option in that it lacks Java Enterprise Edition support and does not benefit from the management and monitoring environment that WebSphere Network Deployment provides. It does, however, provide a simple footprint and runtime for hosting grid containers and catalog servers. While it may be economical to

2. Catalog servers hosting

a. Catalog Server on WebSphere managed node: As mentioned before, The catalog server is the engine that drives the grid operations. Hosting Catalog servers on WebSphere managed node, presents similar benefits, to that of hosting grid servers on WebSphere managed node.

b. Catalog Server on standalone Supported JVM : In some cases, it may be

imperative to host catalog servers on standalone JVMs. An example may include when using WXS enabled grid to cache HTTP session, across geographically disperse data centers. In this specific scenario, while the grid servers may be hosted in WebSphere managed application servers, and the cache to be replicated across data centers, the catalog servers, due to HA managers services and quorum requirements, would be required to be hosted in stand alone JVMs. This will enable a core group of standalone catalog servers.

c. Multi data Center Catalog Server support : Multi data center would imply a

WXS zoned configuration. In a Zoned configuration, usually one catalog server per zone ( or data center) is a good start.

3. Co-locating Grid client and server (Application and Grid container in same JVM): a. This is when the ObjectGrid servers are collocated in the same processes where

the servlets run. The WXS session manager can communicate directly to the local ObjectGrid instance, since it is collocated with in the same server process.

b. When WebSphere Application Server as runtime and grid container, simply place the supplied objectGrid.xml and objectGridDeployment.xml files into the META-INF directories of your WAR files. ObjectGrid will automatically detect the presence of these files when the application starts and will then automatically launch the ObjectGrid containers in the same process as the session manager. You can modify the objectGridDeployment.xml depending on which type of replication (synchronous or asynchronous) and how many replicas you want

configured. This step assumes that your have augmented the application server profile/instance with WXS.

c. When the grid container is collocated with the enterprise application in the same

application server, the grid container becomes more vulnerable. Issues in the application (for example, memory leaks, deadlock situations) can bring the grid container down.

4. Isolating the application from the grid (i.e. application and grid in separate JVMs):

a. Unlike the topology discussed above the server process hosting the application is different from the server process hosting the JVMs. This is beneficial for the application with either voluminous HTTP session traffic or relatively larger http session objects. In this case the WXS session manager, which resides on the application server process, communicates to a remote ObjectGrid server processes. In effect the WXS session manager becomes the client, to the Object grid servers, which are the hosts to grid containers.

b. To use a remote, ObjectGrid, the WXS session manager must be configured to communicate with the catalog servers and grid servers. The session manager will then use an ObjectGrid client connection to communicate to the catalog server and the ObjectGrid servers, wherever they reside. If the ObjectGrid servers are to be started in independent, standalone J2SE processes, then it is advisable to launch the ObjectGrid containers using the objectGridStandAlone.xml and objectGridDeploymentStandAlone.xml files supplied in the session manager samples directory. This differentiation is to indicate that the HTTP session store is standalone and isolated. The core functionality of the file remains, i.e. like objectGrid.xml, objectGridStandalone.xml defines the grid for sessions, and objectGridDeploymentStandalone.xml defines the mechanics of grid deployment.

5. Grid server sizing

a. Calculate the number of JVMs required from the partitioning configuration. b. When deciding on the number of JVMs required for a grid, a good rule of thumb

is to start with the number of partitions needed multiplied by the number of replica shards per partition. For example, if you have two partitions and each partition has one primary and one replica shard, then you would need at least four JVMs as a starting point for a highly available grid.

c. It is important to plan the number of partitions you will need for both current capacity and future growth. Once you have deployed your applications you cannot simply add partitions or scale using more JVMs. A good guideline to follow is that at the end of your planning horizon, you should end up with 10 shards per partition.

6. Catalog server sizing

a. Always more than one for high availability b. Recommended prime number greater than 2. This is due to concept of a catalog

service quorum. A catalog service quorum is the minimum number of active catalog server members required for the grid system to operate correctly (that is, to accept membership registrations and changes to membership to ensure proper routing and shard placement). This approach of ensuring the registration and

consistency of grid servers is achieved only when a quorum is established between catalog servers. Writes to the catalog service state are committed only when the majority of the catalog servers participate in the transaction.

c. The decision on the number of catalog servers depends on overall grid size, desired high availability, and zones configuration.

d. In a Zoned configuration, usually one catalog server per zone is a good start.

7. Secure Transport (SSL) a. Understanding which end points to secure : It is vital to understand different

transports that ObjectGrid uses in different instances. WXS uses the ORB for client-server, server-server, client-catalog, server-catalog communications. This is secure and was added support for SSL-Required in 6.1.0.5.

In addition to the ORB WXS the HAM/DCS/RMM stack for membership only by the containers. Further the intra-zone (data center) grid servers use group services (HA manager construct) for server state information. So, the communications on this protocol is kept local and only contains membership information, but this is not yet secured. This should not be alarming to enterprise, as there is no application data shared, but only server state data ( which included things like server, hostname and port information).

b. Supported Transport layer security: SSL Supported and SSL required are two

supported transport Type. SSL required, transportType support was added in 6.1.0.5. SSLv2, SSLv3, and TLSv1 are all considered SSL protocols.For Transport, One can choose to use Secure Sockets Layer (SSL), TCP/IP or both as the inbound transport that a server supports. If you specify TCP/IP, the server only supports TCP/IP and cannot accept SSL connections. If you specify SSL-supported, this server can support either TCP/IP or SSL connections. If you specify SSL-required, then any server communicating with this one must use SSL.

SSL Supported: If you select SSL-supported, then the server opens both a TCP/IP and an SSL listener port and most inbound requests are received using SSL SSL Required : If you select SSL-required, then the server opens an SSL listener port only and all inbound requests are received using SSL.

If you specify SSL-supported or SSL-required, decide which set of SSL configuration settings you want to use for the inbound configuration. This decision determines which key file and trust file are used for inbound connections to this server. In dynamic deployment topology, the ORB protocol is used for client/server communications. ORB transport security, Transport Layer Security (TLS) or Secure Sockets Layer (SSL) is supported. However, transportType property must be set to "SSL-Supported" in both the client and server security property files. If set to "SSL-Required", the container server will fail to initialize with an error message: "initial and forwarded IOR inaccessible". Please note that prior to 6.1.0.5 use SSL-Supported or can use SSL-Required if 6.1.0.5 is applied. Either case SSL is supported, although with a different type of support.

c. TCP and Firewall considerations : Please not that all the relevant ports would need to be considered while designing a WXS topology. To ensure communication flow between client-server, server-server, client-catalog, server-catalog is vital to a smooth grid function. These communication flow ensures grid data replication, client to grid access, registering membership changes and subsequent re-routing of client due to membership changes. The following ports would need to considered, in relation to TCP transport and enterprise firewall while planning a WXS topology:

i. Client port : port used for client catalog cluster communication ii. Peer port: port used for peer catalog cluster communication

iii. Listener port : Client’s need to know Catalog service JVMs listener end points.

iv. JMXServicePort – if using JMX ports.

Note: you can set the transportType property in the following client and server

security configuration files: • Client ( application - in Managed WAS env) : The transportType property in the

security.ogclient.props file. • Container or Grid server or cache server: The transportType property in the

containerServer.props file. • Catalog Server: The transportType property in the security.ogserver.props file or

containerServer.props file, depending on which file you use.

8. Testing the Infrastructure

Testing Methodology : One of the primary purpose of testing is to uncover software defects and failures and subsequently correct them. Any broad goals of testing include unit testing, integration testing, system and system integration testing. Each testing process has a specific goal and objectives. From extreme scale testing, it is vital to isolate the application test from the WXS ‘Grid’ tests. The implication of such an approach assumes that ONLY the WXS features are tested independently while the application itself has been tested for component features. The WXS ‘Grid’ tests would include testing the following:

1. Application accessing the Grid 2. WXS grid’s replication mechanism 3. WXS grid container failover and HA mechanism. 4. Testing of the WXS Grid infrastcruture, including the catalog servers and

shard placement. 5. Various use of configurable grid features such as Memory threshold, zone

replication, locking mechanism , batched update etc. Integration and system integration testing would however include tests cases that include verification of interaction between the integrated components, and system integration tests verifies the integrated system is well aligned with core objectives of system requirements. It is vital to understand that testing at each level to be completed, prior to proceeding to integrations or system level testing. This approach will ensure that each component or module is verified prior to their interaction in a system as a whole.

9. Monitoring: Monitoring while important is a complex discussion topic, and many decision points around monitoring focus on “what to monitor?” Conceivably one can monitor many aspects of an enterprise application infrastructure, i.e

a. Infrastructure monitoring – Include Monitoring of the overall infrastructure health, including the resource usage, JVM health and availability and so on.

b. Application Monitoring – Includes application performance, resource usage and bottlenecks.

WXS introduces another level of monitoring, which is completely optional, and that is the GRID monitoring.

c. Grid Monitoring – Includes monitoring around cache statistics, cache entries and cache policy information, grid replication and information on shard placements.

When WXS is hosted in WebSphere managed environments, the WebSphere

Application Server provides a sophisticated performance monitoring infrastructure (PMI). It can be used to gather performance statistics of all components inside an application server. WebSphere eXtreme Scale takes advantage of this infrastructure to collect performance statistics relevant to the grid. Metrics provided include the following:

d. Actual cache size e. Cache hit rate f. Transaction response time g. Loader update response time.

However, in a standalone environment, The ObjectGrid APIs and MBeans are designed to allow for easy integration with third-party monitoring applications. JConsole or MC4J are some examples of lightweight JMX consoles that can be used to quickly digest information about an ObjectGrid topology. Besides the JMX based monitoring, WXS, beginning with WXS 6.1.0.3 provides a sample application called ‘xsadmin’. This nifty application will provide with ‘point-in-time’ information on topology, shard placement and map sizes in the WXS enabled grid. These tools are only made available to facilitate WXS specific grid level monitoring and management. It is assumed that the infrastructure monitoring tools ( example IBM’s ITCAM and CA’s Wily Introscope) will still be employed to monitor the underlying application environment.

Acknowledgement Robert Ringo – WebSphere WW Technical Sales Douglas Berg – WebSphere Extreme Scale Development Joshua Dettinger - WebSphere Extreme Scale Development References: 1. RedBook - Users Guide to WebSphere Extreme Scale www.redbooks.ibm.com/redpieces/pdfs/sg247683.pdf 2. WebSphere Extreme Scale – Wiki – Users Guide. http://www.ibm.com/developerworks/wikis/display/objectgrid/Getting+started 3. Billy Newports Blog - http://www.devwebsphere.com/