NSS Workload Balancing Best Practices[1]

7/27/2019 NSS Workload Balancing Best Practices[1]

1/39

NS Solutions Corporation

How to maximize workload balancing

with Oracle Real Application Clusters


2/39

2

Copyright 2006 NS Solutions Corporation, All rights reserved.

Agenda Company Background

Load Balancing Challenges & Technologies

Test Environment

Test Cases & Results

Conclusion


3/39

3




Test Environment


Conclusion


4/39

4


Name : NS Solutions Corporation

Start of Company : April 2001

Business integration of ENICOM(founded in 1988) and the EI Division(organized in 1986) of Nippon Steel Corporation

Common Stock : 12.9 billion yen

Headquarters : 2-20-15, Shinkawa, Chuo-ku, Tokyo 104-8280, Tel:+81-3-5117-4111

Number of Employees : 2,060 (consolidated 4,032) as of Mar 31,2006

Net sales : 133 billion yen (consolidated 148 billion yen) FY20052005.4.12006.3.31

NS Solutions Group

Hokkaido NS Solutions NS Solutions Kansai

Tohoku NS Solutions NS Solutions Chubu

NS Solutions Tokyo NS Solutions Nishinihon

NS Solutions Oita

JV

NCI SYSTEMS INTEGRATION

NSSOL 51%/ITOCHU Gr. 49

Nittetsu Hitachi Systems Engineering

NSSOL 51%/Hitachi 49

Regional

Shareholders

China NS Solutions Software(Shanghai)

IT Service NSSLC Service

Corporate Outline


5/39

5


Industries (ERP, PLM, SCM etc.)Financial business (financial derivatives, risk management, businessadministration support, etc.)Society and public (advanced science and satellite, education,telecommunication, total document management, etc.)

Internet (large-scale portal site, contents delivery system, etc.)

Advanced IT infrastructure solutions for multi products

accommodating large-scale and mission-critical requirements

Operation service

excellent in reliability

Technological capability in verification, design and set-up of

software platforms and network platforms,

A neutral (vender independent) provider of solutions, offering integrated servicesthrough system life cycle supported by sophisticated technologies.

Systems Research & Development Center Corporate IT Consulting Center

Technology support with advanced expertise

Business solution focused on specific fields

NS Solutions provides 24365 services

using state-of-the-art technologies at

the data centers and the networkcenters deployed nationwide.

NS Solutions provides integratedoutsourcing services ranging fromapplications to IT platforms.

Consultation Design Development and delivery Operation andmaintenance

System life cycle

Business of NS Solutions


6/39

6


Business Background Provide IT infrastructure solutions for large-scale and

mission-critical systems

Research Center has hardware from multiple vendors andis unique.

NS Grid/Utility Computing Center (NSGUC)

RAC license sales : +1400 CPUTotal : +100 of RAC implementations soldOracle 10g +40 RAC implementations sold


7/39

7


NS Grid/Utility Computing Center (NSGUC) Establishment in December 2005, in cooperation with 14

companies

Purpose is to apply grid technology to Managing service

Utility data center

resource rent by time, on demand use (for event, for seasonalalternation)

Utility based ASP service

1 system by 1 customer

Grouping together customers who use same type of applicationand produce effective use of resources

Engineering service

constructing a value-added system for each customer

Server integration, Grid construction as LR(Legacy Re-engineering)

extension


8/39

8


Realizemaximizing and optimally utilizing

system resources

NS Grid/Utility Computing Centers Goal

Stable

performance

for each service

Stable

performancefor each service

Start from small

systems and

expand

Start from small

systems andexpand

Resource

management tohandle load peaks

Resource

management to

handle load peaks

Provide the answer to usability, stability

in actual case.

Provide the answer to usability, stabilityin actual case.

Oracle10g Provides necessaryfunctions


9/39

9


Service A Service C

Service B

Oracle10g RAC

ClientOracle10g AS Spare

HP ProLiant

Hitachi BladeSymphony

IBM BladeCenter

Oracle 10g Test Goal

OEM

Design and Construct large-

scale, multi-node system inDB layer, AS layer and

storage layer

Achieving system

infrastructure consolidationby service definition

In GRID environment.

Optimize resourceutilization by using load

balancing functions.

Achieving reduction of

administration costby using OEM

in GRID environment.

NAS

Blade Serversconsist of various platforms

NetApp

FAS980/FAS3050

Optimize resourceOptimize resource

utilization by using loadutilization by using load

balancing functions.balancing functions.

Total Nodes 108 nodesTotal CPU 232 CPUs


10/39

10




Load Balancing Advisory

Runtime Connection Load Balancing

Test Environment

Test Cases & Results Conclusion


11/39

11


Clients

Load Balancing Advisory (LBA)

LBA Advice:

OLTP Service

instance1 33%

instance2 33%

instance3 34%

LBA Advice:

OLTP Service

instance1 33%

instance2 33%

instance3 34%

LBA Advice:

OLTP Service

instance1 80%

instance2 10%

instance3 10%

OLTP

Servi

ce

Batch

Job

Connection Pool

Load Balancing Advisory is an advisory forbalancing work across RAC instances.

Load balancing advice

Is available to ALL applications that send work. Directs work to where services are executing well

and resources are available.

Adjusts distribution for different power nodes,different priority and shape workloads, changing

demand. Stops sending work to slow, hung, failed nodes

early.


12/39

12


Runtime Connection Load Balancing (RCLB)

works with JDBC, ODP.NETSolves the Connection Pool problem!

- Client connection pool is integrated with RAC Load Balancing Advisory

Easiest way to take advantage of Load Balancing Advisory

No application changes required, just enabled by parameter- When application does getConnection, the connection given is the one that will provide

the best service.

Service A

instance1 50%

instance2 50%

Service B

instance2 50%

instance3 50%

Service C

instance3 50%

instance4 50%

CLIENT requests

connection

Service A

instance1 80%

instance2 20%

Service B

instance2 50%

instance3 50%

Service C

instance3 20%

instance4 80%

instance 1 is bored.instance 1 is bored.

instance 2 and 3 is busy.instance 2 and 3 is busy.

instance 3 is so busyinstance 3 is so busy ..

CLIENT requestsconnection

RAC

Database Server


13/39

13


THROUGHPUT - Work requests are directed based onthroughput

maximize throughput

SERVICE_TIME - Work requests are directed based onresponse time.

minimize response time

None - Default setting, turns off advisory

Load Balancing Advisory Goal Options


14/39

14


We attempted to figure out

Can RCLB deal with nodes of differing power?

Can RCLB deal with multiple workloads runningasymmetrically?

What are the differences between using Throughput or

Response Time goals? If I have multiple workloads, what is the optimal way of

distributing them?

Any tricks to get it working?

Relating RCLB to our Hosting Grid


15/39

15




Test Environment


Conclusion


16/39

16


Test Environment

ProLiant AreaBladeCenterArea

Cisco Catalyst 3750

NetApp F3050NetAppF980

StorageArea(NAS)

Cisco Catalyst 4506

Gbps N/W (public)

Gbps N/W (private + data I/O)

Gbps N/W(data I/O)

x4 trunk x2 trunk

x4 trunk x4 trunk

x3 trunk x4 trunk

NAS I/O and RACInterconnects share

the network segment.

NAS I/O and RACInterconnects share

the network segment.

70nodes 8nodes


17/39

17


Specifications

IBM BladeCenter HS20- Intel Xeon 3.6GHz x 2

- 4GB RAM / 36GB Local Disk

- RedHat EL4 AS (Update3)

- Oracle DB 10gR2 (10.2.0.2)

HP ProLiant BL25p- AMD Opteron 275 2.4GHz x 1

- 4GB RAM / 36GB Local Disk

- RedHat EL4 AS (Update3)

- Oracle DB 10gR2 (10.2.0.2)

NetApp FAS3050- 72GB x 28 (1d+2p:RAID-DP,

9d+2p: RAID4-DP,12d+2p:RAID-DP)

- Data ONTAP 7.0.4

NetApp FAS980

- 72GB x 26 (24d+2p :RAID-DP)- Data ONTAP 7.0.3

Server & Client Disk Storage (NAS)


18/39

18


Service2

Service1

Test Overview

Client(Oracle10g JDBC)

DB Server (A)

(Oracle10gR2 RAC)

DB Server (A)

(Oracle10gR2 RAC)

Listener_

A

Listener_

A

DB Server (B)

(Oracle10gR2 RAC)DB Server (B)

(Oracle10gR2 RAC)

Instance

AInstance

A

Listener_

B

Lis

tener_

B

Instance

BInstance

B

NAS

Client

(Oracle10g JDBC)

JDBC(Connection Pool)

NAS I/O and RACInter Connect

share the networksegment.


19/39

19




Test Environment


Nodes of Distinct Capacity

Multiple Workloads Running Asymmetrically When to Use Throughput or Response Time Goal

Optimal Distribution of Multiple Workloads

Conclusion


20/39

20


node#1 node#2 node#3 node#4

service1

Prior measured performance comparison IBM : HP = 77.5 : 100


IB M IB M H P H P

Test Case

Number of nodes : 4 (IBM 2-node, HP 2-node)

Number of services : 1

Test Goal

Test how RCLB balances workload in multiple vendor environment.

Node construction


Service Allocation


21/39

21


THROUG HPUT - LBA O N -

0

50

100

150

200

250

300

rac15(IBM ) rac16(IBM ) rac29(HP) rac30(HP)

throug

hpu

t/s

(A

vg

)

THROUG HPUT - LBA OFF -

0

50

100

150

200

250

300


throug

hpu

t/s

(A

vg

)


Throughput

209.7

274.9

Average of 4

nodes 242.3 Average of 4

nodes 227.8

Transactions are distributedbased on CPU performance.Ratio of average throughput

IBM : HP = 76.3 : 100

Transactions are distributedbased on CPU performance.Ratio of average throughput

IBM : HP = 76.3 : 100LBA ON LBA OFF

When LBA is ON,transactions are distributed based on CPU performance.

When LBA is ON,transactions are distributed based on CPU performance.


22/39

22


R ESP O NSE-TIM E

0 5 10 15 20 25 30 35 40

LBA O FF

LBA O N

response-tim e (Avg m sec)


24.6

25.8

Average responsetime of 4 nodes

Average responsetime of 4 nodes

5% Improvement5% Improvement


Response Time

LBA ON

LBA OFF

When LBA is ON,response time of each node is equalizedand average response time is improved.

When LBA is ON,response time of each node is equalizedand average response time is improved.

IBM

IBM

HP

HP


23/39

23



CPU Utilization

C PU

0

10

20

30

40

50

60

70

80

90

100

LBA O N LBA O FF

C

PU

(U

S

R+

SY

S)

U

sage

(Avg

%)

CP U rac15(IBM ) C PU rac16(IBM ) CPU rac29(H P) CPU rac30(H P)

CPU utilization

equalized

CPU utilization

equalized

CPU utilization

differs

CPU utilization

differs

LBA ON LBA OFF

When LBA is ON,CPU utilization is equalized

even on different specd machines.

When LBA is ON,CPU utilization is equalized

even on different specd machines.

HPIBMHPIBM


24/39

24


Multiple Workloads Running AsymmetricallyTest Case

Number of nodes : 8

Number of services : 1 (Load Ratio service1:service2 = 5:2)

Test GoalTest how RCLB balances asymmetric multiple workload and find outgood configuration.

Service Allocation

service2

#1 #2 #3 #4

service1

#5 #6 #7 #8

Loading Method

20 min 10 min

stress to service2 (weight 1)



START

END

10 min 5 min


25/39

25


FAN-EVENT & TH RO UG HP UT (service1)

0

10

20

30

40

50

6070

80

90

100

20:17:24

20:19:54

20:22:24

20:24:54

20:27:24

20:29:54

20:32:24

20:34:54

20:37:24

20:39:54

20:42:24

20:44:54

20:47:24

20:49:54

20:52:24

20:54:54

20:57:24

20:59:54

21:02:24

21:04:54

tim e

throug

hpu

t

A

vg

0

10

20

30

40

50

6070

80

90

100

fan

A

vg

fan[rac1-4] tps[rac1-4]

FAN-EVEN T & TH RO UG HPU T (service1)

0

10

20

30

40

50

6070

80

90

100

20:17:24

20:19:54

20:22:24

20:24:54

20:27:24

20:29:54

20:32:24

20:34:54

20:37:24

20:39:54

20:42:24

20:44:54

20:47:24

20:49:54

20:52:24

20:54:54

20:57:24

20:59:54

21:02:24

21:04:54

tim e

throug

hpu

t

A

vg

0

10

20

30

40

50

6070

80

90

100

fan

A

vg


Multiple Workloads Running AsymmetricallyInitial Test Case Throughput doesnt follow LBA

RAC# 1 - 4 RAC# 5 - 8

LBA value is almost the same as we expected.10% difference between LBA value and actual throughput

ratio could be seen.

LBA value is almost the same as we expected.10% difference between LBA value and actual throughput

ratio could be seen.

Throughput

LBA

Throughput

LBA

time time

10% difference between LBA value and

actual throughput

10% difference between LBA value and

actual throughput


26/39

26


RAC#1-4s throughput is increased when service2 started,but it didnt follow the Load Balancing Advisory well.

When CLB_GOAL is LONG,even number of connections

are created for each database instance. But all of theconnections are used and no spares are available to giveRCLB flexibility.

Multiple Workloads Running Asymmetrically

Possible Cause

What occurs when we increasethe number of available connections?

What occurs when we increasethe number of available connections?

Why did such an situation occur?


27/39

27


0

10

20

30

40

50

60

70

80

90

100

15:58:17

16:00:17

16:02:17

16:04:17

16:06:17

16:08:17

16:10:17

16:12:17

16:14:17

16:16:17

16:18:17

16:20:17

16:22:17

16:24:17

16:26:17

16:28:17

16:30:17

16:32:17

16:34:17

16:36:17

16:38:17

16:40:17

16:42:17

16:44:17

16:46:17

16:48:17

16:50:17

tim e

throughputAvg

0

10

20

30

40

50

60

70

80

90

100

fan

Avg


0

10

20

30

40

50

60

70

80

90

100

15:58:17

16:00:17

16:02:17

16:04:17

16:06:17

16:08:17

16:10:17

16:12:17

16:14:17

16:16:17

16:18:17

16:20:17

16:22:17

16:24:17

16:26:17

16:28:17

16:30:17

16:32:17

16:34:17

16:36:17

16:38:17

16:40:17

16:42:17

16:44:17

16:46:17

16:48:17

16:50:17

tim e

throughputAvg

0

10

20

30

40

50

60

70

80

90

100

fan

Avg


Multiple Workloads Running Asymmetrically2nd Test Case Throughput follows LBA

RAC# 5-8RAC# 1-4

LBA value is almost same as throughput ratio.LBA value is almost same as throughput ratio.

Throughput

LBA LBA

Throughput

time time

Actual throughput follows LBA value.Actual throughput follows LBA value.


28/39

28


Connection Load Balancing

CLB_GOAL = LONG CLB_GOAL = SHORT

Client

Connection CacheManagerConnection CacheManager

Java ThreadJava Thread

Daemon ThreadDaemon Thread

ONSONS

DB Server ADB Server A

Listener_A

Listener_A

Instance AInstance A

ONSONS

DB Server BDB Server B

Listener_B

Listener_B

Instance BInstance B

ONSONS

Java Virtual Machine

LBA = 70 %

LBA = 30 %

Client

Connection CacheManagerConnection CacheManager

Java ThreadJava Thread

Daemon ThreadDaemon Thread

ONSONS

DB Server BDB Server B

Listener_B

Listener_B

Instance BInstance B

ONSONS

Java Virtual Machine

DB Server ADB Server A

Listener_A

Listener_A

Instance AInstance A

ONSONS

LBA = 70 %

LBA = 30 %

Balancing of available connections by LBA

1 2 3 4 5 6 7 8 9 10

Connection Cache

Balancing of available connections by LBA

1 2 3 4 5 6 7 8 9 10

Connection Cache

Multiple Workloads Running Asymmetrically

Connection Cache Manager Architecture


29/39

29


When to UseThroughput or Response Time Goal

Test Case

Number of nodes : 4

Number of services : 2 (Load Ratio service1:service2 = 7:4)

Test Goal

Compare the difference of balancing by LBA Goal Option and find

out how to choose it.

Service Allocation


service1

service2

20 min 10 min




START

END

10 min 5 min

Loading method


30/39

30


When to Use Throughput or Response Time Goal

Throughput

THROUGHPUT (Policy:THROUGHPUT)

0

50

100

150

200

250

16:30:22

16:31:22

16:32:22

16:33:22

16:34:22

16:35:22

16:36:22

16:37:22

16:38:22

16:39:22

16:40:22

16:41:22

16:42:22

16:43:22

16:44:22

16:45:22

16:46:22

16:47:22

16:48:22

16:49:22

16:50:22

16:51:22

16:52:22

16:53:22

16:54:22

16:55:22

16:56:22

16:57:22

16:58:22

16:59:22

17:00:22

17:01:22

time

throu

ghput/s

rac25 rac26 rac27 rac28 rac25 rac26 rac27 rac28

Throughput of each node

converges


converges

Policy: THROUGHPUT Policy: SERVICE_TIME

time

throu

ghput/s

THROUGHPUT (Policy:SERVICE_TIME)

0

50

100

150

200

250

17:55:16

17:56:16

17:57:16

17:58:16

17:59:16

18:00:16

18:01:16

18:02:16

18:03:16

18:04:16

18:05:16

18:06:16

18:07:16

18:08:16

18:09:16

18:10:16

18:11:16

18:12:16

18:13:16

18:14:16

18:15:16

18:16:16

18:17:16

18:18:16

18:19:16

18:20:16

18:21:16

18:22:16

18:23:16

18:24:16

18:25:16

18:26:16

service2service2

startedstarted

THROUGHPUT: Throughput is converged and stable.SERVICE TIME: Throughput is not converged and fluctuates

THROUGHPUT: Throughput is converged and stable.SERVICE TIME: Throughput is not converged and fluctuates

Throughput shows

fluctuation

Throughput shows

fluctuation

service2service2

startedstarted


does not converge


does not converge

service 1 runsservice 1 runs





31/39

31


FA N-EV ENT (Policy:THRO UG HPUT)

0

5

10

15

20

25

30

35

40

45

50

16:30:36

16:31:36

16:32:36

16:33:37

16:34:37

16:35:37

16:36:37

16:37:37

16:38:37

16:39:37

16:40:37

16:41:37

16:42:37

16:43:38

16:44:38

16:45:38

16:46:38

16:47:38

16:48:38

16:49:38

16:50:38

16:51:38

16:52:38

16:53:38

16:54:38

16:55:38

16:56:38

16:57:38

16:58:38

16:59:38

17:00:38

17:01:38

17:02:38

time

Ratioofthelo

adforeachnode(%)

rac25 rac26 rac27 rac28

FA N-EV EN T (Policy:SER VIC E_TIM E)

0

5

10

15

20

25

30

35

40

45

50

17:53:05

17:54:06

17:55:06

17:56:06

17:57:06

17:58:06

17:59:07

18:00:07

18:01:07

18:02:07

18:03:07

18:04:07

18:05:07

18:06:07

18:07:07

18:08:07

18:09:08

18:10:08

18:11:08

18:12:08

18:13:08

18:14:08

18:15:09

18:16:09

18:17:09

18:18:09

18:19:09

18:20:09

18:21:09

18:22:09

18:23:09

18:24:09

18:25:09

18:26:09

time

Ratioofthelo

adforeachnode(%)

rac25 rac26 rac27 rac28


LBA Value

6 minutes6 minutes 3 minutes3 minutes

LBA value

changes quickly.

LBA value

changes quickly.LBA value

changes slowly.

LBA value

changes slowly.


service2service2

startedstarted

service 1 runsservice 1 runs service 1 runsservice 1 runs

service 2 runsservice 2 runsservice 2 runsservice 2 runs

service2service2

startedstarted

SERVICE TIME: LBA value responds more quicklyto load variations

SERVICE TIME: LBA value responds more quicklyto load variations


32/39

32


tim e

throughputs/s

tim e

throughputs/s


Response time

service 2 started.service 2 started.service2 added.service2 added.

service 2 ended.service 2 ended.



service 2 started.service 2 started.service2 added.service2 added.

service 2 ended.service 2 ended.


When extra load is addedTHROUGHPUT: Response time gets worse.SERVICE TIME: Response time remained stable.

When extra load is addedTHROUGHPUT: Response time gets worse.SERVICE TIME: Response time remained stable.


33/39


34/39


35/39

35


Agenda

Company Background


Test Environment Test Cases & Results

Conclusion


36/39

36


Conclusion

Our test results show LBA can deal with

Nodes of distinct capacity

Multiple workloads running asymmetrically

Connection pool architecture

LBA effectively optimizesresource utilization in the grid environment

LBA effectively optimizesresource utilization in the grid environment


37/39

37


Tricks Learned & Recommendations

Ensure extra connection capacity in the pool

If there arent enough connections, LBA cant balance workloadthoroughly

=> Need to create large enough connection pool beforehand

CLB goal will determine where new connections go

Short follows LBA determination

Long equalizes distribution=> To use LBA for connection creation, set CLB goal to Short.

LBA policy is a quite effective way to balance workloads

THROUGHPUT keeps workload distribution stable

SERVICE_TIME - reacts faster, but has more fluctuation

=> For throughput stability, THROUGHPUT is preferable.


38/39

38


Service A Service C

Service B

Oracle10g RAC

ClientOracle10g AS Spare

How We Will Use RCLB

OEM

Design and Construct large-

scale, multi-node system inDB layer, AS layer and

storage layer

Achieving system

infrastructure consolidationby service definition

In GRID environment.

Optimize resource utilizationby using load balancing

functions.

Achieving reduction of

administration costby using OEM

in GRID environment

Data Center

Blade Serversconsist of

various platforms

Sales

Finance

HR

Resource management to handle load peaksResource management to handle load peaks

Realize

maximizing and optimally utilizingsystem resources

Stable performance for each serviceStable performance for each service


39/39

39


ORACLE is a registered trademark of ORACLE Corp.

All other names are trademarks or registered trademarks of their respective companies

NSS Workload Balancing Best Practices[1]

Documents

Transcript of NSS Workload Balancing Best Practices[1]