2

3

Have you ever had a performance related emergency?

4

5

TIME IS MONEY

6

TIME IS MONEY

7

GREATER CONSEQUENCES

War Stories

8

A Park’s Perf War

9

10

PARK RESERVATION PERF WAR

11

WORKLOAD

Business Goals

Requirements Current Usage

HistoricalUsage

Projections

!Success

12

PARK RESERVATION WORKLOAD

HISTORICAL RESERVATIONS COUNTS

Peak Reservations / hr(Opening day for 9 parks) 3509

Peak Reservations on opening day 6950

Peak Reservations / hr(Opening day for 5 other parks) 3400

Peak Reservations on 2nd busiest day 7900

13

WHAT DATA WAS ANALYZED• Perfmon, PAL, and Yslow• Key transaction response times• Resource utilization• Validation of proper application calls

14

BUSINESS LESSONS LEARNED• Performance Considerations• Solid Workload Model• Peak busy min… High traffic days

15

TECHNICAL LESSONS LEARNED• Not what it seems to be• Always suspect the DB• Disk I/O• Front End Performance• Validation of proper application calls

A Retailer’s Perf War

16

17

OVERVIEW• Large Scale Test• Quick Ramp to Full Load• Need to Come from Multiple GEOs

18

BATTLE WOUNDS

19

RETAIL LESSONS• Speak to CDN provider prior to any load test• Ensure at least a 15 minutes ramp-up to full load• Testing off hours• Ensure Load distribution is across as many available

regions as possible• No more than 300 Mb/s traffic should be generated

from one geo area + ISP combination

A Bank’s Perf War

20

21

THE CHALLENGE• 1000+ Branches • New software deployment

22

BANKING BATTLEGROUND• IBM AIX Environment• IBM HTTP Server ( Apache), WebSphere

Application Server, Oracle 10• Web client front end (Javascript)• F5 Load Balancers

23

BANKING STRATEGY• Examine how traffic was being

distributed to the servers • Suspected problem with app traffic load

balancing

24

BANKING VICTORY• Investigated load balancer algorithms • Made the change – problem solved!

Software performance testingand its best practices.

25

26

MONITOR ALL RELEVANT COMPONENTS

27

HOW DOES THE DATA FROM THE TOOLS HELP?

28

ALSO• Make use of collection and analysis tools

readily available for the platform• Examine performance results data against

platform thresholds• Document resource utilization issues• Establish origins of adverse performance

events

29

GET THE LOW HANGING FRUIT

Trend Micro Virus ScanOverall Counter Instance Statistics

30

GET THE LOW HANGING FRUIT

CONDITION\PROCESS(*)\IO DATAOPERATIONS/SEC

MIN AVG MAX HOURLY TREND

STD DEVIATIO

N

10% OF OUTLIERS REMOVED

20%OUTLIERS REMOVED

30% OF OUTLIERS REMOVED

More than 1000 data IO operations (network,

disk, or device IO) per second

SQL1/NTRTScan 0 99 1,985 29 404 1 0 0

31

SOFTWARE PERFORMANCE ANTI-PATTERNS

Traffic didn’t crash the Obamacare site alone. Bad coding did too.

Excessive DB Hits OrganicInheritance

Insufficient DBExpertise

No Active SystemMonitoring for

Continuous Improvement

Non-technicalPMs

The fastest path to a slowdown requires no action

32

ANTI-PATTERNS CONT’D

Excessive ObjectCreation One Lane Bridge

UnmeasureCache

Behavior

33

ANTI-PATTERNS EXAMPLETo prioritize the issues for resolution, one possible ordering is as follows:

1. SQLServer: SQL Errors Errors/sec

2. SQLServer: Access Methods Workfiles Created/sec

3. SQLServer: Locks Lock Timeouts/sec

4. SQLServer: Latches Latch Waits/sec

5. SQLServer: Deprecated Features Usage

34

DOCUMENT CHANGES BETWEEN DELIVERED BUILDS

35

DON’T IGNORE FRONT-END PERFORMANCE• An under performing front end can

negate all downstream optimizations to the end user

• Modernize legacy code• Simplify functionality• Handle exceptions• Sometimes problem(s) exist on

client platform and not application• 80 % of total response time is spent

on the front end

FRONT END ISSUES

Fewer HTTP Requests

Using a CDN

Some content not HTTP compressed

Image optimization opportunities

Minify JS and CSS

Large ASP.NET ViewState

Questions?

36

Amit PatelTotal Performance Consulting

@aapatelapatel@totalperform.com

@TotalPerformwww.totalperform.com

37

Thank You.