Mark Friedman

Demand Technology Software

markf@demandtech.com

http://performancebydesign.blogspot.com

Monitoring Web Application Response

Times in Windows

1

Why is this web app running slowly?

2

Main Topics

Building scalable web applicationsYSlow scalability model of web page compositionPage Load Time and Round tripsLimitations of the approach

W3C Nav/Timing APIGoogle Analytics & RUM

Progress report on an ETW-based approach to web application performance monitoring

plus, sprinkle in some case studies & best practices

Presentation Outline

3

Themes:The Value of Response Time measurements

Service Level reporting Application response time measurements correlate with measures of customer

productivity & satisfaction

Queuing models, decomposition & other analytic techniques

Obstacles: Measurement data missing from standard Windows ASP.NET counters, but it is

available from other sources Request-Response boundaries are blurred in many AJAX applications Understanding how to set good response time objectives

Since human beings are adaptable & “ Good” and “Bad” are often relative to the application context See “Engineering Time,” by Dr. Steve Seow

Web Application Performance

4

Themes:Web application programming models and fashion change faster

than tools can adaptAJAX

e.g., Auto-complete in Google Searchachieved using client-side Javascript & Asysnchronous Http web service

RequestsHigh Availability and Scalability using n-tiered architectures

typically, a Presentation Layer, Business Objects layer, and a Data Access LayerHTML5

Effective performance tools are usually one or two generations behind emerging technologye.g., http://webpagetest.org

Web Application Performance

5

Fundamental concept in software performance engineering (SPE)

namely, f(x), such that f(x) reliably predicts Response time.

Factors can be linear (m+n…)mutliplicative (m * n)exponential (mn)

Scalability models

6

Why is my web app running slowly?

Part 1.

7

Case Study.

Developed using ASP.NET (Server-side controls)Multiple tiers: presentation/business objects/data layer

Uses the Model-View-Controller (MVC) pattern

Key elements of the web Pagedata-rich Charting component (.NET Chart, based on the Dundas component)Chart definition used to generate the PDB Query and the results are mapped

to a Chart instance Library of Chart templates Machine selection Date/Time selection

8

Why is this web app running slowly?

Ask YSlow

9

Why is this web app running slowly?

Based on the influential work of Steve Souders*originally at Yahoosince migrated to Google

Google Chrome extensionRule-based Influenced:

Chrome PageSpeed Insights IE Developer ToolsFiddlerGlimpseetc.

* High Performance Web Sites, O’Reilly Media, 2007

10

What is YSlow?

Optimize for Page Load Time Request.Start DOM.Complete

Make specific recommendations on how to improve Page Load Time for a specific web pageInventorying the Document Object Model (DOM) after the composition &

rendering the of web page: Calculate # of Http objects and their size

Does not attempt to actually measure Page Load Time, however.

Lead to a standardization effort to wire performance timing data to the DOM & create a consistent way to access it

Navigation Time, Performance Timing & a High Resolution Clock

11

What is YSlow?

12

Page Load Time

Web Browser Http Server

Http GET Request (Uri)

Compose&

Render

Http RESPONSE

Note: Http is a sessionless protocol

Measurement techniques

• sniff network packets• monitoring service to

submit synthetic Requests• Real User Measurements

• attach handlers to DOM window.unload and window.load events

13

YSlow Chrome extension

14

Chrome PageSpeed

Response message often contains embedded references to additional resources needed to render the Page

e.g., image files style sheets javascript files 15

Page Load Time

Web Browser

Http Server

Http GET Request (Uri)

Compose & Render

Http RESPONSE

HTTP interacts with the underlying TCP/IP networking protocols HTTP Response messages > Ethernet packet size (~ 1500

bytes) require multiple IP packets With large cookies and a large number of parms, GET Request

messages can even exceed the Ethernet packet size16

Page Load Time

Web Browser

Http Server

Http GET Request (Uri)

Compose & Render

Http RESPONSE

YSlow scalability model:

assuming web client processing time is minimal, then

Render Time RoundTrips * RTT

RoundTrips =

17

Page Load Time

Ask YSlo

w

18

Why is this web app running slowly?

YSlow scalability model:Browser Render Time RoundTrips * RTT

Web Browser performs page composition using the Document Object Model, or DOMYSlow Rule: Make fewer HTTP requestsYSlow Rule: Improve cache effectivenessYSlow Rule: Reduce the number of DOM elementsYSlow Rule: Compress the objects the page does need to load

Tuning is a process that attempts to drive # RoundTrips 1

RoundTripTime 0 19

YSlow Scalability model

YSlow never actually measures RTT, but other related tools can

RTT may vary across RequestsObjects can be geographically dispersed

Local cluster, remote, cloude.g., referencing 3rd-party, advertising services

TCP adaptive window scaling and other network congestion avoidance strategies

Ignores variability in the execution time of client and server-side codee.g., sort() a large list of elements, ora hi-res visualization component that scales nonlinearly with the size of the result

setCompression recommendations can be at odds with good code

maintainability practices

20

YSlow Scalability model complications

Web Browsers create multiple TCP sessions to download referenced objects in parallelYSlow Rule: take advantage of parallel sessions by loading scripts last

Accurate rendering of the DOM requires that downloading a Javascript file blocks parallel downloadsScript may add elements to the DOM dynamically, call other scripts or reference

additional resources Browser assumes DOM rendering can only resume after the Javascript code

executes

Page Load Time = Browser Render Time +Script execution Time +

(RoundTrips * RTT)/Sessions

21

YSlow Scalability model complications

Despite the many complications, the YSlow scalability model has proved very influentialBrowser Page Load Time is an end-to-end measurement of

service time, which is apt to be correlated with customer satisfaction

see, for example, http://www.slideshare.net/Strangeloopnet/37-lessons-ive-learned-on-the-performance-front-lines

Inspired development of related tools to measure Page Load Time

22

YSlow Scalability model

The YSlow scalability model is useful, but it is not adequate for many web applications

23

Case study: YSlow scalability model assessment

Page Load time is a measure of end-to-end response timeNavigation Timing measurements decompose

overall response time into Network, Server, and Client (i.e., web browser) components

YSlow is silent: the scalability of server-side componentsthe diversity of web client hardware (PCs, tablets,

phones) & software (iOS, Android, Windows)the performance of the client’s network

connectione.g., Internet vs. Intranet connections

24

YSlow Scalability model assessment

Despite the many complications, the YSlow scalability model has proved very influential

Sparked a standardization effort so Javascript developers could access the PLT measurements reliably across Browsers

Creation of standard DOM performance objects that are accessible to Javascript

Finally, now that actual web application performance data is available in the web browser, how do I get that data back to my data center for optimization & capacity planning?

25

YSlow Scalability model

26

Standard DOM NavTiming object

Prompt for unload redirect App

cache DNS TCP Request Response Processing load

unload DOMContentLoaded

navigationStartredirectStart

redirectEnd

fetchStartdomainLookupStart

domainLookupEnd

connectStartsecureConnectionStart

connectEndrequestStart

responseStartresponseEnd

loadEventStartloadEventEnd

unloadEventStart

unloadEventEnd

domLoading

domInteractive

domContentLoadedEventStart

domContentLoadedEventEnd

domComplete

Http Request/Response and Rendering events

Web Performance Working GroupPerformance Timeline, Navigation Timing,

and High Resolution Time specs Supported in IE, Chrome, WebKit, and FoxFire

Event timings can be use to calculate:

Network latency (from the standpoint of the web client) Page Render time (once the page components are

received from the server) Entire sequence from navigation to page load

completion:

27

Page Load Time Measurements

Navigation Timing spec: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html#processing-model

Calculate network latency: responseEnd - fetchStart

28

Page Load Time Measurements

Calculate Page Render Time:loadEventEnd - responseEnd

29

Page Load Time Measurements

Calculate entire sequence: loadEventEnd - navigationStart.

30

Page Load Time Measurements

see https://developer.mozilla.org/

31

performance.now High Resolution Clock

<html> <head> <script type="text/javascript"> // Add load event listener. window.addEventListener("load", loadTime, false);

function loadTime() { // Get current time. var now = window.performance.now(); // Calculate page load time. var page_load_time = now - performance.timing.navigationStart; // Write the load time to the F12 console. if (window.console) console.log(page_load_time); } </script> </head><body> <!- Main page body is here. --> </body> </html>

Now that actual web application performance data is available in the web browser, how do I get that data back from the web client?What about the volume of measurement data that requires processing

& analysis?

Google Analytics uses web beacons to send the response time data to Google’s data center for analysis and reporting

Yahoo Boomerang project can be used to send web beacons with the data back to your data center

32

NavTiming API

Architecture of an ASP.NET applicationWindows OSTCP/IP(Clustered) IIS front end Web ServersASP.NET.NET CLRADO.NET (data layer)

33

Why is my ASP.Net app running slowly?

Dynamic HTML is session-oriented behavior layered on top of the HTTP

Web applications require state Who you areWhere you are (mobile apps)

They preserve state:During a session

TCP protocol is session-orientedRequires a connectionCalculates RTT per connection (used in re-try logic)

Between sessions (using cookies, etc.)

34

ASP.NET Application Architecture

Server-side Request processingEvent-oriented programming model (Postback)HttpContext wrapper around the HTTP RequestPersistent State

ViewStateSession StateApplication and Page Cache objectsetc.

ASP.NET Application Performance

35

36

IIS Architecture

User

Kernel

HTTP

TCP

IP

Network Interface

HTTP Kernel Mode Driver

(http.sys)

HTTP Response Cache

(Physical Memory)

LSSAS

WindowsAuthentication

SSL

Inetinfo

IIS Administration

FTP

Metabase

SMTP

NNTP

SVCHOST

W3SVC

W3wpW3wp

Mscoree.dll

http

<code-behind>.dll

Default.aspx

Application Pool

W3wpW3wp

Mscoree.dll

<code-behind>.dll

Default.aspx

Application Pool

WAS

Cache

net.tcp net.tcp

http

HttpApplication Event Handlers (IHttpModule)

IIS 7.0 Integrated Mode (pipeline)

Event Event

BeginRequest PreRequestHandlerExecute

AuthenticateRequest PostRequestHandlerExecute

PostAuthenticateRequest ReleaseRequestState

AuthorizeRequest PostReleaseRequestState

PostAuthorizeRequest PostMapRequestHandler

ResolveRequestCache PostMapRequestHandler

PostResolveRequestCache PostMapRequestHandler

MapRequestHandler UpdateRequestCache

PostMapRequestHandler PostUpdateRequestCache

AcquireRequestState LogRequest

PostAcquireRequestState EndRequest37

Extends the ASP.NET event model to all Http RequestsHttpApplication Event Handlers (IHttpModule)

IIS 7.0 Integrated Mode (pipeline)

38

public class BoomerangBeacon : IHttpModule        {            public MyHttpModule()            {}            public void Dispose()            {}            public void Init(HttpApplication application)            {                application.BeginRequest += (new EventHandler(this.Application_BeginRequest));            }

}

w3wp.exew3wp.exe

Common Language Runtime (CLR)

JIT compilerGarbage Collection threads

mscoree.dllmscorsvr.dllMyApp.dll

ASP.NET Application Architecture

39

Many, many Windows components are instrumented with ETWKernel objects include disk IO, process,

thread, paging, CPU TCP/IP, HttpServer, CLR

Providers may issue a current status Rundown

CallStacks can be captured

Event correlation:CPU ID, Process ID, Thread ID, File

Handle, Tcp Session ID (Port)

40

ETW: Event Tracing for Windows

Many obstacles to using trace-based instrumentation effectively in performance investigationsSheer volume of trace data that can be generated (e.g., OS Dispatcher

ContextSwitch events) Cannot always be filtered effectivelyVery little documentation on specific events and how they are logically

related

Inconsistent semantics (but only a few common patterns)Sequence: Begin, Step, EndStateChange(oldState, newState)Fork:JoinSend:Receiveetc.

41

ETW: Event Tracing for Windows

Resource MonitorWindows Performance tools (aka xperf) VS Profiler’s Concurrency Visualizer

inspired by Rico Mariani’s ETLStackBrowser programleveraged Vance Morrison’s TraceEvent .NET libray

and, introducing the Web Application Trace ExplorerGathers, analyzes, filters & reports on server-side events from TcpIP,

HttpServer, and instrumented application Scenarios, plus Boomerang Beacon data from web clients

Designed to help explore the semantics of these sparsely documented event streams 42

ETW tooling

IHttpModule to intercept the Boomerang beacon requests and turn them into ETW events

43

ETW tooling private void Application_BeginRequest(Object source, EventArgs e){

// Create HttpApplication and HttpContext objects to access         // request and response properties.         

HttpApplication application = (HttpApplication)source;HttpContext context = application.Context;

                string beaconUrl = GetBeaconUrl(); // Helper routine for

beacon url     string filePath = context.Request.FilePath;     string fileExtension = VirtualPathUtility.GetExtension(filePath);

if (fileExtension.Equals(".gif"))         {          if (filePath.Contains(beaconUrl))            {

... // Process the beacon parmsapplication.Response.End();

} }

}

Demo using sample data from webscorer.com

44

45

Main Navigation Window

46

47

HTTP activity by URL

48

Navigate to URL Detail

Boomerang beacon supportAdd the boomerang.js script to your web page HTML:

Initialize boomerang:

49

Web Application Trace Explorer

<script src="javascript\Boomerang\boomerang.js" type="text/javascript"></script><script src="javascript\Boomerang\plug-ins\navtiming.js" type="text/javascript"></script><script src="javascript\Boomerang\plug-ins\rt.js" type="text/javascript"></script

<script lang="javascript"> BOOMR.init({beacon_url: "boomerang.gif"});</script>

Boomerang beacon support

Add the BoomerangBeacon HttpModule from MeasurementWareWebServices.dll to web.config

50

Web Application Trace Explorer

  <system.webServer>     <modules>       <add name="MeasurementWareWebServices“              type="MeasurementWareWebServices.BoomerangBeacon"/>     </modules>   </system.webServer>

BoomerangBeacon classderives from IHttpModuleadds an EventHandler for HttpApplication.BeginRequest

In the BeginRequest event handler,intercepts the boomerang.gif GET Requestsparse the beacon parmsgenerate a MeasurementWareWeb ETW event whose payload

includesPage Load Time measurementsplus, IP Address and Tcp Port (unique session ID) of the Sender for correlation

with other HttpServer and TcpIP events

51

Web Application Trace Explorer

52

53

Beacon data summarized by URL

54

Navigate to URL Detail

55

Initial stage of the work to add Boomerang beacon data to the event collection and the displays is functionally complete

Limited ToDo List prior to release as a github Open Source projectWaterfall ViewIIS ServerVariables: (browser, platform, etc.)

Contact me if your org is interested in participatingmarkf@demandtech.comSee my blog at

http://performancebydesign.blogspot.com for the latest status (until it reaches the Release to github stage)

Web App Trace Explorer: Project Status

?56

Questions