Troubleshooting ASP.NET and IIS Scalability Hotspots

Please Standby: Webcast to Begin Shortly

Unable to listen via your speakers? Dial in: Toll Free: (877) 309-2074 Access Code: 119-063-034

International: Click “telephone” in audio section,then click “additional numbers”

Troubleshooting ASP.NET and IIS Scalability Hotspots

Confidential, Dynatrace, LLC

Asad AliDirector, Product SpecialistDynatrace

Troubleshooting ASP.NET and IIS Scalability Hotspots

• Real Use Cases• IIS/ASP.net Metrics• High Latency Application Example• Synchronization Issues• Sharepoint Example

Agenda

Confidential, Dynatrace LLC

Real Use Cases

Online Room Service

22% spent in IIS

Online Room Service

Tip: Elapsed Time tells us WHEN a Method was executed!

Online Banking: Slow Balance Check1.69m (=101s!) To

Check Balance!

87% spent in IIS

Time really spent in IIS

Tip: Elapsed Time tells us WHEN a Method was executed!

Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET

Tip: Thread# gives us insight on Thread Queues / Switches


• Modules Installed

• IIS/ASP.net Metrics

What To Look


IIS Modules

IIS/ASP.net Metrics


Arrival Rate• Rate at which requests are arriving in the queue

• Perfmon Counter: Http Service Request Queues\ArrivalRate

• Should be >= W3WP_W3SVC\Requests / sec

IIS/ASP.net Metrics


MaxQueueItemAge• Incoming requests are waiting at this long to be processed

• Perfmon Counter: Http Service Request Queues\MaxQueueItemAge

IIS/ASP.net Metrics


CLR ThreadPool Queue•Requests are queued if all the CLR threads are busy

• Perfmon Counter: ASP.NET v4.0.30319/Requests Queued

• Symptom: ASP.net returns 503 Service Unavailable

IIS/ASP.net Metrics


Number of Free DB Connections

• Number of free ADO.net connections available

• Perfmon Counter: .Net Data Provider for SqlServer/NumberOfFreeConnections

• Alert: If value == 0

ADO.Net Metrics


Number of Active DB Connections

•Number of Active ADO.net connections

•Perfmon Counter: .Net Data Provider for SqlServer/NumberOfActiveConnections

ADO.Net Metrics


High Latency Application Example


Too Many Queries crash

Undersized Pools

Online Banking: Slow Balance Check1.69m (=101s!) To

Check Balance!

87% spent in IIS 600! SQL Executions

SQL ExecutionsFinding: EVERY SQL

statement is executed on ITS OWN Connection!

Tip: Look at “GetConnection”

Lesson Learned

ASP.NET Worker Thread Pool Sizing!

DB Connection PoolsMore Efficient SQL

SQL ExecutionsFinding: EVERY SQL

statement is executed on ITS OWN Connection!

Tip: Look at “GetConnection”


SQL Executions

#1: Same SQL is executed 67! times

#2: NO PREPARATION because everything

executed on new Connection


• Idle vs. Busy Threads• # SQLs / Request• # GetConnection

Tips To Apply


SynchronizationIssues


Synchronization Issues

Load vs. Memory vs Response Time vs. Threads


Increasing MemoryAvg Response Time: Increases over Time

Max Response Time correlates with # Threads

Max Response Time correlates with # Threads

Load: System can’t handle constant load over time

# of Worker Threads increase with constant load

Layer Breakdown: Performance Problem is in their own code (Sync Time)


96% Synchronization TimeMonitor.Enter: By far biggest

performance contributor

Called from GetStatus and _GetDebugStatus. Why also from Debug method? Bug?


Excessive file attribute access second hotspot

Can FileExist check be cached to reduce

File Access?

File Access next hotspots


• # of Threads vs # Incoming Requests

• Time Spent in Sync• Time Spent in I/O

Tips To Apply


Sharepoint Example


Frustrated user report bad response times

Frustrated User

Slow Page Load caused by Browser JS Time

Slow Page Load caused by Server-Side Processing


Really slow page6.8s to deliver Default.aspx page

Involved Web Parts

Most of the Time spentIn waiting


WebPart uses multiple parallel threads

Async Threads are busy with I/O


First remote call is very slow

Web Service call by ContentEditorWebPart

HttpWebRequests uses ServicePoint internally

First Web Serivce Requests takes 5.8s to return


Thread limit lets all other threads wait!

We have 10 parallel calls in our background threads

The other background threads spend their time“waiting” in the ServicePoint


Solution: Change Defaults


• Review installed IIS modules• Pay attention to IIS/ASP.net metrics• Optimize thread synchronization• Reduce I/O • Analyze SQL statements• Use DB Connection Pool Optimally

Tips To Apply


Free Trial: bit.ly/dttrial

Thank you!

Time for Q&A

Troubleshooting ASP.NET and IIS Scalability Hotspots

Technology

Transcript of Troubleshooting ASP.NET and IIS Scalability Hotspots