Configuring SIP Trunks between Avaya Communication Manager ...
Configuring Service Manager for Performance and Scale - MMS2013 Presentation
-
Upload
concurrency-inc -
Category
Technology
-
view
517 -
download
2
description
Transcript of Configuring Service Manager for Performance and Scale - MMS2013 Presentation
For more information: www.concurrency.com
Configuring Service Manager for Performance and ScaleNathan LasnoskiInfrastructure Architect, Microsoft MVPConcurrency
SD-B312For more information: www.concurrency.com
Experiences with Service Manager and OrchestratorVisibilityUnderstand what you deliver, how its configured, and how you support it
… One configuration database for 20,000+ servers, 1000+ SQL databases, and work items
Process ImprovementOptimize or re-shape processes for enterprise organizations
… 1,000,000+ work instances, each of which saved on average of 15 minutes of IT time
Automation (Client or Datacenter)Take manual processes and replace them with automation
… 200,000+ user and asset automations, each of which saving an hour of time
For more information: www.concurrency.com
What does performance mean?Console OperationsIncidents, problems, and changes open and close quicklyRead operations provide content quickly to IT users
Portal PerformanceNo or minimal “please wait” on the portalSubmissions complete without failure
Automation PerformanceAutomation execution time. Application deployment in 1 vs. 5 minutes makes a difference
Report PerformanceReport data is delivered quickly and user expectations are setFor more information: www.concurrency.com
What does scale mean?Massive User CountsCustomers with 200,000+ Users
Huge DatacentersDatacenters with 20,000+ Servers
GeographyGeographically dispersed users“Follow-the-sun”
Quantity of RequestsMillions of incidents, requests, and changes
For more information: www.concurrency.com
Agenda
Architecture for “Large” Scale Deployments (50,000+)SQL Server Configuration TipsUser Experience OptimizationWorkflow and Connector OptimizationData Warehouse Optimization
For more information: www.concurrency.com
Deployment Architecture for Large Scale
For more information: www.concurrency.com
Assumptions for Large Scale DeploymentsGeneral AssumptionsGeographically dispersed IT analysts and end usersConstant availability without large maintenance windowsFailover scenarios for alternate siteNeed for offloading functions to scale as needs arise
General AssumptionsLarge scale of users (50,000+ users)Large scale of work items (ex: millions of incidents per year)Large scale of configuration items (hundreds of thousands of computers, software, servers, etc.)
For more information: www.concurrency.com
Hardware Recommendations for Large ScaleManagement Servers4 – 8 core16 GB RAM(console and workflow servers)
SQL Servers16 – 24 core32+ GB RAMHigh performance disk
DW SQL Servers16 – 24 core32+ GB RAMHigh performance disk
For more information: www.concurrency.com
Hardware Recommendations for Large ScaleSQL Servers and DW SQL Servers8 – 16 core32+ GB RAMHigh performance disk
SharePoint and Web Content Servers4 - 8 core16 – 32 GB RAM80 GB HD
Orchestrator Management and Runbook Servers4 – 8 core16 – 32 GB RAMHigh performance disk
For more information: www.concurrency.com
Example Architecture for 50,000+ UsersManagement ServersDedicated workflow mgmt. serverDedicated console servers with load balancerDedicated mgmt. servers for Orchestrator
SQLClustering for all componentsAlwaysOn for operations databases
OrchestratorRunbook Servers
OrchestratorManagement
SM Management Servers
SM Connection Servers
SM Portal
SharePoint / WCS
SM Orchestrator Server
SM ORCH Target Server
Service Manager DBSQL 2012
Datawarehouse DBSQL 2012
Windows Server 201224 – 48 vCPUs, 32 – 192 GB RAM
80 GB OS, 600 GB Data
Windows Server 20124 – 8 vCPUs, 8 – 16 GB RAM
60 GB OS, 1 TB+ Data2-4TB of disk for 1095 days
DR: RTO 24 hours
Windows Server 20122 vCPUs, 8 GB RAM
80 GB OSScale number as
workloads demandRTO: 30 min?
Windows Server 20128 vCPUs, 16 GB RAM
60 GB OSScale as workload demands
RTO: 15 min
Windows Server 20124 vCPUs, 8 – 16 GB RAM
60 GB OSDR: RTO 24 hours
Windows Server 20124 – 8 vCPUs, 8 - 16 GB RAM
60 GB OS80-100 concurrent sessions/server
DR: RTO 30 minutes
Windows Server 20124 vCPUs, 16 GB RAM
First is failover for PrimaryScale as workloads demand
RTO: Globally Coninuously Available?
Windows Server 20124 vCPUs, 16 GB RAM
60 GB OSRTO: Not clear
Windows Server 20124 vCPUs, 16 GB RAM
First is failover for PrimaryScale as workloads demand
RTO: Globally Coninuously Available?
Windows Server 20122 vCPUs, 8 GB RAM
80 GB OSScale number as
workloads demandRTO: 30 min?
Service Manager DBSQL 2012
Windows Server 201224 – 48 vCPUs, 32 – 192 GB RAM
80 GB OS, 600 GB Data
Windows Server 20128 vCPUs, 16 GB RAM
60 GB OSScale as workload demands
RTO: 15 min
Windows Server 20128 vCPUs, 16 GB RAM
60 GB OSScale as workload demands
RTO: 15 min
Active DirectorySCOM, SCCM,Other Systems
For more information: www.concurrency.com
SQL Server ConfigurationTempDBHigh performance LUN, split IO to log volumeMultiple TempDBs (1 per 2 cores), but only one log, normalized sizeTempDB performance is critical to Service Manager performance
ServiceManager DatabaseHigh performance LUN, split IO to log volumeTransaction log set to BASIC by defaultTest performance of your disk:“sqlio -kW -t2 –s30 -o1 -frandom -b64 -BH -LS c:\Testfile.dat “
RAM AllocationConfigure for 2 GB less than the total RAM on the SQL server
For more information: www.concurrency.com
SQL Server ConfigurationSQL BrokerDrastic performance difference with workflowsValidate SQL Broker set to 1 vs. 0
Checking SQL BrokerSELECT is_broker_enabled FROM sys.databases WHERE name = 'ServiceManager‘
Setting SQL Broker (change window only)ALTER DATABASE ServiceManager SET SINGLE_USER WITH ROLLBACK IMMEDIATE ALTER DATABASE ServiceManager SET ENABLE_BROKERALTER DATABASE ServiceManager SET MULTI_USER
For more information: www.concurrency.com
Max Degree of ParallelismDefines parallel processing rules in SQLDefault is “0” and allows a single query to use all processor coresBetter results in our environments with “1” to “4”.
Learn more and set as required. Test, as your experience may vary.http://msdn.microsoft.com/en-us/library/ms181007(v=SQL.105).aspx
SELECT name, value FROM sys.configurations WHERE name = ‘max degree of parallelism’
Validate Read Committed Snapshot Isolation (RCSI)Validating isolation levelsSELECT name, is_read_committed_snapshot_on FROMsys.databases where name = 'ServiceManager‘
ALTER DATABASE ServiceManager SET READ_COMMITTED_SNAPSHOT ON
For more information: www.concurrency.com
SQL Performance for ServiceManager DBRAM Usage Page File % Usage should be less than 1%Memory – Available Mbytes should not be below 100mb (indicates starved OS)
Disk PerformanceAve. Disk Sec/Read and /Write should be less than 20 msec typically (some spikes ok)Ave. Disk Queue Length can indicate disk IO issues, though less valuable than Ave. Disk Sec/Read
CPU PerformanceValidate SQL service and CPU usage.
Change Grooming SettingsMinimize Retention Settings – ideally less than a week or twoCreate custom grooming rules
For more information: www.concurrency.com
SQL Logs and Recovery ModelCheck your log size and expansionDoes your log write, clear, and have expansion space?
Check your recovery modelUse SIMPLE, or configure a transaction log backupSuggestion is to use SIMPLE
For more information: www.concurrency.com
High AvailabilitySQL 2012 AlwaysOn High availability with AlwaysOnhttp://blogs.technet.com/b/babulalghule/archive/2013/02/17/how-to-install-service-manager-2012-sp1-with-a-sql-2012-alwayson-availability-groups.aspx
For more information: www.concurrency.com
DEMO: Checking your SQL Settings
For more information: www.concurrency.com
Pick the Right User ExperienceSelf-Service PortalEnd user self-service interactions or IT interactionsService Requests are excellent for “low / no training”, interactions
ConsoleHigh bandwidth, low latency scenariosRemoteApp necessary for high bandwidth scenariosLess than 100 msec or less150 – 200 msec has 40% degradation
Web Consoles (GridPro and Cireson)Excellent for low bandwidth and/or latency scenariosSome match with console experience (GridPro)
For more information: www.concurrency.com
Console OptimizationConsole OptimizationSlower performance when maximizedApply SP1 for console memory leak
Minimize Quantity of ViewsDelete views not needed. More views slows console load time.
Use Search vs. ViewsUse searching vs. views to find data quickly
Configure the Global Operators GroupLimits uses selectable when assigning work items (drastic performance improvement)
For more information: www.concurrency.com
View OptimizationBasic ViewsBring back only information from one classThe fastest view to create
Type Projections / Combination ClassesCombine classes to bring back information that includes relationshipsUse the smallest projection possibleDownload type projections for Incident, Problem, Change, and Service Request
Advanced ClassesDo not use the (Advanced) class in views. Use a type projection. The (Advanced) class can be used in searches that return small quantitieshttp://blogs.technet.com/b/servicemanager/archive/2010/12/02/faq-why-is-my-custom-incident-view-so-slow.aspxhttp://blogs.technet.com/b/servicemanager/archive/2011/09/19/new-change-request-type-projections-management-pack.aspx
Scoped User Roles vs. Non-ScopedScopedScoped user roles facilitate filtering based on groups and queuesMinimize use of scoped user roles due to additional table join in the database
For more information: www.concurrency.com
Portal PerformancePortal and IconsMore icons means longer load timeConsider using a “start page” to point to Service Offerings
Scope Request OfferingsUse user roles to scope access to service offerings and request offeringsThe more service offerings and request offerings, the longer the load
Disable App Pool Recycling for SharePoint and SSPDefault is nightly recycling which causes slow initial performanceMake sure not to set recycling to high memory usagehttp://technet.microsoft.com/en-us/library/cc753179(v=WS.10).aspx
For more information: www.concurrency.com
Portal PerformanceUse Searching in Request OfferingsDon’t increase the size of the query resultsUse pre-search filters to allow searching of hundreds of thousands of CIs in secondshttp://blog.concurrency.com/infrastructure/service-manager-request-query-result-filtering/
Use Known InformationUse previous questions and default values to filter queriesUse known information (such as user name and relationship to computer)Avoid MP ENUM questions as they cannot be used in post-selection filters
For more information: www.concurrency.com
Server and Workflow Optimization
For more information: www.concurrency.com
Dedicated ServersUse a Dedicated Workflow ServerAlways the first server in the management group by default
Use a Dedicated Orchestrator Target ServerAny server in the management groupUsed for any Orchestrator interactionUse a dedicated account for Orchestrator automations
For more information: www.concurrency.com
Workflow TweaksImplied Permissions WorkflowConsider disabling implied permissionsUtilize direct permissions with un-scoped operator role
New Priority CalculationDisable or use priority calculation ruleDisable Incident_Adjust_PriorityAndResolutionTime_Custom_Rule.Add if using SLOs
First Assigned RelationshipDisable “WorkItem_SetFirstAssignedTo_RelationshipAdd_Rule” if not used
SLO ApplicationLimit to what needed, as SLO application is performance heavyOrchestrator can apply SLOs based on complex rule matrixes.
Group Calculation IntervalMinimize Group Calculation IntervalDefault is every 30 seconds, 6000000 = 10 minutes
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\System Center\2010\Common\GroupCalcPollingIntervalMilliseconds
For more information: www.concurrency.com
Troubleshooting Performance DelaysHow to find a problematic workflowUse queries in blog post:http://blogs.technet.com/b/servicemanager/archive/2013/01/14/troubleshooting-workflow-performance-and-delays.aspx
Setting the WatermarkArtificially moving the watermark forward… only last resort
For more information: www.concurrency.com
OrchestratorWhen to Use OrchestratorComplex workflows, ex. Incident categorization, routing, notificationsUse vs. many individual Service Manager workflowsHigh capacity automation performancePerformance distribution of workflow processes
Incident CategorizationMonitor Object or “Get” initiationComplex routing and translation
For more information: www.concurrency.com
OrchestratorUsing Orchestrator with NotificationsComplex logic and offloading notificationshttp://blog.concurrency.com/infrastructure/scsm-notifications-with-orchestrator-roll-up/
For more information: www.concurrency.com
OrchestratorRunbook Initiation Option 1Service Manager workflow initiates runbook activityEasier and out of box
Runbook Initiation Option 2Orchestrator initiator runbook to find active runbooksDrastically faster than Service Manager OOB initiation (about 4 times faster!!)
For more information: www.concurrency.com
DEMO: Using Orchestrator for Workflow
For more information: www.concurrency.com
Connector ConfigurationConstrain SyncOnly sync those groups, users, and configuration items you need
Null is BadDo not sync “null” values on connectors. Connectors fight with each other.
For more information: www.concurrency.com
Connector ConfigurationDCM SynchronizationDisable DCM workflowhttp://blogs.technet.com/b/mihai/archive/2012/11/30/configuration-manager-connector-s-dcm-rule-can-cause-massive-performance-issues-in-service-manager.aspx
Using the DNS Trick for Active DirectoryDNS trick (fix in UR2)
Sync Custom VariablesSync variables from Active Directory not covered by connector, or fasterhttp://blog.concurrency.com/featured-post/how-to-sync-other-properties-from-active-directory-to-service-manager-using-orchestrator/ (password last set variable)
For more information: www.concurrency.com
Workflow / Connector SchedulingOption 1: Configure in XML or PowerShellCan cause workflows to run more efficiently
Option 2: Configure Orchestrator to Turn Jobs on and offConfigure Orchestrator job with schedule to turn on and off / start / end
For more information: www.concurrency.com
Batch SizeTweak the batch sizeCan cause workflows to run more efficiently
<Rule ID="CIListRule" Enabled="true" Target="OMConnectorLibrary!Microsoft.EnterpriseManagement.LinkingFramework.OpsMgrConnector.SyncWorkflowTarget" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100"> <Category>Maintenance</Category> <DataSources> <DataSource ID="DS1" TypeID="Subscriptions!Microsoft.SystemCenter.CmdbInstanceSubscription.DataSourceModule"> <Subscription> <InstanceSubscription Type="$MPElement[Name='OMConnectorLibrary!Microsoft.EnterpriseManagement.LinkingFramework.OpsMgrConnector.OpsMgrCIs']$"> <UpdateInstance /> </InstanceSubscription> <!--<StartWatermark>1</StartWatermark>--> <PollingIntervalInSeconds>10</PollingIntervalInSeconds> <BatchSize>1000</BatchSize> </Subscription> </DataSource> </DataSources>
For more information: www.concurrency.com
Orchestrator Topologies and Service ManagerOrchestratorUse a dedicated SQL serverUse dedicated runbook serversUse dedicated runbook and management servers for specialized functions
Service ManagerTarget specific Service Manager server, not used for consoles or workflows
HardwareRunbook servers: 4 core, 16 – 32 GB RAMManagement servers: 4 core, 16 – 32 GB RAMSQL servers: 4 core, 16 – 32 GB RAM
For more information: www.concurrency.com
Orchestrator Topologies and Service ManagerRunbook DesignUse smaller, more easily segmented runbooksDO NOT use “activity specific” logging in production (is substantially slower by 3 to 1 ratio)
Runbook ExecutionUse dedicated runbook serversUse dedicated runbook servers for specialized functions
Clear the LogsAutomatically clear logs, unless the cycle is to rapid, causing it to lock up the systemManually clear the logs or use Orchestrator job:http://blog.concurrency.com/infrastructure/virtualization/manually-clearing-orchestrator-logs/ For more information: www.concurrency.com
Orchestrator Topologies and Service ManagerBeware of the “Licensing Issue”Make sure to upgrade to SP1, beware of “license” issue in earlier Orchestrator builds
Hardware and LoggingMake sure your Orchestrator servers are on great hardware, especially disk hardwareDo not configure your logging improperly (use either simple or full with transaction log backups)
For more information: www.concurrency.com
DEMO: Manually Clearing the Logs
For more information: www.concurrency.com
DW and SQL PerformanceScalability and SQL Sizing ImportanceUse a dedicated SQL serverAllocate enough RAM and don’t starve the OSUnderstand analysis services impact and potentially offloadBeware cube processing impact
Running Jobs ManuallyHow to execute a job to correct issuesOrder of events when updating the DW
Snapshots in CMDBUtilize snapshot driven CMDB reports for accessing non-DW data without performance impactProvide reports in self-service driven model, such as through Reporting Services + SSP
For more information: www.concurrency.com
Configure Transform Batch SizeConfigure the Batch Size of the TransformAdjust to larger batch
The transform module default batch size is 50,000 items. The batch size can be adjusted by inserting rows into the DWRepository.ETL.Configuration table as follows:insert into DWRepository.ETL.Configuration( ConfigurationFilter, ConfigurationPath, ConfiguredValueType, ConfiguredValue)values ( 'etl.Transform', 'BatchSize', 'Int32', '100000') For more information: www.concurrency.com
Top Tweeted QuestionsQuestions…Why is Service Manager so cool?
For more information: www.concurrency.com
Evaluation
Complete your session evaluations today and enter to win prizes daily. Provide your feedback at a CommNet kiosk or log on at www.2013mms.com.Upon submission you will receive instant notification if you have won a prize. Prize pickup is at the Information Desk located in Attendee Services in the Mandalay Bay Foyer. Entry details can be found on the MMS website.
We want to hear from you!
For more information: www.concurrency.com
Resources
http://channel9.msdn.com/Events
Access MMS Online to view session recordings after the event.
For more information: www.concurrency.com
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
For more information: www.concurrency.com