Project MedEvac 1.0
-
Upload
abdul-kader-s -
Category
Documents
-
view
111 -
download
0
Transcript of Project MedEvac 1.0
CITRIX MEDEVAC
GLOBAL ESCALATION TEAM
2 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Introducing Citrix Medevac 1.0
• What is Citrix Medevac 1.0?
• How does it work?
• Architecture
• How can Citrix Medevac help me?
• Case Study
• When can I use it?
• Limitations
3 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
What is Citrix Medevac 1.0?
• Citrix Medevac 1.0 is a troubleshooting Utility developed by the Citrix Escalation Team to aid in troubleshooting and resolving complex and time consuming issues that are XML, Terminal Services, IMA and RPC related
• It is a standalone Utility that does not require a Windows Service or Server Side components to be installed for its use
• It can be run from any machine on the network
4 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
What is the Health Check Agent?
• The Citrix Health Check Agent runs as a service on each Citrix Server
• This Agent has the ability to perform actions based on test results
• The tests monitor for the following: XML ticketing issues IMA Service application resolution issues Terminal Services health
• This tool is extensible by administrator to be configurable for monitoring other custom events.
5 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE
How does it work?
A Windows 32-bit Utility used to simulate the ICA client browsing requests to test XML,Terminal Services,IMA & RPC functionality
Uses method of Isolation based on Citrix Presentation Server Farm architecture to isolate Farm functionality
6 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
“Medevac”
ARCHITECTURE
XML BrokerXML Broker
IMA Service
TermSrv
XML Service
Server Farm
Request the Least Loaded Server
DirectLL ServerQueries
RPC
Zone Data Collector
LHCDS
IMA
7 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE…
XML TICKETING TEST:
By simulating a client connection request we are able to test XML ticketing using Medevac as if it were an actual ICA client.
RPC TEST:
Based on input provided by the user, RPC tests can be verified by making RPC calls to the specified Server testing its functionality and returning values based on the result.
TERMINAL SERVICES TEST:
By querying information from Terminal Services remotely using Medevac, we are able to test the responsiveness of Terminal Services to ensure that it is functioning properly.
8 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
IMA (Local DataCollector)
XML Service
IMA (Remote DataCollector)
Web InterfaceWeb Server
XML Broker
Or
XML Least Loaded Server Request
XML – Launching an Application.
9 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
The Least Loaded Server
Termsrv
wsxica.dll
CCTicket.dll
CCTicket.dll
XML Service
CCTicket.dll
XML Ticket
Request
IMA Service
CCTicket.dll
or
XML IMA Service
XML Broker
Web InterfaceServer
XML Ticket
Request
credentialsticket
credentialsticket
credentialsticket
XML – Launching an Application.
10 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE
The Least Loaded Server Tests:
Performs XML, RPC & Terminal Services tests• It uses the XML Broker’s XML service to request the least loaded
farm server for a specified published application• It uses that least loaded server IP address to make health checks
directly against the least loaded server out of the servers that have that published application available
• This is a more efficient method for checking the availability of critical environment applications rather than a “shotgun” approach
Example: If your environment has a couple of hundred servers it becomes increasingly difficult to isolate a problem server
• More importantly, this utility can proactively find a problem least loaded server before users do, reducing user outages and help desk calls
11 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE
What is actually verified on the listed server farm:
The XML service
The IMA service
The Termsrv Service
The RPC service
12 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Enhancements for Medevac 2.0
• Save test configuration to a configuration file.
• Save and import a server lists.
• Command line parameter implementation.
• Application name browsing.
• E-Mail alerts (SMTP).
• Full ICA Connection test.
• Multi-threaded.
• Allow administrators to restart certain services. (Maybe)
13 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
How can Citrix MedEvac help me?
• Imagine a situation where you have an outage in your entire Citrix Farm during peak production time. None of your users can launch their published applications
• Everything else seems to check out fine
• Where would you begin?
• Any time wasted has severe business impact
Your Solution:
CITRIX MEDEVAC 1.0
14 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Typical “Black Hole” Problem
Symptoms
• Server cannot accept user connections
• Server still responding to Data Collector’s heart beats
• Data Collector views server as “least loaded”
• Data Collector directs incoming user connections to this “least loaded” server
• Denial of Service occurs to users connecting to these problematic servers.
15 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
“Black Hole” Problem(s)
DataCollector
Problematic & “Least Loaded”Presentation Server
IMA TermSrv
XML
“Heart Beat”
IMA
Healthy Presentation Server
IMA TermSrvXML
Server Load Levels
“Heart Beat”
XML
User Connections
16 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Medevac 1.5 Demo
Demo
17 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
USERS UNABLE TO CONNECT TO THEIR PUBLISHED RESOURCES
18 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
PROBLEM DEFINITION:
• Users are unable to connect to published resources in the Farm
ENVIRONMENT:
• CPS Version: Citrix Presentation Server 4.0
• CPS Hotfixes: PSE400W2K3R01
• OS Version: Windows Server 2003 Enterprise/wSP1
• Data Store: Microsoft SQL 2000/wSP3a
• CPS Servers: 50
• Zones: 1
• Connection Method: Web Interface 4.0
• ICA client: 9.x
19 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
TROUBLESHOOTING METHODOLOGY:
THE APPROACH:
Description of issue:
User attempt to launch a published application and receives an error - "Server - unavailable" or "Cannot connect to the Citrix MetaFrame Server. I/O error"
Some things we know:• Issue is affecting all users• Issue has never occurred before in this Environment• Nothing has changed in the Environment• Users are connecting through Web Interface 4.0• Web Interface does enumerate the published resources• Connecting directly to a Server works
Now Lets take a closer look at an ICA connection….
20 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
IMA (Local DataCollector)
XML Service
Problems Launching An Application (Step 1)
Problems Launching An Application (Step 1)
IMA (Remote DataCollector)
Web InterfaceWeb Server
XML Broker
Or
XML Least Loaded Server Request
Local Host Cache
Dynamic Store
21 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
The Least Loaded Server
Termsrv
wsxica.dll
XML IMA Service
CCTicket.dll
credentialsticketcredentialsticket
credentialsticket
CCTicket.dll
Web InterfaceWeb Server
XML Ticket Request
Problems Launching AnApplication (Step 2)
Problems Launching AnApplication (Step 2)
XML Broker
End User Client
Connection / Logon To The Citrix Server’s
Published Application
22 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
TRADITIONAL APPROACH:
• QFARM
• Zone Isolation
• Isolation based on application hosting (which servers host certain apps)
• Direct server connections
• Manual testing using custom ICA connections
• Forced ZDC elections…
• Etc.
23 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
INSTEAD…
Let’s use MedEvac to assist with troubleshooting
• Using MedEvac on a workstation from the same Domain (with Admin privileges) we can begin
24 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• Due to the architecture of Load Balancing, we can query the XML Broker to determine the Least Loaded Server....and run some basic tests against the Least Loaded Server that is returned….
XML Broker
Least Loaded Server
Tests run against the
LLS
25 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
Now we can run further health checks against the Least Loaded Server that was returned…
• So now let us run a XML Ticketing Test, as we have already tested Terminal Services and RPC
26 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• XML TICKETING TEST
Least Loaded Server specified
Test results
XML Ticketing
Test specified
27 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• MedEvac identified the “Least Loaded Server” so we know which server the ZDC is routing connections to…
• So how do we verify the IMA service is functioning properly on the “Least Loaded Server?”
• By running the LLS test and specifying the Least Loaded Server in MedEvac, we are making the LLS use it’s IMA service to communicate to the Zone Data Collector to return itself as the Least Loaded Server…
• If this test returns success, we know that the IMA service on the Least Loaded Server is working properly…
28 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
LLS Test against the Least Loaded Server (IMA TEST)
FAILURE
29 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• In this scenario, the Least Loaded Server test failed when run against itself.
• So now we can then focus all troubleshooting towards that server’s IMA service…
• Investigation indicated that the IMA service on that server was “hung” since it could not be stopped or re-started
• Although IMA still responded to the Data Collector’s “heartbeat” messages, it could not accept any connections due to its “hung” state
30 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
RESOLUTION:
• Using PSKill from http://www.sysinternals.com we manually forced termination the IMA service
• Recreated the local host cache
• Restarted IMA - started successfully
• Users able to launch their published resources as normal now
31 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
ADDITIONAL INFORMATION:
• Detailed information on Citrix Medevac and its usage can be found at the following locations:
http://support.citrix.com/article/CTX107935
32 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
When can I use it?
• The Citrix Medevac Utility can be used anytime where isolating a problem Server is difficult (with respect to load balancing, XML, Terminal Services and RPC)
• To troubleshoot connectivity issues with published resources, such as the mysterious ‘black hole’ issue
• To run direct health checks against Citrix Presentation Servers
33 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
KNOWN LIMITATIONS
• Please see the ‘Readme’ and requirements posted on the Citrix Online Support Knowledgebase
http://support.citrix.com/article/CTX107935
• Recommended to register to the Citrix Developers Network for access to Tools and Utilities
Disclaimer• These software applications are provided to you as is with no representations, warranties or conditions of
any kind. You may use and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that (a) the software application may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the software application fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the software application. In no event should the code be used to support of ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SOFTWARE APPLICATION, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the code.
34 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Questions?
35 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.