OnBase TroubleshootingGreg Wheeler, Customer Care Team Manager

What went wrong? Something is misconfigured Bad data causes recurring errors Your specific environment causes a strange problem

with the software A performance bottleneck causes errors Custom Code is failing You may have discovered a bug in the software

Documents stopped appearing in a Workflow queue“Problem with the environment”

Unity Client error/ performance troubleshooting“Problem with the software’s behavior”

(based on actual Customer Care Team solved cases)

Two sample scenarios

Scenario #1: Where are my files? Users report that a Workflow queue that normally contains

documents is empty

These Files are images that are released from another system

Just started this morning!

Uh, where are they?

?

Where do we start? Most problems are way too big—we need to make them

smaller and more manageable/testable

Data Collection We need to know what the user, application,

script or service was doing We need to try to differentiate between a

coding, data or environmental cause Can it be reproduced at will? What about in

another environment? What do the logs say?

Narrowing an issue down Is it a problem with thick client, Core Services, or both?

What servers can we take out of the equation?

What modules can we eliminate?

What functionality can we exclude?

Diskgroup Storage (SAN)

Database Server

Application Server

OnBase Thick Client(Workflow TImers)

Unity Client

OnBase Thick Client(DIP)

Diskgroup Storage (SAN)

Database Server

Application Server

OnBase Thick Client(Workflow TImers)

Unity Client

OnBase Thick Client(DIP)

HTTP

OD

BCSM

B/CI

FS

First Step: Is Workflow…working? Is it limited to this queue/lifecycle in Workflow, or are there

others affected?Diskgroup Storage

Database Server

Application Server

OnBase Thick Client(Workflow TImers)

OnBase Thick Client(DIP)

?

Thoughts We see timers are working – We see other documents

moving around

We can look at the timer machine to make sure THIS timer is still running

Let’s trace further back

Is DIP…you know, DIPping?

Diskgroup Storage

Database Server

Application Server

OnBase Thick Client(Workflow TImers)

OnBase Thick Client(DIP)

?

What to check? DIP is importing documents as a service on the DIP machine

To troubleshoot DIP, we would stop the service, then fire up a client with the SCHED switch

If you see this…it’s not OnBase

If you see this…there’s a problem

Where did it fail?

Sys – Verification Reports

Let’s look at the directory

ProcMon Demo

Process Monitor ProcMon can be used to see behind-the-scenes Windows

calls and access requests

http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

Solution

Issue #2: Unity Performance Users are experiencing slow client performance first thing

every morning

Workflow queues taking a long time to navigate between documents

Sometimes happens in custom queries, document retrieval

All users affected

Diskgroup Storage

Database Server

Application Server

OnBase Thick Client(Workflow TImers)

Unity Client

OnBase Thick Client(DIP)

HTTP

OD

BCSM

B/CI

FS

Diskgroup Storage

Database Server

Application ServerUnity Client

HTTP

OD

BCSM

B/CI

FS

Options Observe Unity directly on Application Server (still slow?)

What about testing the thick client? (Verbose)

Look at Application Server performance (perfmon…or resmon)

Look at connectivity to DB/Diskgroups (ping/WireShark)

Look at Diagnostics Console

Let’s run Unity from the AppServer

Diskgroup Storage

Database Server

Application Server

OD

BCSM

B/CI

FS

Sometimes Thick Client is valid…

Test the Database Choose a SELECT query and run it directly on the database

server.

Compare times to complete

Is the query coming back quickly enough?

Perfmon

Consider ResMon

Diagnostics Console - Service Tab

Change to

“True”

What can ResMon tell you?

Test Connectivity Application Server > Database

Application Server > Diskgroup

Diagnostics Console

“What went wrong?”

“What queries are running,

and how fast?”

“What is OnBase doing right now?”

Advice Beware the Database tab in Production

Consider logging to a file to review on another machine

Log the least amount of information needed to troubleshoot

The rest of the story Grabbed some queries from the Database tab from around

the same time

Ran them in SQL Server Management Studio

Slow there too!

Unoptimized indexes

The Smoking Gun

Your Issues!