SCHOPS NOC Standard Operating...

- - :

Mo

nito

rin

g P

roce

ss

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential

For internal use only. SCHOPS Network Operations Center

Version 1.1 of 23 Date of Issue: 03/01/2012

SCHOPS NOC Standard Operating Procedures

The Basics

Title: SCHOPS Network Operations Center (NOC) Standard Operating Procedures – Formerly: SCHOPS Code of Conduct

Classification: Confidential -ZNA

Status: Final

Written by: R. Larson

Document Owner: SCHOPS NOC

QA Coordinator D. Langland

Initial Creation 04/15/2008

QA Approval 03/01/2012

- - :

Mo

nito

rin

g P

roce

ss

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



Introduction:

This document describes the details and processes for SCHOPS Network Operations Center (NOC) Standard Operating Procedures.

Scope:

This document targets the SCHOPS Network Operations Center (NOC).

Purpose:

To provide supporting documentation regarding Monitoring, Reporting, Escalation, and Ticket Creation.

- - :

Mo

nito

rin

g P

roce

ss

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



Contents

1. Monitoring Process 4 2. Monitoring & Escalation Process Flowchart ....................................................... 6

3. Escalation Procedures 7

4. Ticket Creation 9

5. Tech Call Procedure 12 6. Reporting High Severity Problems .................................................................... 15 8. Early Morning Validations ................................................................................. 19 9. Network Operations Center Activity Information.............................................. 19

This manual will address the following subjects:

Monitoring Process

Monitoring & Escalation Process Flowchart

Escalation Procedures

Ticket Creation

Tech Calls

Related documents

- - :

Mo

nito

rin

g P

roce

ss

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



1. Monitoring Process

Monitoring and Escalation Process Procedures

The SCHOPS Network Operations Center (NOC) monitors the following applications. AS400 ARCserve Early Morning Validations Cameras.in Server Rooms and Secure Areas (screen 1) TSM (screen 2) WhatsUpGold (screen 3) Tivoli (screen 4) HP Openview Map and Alert screen (screens 5&6) Sitescan (screen 7) Weather Map (screen 8) News (screen 9) BMC (screen 10)

The BMC, Tivoli, Sitescan, and WhatsUpGold applications can create an auto-ticket. Schaumburg NOC Monitor Screens:

Monitoring Procedure

- - :

Mo

nito

rin

g P

roce

ss

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



When SCHOPS views an application event, the operator creates a ticket and contacts the specified support group to make the on-call technician aware. The support group has the ticket in the queue for the tech to respond to and report on until the resolution of the ticket. If the event involves one of the Business Critical Applications (Platinum applications), further escalation processes will occur. At this point SCHOPS management will be notified and the situation management group will be contacted as well. A Tech Call is initiated and the minutes created. Additional personnel will be involved if needed to resolve the event. The initial page is sent out. Situation Management (SM) will oversee an escalated event until concluded. Update pages are sent every hour until the resolution, then a final page will be sent out. New Additions to Procedures The EPP Platform SQL Bulk Screening Process has been added to the list of Early Morning Validated items. Each shift will be monitoring ManageNow approximately twice an hour.

- - :

Mo

nito

rin

g P

roce

ss

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



2. Monitoring & Escalation Process Flowchart

- - :

Esca

latio

n P

roced

ure

s

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



3. Escalation Procedures

This section describes the procedure for NOC Escalations. When the support group calls SCHOPS with an issue, SCHOPS will create a ticket. When five or more users report an issue to the Help Desk, the Help Desk creates a ticket. These issues are all reported to SCHOPS. This begins the initial Escalation. The on call support for the specific support group is contacted. If the on-call support Technician is unable to resolve the issue or too much time is needed, then management is notified and, also, Situation Management is notified. At this point the Initial page and later the Update pages are released until the Issue is resolved and the final page is released. If the issue is unresolved, a TECH CALL is initiated. Meeting Minutes from the TECH CALL are recorded. The Situation Management will send e-mails to the standard list plus the application list until the matter is resolved. The Initial page and the Update pages are released and the Final page will report the solution. Notifying Management The Technical Support On-call will be notified first on an issue. The On-call tech will take ownership of finding a resolution. SCHOPS will determine the severity level using the tools within the paging system and then confirming this with the Situation Manager if needed. (We should always lean toward the higher severity.)

- - :

Esca

latio

n P

roced

ure

s

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



Times when the appropriate management needs to be notified: Technical Support On-call

Operations technical issues (i.e. Who should this alert be escalated to; This application will not restart; What do I do?, etc.)

If the issue is a known error, then operations staff notifies Management with any issues that there are questions on.

Initial escalation issues that could turn into reportable issues.

Operations Supervisor

Anytime Tech Support cannot be contacted.

Times when the Tech Support has questions on the reportability of an issue.

Severity 1 – 5 issues: This will be any issue that will be paged or reported.

Escalation issues that require a director or above to be notified.

If a Tech Call has been invoked.

Operations Manager

Anytime the Supervisor cannot be contacted.

If an issue becomes a Severity 1 – 2 issue.

If a Tech Call has been invoked.

- - :

Tic

ket C

rea

tion

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



4. Ticket Creation

There are several sources for ticket creation with currently used ticketing systems.

SCHOPS & Help Desk Ticket: Triggered by Users calling Help Desk

Ticket: Triggered by Support Group Calls

SCHOPS Event Ticket: Triggered by a Monitored event in SCHOPS Network Operations Center (NOC)

SCHOPS Auto Ticket: Triggered by an application that creates auto tickets

- - :

Tic

ket C

rea

tion

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



Sample Ticket:

- - :

Tic

ket C

rea

tion

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



PROBLEM Audit Trail - Continued

CREATED

OPENED

WORKING

RESTORE -

> UAT

RESTORE

FIX

APPROVED

RCA ->UAT

RCA FIX

APPROVED

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



5. Tech Call Procedure

The Schaumburg Network Operations Center (NOC) becomes aware of a problem in several ways.

Notification from the Help Desk

Notification from a Support group

Notification from an independent Application Support desk

Alerts from monitors

These notifications and the steps to be taken in each case are explained in detail in the first chapter of this documentation along with supporting documentation. After the Schaumburg Network Operations Center (NOC) has received notification of an issue and followed the appropriate steps listed in the documentation, it maybe determined that a Tech Call is needed. This decision will be based on several factors. A Tech Call is needed

When a Key application is affected.

When the ETA to resolution is unknown or is not imminent.

When the problem is a re-occurrence of a former issue.

When a sev3 outage has been going on for too long or getting worse.

When the correct Support group cannot be identified.

When there is apparent danger of the severity increasing to a higher severity due to elapsed time, larger number of users affected etc.

When it has been determined the issue is a sev1 or sev2.

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



How to start a Tech Call

1. Notify the Situation Management on call person, via Sametime. In your message, state the issue, ticket number, as well as the bridge line and access code.

2. Open up a bridge line for the call.

3. Turn on the recording. (*17*)

4. Take note of the recording number and send it in an email to Situation Management and carbon copy SCHOPS after the Tech Call page has been sent.

5. Build a Tech Call page and send it.*

* Remember to include the Availability and Recovery team, the application support group involved, and any other group that has been requested.

During a Tech Call

The Situation Manager might re-evaluate the severity based on information received.

Send hourly update pages to the same list* above until the issue has been resolved.

Remain in touch with the Situation Manager via Sametime.

During the call you may be asked to contact additional resources, do so when requested.

If a Sev1 occurs, VIP calls from the Situation Manager will be necessary to notify and keep Upper Management informed of the status.

When to keep a Tech Call going

When the service restoration calls for active monitoring by multiple support groups.

When an immediate action could be necessary.

When the outage is highly visible.

When it is a Sev 1 issue or Sev 2 issue.

When service is restored, but the root cause is not known and can be investigated with the members on the call.

Per the request of Technical Support or the Situation Manager.

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



Notify the Management team. Refer to the ISP Phone / On-call database

Note: NOC Escalation Handler will be responsible for opening up the VIP calls as the moderator only. The Situation Manager will oversee the VIP calls and provide the information to Upper Management. Let the Situation Manager know in advance what bridge Line will be opened for the VIP call, along with the code number. During the VIP call, the Situation Manager may leave the Tech Call and may have another member of their group join the Tech Call in their absence.

What is a Sev1?

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



An emergency condition, which prevents a significant number of users from using a system or application. A user is defined as any functional person or business group supported by the IT Organization. The purpose of a Sev1 Our Management team needs to be contacted for every Sev1 issue. To restore critical business functions, when services are interrupted. To minimize the impact to business functions and to business partners. To provide a predictable time for restoration of critical systems.

6. Reporting High Severity Problems A high severity (severity 1 or 2) is one that is assessed at a sum of value of 6 or higher on the global Impact Assessment scale. Such a problem is a major or critical outage, in terms of the number of users affected, the potential duration of the Sev, the impact to users, and the business sensitivity. See the Impact Assessment scale below for further information. Impact Assessment Scale

Sum of Values Assigned Severity

0 - 1 Sev 5 – Very Low

2 - 3 Sev 4 – Low

4 - 5 Sev 3 – Medium

6 - 7 Sev 2 – Major

8+ Sev 1 - Critical

The values of each area for the severity calculation are added to determine the initial severity.

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



As soon as a high severity is recognized, identified, or assessed by any IT or business user, that user should do the following:

1. Report the problem to the appropriate help desk.

a. All ZNA employees and Farmers/Foremost Management Company employees call the Global IT Service Desk:

i. ZNA 866-655-2422 ii. Farmers/Foremost 866-417-5441 iii. Escalation Line 866-781-3368

Note: Call the escalation line only when the problem is a high severity outage and only if the appropriate primary help desk number is not answered quickly.

b. Farmers/Foremost Exchange users call:

i. Farmers OKC Help Desk 877-224-2894 ii. Foremost Claims Help Desk 616-956-4000, option 2

2. When reporting the problem, include the following:

a. Description of problem b. When the problem started c. Which application(s) are affected d. Which business unit(s) are affected e. How many users are experiencing the same problem f. How the users are impacted (eg. slow response time, intermittent

problem, system unavailable) g. Business sensitivity of the outage.

3. Obtain the ticket number. 4. Confirm with the help desk analyst that s/he correctly assessed and understands

the severity and will escalate the problem immediately. When a problem ticket is created manually in ManageNow or Service Center by anyone other than a help desk employee as a high severity (severity 1 or 2), the creator of the ticket should also immediately call the appropriate help desk to inform them of the presence of the ticket and request that it be escalated immediately. When any support group receives a problem ticket that is lower severity than severity 2, and the technician subsequently assesses it at a higher severity, the technician should immediately call the appropriate help desk to inform them of the higher severity, and request that it be escalated immediately.

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



Guidelines for scoring severities:

Factor Value Value for Severity

Calculation

Number of Users affected

0 0

<1% (1-50) 1

1-10% (51-500) 2

10%+ (>500) 3

7. Outage Duration

No outage 0

<1 hour 1

1-4 hours 2

>4hours 3

Impact

None/Workaround 0

Reduced Performance/ response times 1

Intermittent 2

System unavailable/unable to work 3

Business Sensitivity

Minimal Impact 0 Medium

(eg internal teams/functions) 1

High Impact (eg Agents, ZFS Clients and Consumers,

business critical/sensitive timing) 2

Other

External customer impact/brand damage (payments, letters etc.)

3

Legal or regulatory compliance issues 8

Add for Sum of Values _______

Impact Assessment Scale

Sum of Values Assigned Severity

0 - 1 Sev 5 – Very Low

2 - 3 Sev 4 – Low

4 - 5 Sev 3 – Medium

6 - 7 Sev 2 – Major

8+ Sev 1 - Critical

The values of each area for the severity calculation are added to determine the initial severity.

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



START AN IMMEDIATE TECH CALL AFTER SITUATION MGMT NOTIFICATION ZNA/FARMERS/Foremost Key Applications

ZNA: C/OS / Ezsb (EPP Platform Applications)

ZNA: CDW

ZNA: EDW (Enterprise Data Warehouse)

ZNA: eZACCESS

ZNA: eZBUY

ZNA: Insure90

ZNA: ZIP (Canada)

ZNA: PMS (Canada)

ZNA: PRIDE

ZNA: ZORBA

ZNA: Surety Express

Farmers: eAgent: (APPS/FPPS)

Farmers: CRN

Farmers: CLS/eCLS

Farmers:EASYPAY

Farmers: LIFENET/LIFECOM

Farmers/ZNA: Lotus Notes

Farmers/ZNA: SAP

FOREMOST STAR

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



8. Early Morning Validations These begin validating at 5am – to be completed by 6am

List of 24 Early Morning Validated Platinum Applications:

Ariba BO (Business Objects) Canada PMS

CDW EDW EPP Platform

eZACCESS / ACCESS eZSB eWork / eJournal

Insure 90 My UUG eZTransition

RightFax Risk Access Report Pride

Risk Intelligence Report Risk Intelligence Express Claims

Risk Intelligence Document

Web Intelligence Web Insure Underwriting Reports

Zips Zorba ZDW

9. Network Operations Center Activity Information

For a list of activities by shift See PROD CC NOC Activities CHECKLIST V.93.doc 1. Data Center Checks – Data Center checks will be conducted at 02:00, 06:00,

10:00, 14:00, 18:00, and 22:00 Central daily. These checks include making sure there are no alarms on the Liebert units and Halon systems, no “hot” spots, no unauthorized personnel present, and anything that might be out of the norm for the Data Center. After the checks have been completed, the operator shall initial the check sheet located in the Data Center.PROD CC NOC Activities CHECKLIST V.93.doc

2. BMC – Screen 10 - Alerts that appear on the console need to have a ticket

created (unless auto generated), be escalated to the appropriate group for resolution, and logged into the turnover log (when turnover log is implemented). Follow up for resolution to the issue should be conducted until issue has been resolved.

PROD%20CC%20NOC%20Activities%20checklist%20v.93.doc



- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



3. What’s Up Gold – Alerts that appear on the console need to have a ticket created (unless auto generated), be escalated to the appropriate group for resolution, and logged into the turnover log/problem minutes report. Follow up for resolution to the issue should be conducted until issue has been resolved.

4. Site Scan – Alerts that appear on the console need to have a ticket created,

investigated to see if the problem is true, then escalated to the facilities group for resolution, and logged into the turnover log (when turnover log is implemented). Follow up for resolution to the issue should be conducted until issue has been resolved. SiteScan monitors environment in secure raised-floor areas for temperature, humidity, fire and water leaks. Sitescan, Monitors Schaumburg, OKC, LA and Toronto, Canada.

5. HP OpenView – Alerts that appear on the console need to have a ticket created

(unless auto generated), and be escalated to the appropriate group for resolution, and logged into the turnover log (as turnover log is implemented). Follow up for resolution to the issue should be conducted until issue has been resolved.

6. BrightStor ARCserve Backups – Runs nightly during the week and requires close

monitoring to ensure the successful process/completion of each backup. NOTES full back is processed via ARCserve along with files restores. Operator will monitor job queue status, activity log, restore processes, media pool, and associated device activity. ARCserve resides on server USZZNSCHNAP07 and USZZNSCHNAP08.

7. TSM Backups – Runs nightly and requires close monitoring to ensure the

successful process/completion of each backup. An operator will also monitor status of session activity, tape mount activity, utility log percentage, drive status/activity and storage factors. Additional process activities as reclamations, migrations, and expirations are also monitored. The Network Operations Center (NOC) monitors four TSM boxes, TSM1, TSM2, TSM-TORONTO, and TSM LA. Failures or disruptions to any and all process on TSM platforms are reported via Manage Now.

8. ManageNow – Problem Management software tool use to report and record

problems or issues. Problems related to IT hardware failure, failed application services, environmental disruptions causing impact to user services, etc.

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



9. Service Center (Peregrine) – A prior primary Problem Management tool used to report and record problems. The Network Operations Center (NOC) primarily uses the Service Center to record information on anyone that requires access to restricted Data Center areas and their nature of business.

10. AS400 – Primary hardware located in Canada, however, tape drives to back up

the system are located in Schaumburg. There are three systems that are monitored; production CE9001, development CE9004, and testing CE9003. An Operator checks for outstanding or pending messages that will impede processing.

Also monitored, would be backup activity/completion, production nightQ processing, storage percentage, and overall system status.

11. Storage Tek SN6000 – STK silo hardware. Operator monitors physical status of

online drives, tape summaries, event log, tape mounts/dismounts, error messages/codes, connections, and checks for appropriate functionality.

12. Security Cameras – Used to view restricted areas of the Data Center. Cameras

are located through out the Data Center 5 and 6 floors with a main viewing console within the Network Operations Center (NOC).

13. Halon Alarms – Indicates an environmental issue like smoke or fire that has

been detected within the Data Center. 14. Bulletin Boards – Bulletin boards should be updated weekly with the current Core

Application & ISP Scorecards, important procedures, contact information, and schedules.

15. Change Management Documents – A hard copy of the morning and evening

change report should be kept in the NOC for 30 days. Copies over 30 days can be destroyed.

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



List of Business Critical - Platinum - Applications:

Access EZSB COS Property Workstation RELAY (Reinsurance Layoff) Risk Access / EZ Link

Ariba EZUmbrella MVR Order Management System (Motor Vehicle Report)

Builders Risk CARDS (Corporate Agency Related Database System) works with PIMS

MyGateway

Care Center Doc Pro (Document Policy Issuance)

PMS (Policy Management System)

CDW (Claims Data Warehouse) DPN (Data Processing of Nevada)

Risk Intelligence

Cesar EDW (Enterprise Data Warehouse)

Risk Intelligence Express

Channel Z – Zurich NA Intranet EPP Platform RMS – Risk Browser System

CIID (Centralized Insurance Information Database)

EZACCESS SAM (System of Account Management)

CIM Reporter (Computer Integrated Manufacturing)

Global Zuess U PASS (User Profile and Security System)

Clearance and Registration LSI (Loss Sensitive Invoicing) Webi (Web Intelligence)

CWS (Claims WorkStation) Risk Intelligence ZDW

CMS (Customer Management System)

Insure 90 (I90) ZARC (Zurich American Registration & Clearance)

D&B (Dunn and Bradstreet) WTP (World Travel Protection) ZUESS (Zurich Executive Underwriting Support System)

DDC (Document Distribution Center)

MCSC Time and Billing - MVR Order Management System

ZORBA

DICE (Deductible Information Coverage Entry) works with Access

MEDATA bill review system ZPS

DMS works with PIMS (Producer Information Management System)

PRIDE (Policy Rate Issue Dynamic Environment) works with Citrix

- - :

Tech

Call P

roced

ure

PROD NOC OPERATIONS

PROCEDURE

Zurich Confidential



Related Documents are available for the following subjects:

1. Demand Management Requests – DMR.doc LINK

2. Early Morning Validation Template.doc LINK

3. Application Validation 2007.doc LINK

4. Sit Mgmt expectations of Ops Rev4.doc LINK

5. Tech Call information.doc LINK

6. Severity Codes and Escalations.doc LINK

7. Sev 1 and Sev 2 escalation.doc

8. ArcServe_Drivemapping_update.doc LINK

9. ArcServe Restart.doc LINK

10. Check Sync-Split process.doc LINK

11. Logging into ArcServe_update.doc LINK

12. Tape Drive Failure.doc LINK

13. Iron Mt. Authorization.doc LINK

14. Beta Backups.doc LINK

15. Failed Backup Procedures.doc LINK

16. Incomplete Backups.doc LINK

17. Procedure for Reporting High Severity Problems.doc

18. WUG_update.doc LINK

19. SQL Bulk Screening Procedure.doc LINK

20. Paging Standards.doc LINK

LINK

SCHOPS NOC Standard Operating...

Documents

Transcript of SCHOPS NOC Standard Operating...