TOPS AOS Handbook v1 0 (2)

Applications Operations and Support Handbook

Project Name: TOPS - Production Support Client Name: TOPS Markets, LLC Account

Document Revision History

Change Request# Document Version Approval Date Modified By Section, Page(s) and Text Revised Approved By

1.0 26-Feb-11 C.J. Vijeyprabakar

Final Chitra

Table of Contents

1. PURPOSE.................................................................................................................................................2

2. RESPONSIBILITY MATRIX......................................................................................................................2

3. APPLICATION CHANGE REQUEST APPROACH..................................................................................43.1 Change Request Standards and Procedure.....................................................................................43.2 Scope of Application Change Service...............................................................................................53.3 Testing Strategy................................................................................................................................53.4 Release Strategy...............................................................................................................................63.5 Deployment Procedure.....................................................................................................................73.6 Application Standards.......................................................................................................................8

4. MONITOR OPERATIONS APPROACH...................................................................................................84.1 Scope of Monitor Operations Service...............................................................................................84.2 Restart/Rerun Procedure..................................................................................................................94.3 Job Schedulers Procedure................................................................................................................94.4 Backup/Recovery Procedure............................................................................................................94.5 Equipment Maintenance Procedure................................................................................................104.6 Technical Standards and Procedures.............................................................................................104.6.1 Application Monitoring.....................................................................................................................104.6.2 On-Call Support procedure.............................................................................................................104.6.3 Handling Abends.............................................................................................................................114.6.4 Escalation Procedure......................................................................................................................114.6.5 Downtime Reporting........................................................................................................................124.6.6 Security...........................................................................................................................................124.6.7 Exceptions Handling.......................................................................................................................124.6.8 Environment Change......................................................................................................................124.6.9 Disaster Recovery Drill....................................................................................................................12

5. MANAGE APPLICATION TICKETS........................................................................................................125.1 Application Ticket Management Tool..............................................................................................125.2 Application Ticket Categorization Procedure..................................................................................145.3 Application Ticket Processing Procedure........................................................................................155.4 Scope of Application Tickets...........................................................................................................175.5 Communication & Escalation Matrix...............................................................................................17

6. PROJECT MANAGER STANDARDS AND PROCEDURES..................................................................176.1 User Procedures.............................................................................................................................176.2 Problem Reporting..........................................................................................................................186.3 Requesting Changes.......................................................................................................................186.4 Use of input forms...........................................................................................................................186.5 Making special requests..................................................................................................................18

MphasiS Internal Application Operations and Support Handbook 1

1. PURPOSE

The purpose of this Production Support Handbook is primarily to provide procedural guidelines and assist TOPS Production Support Team member to perform all necessary tasks in order to fulfill the objectives of providing Production Support, Maintenance & Enhancement and to provide below quality services to the client as agreed by MphasiS as HP’s supplier for the project on a time and expenses basis:.

Application Maintenance Application Production Support Production On-Call Enhancement Project Management Documentation

This document is an umbrella document containing links to the documents that make up the Production Support procedures and is intended for use by TOPS account team.

2. Responsibility Matrix

Role ResponsibilitiesDelivery Head Review project progress and suggest improvements where required.

Review risks and provide guidance where required to Delivery Manager to mitigate them

Delivery Manager

Stakeholder Management: ensures commitments are fulfilled by stakeholders Identifies and builds appropriate blend of resources to meet project/program

needs including sub-contractor selection During project execution oversees/initiates appropriate corrective/preventive

actions to ensure fulfillment of program/project objectives Responsible for ensuring communication across all stakeholders Develops Project managers and Leaders. Works with Project Managers and

Project Leaders to Create positive working environment for the team Determines, monitors & reviews project/program economics to include costs,

operational budgets, staffing requirements, sub-contractors, resources, and risk Supports fulfillment of requests related to project/Program proposals, bids,

contracts, solution architecture estimates, and schedules and adds value Supports the TOPS Account Team in developing Account/Program Strategies for

existing business to enhance delivery capabilities, and add-on opportunities with Best Shore inputs

Plans and provide inputs on capacity, resource, finance etc into organizational planning process and provide periodic inputs on resource utilization, forecasting for governing purposes

Initiates/participates in Project Mgmt reviews and ensures closure of findings Work with the TOPS Account team on the financials for the MphasiS part of the

team


Role ResponsibilitiesProject Manager

Performs project management activities to enable project success & client satisfaction

Reviews, establishes project scope and defines the acceptance criteria for project

Obtains stakeholder commitments & coordinate with them to ensure fulfillment Plans, schedules, monitors, escalates and reports on activities related to the

project, including sub-contractor monitoring Work with the TOPS Account PMO to develop/adhere to appropriate project

control and reporting procedures, to be used for planning, scheduling, and tracking projects through integration of various Project Management tools

Reviews and approves project estimates and obtains client approval Controls project requirements, scope, and change management issues Leads team members to accomplish project goals Conducts status review meetings for project and mgmt teams and clients Interacts with client on requirements, scope, estimates and deliverables Ensures timely reporting of status, escalation of issues for resolution Integrates, tailors, and uses methodologies and processes defined by the

organization / TOPS Account PMO to produce operational plans Initiate/participate in peer reviews of Work Management documents, and ensure

closure of the findings Assesses the risks on a continuous basis for their impact on delivery and

manages them proactively Drives Knowledge Management process within the team

Project Tech Leader

Manage the assigned module/application within budget with high quality Owns delivery of assigned modules/application and ensures delivery of only

tested and reviewed products Plan development of module Prepare module level schedules Prepare / Review estimates Define acceptance criteria for software work products Test strategy–plan & execute , set up environment , review test cases & results Review and approval of sub-ordinate tasks - plan and assign work in PS projects Evaluate alternate solutions to provide best fit Monitor, collect, analyze and control module specific metrics Escalation for timely resolution of issues with appropriate stakeholders Risks Management ( specific to the module) Work with Project Manager to obtain clarifications, resolutions on the work

component

Application Support Team Member

Provide support, maintain and enhance software as per project defined software process

Ensures traceability between requirements and LLD where applicable Performs coding, unit testing and related documentation as per project standards Applies testing concepts, tools and techniques to ensure product quality Works with appropriate stakeholders to resolve project technical issues Identifies issues and escalates if not resolved in time Application and adherence to project defined processes Contributes in peer reviews, conducts code reviews and adds value to process

and product Demonstrates understanding of estimation methodologies Continuously learns and improves knowledge on application / project


3. APPLICATION CHANGE REQUEST APPROACH

3.1 Change Request Standards and Procedure

Applications Enhancement and New Development work in response to a problem or a proposed environment or a proposed enhancement are handled as Change Requests/Service Requests. They are discussed at the Service Request Board (SRB) meeting and are assessed, approved, prioritized, and scheduled into a release.

Deliverables

Listed below are the deliverables to be provided for any Enhancement & Change Request:

Scope High Level Estimate Detailed Technical Design Detailed Level Estimate ( if there is a change in the initial estimate ) Milestone Dates Code & Unit Testing Results Work Product Reviews Implementation Plan Post Implementation Validation

Initiate Service Request

Anyone from TOPS can initiate a service request, by filling the Service Request form and submitting it to direct supervisor and/or manager for approval.

After direct supervisor / mangers approval the SR form has to be approved by TOPS Directors or Vice Presidents, and then by TOPS VP of IT Services. If the request is rejected by any one of the above it will be returned to the requestor with a proper explanation.

Once the SR is approved by the TOPS VP of IT Services it will be scheduled for presentation in the subsequent TOPS executive committee meeting.

At TOPS executive committee meeting, priority of the SR is evaluated and business justification will be approved and communication will be sent to announce SR approval and on next steps.

The completed and approved SR forms will be mailed to SRB coordinator. The SRB coordinator will record submitted service requests into the SR repository and will send out

a communication to Service Request Board (SRB) and HP/MphasiS Support teams. If HP/MphasiS team has a service request to submit it will be submitted directly to the SRB. A weekly review of submitted SRs will be conducted by the joint HP/MphasiS and TOPS SRB to:

o Review newly submitted SRso Prioritization of all SRs submitted across all functional areaso Assignment of SRs to resources for assessment (estimating)o SR status is set to “estimating”o Determine target dates

Evaluate Change Impact


Once the SR was approved and prioritized by the SRB, an HP/MphasiS resource will be defined as the overall “PM” for the SR who will be managing completion of the SR Assessment Phase.

Account PMO will communicate on the SR to the respective PM. The TOPS BSA is responsible for representing the TOPS Business and will include TOPS business

personnel and other TOPS stakeholders (e.g. vendors) appropriately. TOPS BSA along with the HP/Mphasis/their representative is responsible to complete a high level

assessment of the SR, attend weekly SRB meeting. The first step is to estimate the effort hours required to complete the SR assessment phase and

communicate to SRB via e-mail or in the weekly SRB meeting. After the estimation of SR assessment phase, the SR requestor and/or TOPS business contact

should define and document high level requirements on the “High Level Requirements” tab of the SR form. .

The assigned HP/MphasiS SR Assessment PM should schedule a formal review of high level requirements with the SR requestor and all stakeholder/points of contact for the SR to minimize the risk of delivering solutions.

Once the high level requirements have been approved they are officially baselined, therefore any changes submitted for high level requirements will require change control.

The HP/MphaisS SR Assessment PM will identify and lead necessary resources from all stakeholder groups to define and document Scope on the “Scope” tab of the SR form and High Level Estimate on the “ROM – High Level Estimate” of the SR form.

Then it has to be reviewed first by HP/MphasiS internal team and then by TOPS stakeholders – at a minimum the TOPS BSA. The review participants should be given 72 hours to prepare and review the meeting materials.

Obtain SRB Approval

The HP/MphasiS SR Assessment PM will submit the SR to the AE Staff for costing and presentation to the TOPS executives. The completed and reviewed SR ROM and costing will be presented by HP ADM/Architect to the TOPS VP of IT and to the TOPS Executive Committee for approval or rejection.

If the SR is approved in the SRB meeting it will go into an Active status which marks the beginning of a project.

Receive and Evaluate

Team responsible for processing service request should receive the SR Form and identify if the SR raised is complete in all sense with all required information.

The received SR is first analyzed/evaluated briefly. Evaluate the impact of SR on current CIs, in terms of volume and impact of change.

Plan and prepare for change from development through implementation, including the following:o Identification of configurable items to be deployedo Whether it is added, changed or deletedo Integration and interface considerations

Identify the applications solutions that best address the requirements identified and that conform to the application architecture.

Detailed Estimate


Based on the detailed Impact Analysis above, estimate should be derived for working on each impacted CI.

Analyze what work products (e.g. each component, module, deliverables etc.) are required to meet the estimating objectives, based on the requirements of the project.

Using the Size and Complexity based estimation approach defined in our account specific Estimation Guideline documents (available at \\Sgpfileclus01.sgp.hp.com\M_GR_TOPS_RTL_WEBTOOLS\02_Busn_Case_Relshp\Delivery_Blueprint\Estimate\Estimation Guidelines), arrive at the bottom up estimate by providing effort hours for each element of the Standard WBS defined in the ‘High Level Estimate’ tab of the SR Form.

Make adjustments based on the team’s experience level, complexity of the project like number of interfaces, inter dependencies, external dependency et al.

Document any assumptions you used to complete the estimates. Estimation should be revisited whenever the requirements and assumptions made during the

estimation changes. Get the estimate reviewed and approved first by HP/MphasiS internal team and then by TOPS

stakeholders – at a minimum the TOPS BSA.

Design the Changes

Design the behavior and relationships among the components of the application, Design user-interface formats, media, and behavior; design the application’s interface with other applications and with the technical environment.

Depending on individual requirement for the CR, solution should be proposed for meeting each requirement.

Any known limitations in satisfying a particular requirement should also be recorded. Impacted Stakeholders should sign off the Design. Ensure that deployment, release and implementation assumptions, constraints, dependencies and

parameters are considered during the design of the CR.

Plan for verification:

Distribute the phase’s major deliverables to the lead technologist, giving adequate review time.

Validation of the change: If the change is an Application Change, for each listed requirement, develop the test cases to verify

if requirements are met, once the changes are done.o Ensure test cases include all possible scenarios while testing a requirement for being met.

For all other changes, plan for any verification and validation of the change to ensure that the change when implemented meets the requirements of the change.

Build the Changes

Develop or acquire the application components according to the specified design and any applicable standards.

Make the required changes on the impacted CIs, after checking them out from the Configuration Library.

Ensure they are also marked “Checked Out” in the CMDB, to help addressing other CRs impacting a specific CI from this CR.

Verify the Components Impacted


Check the changed CIs against developed test cases to verify and validate if all requirements are met.

Log all defects, and track the same to closure. Obtain approval from stakeholders on the CIs, before planning for their release.

Update Support Documentation

If applicable, update all Supporting Documentation and User Manuals related to impacted CIs, to align to the changes made.

Package Release Components

Logically package all applicable components of Release, like Impacted CIs, Support Documentation etc. , validate that the package is complete and meets the applicable detailed requirements.

Using the configuration management and measurement data, identify the status of individual configuration items that are part of the release. Build the release from the appropriate components and store it in a controlled environment.

Validate that the release package is complete and meets TOPS’ requirements. Obtain commitment from the organization and baseline the system in the release repository according to the configuration management plan.

Move the system to the distribution environment, as needed. Create the release documentation (Software Release Note) with the details of this release.

Communicate to TOPS that the release is ready for implementation in production.

Coordinate Formal Acceptance Testing:

If the change is an Application Change, coordinate formal acceptance testing by the End Users. For all other types of changes ensure end users or those impacted by the change validate and

agree to the changes made prior to deployment.

Establish Production Environment

Determine and establish the production environment for the release.

Deploy in Production Environment

Deploy the Application on Production Environment or make the CIs available in the Production Environment according to the implementation plans.

For Non Application Changes:o For CIs added as part of the CR. Make new entries available in the Production environment

and CMDBo For CIs modified as part of the CR: Replace old versions with the new ones in the

production environment and update the CMDBo For CIs Deleted as part of the CR: Remove CIs from the Production environment, Archive

them as per applicable policy and update the CMDB

Perform Deployment Testing

Work with the other stakeholders to validate that key application functions perform appropriately in the production environment.

Assist in obtaining appropriate sign-off, as required.

Post Deployment Support

Communicate the release of the Application/CIs under the CR to the impacted stakeholders Support the user base during implementation of the Application or new CIs


Collect feedback from the users, and raise required Change Requests if any Collect and store post-release defects discovered.

Close the Change Request

After the predefined period of Deployment support is over, close the CR in the tool/CR Log Maintain all records of the closed CR for future reference.

3.2 Scope of Application Change Service

#. Service Request Assignment Steps Responsible1 Approve or reject a SR TOPS Executive Committee

2 Present ROM and Cost Card to the TOPS Executive Committee for approval or rejection

TOPS VP of IT

3 Communicate to the SRB coordinator on whether or not the SR has been approved or rejected

TOPS VP of IT

4 Log and document decisions and submit a request to the HP delivery managers for resource assignments for approved active projects

SRB Coordinator

5 Communicate all SR decisions to all stakeholders associated to the SR

SRB Coordinator

6 Initiate Project Start Up for approved SR projectAssign Project Manager

HP Delivery Manager

7 Assign Project Resources Assigned SR Project Manager

8 Announce Project Assigned SR Project Manager

9 Execute Project as per Application Enhancement and New Development Procedure

Assigned SR Project Manager

10 Report on Project in Weekly TOPS Programme Status Meeting

PMOAssigned SR Project Manager

11 Closedown Project following the TOPS Closedown procedure (includes invoicing)

PMO/ADM/ Assigned SR Project Manager

3.3 Testing StrategyTesting evaluate an application, system or a module against the given requirements. The purpose of the Testing is to define the processes and the activities which include to gather the test requirements and decide on test approach, design test, execute and report test results. The end result of the testing would be to deliver the qualified testing product.

Production support testing is performed when fixes to defects are delivered to the production support environment. The objective of production support testing is to ensure that changes made to an existing build/release of an application have not resulted in adverse affects. Production support tests are executed in the environment that contains a copy of the database identical in structure to the production database.

Out Testing Strategy covers the standard types of testing required to verify and validate the changes. Testing strategies that we follow are:


Define Test Requirementso Elicit the test requirements, document, and analyze

Unit Testingo Create the Unit Test cases in the test design and report templateo Review all the test cases with the functional area SME using respective Work Product

Criteriao Log the review comments and rework to close the commentso Execute Unit Testing, compare actual results to expected results and resolve any

discrepancieso If any of the actual results do not agree with defined expected results, track the defects

identifiedo Resolve the defects identified and re-test the defects as appropriateo Collect, analyze, and store pre-release defects discovered during testing

Integration Testingo Create the System Integration Test cases in the test design and report templateo Review all the test cases with the functional area SME using respective Work Product

Criteriao Log the review comments and rework to close the commentso Integrate the System components and conduct System Integration testing for the changeo Compare actual results to expected results and resolve any discrepancieso If any of the actual results do not agree with defined expected results, track the defects

identifiedo Resolve the defects identified and re-test the defects as appropriateo Collect, analyze, and store pre-release defects discovered during testing

UAT Testing

Production Implementationo Create the Implementation Plan and get the approval from client.

Above test strategies may be tweaked depending on the business requirement/change request.

Below listed are the steps followed during Testing in TOPs:

We will follow the standard SDLC model for all the SR’sMainframe:We will raise a CAB in TRB site, which has to pass through two weeks CAB Meeting which normally happens on Wednesday, they will discuss abort the component, which is moving in production.

Package Execution will be taken care by MM/SC team and it has to be approved by the two approvers (Managers/Lead) to move into production.

Steps followed in moving the code to production Retrieve the elements from the ENDEVOR to WIP with checkout option “Y” Change the code accordingly to requirement in WIP data set Add the element to ENDEVOR regiond “T” Do the code review by unit testing and once done with unit testing with UTR move the elements

from region “T” to “Q” region System Testing done in “Q” region and create the STR with review Please follow the below steps to move the element into production:

o Create the package ido Castingo First level reviewer will approve the packageo Second level person can EXECUTE the packageo Ensure that the elements has been moved into production after the 2nd person approved

the packageWe will raise a CAB in TRB site, which has to pass through two weeks CAB Meeting which normally happens on Wednesday, they will discuss abort the component, which is moving in production.



Once the implemented components is successfully executed in production, we will inform the BSA’s for formal closure. Based on the BSA’s Comments, we will formally close the CAB.

Web:

Download existing code for that application from CVS (Head) Create Dev branch if it is not exist in CVS Change the code accordingly to requirement in local machine Put the changed code in Development Server Internal Review Check in to CVS (Dev) and update the new code in Dev branch then Commit and check out After completion of Unit Testing. The application will be moved to UAT Implementation in Production Server (TRB – Open, Close) Check in to CVS (Head) and transfer the code from Dev to Head branch

MM-Midrange:

Need to copy the code from Production source where changes need to be done Take the backup of the code from Dev and paste the copied source code from production Make the appropriate change to the code Get it reviewed by offshore team Testing need to be done in dev environment after the DB sync Send the test result and get the approval from the business/onsite Create TRB and take the backup of existing code from production and move the new code Monitor for the first run and send mail across the change went success

Retail: Login to Dimension Create a Change Document Check out the existing script Change the code accordingly to requirement in dev machine Internal code review Testing need to be done at any of lab store Get the approval from Team Lead Check in the update code in Dimension Create a Baseline Create a Release Create distribution package Create an IMAC request to distribution team (TOPP_SOFTWARE_DIST_TIVOLI) Distribution team will distribute the update code to Prod Stores Validate the distribution Verify the first run after the distirbutiondistribution

[3.4] Release Strategy

This process contains activities to:


Install all components of the development effort in the production environment Test installed applications

Following table describes activities in the Release System process:

Step Release System Activities Release System Description / Tasks1 Establish Production

EnvironmentAssist in establishing the production environment according to the implementation plans and the technical architecture specifications, including all aspects of the environment related to the application’s security posture. This effort may involve coordination with vendors, other HP/MphasiS site personnel, and TOPS’ production site personnel.

2 Install Application Data Install the application data and data structures into the production environment according to the implementation plans. Allocate production files and databases, convert data from the current system using the conversion components created, and load data into the new production files and databases. After thoroughly verifying the data, as directed by the implementation plans, back up the application data for reloading after installing and testing the application. Document the results of the database installation.

3 Install Application Software If the production environment differs from the distribution environment, migrate the application software components to the production environment according to application design specifications and implementation plans. Migrate program source components, with executable and other components, into the controlled production environment according to established promotion procedures. If the application is being installed for the first time, verify the application software and data structures and back them up to provide a baseline image of the application.

4 Install Business Organization Components

As needed, assist in installing any workplace, workspace, documentation, organization, and training components according to business organization designs and the conversion strategies.

5 Perform Release Testing Work with the other stakeholders to validate that key application functions perform appropriately in the production environment. As needed, perform any test cases called for by the test plans. Assist in obtaining appropriate sign-off, as required.

6 Obtain Release Commitment After all portions of the system are installed and verified in production, work with the other HP/MphasiS stakeholders and suppliers to confirm that the installation was successful. Ensure that all aspects of the implementation have been completed as planned. Based on the installation results, jointly decide either to go with the installed system or to execute the contingency considerations of the implementation plan.

7 Conduct Release Training Conduct training in accordance with the training plan. Training may be conducted using a separate environment and scheduled in parallel with other implementation activities.

8 Monitor Production Application Observe the system to determine whether functions operate as TOPS expects and meet expected performance levels. Collect and store post-release defects discovered during the warranty period. Baseline appropriate measurement data.

9 Turn Over Release Following the production support turnover plan, turn the application over to the support staff who will be responsible for ongoing support. Communicate known errors and workarounds to the people who will support the application. Ensure that the application is performing


acceptably and address any concerns among the support staff.

3.4[3.5] Deployment Procedure

Mainframe

We will raise a CAB in TRB site, which has to pass through two weeks CAB Meeting which normally happens on Wednesday, they will discuss abort the component, which is moving in production.


Once the implemented components is successfully executed in production, we will inform the BSA’s for formal closure. Based on the BSA’s Comments, we will formally close the CAB.

Midrange

The deployment procedure Includes packaging the application or solution for delivery, performing formal acceptance testing, gaining final approval, and making the package available for release. Below table describes the activities in the deployment system process:

Step Deploy System Activities Deploy System Description / Tasks1 Package Release Components Using the configuration-management and measurement data, identify the

status of individual configuration items that are part of the release. Build the release from the appropriate components and store it in a controlled environment. Update system documentation as appropriate.

2 Harvest Reusable Assets Identify the project artifacts that are eligible for reuse (for example, objects, components, and supporting documentation). Identify the artifacts that reduce the time required to develop software, improve software quality, reduce the learning curve for new technologies, or reduce the risk of project failure and that can be leveraged for similar development efforts. Package the assets for engagement consumption.

3 Compare Results with Approach

Evaluate the iteration goals against iteration results. Document iteration shortcomings where goals were not met, for incorporation into future iterations. After reviewing iteration results, gather and evaluate the iteration metrics. Where possible, use metrics-evaluation results to influence future iterations, to identify additional reuse opportunities, and to improve the design. Capture other lessons learned from this iteration to improve later iterations.

Verify that work products (deliverables) created before this point in the project are updated based on the iteration results. Review the Configuration Status Accounting Report to verify that all configuration items are listed.

4 Perform Cross Capability Integration

Integrate the system components developed by all capabilities into a seamless system. Ensure proper coordination across capabilities. As needed, work with other stakeholders and suppliers to integrate the application assets with other solution components (such as hardware, software, and documentation) into a package that can be tested and


released into production.5 Perform Programme and

Enterprise Level TestingWork to plan for and conduct program and/or enterprise testing for the application(s). Execute Consolidated Integration Testing based on client requirements and development-team needs.

Ensure that the test environment is ready. Load base data and parameters as needed.

Execute each test sequencing as defined in the test plan. Ensure that instructions/scripts contained in the test case are followed exactly. Record test results as each test case is executed. Capture all relevant information for investigating any identified defect. Ensure that version numbers of all components being tested are recorded.

Compare test results against previously defined expected results. Evaluate results to determine success. Document discrepancies for analysis. Track all problems discovered to resolution. If applicable, collect, analyze, and store prerelease defects discovered during testing. Retest as appropriate.

Secure completed components and update documentation according to configuration management plan.

6 Conduct Technical Deploy Review

Distribute the phase's major deliverables to the lead technologist, giving adequate review time. Determine whether performing a self-assessment or facilitating an assessment with the lead technologist is necessary. Complete the technical review checklist. Work with the lead technologist to determine next steps where findings indicate that additional work is needed.

7 Obtain Deploy Commitment Validate that the release package is complete and meets TOPS’ requirements. Obtain commitment from the organization, and baseline the system in the release repository according to the configuration management plan.

8 Coordinate Formal Acceptance Testing

Coordinate formal acceptance testing using the approved formal acceptance testing specifications and the test strategy. Record and retain testing results. Track all problems discovered to resolution. If applicable, collect, analyze, and store prerelease defects discovered during formal acceptance testing. Retest as appropriate.

9 Make System Available Using the configuration management data, ensure that the proper system configuration is built. Collect final size metrics. Move the system to the distribution environment, as needed. Create the release documentation as needed. Communicate to TOPS that the release is ready for implementation in production.

[3.6] Application Standards

Design the application processes needed to support each application logic function. Design the interfaces between components, with other applications, with the user, and with the environment. Document the schedule for all normal and special-request executions of application components. Create specifications for system documentation. Refer to the project's reuse plan to identify reusable assets and to design assets with reuse in mind. Design any necessary changes to associated business roles or job functions. Document any necessary changes to business tasks and procedures to provide optimal value to TOPS’ business. Maintain forward and backward traceability between detailed requirements and design components.

Package the design documentation to include all applicable design components. Validate that the package is complete and meets the applicable detailed requirements. Obtain commitment from the organization and TOPS. Secure and baseline the design components in accordance with the configuration management plan.

Below attached spreadsheets has the details of naming conventions used for naming configuration items:

Mainframe (Finance / HR Payroll Applications):


Midrange (Retail Applications)

4. MONITOR OPERATIONS APPROACH

4.1 Scope of Monitor Operations ServiceN/A

4.2 Restart / Rerun Procedure

The following procedures apply to all jobs that can be restarted within the daily batch.• For problems that had occurred earlier• Investigate the problem and check restart/rerun history log for available immediate fixes• For problems that have not occurred earlier

1. Investigate the problem2. If modification of JCL and/or dataset is required, refer the TOPS Temporary / Emergency

fix procedures3. Before deleting any files, ensure that the original files is backed-up4. Un-catalog / delete any files generated by the abended program / step if necessary5. Reload the necessary files6. Verify that all the necessary input files are available7. Restart / Rerun the job

Record the details in the restart / rerun history log

4.3 Job Schedulers ProcedureAll development and production support teams will use the Job Request System (JRS) to request for temporary or permanent changes to the Batch Job Schedules for Mainframe and Midrange.

For emergency changes, the support teams will raise a JRS request and call the 24x7 Batch Scheduling Support team for immediate assistance. The contact numbers for the Batch Scheduling teams are listed in the URL for the JRS application listed below.

The URL for the JRS application is https://www.ehbatchscheduling-eds.com/main.asp

Refer https://www.ehbatchscheduling-eds.com/online/searchpg.htm to access the Online Help for using the JRS application.

4.4 Backup/Recovery ProcedureBackup:

Backups are automated and scheduled to run to tape via Legato Networker. Backups not controlled by Legato are the Oracle backups which are run from the client side via rman and controlled by the database administrators. Backups are run each night beginning at 18:00 with incrementals each day and full backups performed on a weekly (Friday), monthly and yearly basis.

Retention:


https://www.ehbatchscheduling-eds.com/online/searchpg.htm

https://www.ehbatchscheduling-eds.com/main.asp

Offsite tape storage will support TOPS’ business processes that require copies stored offsite for protection. Backups are run to onsite tapes (LTO4) and then cloned (copied) to another tape which is labeled offsite and sent to Iron Mountain in Charlotte. The offsite storage is on a rolling scheduling are as described in the below table:

Recovery:

Data can be recovered from a backup to resolve a production incident or perform investigation or analysis in the test region for providing a permanent fix to an existing problem.

The following steps will be followed to restore files from previous backups:1) Determine which file(s) need to be restored2) Check if necessary disk space is available3) Submit a request to load the necessary tapes / other backup media4) Create / Modify required JCLs / programs / scripts to restore the data from the backup media5) Submit the job(s) after confirmation that the backup media have been loaded6) Recovery process owner (requestor) should confirm that the restore jobs have completed

successfully

If any production files / data need to be replaced to resolve a production incident, the following additional steps will be followed:

1) Perform thorough analysis on the problem to be resolved2) Determine which production files / data need to be restored from backup3) Obtain approval from Production Support Manager4) Perform the recovery steps mentioned above in test region to validate that the restore will

fix the problem 5) Notify the actions to be taken to the TOPS BSA and obtain approval6) Proceed with recovery steps in production

4.5 Equipment Maintenance ProcedureN/A

4.6 Technical Standards and Procedures

4.6.1 Application Monitoring

N/A

4.6.2 On-Call Support procedureProduction On-call is a service to respond to EON’s and Severity Level 1, 2, 3 and 4 service Incidents during support coverage hours.

Offshore team support their respective applications from 8.30 PM – 10.00 AM EST

Onshore team support the respective applications from 10:00 AM – 8:30 PM EST


DATA CATEGORY RETENTION

Database 15 Days

OS 30 Days

Weekly Fulls 90 Days

Monthly Fulls 360 Days

Yearly Fulls 7 Years

If there is any issue, team will escalate the issue to appropriate BSA/Senior ManagementAOS\TOPS Escalation Package v30.xlsx - Escalation PackageAOS\TOPS Buisness Notification List v2 Oct 2009.xls - Business Escalation Contact List

4.6.3 Handling Abends

EON stands for EDS On-call Notification. EON is an automation tool that enables electronic messages to be sent to specific individuals when events occur on monitored devices in the environment. It is also the corporate direction for manual messaging and automated notification tool for system alerts globally. Electronic messages can be in the form of e-mails or text messages to wireless devices using protocols such as TAP/IXO, SMS, HTTP, etc.

Escalations are messages sent to individuals that require a response from the recipient. Escalations can only be sent to devices that guarantee delivery, such as pagers or cell phones. E-mail cannot be used as an escalation in EON.

An escalation is sent to only one person at a time, each person who receives the message has a set amount of time to respond before the next person on the escalation chain is sent the message. When an individual receives a message from EON as an escalation, they must respond to the message by either calling the EON server that sent the message using the phone number included in the message, or by using the Web Interface. In either case they enter the event number from the message to stop the escalation from counting.

When an individual does not respond to EON in the specified amount to time, the escalation will continue to the next person on the escalation chain, until the last person in the escalation chain has been sent the escalatedion, at which point if no one has responded to the escalation, the event is sent to the Operations Console for manual intervention.

If an individual is defined as a notification for an escalation chain, that individual will receive a page EVERY time an action is taken on an event. Whether this action was generated based on someone extending the event escalation to resolve the event, or when some responds to EON to close the event, notifications will be sent in each situation.

Automated monitoring alerts are collected through the leveraged monitoring infrastructure. These alerts are received from monitored devices, servers, applications and network devices that have monitoring tools enabled to watch the application or hardware for failures or other events that require human intervention. Alerts from these environments are passed to EON for escalation/notification purposes.

We have a constraint in EON page to view only the 7 days worth of data.

HP/MphasiS Production Support Analyst will perform initial research of the problem escalated by EON to determine the business impact, a resolution to restore service and if additional help is needed who else needs to be involved in resolution. And will update the EON accordingly by selecting the event number of the event, and updating the appropriate status as given below:

Acknowledged/Extended – Escalation contact is aware of the issue and resolution is in process – requires selection of “repair time”Closed – Resolved – The event has been resolvedClosed – Invalid – The event was considered invalid and should not have been processedClosed – Misrouted – The event was escalated to an incorrect escalation list – requires secondary action from respondent to ensure correct escalation team is engaged to address the original issue.

The responsibilities of the Applications Support teams are listed below:1. Acknowledge the escalation event in EON2. Proceed with the TOPS Incident Management procedures until the Abend has been resolved3. Update the status of the event in EON


4. In the event of any critical event not being resolved or having business impact, alert through the Escalation matrix for support in resolution and / or communication of impact to the business users in a timely fashion

5. With the help of batch monitoring team restart jobs / job streams as required to resolve the incident

4.6.4 Escalation Procedure

On-Call Escalation Process and procedures are maintained and documented in the TOPS Applications Escalation matrix.

http://teams3.sharepoint.hp.com/teams/1605/Shared/Forms/AllItems.aspx?RootFolder=http%3a%2f%2fteams3%2esharepoint%2ehp%2ecom%2fteams%2f1605%2fShared%2fTOPS%20Escalation%20Package&FolderCTID=0x01200053649A71F199C241BD02229EF9AEA4DB

4.6.5 Downtime Reporting

Application availability report is provided to business on monthly basisAppropriate BSA will be intimated about the downtime of application.

4.6.6 Security

Security process is in place, whenever the new resources are inducted in project team. Appropriate security forms will be forwarded to McMaster, Angela E/ Bustos Argañaras, Carlos to get the access for DW, Mainframe, EON & Citrix applications. Similar forms sent to the same team to revoke the access of above applications.

4.6.7 Exceptions Handling

N/AOperation report. Communication to TOPS BSA

4.6.8 Environment Change

N/A

4.6.9 Disaster Recovery Drill

Disaster Recovery process is in place for all the application. Please refer the below path Z:\04_System\MM\DR-docs

5. MANAGE APPLICATION TICKETS

5.1.1.1 Application Ticket Management Tool

The application problem may be identified by TOPS, HP, or a Third Party Vendor. The problem may include items such as software problems, configuration problems, or infrastructure problems. The problem identified will be processed as Application Incident through two possible channels:

Corporate users contacting the HP Pune Help Desk Store users contacting the TOPS Retial Help Desk

The HP Helpdesk or TOPS Retail Helpdesk will capture all pertinent information that will be needed to research the problem and enter an incident ticket in DW System (Application Ticket Management Tool).


Z:\\04_System\\MM\\DR-docs\\ADRPS_SCMM.eds.zip

Z:\\04_System\\MM\\DR-docs\\ADRPS_SCMM.eds.zip

http://teams3.sharepoint.hp.com/teams/1605/Shared/Forms/AllItems.aspx?RootFolder=http%3A%2F%2Fteams3.sharepoint.hp.com%2Fteams%2F1605%2FShared%2FTOPS%20Escalation%20Package&FolderCTID=0x01200053649A71F199C241BD02229EF9AEA4DB



The severity will also be assigned by the HP Helpdesk based on the criteria provided by the end user or TOPS Helpdesk as defined within the DW or Remedy system.

Digital Workflow

Digital Workflow (DW) is a global initiative aligning global activities to identify, develop, integrate and deploy standardized end-to-end business processes supported by a common toolset. Service Center (SC) is the common toolset.

The following processes are in-scope for TOPS:

REQUEST: This process is the direct interface for all client-initiated requests for service and/or status. It receives, initially approves, determines the required action process for fulfillment, provides status as needed, and routes all Client Requests appropriately.

IMAC: IMAC Management process is used to request an install, move, add, change or delete of predefined and pre-approved products and services into a Service Delivery Environment.

CHANGE: This process is initiated when an IMAC Management, Asset/Configuration Management, Problem Management or Incident Management requires a change action. This action may be for approval to proceed with activities that can impact the environment, scheduling of a time period (change window) to perform required work, or planning and testing of specific activities to determine the impact to the computing infrastructure.

INCIDENT: To restore normal service when an event occurs that is not part of the standard operation of a service and that causes, or may cause, an interruption to, or a reduction in, the quality of that service.

PROBLEM: The Problem Management process is used to perform Root Cause Analysis to identify Corrective Action required for resolving and preventing recurrence of service interruption of degradation, and to determine solution propagation. This is initiated through the Incident Management process and/or proactive actions.

Service Center

Service Center (SC) is the common toolset. All Service Center records will follow a specific format:xxx-yy-zzzz (where xxx = Instance, yy = Record Type, zzz = Unique Record Identifier)

Instance:100 – Plano Instance

Record Type:01: Call Record02: Incident Record03: Change Record04: Problem Record05: IMAC06: Known Error Record

When you sign in, you will be on the Main Menu Screen. From there, you click the ICON titled Incident, Call and Change Queues to enter the Queue Lists. The Incident Queue is the Default. You can switch views and queues by either selecting from the current inbox drop-down or clicking the appropriate queue along the top. The Default Queue Lists when you click the ICON are:

Incident – Open Incidents by Assignment Groups Problem – All Open Problems Known Error – All Known Errors Change – Changes assigned to you


IMAC – IMAC’s assigned to you IMAC Line Items – Line Items by Assignment Groups

Any records matching the description of the queue will be displayed. Double click to open. If you need to find records not displayed in this queue, click the search link (on the left) to enter the Search Screen, where you enter your search attributes to find your record.

5.2 Application Ticket Categorization Procedure

SEVERITY LEVEL

DEFINITION ESCALATION / CONTACT TARGET

RESTORATION TARGET

Severity 1 Critical Impact - Complete loss of a critical business function already in production; no reasonable workaround exists. Business-critical applications are applications where loss of the system/application leads to unrecoverable consequences to the business.

1 Hour 4 hours

Severity 2 Major Impact - An Incident causing a significant interruption or degradation of service delivery to the affected client, environment or business operation. Partial loss of critical business function already in

production and/ or significant degradation of ability to provide service to the customer.

Problems with any application which are important to a client’s business or operations and which make the application unusable or unavailable.

Problems for which a workaround exists but requires extensive effort.

4 Hours 8 hours

Severity 3 Moderate Impact – An Incident causing a degradation or loss of non-critical business functions already in production. Users can continue operating with the results being adequate to perform needed functionality (although the process or format may be less than desirable).

Problems for which a workaround exists with minimal effort.

Problems which degrade system functionality or business performance; but major functions of the application still work.

Problems affecting a single user – preventing completion of a critical task.

2 Business Days

2 Business Days

Severity 4 Minor Impact - An Incident causing a degradation or loss of production functionality that affects individuals or small workgroups with minimal impact:

Problems that prevent completion of a non-critical task but NOT impacting other aspects of the user’s workstation.

Problems which do not degrade system functionality (tolerable or deferrable problems); major functions of the application still work.

5 Business Days

5 Business Days

Severity 5 No Current Impact - An Incident that does not affect normal service delivery for a client, environment or business operation. Includes issues with the potential to cause impact if not proactively addressed, and those that began as a higher severity due to Potential Impact, but were resolved prior to causing Actual Impact.

None None


5.3 Application Ticket Processing Procedure

The Application Ticket Processing Procedure for TOPS is unique in there is two possible channels or entry points for TOPS end-users to contact HP/MphasiS:

Corporate users contacting the HP Pune Help Desk Store users contacting the TOPS Retail Help Desk

User who reports the problem will capture all pertinent information that will be needed to research the problem. The HP Helpdesk or TOPS Retail Helpdesk will enter an incident ticket using Digital Workflow. CITRIX Tool is used to access the Digital Workflow from remote locations. The severity will be assigned by the HP Helpdesk or TOPS Retail Helpdesk as defined within the DW or Remedy system. The HP/MphasiS Production Support Analyst will be notified of the problem and given the initial incident details.

The HP/MphasiS Production Support Analyst will accept the DW ticket and determine if this ticket has been assigned to the appropriate group. If not, HP/MphasiS Production Support Analyst will reassign the ticket or contact the HP Helpdesk to reassign the ticket.

If a user reported the problem, HP/MphasiS Production Support Analyst will contact the end user to let them know work has begun on the ticket and to obtain any additional information they may need to research the identified problem.

HP/MphasiS Production Support Analyst will perform initial research of the identified problem to determine the business impact, a resolution to restore service and if additional help is needed who else needs to be involved in resolution. Support Analyst will also refer to the KEDB (Knows Error Database) for any similar known errors resolved in the past, before deciding on the resolution.

Note: As the team performs research and learns more about the business impact in this step and the following steps they will re-evaluate the severity level of the incident. This may result in an upgrade or downgrade of a severity level.

HP/MphasiS Production Support Analyst will notify appropriate parties as needed based on the business impact and severity level. HP/MphasiS will identify and involve those who will be needed to restore service. This can include other HP/MphasiS team members, TOPS or Third Party personnel. If leveraged ITO support is required, a DW ticket will be sent to notify the on-call support teams.

If the severity of the problem is high, a conference line should be used to have continual status updates and communication with TOPS. The HP/MphasiS Project Manager will communicate status, severity of the problem, and the estimated time of when the problem will be resolved to TOPS, the HP/MphasiS Delivery Manager(s), and the HP/MphasiS Account Executive, if appropriate. Also, agree to the update status intervals with TOPS. As required a problem will be escalated. Refer to the HP/MphasiS Escalation Matrix listed below.

If the problem is Third Party Vendor related, the Third Party Vendor will be contacted to research the problem to determine a resolution to restore service. Refer to the Vendor Contact list.

HP/MphasiS Production Support Analyst working with required support teams will determine possible resolutions to restore service. In certain situations, these will be jointly determined with TOPS or presented to TOPS and approval to move forward will be obtained. HP/MphasiS will take into consideration the size and impact of the resolution as a basis for criteria on when to involve TOPS and obtain approval.


User Support Process – HP ApplicationsH

P H

elp

Des k

T

O

PS

Cor

po r

ate

Use

rs

TO

PS

Ret

ail H

elp

Desk

HP

A ppli

cati

on

sT

OP

S St

ore

Use

rs

Open a Remedy ticket

Call Retail Help Desk (716) 635-5555 (option 1)

Close ticket

Document incident/request and assign ticket to appropriate Applications Team

Monitor Digital Workflow Queue

Enough Info in ticket to proceed?

Yes

No

Resolve Incident / Complete Request

Confirm resolution and ticket can be

closed

Provide additional

information

Call HP Help Desk

(Pune)(716) 635-5555 (option 2) or

1-866-593-7845Or Send email to TOPS PUNE MBX

Open a Digital Workflow (DW )

ticket

Close ticket

Document incident/request and assign ticket

to appropriate Apps Team per instructions in

the Knowledge Base

Interface from Remedy to DW

Respond to User

Severity 1 or 2?

Link to Incident Mgmt

Process

Yes

No

Validate and Research

Severity 1 or 2?

Link to Incident

Mgmt Process

NoYes

Initiated in Remedy?

Yes

Email Notification (Open, Update, Close)

Page Notification for Sev1 & Sev2 Incidents

Known Error Database

HP/MphasiS Production Support Analyst will make changes to restore normal service. HP/MphasiS Production Support Analyst will update the DW Ticket. Once production processing has resumed, HP/MphasiS Production Support Analyst will notify the HP Helpdesk or TOPS Retail Help Desk that the problem is resolved.

If at anytime the Resolution Time on the DW Ticket is exceeded, the problem will be escalated.

If the problem is Third Party Vendor related, the Third Party Vendor will provide the solution for an identified problem. Vendor may release a patch for the given problem or may release a new maintenance version.

For user opened tickets, the HP/MphasiS Production Support Analyst, HP Help Desk or TOPS Retail Help Desk will verify with the user that the problem has been resolved. The DW ticket will be updated documenting that the user has verified the restoration of service.

Once service has been restored, HP/MphasiS Production Support Analyst will perform a root cause analysis on the problem. The results of this root cause analysis will be documented in DW along with any pertinent information to aid in helping with future problems.

Problem ticket (root cause analysis) associated with Sev1 and select Sev2 incidents will also follow the TOPS Account 5-Phase process. Completing the “5 Phase Problem Resolution Template.doc” is a requirement of this process.

Based on root cause analysis a service request may need to be entered by HP/MphasiS Project Manager for the permanent resolution. Permanent Fix Service Requests are only to address the work for permanent


fixes after production processing has resumed. The problem ticket may incur significant effort and significant code changes to correct the root cause of the processing error.

The HP/MphasiS Production Support Analyst will close the DW Ticket.

Listed below are the deliverables to be provided relate to Incident Management / Production Support: Impact Statements for critical tickets Root Cause Analysis for Priority 1 and Priority 2 tickets. Permanent Fixes

5.4 Scope of Application Tickets

S. No. Application Name Activity Responsible

1. TOPS Production Support User reported problem – identify problem

TOPS Retail Help DeskHP Help Desk

2. TOPS Production Support Perform Initial Research Production Support Analyst3. TOPS Production Support Reassign Ticket Production Support Analyst

HP Help Desk4. TOPS Production Support Contact end user to notify

work has begun and obtain more information

Production Support Analyst

5. TOPS Production Support Involve appropriate support teams and clients


6. TOPS Production Support Perform necessary research and fix

Production Support AnalystThird Party Vendor

7. TOPS Production Support Determine options to restore normal service


8. TOPS Production Support Restore normal service Production Support AnalystThird Party Vendor

9. TOPS Production Support Verify and document restoration of service

Production Support AnalystHP HelpdeskTOPS Retail Help Des

10. TOPS Production Support Perform Root Cause Analysis and document problem resolution


11. TOPS Production Support Open Service Request for permanent fix

Project Manager

12. TOPS Production Support Close Ticket Production Support Analyst

5.5 Communication & Escalation MatrixRefer to the Escalation Procedure define in section 4.6.4.

6. PROJECT MANAGER STANDARDS AND PROCEDURES

6.1 User Procedures

Work Area

Client visits necessitates the need to keep our immediate work areas in a clean and tidy state and portray a professional, efficient and highly organized image of our team.

Having a tidy and organized desk makes it easier if you are absent for a colleague to find any information on your desk that may be needed, e.g. manuals:


Personal workstation areas should be kept as tidy as possible Valuable items should not be left on desks overnight Care should be taken when temporary cables are laid across floor spaces and if necessary tape the

cable to the floor to lessen the risk of someone tripping over the cable.

Personal workstations should be tidied at the end of your working day, as clients sometimes visit after hours and someone may require the use of your workstation.

Expenses

Refer to the current Policies and Procedures documentation on the MphasiS Intranet.

Travel and Accommodation

Refer to the current policies and procedures documentation on the MphasiS Intranet.

6.2 Problem Reporting<<Mention the procedure on how problems, incidents and issues would be addressed and reported >>

6.3 Requesting Changes<<Mention the procedure on how to request for any change identified in the application, which could be an improvement>>

6.4 Use of input forms<<Mention the procedure on how to update input forms (other than which is already in use), if any, mandated by the client>>

6.5 Making special requests<<Mention the procedure on how to make a special requests, if required>>


TOPS AOS Handbook v1 0 (2)

Documents

Transcript of TOPS AOS Handbook v1 0 (2)