A Rationale Outlook on SAP Testing

- The Avengers Act

Software Testing Conference-2012

Authors: •Abdul Waseem Mujahid

•Monika Raj

Agenda

• Introduction

• Testing SAP- A different Outlook

• Getting the Act Right

• Case Study• Case Study

• Conclusion

• References

Introduction

� Are you testing an application which has been built by the client IT team

and is currently in SIT phase And your team has limited visibility to the

architecture or business workflows?

� Are you testing an SAP application bound with aggressive timelines and

resource crunch?

� Is your Testing project based out of a legacy system or is a separate � Is your Testing project based out of a legacy system or is a separate

entity altogether with an external /intercompany Development team?

� Does your testing team have limited SAP functional knowledge, but

being quality centric you are looking for a business driven testing?

Introduction

With global economic uncertainty, almost every institution is focusing on cost reduction.

Consolidations of the independent business applications and processes have become an integral

part of this strategy.

Testing SAP- A Different Outlook

• Market offers “out of the box” SAP solutions, but they still need to be customized, configured

and thoroughly tested to meet the specific business needs of one’s organization. This can be

achieved only when customization is performed in a right way and to do so one needs to

understand the different systems in place.

• Implementation/consolidation of different legacy systems is a challenge, and on top of that

performing multi facet testing and assuring that this was validated and tested is again a

challenge in itself.

• Testing a SAP solution requires a very thorough understanding of the system and its interaction

with the various modules and their independent workflows. Test Engineers would be effective

in implementing different test methodologies, planning and designing the test strategies,

sequencing and prioritizing the test cases and defining the entry and exit criteria.

• Engineers also help in validating and verifying the conversions, interfaces, and security

processes within the SAP applications and also ensure the correct dataflow – inbound/

outbound within the system.

Testing SAP- A Different Outlook

• A Test engineer can help in ensuring that the security of the system meets all requirements

and that certain tasks such as segregation of duties, role creation, etc are carried out correctly.

But testing SAP in itself requires understanding of the business to which it’s being customized.

That’s when the relevance of Subject Matter Expert (SME) comes into picture.

• Understanding what and how the system needs to be tested, sequencing of the tasks across

different SAP modules, creation of real time test data, and much more can be achieved by

collaborating the testing team with the subject matter experts.

• A Subject Matter expert can share his insights for a project implementing a software system as

he has special in depth knowledge of a business area which in turn helps the team to enhance

their understanding of a given legacy process.

• SME helps in defining the corresponding business needs, test requirements and software

functionality, thereby acts as a stakeholder for a business

team that has recognized expertise and authority for a given business process. Hence a SME

can act as a trainer, peer-reviewer, approver or knowledge sharer; they can also help in

designing the architecture of a system.

Getting the Act Right

• Test Engineers and the SME’s have expertise in their respective streams. When collaborating it

becomes imperative to blend this unique personality. But how this two teams should interact,

what exactly they’ll be expected to contribute are often nebulous concepts.

• The SME- Testers relationship can be easily obfuscated, muddled, or diminished if the project

does not take appropriate steps to ensure that each of them understands their responsibilities.

Getting the Act Right

Below are a few guidelines to help projects maximize the effectiveness and productivity

of Testers and SME’s.

1. Defining the Roles and Expectations

2. Administrative Assistance

3. Standardize Meetings

4. Peer Review Procedures

5. Documentation

6. Identifying a POC

7. SME/Tester Selection Criteria

Getting the Act Right

1. Defining the Roles and Expectations

The main objective is to ensure that the SME’s and the testing team have well-defined tasks,

deliverables, and due dates. In order to maximize the productivity, before arriving to an event, the

Teams should have had the opportunity to ask questions about their role to ensure that they

understand it clearly.

Having a wrong set of expectation could not only cost the project from deliveries perspective but

also bring down the team’s morale.also bring down the team’s morale.

Examples of tasks or deliverables for an SME :

• Review test cases for integration testing associated with other systems (Legacy).

• Help validate user requirements for the application.

• Conduct code walkthrough for accounts payable interface to legacy system.

• Review requirements traceability matrix, and ensure that requirements have coverage.

Getting the Act Right• Help refine and determine feasibility, correctness, and completeness of end-user’s

requirement.

• Provide input for the design, construction and prioritization of test cases dependent on

business scenarios, if any.

• Help answer questions associated with the design of the application, its features, and its

capabilities.

• Help the testing team with the valid test data.

• Validate executed test results.

Examples of tasks or deliverables from a Testing team :Examples of tasks or deliverables from a Testing team :

• Understand the requirements and come up with the test plan, test strategy.

• Creation of test cases and execution of the same.

• Determination of the testing methodology and identification of Test automation tools.

• To state and define the Metrics and QA standards for the project.

• Creation of requirements traceability matrix.

• Run and maintain automated test scripts

• Logging of the defects and verifying the defect fixes

• Preparation of Query log for all the clarifications/queries and ambiguous requirements related

to the functionality.

Getting the Act Right

2. Administrative Assistance

• Generally, SME’s do not align to a particular project or team but

act as supportive resources to the team that is implementing a

software system.

• Consequently, they do not have access to critical project

information and knowledge on test case templates, shared

drives, repositories, databases, and test and requirements

management tools.

• SMEs without access to this project-specific information are at a • SMEs without access to this project-specific information are at a

disadvantage and may be unable to provide significant

contributions to the project.

• Likewise, the testers may have limited or controlled access to the

SAP system (In terms of the transaction codes or access to the

master data), thereby restricting them to a conservative testing.

• The QA manager should create an on-board binder for the teams

to help them become familiar with the project’s objectives, goals,

QA standards, test procedures, and location of the testing

information.

Getting the Act Right

3. Standardize Meetings

• Team meetings being organized need to be standardized in terms of their adherence to a

specified and defined process. If each business process team is executing a meeting without

any defined process, this would likely create confusion for the team.

• To effectively absorb the skills and the expertise of the team, it is imperative for the meeting

facilitator to understand the role of the SME/Testers.

The SME’s are frequently part of the audience during the analysis and requirements gathering • The SME’s are frequently part of the audience during the analysis and requirements gathering

phases, they’ll need to know what contributions they’ll make to the meeting. The Testers

would play a major role in terms of the test execution, understanding the entry/exit criteria for

each phase would be a major determinant of how the execution phase gets carried out.

• The facilitator needs to understand how to best provide the means for the SME/Testers to be

effective.

• The QA manager should provide the guidelines and criteria for conducting consistent,

homogenous meetings.

Getting the Act Right

4. Peer Review Procedures

• SME’s are often involved with the refining of test requirements and test cases. But does the

SME understand what forms/templates will be used for providing peer reviews? And more

importantly, how does the project know that the SME has reviewed all the necessary material

and been given a chance to report their feedback?

• Peer review is an important arena which could enhance the team’s performance. The Testing

team can make an effective use of this opportunity to not only bring out quality test cases but

also understand the business perspective.also understand the business perspective.

• The QA manager should elicit feedback from the SME and establish recurring meetings that

ensure that all of the SME’s and testing team’s questions, concerns, and issues have been

answered.

• Furthermore, the QA manager should determine whether the Team met and fulfilled all their

roles, objectives, and expectations. For instance, if the SME was only able to review 70% of all

testable requirements, the QA manager may want to arrange an alternative so that peer

review happens for all of the testable requirements.

Getting the Act Right

5. Documentation

• To foster an environment of continuous process improvements; the QA manager should strive

to document all issues, suggestions, and problems that the Teams report.

• The Team may find that there were problems with workshops or with the preparation of

materials, be it the test data for test execution, or the knowledge transition from the SME’s to

the testing team. Likewise an SME may discover that the business area was not properly

represented or that they did not have proper training on a test management tool.

• No matter how trivial the suggestion or reported issue seems, the QA manager should identify

and document the problem and take corrective action. The objective is to eliminate the

problem or implement the suggestion and ensure that the team’s inputs are not ignored.

Getting the Act Right

6. Identifying a POC

• The QA manager can assign few Testers a point of contact SME. The POC can help the testers

with the creation of test cases and help in establishing the approach for test execution.

• Also, the POC can help the team understand the business workflows and the transaction codes

that could help in data mining and creation of valid test data.

• It can turn out to be a mutual knowledge sharing platform, where in the testers can help each

SME with the testing methodologies or the strategies being adopted, helping them understand

the test automation tools and the test management tool specially towards the reports/metrics

generation.

• This atmosphere would not only favor the project but also the individuals creating a win-win

situation.

Getting the Act Right

7. SME/Tester Selection Criteria

• Domain Expertise

The SME’s and the test engineers should be competent in their respective fields.

Does the test engineer have the required knowledge on the test management/automation tools?

Does the test engineer have the client related domain (like Finance, Health etc) experience?

Does the SME understand the business workflows and the functionality of the application being

built?

Does the SME have knowledge about the roles and duties associated with different user profiles in Does the SME have knowledge about the roles and duties associated with different user profiles in

SAP or how the application interacts with other modules/applications?

• Business Process Expertise

SME needs to be aware of the client’s business process and the industry standards, a test engineer

with awareness on the business workflow will be an added advantage.

Does the SME have knowledge of the existing legacy systems, and if so, can they provide test data

from the application?

Does the SME know the levels of approvals and notifications across different modules such as FICO -

AP, FA GL?

Getting the Act Right

• Methodology Expertise

If the project is adhering to a specific methodology; does the SME and the test engineers understand

this methodology?

Does the team understand the procedures for documenting peer reviews or in which automated test

tools the requirements and test cases are stored?

• Recognized Competence• Recognized Competence

The relevance of the Test engineers or even the SME with the technology, their past experiences on

similar client work can be of help to the project.

Is the SME recommended by her peers and managers based on years of experience?

What are the Test engineers/SME’s qualifications, and how will those qualifications help the project

implement or upgrade a new system?

Do they possess outstanding performance appraisals?

Getting the Act Right

• Independence

Is the SME under pressure to keep her team’s legacy systems?

Is SME willing to provide valuable input and insight to help implement a system that can phase out

the system that he/she currently supports?

Does the tester have the “Test to Break” attitude?

Do they have the liberty to perform ad-hoc tests or do they have the necessary support system in

terms of the accesses or the environment to do so?

• Availability• Availability

Are the SME’s and the test engineers available to attend workshops and meetings related to the design and

implementation of a system?

Will they be present for all necessary meetings during the analysis phase?

• AuthorityCan the SME approve and sign off on the test cases and the execution report being prepared by the

test engineers or do they need to get approval from superiors or seek advice from other groups or

teams?

Case Study

Client Information:

The client is a leading bank with global presence.

• This methodology of SME and Tester’s collaboration was adopted in one of the projects being

executed for the client. The project was a SAP workflow system that integrates and enhances

the bank's internal financial reporting systems.

• The enterprise resource planning (ERP) application, based on a scalable platform, provides a

comprehensive global operating system to improve efficiency and support through continued

growth.growth.

• The SAP ERP Financials solution included unique features to support the bank's long-term

financial systems requirements, including:

� Enabling a unified, global financial management process

� A flexible operating model to quickly respond to changing regulatory mandates and shifts

in the marketplace

� Incremental enhancement packages without comprehensive system upgrades

Case Study

Project Description & Business Challenge:

• As the client continued to grow in size, the complexity of Invoices workflow faced increasing demands in the number and types of invoices the system processes. This demand surfaced several process and system-driven limitations which made it difficult to proactively manage the cost, expenses and services to its clients.

• Experimental SME – Testers Test Management methodology was incorporated in implementing the project; and test execution delivery has been aligned with the delivery of new functionalities. Due to lack in availability of resources with the required skillset and knowledge of SAP, Functional SME’s from AP/FA and GL modules of FiCO were called for to assist the testing team.

• Additionally, mergers with other financial institutions brought in huge volumes of data from both • Additionally, mergers with other financial institutions brought in huge volumes of data from both the systems. The execution was validated with the inclusion of Systems Integration Testing.

• Project concentrated specifically on Invoice Management process and generation of the BI/ECC reports based on cash flows. Invoices flow through various interfaces like eLedger, manual entries and Ariba system. A middleware portal existed to handle the invoice management workflow before invoices were posted into ECC. The R4 release was a core test execution project for FA, AP and GL modules of SAP across multiple test cycles. The scope of integration testing was to validate the complete flow of data from bank’s legacy system to ECC and BI via CE/PI. Few of the workflow activities included –

– Creation of Assets, Deactivation of assets, transfer of assets through ECC.

– Upload of data using different templates and validating the same in reports.

– Generation of different BI reports and validation of generated data with ECC.

Case Study

Scope:

The scope of this assignment included SIT cycles 1 & 2 and re-engineering of test scripts where the

existing scripts were updated for future regression cycles and Systems Integration Tests for Cycles

1, 2 & 3.

Engagement Details:

• The SIT testing was to be done on SAP Net weaver version and the test management tools that

client opted is Quality Center. client opted is Quality Center.

• The SME’s provided the required KT to the testers and helped the team with the business

flows of AP/FA and GL modules.

• As the scope involved only Test Execution, Quality Center was extensively used to execute the

existing scripts and perform V&V (Verification & Validation).

• Some of the functionality that was in scope included BI Reporting, Invoice Uploads, Invoice

processing workflow like creating invoices, approval process etc. Also the system was validated

in ECC and BI and the same compared with ECC data.

Case Study

Results & Value Adds:

• After the kickoff, there was a small delay to start the SIT and during this interim period, the Testers and the SME’s invested in getting the cross trained on the AP, FA & GL sub-modules under FICO and Quality Center respectively.

• This helped the team with effective utilization of the time and also helped the project build skills to anticipate future contracts.

• The testers were also trained on SAP BPC10. This has played a key differentiating factor when the client was to decide between the current vendor and the competitors.

• As an upshot the contract for BPC10 R5 release testing was finally bagged by the same vendor based on the future planning and the quality which was delivered during R4 SIT cycles.

• BPC10 is a new release of SAP in the market as well as a new implementation to the client. Even for the vendor, it is a new competency that is being built up.

• With SME’s in place, planning of work allocation was done in such a way that the scripts completion timelines were accomplished much ahead of the actual dates. This was a major milestone achievement during all the SIT Cycles that were assigned.

• Quality was the primary perspective of the assignment. Based on the delivered quality of the reengineered scripts and test execution of SIT cycles 1 & 2, the client had awarded additional SIT Cycle3.

Case Study

Project Size & Delivery Model:

• The resources requirement demanded to have competencies from different Service lines and

there was a fusion of resources from Systems Integration and Asset Management System for

testing expertise along with SAP functional consultants who possess the core FICO functional

skills.

• The initial challenge was to get the functional and testing expertise sync with each other on

the understanding perspectives, testing process etc. But with the ongoing challenges that got

presented, the perfect blend of functional and testing expertise was leveraged and was visible

with the successful completion of SIT Cycles ahead of time.with the successful completion of SIT Cycles ahead of time.

Industry & Service Line: Finance / Systems Integration

Tools: SAP Net Weaver, HP – QC

The success trademark was – “To maintain Quality, Timelines and Healthy relationship

with the client”.

Conclusion

� Leveraging the SME’s and Tester’s knowledge to the best use of the project is a Quality centric

and cost effective technique. Proposal of this methodology is an attempt towards bringing in

more radical and rational testing when implementing an SAP solution.

� The above stated Graph gives a comparison between a regular project and a project executed

with SME’s inclusion in terms of Time, Cost and Quality. There is a significant increase in

Quality with reduction in time, however with slight variation in the cost factor.

Conclusion

� SME-Testers collaboration technique can be extended across different industry domains and

applications/platforms of testing.

� This methodology when implemented considering the earlier stated guidelines can help the

organization build eminence. Also Cross training of the resources with secondary skills would

help create a win-win situation for both the practitioners and the employees.

� The paper attempted to identify the Business limitations of test engineers and proposed a

solution to corner it. It also shared the live project experiences incurred upon adopting this solution to corner it. It also shared the live project experiences incurred upon adopting this

particular methodology.

References

This paper has drawn inspiration from:

• Maximizing the Effectiveness of a Subject Matter Expert, Jose Fajardo

• Testing SAP R/3- A Manger’s Step by Step Guide, J. Fajardo and E. Dustin, 2007

• Open ERP:A modern approach to integrated business management, Fabien Pinckaers and

Geoff Gardiner

Thank You