RM6148  Quality  Assurance  and  Testing  for  IT  Systems  2  DPS    

Customer guidance document

Developed with the Home Office

Contents    

1.  Introduction    

2.  Key  facts  summary    

3.  What  services  are  available  

4.  Buying  options  

5.  Buyer  scenarios  

6.  Buying  process  

7.  Running  a  mini  competition  

8.  Using  the  order  and  joint  schedules  

9.  Case  studies  

10. Savings  

11. Templates  and  documentation  

12. Contact  details  

Introduction   1  

As the public sector uses and develops more online and digital services, there is an increasing need for continuous testing of these systems throughout the delivery lifecycle. The Quality Assurance & Testing for IT Systems 2 (QAT 2) DPS Agreement is an easy and flexible route to market for you to buy specialist, independent testing services. This agreement provides a regulated route to market for public sector buyers to buy specialist, independent assurance and testing services, throughout your IT project lifecycle.

Key  facts  summary   2  If you are a public sector buyer, our Quality Assurance and Testing (QAT) DPS agreement is the perfect solution to test and assure the quality of your systems and designs.

Here are some of the reasons why:

• This agreement was developed in collaboration with the Home Office for Public Sector departments (including the Home Office) who will use this agreement for their internal shared managed service for IT testing

• This agreement compliments our other digital agreements, such as Digital Outcomes and Specialists (DOS) and G-Cloud

• Simpler, quicker process – accessible for both SMEs and other suppliers seeking opportunities to provide services to the public sector

• Dynamic – Buyers can create bespoke specifications (within the overall DPS scope), competitions and contracts

• Filtering of Supplier offering - ensures Suppliers receive notifications of competitions that are relevant to their Service offering and that Buyers are able to easily identify potential Suppliers

• Simply fill out your Statement of Requirements and let the suppliers tell you how they’ll deliver the service

• Government specific Terms and conditions have been created based on an agile approach, using Public Sector Contracts

• The agreement is compliant with the Public Contracts Regulations 2015

• Government owned IPR, means that you can continue to develop yourself or with any third party, or share and reuse

What  services  are  available?   3    Services available under this agreement are split into 10 categories listed below. This enables specialist suppliers (including SMEs) to offer services to you through the central route to market.  

Service 1: QAT Specialists Service 2: QA & DevOps Service 3: Load & Performance Testing Service 4: QA & Testing Service 5: Infrastructure Testing Service 6: Operational Acceptance Testing Service 7: Strategic QA Consultancy Service 8: Accessibility QA and Testing Service 9: Security QA and Testing Service 10: QA Capability Development

Service 1: QAT Specialists   Test specialist professionals, bought ‘as a service’ and capable of delivering services across the full life-cycle of your project or design process Service 2: QA & DevOps To support cost effective continuous release methods within an agile or Dev-Ops environment Service 3: Load & Performance Testing To determine how systems perform against specific performance needs and comparative baselines, including under load and stress Service 4: QA & Testing To establish and manage an appropriate level of QA and testing in line with programme delivery plans, validation and verification of a system against specifications and requirements covering functional and non-functional aspects. Service 5: Infrastructure Testing To test the performance of system infrastructures, including network provisions and hosting across LAN, WAN and cloud infrastructures. Service 6: Operational Acceptance Testing This service focuses on testing and assuring the operational readiness of your system(s). This may include Service Readiness Testing, resilience and disaster recovery.

Service 7: Strategic QA Consultancy Strategic consultancy support across all aspects of the test and systems development life cycle. Service 8: Accessibility QA and Testing To assess how far a product or service is easy for its intended audience to use (includes users who access the service via a range of assistive technologies like screen readers, voice recognition and input devices). Service 9: Security QA and Testing To identify threats and measure potential vulnerabilities. The testing scope includes the whole system and not just the software involved. Much of the testing will be automated, supported by advanced exploratory testing and cyber-related defence and assessments. Service 10: QA Capability Development Support for all aspects of growing civil servant capability within the QA specialism including learning and development, recruitment, knowledge management and graduate/apprentice onboarding.

What is out of scope? Products and services out of scope include but are not limited to: • Any hardware and infrastructure • Hosting • Software or software licensing (other than any testing

tools) • Networks or connectivity services (the QA and

Testing services for networks and connectivity are in scope)

• Provision of interim staff or contractors – this is not a resource agreement

To know more about the agreement and services available, please read the RM6148 Buyer Needs document.

 Buyer  Scenarios   4  

Scenario 1:

An NHS trust is streamlining its approach to IT system delivery and wants to establish how testing practices can be improved across the end to end testing and release process. The trust wants a supplier to carry out a strategic review to look at opportunities to improve the efficiency and effectiveness of current testing practices.

Route:

Service 7: Strategic QA Consultancy

Filter Category Level 1, Service 7 suppliers can provide consultancy support and strategic reviews of current operations. The likely output is an options and recommendations report that sets out the rational for change, the associated costs and benefits and an outline implementation plan.

Scenario 2:

A local authority is implementing a new HR and Finance system, which is required to integrate with multiple other existing authority systems. They want to automate the testing as far as possible to improve the efficiency of the process but lack the skills to develop an automated approach within the existing team.

Route:

Service 1: QA & Testing Specialists

Filter Category Level 1, Service 1 suppliers can provide Test specialist professionals, specifically Test Engineers, to work with your test delivery to define and implement the automation approach as a service.

Scenario 3:

The Home Office is launching a series of web hosted services and needs to ensure that each is capable of performing during periods

of heavy demand from the public. Services that slow or crash when under heavy load will lead to significant criticism and jeopardise the delivery of mission critical Departmental objectives.

Route:

Service 3: Load and Performance Testing

Filter Category Level 1, Service 3 suppliers can plan and execute tests that will simulate performance of services under varying degrees load. This will provide assurance to the department that key services will continue to operate even at busy times. This testing service can potentially be completely outsourced depending on the security and technical limitations in place

Our  4  Buying  Options     5  There are four options for running your Mini Competition under this agreement. Which option is best for you depends upon your project timescales and the complexity or scale of your Statement of Requirements.

Depending on which route you choose, you can ask suppliers to provide:

‘Light Touch’ Brief project plan or supplier CV and their prices

‘Short Tender’ More in depth written question responses and their prices

‘Medium Tender’ Option 2 plus a face-to-face clarification session

‘Full Tender’ Option 2 plus a practical demonstration and scrutiny session

Buyers who want to use the Quality Assurance and Testing for IT

Systems Agreement have to follow a Mini Competition process.

This process allows buyers to open competition under the

framework.

Depending on which of ‘Our 4 Buying Options’ you decide to use,

suppliers are required to tell you how they plan to deliver your

requirement and how much they will charge to do so.

Before starting your mini competition process it is essential to

identify and define your user and business needs.

Make sure you:

• Get any spend approval you may require

• Outline your user and business needs in your Statement of

Requirements

• Decide on which of the 4 buying options you will use

• Complete the relevant ‘Your Offer’ template with what you

want to ask the suppliers

• Explain how you will evaluate and score the suppliers bids

inline with the award and evaluation criteria by clearly

describing your evaluation model, including criteria and their

relative importance

• Set your project timescales (allow suppliers time to prepare

and submit their bids), include precise details of the closing

time and date in your Statement of Requirements

You may find engaging with the suppliers on the agreement before

issuing a mini competition helpful. During this pre-market

engagement you can ask suppliers questions to help refine your

requirement; and also give an early insight into the opportunity. If

you do decide to carry out any pre-market engagement activities it

must be fair, transparent and open. You can engage by sending a

Request for Information (RFI) by email or by using the free CCS

eSourcing tool.

Before you run your Mini Competition, find out which suppliers

could meet your requirements using the Dynamic Purchasing

System (DPS) Marketplace. This DPS is organised into four

distinct categories: services, location, clearance and scalability.

The 5 steps below outline how to search suppliers, create a shortlist and carry out a mini-competition.

Buying  Process   6  1 Shortlisting suppliers

Before you run your Mini Competition, find out which suppliers could meet your requirements using the Dynamic Purchasing System (DPS) Marketplace. This DPS is organised into four distinct categories: services, location, clearance and scalability.

The steps below outline how to search suppliers, create a shortlist and carry out a mini-competition.

1. Register as a buyer on the Dynamic Purchasing System Marketplace. Note: although pages are headed ‘Supplier registration’ this is also where you can register as a buyer.

2. Navigate to the Quality Assurance and Testing for IT Systems 2 DPS under Technology and choose the "Access as a buyer" option and confirm your acceptance of the customer access agreement.

3. On your Manage your DPS Category Exports page you now have 2 options:

1. View appointed suppliers

2. Click on the link to create a new category export. This is how you filter the suppliers that can meet your requirements

4. When creating a new category use the filtering tool (select all that apply) to specify your requirements and create your supplier shortlist. Search results will show all suppliers who can meet all service requirements (the search works on an ‘AND’ basis not ‘OR’).

5. Save your filtered list of capable suppliers by clicking the Save Category option and give the shortlist a unique name specific to your competition. From your Manage your Category Exports page you can then export the list of suppliers and contact details (export is to an Excel spreadsheet)

6. Quit the DPS

7. Use the list to produce a shortlist of appointed suppliers to invite to a Further Competition. Invite the suppliers identified by the DPS to bid against your detailed specification, evaluation criteria and contract format (All eligible suppliers must be invited to bid).

1. The list is valid for only 2 working days, from the date of its creation, because new suppliers may apply to join at any point. If your Call for Competition excludes eligible suppliers it potentially breaches procurement regulations. You will need to get a fresh supplier list (steps 3.2 to 5) if you don’t use your list within 2 working days by clicking the Search Again option for your export.

2. You can use CCS’s eSourcing tool to run a Call for Competition. The tool is free to use for public sector buyers, or you can use your organisation's own procurement tool.

8. Your tender process should be proportionate to the value and complexity of your requirement. Suppliers will need a reasonable period of time to review and prepare a response. This includes allowing sufficient time for suppliers to ask clarification questions and consider your answers before the deadline for written responses.

9. Evaluate the responses

10. Request from winning supplier(s) any required evidence that they can meet your requirements. Set them a deadline for submitting evidence of 2 days.

11. If you’re satisfied with the evidence, award the contract

12. Where appropriate, notify of the award on Contracts Finder.

More information about filter categories can be found in the RM6148 Buyer Needs document and DPS Schedule 1 Specification.

2 Running a mini competition

Buyers who want to use the Quality Assurance and Testing for IT Systems 2 DPS Agreement have to follow a Mini Competition process. This process allows buyers to open competition under the agreement.

Individual Contracting Authorities requirements are sent to the registered suppliers who are then invited to submit a specific tender to fulfil these requirements. This helps to ensure that Contracting Authorities are able to thoroughly test the market to ensure the best quality of products or services are received at the most competitive rate.

All eligible suppliers based on the Service, Location, Clearance and Scalability will be invited to bid.

Depending on which of ‘Our 4 Buying Options’ you decide to use, suppliers are required to tell you how they plan to deliver your requirement and how much they will charge to do so.

You will invite suppliers by conducting an Order Procedure for its Deliverables in accordance with the Regulations and in detail in DPS Schedule 7 Order Procedure and Award Criteria.

If the Supplier’s bid has additional detail that you would like included in the contract, insert the Supplier’s bid in Order Schedule 4 (Order Tender)

Your tender documents (Statement of Requirements) will set out:

• the specifics of the service being procured • timescales for responding • the expected format of the response • how bids will be evaluated

There are four key elements covered in tenders:

• supplier proposed solution • examples of experience • personnel • price

Suppliers will submit prices for order contracts, against staff roles and the grade structure set out in Annex 1 of DPS Schedule 1.

The maximum price that can be tendered for any Supplier Staff member to fulfil a Role is £1000 per day (excluding VAT). DPS Pricing and order charges are explained in Schedule 3.

During this Mini Competition process you must keep an audit trail of any dialogue and communication with the potential suppliers.

This can be done using your own procurement system or the free CCS eSourcing tool, Jaggaer.

Where this DPS agreement can be used to meet a public sector requirement by running a further competition, CCS makes its eSourcing System available free of charge to those buyers.

Access is not provided to public sector organisations to conduct further competitions from non-CCS commercial agreements or for their wider procurement activities.

CCS eSourcing tool can be accessed via this link:

https://crowncommercialservice.bravosolution.co.uk/web/login.html

Guidance on how to use the eSourcing tool can be found here:

https://www.gov.uk/government/publications/esourcing-suite-guidance-for-customers

3 Clarification

While your Mini Competition is live and you are waiting for suppliers to bid, we recommend setting a clarification window. This gives potential suppliers with the opportunity to ask questions about your requirement.

All questions asked, and their responses, must be published to all potential suppliers.

4 Evaluation

During your bid evaluation, you must treat all suppliers equally and fairly using the most economically advantageous tender (MEAT) criteria. It is also important that you keep an audit trail to make sure that you have evaluated all bids in the same way you said you would in the evaluation criteria.

5 Award

Stages of the award process

Intent to Award

Issue ‘Intent to Award’ letter to successful supplier, and debrief letters to the unsuccessful bidders. These letters are not a formal commitment.

Standstill

Once you have identified the winning bid, we recommend you apply a standstill period - usually a minimum of 10 working days.

Award

After the standstill period, the Order Contract can be sent out to the winning supplier and the contract is officially awarded. You must tell all the suppliers of the outcome of the Mini Competition by email or letter. Unsuccessful suppliers will receive feedback detailing the relative advantages of the successful bid. Providing feedback is part of the EU procurement regulations; and it is also invaluable information for suppliers, which may help them improve their future bids.

You are now ready to award your contract. Engage with the successful supplier to develop your Order Contract and to begin the project. Publish to Contracts Finder Once you have awarded the contract, you should visit https://www.gov.uk/contracts-finder and publish your award

NB. Non-compliant buying will directly impact the legality and reputation of the DPS agreement and is strictly prohibited. Buyers who do not follow the correct buying processes will be at risk�of legal challenge, fines and claims for loss of earnings, and the terms of the framework may be null and void in these cases.

Award  criteria               7   DPS Order Schedule 7 sets out the relevant weightings for quality and price criteria, in the Order Award Criteria annex. Quality must be between 60-90% and price must be between 10-40%. You may select any combination of the following criteria to make up the relative weighting percentages:

• Technical Merit • Approach To Delivery Of The Services • Added Value / Innovation • Soft Skills & Cultural Fit • Implementation Timescales

You will have the flexibility of how you choose to assess those criteria. Written proposal or demonstration can be used to assess technical merit, for example. Experience and personnel can be used to assess soft skills and cultural fit. This will depend on your project timescales and the complexity or scale of your requirement.

Criteria Relative Weighting Percentage

A

Quality - Which may consist of any combination of the following criteria:

• Technical Merit ��� • Approach To Delivery Of ���The Services ��� • Added Value / Innovation ��� • Soft Skills & Cultural Fit ��� • Implementation ���Timescales ���

90-60%

B Price 10-40%

Scoring  criteria   8  

You have the flexibility to set out the scoring criteria for each mini-competition, to ensure a more meaningful evaluation process. There are two examples of scoring mechanisms set out below. Choose the scoring criteria that you feel provides a clear distinction between bidders’ total scores.

Example 1

Marks Criteria

3 The resources put forward by the supplier clearly demonstrate the supplier’s ability to fully address each element of the response guidance.

2 The resources put forward by the supplier clearly demonstrate the supplier’s ability to fully address two of the three elements of the response guidance.

1 The resources put forward by the supplier clearly demonstrate the supplier’s ability to fully address one of the three elements of the response guidance.

0 The resources put forward by the supplier were unable to address any of the three elements of the response guidance or were unable to give any answer to the question.

Example 2

Marks Criteria

0

Very Poor (does not meet any of the requirement)

The response is significantly below what would be expected because of one or all of the following:

• The response indicates a significant lack of understanding • The response fails to meet the requirement.

1

Poor (meets some of the requirement)

The response meets elements of the requirement but gives concern in a number of significant areas. There are reservations because of one or all of the following:

• There is at least one significant issue needing considerable attention • There is insufficient evidence to demonstrate competence or understanding • The response is light and unconvincing.

4

Fair (meets most of the requirement)

The response meets most of the requirements but there is at least one significant issue of concern, or several smaller issues. These would require some further clarification or attention later in the procurement process (except for final stage), and may arise through lack of demonstrated capability and/or appropriate evidence. The response therefore shows:

• Basic understanding of the requirements • Sufficient competence demonstrated through relevant evidence • Some areas of concern that require attention.

7

Good (meets the requirement)

The response broadly meets what is expected for the criteria. There are no significant areas of concern, although there may be limited minor issues that need further exploration or attention. The response therefore shows:

• Good understanding of the requirements • Sufficient competence demonstrated through relevant evidence • Some insight demonstrated into the relevant issues.

10

Very Good (Adds value by exceeding requirements)

The response exceeds the needs of the project but adds value and provides value for money by doing so. Leaves the project and its stakeholders in no doubt as to the capability and commitment to deliver what is required. The response therefore shows:

• Deep understanding of the requirements of the project • Considerable competence demonstrated through relevant evidence • Considerable insight into the relevant issues • The response is also likely to propose additional value in several respects above that

expected.

Terms  and  Conditions           9   The DPS access agreement is between Crown Commercial Service (CCS) and the Supplier and allows the Supplier to be considered for Order Contracts to supply deliverables in Service Filter Categories for quality assurance and testing services. The DPS contract is formed by:

• the DPS Appointment Form, including any DPS Special Terms (see summary below)

• Core Terms - which are formed with Public Sector Contracts

• a combination of DPS Order Schedules and Joint schedules

Each Order Contract:

• is a separate Contract from the DPS Contract • is between a Supplier and a Buyer • includes Core Terms, Schedules and any other changes

or items in the completed Order Form • survives the termination of the DPS Contract

Buyers may enter into a contract with a supplier for a period of their determining which may exceed the RM6148 QAT2 DPS Agreement should this agreement be terminated at any time. The flexibility of the contracting period allows buyers to determine appropriate contracting timelines required to enable the supplier to meet customer needs. The Order Contract structure is flexible so that it can cope with deals of a wide range of complexity. To learn more about using the Order and Joint Schedules, read Using the Order and Joint Schedules for Quality Assurance and Testing Order Contracts customer guidance document. Notwithstanding the fact that the Buyer has followed a procedure as set out above in Paragraph 2, the Supplier acknowledges and agrees that the Buyer shall be entitled at all times to decline to

make an award for its Deliverables and that nothing in this Contract shall oblige the Buyer to award any Order Contract.

Case  Studies                      10  Where a buyer has a good news story, we encourage buyers to share their experience of the great work they are doing and the savings and benefits we are helping our customers to achieve.

One way to do this is by writing a case study.

Here are some examples of CCS case studies to get you started.

If you would like to write a case study for Quality Assurance and Testing for IT Systems 2 DPS agreement, please contact us for a template and guidance at cloud_digital@crowncommercial.gov.uk.

Savings

Once you have completed your Mini Completion and awarded your Order Contract to the successful supplier, we would like to hear from you.

We have developed a benefits methodology to help us capture and report savings together. As a public sector buyer this helps demonstrate value for money and can help CCS to identify where savings are being driven and where we could do more.

If using the CCS free eSourcing tool Jaggaer, you are required to do some ‘house-keeping’, once you have completed your contract award. The system requires you to fill in very basic information about your organisation, the successful supplier and the contract awarded to enable CCS to monitor savings and the National Audit Office Auditable savings and reporting.

Contact  details   11    

 

 

Contact us

If you need general advice about using Crown Commercial Service or further information in relation to Quality Assurance and Testing services, please contact:

0345 410 2222

info@crowncommercial.gov.uk

www.crowncommercial.gov.uk

@gov_procurement