Download - State of Collaboration - osc.state.ny.us · Vendor 3 FF Vendor 4P PPPPP Vendor 5 P F Vendor 6P PPPPP. 10/17/2019 7 19 • Issues Raised: 1. Vendor one (1) was evaluated by all three

10/17/2019

1

11

State of CollaborationAdvance Your RFP Knowledge

2

• Introduction

• Nine RFP Case Studies

• Q&A

• Conclusion

Agenda

3

CASE STUDY 1CASE STUDY 1

10/17/2019

2

4

• What is a Solicitation of Interest (SOI)?

• Process to determine whether there is any interest in a procurement

• A method of outreach to support a single source justification

• A vendor’s response does not create obligation or commitment of any kind

• A vendor’s response is neither a binding agreement nor an agency’s acceptance of offer.

Solicitation Of Interest (SOI)

5

• How should an SOI be used?

• Announce required services in the Contract Reporter for 15 business days

• Describe required activities

• Identify minimum qualifications


6

• Things to consider after an SOI has been conducted:

• Not all vendors that provide this service may respond to an SOI.

• SOI does not typically request cost information and cannot be used for evaluation and award.

• Vendors that do not respond to an SOI should not be excluded from participating any future formal procurement.

• Agency must still submit CRER to OSC and post advertise.


10/17/2019

3

7

• Agency Z requires specialized services that they believe can only be offered by Vendor X.

• Agency Z conducts an SOI to determine if there are other vendors that can also provide these services.

– The SOI identifies required activities and minimum qualifications.

Solicitation Of Interest (SOI) – Case Study

8

• Recommended:

• Only one vendor responded to the SOI and provided their qualifications.

• As a result of the SOI, Agency Z supported the single source nature of the services.

• Agency Z submitted a single source Contract Reporter Exemption Request (CRER) to OSC including:

– Justification of vendor selection

– Reasonableness of cost

Solicitation Of Interest (SOI) – Case Study A

9

• Recommended:

• Two vendors responded to the SOI and provided their qualifications.

• As a result of the SOI, Agency Z determined there are other potential vendors for these services and conducted a formal competitive procurement.

• Both vendors were included on the bidders list.

Solicitation Of Interest (SOI) – Case Study B

10/17/2019

4

10

• Not Recommended:

• Four vendors responded to the SOI and provided their qualifications.

• Agency Z evaluated each response and determined that Vendor 4 was the most qualified.

• Concerns with this process:

– SOI is not a procurement vehicle and should not be evaluated.

– SOI is not a formal procurement. When the potential for competition exists, the process shall not result in a contract.

[SFL§163.7 Method of Procurement]

Solicitation Of Interest (SOI) – Case Study C

11


12

• Research* indicates that suppliers add, on average, approximately 20% to most fixed price contracts because of uncertainty and risk caused by unclear specifications.

• The research also revealed that most public organizations consider their distributed specifications to be of a high quality and easy to comprehend.

* Pettijohn, Carole and Qiao, Yuhua (2000). Procuring Technology: Issues Faced by Public Organizations, Journal of Public Budgeting, Accounting, and Financial Management 12 (2): 441‐461

RFP Requirements

10/17/2019

5

13

• What are Mandatory Requirements or Minimum Qualifications?

Lowest required level of knowledge, ability or experience that a company as a whole, or any proposed staff, must possess to be considered for award of the RFP.

Mandatory Requirements

14

• How do you write GOOD Minimum Qualifications?

• What may be clear to you and your agency may not be clear to a potential vendor.

• Minimum qualifications should encompass specific skills and abilities that your agency feels are “non‐negotiable”.


15

• Measurable

• Pass/Fail

• Review by individual or team

• Make sure you NEED it

• Must be met without alteration

• Follow evaluation plan

• Match evaluation instrument


10/17/2019

6

16

Mandatory Requirements – Case Study

• Agency X issued an RFP for a Point of Sales System:

• Mandatory Requirement:

– Two (2) Pass/Fail Requirements

– Three Evaluators

• Technical Evaluation: 70 points

• Financial Evaluation: 30 points

17


• Does this vendor fail to meet Mandatory 1, Mandatory 2 or both?

Vendor A’s Proposal

1

The Bidder must have successfully managed a point of sales system, for more than five (5) consecutive years and within the past eight (8) years. The contract for this engagement must be included as part of meeting requirement 2.

2The Bidder must have managed a minimum of three (3) contracts, within the past eight (8) years. At least one of the contracts evidenced for this requirement must meet requirement 1.

Point of Sale System Contract with NYC – 6 years

1. Rensselaer – 4 years2. Albany – 3 years3. East Greenbush – 6 years

Evaluator 1

M 1 M 2

Vendor A ? ?

point of sale system

five (5) consecutive years within the past eight (8) years

three (3) contractswithin the past eight (8) years.

18

• Mandatory Requirement Evaluation Result:


Evaluator 1 Evaluator 2 Evaluator 3

M 1 M 2 M 1 M 2 M 1 M 2

Vendor 1 F F P F P P

Vendor 2 P P P P P P

Vendor 3 F F


Vendor 5 P F


10/17/2019

7

19

• Issues Raised:

1. Vendor one (1) was evaluated by all three (3) evaluators, which agrees with the evaluation plan. However, the pass/fail conclusions do not agree among evaluators.

2. Vendors three (3) and five (5) were evaluated by one evaluator. The evaluation process did not follow the evaluation plan.

3. The mandatory requirements were confusing and possibly too restrictive.


20


21

Technical Evaluation

• Technical Criteria:

• To get your RFP evaluation right, you have to consider:

– What you want to score in your RFP responses (desirables)

– How you’re going to score the vendor’s responses (type of scale, weight)

– Who is going to do the scoring (individuals or team)

10/17/2019

8

22

Technical Evaluation

• Helpful Hints:

• The RFP must present the criteria that will be used for the evaluation of proposals.

• Words such as “must,” “will” and “shall” will make a technical requirement a mandatory.

23

0Meets no aspects and/or expectations of the requirement or information is not present as required

1Meets few aspects and/or expectations of the equipment; shows major gaps in knowledge and understanding of material; substantial inaccuracies

3Meets most to all aspects and/or expectations of the requirement; shows understanding of key aspects and requirement

5Exceeds aspects and/or expectations of the requirement; highly detailed knowledge and understanding of the requirement

Technical Evaluation – Case Study

• Technical Evaluation – 70 points:

• 14 evaluation criteria

• Score Key, as provided to evaluators by Agency X:

24

• Non‐Recommended Sample of Technical Requirements:


# RequirementMandatory or Desirable

Score

1The system must be capable of sending a confirmation to the customer’s

designated preferred contact method. M

2The system is capable of offering alternate options to customers in the event that

the customer’s preference is unavailable. D

3 The system offers a mobile application(s) for authorized users and/or customers. D

4 Authorized users must be able to set and modify business rules. M

5 The system must have a process in place to ensure that profiles are unique. M

6 Contractor mustmanage customer logins and authentications. M

7 The system must allow for variations in fees based on business rules. M

10/17/2019

9

25


• Issues Raised:

• Each technical criteria using the word “must” makes the requirement a mandatory.

• Mandatories are Pass/Fail and should not be scored or included in the technical evaluation.

26

• Recommended Sample of Technical Requirements:


# Requirement Score

1The system is capable of sending a confirmation to the customer’s designated preferred

contact method.

2The system is capable of offering alternate options to customers in the event that the

customer’s preference is unavailable.

3 The system offers a mobile application(s) for authorized users and/or customers.

4 Authorized users are able to set and modify business rules.

5 The system has a process in place to ensure that profiles are unique.

6 Contractor manages customer logins and authentications.

7 The System allows for variations in Fees based on business rules.

27


10/17/2019

10

28

State Finance Law §163.7:

“Where the basis for award is the best value offer, the state agency shall document……the determination of the evaluation criteria, which whenever possible, shall be quantifiable, and the process to be used in the determination of best value and the manner in which the evaluation process and selection shall be conducted.”

Weighting Technical Evaluations

29

• A breakdown of values for cost vs technical criteria should be included as part of your RFP.

• Further breakdown of the Technical Criteria may be provided in the RFP.

• The cost weight should not be less than 20% of the entire value of the procurement.

• If normalization or perfecting will occur, it must be included as part of the evaluation instrument.

• Evaluation instrument must be determined prior to the receipt of proposals.

Weighting Technical Evaluations

30

Component Maximum Points

Company Experience and Program Structure 15

Proposed Approach 30

Expansion of Training and Consultation Services 10

Personal Experience 10

References 5

Total Technical Points 70

Cost Evaluation 30

Total Proposal Score 100

Weighting Technical Evaluations – Case Study

10/17/2019

11

31

Component Maximum Points

Company Experience and Program Structure 30

Proposed Approach 60

Expansion of Training and Consultation Services 15

Personal Experience 15

References 15

Total Technical Points 135

Cost Evaluation 30

Total Proposal Score 165


32

• Here is an example of how the Technical Score could be normalized:

– (Proposer’s Raw Points ÷Maximum Raw points) x Maximum weighted points = weighted technical score

– Examples: Vendor A: (119 ÷ 135) x 70 = 61.70

Vendor B: (121 ÷ 135) x 70 = 62.74

• Normalization process must be included in the evaluation instrument.


70% Technical Weight135 Maximum Raw Technical Points

Vendor A Vendor B

Proposer’s Raw Technical Points 119 121

Proposer’s Normalized Technical Points 61.70 62.74

33


• Total points were used for evaluation purposes.

• No longer 70% Technical vs. 30% Cost (state actual %s)

• Not in accordance with RFP and evaluation instrument.

Vendor Technical Score Cost Score Composite Score

A 119.00 30.00 149.00

B 121.00 28.50 149.50

C 125.00 10.50 135.50

D 101.00 21.25 122.25

10/17/2019

12

34

• The composite scores are no longer weighted at 70% Technical and 30% cost as indicated in the RFP.

• Agency award was based on the total of 165 points (135 Raw Technical points + 30 Cost points) which changed the Technical weight to 82%.

• The evaluation process omitted the step of normalizing the Technical score to agree with the weights specified in the RFP.

• Evaluation results in accordance with the RFP:

Evaluation Vendor A Vendor B Vendor C Vendor D

Technical (70%) 61.70 62.74 64.81 52.37

Cost (30%) 30.00 28.50 10.50 21.25

Composite 91.70 91.24 75.31 73.62


35


36

• Evaluation Process Consistent with RFP:

• Criteria being evaluated

• Maximum scores per criteria

• Evaluator instructions:

– Rating Scale

– Group discussion

– Shortlisting

– Interview/presentation

Documenting Technical Evaluation

10/17/2019

13

37

• Individual Evaluation Sheets:

• Identify the Vendor & Evaluator

• Signed & Dated

• Criteria with Maximum Allowable Points

• Rating Scale with Explanation of Point Value

• Adequate Spacing for Comments & Scores


38

• Individual evaluator comments should be:

• Legible

• Detailed

• Appropriate

• Relevant to Procurement Being Reviewed


39

Documenting Technical Evaluation ‐Case Study

Vendor Name:_____________ Evaluator Name:_____________

Rating Scale 1 2 3 4 5

Signature:_____________ Date:_____________

Evaluation Criteria Max Points Initial Score Comments

1a. Staffing Plan 5

1b. Key Personnel 5

1c. Availability 5

10/17/2019

14

40

Documenting Technical Evaluation ‐ Case Study

Example of Irrelevant Comments:


1a. Staffing Plan 5 1 Bad Handwriting

1b. Key Personnel 5 4

1c. Availability 5 2 Poor Grammar

41


Example of Relevant Comments:


1a. Staffing Plan 5 1Only one manager listed, insufficient for all sites


1c. Availability 5 2Vendor unable to provide weekend staff

42


Scoring Matrix:

Evaluation Criteria Evaluator 1 Evaluator 2 Evaluator 3 Final Score Max Points

1a. Staffing Plan 2 1 5 2.7 5

1b. Key Personnel 3 4 4 3.7 5

1c. Availability 2 2 5 3 5

Total 9.4 15

10/17/2019

15

43


1a. Staffing Plan 5 5


1c. Availability 5 5

Documenting Technical Evaluation – Case Study

• Evaluator 1:

• Evaluator 2:

• Evaluator 3:


1a. Staffing Plan 5 1 Only one manager listed, insufficient for all sites


1c. Availability 5 2 Did not mention coverage for time off


1a. Staffing Plan 5 2 Inadequate staff for multiple sites


1c. Availability 5 2 One of four preferred qualifications met

44


45

• NYS Procurement Guideline V.I. Conducting the Cost Evaluation:

“Methods for calculating costs vary depending on a mix of factors concerning the nature and extent of the services, the costs associated with utilizing the services, and the impact of the services on agency programs and operations (State Finance Law§160.5 and 6).”

Financial Evaluation

10/17/2019

16

46

• Best Practice for Conducting Cost Evaluation:

• Must be evaluated objectively

• Should not be weighted at less than 20%

• Vendor should use the cost form included with the RFP

– Costs submitted must be completed in accordance with RFP. Do not accept narratives.


47

• Cost Proposals shall be equivalent.

• Can you compare “Apples to Apples”?

• Agency must be able to tell how the contract value was calculated based on the proposed cost.

• Agency must be able to justify how the costs are reasonable.

• Agency may clarify that vendor fully understands scope if the cost appears to be out of line with the other proposals.

Financial Evaluation Considerations

48

• Evaluating cost is a calculation, not an analysis

• Recommended formula for cost scores:

Cost Score = (Lowest Bid ÷ Bid Being Evaluated) X Maximum Cost Points Available


10/17/2019

17

49

• Must identify all titles for a vendor to bid on

• Should require that a vendor provide rates for all titles listed

• Should provide estimated usage hours for vendors to bid equally

• May weight each title separately within the cost score

• Should not take simple average of all hourly rates across the board

• Should not allow vendors to bid a range of rates

Financial Evaluation on Hourly Rates

50

• Sample Cost Proposal Form:


Job Title Hourly Rate Estimated Hours Total Cost

Title 1 To be entered by vendor To be entered by agency $




Total Bid Amount $

51

• Vendor A • Vendor B

• Averaging the hourly rates does not provide reasonable representation of actual costs for evaluation purposes.

• No estimated usage was provided by the Agency.


Non‐recommended options for evaluating hourly rates:

Job Title Hourly Rate

Manager $160

Sr. Analyst $90

Analyst $55

Tech Staff $45

Avg. Hourly Rate $87.50

Job Title Hourly Rate

Manager $130

Sr. Analyst $110

Analyst $65

Tech Staff $55

Avg. Hourly Rate $90

10/17/2019

18

52


• Agency provided the estimated usage hours.


Recommended options for evaluating hourly rates:

Job TitleHourly Rate

# of Hours Total

Manager $160 3,000

Sr. Analyst $90 1,000

Analyst $55 2,000

Tech Staff $45 4,000

Total

Job TitleHourly Rate

# of Hours Total

Manager $130 3,000


Analyst $65 2,000


Total

$480,000

$90,000

$110,000

$180,000

$860,000

$390,000

$110,000

$130,000

$220,000

$850,000

53

• RFP for a Consulting Service

– Cost will be based on hourly rates.

– Cost sheet requires vendors to provide a rate for each title listed below.

– Two (2) vendors submitted proposals.

– Vendor A was deemed non responsive.


Financial Evaluation on Hourly Rates – Case Study A

Job Title Rates # of Hours

Manager $140 ‐ $160 3,000

Sr. Analyst $80 ‐ $90 1,000

Analyst $55 ‐ $75 2,000

Tech Staff $40 ‐ $60 4,000

Job Title Rates # of Hours

Manager $150 3,000


Analyst $55 2,000


54

• An agency is implementing a new system for invoice tracking.

• There is no previous contract for these services.

• An agency is unsure of the estimated number of hours for each job title that is required for this contract.

Financial Evaluation on Hourly Rates – Case Study B

Title Rate Weight Points

Project Manager $300 70% 21

Programmer $175 20% 6

Support Staff $100 10% 3

Total 100% 30

10/17/2019

19

55


56

• An agency issued an RFP.

• The RFP specified the following:

• Evaluation would be 70% Technical, 20% Cost and 10% Presentation.

• Vendors will be shortlisted for a presentation.

• Two awards would be made.

Financial Evaluations – Case Study

57

• Vendors must propose their costs in the following manner:

Vendors could propose either a % of compensation with a minimum fee OR vendors could propose a flat fee.


PROFESSIONAL FEES

% of Compensation %

Minimum Fee $

Flat Fee $

10/17/2019

20

58

• Vendors’ proposals were submitted in the manner per the table below:

• Cost Proposals are NOT equivalent.

• Cannot be evaluated “apples to apples”.

• This is NOT recommended.


Vendor A Vendor B Vendor C

PROFESSIONAL FEES Pricing Score Pricing Score Pricing Score

% of Compensation (10 pts) 30% 9.33 28% 1.00

Minimum Fee (10 pts) $45,000 10.00 $72,000 16.25

Flat Fee (20 pts) $60,000 20.00

TOTAL COST SCORE 19.33 20.00 16.25

59


60

State Finance Law §163.9(a)(b):

“The evaluation criteria and methodology for evaluating proposals must be completed prior to the initial receipt of proposals.”

Oral Presentations and Shortlisting

10/17/2019

21

61

• There are multiple ways interviews and oral presentations can be used in the evaluation:

• Can be scored as its own separate category

• Can be used to clarify items from the vendor’s technical proposal

• Evaluators can revise technical scores based on information provided in an interview/oral presentation. If this method is used, evaluators must address any changes to the original scores.


62

• There are multiple ways interviews and oral presentations can be used in the evaluation (continued):

• There should be a script or set of questions that are followed so that all evaluators ask the same thing of all vendors.

• Same evaluators should participate in each interview and oral presentation.

• Evaluation method needs to be documented in the procurement record before proposals are received.


63

• What is Shortlisting?

• Process that limits the number of proposers that are allowed to continue through the final steps of the evaluation process.

• When shortlisting, agency MUST take into consideration BOTH technical and cost scores.

– Shortlisting without considering cost is arguably inconsistent with the SFL “best value” concept.


10/17/2019

22

64

• What is Shortlisting? (continued)

• Shortlisting is an optional technique and should only be done if deemed necessary by an Agency.

• Shortlisting must be identified in the evaluation instrument, if an agency plans to utilize this technique.

• Shortlisting should include all vendors that were susceptible to award after the preliminary technical and cost scores.

– Example: If the oral presentation and interview is worth 10 points, anyone within 10 points of the highest scoring vendor is susceptible to award and must be shortlisted.


65

• An agency issued an RFP.

• The RFP specified the following:

• Evaluation would be 70% Technical, 20% Cost and 10% Presentation.

• Vendors will be shortlisted for a presentation.

• Two awards would be made.

Oral Presentations and Shortlisting – Case Study

66

• Preliminary Scores:

• Short Listed Vendors:



A 53.33 19.33 72.66

B 54.00 20.00 74.00

C 42.97 16.25 59.22

D 59.00 16.70 75.70

E 62.00 11.96 73.96

F 56.00 16.76 72.76


D 59.00 16.70 75.70B 54.00 20.00 74.00E 62.00 11.96 73.96F 56.00 16.76 72.76

• RFP Stated:

The proposal that receives the highest composite score and the next three highest scoring proposals, if they are all within 10 points of the highest total score, will be shortlisted and invited for an oral presentation.

10/17/2019

23

67

• The four shortlisted vendors were invited to oral presentations and the results were as follows:

• Agency awarded contracts to Vendors D and B.


Vendor Technical Score Cost Score Oral Presentation Composite Score

D 59.00 16.70 6.50 82.20

B 54.00 20.00 8.00 82.00

E 62.00 11.96 7.00 80.96

F 56.00 16.76 9.00 81.76

This is incorrect.

68

• Preliminary Scores:

• Short Listed Vendors:



A 53.33 19.33 72.66B 54.00 20.00 74.00C 42.97 16.25 59.22D 59.00 16.70 75.70E 62.00 11.96 73.96F 56.00 16.76 72.76

• All Vendors Susceptible to Award were shortlisted.

• After the Oral Presentation, Vendors A and D have the highest composite scores and should be awarded contracts.

Vendor Technical Score Cost Score Composite Score Oral Presentation Total Score

A 53.33 19.33 72.66 10.00 82.66

B 54.00 20.00 74.00 8.00 82.00

D 59.00 16.70 75.70 6.50 82.20

E 62.00 11.96 73.96 7.00 80.96

F 56.00 16.76 72.76 9.00 81.76

69


10/17/2019

24

70

• What is a Clarification?

• A procurement tool to provide agencies an assurance of the offerer’s full understanding of responsiveness to the solicitation requirements

• Where provided in the solicitation, state agencies may require clarification [SFL 163.9(c)]

Clarification

71

• Things to consider when clarifying a vendor’s response:

• All clarifications must be addressed prior to award.

• All offerers shall be accorded fair and equal treatment.

Clarification

72

• What is an allowed vendor clarification?

• A mathematical error

• Allows additional information/further explanation

• What is not allowed in a vendor clarification?

• The opportunity to remedy a technical and/or cost proposal

• Remedy a prior omission with new material change

Clarification

10/17/2019

25

73

• Agency Q advertised an RFP for computer programming services.

• Requirement ‐ Experience in JavaScript:

– Vendor 1: “I have experience in JavaScript”

– Vendor 2: “I have experience in the required program”

– Vendor 3: “I have experience in programs that are similar to JavaScript”

Clarification – Case Study

74

• Recommended Steps:

• Reach out to the bidders who didn’t clearly confirm they can meet the requirements

– Ask Vendor 2 and Vendor 3 if they have experience in JavaScript

– Give the vendors a chance to further explain

• Submit the written communication as part of your procurement record

Clarification – Case Study A

75

• Agency Z has an RFP for a one year term for software maintenance services. Vendors were required to provide the following as part of their cost:

• Hourly rates for the following titles:

» Job Title A » Job Title B » Job Title C

• One‐time start up fee

• Annual maintenance fee

• Three vendors responded

• Each of the three proposals had one cost item that required clarification

Clarification – Case Study B

10/17/2019

26

76

Required Vendor A Vendor B Vendor C

Job Title A $100 $ $100

Job Title B $80 $80 $80

Job Title C $60 $60 $60

Estimated 2,080 Hours for Each Job Title

$499,200 $291,200 $499,200

One Time Start‐Up Fee $ $3,000 $300,000

Annual Maintenance Fee $4,500 $4,500 $4,000

Total $503,700 $298,700 $803,200

Clarification – Case Study B (Before Clarification)

77

Required Vendor A Vendor B Vendor C

Job Title A $100 $80 $100

Job Title B $80 $80 $80

Job Title C $60 $60 $60

Estimated 2,080 Hours for Each Job Title

$499,200 $457,600 $499,200

One Time Start‐Up Fee $0 $3,000 $300

Annual Maintenance Fee $4,500 $4,500 $4,000

Total $503,700 $465,100 $503,500

Clarification – Case Study B (After Clarification)

• Vendor A: Confirmed no omission, $0 dollar fee – not changed Proposal

• Vendor B: Was supposed to include $80 hourly rate – changed proposal

• Vendor C: Typographical error, should list $300 – changed proposal

78

Any Questions?

10/17/2019

27

79

• Angelina Cadena, (518) 474‐0039, [email protected]

• Priscilla Cassidy, (518) 402‐2969, [email protected]

• Amanda Colomb, (518) 486‐5197, [email protected]

• Alyssa Connelly, (518) 474‐4028, [email protected]

• Christina Crasto, (518) 408‐3895, [email protected]

• Jenny Fung, (518) 473‐6765, [email protected]

• Michelle Ko, (518) 479‐2401, [email protected]

• Ashley Markowski, (518) 486‐3165, [email protected]

• Matthew Mirabile, (518) 474‐4358, [email protected]

• Zachary Nash, (518) 474‐6938, [email protected]

• Dan Robinson, (518) 473‐7107, [email protected]

• Jennifer Waithe, (518) 408‐4389, [email protected]

• Melanie Young, (518) 486‐6231, [email protected]

• Melissa Taub, (518) 486‐3033, [email protected]

• Jenny Reyes, (518) 474‐8946, [email protected]

• Dave Burmaster, (518) 486‐3075, [email protected]

• Randy McConnach, (518) 486‐1250, [email protected]

Contacts

80