MTAC WG 114

MTAC Workgroup 114

Service Performance Measurement

Review and Discussion

Full Workgroup Meeting

July 31, 2007

MTAC WG 114

IBM presentation on measuring service performance (April 13)

USPS Intelligent Mail/Seamless Acceptance presentations (Feb 21, April 13, May 15, June 13, July 11 & numerous subgroup meetings)

International Post Corporation (IPC) (July 11)

Comments to PRC Notice on Service Standards/Measurement (July 16-30)

Countless hours of workgroup/subgroup discussion…

What Have We Learned About Service What Have We Learned About Service Performance Measurement?Performance Measurement?

MTAC WG 114

Test mail sampling should be representative of real mail flows (base design on real data)

Define the measurement period and precision level

Follow live mailing patterns to extent possible

Meet desired stratification and geographic coverage by spreading inductions and receipts accordingly

Sample size determined by precision level desired, time period, variability of population, number of sample groups to be measured (larger sample sizes can improve precision)

IBM Take-Aways…IBM Take-Aways…

MTAC WG 114

Start the Clock and Stop the Clock validation rules needed

Monitor management a critical component

Data analysis/data exclusion processes needed

Quality assurance processes critical

IBM Take-Aways…IBM Take-Aways…

MTAC WG 114

Intelligent Mail will provide the best measurement solution

There are some smaller volume product mailstreams where Intelligent Mail is not a viable solution

Start the Clock validation is key for data inclusion in service performance measurement

For some products, last equipment scan can serve as proxy for actual delivery date

Mailings that do not meet established preparation requirement tolerances will not be included in service performance measurement

USPS Take-Aways…USPS Take-Aways…

MTAC WG 114

“The focus should be on improving service performance, not measuring it…”

“There is a tendency to over-complicate measurement…”

“Precision in measurement should be approached with care because it can be a significant cost driver…”

“Variability in service performance may require more measurement…if performance is consistently high, less measures are needed, but if performance drops below an acceptable point then more measurement may be needed to identify issues and improve service (e.g., if performance is 50%, +/- 10% precision may be ok)”

IPC Take-Aways…IPC Take-Aways…

MTAC WG 114

Measurement as a by-product of operations using real mail is best

Measurement does not always have to be end-to-end mail flow; measurement of network segments (using different methods) can provide end-to-end as a by-product

“There can be enormous expense to measure small volumes with as much precision as high volumes…”

In the IPC member countries, the tendency now is to have the post perform the measurement with audit versus a separate system performed by the regulator.

IPC Take-Aways…IPC Take-Aways…

MTAC WG 114

Intelligent Mail/internal measurement solutions preferred where available

Measurement should lead to continuous service improvement

Consolidated/Aggregate measurement data needed by business mail users to manage their business or add value to mail

CETs must be linked to standards and measurement

Measurement systems may be evolutionary

Periodic review of standards/measurement systems

PRC Notice Comments Take-Aways…PRC Notice Comments Take-Aways…

MTAC WG 114

Does the workgroup want to make specific recommendations on measurement sampling methodologies? Or over-arching recommendations on measurement needs?

Acknowledge that in the remaining time, the workgroup will not be able to fully evaluate and complete very specific measurement recommendations…

– Make as many guiding recommendations as possible

– Recommend further review process (service performance measurement implementation group? MTAC workgroups? This workgroup reconvened in x months?)

Workgroup Decisions on Measurement…Workgroup Decisions on Measurement…

MTAC WG 114

Business mail users need service performance measurement systems that:

– Balance cost of measurement with service needs of users

– Provide the USPS with actionable data to identify service issues and improve performance

– Provide business mail users with data to better manage their businesses and add value to the mail

General Recommendations?General Recommendations?

MTAC WG 114

Business mail users prefer Intelligent Mail measurement solutions, where available, because:

– Less costly, largely passive technology

– Measurement as by-product of operations, more volume can be measured easily, more accurate data

– Provides the USPS with very granular, real-time, and actionable data to identify service issues and improve performance

– Provides more data than likely is needed for statistical representation, allowing for in-depth diagnostics and sufficient data after data exclusion processes

– Data and reporting consistency (Confirm data vs service performance measurement data)

– Provides business mail users with data to better manage their businesses and add value to the mail

General Recommendations?General Recommendations?

MTAC WG 114

Business mail users recognize there may be a need for some external (seed) measurement solutions, but recommend:

– Costs be minimized (e.g., replace existing and new external measurement with IM measurement when possible, evaluate existing industry solutions, etc.)

– Ensure that test mail is not the major flow of product volume mailstream being measured

– Explore industry seed programs– Proven, existing measurement solutions– Fabrication of test mail impossible or impractical

General Recommendations?General Recommendations?

MTAC WG 114

Measurement quality metrics are needed

– There need to be quality standards for Start the Clock processes/scans and for Stop the Clock processes/scans

– Plan for improvement over time

– Recommendations need to be formed around such metrics

General Recommendations?General Recommendations?

MTAC WG 114

Other measurement recommendations…

– Internal USPS measurement systems should be audited

– Measurement systems will be evolutionary and should be reviewed periodically (USPS and customer review)

General Recommendations?General Recommendations?

MTAC WG 114

Measurement systems may be implemented over time, taking costs and product volumes into account

– Initial focus on high volume product mailstreams

– Not ALL measurement needs to be fully implemented in the next year (or two?)

– Review by USPS/mailers as measurement systems are implemented (learn as we go)

General Recommendations?General Recommendations?

MTAC WG 114

How to approach small volume product mailstreams where IM is not a viable solution?

– Measure over longer period? Less statistical precision?

– Increased measurement if consistent performance issues?

– External measurement with sampling methods that recognize costs and size of mailstream?

– Does this workgroup have recommendations, or should the USPS/PRC decide? Over-arching recommendations from workgroup, details by USPS/PRC?

Open Issues for Discussion…Open Issues for Discussion…

MTAC WG 114

What data is included/excluded from Service Performance Measurement system (versus raw data provided to mailers through Confirm/DelCon, etc.)

– Mailers want to be able to discuss service issues with USPS using the same data

– Mailers feel there is value in analyzing all data (e.g., identifying USPS scanning issues or mailer issues)

– Including data on mailings with presort/barcode/address quality errors but reporting that data separately can be a valuable tool for USPS and mailers to improve mail prep and diagnosing service issues

Open Issues for Discussion…Open Issues for Discussion…

MTAC WG 114

Measurement Approaches

– Is end-to-end network measurement always necessary?

– Measure network segments?

– The goal is to improve service…

Open Issues for Discussion…Open Issues for Discussion…

MTAC WG 114

MTAC Workgroup 114

Thanks for your Participation!