KPIs and Management Report

Report to the Management Board and General Assembly on Q1 2016

▪  KPIs ▪ Content ▪  Education & Product

Support ▪ HR ▪ CRM & Collaborations ▪  Technical Services ▪ MLDS Affiliate Report ▪ Work Item Status Update

Contents

▪ Updates by Line of Business (each with Q1 Achievements & Q2 Key Objectives) ▪ Content ▪  Education ▪ CRM ▪ Collaboration ▪ Delivery ▪  Technical Services

KEY PERFORMANCE INDICATORS

Content KPIs (slide 1 of 4)

▪ Request submission for the January 2016 release

KPI Target Achieved

All requests will be reviewed and have their status updated from NEW within 7 days of submission

100% 100%

All requests received before the request close date will be completed, closed or be allocated to content tracker item

100% 100%

Authored 2807 63.32% Request closed not authored 1104 24.90% Assigned to Content Tracker item 255 5.75% Awaiting resolution 267 6.02%

Total 4433 100.00%

January 2016 release summary

Content (slide 2 of 4) SIRS Backlog

Figures exclude CMT related submissions

▪  Figures for 2016 (1st January 2016 – 4th April 2016) ▪ General requests received – 907 ▪ CMT requests received – 434

▪  Service level agreement targets ▪ Requests submitted 1st Jan 2016 to 30th March 2016

▪ Total - 777 ▪  'completed' for SLA target - 35% (Target 90%)

Service levels predictably fell following the introduction of the new authoring tool, as the tool is still under improvement and as the authors got accustomed to an entirely new workflow. Service levels have already improved and will continue to do so, with the final target measurable at the end of the editing cycle. Due to the new tooling and workflow, authors will have an additional two months of authoring every release cycle, as QA processes will be done continuously rather than at the end of the cycle. We should see the full effects of this with the January 2017 release.

2015 2014 2013 2012 2011 2010 2009 2008 Reviewed only 406 7 0 0 0 0 0 0 Allocated to content tracker item 394 815 745 132 10 20 20 0

Content KPIs – release summary (concepts) (slide 3 of 4)

Num

ber o

f con

cept

s

0

5000

10000

15000

20000

25000

30000

New Concepts Changed concepts

Release Date in form of YYYYMMDD

Content KPIs – release summary (descriptions) (slide 4 of 4)

Num

ber o

f des

crip

tions

Release Date in form of YYYYMMDD

0

20000

40000

60000

80000

100000

120000

140000

160000

New descriptions Changed descriptions

Education and Product Support KPIs (slide 1 of 3) Presentations available at end of reporting period

Public accessible presentations Course-specific presentations

15-Q4 5 64

16-Q1 8 81

Assessments available at end of reporting period Public accessible assessments Course-specific assessments

1 13

1 17

E-Learning development and delivery during reporting period Presentations developed Assessments developed Interactive webinars delivered as part of E-Learning courses

16 6 47

17 4 67

Usage of public E-learning materials during reporting period Public accessible presentation views Public accessible assessments taken Average mark for assessment taken

856 248 70%

840 407 68%

Course enrollment and completion during reporting period Students starting Foundation Course Students currently taking Foundation Course Students successfully completing Foundation Course Drop out/did not successfully complete Foundation Course Students starting Implementation Course Students currently taking Implementation Course Students successfully completing Implementation Course Drop out or did not successfully complete Implementation Course

15-Q4 423 809 291 190 109 154

0 6

16-Q1 215 347 197 431

94 160

29 25

Course demand in this reporting period New Foundation Course applicants New Implementation Course applicants

276 157

196 120

Unmet course demand at end of reporting period - students applying for… Foundation Course with confirmed place allocated in next 6 months Foundation Course without a confirmed place in next six months Implementation Course with confirmed place in next 6 months Implementation Course without a confirmed place in next 6 months *Maximum backlog now 6-8 weeks so this is just those staring Jan 2016 ** These were delayed by choice to synchronize with other activities

79*

0 150 7**

46*

0 160

0

Education and Product Support KPIs (slide 2 of 3)

Education and Product Support (slide 3 of 3) Helpdesk tickets and response times and resolution rates

Num

ber o

f tic

kets

and

min

utes

Num

ber o

f tic

kets

and

per

cent

ages

Human Resources KPIs

0

5

10

15

20

25

30

35

40

45

50

Q1 2014 Q2 Q3 Q4 Q1 2015 Q2 Q3 Q4 Q1 2016

Num

ber o

f Em

ploy

ees

Quarter and Year

Number of Departing Employees

Number of New Employees during Q

Total Number of Employees end of Q

2015 Q1 Q2 Q3 Q4 2016 Q1

# Members 27 27 27 28 28

# Large Vendor Agreements 0 0 0 0 0

# New Collab Agreements 0 1 0 0 0

Appropriate Representation at External Events

100% 100% 100% 100% 100%

% Survey Results Showing Strong Product/Service Levels

90.5% (Member survey)

n/a n/a n/a 95,6% (member survey)

Member satisfaction is at least 80% for CRM (annual measurement)

Yes 95.2%

n/a n/a n/a Yes 100%

Member satisfaction of IHTSDO is at least 75% (annual measurement)

Yes, 90.5%

n/a n/a n/a Yes 95,6%

Customer Relations & Collaborations KPIs (slide 1 of 2)

2015 Q1 Q2 Q3 Q4 2016 Q1

Customers receive a response to enquiry with confirmation of assignment in 48 hrs

5h26m avg first

response

3h21m avg first response

3h59m avg first response

3h35 m avg first response

4h57m avg first response

25% of enquiries are resolved in first contact by assigned agent

Yes, at 68%

Yes, at 64%

Yes, at 62%

Yes, at 67% Yes, at 62%

Avg time to resolution of enquiries is 5 working days/one week

2.75 days 0,94 days 1.51 days

0,64 days 2,3 days

Completing customer engagement plan 75% targets met

On track On track On track

Met On track

Collaboration operational plan 2015/16 – 75% targets met

Plan being updated

Customer Relations & Collaborations KPIs (slide 2 of 2)

Note: Q3 performance still better than targets but affected by particularly challenging tickets and summer holidays of staff and customers.

Technical Services - KPIs

▪  SLAs are detailed here –

▪  https://confluence.ihtsdotools.org/display/IT/Tooling+Key+Performance+Indicators

Technical Services KPIs Customer Service (internal & external)

Tickets On time resolution (%)

First contact resolution (%)

Average first response (mins)

Average response (mins)

Average resolution (mins)

Q2 2015 103 90 73 146 96.82 2,401 Q3 2015 92 80 67 206 123 7,980 Q4 2015 172 91 59 317 310 1,367 Q1 2016 100 83 91 215 288.52 1,185

Target 85 75 300 240 2880

Note: Q1 2016 longer response times is longer to the complexity of some of the issues raised, and time taken to resolve

Num

ber o

f tic

kets

and

per

cent

ages

Num

ber o

f tic

kets

and

min

utes

Technical Services KPIs Infrastructure Uptime (%)

Average Uptime (%) 99.41 Target Uptime (%) 99.9

The regular downtime on the Mapping tool is planned downtime for upgrades, daily data import, and necessary patch fixes. This took place generally outside of normal working hours and had no impact on availability to users. In future quarters this will be not included in downtime stats. Blank cells are for services which had not been tracked previously.

Per

cent

age

MLDS Affiliate Report 2016 Q1

Technical Services KPIs Online Service User Sessions

Num

ber o

f ses

sion

s

Tool

Technical Services KPIs Online Service User Sessions By Country & City

Work Plan – Work Item Status Update

Q1 2016 – Details of Work Items reported with Red status: 1. Content – Customer Requests & Admin: The implementation of the new authoring tool has led to a dramatic slow down in the delivery of content changes. This will continue to be monitored closely, and updates communicated to the Community of Practice. The agreed set of service levels specified for the management of SIRS requests, will continue to be reported to the Member Forum. The closure date for requests for the July 2016 release, is 5th, April 2016. The current velocity of content authoring using the new tool is set to deliver around 2000 new concepts for the July 2016 release.

No. of Work Items

Red = would not complete by financial year end Amber = behind schedule but would be complete by financial year end Green: = on track to complete by financial year end 0

10

20

30

40

50

60

70

Q1 2015 Q2 2015 Q3 2015 Q4 2015 Q1 2016

41 40 40 41

61

3 2 1

1

2 3 3 Green

Amber

Red

UPDATES BY LINE OF BUSINESS

▪  CMT – process documentation has improved communication and streamlined authoring activities

▪  Editorial and Content Managers AG’s continue to successfully deliver against their annual work plans

▪  Publication of SNOMED CT/LOINC technology preview, using an updated modelling approach

▪  Successful inclusion of GMDN data into Map tool to provide improved quality assurance of linkage table

▪  Process to receive monthly updates of GMDN data ▪  Embedding new authoring tool in working practices

Content – 2016 Q1 Achievements

▪  GMDN – development of updated end to end process to provide enhanced quality assurance (Content and Release teams)

▪  Substances – development of a detailed plan for delivery of the reworking of substance hierarchy, including resource allocation

▪  Default drug model – allocation of contracts to deliver specific work items (use case summary, draft default concept model, standards position statements)

▪  Anatomy – continuation of work with University of Washington, followed by updating of the anatomy database

▪  Refinement and documentation of new authoring processes utilising new authoring tool

▪  Discussion with Content Managers AG, to develop and agree a policy for the implementation of large scale content changes

Content – 2016 Q2 Key Objectives

▪  Foundation course running in steady state maintenance mode ▪  215 people entered course and at end of quarter 347 were active on course ▪  197 people successfully completed course

▪  Implementation course development complete ▪  94 people entered course and at end of course 160 were active on the course ▪  29 completed in Feb 2016 and 79 are on track to complete in Apr/May 2016 ▪  All completers providing feedback (22) would recommend course to others

▪  Content course pilot development 70% complete ▪  40 people entered pilot intake in Jan 2016 and on track to complete in May

▪  Product Support activities ▪  Documentation Migration to Confluence started (phase 1 should complete in April) ▪  Machine Readable Concept Model draft specification and preview delivered for

comment ▪  Work with HL7 on FHIR Terminology binding to ‘Condition resource’ draft

delivered ▪  Recruitment of two new staff completed to add to service resilience ▪  Building vendor contact to enable collaboration on enhanced product support

Education & Product Support – 2016 Q1 Achievements

▪  Improve E-Learning and Product Support service resilience ▪  Induction of new Implementation Specialists ▪  Updates to E-Learning server and E-Learning process guides

▪  Implementation Course ▪  Complete course delivery to 2nd intake and continue for 3rd intake ▪  Start 4th course intake

▪  Content Development Theory course ▪  Complete course development and delivery to pilot intake ▪  Start 2nd course intake ▪  Increase capacity for 3rd course intake using external resource

▪  Foundation course ▪  Continue running as business as usual ▪  Initiate review and update process

▪  Documentation migration and updates ▪  Complete migration of TIG and other documents to Confluence ▪  Documentation restructuring, review and revision to commence ▪  Complete Terminology Binding document ▪  Initiate work on Decision Support and Refset Guidance documents ▪  Initiate enhanced product support delivery (dependent on vendor timeline)

Education and Product Support – 2016 Q2 Key Objectives

▪  Product and Services catalogue produced Product and Services Roadmap drafted

▪  Translation project delivery drafted ▪  Mobile licensing framework developed ▪  Switzerland onboarding support moving forward, successful event in January ▪  Vendor portal beta live ▪  Vendor Forum – 1st meeting held at HIMSS ▪  Operational plan submitted ▪  Planning work for SCT Expo in New Zealand progressing ▪  Selection of EXPO host 2017 successful ▪  Renewed Czech connections – positive movement forward ▪  Successful meeting with Ministry of Health France ▪  Successful meeting with Ministry and stakeholders in China ▪  Non-Member translation agreement in place with Japanese and French

companies ▪  Successful HIMSS conference executed

Customer Relations – 2016 Q1 Achievements

▪  Additional intelligence gathered on non-Member countries and maturity level with regards to standards

▪  Potential further work on translation policy ▪  Usage information survey developed and sent out to stakeholders to

gather implementation information ▪  Continuous work on vendor engagement in identifying regional

needs and requirements ▪  2015 Member Survey results shared and action plan developed ▪  eHealth week Europe 2016 participation ▪  eHealth – Canada participation ▪  Finalized Products and Services Roadmap ▪  SNOMED CT presentation at the China Hospital Information

Network Conference 2016

Customer Relations – 2016 Q2 Key Objectives

▪  DICOM SNOMED CT Licensing Agreement and GSI Collaboration Agreement signed with agreed joint work

▪  Development of Collaboration Operational Plan 2016/17 ▪  HL7

▪  Draft Affiliate Development License agreement for MB/GA review/feedback ▪  Draft SNOMED CT Licensing FAQs focused on HL7 queries

▪  Deliverables against collaboration agreements continue on schedule with regular touch points internally and externally

▪  Proposal for way forward with IHTSDO/WHO collaboration and supporting governance structure

▪  2nd external global call for clinical reviewers for SNOMED CT/ICD-11 alignment work

▪  Agreed plan with International Dentistry SIG for delivery of General Dentistry subset (in conjunction with ADA)

▪  AMA agreement near completion

Collaboration 2016 Q1 Achievements

▪  HL7 – ▪  Sign off and publication licensing FAQs ▪ HL7 Affiliate License in place ▪  Start review of agreement

▪  Updated Collaboration Agreement with ADA signed ▪  Work plan agreed with GS1 and IHTSDO mechanism for working

with members operational ▪  OpenEHR and CDISC – agreement of managing SNOMED CT

licensing references in their products ▪  Coordinate input from IHTSDO Member experts to JIC UDI

Implementation Guidance ▪  ICN – release Nursing Interventions Equivalence table, start review

of agreement and NI 2016 participation ▪  Progress with IHTSDO/WHO activities – IF there is agreement to

proceed with collaborative activity.

Collaboration – 2016 Q2 Key Objectives

▪  Delivered the Production release of the January 2016 International Edition to our community on January 31

▪  Delivered the GMDN Linkage table to GMDNA on March 31, using the new functionality of the mapping tool to provide the source data

▪  Continued improvements of Release integration with in-house SNOMED CT products (Single Concept Authoring tool & Terminology Server) – this will be used to produce the July 2016 International Edition.

▪  Further increased the coverage of the Release Validation Framework (RVF)

▪  Delivered new improvements to the automated Release service (SRS)

Delivery – 2016 Q1 Achievements

▪  Delivery of the new Beta release of the April 2016 Spanish edition ▪  Delivery of the April 2016 Spanish Production edition ▪  Delivery of all three products of the LOINC phase 3 Alpha edition ▪  Delivery of SNOMED to ICNP Diagnoses Production edition, in line

with January 2016 SNOMED CT International Edition ▪  Delivery of the new SNOMED to ICNP Interventions Production

edition ▪  Delivery of SNOMED to GPFP/ICPC-2 Production editions, in line

with January 2016 SNOMED CT International Edition

Delivery – 2016 Q2 Key Objectives

▪  Phase 3 of Authoring Tool development and migration off Workbench on-going, working on improving user experience, performance and adding functionality.

▪  Terminology Server source code made available on request – currently to UKTC, Nictiz, Swedish NRC, and Kaiser Permanente

▪  July 2016 release is being authored on the new platform ▪  Refset Management and Translation Tooling in testing with Tooling

User Advisory Group, and improvements/enhancements made based on feedback

▪  More Members started on MLDS (IS, CH) ▪  Content Request Service beta version testing completed ▪  Production of Technical Services Operational Plan ▪  Service Management processes published along with drafts of

Information Security policy. ▪  Planning migration from the Workbench to new platforms for

Managed Service Members (DK & SE) started

Technical Services - 2016 Q1 Achievements

▪  Continued documentation improvements across all tooling services ▪  Stabilization and continued improvement of Single Concept

Authoring Tool and Terminology Server ▪  Terminology Server source code made open source and packaged

appropriately ▪  Reference Set Management and Translation Tooling in use by

external users, and packaged for deployment by Members ▪  Complete testing and any new necessary features of Content

Request Service. Plan migration off of SIRS Q2, early Q3 2016. ▪  Development of Template Based Authoring will be starting once

requirements have been detailed.

Technical Services - 2016 Q2 Key Objectives