1

Developing Indicators of Developing Indicators of Quality for AT Reuse Quality for AT Reuse

ProgramsPrograms

Lindsey Bean Kampwerth and Trish RedmonModerators: Carolyn Phillips, Joy Kniskern and Liz Persaud

Pass It On CenterPass It On Centerwww.passitoncenter.orgwww.passitoncenter.org

2

AgendaAgenda

– Vision and goals of Indicators of QualityVision and goals of Indicators of Quality– Overview of the development processOverview of the development process– Themes of feedback on first two Themes of feedback on first two

versionsversions– Design of Prototype Design of Prototype IQIQ--ATR Online ATR Online

Assessment ToolAssessment Tool– Next steps for IQ and Online ToolNext steps for IQ and Online Tool

3

Vision and Goals for Vision and Goals for Indicators of Quality for AT Indicators of Quality for AT

ReuseReuseIQIQ--ATRATR

• IQIQ--ATRATR is a living document that grows and is a living document that grows and develops as AT Reuse Programs develop.develops as AT Reuse Programs develop.

• IQIQ--ATRATR Online will be a dynamic, interactive Online will be a dynamic, interactive tool to assess an AT Reuse Program. tool to assess an AT Reuse Program.

• IQIQ--ATRATR will link to Pass It On Knowledge Base will link to Pass It On Knowledge Base to provide references to information and to provide references to information and models to assist with improving the Reuse models to assist with improving the Reuse Program.Program.

3

4

Vision and Goals for Vision and Goals for IQIQ--ATRATR

4

Knowledge Base supportsIQ-ATR Self-Assessment Tool

IQ-ATR andKnowledge Base inform Successful Practices

Successful Practices inform Knowledge Base

IQIQ--ATRATR

5

This is not the first project:This is not the first project:Other Related QI InitiativesOther Related QI Initiatives• The SMART ExchangeThe SMART Exchange

– Led by Shelley Kaplan, UCP in AtlantaLed by Shelley Kaplan, UCP in Atlanta• 1989-1991, NIDRR grant1989-1991, NIDRR grant• Developed indicators for AT useDeveloped indicators for AT use

The rationale for many of the SMART The rationale for many of the SMART Exchange indicators are relevant to AT Exchange indicators are relevant to AT reuse indicators.reuse indicators.

6

Other Related QI Other Related QI InitiativesInitiatives

• QIATQIAT– Quality Indicators for Assistive Quality Indicators for Assistive

Technology Services in School SettingsTechnology Services in School Settings– Developed in 1998Developed in 1998

QIAT is an outstanding working model QIAT is an outstanding working model of quality indicators for AT use in a of quality indicators for AT use in a specific environment.specific environment.

7

Validating the IQ-ATRValidating the IQ-ATR

1.1. Have we defined a set of factors or Have we defined a set of factors or processes known to lead to successful processes known to lead to successful outcomes?outcomes?

2.2. Can we create a review instrument to Can we create a review instrument to determine if the factors/processes are in determine if the factors/processes are in place?place?

3.3. Is the indicator useful for internal Is the indicator useful for internal quality improvements?quality improvements?

4.4. Can the indicator be used as an external Can the indicator be used as an external measure of quality across programs of measure of quality across programs of the same type and activities?the same type and activities?

8

Developing theDeveloping the IQIQ--ATRATR

• Two Webinars on Quality IndicatorsTwo Webinars on Quality Indicators• A meeting of program professionals and A meeting of program professionals and

PIOC team members in Atlanta to develop PIOC team members in Atlanta to develop initial draftinitial draft

• Distribution of draft to developers and Distribution of draft to developers and interested partiesinterested parties

• Editing of draft based on feedback, Editing of draft based on feedback, including alteration of restrictive formatincluding alteration of restrictive format

• Development of assessment tool prototypeDevelopment of assessment tool prototype

9

Developing the first Developing the first draftdraft

• Reviewed past quality indicatorsReviewed past quality indicators• Reviewed guidelines for writing quality indicatorsReviewed guidelines for writing quality indicators• Small groups edited and added to draft of quality Small groups edited and added to draft of quality

indicatorsindicators

10

First draft done: First draft done: Critique time!Critique time!

QI for AT Reuse Development Meeting in Atlanta, QI for AT Reuse Development Meeting in Atlanta, March 23 & 24 with occupational therapists, March 23 & 24 with occupational therapists, program directors and other key informants. program directors and other key informants. Thank you!Thank you!

11

Categories of IndicatorsCategories of Indicators

• Program Program OperationsOperations

• User ServicesUser Services• SustainabilitySustainability• ManagementManagement• Human ResourcesHuman Resources

• MarketingMarketing• AccountingAccounting• Organizational Organizational

StructureStructure• Emergency Emergency

PreparednessPreparedness• Partnerships and Partnerships and

CollaborationCollaboration

12

Suggestions from Suggestions from reviewers:reviewers:

• Reorganize format: categories and Reorganize format: categories and sequencesequence

• Combine some topics; separate some Combine some topics; separate some topicstopics

• Consider strictures of tabular formatConsider strictures of tabular format• Add rationale for categoriesAdd rationale for categories• Reorganize into “core” indicators and Reorganize into “core” indicators and

options based on program type options based on program type • Add specific AT State program indicatorsAdd specific AT State program indicators

13

Version 2Version 2

• Restrictive five-cell Restrictive five-cell response format was response format was abandoned in favor of a abandoned in favor of a flexible checklist of key flexible checklist of key factors for considerationfactors for consideration

• Indicators were Indicators were combined or split as combined or split as neededneeded

• New version was New version was presented to National presented to National Task Force – which Task Force – which recommended a thorough recommended a thorough language review language review

14

Version 2.1Version 2.1

• NTF sub-committee (Rik Opstelten, Brian NTF sub-committee (Rik Opstelten, Brian Bard, Kerri Morgan and Lindsey Bean) Bard, Kerri Morgan and Lindsey Bean) reviewed the language and recommended reviewed the language and recommended changes.changes.

• Oops! Name change! QIAT recommended Oops! Name change! QIAT recommended against expansion of their name and against expansion of their name and possible confusion. (Hence, IQ-ATR.)possible confusion. (Hence, IQ-ATR.)

• Language updated, and some major issues Language updated, and some major issues were confronted before building the were confronted before building the Online Tool.Online Tool.

15

Major ChallengesMajor Challenges

• OrganizingOrganizing the indicators into CORE the indicators into CORE indicators and implementing indicators and implementing appropriate ‘branching’ based on appropriate ‘branching’ based on program typeprogram type

• WeightingWeighting the categories and the the categories and the indicators to create an appropriate indicators to create an appropriate “result” measure for the program“result” measure for the program

• ImplementingImplementing online assessment tool online assessment tool and links to Knowledge Baseand links to Knowledge Base

16

Organizing the Organizing the IndicatorsIndicators

Variables affecting grouping:Variables affecting grouping:• Type of organizationType of organization• AT reuse activities (reassign, AT reuse activities (reassign,

refurbish, exchange, recycle)refurbish, exchange, recycle)• Types of equipment (DME, Types of equipment (DME,

computers, all?)computers, all?)

How to deal with the possible combinations?How to deal with the possible combinations?

17

Weighting Categories & Weighting Categories & IndicatorsIndicators

1. Survey designed and individuals asked 1. Survey designed and individuals asked to rank: to rank:

• Categories in order of importanceCategories in order of importance• Indicators in order of importance Indicators in order of importance

within categorieswithin categories

2. Ph.D. statistician recommended 2. Ph.D. statistician recommended weighting methodology based on weighting methodology based on ranking, but it required a standard ranking, but it required a standard number of responses.number of responses.

18

To weight or not to To weight or not to weight?weight?

• Based on the ranking of categories Based on the ranking of categories and indicators, it is possible to and indicators, it is possible to produce a numeric “score” from the produce a numeric “score” from the online tool.online tool.

• For the initial assessment tool, it For the initial assessment tool, it was decided not to produce numeric was decided not to produce numeric results, but advisory messages results, but advisory messages supported by lists of resources for supported by lists of resources for each indicator.each indicator.

19

Version 3Version 3• Challenge:Challenge: A return to the five-cell format A return to the five-cell format

would lose ease-of-use that made the flexible would lose ease-of-use that made the flexible checklist of key factors so attractivechecklist of key factors so attractive

• Challenge:Challenge: Too many variables to group Too many variables to group indicators by program typeindicators by program type

Solution: Solution: Retain the flexible lists of key factors for consideration. Retain the flexible lists of key factors for consideration. Create standard set of responses to assess compliance.Create standard set of responses to assess compliance. Allow program to determine if indicator applies.Allow program to determine if indicator applies. No numeric scores – even hidden – at this timeNo numeric scores – even hidden – at this time

20

IQ-ATR and theIQ-ATR and theOnline Program Assessment Online Program Assessment

ToolTool• Sharon Meek and the PIOC web Sharon Meek and the PIOC web

developers from REACH, Inc. developers from REACH, Inc. developed the prototype of the developed the prototype of the interactive toolinteractive tool

www.passitoncenter.org/iq-atreusewww.passitoncenter.org/iq-atreuse

21Home Page: A brief introduction to how the IQ-ATR tool works

22

RegistrationRegistration

Create an account for personal use or register as a guest.

23Directions explain the response choices.

24

Category in the yellow bar. The name of the indicator and the text of the indicator are followed by the Rationale and the Key Factors for Consideration. The standard set of responses follows. User selects a response and clicks Next.

25

User receives a results page for each category and may stop or continue to the next category. KB or other sources are suggested for indicators that do not meet all Key Factors for the promising practice.

26

Next StepsNext Steps

IQ-ATR:IQ-ATR: Share and solicit additional feedback Share and solicit additional feedback Revise Revise IQIQ--ATRATR based on based on

recommendationsrecommendations Design a validation processDesign a validation process

27

Next StepsNext StepsOnline Program Assessment Tool:Online Program Assessment Tool: Gather feedback from pilot usersGather feedback from pilot users Revise online tool and make “updatable” to Revise online tool and make “updatable” to

keep synchronized with Knowledge Base (KB)keep synchronized with Knowledge Base (KB) Allow user to choose sequence of categoriesAllow user to choose sequence of categories Have NTF review KB content for Have NTF review KB content for

appropriateness and usefulness as appropriateness and usefulness as recommended at May meetingrecommended at May meeting

Identify or create content to fill reference Identify or create content to fill reference gaps in Knowledge Basegaps in Knowledge Base

Decide whether responses should be Decide whether responses should be weighted and tallied for a quantitative (rather weighted and tallied for a quantitative (rather than general) result.than general) result.

28

Not enough Not enough knowledge. . .knowledge. . .

• Referencing Indicators of Quality to Referencing Indicators of Quality to content in the Knowledge Base highlighted content in the Knowledge Base highlighted the gaps in supporting information.the gaps in supporting information.

• The content that does exist needs to be The content that does exist needs to be strengthened with more comprehensive strengthened with more comprehensive presentation of needed policies and presentation of needed policies and procedures.procedures.

• The existing content has not been The existing content has not been reviewed by NTF panels as recommended reviewed by NTF panels as recommended in May 2009.in May 2009.

29

How you can helpHow you can help

• Review the IQ-ATR and send commentsReview the IQ-ATR and send commentshttp://passitoncenter.org/contenthttp://passitoncenter.org/content • Try the online tool and send feedbackTry the online tool and send feedbackhttp://passitoncenter.org/IQATReusehttp://passitoncenter.org/IQATReuse • Contribute content or suggest resources Contribute content or suggest resources

for the Knowledge Base for the Knowledge Base

Please send comments and suggestions to:Please send comments and suggestions to:Lindsey@passitoncenter.orgLindsey@passitoncenter.orgTrish@passitoncenter.orgTrish@passitoncenter.org

30

DISCLAIMERThis work is supported under a five-year cooperative agreement # H235V060016 awarded by the U.S. Department of Education, Office of Special Education and Rehabilitative Services, and is administered by the Pass It On Center of the Georgia Department of Labor – Tools for Life. However, the contents of this publication do not necessarily represent the policy or opinions of the Department of Education, or the Georgia Department of Labor, and you should not assume endorsements of this document by the Federal government or the Georgia Department of Labor.