Sustainability forATE Projects  

Nancy Maron, Program Manager, Ithaka S+RKate Wittenberg Managing Director PORTICOKate Wittenberg,  Managing Director, PORTICO

October 27, 2011

Who We Are

• Ithaka S+R is a research and consulting service• Ithaka S+R is a research and consulting service that focuses on the transformation of scholarship and teaching in an online environment, with the goal of identifying the critical issues facing ourof identifying the critical issues facing our community and acting as a catalyst for change.

• Research• Consulting• Consulting

The Sustainability Problem

We have seen:

Educational projects returning to funding agencies for additional grants to support core operations g pp ponce a project is up and running

Funding agencies seeking ways to encourage g g g y gprojects to become sustainable after the grant period

Project leaders seeking ways to support the resources they have developed after the grant endsends

Our work has addressed

Mindsets

Business Models

Strategies

Practices

Our definition, so far:

For those projects that will require ongoing supportrequire ongoing support, ‘sustainability’ is the ability to generate or gain access to the resources—financial or otherwise—needed to protect and pincrease the value of the content or service for those who use itthose who use it.

There is no single ‘right’ sustainability model

Each depends on the circumstances and aspirations of the people and projects involved.

Experimentation will be necessary to see which models work best.

There is no SUSTAIN-O-MATIC…

But there are some basic principles…

It’s not just about the money It’s not just about ‘getting by’ It’s all about identifying value to users… …and knowing that the particular value-

to-stakeholder equation is almost certain to change over time…

Case Studies in Sustainability

• Major support from the JISC-led Strategic Content Alliance; additional fundingAlliance; additional funding from NEH and NSF in 2009

• 12 case studies

2009

• Diversity of geography, disciplines, project type, institutional home 2011institutional home

• How did project

2011

leaders define and address their post-grant sustainabilitygrant sustainability goals?

What did we find? Plenty of challenges

MINDSETS Making the s itch from a f nded “project” to anMINDSETS. Making the switch from a funded “project” to an entrepreneurial “ongoing enterprise” is one of the most difficult but important challenges.

COSTS. Hidden cost contributions can obscure the real costs of operating a resource. Leaders must have a clear understanding of total project costs.

REVENUE. Reliance on a single revenue source can putREVENUE. Reliance on a single revenue source can put projects at risk. Hybrid business models and a portfolio of sustainability strategies is most effective.

Five Things Sustainable Projects Do

Empower leadership to define the mission and take action

Create a strong value proposition

Creatively manage costs

Cultivate diverse sources of revenue

Establish realistic goals and a system of accountability

A diAudience

CASE STUDY: eBird

Crowd-sourced database of birddatabase of bird observations

Initial version of the project was aimed at creating valuable datacreating valuable data for ornithologists

A shift to user-focused strategy led to major changes and steep increase in popularity of site

Lessons Learned: Sources of Support

A sustainable model may include a variety of types f t fi i l d fi i lof support, financial and non-financial

Some diversity of sources is good but reliabilitySome diversity of sources is good… but reliability trumps diversity.

Some good examples of using private partnerships to reach mission and/or revenue goals

And a willingness to keep trying new approaches

Partners

CASE STUDY: The National Archives (UK)

Commercial partners bid to digitize series thatto digitize series that are pre-selected and bundled by TNA staff

TNA staff choose vendor and negotiate termsand negotiate terms

Huge cost savings andLicensed Internet

Associates ProgramHuge cost savings and some revenue,

But some constraints…

Associates Program

Lessons Learned: Partnerships

Can take many forms, from a “partnership of equals” to more of a vendor client relationshipequals to more of a vendor-client relationship

When partner roles are well articulated, strategic partnerships can help maximize reach and p p pinnovate without requiring major investment

HHowever… Must carefully balance benefits and constraintsIf a partner is not able to contribute as plannedIf a partner is not able to contribute as planned,

need to have a backup

Aligningg g

Aligning with Host Institution

Host support played a bigger role than before in many of the places we studied:

Wh t t d l d• When grants stopped or slowed• When a business plan to generate revenue did

not reach targets ot eac ta gets• When changes to a government funding formula

reduced expected levels of support

Lessons Learned: Aligning With the Host

• Focus on activities that directly support host’s missionmission

• Develop plan to communicate value to institutional administrators

• Two places even changed their names to more clearly express their mission:y p

Center for Computing in the Humanities became: Department of Digital HumanitiesDepartment of Digital Humanities

BOPCRIS, a digitization center, became:The University of Southampton Library Digitisation Unit

Great examples. Now how do we get there?

Empower leadership to define the mission and take action

Clarifying Org Roles

Create a strong value proposition User Needs Assessment

Creatively manage costs Cost Analysis

Cultivate diverse sources of revenue

Revenue Modeling

Establish realistic goals and a system of accountability

Organizational Planning

Framework for Post‐Grant Sustainability Planning

Developing a plan for post-grant sustainability

1. Describe the post-grant impact you want2. Set specific goals for each element you will

need to sustain to achieve thisneed to sustain to achieve this3. Identify activities that will be needed to achieve

these goalsg4. Determine the costs of these activities5. Build a revenue plan to obtain financial and

th d d t t iother resources needed to support ongoing activities

Framework for Post-Grant Sustainability Planning

Content & Curriculum 

Events   Outreach & Audience  

Technical & Access Needs 

Staffing  Other 

Set Goals To achieve 

Will you need to develop and d

Will you need to hold events 

What outreach activities must 

Will the project’s 

b

Who is needed to maintain 

Are there other components 

lpost­grant impact, what must be sustained?  

update content and/or curriculum? 

post grants? continue; to whom will they be directed? 

website continue? Will online resources need storage? Archiving?

post grant activities? 

critical to post‐grant impact? 

Archiving?Identify Activities What ongoing activities will be needed to 

Labor (of staff or volunteers) in developing or editing content. 

Event planning and execution.  

Development and execution of outreach plan. 

Maintaining and updating software; labor of staff; hardware 

Succession planning for leadership; staff training & retention, 

What ongoing activities are needed to accomplish the other components 

accomplish the goals above? 

updates  volunteers as needed.  

listed above?

Determine Costs What resources ill b i d

Included here should be direct costs as well as needs for non‐financial resources, including volunteer labor and in‐kind services.  

will be required to support activities? Build Revenue Plan Where will

Included here should be a plan addressing all possible sources of revenue, including direct and indirect support from host institutions, potential earned income, and additional grants, donations, or endowment payoutsWhere will 

project obtain resources needed to cover costs? 

payouts.  

*This chart is adapted with kind permission from the Ithaka S&R Framework for Post­Grant Sustainability Planning.

12 Case Studies, Many Lessons

For an “index” of the topics covered in thetopics covered in the 12 case studies and case study updates, check out our

BRIEFING PAPERonon

www.ithaka.org

Thank you.

Nancy Maron, nancy.maron@ithaka.org

Kate Wittenberg, kate.wittenberg@ithaka.org

ATE Targeted Research Reportg pWayne Welch, University of Minnesota (ret.)

Assessing the Sustainability of the Advanced TechnologicalAdvanced Technological Education (ATE) Program

(November 2011)

Purposes of this presentation

1. Explain how we measured sustainability.

2. Describe what we learned.

Background

NSF/ATE Umbrella grant, 2008U i it f C l dUniversity of ColoradoLiesel Ritchie, PIO f i j tOne of nine projectsWayne Welch, Co-PI

University of Minnesota Rainbow Research IncUniversity of Minnesota, Rainbow Research, Inc.Advisory Committee: 3 ATE PIs, 3 test experts

How we measured sustainability

1. Identify what is to be measuredD fi th l t2. Define the elements

3. Develop measures4 Review and pilot test4. Review and pilot test 5. Select population6. Distribute and follow-up6. Distribute and follow up7. Process and check data8. Analyze9. Report

1. Identify what is to be measured

Th t i bilit f thi th t i i● The sustainability of things that remain in place after ATE grant funding ends

• Sustainability: continuation or persistenceThi li h t ti iti• Things: accomplishments, activities, changes, materials, graduates

2. Define the elements

Literature reviewAdvisory committee InterviewsWMU survey

Define the Elements: WMU Survey

WMU Annual Survey Question (2008)“Imagine your institution in the future,Imagine your institution in the future,

perhaps 3 years after the ATE grant has ended: What things started during the grant will be or are still in place?”

Define the Elements: PI statements

“Teaching methods adapted by our faculty as part of ATE ill ti t b d ”ATE will continue to be used.”

“Changes made in our technological education C a ges ade ou tec o og ca educat oprogram will keep going after our current grant ends.”

“The recruitment efforts developed through the grant will be incorporated into the college’s generalwill be incorporated into the college s general recruitment activities.”

D fi El t St t tDefine Elements: Statements were made about these components

• Programs• Collaborations• Educational Materials• Faculty• Facilities• Students• Institution• Revenue

Develop measures: Quoted statements

Indicate the extent to which you Disagree or Agree …

“Changes made in our technological education program will continue after our ATE grant has ended.” (Circle one)ended. (Circle one)

AA Strongly AgreeA Agree U U t iU Uncertain D Disagree DD Strongly DisagreeNA Not Applicable

Review and Pilot Test

● First draft 65 statements● Expert review 35 statements● PI review 28 statements● Final version 23 items (10 minutes)

● Named “Peer-Generated Likert Items”

Select Appropriate Population

● Active grants at least one year oldG t i d b t 2007 d 2009● Grants expired between 2007 and 2009

● Remove duplicates, planning grants, etc.

● Final population: 261 projects and centers

Mail Survey and Conduct Follow-Ups

● Mailed initial packetFi t f ll R i d l tt● First follow-up: Reminder letter

● Second follow-up: Reminder e-mail● Third follow up: Re mailed packet● Third follow-up: Re-mailed packet

(Approximately two weeks between contacts)

Process and check data for accuracy

● Received 212 completed surveysR t f 81 2%● Response rate of 81.2%

● Conducted a nonresponse bias study:Found no evidence that respondentsFound no evidence that respondents differed from nonrespondents

What we learned

● Computed percent Strongly Agree and Agree for positively stated itemspositively stated items

● Example: “Changes made in our technological● Example: Changes made in our technological education program will keep going after our current grant ends.”

● 90.9 % Agreed or Strongly Agreed

What we learned

● Computed percent Strongly Disagree and Disagree for negatively stated itemsDisagree for negatively stated items

● Example: “Most of the programs and activities● Example: Most of the programs and activities started during our ATE grant will come to an end when our NSF funding stops.”

● 78.2 % Disagreed or Strongly Disagreed

Mean % affirmation by element(N of statements per element in parentheses)

What we learned

ATE PI ill id h i f ti● ATE PIs will provide research information● Evidence of considerable sustainability● Difficult of obtain non NSF revenue● Difficult of obtain non-NSF revenue● Focus on expected success areas ● Have reasonable expectationsHave reasonable expectations

Areas of further study

• Yea-saying response bias• Handling “Not Applicable” responses• Additional validation evidence• Using survey as a sustainability check list

Concluding Remarks

● Described method to measure sustainabilityP G t d Lik t It● Peer-Generated Likert Items

● Presented evidence of ATE sustainability

● Working on new grant for continued research● Group comparisons● Predictor variables● Scale scores● Survey validation● Survey validation

Want more information?

● Welch Report 1: The Sustainability of ATE Program. Available at these web sites.

● http://evalu-ate.org/

● http://www.colorado.edu/ibs/decaproject/

Questions

• Does this information match your experiences?

• What problems have you encountered in t i i t?sustaining your grant?