Overview of Education Research & Development Centers Competition (84.305C)

ies.ed.gov

Connecting Research,Policy and Practice

Overview of Education Research & Development Centers Competition

(84.305C)

Rebecca McGill-Wilkinson, Ph.D.James Benson, Ph.D.

Erin Higgins, Ph.D. & Jonathan Levy, Ph.D.National Center for Education Research

ies.ed.gov

Organizational Structure of IES

2

National Board for Education

SciencesStandards &

Review Office

Office of the Director

National Center for Education Evaluation

National Center for Education Statistics

National Center for Education Research

National Center for

Special Education Research

ies.ed.gov

Missions of the Research Centers

• NCER– supports rigorous research that addresses the nation’s

most pressing education needs, from early childhood to adult education

• NCSER– sponsors a rigorous and comprehensive program of special

education research designed to expand the knowledge and understanding of infants, toddlers, and students with or at risk for disabilities from birth through high school

3

ies.ed.gov

IES Grant Programs: Research Objectives

• Develop or identify education interventions (practices, programs, policies, and approaches) – that enhance academic achievement– that can be widely deployed

• Identify what does not work and thereby encourage innovation and further research

• Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness

4

ies.ed.gov


Education Research & Development Centers (84.305C)

ies.ed.gov 6

Research & Development Centers: Purpose

• Contribute to solving U.S. education problems, improve the education system, and increase student achievement through– Research – Development – Evaluation – National leadership

ies.ed.gov 7

Research & Development Centers: Activities

• Focused research program– Contribute to solution of specific education problem within

the Center’s topic– Generate new knowledge and theories

– Detailed plan of focused program required– 75% of activities should be for focused program of

research

ies.ed.gov 8


• Supplemental activities– Meetings, smaller-scale studies– Work cooperatively with the Institute to select and design

– No detailed plan needed– Show capacity for conducting supplemental activities– Set aside up to 5% of budget annually

ies.ed.gov 9


• Leadership and outreach– Engage with researchers, practitioners, and policymakers– Website with reports, papers, etc.– Innovative uses of technology (e.g., webinars, podcasts)

– Describe audiences and activities– Show capacity for leadership and outreach

ies.ed.gov 10


Research & Development Center on Knowledge Utilization

ies.ed.gov 11

Purpose

• Research– Develop tools for observing and measuring research use in

schools– Illuminate the conditions under which practitioners use

researcher and factors that promote or inhibit research use in schools

– Identify strategies that make research more meaningful to and impactful on education practice

• Leadership and outreach– Demonstrate effective means of knowledge transfer to

improve school performance and student outcomes

ies.ed.gov 12

Background

• Findings from high-quality research and evaluation on impact of education policies and interventions

• Concern that this information is not used by practitioners– To adopt interventions with beneficial effects on students– To discard interventions with adverse or null effects

ies.ed.gov 13

Background: Potential Causes

• Misalignment between research questions (e.g., does it work?) and practitioners’ concerns (e.g., how do I use it with my students?)

• Misunderstanding of social context of practitioners– Research use is multilevel phenomenon (e.g. involves

districts, school leaders, and teachers)• Need new ways of working with practitioners– Partnerships– Bidirectional model of relationship– Intermediary organizations

ies.ed.gov 14

The Center’s Focused Program of Research

• One measurement study– Develop (or modify) and validate tools for observing and

measuring research use in schools and school districts• Two descriptive studies– Understand the conditions under which research is used

and factors that promote or inhibit research use in schools– Identify practices that are associated with greater use of

research in schools and school districts

ies.ed.gov 15

The Center’s Focused Program of Research

• Option to conduct Development Study or Efficacy Study– As alternative to one or both descriptive studies

• To develop a pilot intervention to improve research use in schools

• Or to conduct an efficacy study to assess the impact of a particular practice on the use of research in schools or school districts

• Study may range along a continuum of rigor

ies.ed.gov 16

Significance of Center

• Significance section at start of Research Narrative• Discuss the significance of your proposed Center

– Your vision of the Center – What problems it is intended to address– How your set of research and leadership activities will work

together to promote the use of scientific education research in schools and districts

• Discuss significance of three studies in focused research program

• Describe how you will engage researchers and practitioners in the work and communication of findings

ies.ed.gov 17

The Measurement Study: Research Questions

• What are the right indicators to know if school staff are using education research in their decision-making?

• How to capture variation in research use across schools, school district, and over time?

• You may expand or add additional research questions (and should provide justification for doing so)

ies.ed.gov 18

The Measurement Study: Methodological Requirements

• Methods of developing (or modifying) and validating assessment(s)

• Approach for gathering psychometric evidence• Sample and setting• Characteristics, size, and analytic adequacy of the

sample, including justification for inclusion and exclusion criteria

• Timeline

ies.ed.gov 19

Descriptive Study 1: Research Questions

• What role do State and local education agencies play in making research available to school administrators, faculty, and staff?

• What role do school leaders play?• How do education agencies, school leaders, and teachers obtain

and process scientific information?• How do social networks within schools and school districts appear

to support or hinder research use?• What are the major incentives or disincentives for practitioners to

better understand and apply education research?• What contextual factors seem to affect the use of research in

schools and school districts?• You may expand or add additional research questions (and should

provide justification for doing so)

ies.ed.gov 20

Descriptive Study 2: Research Questions

• Do certain communication strategies used by researchers or intermediaries correlate with greater use of research by practitioners?

• Is the presence or absence of intermediaries associated with greater use?

• Are particular technological innovations associated with greater use of research by practitioners?

• Are partnerships between researchers and practitioners associated with greater research use? To what extent do researchers perceive partnerships with practitioners as an enhancement or burden to their work?

• You may expand or add additional research questions (and should provide justification for doing so)

ies.ed.gov 21

Descriptive Studies: Methodological Requirements

• Population from which you’ll select your sample and sampling approach

• Measures• Data collection and analysis plan• Timeline

ies.ed.gov 22

Optional Development or Efficacy Study: Methodological Requirements

• Clear description of intervention to be developed or evaluated, including a theory of change

• Research questions• Sample and setting• Study designs may fall along a continuum of rigor but

efficacy studies should meet What Works Clearinghouse evidence standards

• Detailed power analysis

• Read Goal 2 and Goal 3 of 84.305A RFA for more details on methodological requirements

ies.ed.gov 23

Optional Development or Efficacy Study: Methodological Requirements

• Measures, including of fidelity and comparison group practices

• Key moderators and mediators• Plan to analyze implementation fidelity and

comparison group practices• Data analysis plan• Timeline

ies.ed.gov 24

Leadership & Outreach

• Innovative approaches to fostering dialogue among practitioners and researchers

• You may coordinate leadership/outreach with annual conferences to save time/money

• Institute encourages website• Institute open to learning how other applied social

sciences encourage research use among practitioners

ies.ed.gov 25

Maximum Awards

• 5 years• $5,000,000• Maximum of one award• Focused program of research must comprise at least

75% of the total budget

ies.ed.gov 26


Research & Development Center on Standards in Schools

ies.ed.gov 27

Motivation

• Standards being rapidly deployed• >40 states currently implementing the Common Core State

Standards• Others implementing some form of college- and career-

ready standards• Little known about the actions that districts are taking to

implement the standards and the influence that the standards are having in schools and classrooms

ies.ed.gov 28

Motivation

• Need for a Network– To coordinate a research agenda on standards that

addresses the needs of practitioners and policymakers for timely information

– To coordinate efforts of researchers from multiple areas of expertise

ies.ed.gov 29

Purpose

• Timely and accurate information– The Center will be a source for unbiased information

regarding implementation of the standards. • Leadership, Outreach and Networking

– The Center will bring together researchers, practitioners, and policymakers who are actively engaged in standards-related research, practice, and policy development.

• Assess Early Outcomes of College- and Career-Ready standards – The Center will be at the forefront of assessing whether the

standards are making a difference, and for whom.

ies.ed.gov 30

Main Focus Areas for the Center

The Center must carry out three focused activities: 1. An Implementation Study2. A Longitudinal Outcomes Study 3. A Leadership and Outreach Program

The Center may also elect to conduct a Measurement Study.

ies.ed.gov 31

Implementation Study Goals

• How is implementation taking place?– What strategies are SEAs and/or LEAs taking to implement

college- and career-ready standards?– How do these strategies vary across sites? – How are implementation strategies seeking to address the

needs of key student groups including ELLs and students with disabilities?

– What changes, if any, are college- and career-ready standards bringing about in schools and classrooms?

ies.ed.gov 32

Implementation Study

• RFA encourages an expansive view of implementation, including several key dimensions, such as: – State, district and school leader support for implementation– Professional development for teachers – Curriculum materials– Teacher practice – Student assessment – All of the above as they pertain to ELLs and students with

disabilities – Broader contextual factors that may affect implementation

ies.ed.gov 33

Longitudinal Outcomes Study

• OPTIONS: Exploratory or Efficacy Study

• Twin goals 1. Assess the relation between standards implementation

and student outcomes2. Assess the factors that explain variation in student

performance across sites

ies.ed.gov 34

Possible Research Questions

• Should reflect the broad objectives for the Center

• As CCRS standards are implemented… – Do student outcomes improve or worsen as CCRS are

implemented, including for sub-groups? – Do achievement gaps between sub-groups improve or

worsen over the study period? – Which factors (implementation or contextual) moderate

the relation between the standards and student outcomes?

ies.ed.gov 35

Leadership & Outreach Activities

The Center should be a source of objective information for a variety of audiences from federal policymakers to local audiences.

• Minimum requirements – Provide specifics about the targeted audiences and

strategies to reach those audiences– Maintain a website– Convene researchers and stakeholders to discuss research

findings and needs

ies.ed.gov 36

Application Sections

• Significance of the Focused Program of Research• Research Plan for the Focused Program of Research• Plans for Other Center Activities (Leadership &

Outreach)

• Management and Institutional Resources• Personnel

ies.ed.gov 37

Application: Significance Section

• Opportunities and Challenges posed by college- and career-ready (CCR) standards– Major research issues in need of attention– How will your approach address these issues, and the

goals of the Center?• Conceptual Framework and Literature Review– What is your conceptual framework? – How is it grounded in theory on standards?– How will it guide research design, data collection and

analysis plans?

ies.ed.gov 38

Application: Significance Section

• Sites– Describe geography, school setting, student population,

implementation of standards– How will findings inform policy discussions and analysis? – Describe partnerships with SEAs, LEAs

• Letters of agreement (Appendix D)

• Research Agenda– Areas of focus for your Implementation Study– Research questions for your Outcomes Study– Justification for measures

• IF you propose measure development

ies.ed.gov 39

Application: Research Plan

Implementation Study– Sample selection, including population and sampling

method– Measures you will use– Data collection strategy– Analysis plan– Timeline

ies.ed.gov 40


Longitudinal Outcomes Study• Exploration Study

– Population and sample selection– Measures you will use; student outcomes– Data collection and analysis plans– Timeline

• Efficacy Study – All of the above– Research design to produce impact estimates– Power analysis – Data analysis procedures– Dissemination plan

ies.ed.gov 41


Optional Measurement Study• Provide a rationale for a new measure or substantial

modification to an existing measure– Instructional practice– Classroom activities– Opportunities to learn– Alignment of instruction, classroom activities, or curriculum

materials to college- and career-ready standards

ies.ed.gov 42

Application: Leadership and Outreach

• Target Audiences: Who will you be speaking to and what will be the key topics for each group?– National audiences (including federal policymakers and

national organizations)– Local audiences where the research will be conducted

• Role of practitioners in reviewing findings• Website (required) • Other Dissemination Venues – Products for downloading– Social media

ies.ed.gov 43

Maximum Awards



ies.ed.gov 44


Research & Development Center on Virtual Learning

ies.ed.gov 45ies.ed.gov

Virtual Learning Lab (VLL): Background

• Many open questions:– Whether education technologies lead to meaningful

improvements in teaching and learning– How “big data” can be used to improve teaching and

learning

• Infrastructure is needed to:– Support the analytic and instructional use of the data– Store and share these data with others, and to mine data

generated from the use of education technologies


VLL: Broad Purpose

• Evaluate and improve instructional practices, content, and/or learning tools offered by one or more widely used online instructional delivery platforms– Focus: Improving outcomes of students in K-12, particularly

low-income and low-performing students– One platform must be for K-12 students, but additional

platforms can be for either K-12 or postsecondary students

• Address how large amounts of data generated within these platforms can be used to improve teaching and learning


VLL: Broad Purpose

• Inform stakeholders about the use of "big data" for education research and practice

• Create a “hub”

• Offer workshops and other activities


Definition of “Widely Used Online Instructional Delivery Platform”

• The technological infrastructure used to: – Deliver any of a number of education technologies, and – Collect large amounts of information about students

• Platform(s) must be used across multiple states, in the majority of districts in a single state, or in the majority of schools in one or more large districts


Definition of “Widely Used Online Instructional Delivery Platform”

• The online instructional delivery platform(s) you choose may be used to provide any of the following:– Online courses offered by a state, school district, or

postsecondary institution – Online courses with open enrollment (i.e., MOOCs)– Online schools or certificate/degree programs– Intelligent tutors– Blended learning models (combine online/classroom

instruction)– Flipped models (instruction online and homework/other

activities during class)


Examples of Research Questions

• How can online and blended learning be used to address the instructional needs of learners across the spectrum – typical, advanced, and struggling – more effectively?

• How can online instruction be designed and implemented to promote the learning and retention of material, not just in the short term, but also over longer school-relevant time periods?

• How can online courses be designed so that students are more likely to persist in and complete the instruction?

• How can online instruction be targeted and delivered to meet the needs of low-income students, and to reduce achievement gaps between higher- and lower-performing students?

50


Examples of Research Questions

• What data, metadata, analytical tools, and models are most useful for understanding and improving student learning?

• How can "big data" be used to predict students’ performance and help them receive the instruction and services they need?

• What practices, protocols, and policies should researchers and Institutional Review Boards (IRBs) be attentive to as they strive to protect students’ privacy?

• Which research and academic skill sets are needed to effectively conduct this type of work, and what training can be employed so others can develop the necessary skills?


Important Partnerships

• The Institute strongly encourages partnerships between: – Researchers– Platform developers– Practitioners within states, school districts, and/or colleges

and universities who use online instructional delivery platforms.

• Must attach letters of agreement from all the key partners in Appendix D


Project Narrative Requirements

• Significance of the Focused Program of Research• Research Plan for the Focused Program of Research• Leadership & Outreach Activities


Significance of the Focused Program of Research

• Explain vision for the VLL• Identify problems or issues that the VLL will address– Review of research literature

– Provide data to support your argument (if possible)

• Discuss significance of studies proposed in the Research Plan– Explain how you will ensure generalizability of findings


Significance of the Focused Program of Research

• Must identify at least ONE widely used online instructional delivery platform

• Platform must be intended for use by students in an authentic K-12 education setting – Additional platforms may be intended for use either by K-

12 or with postsecondary students


Overview of Remainder of Presentation

• Components of the focused program of research– Experimental studies (min = 2)– Measurement studies (min = 1)

• "Big data"• Leadership and outreach activities


Focused program of research:Experimental Studies

• At least TWO experimental studies– More can be proposed, depending on their scope

• Focus– Examination of strategies to improve instructional

practices, content, and/or learning tools within online delivery platforms

• Particular concern – Differentiation for individual learning needs


What can be studied experimentally?

• Intervention that are…– Fully developed? – still in the development process?

• At least ONE study should be designed with the intention of improving an education technology that is in wide use


What can be studied experimentally?

• Affordances in a digital environment– Easy, quick introduction and changes, based on findings– Support of timely decision making

• Comparison study of different iterations or versions of the intervention


Measurement study (min = 1)

Goals• Explore/create valid and reliable measures• Determine which measures useful in differentiating

student learning needs• Identify mediating variables• Examine strategies to prepare data for user-friendly

consumption


Research Plan: Experimental Studies

• Purpose: To help education technology developers and education practitioners make improvements in the instructional practices, content, and/or learning tools they are using

• See enumerated list (RFA, p. 26)


Research Plan: Data Collection

Describe…• Sources of data

– Digital interactions– Administrative records– Surveys, Observations, and Interviews

• Methodology– Sampling strategy– Data collection plans– Analytic approach

• Rationale for each choice


Studying a fully developed technology

• What are the critical questions?



Research Plan: Measurement studies

• Purpose: What is the study’s justification?



Developing and validating NEW measures

Describe iterative process…

• Field testing procedures & processes for item revision

• Validation activities, including types of evidence to be used for reliability and validation for the specified purpose, populations, and contexts

• See Measurement Goal section in the RFA 84.305A


Concerns with "big data" and education

• Issues, concerns and implications on multiple levels– Practical, Ethical, and Legal

• Confidentiality, protection of PII – (personally identifiable information) from unauthorized

disclosure• Data access– Who should have access to data? For which purposes?

• Sharing, linking• Security rules and protocols


Concerns with "big data" and education

• Negative side-effects of labeling – e.g., he’s a slow learner

• Unintended consequences


Leadership & Outreach Activities

• Role of the VLL: Function as a reliable source• For Whom?– Technical: Researchers & developers– Broader audience: Teachers, administrators, policymakers,

general audience


Dissemination

• What: Publications, products• How: Describe efforts planned efforts at

dissemination– Website (at a minimum)– Other new media (podcasts, video, webinars) that are

relatively inexpensive yet can reach a large audience


Hub for Training & Research (some ideas)

• Researchers, developers, and practitioners will come together (virtually and in person)

• Discuss research goals and methods related to online learning

• Review emerging findings– Foster new research partnerships and collaborations:

(e.g., hosting non-Lab researchers, developers, etc., to work on Lab projects)

– Provide training and education: Host summer workshops, conduct courses for graduate students, non-Lab researchers, practitioners, and others


Hub for Training & Research (some ideas)

• Organizing visits by state or school district leaders to demonstrate how education technologies are being used,

improved, and having an effect

• Your suggestion(s) here: ______

ies.ed.gov 72

Maximum Awards



ies.ed.gov 73


General Requirements

ies.ed.gov 74

Additional Sections of the Research Narrative for All Centers

• In your 35-page Research Narrative, you are to discuss– Significance– Research Plan for the Focused Program of Research – Other Center Activities (Leadership and Outreach)– Management and Institutional Resources– Personnel

• We’ve discussed the first three and now we’ll move to the last two

ies.ed.gov 75

General Requirements: Management & Institutional Resources

• Describe plans and procedures for overall management of the Center and its activities

• Show capacity of supplemental activities and leadership and outreach

• If plan is to work in schools or other education delivery settings in first year, must include letter of agreement from education organization(s)

• If developer/distributer of program or intervention is part of Center, describe role and how objectivity will be maintained

ies.ed.gov 76

General Requirements: Personnel

• Expertise in content area(s)• Methodological expertise• Sufficient experience working with education

delivery settings• Capacity for supplemental activities• Experience relevant to national leadership– Explain why proposed Center staff is qualified to fulfill

leadership role

ies.ed.gov

Important Dates and Deadlines

77

Application Deadline

Letter of Intent Due

Date

Application Package Posted

Start Dates

August 7, 2014

4:30:00 PM DC Time

June 5, 2014 June 5, 2014 July 1, 2015to

Sept 1, 2015

ies.ed.gov

Peer Review Process

• Applications are reviewed for compliance and responsiveness to the RFA

• Applications that are compliant and responsive are assigned to a review panel

• Two or three panel members conduct a primary review of each application– Most competitive applications move to full panel

• At panel meeting, the most competitive applications are reviewed by full panel

78

ies.ed.gov

Notification

• All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS):• Status of award• Reviewer summary statement

• If you are not granted an award the first time, consider resubmitting and talking with your program officer

ies.ed.gov 80

Questions?

http://ies.ed.gov/funding

Knowledge UtilizationBecky McGill-Wilkinson ([email protected])

Standards in SchoolsJames Benson ([email protected])

Virtual LearningJonathan Levy ([email protected])

Erin Higgins ([email protected])

http://ies.ed.gov/funding

mailto:[email protected]




Overview of Education Research & Development Centers Competition (84.305C)

Documents

Transcript of Overview of Education Research & Development Centers Competition (84.305C)