May 29, 2014 Elizabeth R. Albro, Ph.D. Associate Commissioner of Teaching & Learning

ies.ed.govies.ed.gov

Connecting Research,Connecting Research,Policy and PracticePolicy and Practice

May 29, 2014

Elizabeth R. Albro, Ph.D.Associate Commissioner of Teaching & Learning

National Center for Education Research

Jacquelyn A. Buckley, Ph.D.National Center for Special Education Research

@IESResearch

IES Grant Writing Workshop


Purpose of the Workshop

This workshop will provide instruction and advice on writing a successful application to the Institute of Education Sciences’research grant programs, specifically to the:

•Education Research Grants Program (84.305A)•Special Education Research Grants Program (84.324A)


Grant Writing Is A Process

3


Agenda

• Introduction to IES• Grant Writing Tips• General Requirements• Grant Research Topics• Grant Research Goals • Four Sections of the Project Narrative

4


What is IES?

• Research arm of the U.S. Department of Education, non-partisan by law.

• Charged with providing rigorous and relevant evidence on which to ground education practice and policy and share this information broadly.

• By identifying what works, what doesn't, and why, we aim to improve educational outcomes for all students, particularly those at risk of failure.


IES Organizational Structure

Office of the Director

National Board for Education

Sciences

National Center for Education Evaluation

National Center for Education Statistics

National Center for Education Research

National Center for

Special Education Research

Standards & Review Office

6


FY 2015 Funding Opportunities

7


RFA Changes of Note

• Revised formatting, highlighting minimal requirements for an application to be sent forward for peer review

• Identified research gaps in each topic domain• Dissemination plans are required• Appendices have been modified• Submission Guide incorporated into each RFA• Glossary of important terms added

8


Primary Research Grant Programs

• Education Research Grants (84.305A)

• Special Education Research Grants (84.324A)

These grant programs are organized by research topic and research goal.

9


Agenda


10


What You Need to Do

Sell your research idea, promote yourself as the best person to do the research, and build

goodwill and trust.

How?

By demonstrating that you know what the problem is and have a way to address it.


Opening Paragraph

• Sets the scene for readers:– Identifies the significance of the work to be

done and what actually will be done– Readers use it to organize information in rest of

the application– You can lose your readers right off with an

unclear opening


Statement of Purpose

Should:– Be short and attention-getting – Contain the problem statement and your contribution

to solving it.

Your fellow researchers, friends, and family members should be able to understand it and see its relevance.

NOTE: It’s not as easy to do as you may think. You have to know a lot about what it is you want to do before you can describe it succinctly.


Theory of Change (ToC)

• The model underlying your research. • A roadmap to your project narrative.• A source for generating research questions.• Constantly evolving.

NOTE: Some fields and scholars use terms like Logic Model or Logical Framework to mean things similar to what we are calling a Theory of Change here.


ToC & Your Research Plan

In your research plan, you need to specify exactly what it is you’re exploring, creating, validating, or testing. You

also need to specify how you will do these things.

Strategies/Activities What are the pieces that you’ll be exploring, creating, testing, etc.?Outcomes Indicators: What will you measure, and how you will measure it?

Populations: Who and where (both in treatment and control/comparison)?Thresholds: What effect (size) should you expect?Timeline: When should you be collecting what data?


Helpfulness of the Program Officer

• Share your framework and statement of purpose with the Program Officer– Ensure you are submitting to the correct

competition/topic – Springboard for further discussion


Clarity of Writing

• Readers (e.g., application reviewers) often complain about lack of clarity. – Significance too general – Lack of detail regarding intervention,

development cycle, or data analysis– Use of jargon and assumptions of knowledge– Poor writing (e.g., grammar), awkward

constructions, etc.


Resources for Researchers

• Visit links for faculty and researchers on IES websitehttp://ies.ed.gov/resourcesforresearchers.asp

• Review past webinars and participate in future webinars for the FY 2015 competitionshttp://ies.ed.gov/funding/webinars/index.asp

18


Agenda


19


All Proposed Studies Must

• Measure Student Education Outcomes• Be Relevant to Education in the U.S.• Address Authentic Education Settings• Specify 1 Research Topic• Specify 1 Research Goal

20


NCER Ultimate Outcomes of Interest: Student Outcomes

Grade OutcomePrekindergarten School readiness (e.g., pre-reading, language,

vocabulary, early math and science knowledge, social and behavioral competencies)

Kindergarten – Grade 12

Learning, achievement, and higher-order thinking in reading, writing, mathematics, and science; progress through the education system (e.g., course and grade completion or retention, high school graduation, and dropout); social skills, attitudes, and behaviors that support learning in school

21


NCER Ultimate Outcomes of Interest: Student Outcomes

Grade OutcomePostsecondary(Grades 13 – 16)

Access to, persistence in, progress through, and completion of postsecondary education; for students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics

Adult Education(Adult Basic Education, Adult Secondary Education, Adult ESL, and GED preparation)

Student achievement in reading, writing, and mathematics; access to, persistence in, progress through, and completion of adult education programs

22


NCSER Ultimate Outcomes of Interest: Student Outcomes

Grade OutcomeBirth - 5 Developmental outcomes and school readiness

Kindergarten – High School

Achievement in core academic content (reading, writing, mathematics, science), behaviors that support learning in academic contexts, functional outcomes that improve educational results and transitions to employment, independent living, and postsecondary education

23


Who Should Read the RFA?

• You, the proposed Principal Investigator• Your team members, including your co-PIs,

statistician, methodologist, developer• Your assigned sponsored projects officer• Anyone else participating in the preparation of

the application

24


Agenda


25


Grant Topics

• All applications to the primary research grant programs must be directed to a specific topic– Note on SF 424 Form, Item 4b (Agency

Identifier Number)– Note at top of Abstract and Project Narrative


Research TopicsEducation Research (305A) Special Education Research (324)

Cognition & Student Learning Cognition & Student Learning in SpEd

Early Learning Programs and Policies Early Intervention & Early Learning in SpEd

Education Technology Technology for Special Education

Effective Teachers & Effective Teaching PD for Teachers & Related Service Providers

English Learners Autism Spectrum Disorders

Improving Education Systems SpEd Policy, Finance, and Systems

Mathematics & Science Education Mathematics & Science Education

Postsecondary & Adult Education Transition Outcomes for Secondary Students with Disabilities

Reading & Writing Reading, Writing & Language Development

Social & Behavioral Context for Academic Learning

Social & Behavioral Outcomes to Support Learning

Families of Children with Disabilities


Topics: Nota Bene

• Must address student education outcomes• Grade range varies by topic• Your project might fit in more than one topic


305A: Topics and Their Grade Range

29


324A: Topics, Grade Level, Disability Status

30


Choosing among Overlapping Topics

• What literature are you citing?• To which topic is your area of expertise best

aligned?• If your focus is on a specific population of

students/teachers, go to that program/topic:– Is your focus on a specific type of

student/teacher (e.g., English Learners), or are you studying them as a subgroup of your sample?


New This Year: Research Gaps & Considerations

• Every topic now includes a section which describes research gaps and/or considerations that the Institute has identified.

32


Agenda


33


Grant Research Goals

• All applications to 84.305A/84.324A must be directed to a specific goal– Note on SF 424 Form, Item 4b– Note at top of Abstract and Research Narrative

• The goal describes the type of research to be done

• Every application is directed to a specific topic/goal combination


What Topic X Goal Fits Your Project?

35


Program Officer Role

• As you are thinking through which is the appropriate topic and goal for your project, we encourage applicants to reach out to your program officers.

• They are a valuable resource.

36


FY 2015 Research Goals

• Exploration • Development & Innovation• Efficacy & Replication• Effectiveness• Measurement

37


Goal Max. Duration & Award (direct + indirect)

Exploration W/ secondary data or meta-analysis With primary data

2 years, $700,0004 years, $1,600,000

Development & Innovation 4 years, $1,500,000

Efficacy & Replication Follow-up study Retrospective

4 years, $3,500,0003 years, $1,200,0003 years, $800,000

Effectiveness Follow-up study

5 years, $5,000,0003 years, $1,600,000

Measurement 4 years, $1,600,000

Maximum Award Amounts (84.305A)

38


NCER Grants by Goal (2004-2013)


NCSER Grants by Goal (2006-2013)

Percentage of Funded GrantsFY 2006-2013


Goal Requirements

• Your application must meet all Project Narrative and Award Requirements listed for the goal you select in order for your application to be considered responsive and sent forward to review.

• We strongly encourage you to incorporate the recommendations into your Project Narrative.

• All applications must include a Dissemination Plan.

41


Exploration Projects: Malleable Factors

• Malleable factors must be under the control of the education system– Something that can be changed by the system

• Examples– Student characteristics: behavior, skills– Teacher characteristics: practices, credentials– School characteristics: size, climate, organization– Education interventions: practices, curricula,

instructional approaches, programs, and policies


Development & Innovation Projects: Key Features

• Iterative development process• Well specified theory of change• Data collected on feasibility and

usability in authentic education settings

• Fidelity must be measured• Pilot data on student outcomes

43

Development process must be

iterative!


Efficacy & Replication Projects:Key Features

• Testing a causal question.• Ask what might be needed to implement

intervention under routine practice, even if you intend to test under ideal conditions

• Consider role of developer to avoid conflict of interest for developer-evaluators

• Do not require confirmatory mediator analyses but recommend exploratory ones

44


Effectiveness Projects: Key Features

• IES expects researchers to– Implement intervention under routine practice– Include evaluators independent of development/distribution– Describe strong efficacy evidence for intervention (from at

least 2 previous studies)

• Does not expect wide generalizability from a single study– Expects multiple Effectiveness projects to this end– Sample size is not a key distinction from Efficacy

• Does not require confirmatory mediator analyses but encourages exploratory ones

• Cost of implementation is limited to 25% of budget45


Measurement Projects: Key Features

• Assessments may also be developed in other goals, but not as the primary focus

• Primary product of measurement grant is the design, refinement, and/or validation of an assessment

• Include an assessment framework• Must link the assessment to student education

outcomes

46


Expected Products

• Expected Products for each goal can help you identify the right goal for your project

• At the end of a funded project, IES expects you to provide…


Agenda


48


Project Narrative• Four Required Sections

– Significance– Research Plan– Personnel– Resources

• Each of these sections will be scored individually by the peer reviewers

• In addition, reviewers provide an overall score of Scientific Merit

49


The Application’s Project Narrative

• Requirements vary by program & goal• READ THE REQUIREMENTS CAREFULLY• 25 pages, single spaced• Project Narrative is supported by Appendices, but all

critical content for reviewers should be included within the 25 pages of the Project Narrative.


Significance Section

• Describes the overall project– Your research question to be answered; intervention

to be developed or evaluated, or measure to be developed and/or validated

• Provides a compelling rationale for the project– Theoretical justification

• Theory of Change– Empirical justification– Practical justification


Significance

• Do not assume reviewers know significance of your work

• Do not quote back RFA on general importance of a topic, – e.g., RFA paragraph on lack of reading proficiency of 8th and

12th graders based on NAEP data

• Do quote back RFA if your project is addressing a research gap identified in the RFA – e.g., disproportionality in discipline (Social/Behavioral);

impact of early childhood policy initiatives (Early Learning)


Significance – 2 Key Problem Areas

1. Description of Malleable Factor/Intervention– Unclear what intervention is: confuses reviewers

• Many components and may be applied at different times – how fit together – Graphic may help

– Unclear how to be implemented to ensure fidelity– Intervention not shown to be strong enough to expect an impact

• Especially true for information interventions – e.g., provide data on students, short teacher workshops

– Overly focused on actions not content• Ex.: 20 hours of PD held over 10 weeks but no detail on what is to

be covered in the sessions


Significance – 2 Key Problem Areas

2. Theory of change– Why a malleable factor is expected to be related to a student

outcome– Why the proposed intervention should improve outcomes

versus current practice– Why an assessment/instrument will measure a specific

construct– When well laid out, a theory of change makes clear what is

expected to happen and in what order– Easy for reviewers to understand research plan – why

measure certain outcomes – Graphic can be helpful


Theory of Change Should Describe:

• How the intervention addresses the need and why it should work

– Content: what the student should know or be able to do; why this meets the need

– Pedagogy: instructional techniques and methods to be used; why appropriate

– Delivery System: how the intervention will arrange to deliver the instruction

• Which aspects of the intervention are different from the counterfactual condition

• Key factors or core ingredients most essential and distinctive to the intervention


Simple Theory of Change

56


Logic Model Graphics Don’t Do This!

• Overwhelm the reader• Use color as a key because applications are

reviewed in black and white


PLTProfile AnalysisSet instructional goals & WL focus

PLTProfile AnalysisSet instructional goals & WL focus

[4 weeks at end of prior year or beg of current

year]

PLTBegins weekly

meetings

PLTBegins weekly

meetings PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

PLT

WLDebrief

[Processes 4-11 repeat to mid-year]

[PLT appoints SLT1 to address PD Topic 1]

SLT 1Research

SLT 1Research

SLT 1Implement Prof Devt

SLT 1Implement Prof Devt

DEVELOPMENT MODEL FOR “WL” INTERVENTION

DEVELOPED BY “ABC” WITH THE ASSISTANCE OF “DEF”

PLT = Primary Leadership team

SLT = Secondary Leadership team

= Begin Process

= Feedback for next process

= Delegation of PD

KEY

10. ABC reviews/revises

model based on DEF findings



14a. SLT and Coaches

create PD unit

14a. SLT and Coaches

create PD unit

1a. PI Recruits

and Trains Coaches

1a. PI Recruits

and Trains Coaches

6a. Coaches facilitate PLT

identification of annual instr. goal

6a. Coaches facilitate PLT

identification of annual instr. goal

1b. DEF develops

data collection

tools

1b. DEF develops

data collection

tools

11. Coaches share/

implement revisions with PLTs

11. Coaches share/

implement revisions with PLTs

3a. Coaches collect 3 yrs. stud. ach. & demo. data per school

3a. Coaches collect 3 yrs. stud. ach. & demo. data per school

2a. ABC PI matches

Coaches to schools (n=5)

2a. ABC PI matches

Coaches to schools (n=5) 2b. DEF

trains coaches to

use all data tools

2b. DEF trains

coaches to use all

data tools

5. DEF guides Coaches and PLTs

in constructing and analyzing

profiles based on data collected

5. DEF guides Coaches and PLTs

in constructing and analyzing

profiles based on data collected

4. Coaches and DEF

work with PLT to

collect climate

data

4. Coaches and DEF

work with PLT to

collect climate

data

3b. DEF develops electronic

profile develop-ment tool

3b. DEF develops electronic

profile develop-ment tool

7a. Coaches teach PLT to

conduct Learning Walks (WL)

7a. Coaches teach PLT to

conduct Learning Walks (WL)

approx. 3 wks after school begins

8a2. Coaches assist PLTs in

weekly WLs and Debriefing

through mid-year


weekly WLs and Debriefing

through mid-year

6-8b. DEF observes PLTs,

documents implementation

6-8b. DEF observes PLTs,

documents implementation

9. DEF interviews

coaches and PLTs about WL process

9. DEF interviews

coaches and PLTs about WL process

approx. 9 wks after school begins

13a.ABC researchers train SLT to research

best-practices in PD area

13a.ABC researchers train SLT to research

best-practices in PD area

12. Coaches & PLTs choose 2-4 teachers (based on WLs) to become Dynamic

Leadership Team 1]

12. Coaches & PLTs choose 2-4 teachers (based on WLs) to become Dynamic

Leadership Team 1]

13-15b. DEF observes/consul

ts SLTs, documents

implementation

13-15b. DEF observes/consul

ts SLTs, documents

implementation

15a.Coaches assist SLT in

implementing PD with faculty

15a.Coaches assist SLT in

implementing PD with faculty

16. DEF interviews

coaches, SLTs, and PLTs about

PD and WL processes

16. DEF interviews

coaches, SLTs, and PLTs about

PD and WL processes






using all data to ID area for prof

devt


using all data to ID area for prof

devt

[Processes 3-17 repeat twice in Year 2]

18a. ABC and DEF submit

Annual Report to DOE and

schools

18a. ABC and DEF submit

Annual Report to DOE and

schools


Research Plan Section

• Describe the work you intend to do– How you will answer your research question; develop your

intervention; evaluate the intervention, or develop and/or validate your assessment

• Make certain Research Plan is aligned to Significance section– All research questions should have justification in

Significance

• Step-by-step process– A timeline is strongly recommended!


Identify Setting, Population, & Sample

• Identify the places you will be doing research• Identify the population you are addressing• Identify the sample

– Inclusion and exclusion criteria– Sample size (issues of power for analysis)– The importance of attrition and how to address it– External validity: can you generalize to your population or

only to a subset of it• If using secondary data, discuss these for the datasets you will

be using


Specify Your Outcome Measures

• For both proximal and distal outcomes• Sensitive (often narrow) measures• Measures of broad interest to educators• Measures not expected to be linked can be used as

additional evidence• Describe reliability, validity, and relevance• Do not include measures not linked to research

questions• Consider issue of multiple comparisons


Specify Features of All Other Measures

• Measures that feed back into iterative development process

• Fidelity of Implementation– Operating as intended– Able to address comparison groups

• Feasibility


Measures Derived From Qualitative Research

• Describe– Actual items to be used– How items link to constructs – the validity of these

measures– Procedures for collection and coding (address inter-rater

reliability)– How consent obtained for an adequate percent of sample– How qualitatively collected measures are used in analysis of

quantitative outcomes (e.g., test scores)


Measurement Projects

• Alternate forms – horizontal equating• Vertical equating, if measuring growth• Test fairness• Non-student instruments must be validated against

student outcomes

64


Analysis Depends on Design

• Describe how your analysis answers your research questions

• Describe analyses of qualitative data


Analysis (cont.)

• Show your model– Identify coefficients of interest and their meaning– Show different models for different analyses– Include Equations

• Address clustering• Describe plan for missing data – check for

equivalency at start and attrition bias• Use sensitivity tests of assumptions

66

ijk i j ij ijkY


Personnel Section

• Describe key personnel– Show that every aspect of project has person with

expertise to do it• Appropriate methodological expertise• Substantive person for all issues addressed• Do not propose to hire a key person with X expertise• Project management skills

– Show that every aspect of project has enough time from an expert

• Orient CVs so specific to project– 4 pages plus 1 page for other sources of support

67


Personnel Strategies for PI

• Senior Researcher as PI– Show adequate time to be PI– Make credentials clear (not all reviewers may know)

• Junior Researcher as PI– Show you have adequate expertise not only to do work but

to manage project• Continuation of graduate research• Management skills as graduate student

– Reviewers more comfortable if you have senior person(s) on project to turn to for advice

• Co-PI, Co-I, contractors, advisory board• Have them on for enough time to be taken seriously

68


Resources

• Show institutions involved have capacity to support work– Do not use university boilerplate

• Show that all organizations involved understand and agree to their roles– What will each institution, including schools, contribute

to the project– Show strong commitment of schools and districts and

alternatives in case of attrition• If you have received a prior grant award for similar

work, describe the success of that work

69


Resources (cont.)

• Appendix D should back up the Resources section

• Detailed Letters of Agreement from research institutions, States, districts, schools– Do letters show that partners understand their

role in the project (e.g., random assignment to condition; time commitments)?

– Do letters show that you have access to all necessary data to do the proposed work?

70


Budget and Budget Narrative

• Provide a clear budget and budget narrative for overall project and each sub-award

• Provide detail on the assumptions used in the budget (e.g., assumptions for travel)

• Budget categories are described beginning on pg. 102 of NCER RFA, pg. 104 of NCSER RFA

• Check RFA for specific budget requirements for Research Goals

• Ensure alignment among Project Narrative, Budget, and Budget Narrative

71


AppendicesAppendix Content to be included

A 3 page response to reviewers for resubmitted applications

B Figures, charts, tables that supplement project narrative; examples of measures to be used in the project (optional)

C Examples of materials to be used in the intervention or assessment that is the focus of your project (optional)

D Letters of agreement from partners, data sources, and consultants (optional)

E Data Management Plan (Required for Efficacy/Replication and Effectiveness Applications)

72


Application Deadline

Letter of Intent Due

Date

Application Package Posted

Start Dates

August 7, 2014

4:30:00 PM DC Time

June 5, 2014 June 5, 2014 July 1, 2015to

Sept 1, 2015

Important Dates and Deadlines

73


Finding Application Packages

• FY 2015 Application Packages will be available on www.grants.gov

74


Review Application Requirements

Request for Applications Currently available at http://ies.ed.gov/funding

Application Package Will be available on Grants.gov on 6/5/2014

75


Peer Review Process

• Applications are reviewed for compliance and responsiveness to the RFA

• Applications that are compliant and responsive are assigned to a review panel

• Two or three panel members conduct a primary review of each application

• The most competitive applications are reviewed and discussed by the full panel

76


Peer Review Process

• Peer Review of Grants Process– http://ies.ed.gov/director/sro/peer_review/applic

ation_review.asp

• Lists of Peer Reviewers– http://ies.ed.gov/director/sro/peer_review/revie

wers.asp

77


Help Us Help You

• Read the Request for Applications carefully

• Call or e-mail IES Program Officers early in the process

• As time permits, IES program staff can review draft proposals and provide feedback

Don’t be afraid to contact us!

78


Wrap-up and Final Q&A

Elizabeth AlbroAssociate Commissioner, Teaching and Learning, NCER

[email protected](202) 219-2148

Jacquelyn BuckleyNCSER

[email protected]

May 29, 2014 Elizabeth R. Albro, Ph.D. Associate Commissioner of Teaching & Learning

Documents

Transcript of May 29, 2014 Elizabeth R. Albro, Ph.D. Associate Commissioner of Teaching & Learning