Securing External Federal Funding Janice F. Almasi, Ph.D. Carol Lee Robertson Endowed Professor of...

Securing External Federal Funding

Janice F. Almasi, Ph.D.

Carol Lee Robertson Endowed Professor of Literacy

University of Kentucky

[email protected]

Institute of Education Sciences2 Grant Competitions per year

Janice F. Almasi, Ph.D., University of Kentucky

Current Funding Opportunities14 Long-term Programs of Research

Drag picture to placeholder or click icon to add

Be InformedSubscribe to Newsflash at ies.ed.gov/newsflash


IES Research Goals

Goal 1: Identification Identifying programs and practices associated with

better educational outcomes (secondary data analysis)

Goal 2: Development Projects Developing educational interventions

Goal 3: Efficacy and Replication Projects Determine if fully-developed interventions are effective

Goal 4: Scale-Up Goal 5: Measurement Projects


Prior to Peer Review Meeting

Triage identifies top 25 applications

Reviewers read, rate about 8 applications

Reviewers check for COIs

Each application assigned to at least 2 reviewers


Criteria

Significance

Research Plan

Personnel

Resources


Review Criterion Ratings(M

ore

Wea

knes

ses

than

St

reng

ths)

(Bal

ance

of

Stre

ngth

s an

d

Wea

knes

ses)

(Mor

e St

reng

ths

than

Wea

knes

ses)

1

Poor Excellent

765432


What Reviewers Look ForWhere Applications Tend to be Weak


Significance

Goal 1

• Theoretical and empirical rationale for study and practical importance of the intervention (e.g., program, practice) that will be examined

Goals 2 and 3

• Describe (a) the intervention (e.g., features, components) and the logic model for the intervention, (b) theoretical and empirical support for intervention, and (c) practical importance of the intervention


Significance in Goals 2 and 3

Context for Proposed Interventions Provide context for the proposed intervention by including

data on, or reviewing research describing, the attributes of typical existing practices.

Identify shortcomings of current practice and how they contribute to the rationale for the proposed intervention.

Provide context for understanding how much of a change the proposed intervention is intended to achieve.



Intervention, Theory of Change, Empirical/Theoretical Rationale Clearly describe the intervention Clearly describe the theory of change for the intervention

How do the features or components of the intervention relate to each other temporally (or operationally), pedagogically, and theoretically (e.g., why A leads to B)?

Provide a strong theoretical and empirical justification for the design and sequencing of the features or components of the intervention. Enables evaluation of: Relation between the intervention and its theoretical and

empirical foundation (e.g., is the proposed intervention a reasonable operationalization of the theory?)

Relation between the intervention and the outcome measures (e.g., do the proposed measures tap the constructs that the intervention is intended to address?)

Include a Logic Model



Practical Importance of Intervention When the proposed intervention is fully developed will it

have the potential to improve student outcomes in educationally meaningful increments, if it were implemented over the course of a semester or school year?

Would the proposed intervention be both affordable for and easily implemented by schools (e.g., not involve major adjustments to normal school schedules)?


Rese

arch

Pla

n:

Goal 2

Sample• Samples and settings used to

assess feasibility of intervention and for pilot data assessing promise of intervention

Iterative Development Process• Revision• Implementation• Observation• Revision

How do you define“operating as intended?”

What data will begathered to determine

how intervention isoperating?

How will the datagathered be used

to revise the intervention?

What criteria will be used to

determine if the intervention operates

as intended?


Rese

arch

Pla

n:

Goal 2

Feasibility of Implementation• Goal is a fully developed intervention• Data that addresses feasibility of

implementing in small sample of authentic education settings

• Promise of intervention in terms of outcomes

Pilot Study• Pilot data on outcome measures progressing

in right direction• Pilot data demonstrates implementation of

intervention is associated with behaviors consistent with theory of change

• No more than 30% of funds• Data should not be a test of efficacy


Rese

arch

Pla

n:

Goal 2

Measures• Clearly describe procedures for

gathering data to refine and revise the intervention and provide insight into feasibility and usability of proposed intervention• What needs to be observed?• How will observations be

gathered?• Clearly describe measures that will

be used (and reliability and validity if appropriate)


Reearch

Pla

n:

Goal 3

Research Questions• Pose clear, concise hypotheses

or research questions

Sample• Define sample to be selected• Define sampling procedures

(including justification for inclusion and exclusion)

• Strategies to be used to reduce attrition


Reearch

Pla

n:

Goal 3

Research Design• Provide detail!• How will threats to internal/external validity be

addressed?• Studies using random assignment are preferred

where feasible• What is unit of randomization and what

procedures will be used to make assignments to conditions?

Power• What power is needed to detect a reasonably

expected and minimally important effect?• How was effect size calculated?• If clusters are randomly assigned to treatment

conditions be sure to include intraclass correlation and anticipated effect size in power analysis


Rese

arch

Pla

n:

Goal 3

Measures• Justify appropriateness of measures• Are measures of practical interest to

educators and not overly aligned with intervention?

• Include reliability and validity informationFidelity of Implementation• How will implementation be documented and

measured?• How will factors associated with fidelity be

identified and assessed?• How will fidelity data be incorporated into

analyses of impact?• How do conditions in the school setting affect

fidelity of implentation?


Rese

arch

Pla

n:

Goal 3

Comparison Group• How does comparison group compare to intervention

on critical features of intervention?• Using a “business-as-usual” comparison is

acceptable but explain why using it is acceptable• How will contamination be avoided?

Mediating and Moderating Variables• Observational, survey, or qualitative methods are

encouraged to help identify factors that may explain the effect or lack of effect of intervention

Data Analysis• Quantitative: Specify statistical procedures and

include formulas where appropriate• Qualitative: Specific methods used to index,

summarize, and interpret data should be identified• Relation between hypotheses, measures, and

independent and dependent variables should be clear


Personnel

What role will each individual have in the project?

What qualifications, training, and experience do key personnel possess?

How will qualifications be used on the research?

Are key personnel dedicating sufficient time to competently implement proposed research?


Resources

Are the resources adequate to support the proposed activities in terms of: Facilities Equipment Supplies Institutional Support for Managing/Directing Grants and

Supporting Scholarship

Have partners shown support for implementation and support of the project?


Scientist Reviewer Critiques

A brief description of the overall application.

Identify each application’s key strengths and weaknesses in each of the evaluation areas and prepare critical, evaluative comments.

Integrated summary of the overall assessment of the application, including the main strengths and weaknesses of the application.


Overall Score

Overall Score Range Adjectival Equivalent

1.0 - 1.5 Outstanding

1.6 - 2.0 Excellent

2.1 - 2.5 Very Good

2.6 - 3.0 Good

3.1 - 4.0 Fair

4.1 - 5.0 Poor


Peer R

evie

w M

eetin

g P

roce

ss3. Notetaker

Summarizes the discussion orally and in writing

2. Full PanelDiscusses the application, asks questions, and offers additional

critiqueDiscusses the budget

1. Assigned Scientist Reviewers Share Overall ScoresReviewer 1 Summarizes the

application and its strengths and weaknesses in each

Reviewer 2 elaborates on areas of agreement or disagreement


Peer R

evie

w M

eetin

g P

roce

ss6. Assigned Scientist Reviewers

May edit/revise their original written critiques based on panel discussion

5. Full PanelPrivately assigns criteria scores and overall

scores

4. Assigned Scientist ReviewersAdjust initial recommended criteria scores and

overall scores

Securing External Federal Funding Janice F. Almasi, Ph.D. Carol Lee Robertson Endowed Professor of...

Documents

Transcript of Securing External Federal Funding Janice F. Almasi, Ph.D. Carol Lee Robertson Endowed Professor of...