NSF-EHR Framework for NSF-EHR Framework for Improving Improving Science, Technology, Science, Technology, Engineering and Mathematics Engineering and Mathematics

Education R & D Education R & D

Ferdie RiveraProgram Director (Part time; Intermittent Expert)

Division of Research on Learning in Formal and Informal Settings (DRL)Directorate for Education and Human Resources (EHR Directorate)

National Science Foundation (NSF)

National Science FoundationNational Science FoundationDirectoratesDirectorates

Social, Behavioral, and Economic Sciences

Biological Sciences

Engineering

Education and Human Resources

Computer and Information Sciences and Engineering

Mathematics and Physical Sciences

Geosciences

Directorate for Education and Directorate for Education and Human ResourcesHuman Resources

Division of Undergraduate Education Division of Undergraduate Education (IUSE; ATE; NOYCE; STEM-C)(IUSE; ATE; NOYCE; STEM-C)

Division of Graduate Education (GRFP; Division of Graduate Education (GRFP; IGERT; GK 12)IGERT; GK 12)

Division of Human Resource Development Division of Human Resource Development (ADVANCE; AGEP; HBCU; TCUP)(ADVANCE; AGEP; HBCU; TCUP)

Division of Research on Learning in Division of Research on Learning in Informal and Formal Settings (CORE; Informal and Formal Settings (CORE; ITEST; DR K 12; AISL; STEM-C) ITEST; DR K 12; AISL; STEM-C)

Characteristics of Strong STEM R & D ProjectsCharacteristics of Strong STEM R & D Projects

They essentially have a They essentially have a coherentcoherent and and deep scientific deep scientific researchresearch base that informs/guides efforts to meet base that informs/guides efforts to meet national STEM education and workforce development national STEM education and workforce development needs/prioritiesneeds/priorities

They address CORE They address CORE research issues research issues that address that address broad, essential, enduring, and challenges of great broad, essential, enduring, and challenges of great importance and are importance and are designed to inform the designed to inform the transformation of STEM learning and educationtransformation of STEM learning and education

R & D projects are potentially transformative and R & D projects are potentially transformative and innovativeinnovative

NSF EHR CORENSF EHR CORE

STEM LEARNING

STEM LEARNING ENVIRONMENTS

STEM WORKFORCE

DEVELOPMENT

BROADENING PARTICIPATION

IN STEM

Learning of specific STEM disciplinary Learning of specific STEM disciplinary knowledge and practices across different knowledge and practices across different groups of learners, contexts, and groups of learners, contexts, and environmentsenvironments

Learning includes cognitive, social and Learning includes cognitive, social and behavioral competenciesbehavioral competencies

Range of theoretical and methodological Range of theoretical and methodological approaches, including the use of approaches, including the use of ““big databig data””

Focus on learning at the frontiers of STEM Focus on learning at the frontiers of STEM disciplines and interdisciplinaritydisciplines and interdisciplinarity

Core Issues in STEM LearningCore Issues in STEM Learning

Range of formal and informal learning Range of formal and informal learning environments across the K-16 landscapeenvironments across the K-16 landscape

Develop an understanding of how characteristics Develop an understanding of how characteristics of learning environments interact with or support of learning environments interact with or support multiple aspects of STEM learningmultiple aspects of STEM learning

Special focus on emerging learning environments Special focus on emerging learning environments and evidence-based approaches to undergraduate and evidence-based approaches to undergraduate STEM instructionSTEM instruction

Multiple theoretical and methodological Multiple theoretical and methodological perspectivesperspectives

Core Issues in STEM Learning Core Issues in STEM Learning EnvironmentsEnvironments

Preparation of a diverse, globally-prepared and Preparation of a diverse, globally-prepared and highly-skilled STEM workforce - including highly-skilled STEM workforce - including teachersteachers

Focus on entire STEM workforce continuum, Focus on entire STEM workforce continuum, from Pre-K to postdoctoral training to careerfrom Pre-K to postdoctoral training to career

Emphasis on academic and non-academic Emphasis on academic and non-academic STEM career pathways and transitions across STEM career pathways and transitions across changing contexts and climateschanging contexts and climates

Connecting workplace expectations to design Connecting workplace expectations to design of interventionsof interventions

Core Issues in STEM Workforce Core Issues in STEM Workforce DevelopmentDevelopment

Practices that broaden participation, retention, and Practices that broaden participation, retention, and success of individuals underrepresented in STEM  success of individuals underrepresented in STEM  

Preparing students for successful transition to Preparing students for successful transition to further education or training, or the STEM workplace further education or training, or the STEM workplace

Study of accessibility and the impacts of technology Study of accessibility and the impacts of technology on diverse populationson diverse populations

Measures, processes and metrics to assess impacts Measures, processes and metrics to assess impacts and outcomes of broadening participation and and outcomes of broadening participation and institutional capacity building (e.g. on STEM institutional capacity building (e.g. on STEM innovation/productivity)innovation/productivity)

Core Issues in Broadening Participation Core Issues in Broadening Participation in STEMin STEM

NSF projects should be of the highest quality NSF projects should be of the highest quality and have the potential to advance, if not and have the potential to advance, if not transform, the frontiers of knowledgetransform, the frontiers of knowledge

NSF projects should contribute more broadly to NSF projects should contribute more broadly to achieving societal goalsachieving societal goals

Meaningful assessment and evaluation of NSF Meaningful assessment and evaluation of NSF funded projects should be based on appropriate funded projects should be based on appropriate metrics, keeping in mind the likely correlation metrics, keeping in mind the likely correlation between the effect of broader impacts and the between the effect of broader impacts and the resources provided to implement projectsresources provided to implement projects

NSF Merit Review PrinciplesNSF Merit Review Principles

NSF Review CriteriaNSF Review Criteria All proposals are evaluated against the All proposals are evaluated against the

two NSB-approved review criteriatwo NSB-approved review criteria: : Intellectual Merit:Intellectual Merit: criterion encompasses criterion encompasses

the potential to advance knowledgethe potential to advance knowledgeBroader ImpactsBroader Impacts: criterion encompasses : criterion encompasses

the potential to benefit society and the potential to benefit society and contribute to the achievement of specific, contribute to the achievement of specific, desired societal outcomesdesired societal outcomes

NSF Suggested Questions for Intellectual Merit NSF Suggested Questions for Intellectual Merit & Broader Impact& Broader Impact

1.1. What is the potential for the proposed activity to: What is the potential for the proposed activity to: Advance knowledge Advance knowledge and understanding within its own field or across and understanding within its own field or across

different fields (Intellectual Merit); and different fields (Intellectual Merit); and Benefit societyBenefit society or advance desired societal outcomes (Broader or advance desired societal outcomes (Broader

Impacts)?Impacts)?

2.2. To what extent do the proposed activities suggest and explore To what extent do the proposed activities suggest and explore creative, original, or creative, original, or potentially transformativepotentially transformative concepts? concepts?

3.3. Is the Is the planplan for carrying out the proposed activities well-reasoned, for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to incorporate a mechanism to assessassess success? success?

4.4. How How well qualified well qualified is the individual, team, or organization to is the individual, team, or organization to conduct the proposed activities?conduct the proposed activities?

5.5. Are there Are there adequate resourcesadequate resources available to the PI (either at the available to the PI (either at the home organization or through collaborations) to carry out the home organization or through collaborations) to carry out the proposed activities?proposed activities?

Additional Proposal ConsiderationsAdditional Proposal Considerations Project evaluationProject evaluation

Evaluation plan appropriate to project goals and Evaluation plan appropriate to project goals and descriptive of project assessment approachdescriptive of project assessment approach

Substantive external expert review mechanismSubstantive external expert review mechanism Be very clear that the evaluation and research plans are Be very clear that the evaluation and research plans are

distinct. If they are the same, present a strong justification. distinct. If they are the same, present a strong justification.

Project personnel and managementProject personnel and management Research and management roles of senior personnelResearch and management roles of senior personnel Roles and responsibilities of students and other traineesRoles and responsibilities of students and other trainees Mentoring plan for postdoctoral researchersMentoring plan for postdoctoral researchers

Dissemination strategyDissemination strategy

Additional Proposal ConsiderationsAdditional Proposal Considerations

Letters of agreement to participateLetters of agreement to participate All appropriate organizationsAll appropriate organizations Members of advisory committeesMembers of advisory committees

Research design and methodologyResearch design and methodology Appropriate and rigorous research methodsAppropriate and rigorous research methods Well-documented usable, and replicable models, Well-documented usable, and replicable models,

frameworks, data, literature, and measuresframeworks, data, literature, and measures Well-justified methods, consonant with theory, and Well-justified methods, consonant with theory, and

suited to the stated questions or hypothesessuited to the stated questions or hypotheses Justify measures and instruments that are needed to Justify measures and instruments that are needed to

establish evidence establish evidence

Common GuidelinesCommon Guidelinesfor Education Research and Developmentfor Education Research and Development

U.S. National Science Foundation and U.S. National Science Foundation and U.S. Department of EducationU.S. Department of Education

A cross-agency framework that describes:A cross-agency framework that describes:Broad types of research and development Broad types of research and development The expected The expected purposes, justifications, purposes, justifications, and and contributions contributions of various types of research to of various types of research to knowledge generation about interventions and knowledge generation about interventions and strategies for improving learningstrategies for improving learning

1616

What do we mean by What do we mean by ““Common Guidelines?Common Guidelines?””

The American education system needs stronger The American education system needs stronger evidence provided at a faster paceevidence provided at a faster pace

More constrained federal resources demand that More constrained federal resources demand that NSF and ED purposefully build on each other’s NSF and ED purposefully build on each other’s research and development portfoliosresearch and development portfolios

A cross-agency vocabulary and set of research A cross-agency vocabulary and set of research expectations is critical for effective expectations is critical for effective communicationcommunication

Why do we need “Common Guidelines?”Why do we need “Common Guidelines?”

1717

Guidelines can help stakeholders develop a Guidelines can help stakeholders develop a better understanding of what different stages better understanding of what different stages of education research should address and of education research should address and might be expected to producemight be expected to produce Helps stakeholders understand what to expect from Helps stakeholders understand what to expect from

different types of research findingsdifferent types of research findings Supports more informed decisions based on the level of Supports more informed decisions based on the level of

evidenceevidence Provides a shared sense of what is needed as stakeholders Provides a shared sense of what is needed as stakeholders

engage with research to improve educational practicesengage with research to improve educational practices

1818

Implications of the NSF/IES Common Implications of the NSF/IES Common GuidelinesGuidelines

1919

Is not strictly linear, but multi-directional. Is not strictly linear, but multi-directional. Investigations can Investigations can sometimes move directly from development of core knowledge to Scale-sometimes move directly from development of core knowledge to Scale-up. Investigations can sometimes move directly from development of core up. Investigations can sometimes move directly from development of core knowledge to Scale-up. knowledge to Scale-up.

Requires efforts of researchers and practitioners representing Requires efforts of researchers and practitioners representing a range of disciplines and methodological expertisea range of disciplines and methodological expertise

May require more studies for basic exploration and design May require more studies for basic exploration and design than for testing the effectiveness of a fully-developed than for testing the effectiveness of a fully-developed intervention or strategyintervention or strategy

Requires assessment of implementation—not just estimation Requires assessment of implementation—not just estimation of impactsof impacts

Includes attention to learning in multiple settings ( formal and Includes attention to learning in multiple settings ( formal and informal)informal)

Knowledge Development in EducationKnowledge Development in Education

2020

2121

CORE KNOWLEDGE

•Foundational Research•Exploratory/Early Stage Research

2222

Foundational Research Fundamental knowledge that may contribute to improved learning & other education outcomes

Studies of this type:◦Test, develop or refine theories of teaching or learning◦May develop innovations in methodologies and/or technologies that influence & inform research & development in different contexts

2323

Early-Stage or Exploratory Research

Examines relationships among important constructs in education and learningGoal is to establish logical connections that may form the basis for future interventions or strategies intended to improve education outcomesConnections are usually correlational rather than causal

2424

Design and Development Research

2525

Design and Development Research

Draws on existing theory & evidence to design and iteratively develop interventions or strategies

◦ Includes testing individual components to provide feedback in the development process

Could lead to additional work to better understand the foundational theory behind the results Could indicate that the intervention or strategy is sufficiently promising to warrant more advanced testing

2626

STUDIES OF IMPACT

•Efficacy Research•Effectiveness Research•Scale-Up Research

2727

Studies of Impact generate reliable estimates of the ability of a fully-developed intervention or strategy to achieve its intended outcomes

Efficacy Research tests/examines whether an intervention or strategy can improve outcomes under what are sometimes called “ideal” conditions

Effectiveness Research tests/estimates the impacts of an intervention or strategy when implemented under conditions of routine practice

Scale-Up Research examines/estimates the impacts of an intervention or strategy under conditions of routine practice and across a broad spectrum of populations and settings

Another Way of Thinking About Another Way of Thinking About Different Impact Studies (Dearing)Different Impact Studies (Dearing)

Internal validity research is about Internal validity research is about efficacyefficacyExternal validity research is about External validity research is about

effectivenesseffectivenessDissemination research is about Dissemination research is about

efficiencyefficiency Implementation research is about Implementation research is about

compatibilitycompatibility

Important Features of Each Type of ResearchImportant Features of Each Type of Research

2929

PurposeHow does this type of research contribute to the evidence base?

Justification

How should policy and practical significance be demonstrated?

What types of theoretical and/or empirical arguments should be made for conducting this study?

Important Features of Each Type of ResearchImportant Features of Each Type of Research

3030

Outcomes

Generally speaking, what types of outcomes (theory and empirical evidence) should the project produce?

What are the key features of a research design for this type of study?

3131

Comparison, in brief: JUSTIFICATIONComparison, in brief: JUSTIFICATION

A clear description of the practical education problem and a compelling case that the proposed research will inform the development, improvement, or evaluation of education programs, policies, or practices

A strong theoretical and empirical rationale for the project, ideally with citations to evidence

Exploratory/Early Stage

Research

3232

Comparison, in brief: JUSTIFICATIONComparison, in brief: JUSTIFICATION

A clear description of the practical problem and the initial concept for the planned investigation, including a well-explicated logic modelIn the logic model, identification of key components of the approach, a description of the relationships among components, and theoretical and/or empirical supportExplanation of how the approach is different from current practice and why it has the potential to improve learning

.

Design and Development

Research

3333

Comparison, in brief: JUSTIFICATIONComparison, in brief: JUSTIFICATION

Clear description of the intervention/ strategy and the practical problem it addresses; how intervention differs from others; and connection to learning

Empirical evidence of promise from a Design and Development pilot study, or support for each link in the logic model from Exploratory/Early Stage research, or evidence of wide use

Justification for examining impact under ideal circumstances, rather than under routine practice conditions

Efficacy Research

3434

Comparison, in brief: OUTCOMESComparison, in brief: OUTCOMES

Empirical evidence regarding associations between malleable factors and education or learning outcomes

A conceptual framework supporting a theoretical explanation for the malleable factors’ link with the education or learning outcomes

A determination, based on the empirical evidence and conceptual framework, of whether Design and Development research or an Efficacy study is warranted, or whether further Foundational or Exploratory/Early-Stage research is needed

Exploratory/Early Stage

3535

Comparison, in brief: OUTCOMESComparison, in brief: OUTCOMES

A fully-developed version of the intervention or strategy

A well-specified logic model

Descriptions of the major design iterations, resulting evidence, and adjustments to logic model

Measures and data demonstrating project’s implementation success

Pilot data on the intervention’s promise for generating the intended outcomes

.

Design and Development

Research

3636

Comparison, in brief: Comparison, in brief: OUTCOMESOUTCOMES

Detailed descriptions of the study goals, design and implementation, data collection and quality, and analysis and findings

Implementation documented in sufficient detail to judge applicability of the study findings; when possible, relate these factors descriptively to the impact findings

Discussion of the implications of the findings for the logic model and, where warranted, make suggestions for adjusting the logic model to reflect the study findings

Efficacy Research

Project NarrativeProject NarrativeProject Outcomes Project Outcomes

• • Detailed descriptions of the study goals, design and implementation, Detailed descriptions of the study goals, design and implementation, data collection and quality, and analysis and findings data collection and quality, and analysis and findings

Research Plan Research Plan Study design used to estimate causal impact of the intervention on Study design used to estimate causal impact of the intervention on

the outcomes of interest the outcomes of interest Key outcomes of interest for the impact study and the minimum Key outcomes of interest for the impact study and the minimum

impact that would have policy or practical relevance impact that would have policy or practical relevance Study setting(s) and target population(s) Study setting(s) and target population(s) The sample The sample The data collection plan The data collection plan The analysis and reporting plan The analysis and reporting plan

DeadlinesDeadlines

Please go to nsf.gov for deadlines. Please go to nsf.gov for deadlines.

??

3939

QuestionsQuestions

Thank You!!!Thank You!!!