PROJECT SUMMARY

Overview:The emergence of large, sensor-based datasets provides an opportunity to engage students in STEM andimprove quantitative reasoning through open-ended exploration and interpretation of real-world data.Project EDDIE (Environmental Data-Driven Inquiry and Exploration) is a collaboration among STEMdisciplinary and educational researchers. We aim to develop flexible classroom modules using large,publicly available, digital data for undergraduate students in biology, geology, and environmentalscience, as well as provide the associated professional development needed to ensure their effective use.Our previous NSF TUES pilot award allowed us to demonstrate that our modules can be highly effective(Carey and Gougis, 2017; Klug et al., 2017; O'Reilly et al., 2017; Soule et al., in press). Here, we proposeto expand our previous work to 1) develop and validate a much larger suite of curricular modules toimprove student quantitative reasoning, 2) develop a community of practice to engage faculty membersand foster a pedagogical orientation that favors open inquiry with large datasets and, 3) determine whatmechanisms contribute to shifts in an instructor's pedagogical orientation.

Intellectual Merit:EDDIE is designed with a community-based approach that will lead to educators using large authenticdatasets in their classrooms to improve student quantitative reasoning. We will use formative feedback toguide our cycle of innovation, as well as summative assessment of the project. Our theory of change forEDDIE is reflected in the creation of new knowledge in four key areas: 1) community needs forpedagogical tools to teach quantitative reasoning. 2) effectiveness of project-built modules and statisticalvignettes 3) understanding of which types of events and interventions result in a sustainable communityof practice and 4) how the use of EDDIE materials and participation in the development of pedagogicaltools will result in a change in instructors' pedagogical orientation. In addition to the 40 modules and 10statistical vignettes, we will create and validate a post-secondary version of an existing pedagogicalorientation instrument. An independent evaluator will measure and quantify the project-level successusing a multivariate approach to social network analysis and reach channels.

Broader Impacts:Our project is structured to provide community transformation using effective professional developmentpractices. EDDIE will reach over 1,000 instructors and over 13,500 undergraduate students from acrossthe country during the award period. Participants will be recruited and encouraged from 2-year and 4-yearinstitutions serving a broad range of student populations. During the two years of assessing student-learning gains, we will involve up to 4,300 students at three, diverse, institutions including University ofArizona, Illinois State University, and Queens College. Queens College students are 28% Hispanic; it isofficially designated as both an Asian American and Native American Pacific Islander-Serving Institution(AANAPISI) and a Hispanic Serving Institution (HSI). This diversity of institutions will ensure EDDIEmaterials and methods will be impactful across institution types. All project products, including bothcurricular and professional development materials will be freely available on the project website. EDDIEwill grow the STEM workforce by engaging declared and undeclared STEM majors in a variety ofcourses. By creating a community-sourced understanding of the barriers to teaching quantitativereasoning and skills, we will transform how instructors teach with data and create a pathway to STEMopportunities.

1

Collaborative Research: Environmental Data-Driven Inquiry and Exploration (EDDIE): Using large datasets to build quantitative reasoning BACKGROUND The wealth of large authentic datasets online provides an opportunity to engage students in scientific inquiry while simultaneously improving quantitative reasoning. Open-ended exploration through the analysis and interpretation of large datasets can have substantial benefits as students explore the stochastic nature of environmental and Earth systems (Brewer and Gross, 2003; Ellwein et al., 2014; Gougis et al., 2016). Manipulating messy, heterogeneous measurements motivates students to develop and rely on conceptual frameworks or mental models, and interpreting graphs means that students have to focus on pattern and process (Gould et al., 2014). Using authentic and publically accessible online datasets to address real-world questions reinforces the need and rationale for developing quantitative reasoning skills, leading to an increased appreciation for large complex datasets associated with basic environmental monitoring (Ellwein et al., 2014; O’Reilly et al., 2017). In addition, spatially resolved datasets allow students to find place-based data that are meaningful to them, and real-time data allow students to see immediate relevance. Many of these large datasets are based on high-frequency sensor systems that collect hundreds to millions of data points each day, and introducing these methods is a useful tool for engaging students in today’s age of information technology as well as encouraging them in modern scientific understandings.

Increasing availability of large online datasets provides unique opportunities to develop quantitative reasoning skills, particularly those associated with visualizing, analyzing, and interpreting quantitative data. Quantitative reasoning (QR) refers to the ability to interpret data and to reason with numbers in real-world situations (Steen, 2004), and working with large authentic datasets can provide the contextualization needed for meaningful student engagement. Graphing is a key element of QR and scientific literacy because data visualization is a critical step for initial exploration, for fostering scientific knowledge, and for effective communication of complex information (AAAS, 2011). Summarizing, condensing, and displaying quantitative data remains a persistent challenge in science (Glazer, 2011). Undergraduate students’ ability to comprehend and conceptualize data need improvement in areas that include variable identification and interpretation of variable relationships, utilization of the appropriate graph type, detection of data trends, and transformation of simple data into graphical form (Bowen et al., 1999; Picone et al., 2007; Bray Speth et al., 2010; Maltese et al., 2015). At the same time, working with large datasets requires that students build critical skills associated with data management and manipulation (Strasser and Hampton, 2012).

In Project EDDIE (Environmental Data-Driven Inquiry and Exploration), we propose to expand curricular material that improves quantitative reasoning and to develop a self-sustaining community of practice (sensu Lave and Wegner, 1991). We will focus on topics related to the environmental and Earth sciences, allowing the greatest potential for adoption across a wide range of STEM courses, from physical to life sciences. Our goals are to: 1. Develop a suite of flexible modular curricular materials, using large publicly available online

datasets, that contribute to improved student quantitative reasoning. 2. Develop a community of faculty members engaged with materials and professional

development designed to foster pedagogical orientation favoring open inquiry with large datasets.

2

3. Determine what mechanisms contribute to shifts in instructors’ pedagogical orientation towards inquiry-based teaching. Our prior work indicates that undergraduate students can effectively work with large

authentic datasets to address and explore both scientific and quantitative concepts (Carey et al., 2015; Bader et al., 2016). Our original collaboration developed 10 pilot modules using data from hydrology, seismology, biogeochemistry, climate, and limnology through NSF TUES 1245707 “The use of high-frequency data to engage students in quantitative reasoning and scientific discourse.” Students using our modules experienced measurable gains in both quantitative skills and concepts. There were significant self-reported gains in spreadsheet skill competence, with the greatest skill gains reported in students with the lowest initial scores (Klug et al., 2017). The use of the modules led to a greater appreciation for large datasets, computer software, tools, and technology, as well as a stronger understanding of the scientific concepts (Carey and Gougis, 2017; Klug et al., 2017; O’Reilly et al., 2017). Learning gains from the modules occurred even when students did not work through the modules as an in-class activity (Soule et al., in press), suggesting that the modules have strong potential to be used for online courses. The modules from our pilot project have been used by over 1,000 students and assessed across a broad range of courses in different STEM disciplines, from introductory to graduate students and across eight different institutions, including a small liberal arts college, mid-sized state universities, and large R1 institutions. Because the modules were developed by individual instructors and widely assessed (Carey and Gougis, 2017; Klug et al., 2017; O’Reilly et al., 2017; Soule et al., in press), we are confident that this process of using authentic large datasets to improve quantitative reasoning skills can be successfully expanded. PROGRAM COMPONENT 1: DEVELOPMENT AND ADOPTION OF MATERIALS, AND PROFESSIONAL DEVELOPMENT OF PARTICIPANTS The proposed Project EDDIE is designed with a suite of program elements that will lead to a well-established community of users poised for long-term growth and widespread adoption of effective curricular materials (Fig. 1). From ongoing research, we know that developing good materials and documenting their effectiveness is not sufficient to promote widespread adoption (Khatri et al., 2016). Many post-secondary instructors are aware of new teaching strategies and materials, but awareness of materials, even ones they might find helpful, has not led to their adoption (Henderson et al., 2010). Strategies for increasing adoption of teaching interventions include engaging the target audience in the design process, interactive events to encourage material adoption, and professional development support for adopters (Kahtri et al., 2016). Sharing models for curriculum design and program structure with the community will result in a more successful program (Manduca et al., 2010; McDaris et al., 2013). Thus, Project EDDIE begins with a community needs assessment before continuing on to the other program elements—we will

Fig 1: Project EDDIE activities and involvement of instructors and students during the four years of the project.

3

initially create an understanding of the barriers that the community of instructors face in teaching with large data sets and their interest in solutions. We will then focus on three additional program elements designed to mutually reinforce one another and expand the project’s reach and persistence: (1) production of freely available teaching materials and examples of their use in courses that will result in improved quantitative skills; (2) demonstration of second-generation use of modules by non-author adopters that will increase the number of faculty members using EDDIE materials, including examples from novel settings; and (3) a self-sustaining professional development program designed to (a) facilitate use of the materials by providing and gathering examples of use in diverse settings, (b) foster a community of faculty members interested in supporting each other to teach quantitative skills, and (c) encourage shifts in pedagogical orientation towards open-inquiry based practices. Understanding community needs – We propose to begin by engaging participants in building a shared vision for change (Wenger et al., 2011). To create this shared vision, we will first work to understand the perceived gap between instructor needs and the existing resources available to support them in teaching quantitative reasoning and managing large datasets. This gap analysis will be done through targeted interviews as part of our assessment plans. We will also hold a “barriers and solutions” face-to-face workshop on teaching with large datasets and host four webinars during Year 1. This assessment study will generate a community-sourced understanding of needs and barriers for teaching quantitative reasoning and teaching with large data sets and will build community identity for the next phase of Project EDDIE (Gehrke and Kezar, 2016).

Teaching materials: EDDIE Modules – As part of our NSF TUES award, we created a framework for EDDIE modules such that they are scalable across different skill levels, both within and across different types of institutions. The modules are based on a set of learning objectives, a plan for assessing student achievement, and a data set that provides an opportunity to explore a scientific concept or environmental problem. The common goal for all modules is to improve quantitative reasoning and skills associated with data manipulation and visualization (Carey et al., 2015; Bader et al., 2016). Each module has a flexible “A-B-C” structure that is based on the 5E Learning Cycle (Bybee et al., 2006; Carey et al., 2015). In an EDDIE module, Part A engages students in initial data exploration and skill development using simple analyses that bypass some of the technical challenges associated with the manipulation of data. Part B asks the students to explore and explain through more detailed analyses that require them to independently discuss and decide what analyses are appropriate for the data and explain the implications of results. In Part C, students expand by exploring questions that they have developed and choosing data from different sites. Students evaluate their learning through guided discussion. Each EDDIE module provides opportunities for students to practice sophisticated cognitive tasks, such as data visualization, evidence-based reasoning, and discussions of how spatial and temporal resolution affects ability to detect change.

Two main components are incorporated into every EDDIE module to build quantitative and analytical skills. First, the design of the module prompts students to generate and work with a visualization of the large dataset. Because the datasets are large, students learn to use shortcut codes to manipulate and select data, typically using spreadsheet software such as Excel, although modules can also be adapted to computing languages such as R. Second, open-ended questions are incorporated into each module, requiring the students to choose which data to work with,

4

either selecting a subset of data or choosing data from different locations. The goal of these sections is for students to grapple with the inherent issues of spatial and temporal variability within the dataset and to think carefully about what the data actually represents. For example, students make their own decisions about how to split up a temporal dataset to compare conditions “before” and “after” human activity when exploring the effects of urbanization on flooding or contrasting rates of temperature change. For another module, students determine how their results might change if they had a shorter time series. These questions force students to confront how their interpretations are influenced by data availability and variation. Assessment of initial EDDIE modules verified that students report improvement in their Excel skills and a greater appreciation for large datasets and how they can be used to advance scientific knowledge and monitor environmental problems (Carey and Gougis, 2017; Klug et al., 2017; O’Reilly et al., 2017; Soule et al., in press). Teaching materials: EDDIE Statistical Vignettes – Prior studies have shown that explicit instruction on underlying mathematical or statistical principles, in conjunction with practical applications, are among the most effective ways to build conceptual understanding of difficult topics (Gould et al., 2010). Given that a primary goal of EDDIE is to improve quantitative reasoning, instructors need to address student misconceptions of quantitative concepts. Variation, rate of change, uncertainty, and the exclusion of outliers are among the concepts that scientists confront in order to make sense of raw data. Examining variation in real data, along with structured discussion, has the potential to develop statistical conceptions about variation, randomness, and regression. Students who may initially understand variation to mean a superficial difference can come to understand and appreciate the need to capture and quantify variation in a dataset (Gougis et al., 2016). Exploring and identifying outliers can create opportunities to consider appropriate criteria for the exclusion of data (Carey et al., 2015) and can also create opportunities to build the statistical toolbox students need to correctly express

concepts such as significance and uncertainty (McCright, 2012).

To support specific learning of statistical concepts, we will develop a set of EDDIE Statistical Vignettes, each of which will focus on describing a different set of quantitative concepts. Cognitively, the vignettes will help students engage in elaboration: the process of clarifying the relationships between what the student already knows and what is being learned. This scaffolding forms a solid support upon which the rest of the lesson can be built (Glynn, 2008; Ancker and Begg, 2017). We explored these ideas in our TUES project, developing preliminary EDDIE Statistical Vignettes that were used to describe correlation coefficient, R2, standard deviation, probability, and the geometric distribution. Our results suggested that pairing EDDIE Statistical Vignettes with EDDIE modules can increase

Fig 2: An example of a cartoon in a Statistical Vignette. Art by M. Weirathmueller.

5

student learning gains (Soule et al., in press), in part by leveraging the cognitive power of illustrated analogies and metaphors (Glynn and Takahashi, 1998; Braasch and Goldman, 2010; Ancker and Begg, 2017).

The EDDIE Statistical Vignettes will provide a broad suite of materials for instructors to use as stand-alone teaching tools or in conjunction with EDDIE modules. Conceptually similar to Just-In-Time Math, EDDIE Statistical Vignettes consist of brief lectures and supporting materials on statistical topics to explicitly address quantitative concepts presented in our modules. Example topics would include linear regression, independent and dependent variables, correlation coefficient, and the difference between mean, median, and mode. The vignettes are modular and use engaging storylines with a diverse group of characters to step through the theoretical background of various statistical concepts. The storylines are decoupled from any specific module to allow the flexibility to use them as needed. EDDIE Statistical Vignettes will be developed in Years 1 and 2, primarily by PI Soule, with assistance from a master’s student and input from a statistician (S. Juliano), and will include illustrations to engage students (Fig. 2; M. Weirathmueller). EDDIE Module development workshops – We will refine the 10 existing modules from the TUES project, as well 3 currently being developed through funding to Dr. Cayelan Carey (NSF EF 1702506), and recruit new faculty to develop at least 30 additional modules through a structured design process that includes in-class piloting and revision and adherence to a design rubric. Modules will be developed for courses across environmental and Earth science disciplines to be sequenced or used as stand-alone material for in-class laboratory sessions, homework, or online. Modules will be designed so that they can interface with relevant statistical vignettes. Each module will focus on specific scientific concepts and address a set of quantitative reasoning or analytical skills, using high-frequency datasets that are publicly available online. Modules will generally be designed for use with Microsoft Excel as the data management and graphing software, with some modules also having an R version. Module materials will include a presentation, instructor handout, a student worksheet, and a dataset in case students cannot work online or if instructors wish to provide the data in a ready-to-use format. All modules will also be made available in a format appropriate for common course management systems so that they can be adapted for online use.

The new modules in this project will be developed in 3-day, face-to-face workshops in two cohorts starting in Years 2 and 3 with 20 participants in each workshop. The context of module development will be informed by the needs assessment. The workshops will be coordinated by the Science Education Resource Center (SERC) staff together with faculty members who already have expertise in developing and teaching EDDIE modules from their participation in the TUES project (PI’s O’Reilly, Soule, and Meixner). Workshop attendees will include faculty members from across the country that will be recruited through an open call targeting both 2-year and 4-year institutions and from the participant pool in the Year 1 needs assessment workshops. Module authors will be coached in developing materials that adhere to the EDDIE design, including evidence-based pedagogy, quantitative reasoning, use of high-frequency data, and the A-B-C structure. Peer reviews will be facilitated as part of the workshop. Each author will pilot the module in their own course during the following academic year and revise the module as needed. Final independent reviews of the field-tested and revised modules will be conducted by an assessment consultant prior to publication on the project website.

6

EDDIE Module second-generation adoption and broadening reach workshops – Excellent materials alone are not sufficient for the breadth of adoption that we seek. Lasting change requires demonstrated use by those not associated with materials development as well as the propagation of the materials, process, and pedagogy to facilitate additional use (Henderson et al., 2011). Although unstructured modifications to materials can reduce their efficacy (Henderson, 2007), we propose that adaptations are inevitable and that modifications made with the design principles in mind can retain the impact of the original materials. Thus, engaging non-author adopters in ways that allow us to receive feedback about how they use the modules is a key aspect of developing the Project EDDIE platform. This non-author step in the development process will help ensure that EDDIE modules have reach outside the project membership and will lead to a more robust resource that facilitates the scaling up of the project. Thus, our project will focus on several pathways to adoption of the modules by the broader community.

To develop specific adoption strategies, we will host virtual workshops to introduce new instructors to teaching with EDDIE modules. Participants will be recruited from 2-year and 4-year institutions and chosen by application, which will ensure participation from a range of institutional settings. These second-generation adopters will give feedback to the original module authors about their experience teaching the modules, providing an opportunity for community feedback into the cycle of innovation. Critically, they will write instructor stories about their experiences teaching quantitative reasoning in the classroom and provide any materials that they have adapted from the original modules. These stories, along with the adapted materials, will be published to the project website as examples of how the EDDIE approach and materials can be used in a diverse set of courses and institutions. Instructor stories and examples of adoption in different types of courses will become part of the materials used in project propagation in the final year of the program.

For broadening use of Project EDDIE materials, we will host open interactive workshops in the later part of the project. Virtual workshops to share and encourage use of EDDIE materials to will also create examples of EDDIE module use and impact in classrooms. PIs will also host face-to-face workshops at national conferences such as the American Geophysical Union (AGU), the Geological Society of America (GSA), the Association for the Sciences of Limnology and Oceanography (ASLO), the Ecological Society of America (ESA), and the Global Lake Ecological Observatory Network (GLEON) in Year 3 to increase adoption. Adopters will have opportunities to participate in project evaluation and research projects. All materials, development process, examples of use, instructor stories, and evidence of impact on faculty and student attitudes and learning generated through module adoption will be published to the project website and made publicly available through the SERC and professional society communities. Building an EDDIE community through professional development – Building a community of practice leads to individual faculty member growth through a shared commitment to learning (Lave and Wegner, 1991; Wegner et al., 2011). Project EDDIE is structured so as to generate such a community, beginning with the initial community needs assessment to create a shared vision for change. The sequence of activities includes an interactive, iterative development process in which authors are working together to build and test materials. This includes an immediate peer review, followed later by a testing stage involving a new set of practitioners who will bring their own solutions and ideas to the effort. Propagation of project outcomes will focus on not only the materials but also on advice and examples from successful members of the

7

community of practice. This model for community involvement will lead to the project results having ongoing impact past the life of the project (Kastens and Manduca, in press).

The professional development program starts with module authoring as professional development and makes use of an interactive strategy of sharing models for curriculum design and program structure with community support to more successfully facilitate participants changing their teaching practice (Manduca et al., 2010; McDaris et al., 2013; Manduca et al., 2017). In this model, all events are interactive with participants working together to learn how they might use the materials in their own courses with examples of teaching quantitative reasoning in a variety of classroom settings and institution types (Soule et al., in press). Additionally, a series of eight topical virtual workshops, supported by online materials, will encourage module adoption and adaptation. These events will draw from the experiences of the second-generation module adopters and will introduce their instruction strategies to new audiences. The virtual workshops will be widely advertised to faculty members across the EDDIE disciplines, including members of the broad SERC community and the appropriate professional societies (American Geophysical Union, Geological Society of America, Ecological Society of America, and the Association for the Sciences of Limnology and Oceanography). Virtual workshops will be interactive and provide direct opportunities for participants to apply the workshop materials to their own courses; our team has prior experience running highly regarded propagation workshops (e.g., McFadden et al., 2016). PROGRAM COMPONENT 2: ASSESSMENT, RESEARCH AND EVALUATION To determine the success of the project and also generate new knowledge on how to structure community and individual interventions to improve teaching and learning of quantitative reasoning, we have designed elements of internal and external evaluation (led by Iverson and Eriksson, respectively) as well as a research program (led by Gougis) that work together to build a picture of the project impact overall and address the project’s three main questions. Assessment: Understanding community needs – In support of the project creating value by building a shared vision (Wenger et al., 2011), Iverson and Eriksson will lead a qualitative study in Year 1 to investigate “How does the broader community characterize the needs and barriers to adopting curriculum that promotes inquiry with large data sets?” Prior to the needs assessment workshop, we will conduct an initial set of five interviews with leaders within the existing EDDIE community and beyond (e.g. including leaders from data-rich projects such as UNAVCO’s GEOdesy Tools for Societal Issues, IRIS’ Recent Earthquake Teachable Moments, and GeoPRISMS Margins data in the classroom). The initial set of interviews will serve to both test the protocol and to inform the design of the “barriers and solutions” workshop. Following the “barriers and solutions” workshop, Iverson and Eriksson will conduct an additional set of up to 10 interviews with a purposeful sample of faculty selected from participants from the workshop as well as additional key informants from UNAVCO, IRIS, GeoPRISM, and other data-rich curriculum groups. Data from the qualitative study will be analyzed along with products and artifacts from the workshop using a prioritization and causal analysis mapping approach (Altschuld and Kumar, 2009). The results will contribute to the body of knowledge about the needs, barriers, and solutions for faculty adoption of curriculum that promotes inquiry with large data sets. The findings will also ensure that the project aligns to the needs of the community it seeks to reach.

8

Assessment: Student learning gains – In this project we will assess both EDDIE Modules and EDDIE Statistical Vignettes because they can be used independently or in combination. Prior work indicated the effectiveness of EDDIE modules on a range of key areas associated with quantitative reasoning (Carey and Gougis, 2017; Klug et al., 2017; O’Reilly et al., 2017; Soule et al., in press). However, we will also have the opportunity to assess other factors, such as comparing the effectiveness of EDDIE Modules when used with and without Statistical Vignettes and how using multiple modules affects overall learning gains. We will conduct the assessment in several large general education classes with multiple lab sections at Illinois State University (ISU), CUNY-Queens, and the University of Arizona (Table 1). Project personnel are involved in teaching and/or coordinating laboratory activities for these courses.

Given the diverse nature of the courses and large number of students and laboratory sections, we will be able to design assessment approaches to explore a wide range of questions. Learning gains will be assessed using a set of validated multivariate instruments. Our team has experience using a range of assessment approaches (Carey and Gougis, 2017; Klug et al., 2017; O’Reilly et al., 2017; Soule et al., in press). Existing peer-reviewed instruments that would be appropriate for assessment of the QR goals of this project include Watson et al.’s (2003) Distributional Variation Scale, the SUSSI instrument that measures students’ understanding of the nature of science and scientific methods (Liang et al., 2006), and the Test of Scientific Literacy Skills (TOSLS) to measure students’ evaluation of scientific information and arguments (Gormally et al., 2012); we have used and found these instruments to be sensitive to assessing student learning in past work (e.g. Gougis et al., 2016; Soule et al., in press). Specific assessment design will be developed during Years 1 and 2, as the list of QR skills is refined in response to community vision, the specific Statistical Vignettes are developed, and module topics become known during the first development workshop. We will conduct an initial pre-treatment baseline assessment during Year 1 and then conduct assessments during Years 2 and 3. Research: How do we build a community and network of users? – To measure the strength of the network, Eriksson and Iverson will investigate the following question: “How do resources and activities supported by the project (website, virtual activities, etc.) contribute to the growth and reach of the community and to behaviors related to community engagement?” Two methods will be used to investigate this question. First, through survey data administered for all project activities, we will use social network analytic methods to measure growth of connections among participants, assess density of these connections, and understand the relationship between connections and collaborative behaviors among community participants (e.g., PIs, developers, second-generation users). In addition to contributing data to the study, the ongoing findings from the analysis will be used formatively to identify structural holes where we may want to engage

Institution Course # students/ semester

# lab sections

Illinois State University

BSC 101: Fundamental Concepts in Biology

1000 30

Illinois State University

GEO 102: Principles of Geology

700 20

CUNY-Queens

ENSCI 100: Introduction to Environmental Science (majors)

100 5

CUNY-Queens

ENSCI 99: Guide to Environmental Choices (non-majors)

200 10

University of Arizona

HWRS 170: Earth Our Watery Home

150 5

Table1: Description of the courses to be used in assessing student learning gains for modules and statistical vignettes.

9

members using different strategies or to identify key netweavers to capitalize upon their connections within the community (Daly, 2010; Kadushin, 2012). Second, metrics will be collected on reach channels (professional development activities, integrated web portal, e-mail lists, and professional association activities) used by the projects to investigate how specific program elements are leading to engagement of people in the community of practice (Teigland and Wasko, 2004). These metrics include participant demographics, participant listserv engagement, and weblog metrics such as page views, frequency of use, and returning users. We will triangulate the network analysis with reach metrics to document the growth and spread of the community across institutions and sub-disciplines. This study will contribute to the broader scholarly knowledge about the relationship between professional development and reach channels in supporting the growth and engagement of a faculty community of practice. Research: What is the influence on pedagogical orientation? – Very little research has been conducted on post-secondary science educators’ pedagogical orientation and what experiences beget shifts in orientation toward inquiry. Many post-secondary educators have recognized that students are not empty vessels ready to be filled with knowledge transmitted from teacher to learner but rather that students actively construct and continuously restructure perceptions of their world through a back-and-forth dialogue between instructor and student (Cross, 1991; von Glasersfeld, 1995). We can use the construct of pedagogical orientation (Cobern et al., 2014) to capture an instructor’s epistemic approach to teaching science. Instructors with a direct orientation explain scientific concepts and principles, whereas instructors with an inquiry orientation engage students in exploration of scientific concepts and principles so that they may construct, at least partially, the scientific concepts and principles themselves (Cobern et al., 2014). Guided-inquiry orientation allows students to actively explore natural phenomena, concepts, and principles prior to explanation while the instructor guides exploration toward desired scientific content. An open-inquiry orientation allows students to actively explore natural phenomena, concepts, and principles of their choosing while the instructor facilitates sound scientific practice but does not necessarily direct their exploration toward specific content.

Given that EDDIE modules are designed around guided and open-ended inquiry, we will explore whether participation in professional development workshops shifts instructors’ general approach to teaching science toward a model which integrates active learning techniques and inquiry-based instruction. EDDIE modules, if taught with fidelity, facilitate guided-inquiry instruction early in the modules (Parts A and B) and open-inquiry instruction later in the modules (Parts C and more). Thus, it is plausible that a direct-oriented instructor who participates in an EDDIE workshop, adopts an EDDIE module(s), and faithfully implements it in their classroom may discern the feasibility of inquiry-based instruction and experience a shift in pedagogical orientation. We saw this happen anecdotally to some extent in our TUES project. In turn, this shift may influence the instructor’s teaching more broadly. In this project, we will improve our understanding of post-secondary educators’ pedagogical orientation, thus informing professional development for higher education faculty as well as graduate student training that prepares the future professorate. We will address two research questions: (1) When a post-secondary science instructor’s pre-EDDIE orientation is direct, what resources and experiences facilitate a shift from direct to inquiry orientations? and (2) When post-secondary science instructors teach an EDDIE module(s) that facilitates more inquiry-oriented instruction than what they would typically implement, to what extent do they revise non-EDDIE lessons and activities to be more inquiry oriented?

10

We will address these questions by modifying and validating an existing instrument and applying it to EDDIE workshop participants. First, Gougis and a postdoctoral researcher will adapt the existing POSTT for K–12 (Cobern et al., 2014) to create the Post-Secondary Pedagogical Orientations Science Teaching Test (PS-POSTT) to identify a science faculty member’s pedagogical orientation. Initial content validity will be established by conducting 30 faculty interviews, and for validation we will recruit at least 1,000 science faculty through professional societies such as AGU, GSA, ESA, and ASLO (respective memberships of 62,000, 27,000, 9,000, and 4,300). Before attending workshops, EDDIE participants will complete the PS-POSTT and be interviewed about current teaching practices, beliefs, and methods. These data will be used to create short (<15 minutes) explanatory videos of learning theory concepts, active teaching methods, etc., for participants to watch in preparation the workshops. After they have implemented an EDDIE module, Gougis will interview participants to qualitatively explore teaching practices, beliefs, and methods used both within the EDDIE module and in other teaching. Participants will also take the PS-POSTT at the end of the academic year in which module implementation occurred as well as again a year later to gauge long-lasting impact of EDDIE participation. Repeated-measures analysis of variance will be used to compare changes in science pedagogical orientation and qualitative analysis of interview data will allow for identification of specific experiences and resources that enabled a shift in science faculty’s pedagogical orientation toward inquiry teaching.

Evaluation: Overall project – Evaluation will be both formative and summative, using mixed methodology with a combination of surveys, focus groups, and interviews. All project data collected for research and evaluation studies described in this proposal will inform multiple uses. The external evaluation, led by Eriksson, will focus on overall project monitoring and will use a Causal Link Monitoring model (Britt et al, 2017) in which the assumptions and processes that support an initial logic model (Table 2) are examined to inform the leadership team to modify activities and products throughout the project. This is consistent with finding “critical dependencies...and insert(ing) evaluative probes” through a project (Kastens and Manduca, 2017). The overarching project evaluation centers on the question: “To what extent do the various program components lead to achieving project goals?” Initial evaluation will address the extent to which project leadership establishes a robust project plan with clear roles and responsibilities, a plan to collaborate in planning and implementation of activities, and alignment to research-grounded components (Henderson et al., 2011). External evaluation will mine data from various program components to measure medium-term effects to examine program execution and progress (e.g., professional development, EDDIE Statistical Vignettes). The internal evaluation will be led by Iverson and will center on evaluation questions related to professional development. Long-term indicators will look at the relationship of the various program components to achieve the desired outcomes as well as capture unintended or emergent project outcomes. Evaluation: Professional development – The evaluation of the professional development will seek to answer the evaluation question “To what extent do professional development (PD) strategies promote the adoption of curricular materials that support open inquiry with large datasets?” The evaluation will investigate alignment of PD programmatic activities to the goals of the project and to the needs and barriers identified in Year 1. Then, the evaluation will examine the ways that participants attribute short-term and medium-term effects to the PD

11

experience. Short-term methods will include analysis of how program and PD artifacts (activity reviews, essays, virtual discussions) align to goals and needs. In addition, road check surveys and end of workshop surveys will be used to collect participant immediate perceptions of the PD experience. Medium-term effects will be examined through post-workshop and post-field testing reflection surveys and the analysis of weblogs, forums usage, material downloads, and materials reviews. Finally, the PD evaluation will triangulate self-report effects with data related to the reach of the community gathered through the social network analysis methods to articulate a complete picture of the impact of the program on the different types of participants (materials developers, second-generation testers, and extended community members). Table 2. Logic model for Project EDDIE illustrating products and outcomes to improve quantitative reasoning (QR) in undergraduate education.

Inputs Activities Products Program Outcomes Existing Modules from NSF TUES

Framework rubric developed Workshop for developing new modules

Revise and adapt by users

30 new and 10 edited peer-reviewed modules

Sustained Professional Development

Community of Practice in teaching QR

Material Design Rubric

Authors test and provide teaching tips

Examples of Use and Instructor notes from 2nd-Gen users

Website with published modules and supporting materials

Robust resource library for instructors of QR

1000 instructors and 13,500 students reached

SERC, PIs, Faculty expertise

Workshops to facilitate community vision and barriers/needs assessment

Study of broader community needs

Module pedagogy workshops Widespread use of materials

Lessons learned from NSF TUES

Develop Statistical Vignettes Tested tool to measure specific quantitative reasoning skills

Flexible tools for teaching QR

Understanding of student QR

Extended audience has tools for teaching and measuring QR.

POSTT Instrument

Update and validate a version for post-secondary education

PS-POSTT for higher education

Understanding of instructor development

Faculty researcher and postdoc

Research on pedagogical orientation

PS-POSTT for Higher Education

Understanding of module and PD impact on teaching practice

Evaluators’ skills and experience

Evaluation: Prof. Development Project Monitoring

Understanding of community network, mechanisms for strength and growth,

Description of project strengths/ways of working

Mechanisms to strengthen community of practice

Future directions based on program outcomes

COMMUNITY SUPPORT Due to the success and widespread use our previous modules from the TUES proposal, this new Project EDDIE proposal and the community of practice it will establish has the momentum of contributions from scientists of many disciplines. We have the support of our institutions and department chairs to assess innovative teaching approaches for quantitative reasoning in our courses. William Wilcock (Univ. of Washington) will be developing modules focused on Geophysics and Geodesy. Director of Education and Outreach at IRIS, is working to ensure data availability and supporting module development. Glenn Kroeger (Trinity Univ.) is developing

12

the software package Seismic Canvas to support relevant module development. Deborah Kelley, the Director of the Ocean Observatories Initiation regional Cabled Array (Univ. of Washington) will have a presence at our workshops and contribute to the development of modules focused on digital data collected through their seafloor network. The Global Lake Ecological Observatory Network (GLEON) will build on the success of the existing lake suite of modules with have been used internationally and assist with recruitment and host workshops. William Perry (ISU), who has taught several undergraduate short courses in how to use R and coordinates and leads a Learn-to-Use-R working group at GLEON, has agreed to provide assistance as needed with conversion of modules from Excel-based into R-based. Steven Juliano, who is the co-developer of ISU’s MS sequence in biomathematics, run jointly by Mathematics and Biological Sciences, and who teaches introductory and advanced statistics courses for graduate students in Biological Sciences, has agreed to provide a check of the statistical vignettes. Finally, Diana Dalbotten, Director of the Geoscience Alliance, will act as a liaison to recruit faculty from tribal colleges faculty to participate in workshops and webinars. PROJECT STRUCTURE AND MANAGEMENT The management structure will assign clear leadership to personnel for specific segments of the program, with centralization of key support services (Table 3). Support for the leadership team, website, and professional development will be centralized at the Science Education Resource Center (SERC) at Carleton College. Administrative and financial support services will be centralized at the Center for Mathematics, Science and Technology (CeMaST) at Illinois State University (ISU). Each of these centers has extensive experience in these types of projects. For example, SERC coordinated the $10M NSF STEP Center InTeGrate (activities of 10 PIs, team-based authoring of 33, 2-week curricular modules, more than 65 workshops and webinars, 16 institution-based program-scale interventions, and development of the community and website). CeMaST coordinates a series of programmatic activities associated with K–12 and higher education curricula and training, including ISU’s NSF Noyce Scholarship Program and community building, and has successfully run 22 multi-year NSF and NIH-funded curriculum and professional development projects over the past 25 years. Our project structure retains a central communication structure, core administrative facilities, strong project management, and a unified public face. This structure then allows faculty from across the country to be engaged in leadership of program components, facilitating the development of a community network.

The project will be co-directed by O’Reilly at ISU and Orr of SERC at Carleton College. The Leadership Team will include PIs Meixner (University of Arizona) and Soule (Queens College), who each have experience developing and teaching modules that use large datasets, as well as Iverson at SERC. An annual team meeting and monthly video conferences will maintain communication, address management issues, and coordinate planning. We will use a private workspace on the SERC website to organize reporting, share information and documents, track activities, and manage shared responsibilities. The Leadership Team will meet annually to review work and evaluation results from the prior year and plan activities for the coming year (Table 4), with feedback from Gougis and Hunter, as well as Eriksson, the external project evaluator. PIs will each will manage a team of module authors as part of the module development process, mentoring them through development, piloting, revision, and publishing. Project policies will be developed in the first year, to ensure common understanding of processes, goals, and transparency across all activities.

13

Table 3. Project EDDIE key personnel and responsibilities

Underpinning and extending this work is the community website and virtual platform, which

will capitalize on the tools and expertise of SERC (see Facilities Statement). The development teams, second-generation adopters, and professional development activities will be supported by workspaces on the project website. SERC has done pioneering, award winning work on the interactions between workshops and websites to support communities and to foster change in STEM Education (Fox et al., 2005; Manduca et al., 2006; Manduca and Johnson, 2008; Manduca et al., 2010; Ledley et al., 2013; Manduca et al., 2017). SERC hosts one of the largest on-line collections of resources for undergraduate educators, which was developed primarily using the processes of workshops, revision, and adoption described in this proposal. The work of SERC’s On the Cutting Edge project in the geosciences epitomizes the success of this approach, having simultaneously built a geoscience education community that has sustained a workshop leadership model of programming and attendance for over 12 years (Tewksbury et al. 2013) and created an online resource used by over 40% of geoscience faculty with more than 5,000 pages of content.

Table 4: Timeline of activities associated with Project EDDIE Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Community needs assessment Barriers and Solutions workshop Virtual webinars on teaching with large datasets Interviews Material development, adoption, and module use

Catherine O’Reilly, Illinois State University (ISU)

Leadership Team. Overall project management and coordination. Lead development of module structure rubric and revision of existing modules. Assist with coordination of workshops and student learning gains. Assist with coordination of module development and implementation for environmental and Earth sciences.

Dax Soule, CUNY-Queens College Leadership Team. Coordinate module development and implementation for environment and Earth sciences. Assist with coordination of workshops. Lead assessment of student learning gains. Supervise masters student. Develop and assess statistical vignettes.

Thomas Meixner, Univ. of Arizona Leadership Team. Coordinate module development and implementation for hydrology. Assist with coordination of workshops and student learning gains.

Rebekka Gougis, ISU Develop instrument to assess pedagogical orientation of faculty. Supervise postdoctoral researcher. Lead research on pedagogical orientation. Assist with assessment of student learning gains.

Brittany Ciancarelli, Univ. of Arizona Assist with assessment of student learning gains. William Cobern, Western Michigan University

Contribute to developing instrument to assess pedagogical orientation of faculty with Gougis and postdoctoral researcher.

Cailin Huyck Orr, SERC Carleton College

Leadership Team. Overall management of professional development program, workshop planning, and web publishing.

Ellen Iverson, SERC Carleton College

Leadership Team. Internal evaluator. Lead community needs assessment, research on social network analyses, attitudinal change, relationship to engagement, and community building. Assist with assessment of student learning gains.

Sean Fox, SERC Carleton College Develop website infrastructure in support of development, publishing and assessment.

William Hunter, CeMaST, ISU Fiscal oversight. Coordinate administrative support. Provide module review. Susan Eriksson, Eriksson Assoc. External program evaluator. Michelle Weirathmueller Artwork for modules and statistical vignettes.

14

Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Module design rubric and desired specific QR skills list developed Existing modules edited to match rubric Statistical Vignettes developed based on QR skills list Module Development: Workshop, implementation, reviews and revision. Module Adoption: Workshop, implementation, adoption stories Module Use : Open webinars and conference workshops Workshops and materials published on website Assessment and Research Activities Student learning gains: EDDIE Modules and Statistical Vignettes Pedagogical orientation: Assessment tool developed Pedagogical orientation: Data collection and analyses Building a community: Social network analysis and reach External program evaluation Management Team meetings PI and assessment team outcomes summit Data analyses and peer-reviewed publication

BROADER IMPACTS Project EDDIE is designed to maximize the direct impacts on faculty and students during module development and to create mechanisms that allow these curricular materials to propagate far beyond the active life of the program. Exponential growth is a demonstrable characteristic of faculty development when a community of practice is built that takes a systems approach with reinforcing feedback loops (Kastens and Manduca, in press), such as in our program structure. Instructor essays will provide example solutions to common barriers and a list of “pro tips” that will facilitate adoption in diverse settings. Project EDDIE will update the POSTT assessment for use in higher education. Project EDDIE will result in an improved understanding of the level of engagement needed to change faculty practice and the influence of professional development on the pedagogical orientation of faculty members.

Project EDDIE will reach 1,350 faculty members over the first 4 years of the program. Each of these faculty member interactions will provide opportunities for participants to learn how to teach with large data sets, provide information and evidence for why teaching with data is useful, and identify ideas about overcoming the barriers to using this approach. The design and development phase will engage 390 faculty members to generate a community-sourced understanding of needs and barriers for teaching quantitative reasoning and teaching with large data sets. The second-generation adopters will engage 60 faculty members from 2- and 4-year institutions across the United States in the development of 30 modules and ~10 statistical vignettes. Professional development activities associated with the national professional society meetings will engage 120 additional faculty members in face-to-face meetings to facilitate the use and adaption of the modules. The planned virtual workshops will engage 720 additional faculty members in a series of eight materials-adoption virtual workshops. This large engagement of faculty will effect change through the shared process of developing and evaluating new teaching resources and instructional strategies. By creating a community-sourced understanding of the barriers to teaching quantitative reasoning, we will transform how instructors teach with data.

15

Project EDDIE will directly involve a wide range of undergraduate students, impacting at least 13,500 students. If each module is used twice, once by the author and once by a second- generation adopter, with 25 students, this will engage ~2,000 undergraduate scholars. For the module adoption workshops held at national conferences, if 90% of these attendees use a module with 25 students, this will engage an additional 2,700 students. For the webinars, even if only 25% of these participants try an EDDIE module, we would reach an additional 4,500 students. During 2 years of assessing student learning gains, we would involve up to 4,300 students at the three lead institutions. Enrollment at these institutions encompasses underrepresented student populations, with 9.4% Hispanic and 8% African American students at ISU, 25% Hispanic and 4% African American students at the University of Arizona, and 28% Hispanic and 9% African American students at Queens College (QC). QC is officially designated as both an Asian American and Native American Pacific Islander-Serving Institution (AANAPISI) and a Hispanic Serving Institution (HSI). In addition to helping students understand that data exploration is part of the scientific method, visualizing and interpreting large datasets could make students better able to produce and evaluate data presented in public formats. PRIOR NSF SUPPORT: Over the past 5 years, our PIs have received a total of 14 NSF awards. Many of these grants have involved undergraduate and pre-service teacher education in STEM (curricular materials, tools, networks, scholarships, and training), as well as supporting instrumentation and basic research. These projects have engaged hundreds of teachers and thousands of students. Below, we describe the awards most closely related to this proposal. TUES-1245707: The use of high frequency data to engage students in quantitative reasoning and scientific discourse. $217,328. 7/2013-6/2017. PIs: O’Reilly, Gougis. IM: We held 3 workshops in which 8 faculty developed 10 modules that use large or high-frequency datasets for courses in biology, ecology, geology, hydrology and environmental science. These modules were assessed at 8 different institutions and are available online at SERC. We published 7 manuscripts and did 14 conference presentations including a special session. BI: We included 3 graduate students, and some modules were adapted for use with middle school students. DUE-1540591: ISU's Noyce Scholarships for STEM Teachers of Under-Represented Groups. $1,444,790. 2/2016-1/2021. PIs: Gougis, Hunter. IM: We expanded ISU’s successful model of STEM teacher training into a high-needs suburban setting, also exploring several lines of research on teacher development. Thus far, this project has yielded 4 presentations at national conferences. BI: 40 scholarships to new STEM teachers-of-color who will seek employment in Valley View School District (Chicago-area) or other high-need schools, thus ensuring that high quality STEM is taught by a teacher who belongs to an underrepresented group. NSF EAR-1331408: Transformative behavior of energy, water and carbon in the critical zone II. $4.9M. 10/13-9/18. Co-PI: Meixner. IM: The Catalina-Jemez Critical Zone Observatory is developing quantitative links between water, land, and biogeochemistry, including using in situ sensor datasets, with 8 papers so far. BI: Understanding controls on water and carbon will inform decisions made by land and resource managers. DBI-1730273: RCN-UBE: Course-based Undergraduate Research Network 2. $499,925. 9/2017-8/2020. Co-PI: Orr. IM: This new project will build on the Course-based Undergraduate Research Experiences (CURE) Network (CUREnet) to support national transformation of undergraduate biology laboratory instruction to instruction that engages students in research. BI: CUREnet2 will focus on building capacity for CUREs at Historically Black Colleges and Universities (HBCUs) and other under-resourced colleges and universities.

1

References * indicates references associated with the prior NSF TUES award AAAS, 2011. Vision and Change in Undergraduate Biology Education: A Call to Action.

AAAS Washington, DC. Altschuld, J.W., and D.D. Kumar, 2009. Needs assessment: An Overview. SAGE Publications,

Thousand Oaks, CA. Ancker, J.S. and M.D. Begg, 2017. Using visual analogies to teach introductory statistical

concepts. Numeracy 10:2:7 DOI:10.5038/1936-4660.10.2.7. *Bader, N.E., D. Soule, D. Castendyk, T. Meixner, C.M. O’Reilly, and R.D. Gougis, 2016.

Students, meet data: Using publicly available, high-frequency sensor data in the classroom. EOS. DOI:10.1029/2016EO047175.

Bowen, G.M, W.M. Roth, and M.K. McGinn,1999. Interpretations of graphs by university biology students and practicing scientists: Toward a social practice view of scientific representation practices. Journal of Research in Science Teaching 36 (9): 1020-1043.

Braasch, J.L.G., and S.R. Goldman. 2010. The role of prior knowledge in learning from analogies in science texts. Discourse Processes 47(6): 447-470. DOI 10.1080/01638530903420960.

Bray Speth, E., J.L. Momsen, G.A. Moyerbrailean, D. Ebert-May, T.M. Long, S. Wysession et al., 2010. 1, 2, 3, 4: infusing quantitative literacy into introductory biology. CBE-Life Sciences Education 9 (3): 323-332.

Brewer, C.A., and L.J. Gross, 2003. Training ecologists to think with uncertainty. Ecology 84(6): 1412:1414. DOI: 10.1890/0012-9658(2003)084[1412:TETTWU]2.0.CO;2.

Britt, H., R. Hummelbrunner, and J.Greene, 2017. Causal Link Monitoring. Retrieved from: http://www.betterevaluation.org/resources/overview/Causal_Link_Monitoring.

Bybee, R. W., J.A.Taylor, A. Gardner, P. Van Scotter, J.C. Power, A. Westbrook, and N. Landes, 2006. BSCS 5E instructional model: Origins and effectiveness. A report prepared for the Science Education, National Institute of Health. BSCS.

*Carey, C.C., and R.D. Gougis, 2017. Simulation modeling of lakes in undergraduate and graduate classrooms increases comprehension of climate change concepts and interest in computational tools. Journal of Science Education and Technology. DOI:10.1007/s10956-016-9644-2.

*Carey, C.C., R. Gougis, J. L. Klug, C.M. O’Reilly, and D.C. Richardson, 2015. A model for using environmental data-driven inquiry and exploration to teach limnology to undergraduates. Limnology and Oceanography Bulletin, 24:32-35. DOI:10.1002/lob.10020.

Cobern, W.W., D. Schuster, B. Adams, B. A. Skjold, E. Z. Muğaloğlu, A. Bentz, and K. Sparks, 2014. Pedagogy of science teaching tests: Formative assessments of science teaching orientations. International Journal of Science Education, 36(13): 2265-2288.

Cross, K.P., 1991. College Teaching: What do we know about it. Innovative Higher Education. 16: 7-25. DOI:10.1007/BF00911555.

Daly, A. J., 2010. Mapping the terrain, Social network theory and educational change. Cambridge, MA: Harvard Education press.

Ellwein, A.L, L.M. Hartley, S. Donovan, and I. Billick, 2014. Using rich context and data exploration to improve engagement with climate data: Bringing a field station into the college classroom. Journal of Geoscience Education 62:578-586. DOI: 10.5408/13-034.

2

Fox, S., C.A. Manduca, and E. Iverson, 2005. Building educational portals atop digital libraries. D-Lib, 11(1).

Gehrke, S., and A. Kezar 2016. The roles of STEM faculty communities of practice in institutional and departmental reform in higher education. American Education Research Journal 54: 803-833. DOI: 10.3102/0002831217706736.

Glazer, N., 2011. Challenges with graph interpretation: a review of the literature. Studies in Science Education 47: 183-210. DOI: 10.1080/03057267.2011.605307.

Glynn, S.M., and T. Takahashi, 1998. Learning from analogy-enhanced science text. Journal of Research in Science Teaching 35: 1129-1149. DOI: 10.1002/(SICI)1098-2736(199812)35:10<1129::AID-TEA5>3.0.CO;2-2.

Glynn, S.M., 2008. Making science concepts meaningful to students: Teaching with analogies. In S. Mikelskis-Seifert, U. Ringelband, and M. Brückmann (Eds.), Four decades of research in science education: From curriculum development to quality improvement. Waxmann. Munster.

Gormally, C. P. Brickman, and M. Lutz, 2012. Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates’ evaluation of scientific information and arguments. CBE Life Science Education 11: 364-377. DOI 10.1187/cbe.12-03-0026.

*Gougis, R.D., J. F. Stomberg, A. O’Hare, N.E. Bader, T. Meixner, C.M. O’Reilly, and C.C. Carey, 2016. If random is to have no pattern, how can we predict variation in a random sample?: Post-secondary science students’ concepts of randomness and variation. International Journal of Mathematics and Science Education. DOI10.1007/s10763-016-9737-7.

Gould, R., 2010. Statistics and the Modern Student. International Statistical Review. 78: 297-315. DOI: 10.1111/j.1751-5823.2010.00117.x.

Gould, R., S. Sunbury, and M. Dussault, 2014. In praise of messy data. Science Teacher 81: 31–36.

Henderson, C., A. Beach, and N. Finkelstein, 2011. Facilitating Change in Undergraduate STEM Instructional Practices: An Analytic Review of the Literature. Journal of Research in Science Teaching. 48(8): 952–984.

Henderson, C., N. Finkelstein and A. Beach, 2010. Beyond dissemination in college science teaching: An introduction to four core change strategies. Journal of College Science Teaching. 39: 18-25.

Henderson, C. and M.H. Dancy, 2007. Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics. Physical Review Physics Education Research. 3: 020102.

Kadushin, C., 2012. Understanding Social Networks: Theories Concepts and Findings, Oxford University Press, 252 pp. ISBN: 9780195379464.

Kastens, K. and C. A. Manduca, 2017, Using Systems Thinking in the Design, Implementation, and Evaluation of Complex Educational Innovations, With Examples From the InTeGrate Project. Journal of Geoscience Education. 65(3): 219-230.

Kastens, K. A., and C.A. Manduca, in press. Leveraging the power of community of practice to improve teaching and learning about the earth. Change: The Magazine of Higher Learning.

Khatri, R., C. Henderson, R. Cole, J. Froyd, D. Friedrichsen, and C. Stanford, 2016. Designing for sustained adoption: A model of developing educational innovations for successful propagation. Physical Review Physics Education Research 12 (010112).

3

*Klug, J.L., C.C. Carey, D.C. Richardson, and R.D. Gougis, 2017. Integrating high-frequency and long-term data analyses into undergraduate ecology classes improves quantitative literacy. Ecosphere 8(3): e01733. DOI: 10.1002/ecs2.1733.

Lave, J., and E. Wenger, 1991. Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press.

Ledley, T.S., A.U. Gold, F. Niepold, M. Grogan, K.B. Kirk, S.B. Sullivan, S. Lynds, M. McCaffrey, C.A. Manduca, and S.P. Fox, 2013. The Climate Literacy and Energy Awareness Network (CLEAN): Leveraging reviewed educational resources and a diverse community to achieve climate literacy goals. Geological Society of America Abstracts with Programs. 45(7): 501.

Liang, L.L., S. Chen, X. Chen, N. K. Osman, A. Adams, M. Maklin and J. Ebenezer, 2006. Student Understanding of Science and Scientific Inquiry (SUSSI): revision and further validation of an assessment instrument. Annual Conference of the National Association for Research in Science Teaching. San Francisco, CA.

Maltese, A.V., J.A. Danish, R.M. Bouldin, J.A. Harsh and B. Bryan, 2015. What are students doing during lectures? Evidence from new technologies to capture student activity. International Journal of Research and Method in Education 39: 208-226 DOI:10.1080/1743727X.2015.1041492.

Manduca, C. A., E.R. Iverson, M. Luxenberg, R.H. Macdonald, D. A. McConnell, D. W. Mogk, and B.J. Tewksbury, 2017. Improving undergraduate STEM education: The efficacy of discipline-based professional development. Science Advances, 3(2).

Manduca, C., and J. Johnston, 2008. Engaging faculty in discussion of the affective domain: A Practical Strategy, The National Teaching & Learning Forum. 17(3).

Manduca, C.A., H. MacDonald, D. Mogk, and B. Tewksbury, 2006. On the cutting edge: Evolving themes, enduring impact: Fourth-year report of outcomes based on interviews and workshop evaluations. (Acrobat (PDF) 335kB Apr25 06). Northfield, MN: Science Education Resource Center.

Manduca, C. A., D. J. Mogk, B. Tewksbury, R. H. Macdonald, S. P. Fox, E. R. Iverson, K. Kirk, J. McDaris, C. Ormand, and M. Bruckner, 2010. SPORE: Science Prize for Online Resources in Education: On the Cutting Edge: Teaching Help For Geoscience Faculty: Science.327(5969): 1095-1096.

McCright, A.M., 2012. Enhancing Students’ Scientific and Quantitative Literacies through a Sociological Inquiry-Based Learning Project on Climate Change. Journal of the Scholarship of Teaching & Learning 12(4): 86-102.

McFadden, R.R., A.C. Newman, and C.A. Manduca, 2016. Broadening Adoption of Teaching About the Earth in a Societal Context with the InTeGrate Webinar Series. Geological Society of American annual meeting. Denver, USA.

McDaris, J., T. Watson Nelson, A. Egger, C. Manduca, and Q. Williams, 2013. Showcasing successful strategies for supporting minority students in the geosciences. Geological Society of America Abstracts with Programs. 45(7): 379.

*O’Reilly, C.M., R. Darner Gougis, J.L. Klug, C.C. Carey, D.C. Richardson, N.E. Bader, D. Soule, D. Castendyk, T. Meixner, J.F. Stomberg, K.C. Weathers, and W. Hunter, 2017. Using large datasets for open-ended inquiry in undergraduate classrooms. Bioscience. 67(12): 1052-1061.

Picone C, J. Rhode, L. Hyatt, T. Parshall, 2007 Assessing gains in undergraduate students’ abilities to analyze graphical data. Teaching Issues and Experiments in Ecology. Vol. 5

4

*Soule, D, R. Gougis, C.M. O’Reilly, N.E. Bader, T. Meixner, C.A. Gibson, and R.E. McDuff. In press. EDDIE modules are effective learning tools for developing quantitative literacy and seismological understanding. Journal of Geoscience Education.

Steen, L.A., 2004. Everything I needed to know about averages I learned in college. Peer Review 6(4): 4-8.

Strasser, C.A. and S.E. Hampton, 2012. The fractured lab notebook: undergraduates and ecological data management training in the United States. Ecosphere 3: 1-18. DOI: 10.1890/ES12-00139.1.

Teigland, R., and M. M.Wasko, 2004. Extending richness with rich: Participation and knowledge exchange in electronic networks of practice. In P. M. Hildreth & C. Kimble (Eds.), Knowledge networks: Innovation through communities of practice (pp. 230-242). Hershey, PA: Idea Group Publishing.

Tewksbury, B., C.A. Manduca, D.W. Mogk, and R.H. Macdonald, 2013. Geoscience Education for the Anthropocene, The Impact of the Geological Sciences on Society: Geological Society of America Special Paper. 501: 189-201.

von Glasersfeld, E. 1995. Radical Constructivism: A way of knowing and learning. Studies in mathematical education series:6. ERIC ED381352. ISBN-0-7507-0387-3

Wenger, E., B. Trayner, and M. de Laat, 2011. Promoting and assessing value creation in communities and networks: a conceptual framework. Rapport 18, Ruud de Moor Centrum, Open University of the Netherlands. 56 pp.