Developing and implementing materials to teach recurrence intervals and related statistical concepts

Travis Sossaman, Fred Boehm, and Bill Bland

Abstract

We developed and implemented a learning module that incorporates statistical reasoning and methods in the context of longitudinal precipitation measurements in distinct geographic regions. Our instructional materials encouraged students to extrapolate the concept of “recurrence interval” and related notions to other scientific contexts. We designed questions to measure student comprehension before and after the in-­class activity. We analyzed student responses to gain insights into the extent to which we achieved the learning objectives. Our results suggest that students learned statistical language associated with recurrence intervals and learned the usefulness of data visualizations. In future work, we will refine this unit and integrate statistical reasoning skills into other course components. Introduction

Statistical thinking, in which students learn from data in the presence of uncertainty, plays a critical role in scientific research and the scientific method. Despite this, many college courses in science, technology, engineering, and mathematics (STEM) disciplines de-­emphasize statistical methods and statistical reasoning. While most fields certainly attempt to convey the importance of statistics to their students, these attempts often manifest as a single course, or module of a course, within the field’s undergraduate curriculum. Isolating statistics from a scientific discipline’s core concepts may discourage students from recognizing the importance of statistical methods and reasoning in their respective disciplines. To encourage undergraduate students to develop statistical reasoning skills, we developed instructional materials for a module in an introductory science class. The module is the first step in promoting a pervasive inclusion of statistical principles throughout the course. Project Context and Scope

The target audience for the project were primarily undergraduate students enrolled in Soil Science/AOS 132 -­ Earth’s Water: Natural Science and Human Use in the Spring 2014 term. William Bland, a soil science professor, served as lead instructor for the course. The current semester was comprised of fifty-­six students, mostly non-­science majors, of whom approximately 40% were Freshman. The course met Tuesdays and Thursdays for 75 minutes and was structured as a “flipped” classroom that made use of the WisCEL lab located in Wendt Library. The structure of the course throughout the semester consisted of pre-­recorded lectures that included a pre-­class quiz to ensure students came prepared for the activity planned during class time. The activities generally involved small group (2-­4 people) case studies.

The instructional material developed in this project was produced throughout the semester;; therefore, the class time available for its implementation was inherently limited. The intended scope of the project was necessary to define early on before much of the design

process could begin. In order to allow sufficient time to produce the instructional material and strike an appropriate balance between quality and quantity of content, the project would focus on a single 75-­minute lecture with pre-­recorded video lectures as necessary, so as to follow the currently established format of the course. Learning Challenges and Goals

In designing our instructional materials, we used information from previous offerings of the course. Specifically, we used evidence that past students in Soil Science 132 tended to possess weak quantitative reasoning skills when developing our learning materials. The goal of our material design was to impart stronger quantitative and statistical reasoning skills through our learning material, while retaining a content focus germane to the Soil Sciences. Project Design and Implementation

The project was developed throughout the semester using the “backward design” method described by Wiggins and McTighe [1] and was created with a learner-­centered focus following the methods described by Huba and Freed [2]. The procedure was as follows: 1. Establish and define the conceptual goals for the material being developed. 2. Define the specific learning objectives that promote the understanding of the designated

conceptual goal(s). 3. Describe the assessment tools that would measure student learning gains for each learning

objective. 4. Determine and develop the learning activity or activities that provided an effective learning

opportunity for students to engage with the material, accomplish the established learning objectives, and implement the assessment metric that catalogs both student performance and the efficacy of the instructional material developed.

During the design process, the procedure above was utilized iteratively, where repeated

applications worked to refine each aspect of the design process. For instance, a suggested activity or assessment technique would bring to light a new potential learning objective. If it was deemed a good addition, the assessment and activity would be refined to incorporate the newly added objective. We performed several iterations before we finalized the working project outline (Figure 1).

Figure 1. The final project outline used to direct the instructional material developed in the current

project.

While several conceptual goals were initially proposed, the scope of the project (one 75

minute class period) required us to focus on the one that we felt was most important, this being that students learned the language of climatological events within a statistical context. From this goal, our learning objectives were to enable students to learn the basic statistical concepts used in climatology (rank, percentile rank, exceedance, and the recurrence interval) and to make proficient use of the graphs typically used in the field (time series, empirical density, and cumulative distribution function plots).

Our approach for assessment was based upon a pre-­ and post-­activity evaluation comparison to determine student learning gains, as well as a post-­activity survey to gauge student opinion of the activity and obtain useful feedback for future improvement of the module. Both evaluations were conducted using the UW Moodle course management software, which allowed tracking of individual student performance.

Our learning activities included three pre-­class lecture videos that: 1) introduced climatology and the methods by which rainfall is recorded, 2) reviewed several simple statistical concepts needed for the planned activity (mean, median, and standard deviation), and 3) introduced the specific statistical concepts and graphical data representations used in climatology, including rank, percentile rank, the recurrence interval, exceedance probabilities, and the use of time series, empirical density, and cumilative distribution function plots. Each video lecture was developed using PowerPoint slides with the relevant material and an accompanying script used in the discussion of each slide. The PowerPoint and voice-­overs were recorded using the “Screen Capture” Add-­In from Camtasia Studio. The pre-­class evaluation was administered after students viewed these introductory lecture videos.

The final learning activity was administered in-­class and was developed as a mock scenario/design problem that utilized the statistical and climatological concepts outlined above. The premise for the activity had students acting as planner-­engineers for water sustainability

solutions in urban environments, where the three cities of interest (Austin, Madison, and Miami) wanted to divert some amount of recycled waste-­water for maintenance of recreational areas such as parks and golf courses. The students were tasked with determining the number of recreational areas each city could support, based on the available recycled waste-­water and June-­August precipitation data for each location spanning from 1895 to 2013. The pre-­class quiz, in-­class activity with supplementary material, and post-­class exam questions related to the project are provided in Appendices A, B, and C, respectively. We include the initially proposed post-­activity questions in appendix D. The pre-­lecture videos provided to students are available at: http://pages.stat.wisc.edu/~boehm/courses/spring2014/stat692-­IMD/project/project.html. Assessment of Project Efficacy

Our method for assessing the efficacy of the instructional material developed for this project consisted of two evaluative facets: a paired summative/formative evaluation of student performance on pre-­ and post-­activity quizzes and a formative student opinion survey. Student Performance Data

Each question in the pre-­activity quiz (see Appendix A) was based upon one or more learning objectives given in Figure 1. The quiz specifically focused on rank, percentile rank, the recurrence interval, and the exceedance probability. The quiz itself lacked any use of graphs, since this was the focus of the in-­class activity. The post-­activity exam questions (see Appendix C) include the use of percentile rank, the recurrence interval, and exceedance probability with a strong emphasis on the use of graphs, and included a question on the definition of climate normals.

Figure 2. The overall scores of student participating in the pre-­activity quiz (n=51, left) and the

post-­activity exam (n=54, right).

Figure 2 displays the overall performance data for all students participating in the

pre-­activity quiz (n=51, left) and post-­activity exam (n=54, right). Inspection of the two graphs suggests a reduction in total scores below 40% of the total possible points (2 and 2.8,

respectively), with an associated increase in scores greater than 60%. Indeed, the overall average of scores increased from 65.4% to 70.3%. While the observed improvement in overall scores is encouraging, we must consider the nature of the questions in each evaluation before a definitive statement can be made about the efficacy of the developed material based on the data provided in Figure 2.

Of note, Question #4 of the post-­activity exam is identical to Question #4 of the pre-­activity quiz, but contains an additional false multiple choice answer). The results from the pre-­activity quiz indicated that the students had a good general grasp of the concepts explored in the the evaluation. On further inspection of the scores for individual problems, Questions 1, 2, 3, and 5 were answered correctly by at least 58% of students (Q #3) and reached a maximum of 90% (Q #5). By contrast, Question #4 was correctly answered by only 33% of students, indicating that relative relations between recurrence intervals was a challenge for the students before the in-­class activity. The reuse of Question #4 on the post-­activity exam is valuable in direct comparison of learning gains.

Question #5 of the post-­activity exam was an open-­answer question meant to gauge student understanding of the value and utilization of cumulative distribution function plots in the field of climatology. Of all questions in the post-­activity exam, Question #5 required the highest level of thought, utilizing aspects from the Knowledge, Comprehension, and Application classifications of Bloom’s Taxonomy. Student performance on this question would provide the most applicable feedback as to their level of understanding of the material presented in the current project. The comparative results for Question #4 and performance results for Question #5 are given in Figure 3.

Figure 3. The comparison of pre-­ and post-­activity Question #4 (left) containing raw scores

(blue bars) and relative student gains (red bars) for the specific question (Q4 Gain) and across

the two evaluations as a whole (Net Gain). Also includes the results for the open-­answer

application question of the post-­activity exam (right).

From Figure 3 (left), the percent of correct answers to Question #4 increased from 33%

in the post-­activity exam to 59% in the pre-­activity quiz. Of the students participating in both the pre-­activity quiz and post-­activity exam (n=49), 44.9% of them saw an improvement in their score (Q4 Gain in Figure 3). The result may suggest the students’ understanding of relative relationships between recurrence intervals improved during the in-­class activity. Likewise,

among these same students, 61.2% achieved a better normalized score across all questions in the post-­activity exam than the pre-­activity quiz (Neg Gain in Figure 3), which again indicates that student understanding of the intended learning goals improved with the in-­class activity.

While the results obtained in the pre/post comparison are encouraging, there are several factors that must be taken into consideration before attributing these improvements to genuine learning gains. For instance, the pre-­activity quiz was electronically graded and included feedback for the students. In the case of multiple choice questions such as Question #4, this includes the correct answer;; which could mean some fraction of students demonstrating improvement simply remembered the answer from the pre-­activity quiz when taking the post-­activity exam. Likewise, the emphasis on the use of graphs in the post-­activity exam may have tapped into the students’ pre-­existing knowledge more readily than a similarly phrased question that was formatted as calculation. With no pre-­activity baseline question on the students’ skills in the use of graphs, attribution of the nearly 5% increase in overall scores, and 61% net gain in normalized scores to definitive learning gains is not directly possible.

The results from the open-­answer Question #5 in Figure 3 (right) represent the greatest indication of the project’s efficacy based on student performance. Nearly 80% of students achieved a score of 1.5 (50%) or more on the question, with over 20% achieving perfect scores. The question contains three parts, each increasing in difficulty and extending to higher order classifications of Bloom’s Taxonomy. The nature of the question is directly related to the in-­class activity, and represents a very difficult question without sufficient pre-­existing knowledge of the topic and/or without having participated in the pre-­activity quiz or in-­class activity. Indeed, those students who took the post-­activity exam without taking the pre-­activity quiz scored no better than 1.5 (50%) for Question #5, and 4.5 (64.3%) across all associated questions. Neither maximal score by these students exceeded the average score for Question #5 (1.9) or the post-­activity exam (4.9). The above observations are the greatest indication that the improvements we see in student performance can be attributed to genuine gains in student learning. Student Opinion Survey

The post activity survey administered as part of the in-­class activity worksheet (see Appendix B) was used as a method of qualitative formative assessment of the project. The first two questions were used to gauge student interest and opinion of the project as a whole, asking them to identify aspects they liked and disliked. The vast majority of responses indicated that the use of the graphs and the relative comparisons between the three regions was the most enjoyable aspect of the activity. Many comments also included an appreciation for the the premise of the case-­study, and its ties to “the real world”. Sample comments included:

“We liked comparing the different graphs to show similar information. It was very

beneficial to learn a lot about the different graphs and how they function and relate to one

another.“

“We like how the activity is structured that we can consider three specific cities and using

the specific context to familiarize us with the climatology terms and calculations.”

Student responses for what they disliked about the activity correlated directly with their responses for what confused them the most with the activity, and was generally the topic of the suggestions they provided for the activity’s future improvement. The vast majority of students disliked Question 3(c), the math intensive design question included in the activity. All students cited it as the primary source of their confusion with the activity. Student suggestions for future improvement of the activity primarily focused on the need for improved framing of the design problem and greater guidance in it setup and solution. Many indicated more background information on the topic and/or practice questions dealing with the topic before attending class would have been useful. One example comment included the following:

b) What part did you like the least?

“We feel like we needed more understanding of the statistics that we need to do the

calculations. Question 3 is phrased a little confusing and we were not sure how to

approach on this question.”

c) Was any part of this exercise particularly confusing?

“Yes, what question 3 was asking was a little confusing.”

d) What suggestions do you have to improve the activity and/or make it more clear?

“It would be helpful if we can be given some example problems and clearer instructions

on what the questions are asking for. Also, it feels as though we are expected to know a

lot of background information about how to do statistics and about some of these

processes that we have never learned about. Some of these activities seem too

advanced and too reliant on previous knowledge that students may or may not have.”

The results of the student survey were very effective in gauging student interest and issues with the instructional material developed in the current project. It is clear that the design question must be significantly revised to truly be of benefit to the students. Based on students’ responses, the confusion incurred from insufficient framing and background on the problem may have disrupted their ability to grasp the key statistical and climatological concepts intended to be learned through the activity. While the question was designed in part to push the limits of their understanding and encourage high-­order cognition of the importance of statistics in the climatology, it proved to be too involved for the limited background information they were provided. In order to make the current design question viable for achieving the learning gains it was intended for, a greater amount of time would need to be spent on the topic to familiarize the students with the field, provide the necessary background knowledge, framework, and example problems necessary for their success and understanding of the problem. Conclusions

The results from student performances and opinions were not sufficient to make a detailed, definitive statement on the efficacy of the instructional material developed for the current project;; however, they do suggest that students benefitted from the material and achieved many of the learning goals intended for the project. To this effect, our conceptual goal that students would learn the language to express the statistical chance of various climatological effect was a

success. The post-­evaluation results indicated that most students understood the process of and value in using graphical representations of climatological data. Comparative results indicated that learning gains in our target learning objectives were observable, even though uncontrolled variables prevented a significant quantitative pre/post comparison analysis. Finally, the student opinion surveys provided a highly valuable formative assessment for continued refinement of the instructional material. The project provided a good foundation of material that, like all good instructional material, may be iteratively improved to afford greater student understanding and learning gains each semester. References 1. G. Wiggins, J. McTighe, Understanding by Design, Prentice Hall, Columbus, OH, (1998) p. 7-­19. 2. M. Huba, J. Freed, Learner-­Centered Assessment on College Campuses, Allyn and Bacon, Needham Heights, MA, (2000) p. 91-­101.

APPENDICES: Appendix A: Pre-­activity Moodle Quiz Appendix B: In-­class worksheet with figures Appendix C: Post-­activity midterm exam questions (as they appeared on the midterm exam that followed the in-­class activity by one week) Appendix D: Initially proposed post-­activity assessment questions

Appendix A: Pre-­Activity Moodle Quiz Question 1 Imagine that we selected 20 students from the class (our sample) and lined them up ordered by height. How many would be in the tallest 20-­th percentile? Question 2 Think about your bank balance the day before payday (doesn’t matter if weekly or monthly). Looking over records from 50 paydays, you see that the day before payday your balance was less than $100 on 6 occasions. What is the probability of having a less-­than-­$100 balance immediately before a payday? Question 3 Think about your bank balance the day before payday (let's assume you are paid weekly). Looking over records from 50 paydays, you see that the day before payday your balance was less than $100 on 6 occasions. What is the recurrence interval (number and units) of a less-­than-­$100 balance immediately before a payday? Question 4 Think about your bank balance the day before payday (doesn’t matter if weekly or monthly). Looking over records from 50 paydays, you see that the day before payday your balance was less than $100 on 6 occasions (ranged from $99 to $5 (!)). What can you say about the recurrence interval (RI) of a less-­than-­$50 end-­of-­pay-­period balance compared to a less-­than-­$100 RI? Select one: a. The $100-­balance RI is the larger value. b. They are the same. c. The $100-­balance RI is the smaller value. d. The $100 RI is twice the $50 RI. Question 5 Imagine that we selected 20 students from the class (our sample) and lined them up ordered by height. How would you estimate the median? Select one: a. Average the highest and lowest values. b. Order by both shortest to tallest and tallest to shortest and find the average. c. Select the value with the highest rank. d. Find the mean of the middle two values.

Appendix B: In-­Class Worksheet and Survey

Earth’s Water: Natural Science and Human Use Soil Sci/AOS 132 -­‐ Spring 2014

Precipitation: The Statistical Perspective

Precipitation is the most apparent of the various components of Earth’s water cycle. We understand it in several complementary ways: its magnitude and contribution to the water cycle, the physics and chemistry that make it happen, and through statistical analysis of simple observations. This last way of knowing precipitation is vital to designing and managing the many human endeavors that depend on precipitation, such as growing food, minimizing flood damage, and having enough water to drink. Here we will touch on an example of statistical engagement with precipitation data. You are a planner-­‐engineer with a firm that is working hard toward greater water sustainability in urban and suburban areas. The particular focus of your firm is that the discharge from wastewater treatment plants (treated water returned to rivers or lakes) should be used locally for landscape irrigation (this is done to some degree in the arid West). This can both reduce the amount of water extracted from groundwater and surface waters for irrigation, and put the wastewater discharge to valuable use. The challenge is to match the supply of treated wastewater with the needs of nearby recreational grasslands, such as golf courses and parks. Some care in matching the supply to the demand is necessary so that you do not come up short in dry years (lots of parched grass) and do not have excess water to dispose of in rainy years. Your design parameters: Volume of wastewater: Assume that the wastewater utility wants to divert from its usual discharge about 1 mgd (“million gallons per day”-­‐-­‐ call it 4000 m3/day). This would be for a season, 90 days x 4000 m3/day = 360,000 m3. Evapotranspiration loss from park and golf course areas: 450 mm (~18”) during June-­‐July-­‐August period (assume that this figure applies to all locations in this exercise). A typical 18-­‐hole golf course covers 75 acres (303,514 m2). As you work on your design you should be thinking about trying to irrigate one or several (say 2 or 3) courses. How many you could irrigate with the diverted water will depend on precipitation. Precipitation: We have supplied years of precipitation data for three locations in the US: Madison, WI, Austin, TX, and Miami, FL. Three presentations of the data are available: timeseries, density plot, and cumulative probability.

Exercises: The first step in a data-­‐oriented problem is to spend time convincing yourself that the available information makes sense. Linger over the data. 1) Timeseries: do a “reality check” on it:

a) What are the approximate values of the maximum, minimum, and mean (average) for each location? Do they make sense to you given your knowledge of where the cities are located?

b) Do you note any periods of above-­‐ and below-­‐average rain in each series? c) Do you see trends in the mean over time in any of the three? d) For each location, how many years does there seem to be enough rainfall to meet evapotranspiration requirement of a golf course (and all courses in the area)? What is the recurrence interval of sufficient rainfall for each location? (Note: If an area has no instance of rainfall that fully supports the needs of a golf course, simply state “Not applicable”.)

2) Density plots

a) Match the density plots to the correct timeseries. b) Study the symmetry of the distribution of the observed rainfall totals. Are any of the three “skewed,” that is not balanced like a “bell curve” is? What does this mean for the precipitation climate of the place?

3) Cumulative probability plots

a) Again, match the cumulative probability plots to the correct timeseries. b) From these plots, determine the 10, 20, median, 80, 90-­‐th percentile values of summer rainfall for each location.

c) Using the median rainfall for each location, estimate the number of golf courses that could be fully irrigated using half of the amount of available wastewater if necessary. There are two possible outcomes, so follow the directions below: i. If wastewater is needed, round your answer to the nearest integer irrigated course. Next, for each applicable location, calculate the minimum amount of rainfall necessary for these courses to be fully irrigated using all of the available wastewater reserve. What is the probability of having a year where the supply by rainfall and wastewater fails to meet the demand by evapotranspiration? Do you think our procedure strikes a sufficient balance of wastewater reclamation while avoiding over-­‐taxation in abnormally dry years?

ii. If no wastewater is necessary, estimate the probability of needing at least some of the wastewater in a given year using the CDF plot (even if it’s only a few gallons per course). Compare this answer to the probability you can calculate using the recurrence interval (for the appropriate location) in Problem 1(d). Are they the same? Why or why not? Finally, calculate the minimum number of courses the region can support, assuming full use of the wastewater reserve.

4) Please give us your thoughts on this activity. (Note: Your comments here have no impact on the grade you receive for Problems 1-­‐3. This question is designed to improve this exercise for future classes, so any thoughts you may have on improving it are extremely valuable to us.)

a) What part of this activity did you like the most? b) What part did you like the least? c) Was any part of this exercise particularly confusing? d) What suggestions do you have to improve the activity and/or make it more clear?

Appendix C: Post-­Activity Midterm Exam Questions Question 1 (as Question 9) Using the graph that follows, estimate what are the chances of having a total monthly precipitation of less than 100 mm at this location in the US. Select one: a. 75% b. 5% c. 100% d. 50% e. 0.05% Question 2 (as Question 10) Climatologists study the statistical representation of the weather, and they define ‘climate normals' as… Select one: a. the probability of a precipitation event to occur or a temperature value to be reached. b. the 30-­year averages of temperature or precipitation for an area. c. the value of annual precipitation or median temperature that is twice the previous year

average. d. the events that occur once in every hundred years (1%) in a particular area. e. the climatic events that have a recurrence interval of 30 years. Question 3 (as Question 21) Which graph type could you most readily determine the 90-­th % value of a dataset? Select one: a. scatter b. histogram c. cumulative probability d. density e. timeseries Question 4 (as Question 26) Think about your bank balance the day before payday (doesn’t matter if weekly or monthly). Looking over records from 50 paydays, you see that the day before payday your balance was less than $100 on 6 occasions (ranged from $99 to $5 (!)). What can you say about the recurrence interval (RI) of a less-­than-­$50 end-­of-­pay-­period balance compared to a less-­than-­$100 RI?

Select one: a. You can't tell anything about their relationship without calculating them. b. The $100-­balance RI is the smaller value. c. They are the same. d. The $100 RI is twice the $50 RI. e. The $100-­balance RI is the larger value. Question 5 (as Question 33) The graph below was derived from summer (June-­July-­August) rainfall totals at a location. It is one form of data analysis useful in climatology. Describe:

1. how it is created, 2. two examples of the sort of statements that you can make based on it, and 3. a possible application of an analysis like this for a water management challenge (that is,

how might a statement like you wrote for #2 be useful?).

Appendix D: Preliminary questions for post-­activity assessment Suppose that the Wisconsin River depth at Sauk City has been recorded on April 1st of each year for the last 30 years. 1. Which graph could you most readily determine the 90% percentile value of a dataset?

a) timeseries b) density c) cumulative probability d) none -­ calculations are required

2. Using the appropriate graph, you find that 90% of the recordings are below 2m. What is the probability of having a river depth of greater than 2m on this date in a single year? What is the recurrence interval for a river depth (at Sauk City on April 1) of greater than 2m? How is the recurrence interval for river depth greater than 2m related to the exceedance probability for river depth greater than 2m?

a) RI = 1/P b) RI = (1-­1/P) c) RI = P d) There is no relation between the RI and the exceedance probability.

3. Do any of the graphical depictions (timeseries, density plot, cumulative density plot) allow you to immediately determine the mean? (Select all that apply.)

a) Timeseries plot b) Density plot c) Cumulative density function plot d) None, the mean must be calculated

How about the median? (Select all that apply.)

a) Timeseries plot b) Density plot c) Cumulative density function plot d) None, the mean must be calculated

4. What is meant by climate normals?

a) 30-­year averages of temperature and precipitation. b) What scientists believe the weather should be at a location. c) Years when the weather equals the climate d) The climate that best aligns with that of near-­by observation sites. e) A running average of weather at a place.