Are crib sheets sheets worthwhile? · Literature on crib sheets In general, crib sheets are seen as...

Are crib sheets sheets worthwhile?An exploratory data analysis

Ant Edwards & Birgit Loch

[email protected] of Mathematics

Swinburne University of Technology, Australia

07/07/2015

Commonwealth of AustraliaCopyright Act 1968Notice for paragraph 135ZXA (a) of the Copyright Act 1968WarningThis material has been reproduced and communicated to you by or on behalf of Swinburne University ofTechnology under Part VB of the Copyright Act 1968 (the Act). The material in this communication maybe subject to copyright under the Act. Any further reproduction or communication of this material by youmay be the subject of copyright protection under the Act.Do not remove this notice.

Table of contents

1 IntroductionAbout Swinburne and crib sheetsLiteratureThe courseMethods

2 Project stage 1 - Exploratory analysisOpen coding of crib sheetsOverall trendsStructure and layoutSTEM content specific elements

3 Summary

Ant Edwards & Birgit Loch Are crib sheets sheets worthwhile? 07/07/2015 1 / 20

Introduction About Swinburne and crib sheets

Are crib sheets sheets worthwhile?

Introduction Literature

Literature on crib sheets

In general, crib sheets are seen as a useful object for students

Affective hypothesisCrib sheets make students less stressful in exams [and so do better](Butler & Crouch, 2011; Erbe, 2007; Janick, 1990; Weimer, 1989)

Encoding hypothesisBy creating a crib sheet, students engage with material [and so do better](Dorsel & Cundiff, 1979; Lawin, Gorman, and Larwin, 2013)

External storage (retrieval) hypothesisStudents have access to information during an exam [and so do better](Dorsel & Cundiff, 1979; Gharib, Phillips, and Noelle, 2012)

However. . .

Students can see mathematics to be primarily based on surface procedures(Crawford, Gordon, Nicholas, & Prosser, 1994; Frank, 1988; Garofalo,1989)

Cribsheets may encourageI a reliance on procedural surface-level understanding of topics

[Psychology UG] (Dickson & Bauer, 2008)I a replication of what the lecturer thinks is important [Chemistry UG]

(Visco, Swaminathan, Zagumny & Anthony, 2007)I students to ‘fill up’ sheets completely (ibid.)I dependence on crib sheets [Psychology UG] (Funk & Dickson, 2011)I a “search” mentality when stuck on a problem [Psychology UG]

(Burns, 2014)

However. . .

Students can see mathematics to be primarily based on surface procedures(Crawford, Gordon, Nicholas, & Prosser, 1994; Frank, 1988; Garofalo,1989)

Cribsheets may encourageI a reliance on procedural surface-level understanding of topics

[Psychology UG] (Dickson & Bauer, 2008)I a replication of what the lecturer thinks is important [Chemistry UG]

(Visco, Swaminathan, Zagumny & Anthony, 2007)I students to ‘fill up’ sheets completely (ibid.)I dependence on crib sheets [Psychology UG] (Funk & Dickson, 2011)I a “search” mentality when stuck on a problem [Psychology UG]

(Burns, 2014)

de Raadt (2012): Computer science students

Undergraduate students on a programming course:81% of students in the study self-selected using a sheet.

I Did students who created and used a crib sheet get higher grades?

I How about when compared with their earlier attainment?I What was present on crib sheets? How did this relate to grades?

Content and structureTheme Category of description Effect

Layout-basedDensityOrganizationOrdering matches course

Content-based

Examples [of code]Abstract representations [diagrams]

Sample Answers

Duplication

I Did students who created and used a crib sheet get higher grades?

YesDifference in means 8.6 marks out of 50(t-test was significant at the 5% level.)

I How about when compared with their earlier attainment?I What was present on crib sheets? How did this relate to grades?

Content-based

Sample Answers

Duplication

I Did students who created and used a crib sheet get higher grades?I How about when compared with their earlier attainment?

I What was present on crib sheets? How did this relate to grades?

Content-based

Sample Answers

Duplication

I Did students who created and used a crib sheet get higher grades?I How about when compared with their earlier attainment?

Yes.Improvement in [expected] performance was 5.3% for crib sheet users.No methodology or test of significance stated.

I What was present on crib sheets? How did this relate to grades?

Content-based

Sample Answers

Duplication

I Did students who created and used a crib sheet get higher grades?I How about when compared with their earlier attainment?I What was present on crib sheets? How did this relate to grades?

Content-based

Sample Answers

Duplication

Content-based

Sample Answers

Duplication

Layout-basedDensityOrganizationOrdering matches course +13%

Content-based

Examples [of code]Abstract representations [diagrams] +21%Sample Answers -30%Duplication

Introduction The course

MTH00004 Foundation Mathematics

First year mathematics unit for Science students who, in general, had nottaken intermediate mathematics units at school.

First “calculus” course, based on Foundation Mathematics textbook.

Assessment structureI 2 class tests, each worth 14% of the final markI 12 online assessments, one per week, collectively worth 17%I final examination (3hr, 23 questions), worth 55%

Crib sheets were allow in class tests and the final examination, onlineassignments were unmoderated.

107 students, 98 took the final examination, of which 85 used a crib sheet.

Tony Croft

Introduction Methods

Methods

Two stages of analysis

Open coding of Class Test crib sheetsI What is salient on crib sheets?I Was this similar to de Raadt?

Exploratory data analysisI Can we measure / code / classify these features?I Do these features have any relationship with exam performance?

Methods

Project stage 1 - Exploratory analysis Open coding of crib sheets

Preliminary analysis

Initial componentsTheme Category of description Types of difference

Layout

Density Font size, Amount of white space, Location ofwhite space

Emphasis Colouring, Boxing, Headings, Linearity, Box outs,Starring, Separation

Sheet structure Course Structure, Similarity structure, Neatness,Bullets, Orientation, Sub-division

Content

Examples Worked solutions, Boundary examples,Sketches,Transferability of examples

Representations Brevity / Complexity, Calculator procedures, Fillin,the gaps, Other languages

Formulae Listed, Grouped, In situ, Repetition of formulaesheet, Content

Meta-content Reminders, Messages, Thematic commentary,Arrows, Indexing with colour themes

Correctness Errors, Completeness

Four categories of elements

After open coding, we found the following distinction best to work with:

Layout elements

I Structural elements - spacing, boxes, colour, neatnessI Informational elements - density, complexity

these can appear on crib sheets from any discipline

Content — specific to STEM

I Topic elements - what topics are presentI Element types - how are these realised (stating a formula, giving a

specific example etc)

Layout elementsI Structural elements - spacing, boxes, colour, neatnessI Informational elements - density, complexity

Content — specific to STEM

I Topic elements - what topics are presentI Element types - how are these realised (stating a formula, giving a

Layout elementsI Structural elements - spacing, boxes, colour, neatnessI Informational elements - density, complexity

Content — specific to STEMI Topic elements - what topics are presentI Element types - how are these realised (stating a formula, giving a

Project stage 1 - Exploratory analysis Overall trends

Was using a crib sheet worthwhile?

Students who used a crib sheet did better in the examI Used a crib sheet: Mean = 62.5, SE = 6.3I Did not use a crib sheet: Mean = 41.5, SE = 2.1

I Difference is significant: t(96) = 3.58, p < .001, medium effect size:r = 0.34

Confounds aplenty: e.g. students who are not engaged with the course willnot create a crib sheet and not do well on the exam also.This is still true when we control for prior performance

I Partial point-biserial correlation of exam score and crib sheet useI Controlled for both Class Test results and the Online Test result

I Significant difference r = 0.256, p = .016

Linear regression: B = 6.9, so over half a grade’s difference in the exam.

Students who used a crib sheet did better in the examI Used a crib sheet: Mean = 62.5, SE = 6.3I Did not use a crib sheet: Mean = 41.5, SE = 2.1

I Difference is significant: t(96) = 3.58, p < .001, medium effect size:r = 0.34

Students who used a crib sheet did better in the examI Used a crib sheet: Mean = 62.5, SE = 6.3I Did not use a crib sheet: Mean = 41.5, SE = 2.1I Difference is significant: t(96) = 3.58, p < .001, medium effect size:r = 0.34

Confounds aplenty: e.g. students who are not engaged with the course willnot create a crib sheet and not do well on the exam also.

This is still true when we control for prior performanceI Partial point-biserial correlation of exam score and crib sheet useI Controlled for both Class Test results and the Online Test result

I Partial point-biserial correlation of exam score and crib sheet useI Controlled for both Class Test results and the Online Test resultI Significant difference r = 0.256, p = .016

Project stage 1 - Exploratory analysis Structure and layout

The effect of structural / layout elements

Correlations of layout elements was related to information retrievalhypothesis

I Sheets with more ink had more “infomation”I Sheets with “more information” tended to use strategies to ease

information retrieval (colours, lines, titles)

Were any of these related to exam performance?

There is little or no evidence that layout and structural elements arerelated to exam performance.

I Negative correlation between colour and exam performancer = −.278, p < 0.01

(probably spurious)

I Positive correlation between partitioning with lines and examperformance r = .282, p < 0.01

(but no evidence that this is linked toinformation retrieval: sheets with more information and more lines didnot do better than those with just more information)

I Sheets with more ink had more “infomation”

I Sheets with “more information” tended to use strategies to easeinformation retrieval (colours, lines, titles)

(probably spurious)

(probably spurious)I Positive correlation between partitioning with lines and exam

performance r = .282, p < 0.01

I Negative correlation between colour and exam performancer = −.278, p < 0.01 (probably spurious)

I Positive correlation between partitioning with lines and examperformance r = .282, p < 0.01 (but no evidence that this is linked toinformation retrieval: sheets with more information and more lines didnot do better than those with just more information)

Project stage 1 - Exploratory analysis STEM content specific elements

The effect of content elements

For this part of the analysis, we first listed the 29 topics that were presenton the final exam:

I DP / Sig figsI SI unitsI LCM / HCFI Index formI percentagesI Algebra (make the subject)I Geometric ProgressionsI 3D volumeI SoH CaH ToAI sine cosine rule

I trig functions graphsI linear functionsI domainI solve quadraticI complete squareI translating graphsI ellipseI rational inequalityI differentiation “look up”I differentiation “chainrule”

I differentiation ”product”I optimisationI integration “simple”I integration “substitutio”I integration “part”I area under curveI boxplotsI 5 number summaryI correlation

We coded for the presence of each on the crib sheets in four categories:I categories

presentI formulae I specific

examplesI commentary

1014 categories present, 640 examples, 351 formulae, 300 comments

Integration by parts

Comments, Forumla:

Example:

Student-level cluster analysis

We ran a K-means cluster analysis on student-level data (n = 85) ofcounts for “categories”, “specific examples”, “formulae”, and “comments”.

Two, three, four and five cluster solutions were considered

Here a 4-cluster solution was found to be most appropriate, in terms ofcompromise between cluster size and qualitative explanation of clustermembership

Four cluster classification of student crib sheetsStudents whose crib sheets, relative to their peers, included:

I more of everything(scored more highly in continuous assessment, but averagely in others)

I less of everything(scored more highly in summative assignments and examinations)

I more examples(scored worse in all assessment components)

I more comments and fewer examples(scored more highly in almost all assignments)

Is it the case that any of these groups did worse than students who did notuse a crib sheet at all?

Four cluster classification of student crib sheetsStudents whose crib sheets, relative to their peers, included:

I more of everything(scored more highly in continuous assessment, but averagely in others)

I less of everything(scored more highly in summative assignments and examinations)

I more examples(scored worse in all assessment components)

I more comments and fewer examples(scored more highly in almost all assignments)

Is it the case that any of these groups did worse than students who did notuse a crib sheet at all?Ant Edwards & Birgit Loch Are crib sheets sheets worthwhile? 07/07/2015 16 / 20

What is the effect of being in a different cluster?

Regression on examscore for each cluster

I Entered first:scores onpreviousassignments

I Then dummyvariables to codefor being amember of eachof the 4 K-meansclusters

B SE B β p

Step 1Constant 2.82 4.60 p = .541Test 1 (%) 0.44 0.07 .48 p < .001Test 2 (%) 0.37 0.07 .45 p < .001Online (%) 0.05 0.06 .045 p = .406

Step 2Constant 1.66 4.62 p = .723Test 1 (%) 0.43 0.07 .48 p < .001Test 2 (%) 0.36 0.07 .44 p < .001Online (%) -.003 0.07 -.003 p = .960DV All 7.45 4.1 .10 p = .075

DV Example 5.42 3.01 .13 p = .075DV Comment 9.11 3.26 .19 p = .006

DV Sparse 7.59 3.13 .18 p = .018

B SE B β p

DV Sparse 7.59 3.13 .18 p = .018

B SE B β p

DV Sparse 7.59 3.13 .18 p = .018

B SE B β p

DV Sparse 7.59 3.13 .18 p = .018Ant Edwards & Birgit Loch Are crib sheets sheets worthwhile? 07/07/2015 17 / 20

B SE B β p

Summary

OverallI Students who used crib sheets did better than those that did notI This was still true when prior performance was taken into account

Structure and layout elementsI Students used layout to separate information on more dense sheetsI Despite this, having more information on the sheet was not correlated

to increased exam performance

STEM-sepecific elementsI Students who included more examples did not perform significantly

better than students who did not take a crib sheet to the examI Those students who included more comments, or who included

minimal conent, did significantly better than those students who didnot use a crib sheet (≈ 9.1 and 7.6 percentage points better)

Summary

Further work

2015 Semester 1I Testing trends identified with two other cohortsI Formulating ‘good practice’ advice

2015 Semester 2I Interview and focus groups exploring trends identifiedI Educational experiment looking at the effectiveness of ‘good practice’

advice

References

Burns, K. (2014). Security Blanket or Crutch? Crib Card Usage Depends on Students’ Abilities. Teaching of Psychology, 41(1),66-68.Butler, D., & Crouch, N. (2011). Student experience of Making and Using Cheat Sheets in Mathematical Exams. Mathematics:Traditions and New Practices, 131-141.Crawford, K., Gordon, S., Nicholas, J., & Prosser, M. (1994). University Mathematics Students’ Conceptions of Mathematics.Studies in Higher Education, 23(1), 87-94.de Raadt, M. (2012). Student Created Cheat-Sheets in Examinations: Impact on Student Outcomes. Paper presented at theProceedings of the 14th Australasian Coimputing Education Conference, Melbourne, Australia.Dickson, K. L., & Bauer, J. (2008). Do Students Learn Course Material During Crib Sheet Construction? Teaching ofPsychology, 35(2), 117-120.Dorsel, T. N., & Cundiff, G. W. (1979). The cheat-sheet: Efficient coding device or indispensable crutch? Journal ofExperimental Education, 48(39-42).Erbe, B. (2007). Reducing Test Anxiety While Increasing Learning: The Cheat Sheet. College Teaching, 55(3), 96-98.Frank, M. (1988). Problem Solving and Mathematical Beliefs. Arithmetic Teacher, 35(5), 32-34.Funk, S., & Dickson, K. L. (2011). Crib Card Use During Tests: Helpful or a Crutch? Teaching of Psychology, 38(2), 114-117.Garofalo, J. (1989). Beliefs and Their Influence on Mathematical Performance. Mathematics Teacher, 82(7), 502-505.Gharib, A., Phillips, W., & Noelle, M. (2012). Cheat Sheet or Open Book? A Comparison of the Effects of Exam Types onPerformance, Retention and Anxiety. Psychology Research, 2(8), 469-478.Janick, J. (1990). Crib Sheets. Teaching Professor, 4(6), 2.Lawin, K., Gorman, J., & Larwin, D. (2013). Assessing the Impact of Testing Aids on Post-Secondary Student Performance: AMeta-Analytic Investigation. Educational Psychology Review, 25(3), 429-443.Visco, D., Swaminathan, S., Zagumny, L., & Anthony, H. (2007). Interpreting Student-Constructed Study Guides: aConstructivist/Constructionist Perspective. Paper presented at the 6th Annual Meeting, American Institute of ChemicalEnginners.Weimer, M. (1989). Exams: Alternate ideas and approaches. Teaching Professor, 3(8), 3-4.

Are crib sheets sheets worthwhile? · Literature on crib sheets In general, crib sheets are seen as...

Documents

Transcript of Are crib sheets sheets worthwhile? · Literature on crib sheets In general, crib sheets are seen as...