What is the question?IMPACT OF QUESTION LENGTH AND ILLUSTRATION SUPPORT ON THE SUCCESS AND SKIP RATES OF PRE-UNIVERSITY MOCK ONLINE TESTS

Computational thinking beyond CS

Students of technical drawing are required to possess a computational thinking mindset (Wing, 2006)

◦ Require advanced spatial reasoning◦ Solve complex problems◦ Use abstraction and decomposition

Which piece fits perfectly the one on the left?

Online quizzes as educational tools

Quizzes are standardized tests with a restricted set of possible answers, allowing easier (and often automated) grading.

In the European Higher Education Area, the Bologna process emphasizes the importance of self-learning.

How can online quizzes be useful in education, beyond grading?◦ Students can self-assess their learning process anytime, anywhere and any number of times◦ Receive immediate feedback on their performance

Case of study This paper analyzes over 20,000 responses of the preparatory online quiz for the Spanish University Admission Test (Selectividad) of the technical drawing subject.

The Selectividad is closely related to other admission tests to colleges and universities, such as SAT in the US, A Level in the UK, Abitur in Germany or Baccalauréat in France.

With the objective of improving future iterations of the test, the authors wanted to research the variables that:

◦ Increased the number of successful attempts ◦ Lowered the number of questions skipped

Experimental data The analyzed data included the following variables on the results of the online mock test:

◦ Year collected (2009 or 2015)◦ Intake (June or September)◦ Summary of the number of responses passed, failed or disregarded (per question)

In addition, more data was derived from the content of the question:◦ Whether the question was only textual or included an illustration◦ Length of the question (including all responses), as a proxy of reading comprehension difficulty◦ Knowledge subdomain inside the subject of technical drawing

Results: Influence of image support

To research the influence of image support, a series of density graphs were produced showing the pass, fail and skip rates when the question was text only or had the support of an illustration in two the years studied:

◦ The skip rate was much lower when the question had image support

◦ The pass ratio was higher when a figure was present

◦ When overlaying the same graph for both years the results were very similar regardless of the 6 years elapsed

Results: Performance across years

To reveal the changes across the 6-year span, a dot plot was used, depicting the pass, fail and skip rates of each question, reflecting changes in educational methodology or student profile between those years:

◦ On 2009 the questions were more successfully answered and less likely to be skipped than 6 years later

◦ Questions with image support showed higher success rates overall, and their skip ratio was much lower (and in many cases close to zero)

Results: Performance between intakes

The relationship between the results in both intakes was examined in detail in a graph that combined a violin plot with a box plot:

◦ On both years the fail rate was higher in the September intake (the students who take the September exam are the ones who did not pass the June exam)

◦ The skip rates and pass rates of both years were reversed, suggesting that September students were more inclined to answer on 2015 than on 2009 but, were less successful in their attempts

Results: Influence of question length

To measure reading comprehension, the studied ratios were plotted against question length as scatterplots, weighted by number of responses:

◦ The success ratio slightly decreased from 2009 to 2015, but the skip ratio was almost the same on both years

◦ The success ratio greatly increased when a picture was included

◦ The students tended to skip longer questions more often

◦ The questions with illustrations were rarely skipped

Conclusions on the results Questions with image support received more successful attempts (i.e. students selected the correct answer) while at the same time significantly reduced the percentage of questions skipped, compared to the questions without images.

Students were more confident about their knowledge when dealing questions with image support (were skipped less often), and therefore these questions were:

◦ Less difficult◦ Easier to understand ◦ More capable of motivating the students

As the length of the question increased (as a proxy of reading comprehension difficulty), the success rate decreased while the skip rate increased.

Conclusions on quiz design The findings suggest that students struggled to fully understand the longer questions or failed to read all the answers of the multiple choice test. The fact that longer questions were skipped more often supports this hypothesis, because students might abandon the question if they perceived that it was too long either because they felt they would leave less time to answer the shorter questions, or because they had trouble understanding what they were asked about.

Identifying the causes of these results should allow the authors to either include more questions with image support or rephrase the non-graphic questions to be shorter or more easily comprehended.

Conclusions on student profile Although the questions in 2009 and 2015 were the same, the analysis of the variation across the 6 years between studies showed a reduction in the success ratio while the skip ratio stayed relatively the same.

The authors suggest that while the students have quickly adopted new mobile and communication technologies, the curriculum they are taught and the way they are tested has not adapted yet, and this disconnect manifests in lower success rates.

Future work While current results have provided insight and further research questions using just statistical graphical methods, future studies will use statistical hypothesis testing to provide more robust conclusions while at the same time providing a measure of the influence of each factor using multiple regression analysis.

To measure question complexity more accurately, the semantic analysis of the content of the question will also be explored in future studies.

Thank You!

Research supported by the Non-Oriented Fundamental Research Project EDU2012-37247/EDUCof the VI National Plan for Scientific Research, Development and Technological Innovation 2008-2011, Government of Spain,

titled “E-learning 3.0 in the teaching of architecture. Case studies of educational research for the foreseeable future”

David FonsecaArchitecture School – La Salle Universitat Ramon Llull

fonsi@salle.url.edu

Ernest Redondo, Joaquim Regot, Francesc Valls, Lluís GiménezBarcelona School of Architecture – BarcelonaTech

{ernesto.redondo, joquim.regot, francesc.valls, lluis.gimenez}@upc.edu