Post on 24-Dec-2015
Motivation
• Automated assessment systems are very helpful– Reduce load for instructors– Quicker feedback for students
• But have limited scope– Black-box (input-output) tests, unit tests– Lately code style checks
• Software metrics are not used– Utilized only by plagiarism detection– Not to help students improve their skills
Motivation (cont.)
• Software metrics can tell a lot about a program– Have highly diverse methodologies and objectives– Difficult to define quality for various programs in uniform
way– What which metric means for particular assignment?– What metrics matter the most?
• We explore the usage of software metrics in automated assessment– To cover testing of aspects that demand instructor attention– Algorithm complexity– Number and size of programming units and functions
Student solutionMODULE Triplets1;...VAR x, y, z: INTEGER;BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END ENDEND Triplets1.
Student solution
Parsing the solution• eCST generator
(the SMIILE tool)– Recognizes input language– Calls appropriate scanner & parser– Generates an eCST representation
• Enriched Concrete Syntax Tree– Modified Concrete Syntax Tree– Enriched with marker nodes
• Unit: class, module…• Loop Statement: for, while, repeat…• Branch Statement: if, case, switch…
– Independent of programming language
– Basis for calculation of software metrics
Student solution
eCSTParse
Measuring eCST• Metrics calculator
(the SMIILE tool)– Software metrics tool in the
development– Analyzes the eCST representation– Calculates software metrics– Exports results in XML file
• Final result is one XML file with values of all calculated metrics
• Current prototype of SMIILE tool supports several software metrics– Lines of Code– Cyclomatic Complexity– …
Student solution
Metric values
in xml
eCSTParse
Measure
Metric values in XML<?xml version="1.0"?><sourceElement endLine="24" firstLine="1" name="" type=""> <metrics loc="24" cc="4"/> <sourceElement endLine="24" firstLine="1" name="Triplets1" type="CONCRETE_UNIT_DECL"> <metrics loc="24" cc="4"/> <sourceElement endLine="23" firstLine="9" name="FOR" type="LOOP_STATEMENT"> <metrics loc="15" cc="4"/> <sourceElement endLine="22" firstLine="10" name="FOR" type="LOOP_STATEMENT"> <metrics loc="13" cc="3"/> <sourceElement endLine="21" firstLine="11" name="FOR" type="LOOP_STATEMENT"> <metrics loc="11" cc="2"/> ... </sourceElement> </sourceElement> </sourceElement> </sourceElement></sourceElement>
Student solution
Metric values
in xml
eCSTParse
Measure
Transform XML to properties
• XSL Transformator– Uses XSLT stylesheet
• Input XML file– Values of calculated
software metrics
• Output XML file– Can be manipulated
easily inside the testing system
Student solution
Metric values
in xml
eCST
Matric values
as properties
Parse
Measure
Transform
Metric values as properties<?xml version="1.0" encoding="UTF-8"?><metrics> <Triplets1> <loc>24</loc> <cc>4</cc> </Triplets1></metrics>
Student solution
Metric values
in xml
eCST
Matric values
as properties
Parse
Measure
Transform
Testing the values• Testing system (Testovid)
– Implemented as a framework for running domain specific testers
• Domain specific testers– Written as Apache Ant scripts
• Using software metrics– Script runs SMIILE tool– Transforms XML file with metrics
values– Loads calculated values– Freely uses them for grading,
intelligent advice generation…
• Final report– Contains advices– Success/failure information– Presented to the student
Student solution
Metric values
in xml
Script with
metric control values
eCST
Matric values
as properties
Testreport
Parse
Measure
Transform
Test
Test reportCourse: Data structures and algorithmsAssignment: Assignment 1 - Pythagorean tripletsStudent: John DoeTime: 05.09.2012. 11:15:00
Compilation100% (from 2 points)All is well, no errors.
Correctness33% (from 6 points)Not all triplets were found, check loop boundaries.
Optimality50% (from 2 points)Try using Euclid's formula.
-+-+-+-+-+-+-+-+-
Total: 5 out of 10 points.
Student solution
Metric values
in xml
Script with
metric control values
eCST
Matric values
as properties
Testreport
Parse
Measure
Transform
Test
A case study
• Data structures and algorithms course• Testing efficiency of a Modula 2 program• Using cyclomatic complexity metric– detect loop and branch statements
• Created a domain specific tester for Testovid– differentiate between typical student solutions– award points accordingly
• The assignment– Write a program which prints Pythagorean triplets,
positive integer numbers x, y and z for which x2+y2=z2
Solution 1: Naive solutionMODULE Triplets1;...VAR x, y, z: INTEGER; zreal: REAL;BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO zreal := REAL(Sqrt(LONGREAL(x*x + y*y))); z := TRUNC(zreal); IF zreal = FLOAT(z) THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END ENDEND Triplets1.
• Cyclomatic complexity3
• EfficiencyAverage
• Points50%
Solution 2: Brute force solutionMODULE Triplets2;...VAR x, y, z: INTEGER;BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END ENDEND Triplets2.
• Cyclomatic complexity4
• EfficiencyBad
• Points0%
Solution 3: Using Euclid’s formulaMODULE Triplets;...VAR x, y, z, m, n: CARDINAL;BEGIN FOR m := 1 TO Gr DO FOR n := 1 TO m-1 DO x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END ENDEND Triplets.
• Cyclomatic complexity2
• EfficiencyGood
• Points100%
Non solutionMODULE Triplets4;...VAR x, y, z, m, n, w, i, temp : CARDINAL;BEGIN w := 1; n := 0; FOR i := 1 TO Gr DO m := n + w; x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2); temp := w; w := 3*w + 4*n; n := 2*temp + 3*n ENDEND Triplets4.
• Cyclomatic complexity1
• EfficiencyExcellent
• Points100%
• Lose pointsfor correctness
Summary• The greater the cyclomatic complexity
the worst the solution efficiency• Awarded points should be reverse
proportional to the cyclomatic complexity– Inefficient solutions – no points– Average solutions – half the maximum
points– Efficient solutions – all the points
• Instructor should use knowledge and experience to– Choose metrics– Define minimum and maximum metric
values– Define awarded points for those cases
• Testing system can automatically– Classify student solutions– Grade them accordingly
Solution Efficiency Metric Points
Triplets1 Average 3 50%
Triplets2 Bad 4 0%
Triplets3 Good 2 100%
Triplets4 Excellent 1 !!! 100%
Se-ries1
Conclusion
• Utilized software metrics in the assessment process• Increased the scope of aspects that can be covered
by automatic tests• Platform and programming language independent• Support a wide range of metrics• Left great flexibility in selecting interesting metrics• Can provide hints and advices to students• Added intelligent assistance• Improved student learning experience