Data Analytics process in Learning and Academic

Analytics projects

Day 3: Data processing

Alex Rayón Jerezalex.rayon@deusto.es

DeustoTech Learning – Deusto Institute of Technology – University of DeustoAvda. Universidades 24, 48007 Bilbao, Spain

www.deusto.es

Table of contents

● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery

Table of contents

● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery

Data dimensionsSummary

[Verbert2011]

Data dimensions1) Computing

● Software○ Example

■ Q1. Among the tools, which is more representative of the final grade?

■ Q5. Which is the impact of the social networks in the group composition?

■ Q6. Which tools are more prone to foster collaboration?

■ Q7. The use of some collaboration tools has effect on the final grade?

● Hardware● Network

Data dimensions2) Location

● Quantitative● Qualitative

Data dimensions3) Time

● Timestamp● Time interval

Data dimensions4) Activity

● Events● Tasks● Goals● Subject

○ Example

■ Q2. Which are the differences in terms of grades

between this subject and other subjects where we already know the final grade?

Data dimensions5) Physical condition

● Noise level● Lighting● ...

Data dimensions6) Resource

● Physical resource● Virtual resource

Data dimensions7) User

● Basic info○ Example

■ Q3. Is there any gender difference in the use of the tools?

● Knowledge● Interest● Goals

○ Short-term○ Long-term

● Learning styles● Affects● Background

Data dimensions8) Relations

● Social relations○ Example

■ Q4. Are there groups of people that repeatedly collaborate in different tools?

■ Q4. Do these groups repeat over time?

● Functional relations● Compositional relations● Proximity● Orientation● Communication

Table of contents

● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery

ApplicationsWhy do learners use analytics?

[Ferguson2014]

● Monitor their own activities and interactions● Monitor the learning process● Compare their activity with that of others● Increase awareness, reflect and self reflect● Improve discussion participation● Improve learning behaviour● Improve performance● Become better learners● Learn!

ApplicationsWhy do teachers use analytics?

[Ferguson2014]

● Monitor the learning process● Explore student data● Identify problems● Discover patterns● Find early indicators for success● Find early indicators for poor marks or drop-

out● Assess usefulness of learning materials

ApplicationsWhy do teachers use analytics? (Ii)

● Increase awareness, reflect and self reflect● Increase understanding of learning

environments● Intervene, advise and assist● Improve teaching, resources and the

environment

Table of contents

● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery

Data processingTransform menu

Data processingScripting menu

Data processingJoins menu

Data processingStatistics menu

Data processingWEKA plugin

Data processingWEKA plugin (II)

Data processingWEKA plugin (III)

Data processingWEKA plugin (IV)

Data processingWEKA plugin (V)

Table of contents

● Data dimensions● Applications● Data processing in an ETL refined data● Knowledge discovery

Knowledge discoveryIntroduction

[BakerSiemens2014]

This review draws on past reviews (cf. Baker & Yacef, 2009; Romero & Ventura, 2010; Ferguson, 2012; Siemens & Baker, 2012)

Knowledge discoveryIntroduction (II)

Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

Knowledge discoveryClassification

1. Prediction methods

2. Structure discovery

3. Relationship mining

Knowledge discovery1) Prediction methods

● The goal is to develop a model which can infer a single aspect of the data ○ The predicted variable

○ Similar to dependent variables in traditional statistical analysis

● … from some combination of other aspects of the data○ Predictor variables

○ Similar to independent variables in traditional statistical analysis

Knowledge discovery1) Prediction methods (II)

● Prediction models are commonly used: ○ Predict future events (Dekker2009; Feng2009;

MingMing2012)

○ Predict variables that are not feasible to directly collect in real-time

■ Example: collecting data on affect or engagement in

real-time often requires expensive observations or disruptive self-report measures

■ Whereas a prediction model based on student log

data can be completely non-intrusive (Sabourin2011)

Knowledge discovery1) Prediction methods (III)

Source: http://etec.ctlt.ubc.ca/510wiki/Learning_Analytics

Knowledge discovery1) Prediction methods (IV)

● Three types of prediction models are common in EDM/LA:○ Classifiers○ Regressors○ Latent knowledge estimation

Knowledge discovery1) Prediction methods (V)

Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

Knowledge discovery1) Prediction methods (VI)

● Classifiers○ The predicted variable can be either a binary (e.g. 0 or

1) or a categorical variable

○ Some popular classification methods in educational domains include:

■ Decision trees

■ Random forest

■ Decision rules

■ Step regression

■ Logistic regression

Knowledge discovery1) Prediction methods (VII)

Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

Knowledge discovery1) Prediction methods (VIII)

● Regressors○ The predicted variable is a continuous variable

■ For example: if the Grade can be explained by the number of pending subjects and the call number

○ The most popular regressor in EDM is linear regression

■ Note that linear regression is not used the same way in EDM/LA as in traditional statistics, despite the identical name

Knowledge discovery1) Prediction methods (IX)

Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

Knowledge discovery1) Prediction methods (X)

Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

Knowledge discovery1) Prediction methods (XI)

● Latent Knowledge Estimation○ Actually is a special type of classifier

○ A student’s knowledge of specific skills and concepts is

assessed by their patterns of correctness on those skills

○ A wide range of algorithms exist for latent knowledge estimation, being the two most popular:

■ Bayesian Knowledge Tracing (Corbett & Anderson, 1995)

■ Performance Factors Analysis (Pavlik2009)

Knowledge discovery1) Prediction methods (XII)

● Classifiers in WEKA are models for predicting nominal or numeric quantities

● Implemented learning schemes include:○ Decision trees and lists, instance-based classifiers,

support vector machines, multi-layer perceptrons, logistic regression, Bayes’ nets, etc.

● “Meta”-classifiers include:○ Bagging, boosting, stacking, error-correcting output

codes, locally weighted learning, etc.

Knowledge discovery1) Prediction methods (XIII)

Knowledge discovery2) Structure discovery

● Attempt to find structure in the data without an a priori idea of what should be found

● It is, actually, a very different goal than in prediction○ In prediction, there is a specific variable that the

EDM/LA researcher attempts to model;

○ By contrast, there is not a specific variable of interest in structure discovery

○ Instead, the researcher attempts to determine what structure emerges naturally from the data

Knowledge discovery2) Structure discovery (II)

● Include:○ Clustering○ Factor analysis○ Social Network Analysis○ Domain Structure Discovery

Knowledge discovery2) Structure discovery (III)

● Clustering○ The goal is to find data points that naturally group

together, splitting the full data set into a set of clusters

○ Clustering is particularly useful in cases where the

most common categories within the data set are not known in advance

○ If a set of clusters is well-selected, each data point in a

cluster will generally be more similar to the other data points in that cluster than data points in other clusters

Knowledge discovery2) Structure discovery (IV)

● Clustering○ Clusters have been used to group students (Beal2006)

and student actions (Amershi2009)

■ Amershi & Conati (2009) found characteristic

patterns in how students use exploratory learning

environments, and used this information to identify more and less effective student strategies

Knowledge discovery2) Structure discovery (IV)

● Factor analysis○ A closely related method

○ Here, the goal is to find variables that naturally group

together, splitting the set of variables (as opposed to

the data points) into a set of latent (not directly observable) factors

○ Factor analysis is frequently used in psychometrics for validating or determining scales

Knowledge discovery2) Structure discovery (V)

● Factor analysis○ In EDM/LA, factor analysis is used for dimensionality

reduction (e.g., reducing the number of variables) for a wide variety of applications

○ For instance, [Baker2009] used factor analysis to

determine which design choices are made in common by the designers of intelligent tutoring systems

■ For instance, tutor designers tend to use principle

based hints rather than concrete hints in tutor problems that have brief problem scenarios

Knowledge discovery2) Structure discovery (VI)

● Social Network Analysis○ Models are developed of the relationships and

interactions between individual actors, as well as the

patterns that emerge from those relationships and interactions

○ Examples

■ Understanding the differences between effective and ineffective project groups [Kay2006]

■ How students’ communication behaviors change over time [Haythornthwaite2001]

■ How students’ positions in a social network relate

to their perception of being part of a learning community [Dawson2008]

Knowledge discovery2) Structure discovery (VII)

● Domain structure discovery○ Consists of finding the structure of knowledge in an

educational domain (e.g., how specific content maps to specific knowledge components or skills, across students)

○ This could consist of mapping problems in educational software to specific knowledge components, in order to group the problems effectively for latent knowledge

estimation and problem selection [Koedinger2006], or could consist of mapping test items to skills [Tatsuoka1995]

Knowledge discovery2) Structure discovery (VIII)

● WEKA contains “clusterers” for finding groups of similar instances in a dataset

● Implemented schemes are:○ k-Means, EM, Cobweb, X-means, FarthestFirst

● Clusters can be visualized and compared to “true” clusters (if given)

● Evaluation based on loglikelihood if clustering scheme produces a probability distribution

Knowledge discovery3) Relationship mining

● Discover relationships between variables in a data set with a large number of variables

● It has historically been the most common category of EDM research [Baker2009]

● It may take the form of attempting to find out which variables are most strongly associated with a single variable of particular interest

● Or may take the form of attempting to discover which relationships between any two variables are strongest

Knowledge discovery3) Relationship mining (II)

● There are four types of relationship mining○ Association rule mining○ Correlation mining○ Sequential pattern mining○ Causal data mining

Knowledge discovery3) Relationship mining (III)

● Association rule mining○ The goal is to find if-then rules of the form that if some

set of variable values is found, another variable will generally have a specific value

○ For instance, [BenNaim2009] used association rule mining to find patterns of successful student performance in an engineering simulation, to make better suggestions to students having difficulty about how they can improve their performance

Knowledge discovery3) Relationship mining (IV)

● Correlation mining○ The goal is to find positive or negative linear

correlations between variables (using post-hoc corrections or dimensionality reduction methods when appropriate to avoid finding spurious relationships)

○ An example can be found in [Baker2009], where correlations were computed between a range of features of the design of intelligent tutoring system lessons and students’ prevalence of gaming the system

Knowledge discovery3) Relationship mining (V)

● Sequential pattern mining○ The goal is to find temporal associations between

events

○ One successful use of this approach was work by

[Perera2009], to determine what path of student collaboration behaviors leads to a more successful eventual group project

Knowledge discovery3) Relationship mining (VI)

● Causal data mining○ The goal is to find whether one event (or observed

construct) was the cause of another event (or observed construct)

○ For example to predict which factors will lead a student to do poorly in a class [Fancsali2012]

Knowledge discovery3) Relationship mining (VII)

● WEKA contains an implementation of the Apriori algorithm for learning association rules○ Works only with discrete data

● Can identify statistical dependencies between groups of attributes:○ milk, butter bread, eggs (with confidence 0.9 and

support 2000)

● Apriori can compute all rules that have a given minimum support and exceed a given confidence

Knowledge discovery3) Relationship mining (VIII)

Knowledge discovery4) Attribute selection

● Panel that can be used to investigate which (subsets of) attributes are the most predictive ones

● Attribute selection methods contain two parts:○ A search method: best-first, forward selection,

random, exhaustive, genetic algorithm, ranking

○ An evaluation method: correlation-based, wrapper, information gain, chi-squared, etc.

● Very flexible: WEKA allows (almost) arbitrary combinations of these two

Knowledge discovery4) Attribute selection (II)

Knowledge discovery4) Attribute selection (III)

References[Amershi2009] Amershi, S., Conati, C. (2009). Combining Unsupervised and Supervised Machine Learning to Build User Models for Exploratory Learning Environments. Journal of Educational Data Mining, 1(1), 71-81.

[BakerSiemens2014] Baker, R., and George Siemens. "Educational data mining and learning analytics." Cambridge Handbook of the Learning Sciences: (2014).

[BakerYacef2009] Baker, R.S.J.d., Yacef, K. (2009). The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Educational Data Mining, 1 (1), 3-17

[Beal2006] Beal, C.R., Qu, L., & Lee, H. (2006). Classifying learner engagement through integration of multiple data sources. Paper presented at the 21st National Conference on Artificial Intelligence (AAAI-2006), Boston, MA.

[CorbettAnderson1995] Corbett, A.T., Anderson, J.R. (1995). Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.

[Dawson2008] Dawson, S. (2008). A study of the relationship between student social networks and sense of community. Educational Technology & Society, 11(3), 224-238.

[Dekker2009] Dekker, G., Pechenizkiy, M., and Vleeshouwers, J. (2009). Predicting students drop out: A case study. Proceedings of the 2nd International Conference on Educational Data Mining, EDM'09, 41-50

[Fancsali2012] Fancsali, S. (2012) Variable Construction and Causal Discovery for Cognitive Tutor Log Data: Initial Results. Proceedings of the 5th International Conference on Educational Data Mining, 238-239.

[Feng2009] Feng, M., Heffernan, N., & Koedinger, K. (2009). Addressing the Assessment Challenge in an Intelligent Tutoring System that Tutors as it Assesses. User Modeling and User-Adapted Interaction, 19, 243-266

[Ferguson2012] Ferguson, R. (2012). The State Of Learning Analytics in 2012: A Review and Future Challenges. Technical Report KMI-12-01, Knowledge Media Institute, The Open University, UK. http://kmi.open.ac.uk/publications/techreport/kmi-12-01

[Ferguson2014] Learning analytics FAQs [Online]. URL: http://www.slideshare.net/R3beccaF/learning-analytics-fa-qs

[Haythornthwaite2001] Haythornthwaite, C. (2001). Exploring Multiplexity: Social Network Structures in a ComputerSupported Distance Learning Class. The Information Society: An International Journal, 17 (3), 211-226.

[Kay2006] Kay, J., Maisonneuve, N., Yacef, K., Reimann, P. (2006) The Big Five and Visualisations of Team Work Activity. Proceedings of the International Conference on Intelligent Tutoring Systems, 197 – 206.

References (II)[Koedinger2006] Koedinger, K. R., & Corbett, A. T. (2006). Cognitive Tutors: Technology bringing learning science to the classroom. In K. Sawyer (Ed.) The Cambridge Handbook of the Learning Sciences (pp. 61-78). New York: Cambridge University Press.

[MingMing2012] Ming, N.C., Ming, V.L. (2012). Predicting Student Outcomes from Unstructured Data. Proceedings of the 2nd International Workshop on Personalization Approaches in Learning Environments, 11-16.

[Pavlik2009] Pavlik, P.I., Cen, H., Koedinger, K.R. (2009). Performance Factors Analysis -- A New Alternative to Knowledge Tracing. Proceedings of AIED2009.

[Perera2009] Perera, D., Kay, J., Koprinska, I., Yacef, K., and Zaiane, O.R. (2009). Clustering and Sequential Pattern Mining of Online Collaborative Learning Data. IEEE Transactions on Knowledge and Data Engineering, 21(6), 759-772

[RomeroVentura2010]Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state-ofthe-art. IEEE Transaction on Systems, Man and Cybernetics, part C: Applications and Reviews, 40(6), 610–618

[Sabourin2011] Sabourin, J., Rowe, J., Mott, B., Lester, J. (2011). When Off-Task in On-Task: The Affective Role of Off-Task Behavior in Narrative-Centered Learning Environments. Proceedings of the 15th International Conference on Artificial Intelligence in Education, 534-536.

[SiemensBaker2012] Siemens, G., Baker, R.S.J.d. (2012). Learning Analytics and Educational Data Mining: Towards Communication and Collaboration. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge.

[Tatsuoka1995] Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment, 327–359. Hillsdale NJ: Erlbaum

[Verbert2011] Dataset-driven research to improve TEL recommender systems [Online]. URL: http://www.slideshare.net/kverbert/datasetdriven-research-to-improve-tel-recommender-systems

Data Analytics process in Learning and Academic

Analytics projects

Day 3: Data processing

Alex Rayón Jerezalex.rayon@deusto.es

DeustoTech Learning – Deusto Institute of Technology – University of DeustoAvda. Universidades 24, 48007 Bilbao, Spain

www.deusto.es