1. Knewton Adaptive LearningBuilding the worlds most powerfuleducation recommendation engineTable of Contents 2 Introduction 3 What Knewton means by adaptive learning 4 Theories & approaches behind Knewton recommendations 6 Why the Knewton Adaptive Learning PlatformTM is so effective 9 Big data & adaptive infrastructure11 Knewton Adaptive Learning in the classroom12 How Knewton Adaptive Learning engages students13 How Knewton Adaptive Learning empowers teacherswww.knewton.com

2. IntroductionAdaptive learning, n.:A teaching method premised on the idea that the curriculum should adapt to each user.On a basic level, the definition of adaptive learning seems simple. But diga bit deeper, and the nuances of the term begin to reveal themselves.There are many different degrees and types of adaptive learning (single-point vs. continuous, adaptive testing vs. adaptive learning), but oftenthese distinctions arent made clear. As the quest for personalizedlearning gains traction among educators, and more and more productsclaim adaptive learning capabilities, a certain fuzziness has emergedaround the term.Here at Knewton we spend our days and nights thinking about adaptivity,as we iterate on and improve our Adaptive Learning PlatformTM. Thiswhitepaper is intended to clarify our interpretation of adaptive learning;provide a glimpse into some of the theories behind our recommendationengine; and explain the effect Knewton adaptive learning can have in aclassroom setting. We hope you find it informative.www.knewton.com Knewton Adaptive Learning 2 3. I What Knewton means by adaptive learningAdaptive learning is a term that has been tossed around a good deal recently in edtech circles.When most people use this buzzword, what theyre really discussing iseither a) single point adaptivity, which evaluates a students performanceat one point in time in order to determine the level of instruction ormaterial he receives from that point on, or b) adaptive testing, whichdetermines a students exact proficiency level using a fixed number ofquestions.When Knewton refers to adaptive learning, it means a system that iscontinuously adaptive that responds in real-time to each individualsperformance and activity on the system and that maximizes thelikelihood a student will obtain her learning objectives by providing theright instruction, at the right time, about the right thing. In other words,while adaptive testing answers the question, How do I get the mostaccurate picture of a students state of knowledge with a fixed numberof questions?, adaptive learning answers the question, Given whatwe understand about a students current knowledge, what should that figure a.student be working on right now? T H E KNE W T O N R E COMMEN D ATIONTo provide continuously adaptive learning, Knewton analyzes learningmaterials based on thousands of data points including concepts,structure, difficulty level, and media format and uses sophisticatedalgorithms to piece together the perfect bundle of content for eachstudent, constantly. The system refines recommendations throughnetwork effects that harness the power of all the data collected for allstudents to optimize learning for each individual student. www.knewton.com Knewton Adaptive Learning3 4. II Theories & approaches behindKnewton recommendationsNo two students are identical they learn and forget at different rates, come from different educationalbackgrounds, and have different intellectual capabilities, attention spans, and modes of learning. As a result,designing a real-time recommendation engine that is sensitive to the characteristics of each student is animmensely complex task.Here at Knewton we address this challenge head-on, using educationalpath planning technologies and advanced models of student ability.These technologies and models ensure that every student progressesthrough the course material in a way that maximizes his learning.Heres a quick look at some of the theories and approaches behind theKnewton recommendation engine:Item Response Theory (IRT)Imagine that youre teaching a math remediation course full of fourth item response functions forgraders. Youve just administered a test with 10 questions. Of those 10various difficulty valuesquestions, two questions are very simple, two are incredibly hard, and therest are of medium difficulty. Now imagine that two of your students take probability of answering correctlythis test. Both answer nine of the 10 questions correctly. The first studentanswers an easy question incorrectly, while the second answers a hardquestion incorrectly. Which student has demonstrated greater mastery ofthe material?Under a traditional grading approach, you would assign both studentsa score of 90 out of 100, grant both of them an A, and move on to thenext test. But this approach illustrates a key problem with measuringstudent ability via testing instruments: test questions do not have uniformcharacteristics. So how can we measure student ability while accountingstudent abilityfor differences in questions? figure b.IRT models student ability using question level performance instead I T E M R E SPON SE TH EOR Yof aggregate test level performance. Instead of assuming all questionscontribute equivalently to our understanding of a students abilities,IRT provides a more nuanced view on the information each questionprovides about a student. It is founded on the premise that the probabilityof a correct response to a test question is a mathematical functionof parameters such as a persons latent traits or abilities and itemcharacteristics (such as difficulty, guessability, and specificity to topic).Figure B shows two item response function curves generated by anIRT model. The curves illustrate how an IRT model relates a students www.knewton.comKnewton Adaptive Learning 4 5. ability with the probability of answering a question correctly, giventhat questions difficulty and discrimination levels and guessability.While IRT models are atemporal and reliant upon a single measure ofability (and thus reflect only one facet of the science behind Knewtonrecommendations), they help us better understand how a studentsperformance on testing relates to his ability.Probabilistic Graphical Models (PGMs)This framework, which encompasses statistical methods such asBayesian networks and Markov random fields, allows data scientists tocode and manipulate probability distributions over multi-dimensionalspaces in which there are hundreds or even thousands of variables atplay. In other words, PGMs allow Knewton analysts to build complexmodels one effect at a time, relating the many learning activities theyobserve to estimations that are useful for recommendation.One of the ways in which Knewton applies PGMs is by using a studentsknown proficiencies to determine which other topics he may beready to master. For instance, such a model might help the platformdiscover to what degree a mastery of fractions helps students masterdecimals and to what degree a mastery of decimals helps studentsmaster exponentiation. Knewton data scientists can thus determinefigure c.the relationship between mastery of fractions and mastery ofpro ba bi listic g raphical mode lexponentiation. Ultimately, the discovery of these types of relationshipsallows the Knewton Adaptive Learning PlatformTM to continually refineits recommendations. Hierarchical Agglomerative ClusteringIn data mining, hierarchical clustering is a method of analysis whichaims to construct a hierarchy or structure of clusters. At Knewton, thetechnique is used to detect latent structures within large groups andbuild algorithms that determine how students should be grouped andwhat features they should be grouped by. An implementation of thistechnique is incorporated in Knewton Math Readiness, which providesFields of knowledge: correlation of studentsa dashboard that allows teachers to group students who are working performanceon the same material by level of concept mastery. figure d.H ierarchica l a g g lomerative c lusterin gwww.knewton.comKnewton Adaptive Learning5 6. IIIWhy the Knewton Adaptive Learning PlatformTM is so effectiveThe science behind recommendation is enhanced by the disciplinary range and tremendous scope of Knewtonadaptivity. A cross-disciplinary Knowledge Graph, continuous adaptivity, lifelong student learning profiles,and vast network effects combine to produce powerfully personalized learning for every student who takes aKnewton-powered course.Knowledge GraphTMKnewton-powered courses are linked by the Knewton KnowledgeGraphTM, a canonical cross-disciplinary graph of academic concepts.The Knowledge GraphTM takes into account these concepts defined bysets of content and the relationships between those concepts. Knewtonrecommendations steer students on personalized and even cross-disciplinary paths on the Knowledge GraphTM towards ultimate learningobjectives based on both what they know and how they learn. The morecontent that teaches or assesses each concept that is added to thesystem, the more precise the adaptive course experience becomes.LaurenWhen visualized, the Knowledge Graph can provide a sense of aTMstudents potential flow through the course material.Within the Knowledge GraphTM, concepts have prerequisite relationshipsthat contribute to defining a students path through the course. Specialfigure e.L A U R E N S PER SON AL IZEDrelationships that define content as either instructional or assessment L E A R N IN G PATH WAYdetermine what kind of content to deliver to students at any given point.Continuous, as opposed to single-point adaptivityA single-point adaptive learning system evaluates a studentsperformance at one point in time, and from there determines the type ofinstruction she receives. An example of single-point adaptivity would bea course that includes a diagnostic exam, the results of which determinesubsequent course content, with little or no further data mining andpersonalization.Knewtons continuously adaptive learning system, on the other hand,constantly mines student performance data, responding in real time toa students activity on the system. Upon completion of a given activity,Williamthe system directs the student to the next activity. For example, when astudent struggles with a particular set of questions, Knewton will knowwhere that particular students weaknesses lie in relation to the conceptsassessed by those questions and can deliver content to increase thestudents proficiency on those concepts. In this way, a continuouslyfigure f.adaptive system provides each student with a personalized syllabus atW i ll iam S PER SON ALIZED L E A R NIN G PATH WAYevery moment.www.knewton.com Knewton Adaptive Learning 6 7. The following are specific examples of approaches that allow Knewtonto offer truly continuously adaptive learning:Spaced reinforcementIn contrast with massed reinforcement, the standard method of drillingwhich requires students to apply new concepts or skills in a shortperiod of time until they demonstrate mastery, spaced reinforcement(also referred to as distributed reinforcement) is a learning methodin which new concepts or skills are absorbed while previously-taught concepts and skills are reinforced. Because new material isintroduced incrementally and woven into familiar material, spacedreinforcement typically occurs over an extended period of time.Spaced reinforcement allows Knewton recommendations to helpstudents build their skills in a cumulative way and retain understandingonce it is gained.Retention & learning curvesThe Knewton recommendation engine needs to be able to take thedegradation or diminishment of skill (or forgetting) into account.That is, it needs to be able to detect such occurrences and provideactionable recommendations as a result.Inspired by Hermann Ebbinghauss work on memory retention andlearning curves, Knewton data scientists have used exponentialgrowth and decay curves to model changes in student ability whilelearning and forgetting. These curves are governed by the followingpremise: each time students are exposed to content associated withmemorya given topic, they receive a bump in their virtual ability level for atopic; likewise, if they are not exposed to some other topic, they likelyforget that topic over time. The forgetting curve itself that governs time remembered (days)rate of retention is roughly described by the following formula: -tR=e s figure g. the f org ettin g c ur vewhere R is memory retention, S is the relative strength of memory, andt is time.By integrating this curve into engine validation efforts, Knewton datascientists can capture the way a students knowledge waxes andwanes, depending on how and when they are exposed to content.www.knewton.comKnewton Adaptive Learning 7 8. Ultimately, the process allows Knewton data scientists to test thealgorithms that govern a students flow through the course.Student learning profileWith Knewton, students can maintain a continuously updated learningprofile that contains information on what the student knows and howshe learns best. The profile is progressive, which means it keeps gettingsmarter the longer the student remains on the platform.For instance, if a student who has already taken a Knewton-poweredcourse enrolls in another, the course starts warm with that studentsdata (as opposed to starting cold with no data). The course takesinto account the students recently mastered concepts and skills andunique trajectory through the material, and uses this knowledge tomaximize student learning continuously from that point forward. Onceenough data is collected, the platform will uncover patterns in thestudents learning, likely blind spots; modality and medium preferences;and granular strengths and weaknesses. The more often a student usesKnewton-powered courses, the more effective the platform becomesat serving up targeted learning material.In this way, the Knewton Adaptive Learning PlatformTM works tominimize unproductive feelings of frustration and confusion and buildstudent skills in a natural way over time. The implications of this arestraightforward: student engagement can be strengthened if academicwork is imbued with a sense of continuity. Nothing is more dissatisfyingto students than feeling like the challenges they face are essentiallyarbitrary and culminate in nothing. The Knewton learning profileanswers the student need for continuity and meaning by affordingstudents a sense of long-term investment in the learning process.www.knewton.com Knewton Adaptive Learning 8 9. IVBig data & adaptive infrastructureThe term big data is used to describe the tremendous volume, velocity, and variety of data generated byvarious technology platforms, many of which involve the continuous or ubiquitous collection of data. Bigdata refers specifically to data sets that are so large and complex that they are challenging to work withusing traditional database management tools; specific challenges include the storage, search, analysis, andvisualization of data.Big Data & EducationThe advent of big data in areas such as internet search and social mediaBig data unleashes a rangehas disrupted existing industries, created new industries, and led to the of productive possibilitiesextraordinary success of companies such as Google and Facebook. Big in the education domain data unleashes a range of productive possibilities in the education domainin particular, since data that reflects cognition is structurally unique fromthe data generated by user activity around web pages, social profiles,and online purchasing habits. Because there is a very high degree ofcorrelation between educational data (mastery of fractions and masteryof exponentiation, for example), there is tremendous potential to optimizeuser experiences over time and provide tangible value for students.One feature that distinguishes the data produced by students (from thatof consumers shopping online, for example) is the fact that academicstudy requires a prolonged period of engagement; students thus remainon the platform for an extended length of time. Furthermore, there is afocus, intention, and intensity to students activity: they are engaging inhigh stakes situations taking a course for credit, trying to improve theirfuture, expanding their range of skills. The sustained intensity of theseefforts generates vast quantities of meaningful data that can be harnessedcontinuously to power personalized learning for each individual.Adaptive InfrastructureKnewton has established an infrastructure that allows the platform toprocess tremendous amounts of student data. For instance, inferenceon probabilistic graphical models is one example of a class of algorithmscalled graph algorithms. These algorithms are special in that they canbe broken down into units of computation that depend only on otherspecific units and can thus be parallelized very efficiently if the work issplit between computers so that limited coordination is required.figure h.Given the absence of robust, public frameworks for accomplishing A ltnode Frameworkthese computations at a large scale, Knewton has designed its ownframework called AltNode which works by dividing work betweenwww.knewton.comKnewton Adaptive Learning 9 10. machines and then sending continuous updates between the minimalnecessary number of machines. All significant updates are stored ina distributed Cassandra database. If one machine fails, another onenearby automatically takes its place, recovering recent values from thedatabase and resuming work. One unique feature of AltNote is that itallows models to recover from any state and respond to new data as itarrives.www.knewton.com Knewton Adaptive Learning 10 11. V Knewton adaptive learning in the classroomAdaptive learning supports mastery-based learning, a school of teaching founded on the idea that studentprogression through a course should be dependent on proficiency as opposed to the amount of time spent onacademic work. Though it may not always be referred to by name, mastery-based learning describes anysituation in which one is given a set of problems, labs, or activities, and in which progression through thatmaterial is dependent on successful completion of various tasks rather than seat time.Online adaptive learning makes it possible to implement mastery-based learning in a scalable way. Knewton Math Readiness, for example, ASU Remedial Math 200920102011 (Knewton)creates a guided, self-paced environment in which live instruction is Mastery 67% 66%74.5%optimized around targeted group sessions. The course is designed topresent students with personal learning paths as it continually assesses Withdrawals 13.2% 13.6% 5.6%their mathematical proficiency and adapts accordingly. Lessons consist Mastery n/a n/a 50%of videos, online textbook selections, and lesson quizzes, and studentsprogress through the course by earning badges. Early efficacy reports figure i.reflect the success of the program: after one semester of use with nearlyKNE W T O N M ATH R EAD IN ESS2,000 remedial math students at ASU, withdrawal rates dropped by 50%,R ESULTS AT ASUpass rates went from 66% to 75%, and 50% of the class finished 4 weeksearly.This implementation model is often referred to as blended learning, aterm which describes any arrangement in which a student learns, at leastin part at a brick-and-mortar facility and in part through online deliverywith student control over time, place, path, or pace.Irene Bloom, a Senior Lecturer at Arizona State University, was originallya skeptic of online learning. But she says that the classroom dynamic hasBefore this, I worked on thechanged for the better since introducing Knewton into her remedial mathassumption that all studentsclasses: I love looking around the classroom and seeing them working in were at the same place. Now,groups, talking to each other and explaining things to each other Most because they progress atof the time, different groups are working on different things, dependingon where they are in the course. This is very new for me. Before this, Idifferent rates, I meet themworked on the assumption that all students were at the same place. Now, where they are.because they progress at different rates, I meet them where they are.www.knewton.comKnewton Adaptive Learning11 12. VIHow Knewton adaptive learning engages studentsKnewton adaptive learning can improve student engagement by increasing self-confidence, decreasingdiscomfort and frustration, and encouraging productive learning habits.Instant FeedbackStudents are less likely to lose focus if feedback is immediate and theycan quickly self-correct. A continuously adaptive learning system isable to deliver personalized feedback to both multiple choice andfree response questions quickly that is, instantaneously or near-instantaneously. The result is pacing that is conducive to risk-taking,experimentation, iterative development, and rapid learning.Community & CollaborationIsolation can exacerbate the challenges students experience in school.An adaptive system can improve student engagement by weaving a socialcomponent into coursework. Knewton Math Readiness, for instance,provides a dashboard that allows teachers to group students who areworking on the same material together. Using the reporting features,figure j.teachers can also arrange peer review opportunities and form groups ofst u dy g ro u ps and peer rev iewstudents whose abilities complement each other.GamificationWith countless opportunities for students to demonstrate skill andreflect on action and feedback, adaptive courses naturally have muchin common with games. Whats more, adaptive courses keep studentsin a game-like state of flow by escalating the difficulty of the workincrementally and unveiling levels one at a time to increase suspense.These and other game elements can be heightened (as in the KnewtonMath Readiness course, which includes progression bars, badges, andachievements) to transform adaptive courses into truly gamified learningexperiences.The process of unlocking work (as reflected in the card metaphordisplayed above) helps heighten the satisfaction students feel in theirprogress. figure k.g ami f ication visual el ementswww.knewton.com Knewton Adaptive Learning 12 13. VII How Knewton adaptive learning empowers teachersKnewton adaptive learning gives teachers insight into the learning process, specifically in terms of efficacy,engagement, and knowledge retention. The platform also provides an unprecedented flexibility of scope;teachers can grasp patterns in student activity and performance across the whole class or drill down intoindividual student profiles to determine exactly why a student is struggling.Addressing the Diverse Needs of StudentsOne of the biggest challenges facing teachers and school administratorstoday is the growing diversity of the students within their population. Agreater diversity of students means a greater diversity of needs to consider.Some struggle because English is not their first language; others havedifficulty with focus or organization. Others may be particularly weak insome area but possess unusual strengths in another.Knewton adaptive learning allows teachers to address the needs ofdiverse students while gaining insight into the learning process. Theplatform may discover, for instance, that a student who is weak withmath word problems is struggling because he has difficulty with readingcomprehension; the system can then direct the teacher to specificmaterial on syntax and vocabulary and suggest ways that the teachermight coach the student individually in these respects. Later, the instructormay be informed that another student who understands mathematicalconcepts but has trouble with carelessness in arithmetic should receivefeedback about how to develop stronger estimation abilities or checkwork once completed. The instructor can then coach that student with aprecise understanding of his particular weaknesses.For example, the Knewton Math Readiness course dashboard includes anon-track/off-track concept to measure student progress through thecourse. This concept functions as a binary indicator that helps teachersefficiently grasp information about the class as a whole. Using this tool andothers, teachers can see reporting data from two perspectives:The Whole ClassThe Knewton Math Readiness instructor dashboard includes a histogramwhich provides a big picture assessment of the whole class track status.Using the dashboard, teachers can also see how students are performingin individual subject areas; which segments of material are the mostand least challenging for students; and what kinds of patterns in bothperformance and activity emerge across the class. After multiple years ofteaching the same course, teachers will be able to compare data from year figure l.to year; Knewton analytics will help them home in on useful informationg ro u p pro g ress dash b oardwhile leaving them free to interpret the results.www.knewton.com Knewton Adaptive Learning 13 14. Individual StudentsWhile the reporting dashboard in Knewton Math Readiness isstreamlined so that teachers can focus on the big picture, thedashboard also allows teachers to click into the interface anddrill down to each students work in the system. Teachers can seehow students have performed on specific quizzes and exams. If astudent isnt grasping the material, teachers can determine (usinganalytics that guide them to specific data points) where precisely anymisunderstanding is occurring. figure m.This capacity allows teachers to both address the diverse needs of indi vidu a l pro g ress dash b oardstudents and better understand their content, so that they can refine itfrom year to year.Improving & Understanding Content for Long-Term CurriculumDevelopmentAn instructor dashboard that measures the efficacy of content can helpteachers determine the strongest and weakest aspects of their teachingmaterials. As described previously (under Item Response Theoryand Network Effects), Knewton adaptive learning can help teachersunderstand precisely what the content they are working with teachesand assesses. This ensures that content can be analyzed for fine-tunedimprovements from year to year, and that students are never stuck withoutdated or ineffective materials.www.knewton.comKnewton Adaptive Learning 14 15. This is just the beginning.We are continually working to refine and improve our adaptive learning technology.Our goal is to work with teachers and institutions to provide every student with a learningexperience that is both highly effective and engaging. We want not only to help studentsmeet specific learning objectives but also provide them with a greater understanding ofthe transformative power of education, the simple joy of learning something new.Interested in learning more about Knewton adaptive learning, or just want to keep up withus on our journey? Contact us at hello@knewton.com.www.knewton.com