Network Visualization of Human and Machine-based Educational Standard Assignment

IV10 – London, UK

René Reitsma1, Anne Diekema2

1Oregon State University2Utah State University

Network Visualization of Human and Machine-based Educational Standard Assignments – Overview

• Problem: need for automated educational standard assignment.

• Question 1: how are machine-based assignments (e.g., CAT) different from human (cataloger) assignments?

• Question 2: how do humans assign these standards in the first place?

• (Inductive) method:– Generate standard networks of both the human and machine-based

assignments.

– Explore the structure and visual layout of both networks for indications of differences.

The Standard Assignment Challenge

DL Collection K-12 items Estimation method

Teachers’ Domain(www.teachersdomain.org)

3,373 Search query on www.nsdl.org

TeachEngineering(www.teachengineering.org)

812 Document listing on Website

NetTrekker(www.nettrekker.com)

300,000 Promotional video

Engineering is Elementary(www.mos.org/eie/)

78 Direct count on Website

Middle School Portal(www.msteacher.org)

2,562 Search on www.nsdl.org

Curriki(www.curriki.org)

32,222 Resource count displayed on home page

National Science Digital Library(www.nsdl.org)

79,155 Estimate by NSDL official

The Standard Assignment Challenge… cont.’d

• In USA, K-12 teaching & learning is increasingly standards based (‘No Child Left Behind’ Act – 2001)

• Educational standards (K-12) are governed by the 50 states.

• States have different views and opinions

• 50,000+ K-12 Science, Technology, Engineering & Math (STEM) standards: MT: 20, AK: 50, IA: 60, ..., NY: 2,309, IL: 2,881, TN: 3,410

• Teachers must find curriculum to teach to the standards.

• Curriculum providers must assign standards to their curriculum in order to assist the teachers.

• Available curriculum grows quickly; states regularly change/update their standards.

The Standard Assignment Challenge… cont.’d

• Manual assignment is (very) time consuming Automatic assignment tools have been under development for some time.

• Tool of choice: Content Assignment Tool (CAT):

– Center for Natural Language Processing (CNLP) – Syracuse University

– Part of NSF’s National Science Digital Library (www.nsdl.org)

– NLP technology – part-of-speech tags (verbs, nouns, etc.), TF/IDF, etc.

– CAT web service:

• Point CAT to an XML/HTML version of a curricular document.

• Tell it which standard body to assign from.

• Returns a list of assigned standards.

That’s All Very Nice, but…

• How good are those CAT assignments?

• What do we really mean when we say that a standard and a curricular item line up? (Reitsma, Marshall, Zarske (IPM – 2010))

• If the human and CAT assignments are different, are these differences systematic?

• Do the human assignments provide suggestions on how to improve the machine-based ones?

(Inductive) Method & Data

• Notion: visualize both machine-based and human assignments and compare and contrast the visualizations for clues.

• Approach: build networks of standards; layout the networks, interpret their spatial arrangements:

– Networks are based on how standards have been assigned to curriculum.

– Any two jointly assigned standards are considered ‘linked.’

• Data:

– TeachEngineering collection (www.teachengineering.org) – Jan. 2009.

– CO 2007 Science standards.

– CAT & human standard assignments.

CO 2007 Science Standard Assignments

Human Catalogers(CO Curriculum)

CAT (CO curriculum

cataloged by humans)

Curricular items assigned

86 86

Assignments 324 139

Standards covered

63 47

Standard reuse rate

5.14 2.96

Mean number of curricular items

per standard

0.73 0.55

Mean number of assignments per

curricular item

3.78 1.61

CO 2007 Science Standard Assignments... cont.’d

CAT assignments (CO curriculum cataloged by humans)

Yes No Total

Human cataloger

assignments

Yes 25 299 324

No 114 NA

Total 139

– CAT recall = 25 / 324 = .077

– CAT precision = 25 / 139 = .18

‘Curricular units’

– Human network is denser and more clustered.

– Human curricular units are clusters

– Human clusters link through common standards.

– CAT: open structure; less clustering. Has no sense of curricular units.

Weighted or unweighted?

• FR diagrams consider the network unweighted; i.e., all links have equal value/weight.

• Two weights:

– TF/IDF-like: weigh a standard link inversely proportional to the size of its company.

– ‘Fidelity:’ weigh a link between standards proportional to their mutual fidelity across the collection.

• Compute the KK network layouts

• Resulting KK diagrams showed essentially the same properties as the FR diagrams (hierarchical cluster analysis of two-dimensional positions)

Standards: ‘Method’ vs. ‘World’

• CO Science standards can be grouped and subdivided in two categories:

– World standards (W): express facts and principles about the empirical world.

• E.g., S103EC87: Light and sound waves have distinct properties: frequency, wavelength and amplitude.

– Method standards (M!): express ways and means of conducting science.

• E.g., S103ECE9: A controlled experiment must have comparable results when repeated.

– Some method standards ‘contaminated’ with world terms/examples (M):

• E.g., S103ECD4: Technology is needed to explore space (for example: telescopes, spectroscopes, spacecraft, life support systems).

• Question: How do CAT and human catalogers compare on World vs. Method?

Standards: ‘Method’ vs. ‘World…’ cont.’d

Standards M & M!standards

M & M!standards %

M! standards

M! standards %

Humans 63 21 33.33 16 25.40

CAT 47 9 19.15 3 6.38

W = worldM! = (pure) methodM = method with world examples

– CAT under-assigns method.

– Humans: method standards as curricular hubs

– CAT central method hub: S103EC77:

“physical properties of solids, liquids, gases and the plasma state and their changes can be explained using the particulate nature of matter model“

Conclusions & Recommendations

• Visualization of standard assignment networks reveals important differences between CAT-based and human-based classification.

• Tools such as CAT can benefit from contextual knowledge; e.g., that certain lessons are part of a larger set of lessons or a curricular unit.

• Humans organize curriculum around both World and Method standards. Hence, tools such as CAT MUST!! become better at recognizing Method standards.

• In USA strong need for automated standard assignment.

• CAT is a recognized and well-used automated classifier.

• CAT could use improvement, particularly in the Method segment.