Education MED Program Assessment Report for AY 2016-17

April 15, 2018

Program Assessment Overview This 2018 annual report presents the required evidence from the last three academic years that Masters of Education in Education (EDUC MED) candidates preparing to work in schools as teachers know and demonstrate the content knowledge, pedagogical content knowledge and skills, pedagogical and professional knowledge and skills, and the professional dispositions necessary to help all students learn. It should be noted that the program’s name was changed in 2017 from Secondary Education MED to Education MED at the request of the Arkansas Department of Education, because several licensure areas are for K-12th grade instead of just 7-12th grade. For simplicity, we just refer to the new program name across the 2014-15, 2015-16, and 2017-18 Academic Years (AY) throughout this report. The EDUC MED program leads to the MED as well as licensure in several content areas. Due to low numbers of completers in several of these areas, we collapse and aggregate the data across licensure areas because this report will be made public. The data presented below is reported for completers of the program by academic year. The table below shows the number of completers who also received licensure by licensure area by academic year. Please note that some completers may have received licensure in multiple areas. Finally, the numbers of completers data across the assessments vary because we may not have data for each assessment for each academic year for all completers.

Licensure Area 2014-15 AY 2015-16 AY 2016-17 AY

Art 4 1 1

Business Ed 5 1 0

Drama/Speech 1 1 2

English 5 2 3

Foreign Language 1 1 0

Math 1 1 1

Music 1 0 2

Physical Ed/Health 2 0 0

Science 3 8 5

Social Studies 3 1 0

TOTAL 26 16 15

Program Overview: Mission, Goals & Objectives The EDUC MED program prepares teachers for initial licensure in grades K-12 or 7-12 in the following content areas: art (K-12), biology, chemistry, earth science, English, foreign language(K-12), history, mathematics, physical education and health sciences (K-12), physics,

2

social studies, speech and speech/drama. Graduates are required to maintain professional connections with schools, and as graduates, they continue to work collaboratively with school colleagues, parents/guardians, and the community; in addition, they acquire the foundations and dispositions in research, problem-solving, reflective thinking and technology for lifelong learning. The EDUC MED program provides proficiency in professional education, curriculum and instruction, teaching skills, and a teaching specialty. Two tracks are offered within the master’s degree program: the initial licensure track, and the provisional initial licensure track. Assessment data from these two levels are combined in the following report. The EDUC MED program bases this assessment report on the 2016 CAEP Initial Preparation Standards which are mandatory for all initial programs seeking CAEP accreditation. This assessment report focuses on CAEP Standard 1:

Standard 1: Content and Pedagogical Knowledge - The provider ensures that candidates develop a deep understanding of the critical concepts and principles of their discipline and, by completion, are able to use discipline-specific practices flexibly to advance the learning of all students toward attainment of college- and career readiness standards. Candidate Knowledge, Skills, and Professional Dispositions 1.1 Candidates demonstrate an understanding of the 10 InTASC standards at the appropriate progression level(s) in the following categories: the learner and learning; content; instructional practice; and professional responsibility. Provider Responsibilities: 1.2 Providers ensure that candidates use research and evidence to develop an understanding of the teaching profession and use both to measure their P-12 students’ progress and their own professional practice. 1.3 Providers ensure that candidates apply content and pedagogical knowledge as reflected in outcome assessments in response to standards of Specialized Professional Associations (SPA), the National Board for Professional Teaching Standards (NBPTS), states, or other accrediting bodies (e.g., National Association of Schools of Music – NASM). 1.4 Providers ensure that candidates demonstrate skills and commitment that afford all P-12 students access to rigorous college- and career-ready standards (e.g., Next Generation Science Standards, National Career Readiness Certificate, Common Core State Standards). 1.5 Providers ensure that candidates model and apply technology standards as they design, implement and assess learning experiences to engage students and improve learning; and enrich professional practice. (CAEP Accreditation Handbook, March 2016, p. 14)

The required candidate knowledge and skills embedded in Standard 1 are those of the Interstate Teacher Assessment and Support Consortium (InTASC), the Council of Chief State School Officers' project that defined teacher standards (https://www.ccsso.org/resource-library/intasc-

3

model-core-teaching-standards-and-learning-progressions-teachers-10 ). The InTASC standards are also aligned with the Arkansas Department of Model Core Teaching Standards:

Standard #1: Learner Development. The teacher understands how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences. Standard #2: Learning Differences. The teacher uses understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. Standard #3: Learning Environments. The teacher works with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation.

Standard #4: Content Knowledge. The teacher understands the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches and creates learning experiences that make the discipline accessible and meaningful for learners to assure mastery of the content. Standard #5: Application of Content. The teacher understands how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues.

Standard #6: Assessment. The teacher understands and uses multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher’s and learner’s decision making. Standard #7: Planning for Instruction. The teacher plans instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context. Standard #8: Instructional Strategies. The teacher understands and uses a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways.

Standard #9: Professional Learning and Ethical Practice. The teacher engages in ongoing professional learning and uses evidence to continually evaluate his/her practice, particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community), and adapts practice to meet the needs of each learner. Standard #10: Leadership and Collaboration. The teacher seeks appropriate leadership roles and opportunities to take responsibility for student learning, to collaborate with learners, families, colleagues, other school professionals, and community members to ensure learner growth, and to advance the profession.

Data Overview

4

In this report, we present the EDUC MED data as evidence that our EDUC MED initial licensure candidates develop the content and pedagogical knowledge, skills, and dispositions deemed necessary for advancing the learning of all students toward attainment of college and career readiness standards. The mechanism for developing the aggregated data for Unit evaluation of the content and pedagogical knowledge and skills of our initial licensure candidates is referred to as iLab (see Table S1_1.1_INTASC_Crosswalk below for the specific assessments and their alignment with the INTASC standards). These data can be aggregated by program and the entire unit or disaggregated by licensure area within programs. The numerous licensure areas that comprise the EDUC MED use disaggregated data for the various Specialty Professional Association reports for national recognition. However, in this annual report, we have aggregated all specific licensure area data and present the data, analyses, and interpretations for the EDUC MED program as a unit. The assessment system used by the EDUC MED program is part of a broader EPP assessment system for all initial licensure programs that was developed across several years starting in 2006 through the previous NCATE accreditation and subsequently through the various restructuring efforts across the last 3 years. The iLab assessments were developed by faculty to provide a better means for aggregating and comparing candidate performance across our various initial licensure programs. It provides a flexible assessment system that allows for individual program variation in assessments due to various professional goals and standards and specific SPA requirements, while also providing a means for EPP or unit assessment on common rubrics across programs. Each licensure program has its own unique assessment system tied to respective SPA key assessments, but every program is required to include these key assessments of candidate knowledge, skills, and dispositions at common gateways across the programs. Candidates then submit the required performances (assignments, etc.) into their electronic portfolios in Chalk & Wire at both the Program level and the appropriate Unit level (iLab for initial licensure programs). Faculty score these assessments using both their unique program rubrics approved by their respective SPAs for candidate and program assessment and a common EPP/unit scoring rubric for aggregated EPP/unit assessment. The scoring rubrics for iLab were developed by EPP faculty and allow aggregation and comparison across programs using a common scoring standard. Table S1_1.1_INTASC_Crosswalk: TCED iLab Assessments Aligned to INTASC Standards

ILab Assessment INTASC Standards The Learner and

Learning Content

Knowledge Instructional

Practice Professional

Responsibility

1 2 3 4 5 6 7 8 9 10

Graduate Initial Programs

Course Prerequisites* √

Lesson Plan √ √ √ √ √ √ √ √

Case Study √ √ √

TESS – Field Observation* √ √ √ √ √ √ √ √ √ √

Dispositions √ √ √ √

5

Technology √ √

Unit Plan √ √ √ √ √ √ √ √

Teacher Work Sample √ √ √

Teaching Philosophy √ √ √ √

TESS - Internship √ √ √ √ √ √ √ √ √ √

Exit Survey √ √ √ √ √ √ √ √ √ √ * These assessments are not included and data is not provided in this Standard 1 report because data is not gathered

in common across all initial licensure programs for these assessments. CAEP Standard 1.1: Data presented in the following sections of the standard provides evidence that our initial licensure candidates develop the professional knowledge, skills, and dispositions defined by the INTASC Standards. Specifically, evidence, disaggregated by specialty area, is obtained from the following assessments and measures that comprise our EPP/unit assessments, iLab and ADE required Praxis content and pedagogy exams. Table S1_1.1 (above) provides a crosswalk of how the various iLab Assessments are aligned with the specific INTASC Standards. Taken together, the data from the iLab assessments and the state required Praxis content and pedagogy tests provide evidence that our initial licensure candidates demonstrate an understanding of and have acquired the knowledge, skills, and dispositions of the 10 INTASC standards. Data from the iLab assessments and Praxis Exams attached in this standard provide evidence that our completing candidates demonstrate an understanding of the 10 INTASC standards at the appropriate progression levels in the following categories:

1. The Learner and Learning: Lesson Plan, Case Study, Dispositions, Unit Plan, Teaching Philosophy and the TESS-Internship Observation.

4. Professional Responsibility: Dispositions, Teaching Philosophy, and the TESS-Internship Observation.

Analyses of the data/evidence are provided for completers of our initial licensure programs in the 2014-15; 2015-16; and 2016-17 academic years in each of the assessment attachments:

6

S1_1.1_INTASC_Crosswalk: TCED iLab Assessments Aligned to INTASC Standards S1_1.2_Praxis_Content_Exams and_Praxis_Pedagogy_Exams (not inluded due to privacy issues)S1_1.3_Lesson_PlanS1_1.4_Case_StudyS1_1.5__Unit PlanS1_1.6_Dispositions

2. Content Knowledge: Lesson Plan, Unit Plan, Tess-Internship Observation, and the Praxis Content Exams (not inluded due to privacy issues)3. Instructional Practice: Lesson Plan, Technology, Unit Plan, Teacher Work Sample, TESS-Internship Observation, and the Praxis Pedagogy Exams (not included)

S1_1.7_Technology S1_1.8_Teacher_Work_Sample S1_1.9_Teaching_Philosophy S1_1.10_TESS_Internship_Observation Standard 1.2: Data presented in the following attachments provides evidence that our initial licensure candidates use research and evidence to help further develop their teaching knowledge and skills and to assess student progress and the effectiveness of their own teaching. Specifically, evidence, disaggregated by specialty area, is obtained from the following assessments and measures: S1_1.4_Case_Study S1_1.8_Teacher_Work_Sample S1_1.10_TESS_Internship_Observation Analyses of the data/evidence are provided for completers of our initial licensure programs in the 2014-15, 2015-16, and 2016-17 academic years in each of the assessment attachments. Data presented in the attached assessments appear to document that our candidates effectively and successfully use research and evidence for planning, implementing, and evaluating their K-12 students’ progress in their field and clinical placements. Further, it appears that they reflect on their effectiveness and practice and they appear to be able to use data on their students’ performance to assess their progress and to modify instruction and management based on that data. Specifically, review of the Case Study data for initial candidates suggests a majority of our completers of these programs demonstrate a proficient skill in identifying a central issue and developing an action plan for successfully dealing with the issue using specific evidence throughout this process to support evaluations and suggested actions and plans. Further, the candidates do so by demonstrate an understanding of the roles of learner development and differences and the need to collaborate with others in successfully implementing the action plan. Review of the Teacher Work Sample data suggests a majority of our completers of the initial programs demonstrate an emerging to comprehensive understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making. Additionally, candidates show emerging to comprehensive ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context. Finally, they demonstrate an emerging to comprehensive understanding and use of a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways. Review of the pertinent components or subdomains of the TESS Observation Scoring Rubric suggests that a majority of our completers of the initial programs were observed as being proficient to distinguished across the following components related to Standard 1.2: Component 1b: Demonstrating Knowledge of Students; Component 1c: Setting Instructional Outcomes; Component 1e: Designing Coherent Instruction; Component 1f: Designing Student Assessments;

7

Component 3d: Using Assessment in Instruction; Component 3e: Demonstrating Flexibility and Responsiveness; Component 4a: Reflecting on Teaching; Component 4b: Maintaining Accurate Records; and Component 4e: Growing and Developing Professionally. Standard 1.3: Data presented within various Specialized Professional Association (SPA) reports, shows that we ensure our candidates apply content and pedagogical knowledge that meet professional standards. Table S1_3.1_EDUC_MED_Program_Recognition_Status shows that our initial licensure programs that comprise the EDUC MED are either nationally recognized by the Specialty Professional Organizations, endorsed with a similar State review by the Arkansas Department of Education (which uses the NCATE SPA template for its review), or nationally recognized through other professional accrediting bodies (such as NSAD, AACSB, and NASM). Table S1_3.1_EDUC_MED_Program_Recognition_Status

Initial Licensure Programs: Degree Earned

College Specialized Professional Association

Program Status (Spring 2018)

Secondary Education Graduate Initial Licensure (MEd) by Content Area:

Art MED CEHP NSAD*** Nationally Recognized Business Ed MED CEHP AACSB*** Nationally Recognized Drama/Speech MED CEHP ADE ** State Recognized English MED CEHP NCTE* Nationally Recognized Foreign Languages MED CEHP ACTFL* Nationally Recognized Health Science and P.E. MED CEHP ADE (SSPEWL)** State Recognized Math MED CEHP NCTM* Nationally Recognized Science MED CEHP NSTA* Nationally Recognized Social Studies MED CEHP NCSS* Nationally Recognized Music MED CEHP NASM*** Nationally Recognized

* NCATE adopted Specialized Professional Association standards ** Arkansas State Department of Education (ADE) Endorsement (specific SPA/State standards that were followed) *** non-NCATE Specialized Professional Associations/Organizations Standard 1.4: Data presented below suggests that we ensure that our candidates demonstrate skills and commitment to meet rigorous college- and career-ready standards. Specifically, evidence, disaggregated by specialty area, is obtained from the following assessments and measures: ILab: Lesson & Unit Plans; Teacher Work Samples; Case Studies, and TESS field and internship observations, including dispositions. S1_1.3_Lesson_Plan S1_1.4_Case_Study S1_1.5__Unit Plan S1_1.6_Dispositions S1_1.8_Teacher_Work_Sample S1_1.9_Teaching_Philosophy S1_1.10_TESS_Internship_Observation

8

Analyses of the data/evidence are provided for completers of our initial licensure programs in the 2014-15, 2015-16, and 2016-7 academic years in each the above assessment attachments. Taken together, the data provides evidence of our candidates’ ability to provide and differentiate instruction for all students. They plan and teach lessons and units that requires students to apply knowledge in solving problems and thinking critically. Their lessons demonstrate candidates’ ability to include cross-discipline learning experiences and to teach for transfer of skills and to design and implement learning experiences that require collaboration and communication skills. Specifically, review of the lesson plan data suggests that a majority of our completers of the initial programs demonstrate a comprehensive understanding of the use of relevant aspects of student backgrounds in planning instruction; and, candidates clearly articulate instructional goals based on this analysis. Additionally, candidates show mastery in use and selection of appropriate instructional resources and in using evaluation to plan future instruction. Review of Unit Plan data suggests that a majority of our completers of the initial programs demonstrate an emerging to comprehensive understanding of the use of relevant aspects of student backgrounds, student diversity, contextual variables and content knowledge in planning instruction across a unit; and, candidates clearly articulate instructional goals based on this analysis. Additionally, graduate initial licensure candidates show mastery in use and selection of a variety of appropriate instructional resources and in using a variety of assessments to evaluation student learning. The graduate initial candidates also demonstrate emerging to comprehensive understandings of how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues. Review of the Case Study data for initial candidates suggests a majority of our completers of these programs demonstrate a proficient skill in identifying a central issue and developing an action plan for successfully dealing with the issue using specific evidence throughout this process to support evaluations and suggested actions and plans. Further, the candidates do so by demonstrating an understanding of the roles of learner development and differences and the need to collaborate with others in successfully implementing the action plan. Review of the Teacher Work Sample data suggests a majority of our completers of the initial programs demonstrate an emerging to comprehensive understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making. Additionally, candidates show emerging to comprehensive ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context. Finally, they demonstrate an emerging to comprehensive understanding and use of a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways. Review of the Dispositions assessment data suggests that a majority of our completers of the initial programs created a positive environment of respect and their interactions with students, and the interactions of their students with each other, are polite and respectful, reflecting general warmth and caring, and are appropriate to the cultural and developmental differences among

9

groups of students. They also created a classroom culture characterized by high expectations for most, if not all, students and genuine commitment to the subject by both teacher and students, with students demonstrating pride in their work. The completers of the initial licensure programs also promoted the successful learning of all students, adjusting as needed to instruction plans, and accommodating student questions, needs, and interests. Most completers also communicated frequently with families and successfully engaged them in the instructional program. Information to families about individual students was conveyed in a culturally appropriate manner. Finally, our candidates exhibited a high level of ethics and professionalism in dealings with both students and colleagues and complied fully and voluntarily with school and district regulations. Review of the Teaching Philosophy data suggests that a majority of our completers of the initial programs demonstrate an emerging to comprehensive understanding that all students can learn but may require accommodations to individual differences based on the understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas and designs and implements. They also appear to have developed an emerging to comprehensive understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. A majority of our completers also appear to have developed ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation. Finally, they appear to have developed a general and specific theoretical understanding of the need for ongoing professional learning and using evidence to continually evaluate his/her practice, particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community), and adapts practice to meet the needs of each learner. Finally, Review of these tables suggests that a majority of our completers of the initial programs were observed as being proficient to distinguished across all four broad domains and all 22 components or subdomains of the TESS Observation Scoring Rubric. While there is some variation across some programs, in general, our initial licensure candidates:

• plans and practice reflect solid knowledge of the content, prerequisite relationships between important concepts, and the instructional practices specific to that discipline.

• actively seeks knowledge of students' backgrounds, cultures, skills, language proficiency, interests, and special needs, and attains this knowledge for groups of students.

• Instructional outcomes are stated as goals reflecting high-level learning and curriculum standards. They are suitable for most students in the class, represent different types of learning, and can be assessed. The outcomes reflect opportunities for coordination.

• Are fully aware of the resources available through the school or district to enhance own knowledge, to use in teaching, or for students who need them.

• coordinate knowledge of content, of students, and of resources to design a series of learning experiences aligned to instructional outcomes and suitable for groups of students. The lesson or unit has a clear structure and is likely to engage students in significant learning.

• plans for student assessment are aligned with the instructional outcomes, uses clear criteria, and is appropriate to the needs of students. The candidate intends to use assessment results to plan for future instruction for groups of students.

• Classroom interactions between the teacher and students and among students are polite and respectful, reflecting general warmth and caring, and are appropriate to the cultural and developmental differences among groups of students.

10

• The classroom culture is characterized by high expectations for most students and genuine commitment to the subject by both teacher and students, with students demonstrating pride in their work.

• Little instructional time is lost because of classroom routines and procedures for transitions, handling of supplies, and performance of non-instructional duties, which occur smoothly.

• Standards of conduct appear to be clear to students, and the teacher monitors student behavior against those standards. The candidate’s response to student misbehavior is appropriate and respects the students' dignity.

• The classroom is safe, and learning is accessible to all students; the teacher ensures that the physical arrangement is appropriate to the learning activities. The candidate makes effective use of physical resources, including computer technology.

• Expectations for learning, directions and procedures, and explanations of content are clear to students. Communications are appropriate for students' cultures and levels of development.

• Most of the candidate's questions elicit a thoughtful response, and the teacher allows sufficient time for students to answer. All students participate in the discussion, with the teacher stepping aside when appropriate.

• Activities and assignments, materials, and groupings of students are fully appropriate for the instructional outcomes and students' cultures and levels of understanding. All students are engaged in work of a high level of rigor. The lesson's structure is coherent, with appropriate pace.

• Assessment is regularly used in instruction, through self-assessment by students, monitoring of progress of learning by the candidate and/or students, and high-quality feedback to students. Students are fully aware of the assessment criteria used to evaluate their work.

• promotes the successful learning of all students, making adjustments as needed to instruction plans and accommodating student questions, needs, and interests.

• provides an accurate and objective description of the lesson, citing specific evidence. The candidate makes some specific suggestions as to how the lesson might be improved.

• systems for maintaining both instructional and non-instructional records are accurate, efficient, and effective. • communicates frequently with families and successfully engages them in the instructional program. Information to families about

individual students is conveyed in a culturally appropriate manner. • participates actively in the professional community and in school and district events and projects, and maintains positive and

productive relationships with colleagues. • seeks out opportunities for professional development based on an individual assessment of need and actively shares expertise with

others. The candidate welcomes feedback from supervisors and colleagues. • displays a high level of ethics and professionalism in dealings with both students and colleagues and complies fully and voluntarily

with school and district regulations.

Standard 1.5: Data presented below suggests that we ensure that our candidates model and apply technology standards as they plan, implement, and assess students. Specifically, evidence, disaggregated by specialty area, is obtained from the following assessments and measures: ILab: Lesson & Unit Plans; Teacher Work Samples; Technology Assessment, and TESS field and internship observations S1_1.3_Lesson_Plan S1_1.5__Unit Plan S1_1.7_Technology S1_1.8_Teacher_Work_Sample S1_1.10_TESS_Internship_Observation

11

Analyses of the data/evidence are provided for completers of our initial licensure programs in the 2014-15, 2015-16, and 2016-17 academic years in each of the assessment attachments. Our exiting candidates model and apply technology standards in coursework and clinical experiences. They demonstrate knowledge and skills in accessing databases, digital media, and/or electronic sources as well as the ability to design and facilitate digital learning. They also track and share student performance data digitally. Specifically, review of the Technology assessment data suggests that most of our completers of the initial programs demonstrate an above standard to exemplary knowledge and understanding of the use of technology in their teaching. Review of the lesson plan data suggests that a majority of our completers show mastery in use and selection of appropriate instructional resources and in using evaluation to plan future instruction. Additionally, review of the Unit Plan data suggest that a majority of our initial licensure candidates show mastery in use and selection of a variety of appropriate instructional resources and in using a variety of assessments to evaluation student learning. Review of the Teacher Work Sample data suggests a majority of our completers of the initial programs demonstrate an emerging to comprehensive understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making. The use of technology in teaching students in field and clinical experiences is also included in the AR-TESS observation assessment (Component 2e: Organizing Physical Space; Component 3d: Using Assessment in Instruction; and Component 4b: Maintaining Accurate Records). Review of this data suggests that our initial candidates make effective use of computer technology in their instruction, management, and assessment of students and in monitoring of progress by both students and teacher, and high-quality feedback to students from a variety of sources. Finally, the candidates’ systems for maintaining both instructional and non-instructional records are accurate, efficient, and effective. Use of Assessment Results to Improve Program PROGRAM YEAR CHANGES MADE DUE TO ASSESSMENT

EDUC/SCED MED ENGLISH Fall

2013 (1). Content Knowledge: Because candidates come to the program from accredited baccalaureate degrees in English, as required by the State of Arkansas, and the GPA requirement for entry into the program is 2.75, knowledge level is generally assured. Nevertheless, the English Language Arts advisor notes weaknesses and/or any deficiency beyond those listed on the Pre-Admission analysis and advises candidates to take additional coursework before proceeding with the program. The State Department of Education changed the required tests required English Language Arts in 2011; the results do not coincide with previous scores, and the data for some candidates who took earlier tests is not reflected in the grid. Assessment 2 Thematic Unit Plan has been moved from Assessment 6 to replace the generic Unit Plan completed in the curriculum course. The current Thematic Unit is created during Teaching Adolescent Literature and was specifically designed to align with Standard 3.0, as well as other standards that apply to the organization of knowledge for instruction. The assessment is the major assessment for the course,

12

and candidates share their units with each other via the Blackboard community. The theme must also reflect some element of diversity to strengthen that element and to match the unit’s conceptual framework. Because a course in literary theory is seldom offered at the graduate level at UALR, and many English majors have not had a course in it, the program advisor instituted an elective in Literary Analysis in the Secondary English Classroom. This course fills a void in the program and also supplies additional experience with adolescent and multicultural literature. The other graduate elective that is a requirement for nearly all candidates is Teaching Adolescent Literature. With these additions, passing the state required Praxis II is seldom an issue. Another contributing factor is the transcript analysis which is completed for each candidate at entry level whether he or she has a degree in English or not. Any deficits are noted, and candidates rectify that content element from the standards through additional coursework. (2). Professional and Pedagogical Knowledge, Skill, and Dispositions: All candidates undergo rigorous field experiences while being supervised by adjunct faculty who are former English language arts teachers. The program relies heavily on their expertise. The current ELA advisor and teacher of the Specialized Methods course is a former English teacher, grades 7-12. Thus, modeling is an important part of candidate knowledge bases. Instructional skills are developed in two courses on campus, as well as during 60 hours of clinical opportunities in the schools. The Instructional Skills and Classroom Management course provides opportunities for candidates to micro-teach others who are not in their content area as all majors enroll in this introductory course. The Specialized Methods course includes all English language arts majors, plus those candidates in Speech and Drama when the numbers are too low to provide a separate methods course for them.. Thus, candidates experience peer teaching with those who understand their specific content. The ELA program supervisor has also aligned NCTE standards with a Pre-Professional Evaluation Form used by university supervisors during all field experiences. As supervisors continue to supply data, conclusions about candidate expertise in this area can be developed. The Teacher Education Department also conducts, when needed, a Concerns Conference when candidates are exhibiting unprofessional behavior. Domain D on the state-required Pathwise Pre-Professional Form completed during field experiences addresses this area. The Secondary Education Program Coordinator, the program supervisor, the Clinical Placement Director, and the Associate Dean meet with the candidate. The situation is addressed from both sides, and the candidate receives a written copy of the problem, along with the conference outcomes; he/she must sign this form. The three ELA candidates who have been required to attend a conference were asked to voluntarily leave the program. The reasons for this in two instances resulted from unprofessional conduct during the clinical classroom experience. The third failed to enroll in coursework during his Provisional Licensure period; the State Department of Education requires those who are teaching on a provisional license while completing a graduate program to be continuously enrolled during the completion of the degree. Assessment 4 data utilizing the NCTE University Supervisor Observation Form is obtained during the Internship, but previously candidates are evaluated using that

13

same form during the clinical experience during Specialized Methods. Candidates can make comparisons in growth in this category.. Praxis II also contains a section on Pedagogy where each candidate is assessed about instructional knowledge. Although this test is not aligned with NCTE standards, baseline data is being collected and can be used if this alignment occurs. (3) Student Learning: The Teacher Work Sample has been part of the program for all candidates, both undergraduate and graduate, who are in the initial licensure program. However, the rubrics were generic, applying across programs. Each candidate conducts this assessment research and analysis project during the Internship. The ELA advisor aligned the rubric with NCTE standards in the spring of 2007. As more candidates complete the program and are evaluated with the revised rubric, conclusions can be drawn about the role of assessment and its effect on students.

FOREIGN LANGUAGE

Spring 2014

1. Candidate Content Knowledge Prior to the collection period for this ACTFL/NCATE report, DISLS faculty had already begun the discussion of raising candidates’ oral proficiency. We have begun the process of creating a culture for the OPI by requiring a MOPI of all students enrolled in the skills development classes. Since only a few candidates have sat for the official OPI and few students who have been a part of the MOPI process have reached the teacher education candidate stage, the faculty is not yet sure of the impact of this process. Scores on the official ACTFL OPI will continue to drive curricular reform as we tweak the curriculum to better prepare candidates to pass the OPI at the Advanced-Low level. It is an expectation within DISLS that all courses at the 3000-level and above are taught in the target language. No courses are offered in translation. Study abroad is not required of all students, but all teacher education candidates are encouraged to spend time abroad in one of our university locations. The courses taken by foreign language candidates represent a logical distribution of skill-building courses, culture, linguistics, and literature. The addition of work on the 3Ps paradigm in the culture courses and the requirement of three 3000-level courses on the interpretive, interpersonal, and presentational modes of language not only provides candidates with current content knowledge, but also lets them experience these language models in their own foreign language education. 2. Candidate Professional and Pedagogical Knowledge, Skill, and Dispositions We are fortunate here at UALR to have four foreign language pedagogy courses to help develop the pedagogical knowledge, skills, and dispositions of our candidates. These four courses have been in place for over 10 years. Faculty that are current in the field and are productive scholars and recognized nationally teach all four courses. Foreign language education faculty supervise all pre-professional and internship experiences. Faculty also have input in placement of candidates for these experiences. The faculty believe that the pedagogical background of our candidates is exceptional. 3. Candidate Impact on Student Learning We believe that our Teacher Work Sample presented as evidence here is helping the faculty get a handle on our candidates’ impact on student learning. With a separate

14

second language assessment course, our candidates have many experiences creating assessments and appropriate rubrics. Likewise, assessment theories and selection of assessment tools form part of the course content. As more candidates complete student teaching and undertake completion of the Teacher Work Sample, we will have concrete evidence of candidates’ ability to measure what their students know and are able to do and to report the results to appropriate stakeholders.

MATH Fall 2015

more clearly align indicators with the assessment task, rubric, and data analysis. CONTENT KNOWLEDGE: Candidates completing the graduate secondary mathematics initial licensure program enter the program with a strong content foundation. The program utilizes the mathematics content to build equally strong pedagogical knowledge, skills, and dispositions which provide the Central Arkansas area with mathematics teachers of excellence. As indicated in Assessment 1 the performance of UALR Graduate Mathematics Candidates on the Praxis II Exam have a strong passing rate. The requirement that candidates pass Praxis II before entering their Internship will ensure that all candidates possess an appropriate content knowledge base prior to their field experience. Assessment 2 clearly demonstrates candidates are required to meet strong admission standards through their mathematics content as related to the NCTM Standards and GPA. The program of study will include the History of Mathematics and Methods in Mathematical Technologies. Additionally this program of study has changed to meet the NCTM standards by substituting specific courses that will cover discrete math, proof, and history. The strong admission standards ensure candidates have a basic mathematical foundation to build their pedagogical skills on. Assessment 6 was revised to further develop the candidate’s mathematical content in the NCTM standards. PROFESSIONAL AND PEDAGOGICAL KNOWLEDGE, SKILL, AND DISPOSITIONS: In Assessment 3 candidates are required to demonstrate their professional and pedagogical knowledge, skill and dispositions through a unit development that is scored by a rubric based on Arkansas Teacher Licensure Standards and NCTM Frameworks. All candidates met expectations as specified by the rubrics demonstrating an application of pedagogical skills to the mathematics content. In Assessment 4 candidates demonstrate their pedagogical knowledge through two 30 hour pre-professional field experiences and one 12 week Internship. In the past candidates were assessed through a Pathwise Plus form which meets Arkansas State Teacher Licensure Standards now the state is transitioning to TESS. The program has revised NCTM Mathematics Observation Guide based on applicable NCTM standards. All candidates met expectations for successful content instructional planning and delivery, classroom management and professionalism. All of our university supervisors are retired certified mathematics teachers with records of excellence in teaching mathematics and have been trained in Pathwise Plus and will be trained on TESS providing content knowledge for the mathematics NCTM assessment and pedagogy for the knowledge, skills and dispositions needed to be a teacher of quality. STUDENT LEARNING

15

In Assessment 5 the data for the NCTM Effect on Student Learning indicate that candidates have a positive effect on 7th-12th grade students in relation to NCTM standards. The data demonstrate the mathematics candidates are encouraging all students including diverse populations, those with special needs, and English as second language learners to learn with mean ratings that meet or exceed expectations.

SCIENCE Fall 2013

(1) Content knowledge: Our program allows candidates to acquire a master’s degree along with initial licensure. Because of this we require strong content as a foundation for admission to the program. Previous to 2007 candidates were allowed admission with 18 hours of prerequisite courses to complete. Starting in the fall of 2007 candidates are required to lack no more than 12 hours of prerequisite courses. Strengthening a candidate’s knowledge base insures the candidate will pass all Praxis II Life/Earth/Physical Content. We made this decision upon analysis of Praxis II data when it was noted several candidates were able to pass the Praxis life or physical science content first time, yet others took two or three times to pass the Praxis II Earth Science Content. Candidates’ lack of earth science hours was evident in their failure to pass Earth Science Praxis. The Department of Education has now changed the requirements for licensure to just including Life Science and Physical Science. Earth Science is no longer required for licensure. Earth Science is no longer required for licensure. This change occurred in January, 2013. This will change our emphasis in the program of study in the future. The Earth Science majors will have the opportunity to choose whether they want to be licensed in Life or Physical Science. (2) Professional and pedagogical knowledge, skill, and dispositions: We have had 100% pass rate on the first time candidates taking the Life or Physical Pedagogy. Arkansas no longer has the Praxis III administration, but our graduates have always scored at the proficient or above level. Our general instructional methods and classroom management, specialized science methods course and curriculum design all require a field co-requisite that allows candidates to put into practice the theory learned in the courses. The courses were evaluated through Pathwise which is similar to and a foundation for the Praxis III. Candidates were required to demonstrate content, planning, instruction, human relations, professionalism, research & technology, professional ethics and diversity as align with Arkansas Teacher Licensure Standards and the NSTA applicable standards in their exit portfolio. However, Praxis III evaluations are no longer used in Arkansas. In the future a program called Teacher Evaluation Support System (TESS) will used to evaluate our candidates. This data will not be shared with the universities, so evaluation performance the first year in the classroom will not be possible unless a data sharing system is developed at the Department of Education. (3) Student learning: Candidates are required to submit a student work sample during their internship that contains a pre and post test of content taught, an analysis of the test as demonstrated in a graph, a reflection as to the effectiveness of a lesson and a plan to improve how the lesson was taught such as adding or deleting specific instructions or content. Beginning spring 2008 and continuing through Spring, 2011, data was collected on Praxis III for our graduates. We collected and analyzed data on our candidates. No student scored below the required score for licensure. Praxis III utilized trained assessors who have passed the ETS test battery, experienced teachers, administrators, and teacher educators who complete a one-time evaluation which includes pre-interview, classroom observation, and post-interview to determine whether the novice teacher should continue in the classroom without remediation. It is now the goal of the Department of Education

16

and the schools in Arkansas to train all teachers in TESS. Extensive professional development has been conducted to train administrators, faculty, and university faculty in the TESS system. Teachers will be evaluated using this system in all schools in the state beginning in the Fall, 2013. The Dean’s of the state universities are working with the Department of Education to be able to code the graduates of each university so data can be acquired from graduated for evaluation purposes. The system has not been approved as of September 1, 2013, but more work will be done in the future to code graduates and match them to each university. Assessments listed in Section II have not been changed for the three years of reported data. The University is in the process of making the transition to the 2012 NSTA Standards and will rewrite assessments and rubrics to meet the new standards. The current assessment were still able to assess the learning of the candidates in the Secondary Graduate Science Program.

Social Studies Spring 2015

1. Evidence must be presented in this section that assessment results have been analyzed and have been or will be used to improve candidate performance and strengthen the program. This description should not link improvements to individual assessments but, rather, it should summarize principal findings from the evidence, the faculty's interpretation of those findings, and changes made in (or planned for) the program as a result. Describe the steps program faculty has taken to use information from assessments for improvement of both candidate performance and the program, This information should be organized around (1) content knowledge, (2) professional and pedagogical knowledge, skill, and dispositions, and (3) student learning. In addition, for each assessment listed in Section II, describe why or why not the assessment has been changed since the program was submitted previously. (Response limited to 24,000 characters) Table: Transcript Analysis: Content at Admission, Remediation, and at Graduation (attached) has been added to provide deeper meaning to Assessment 2 as part of the Transcript Analysis. This is a visual of evidence identifying candidate deficiencies entering the program as determined by the Transcript Analysis. This table shows the content hours completed by each candidate as they enter the program, the number of hours deficient by content area, and the total number of content hours completed per area upon Graduation. This table reflects candidates for 2010-2011 and 2010-2012 (two administrations). This table will be included in future NCSS reports. Following feedback from the 2014 report, the Prerequisite Evaluation Form (Appendix “D” page 46) has been revised to the Transcript Analysis Form (Appendix “D” page 47) and the following course align is now being administered to be more in alignment with NCSS expectations for all candidates. The Transcript Analysis Form will require 12 hours of US History, 9 hours of World History, 6 hours of Political Science, 6 hours of Geography, 6 hours of Economics, 3 hours of either Anthropology or Cultural/Human Geography (in addition to the 6 hours of geography), and 3 hours of either Sociology or Psychology. Following feedback from the 2014 report, the program has added content course requirement to the Transcript Analysis to strengthen NCSS Standard 1.8. All candidates will be required to complete 3 hours in one of two classes: • HIST 4301 History of Technology - A survey of the role of technology from the Stone Age to the nuclear age.

17

• HIST 4302 Magic, Science, and the Occult from Antiquity to Newton - A survey of humans’ attempts to explain and control the cosmos from antiquity to the emergence of modern science around 1700, including the contributions of pseudo-scientific, occult, and magical world-views; internal developments in the history of science; and the relationship between scientific thought and the historical context. Dual-listed in the UALR Graduate Catalog as HIST 5302.

Stakeholder Involvement/Communication Plan The SOE Assessment Timeline provides a process by which stakeholders are involved in analyzing and interpreting data as well as by offering suggestions for improvement (as members of advisory committees, at stakeholder meetings, and by publicly reviewing and commenting on the annual program reports online). Across the 2016-17 AY, the EDUC MED faculty communicated with and shared the above data and findings with our candidates’ cooperating teachers and with cooperating teachers at the bi-annual Cooperating Teachers Workshop. In both fall 2017 and spring 2018 workshops, we had a chance to discuss the findings with CT’s, who provided suggestions and feedback. Due to the CAEP accreditation visit in October 2017, we completed much of the report above in spring and fall 2017 and the report was shared with stakeholders in our advisory committees and it was also reviewed by the CAEP visiting team in offsite and onsite visits. This report will also be posted online in May 2018 for public review at http://ualr.edu/cehp/assessment/ .

18

S1_1.3_ Lesson Plan Lesson Plan Although the various lesson plan assignments may vary across licensure areas, they all involve the planning, teaching, and evaluating of a lesson in a field site. The lesson plan is an assignment typically administered at the intermediate level in field experiences or at exit in internship. Below is the common iLab scoring rubric used across licensure areas by all faculty to score the assignment and the aggregated data by program. The data tables follow and provide data by program for those EDUC MED program completers who graduated in the 2014-15, 2015-16, and 2016-17 academic years. Review of these tables suggests that a majority of our completers of the EDUC MED initial programs demonstrate a comprehensive understanding of the use of relevant aspects of student backgrounds in planning instruction; and, candidates clearly articulate instructional goals based on this analysis. Additionally, candidates show mastery in use and selection of appropriate instructional resources and in using evaluation to plan future instruction. It is important to note that our candidates submit a number of lesson plans for assessment across their programs. But, the iLab Lesson Plan is typically assessed from their electronic exit portfolio and it allows them to submit their best lesson plans for review. In terms of the validity of the measure, it is important to note that the iLab assessment of performance using the common rubric below is based on an assessment of candidate performance in a key assessment in the field site from each initial program that has been vetted and approved by specific professional associations and the Arkansas Department of Education. Further, the program assessments are also aligned with INTASC, the specific Arkansas Curriculum Frameworks (most based on Common Core), and the Arkansas Teaching Standards (that use INTASC Standards and extend them). These assessments were originally developed by faculty and subsequently revised by a collection of experts such as EPP faculty, P-12 teachers, P-12 administrators, and EPP alumni. Further, this iLab lesson plan assessment has been in place for several years and was evaluated by the previous NCATE Board of Examiners. Since the last NCATE review, the iLab rubric has been realigned with the new CAEP standards and INTASC and we created five separate levels of the rubric from that of a single rubric, thus allowing for five separate scores for the five dimensions of lesson planning in the current iLab rubric (knowledge of students, instructional goals, instructional resources, instructional design, and assessment). In terms of reliability, it is important to note that most of the faculty who are assessing our candidates on the measure have been using this key assessment, both at the program level and at the EPP level for iLab for several years. Validity & Reliability: Validity studies will be performed by an expert panel in the Spring 2018 semester. Please see Standard 5.1 and 5.2 for a description of the future extensive analyses of the validity of this measure that will be undertaken in spring 2018. However, in the spring of 2017 there was an investigation of the inter-rater reliability of the lesson plan assessment. Although there was 84% inter-rater agreement on the 5 categories, the differences appear to be significant, suggesting a need to review and refine the

19

assessment and scoring rubric. An initial analysis of the rubric was performed by the EPPAC in late spring, suggesting a need for a more detailed scoring rubric with more precise levels of performance which is offered below. This new scoring rubric will be implemented in the Fall 2017 semester and inter-rater reliability will be checked again in spring 2018 along with the expert validity studies.

Results from Interrater Reliability Analyses

Agreement

Ratings Differed by 1 Point

Ratings Differed by 2 Points

Cohen’s kappa

Lesson Plan 5 Categories

21/25 = 84% 4/25 = 16% 0/25 = 0% K = 2.61 p = .009*

Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees reviewed the 3 years of data and decided upon benchmarks for candidate performance in subsequent years of data. In Fall 2018, faculty will compare the findings from 2017-18 to the approved benchmark. The benchmark for this assessment, based on initial review of overall means for the initial licensure programs, is a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that did not meet the benchmark will need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual action plans and present this data in an aggregated form in the annual assessment report. The SOE Assessment Coordinator will aggregate these assessment analyses and action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review. Lesson Plan Scoring Rubric Knowledge of Students INTASC St 1, 2, 3

Demonstrates minimal or no understanding and application of knowledge of student background and prior

Demonstrates limited understanding and application of knowledge of student background

Demonstrates emerging understanding and application of knowledge of student background

Demonstrates comprehensive understanding and application of knowledge of

20

knowledge in planning instruction.

and prior knowledge in planning instruction.

and prior knowledge in planning instruction.

student background and prior knowledge in planning instruction.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Instructional Goals INTASC St 7, 8

Demonstrates minimal or no understanding and application of knowledge of instructional goals in planning instruction.

Demonstrates limited understanding and application of knowledge of instructional goals in planning instruction.

Demonstrates emerging understanding and application of knowledge of instructional goals in planning instruction.

Demonstrates comprehensive understanding and application of knowledge of instructional goals in planning instruction.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Instructional Resources INTASC St 4, 5, 8

Demonstrates minimal or no understanding of selecting instructional resources in planning instruction.

Demonstrates limited understanding of selecting instructional resources in planning instruction.

Demonstrates an emerging understanding of selecting instructional resources in planning instruction.

Demonstrates comprehensive understanding of selecting instructional resources in planning instruction.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Instructional Design INTASC St 4, 5, 8

Demonstrates minimal or no understanding of lesson plan sequence and design.

Demonstrates limited understanding of lesson plan sequence and design.

Demonstrates an emerging knowledge and understanding of lesson plan sequence and design.

Demonstrates comprehensive understanding of lesson plan sequence and design.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Assessment INTASC St 6

Demonstrates minimal or no understanding of assessment and evaluation in planning instruction.

Demonstrates limited understanding of assessment and evaluation in planning instruction.

Demonstrates an emerging knowledge and understanding of assessment and evaluation in planning instruction.

Demonstrates comprehensive understanding of assessment and evaluation in planning instruction.

New Lesson Plan Scoring Rubric to be implemented starting in Fall 2017

21

2.1 Lesson Plan TCED G 17 Standard 1: CONTENT AND PEDAGOGICAL KNOWLEDGE The provider ensures that candidates develop a deep understanding of the critical concepts and principles of their discipline and, by completion, are able to use discipline-specific practices flexibly to advance the learning of all students toward attainment of college- and career-readiness standards. The assessment is used as evidence for the following INTASC Standards for Candidate Knowledge, Skills, and Professional Dispositions: Standard #1: Learner Development. The teacher understands how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences. Standard #2: Learning Differences. The teacher uses understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. Standard #3: Learning Environments. The teacher works with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation. Standard # 4 The teacher understands the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches and creates learning experiences that make these aspects of the discipline accessible and meaningful for learners to assure mastery of the content. Standard #5: Application of Content The teacher understands how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues. Standard #6: Assessment The teacher understands and uses multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher’s and learner’s decision making. Standard #7: Planning for Instruction The teacher plans instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context. Standard #8: Instructional Strategies The teacher understands and uses a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways.

1.0 Standard 1 Learner Development

1.0 Unsatisfactory Demonstrates minimal or no understanding and application of understanding of how learners grow and develop.

2.0 Basic Demonstrates limited understanding and application of knowledge understanding about how learners grow and develop.

3.0 Proficient Demonstrates emerging understanding and application of knowledge about how learners grow and develop.

4.0 Distinguished Demonstrates comprehensive understanding and application of knowledge about how learns grow and development.

1.0 Standard 2 Learning Differences

1.0 Unsatisfactory Demonstrates minimal or no understanding of individual differences and diverse cultures and communities.

2.0 Basic Demonstrates limited understanding and application of knowledge of individual differences and diverse cultures and communities.

3.0 Proficient Demonstrates emerging understanding of individual differences and diverse cultures and communities.

4.0 Distinguished Demonstrates comprehensive understanding of individual differences and diverse cultures and communities.

1.0 Standard 3 Learning Environments

1.0 Unsatisfactory Demonstrates minimal or no ability to work with others to

2.0 Basic Demonstrates limited ability to work with others

3.0 Proficient Demonstrates an emerging understanding of working with

4.0 Distinguished Demonstrates a comprehensive understanding

22

create environments that support individual and collaborative learning. and that encourage positive social interaction, active engagement in learning, and self motivation.

to create environments that support individual and collaborative learning. and that encourage positive social interaction, active engagement in learning, and self motivation.

others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation

of working with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation

1.0 Standard 4 Content Knowledge

1.0 Unsatisfactory Demonstrates minimal or no understanding and application of knowledge of student background and prior knowledge in planning instruction.

2.0 Basic Demonstrates limited understanding of the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches.

3.0 Proficient Demonstrates emerging understanding of the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches.

4.0 Distinguished Demonstrates limited understanding of the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches

1.0 Standard 5 Application of Content

1.0 Unsatisfactory Demonstrates minimal or no understanding how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issuesSaveli

2.0 Basic Demonstrates lmited understanding how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues

3.0 Proficient Demonstrates an emerging understanding how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues

4.0 Distinguished Demonstrates comprehensive understanding how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues

1.0 Standard 6 Assessment 1.0 Unsatisfactory Demonstrates minimal or no understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making

2.0 Basic Demonstrates limited understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making

3.0 Proficient Demonstrates emerging understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making

4.0 Distinguished Demonstrates comprehensive understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making

23

1.0 Standard 7 Planning for Instruction

1.0 Unsatisfactory Demonstrates minimal or no ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context

2.0 Basic Demonstrates limited ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context

3.0 Proficient Demonstrates emerging ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context

4.0 Distinguished Demonstrates comprehensive ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context

1.0 Standard 8 Instructional Strategies

1.0 Unsatisfactory Demonstrates minimal or no understanding of and the ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in mearningful ways

2.0 Basic Demonstrates limited understanding of and the ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways

3.0 Proficient Demonstrates emerging understanding of and the ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways

4.0 Distinguished Demonstrates comprehensive understanding of and the ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways

24

Table S1_1.3_ Lesson_Plan_2014-2017 EDUC MED Completers

Lesson Plan

Initial Licensure Program Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

EDUC MED:

2016-17 MED

Knowledge of Students 100% 14 4.0 0.00

Instructional Goals 100% 14 4.0 0.00

Instructional Resources 100% 14 4.0 0.00

Instructional Design 100% 14 4.0 0.00

Assessment 100% 14 4.0 0.00

2015-16 MED

Knowledge of Students 100% 16 4.0 0.00

Instructional Goals 100% 16 4.0 0.00

Instructional Resources 100% 16 4.0 0.00

Instructional Design 100% 16 4.0 0.00

Assessment 100% 16 4.0 0.00

2014-15 MED

Knowledge of Students 14% 86% 35 3.86 0.36

Instructional Goals 14% 86% 35 3.86 0.36

Instructional Resources 14% 86% 35 3.86 0.36

Instructional Design 14% 86% 35 3.86 0.36

Assessment 14% 86% 35 3.86 0.36

25

S1_1.4_Case Study Although the various case study assignments vary across licensure areas, they all involve the planning, administration, and analysis of a relevant case, and the subjective interpretation of data of a case (an individual student or group/class of students) in an applicable educational setting. The case study assignment is administered at the intermediate level for some programs and at the exit level for others. Below is the common iLab scoring rubric used across the licensure graduate programs, by all faculty to score the assignment. The aggregated data across licensure programs follows. The case study involves focusing on an actual student or students with special needs (academic and/or behavioral issues) in a field site. Review of the table below for the EDUC initial candidates suggests a majority of our completers demonstrate a proficient skill in identifying a central issue and developing an action plan for successfully dealing with the issue using specific evidence throughout this process to support evaluations and suggested actions and plans. Further, the candidates do so by demonstrating an understanding of the roles of learner development and differences and the need to collaborate with others in successfully implementing the action plan. Analyses of the data also suggested a possible tendency for scores to be lower for the Learning Environments level of the rubric. This level of the assessment included the ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation. We plan on exploring this potential issue further in subsequent semesters as we gather more data from other candidates. We note that the skills of collaborating with others will be identified as a potential issue in the AR-TESS Internship assessment. Further, a discussion of faculty about the data led to informal observations of several candidates having narrow understandings of student motivation. While we feel that these are potential issues to explore, we believe that we need more data to support these potential preliminary and informal observations. In terms of the validity of the measure, it is important to note that the iLab assessment of performance using the common rubric below is based on an assessment of candidate performance in a key assessment in the field site from most of the initial programs that has been vetted and approved by specific professional associations and the Arkansas Department of Education. Further, the program assessments are also aligned with INTASC, the specific Arkansas Curriculum Frameworks (most based on Common Core), and the Arkansas Teaching Standards (that use INTASC Standards and extend them). These assessments were originally developed by faculty and subsequently revised by a collection of experts such as EPP faculty, P-12 teachers, P-12 administrators, and EPP alumni. Further, this iLab case study assessment has been in place for several years and was evaluated by the previous NCATE Board of Examiners. Since the last NCATE review, the iLab rubric has been realigned with the new CAEP standards and INTASC. Further, we created three separate levels of the rubric from that of a single rubric, thus allowing for three separate scores for the three dimensions of lesson planning in the current iLab rubric (focus/organization learner development, learner differences, and learning environments). In terms

26

of reliability, it is important to note that most of the faculty who are assessing our candidates on the measure have been using this key assessment, both at the program level and at the EPP level for iLab for several years. Validity and Reliability Studies: However, what we have not done is any formal validity and reliability studies of the common iLab measure. As described in this self-study in Standard 5.2, we have now developed a team of experts to undertake such analyses. Validity studies will be performed by an expert panel in the Spring 2018 semester. Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees reviewed the 3 years of data and decided upon benchmarks for candidate performance in subsequent years of data. In Fall 2018, faculty will compare the findings from 2017-18 to the approved benchmark. The benchmark for this assessment, based on initial review of overall means for the initial licensure programs, is a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that do not meet the benchmark in Fall 2018 would need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual action plans and present this data in an aggregated form in the annual assessment report. The SOE Assessment Coordinator will aggregate these assessment analyses and action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review. 2.2 Case Study iLab The assessment meets part of the requirements for CAEP Standard 1: Standard 1: CONTENT AND PEDAGOGICAL KNOWLEDGE The provider ensures that candidates develop a deep understanding of the critical concepts and principles of their discipline and, by completion, are able to use discipline-specific practices flexibly to advance the learning of all students toward attainment of college- and career-readiness standards. The assessment is used as evidence for the following INTASC Standards Candidate Knowledge, Skills, and Professional Dispositions: Standard #1: Learner Development. The teacher understands how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences. Standard #2: Learning Differences. The teacher uses understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. Standard #3: Learning Environments. The teacher works with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation.

27

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Focus/Organization Learner Development INTASC St 1

Demonstrates minimal or no understanding of how learners grow and develop, or recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social emotional, and physical areas, and designs and implements developments appropriate and challenging learning experiences.

Demonstrates limited understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social emotional, and physical areas, and designs and implements developments appropriate and challenging learning experiences.

Demonstrates an emerging understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social emotional, and physical areas, and designs and implements developments appropriate and challenging learning experiences.

Demonstrates a comprehensive understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social emotional, and physical areas, and designs and implements developments appropriate and challenging learning experiences.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Learning Differences INTASC St 2

Demonstrates minimal or no understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet highs standards.

Demonstrates limited understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet highs standards.

Demonstrates emerging understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet highs standards.

Demonstrates comprehensive understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet highs standards.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

1.0 Learning Environments INTASC St 3

Demonstrates minimal or no ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active

Demonstrates limited ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social

Demonstrates emerging ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social

Demonstrates comprehensive ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in

28

engagement in learning, and self-motivation.

interaction, active engagement in learning, and self-motivation.

interaction, active engagement in learning, and self-motivation.

learning, and self-motivation.

Table S1_1.4_Case_Study

Case Study EDUC MED Degree Unsatisfactory

1 Basic

2 Proficient

3 Distinguished

4 N Mean SD

2016-17 MED

Focus/Organization: Learner Development 3 13 16 3.81 0.40

Learning Differences 4 12 16 3.75 0.44

Learning Environment 1 5 10 16 3.56 0.63

2015-16 MED

Focus/Organization: Learner Development 3 9 12 3.75 0.45

Learning Differences 4 8 12 3.67 0.49

Learning Environment 4 3 5 12 3.08 0.90

2014-15 MED

Focus/Organization: Learner Development 5 20 25 3.80 0.41

Learning Differences 6 19 25 3.76 0.44

Learning Environment 3 11 11 25 3.36 0.70

29

S1_1.5_ Unit Plan Unit Plan

Although the unit plan assignments may vary across licensure areas, they all involve the planning, teaching, and assessment of a unit in an applicable educational setting, typically their classroom where they are a teacher of record or their internship site. The unit plan is an assignment administered at the exit level during teaching or internship. Below is the scoring rubric for the graduate initial licensure programs that are used by all faculty to score the assignment and the aggregated data by program and by degree level. Our graduate candidates have more control over the planning and implementation of these units because almost all of them are teachers of record in their own classroom. Review of these tables suggests that a majority of our completers of the EDUC MED initial programs demonstrate an emerging to comprehensive understanding of the use of relevant aspects of student backgrounds, student diversity, contextual variables and content knowledge in planning instruction across a unit; and, candidates clearly articulate instructional goals based on this analysis. Additionally, graduate initial licensure candidates show mastery in use and selection of a variety of appropriate instructional resources and in using a variety of assessments to evaluation student learning. The graduate initial candidates also demonstrate emerging to comprehensive understandings of how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues. In terms of the validity of the measure, it is important to note that the iLab assessment of performance using the common rubric below is based on an assessment of candidate performance in a key assessment in the field site from several, but not all, of the initial programs that have been vetted and approved by specific professional associations and the Arkansas Department of Education. These assessments were originally developed by faculty and subsequently revised by a collection of experts such as EPP faculty, P-12 teachers, P-12 administrators, and EPP alumni. This iLab unit plan assessment has been in place for several years and was evaluated by the previous NCATE Board of Examiners. Further, the program assessments are also aligned with INTASC, the specific Arkansas Curriculum Frameworks (most based on Common Core), and the Arkansas Teaching Standards (that use INTASC Standards and extend them). Since the last NCATE review, the iLab rubric has been realigned with the new CAEP standards and INTASC and we created four separate levels of the rubric from that of a single rubric for the undergraduate unit plan, thus allowing for four separate scores for the four dimensions of unit planning in the current iLab rubric (learner development, learner differences, learning environments, and content knowledge). The current graduate unit plan scoring rubric includes these four dimensions and adds four more (application of content, assessment, lesson planning, and instructional strategies). In terms of reliability, it is important to note that most of the faculty who are assessing our candidates on the measure have been using this key assessment, both at the program level and at the EPP level for iLab for several years.

30

Validity & Reliability: Validity studies will be performed by an expert panel in the Spring 2018 semester. However, in the spring of 2017 there was an investigation of the inter-rater reliability of the unit plan assessment. Although there was only 68% inter-rater agreement on the 5 categories, and the differences were not significant, we were concerned about the relatively low agreement and decided to review and refine the assessment and scoring rubric for the candidates. Inter-rater reliability will be checked again in spring 2018 along with the expert validity studies.

Results from Interrater Reliability Analyses

Agreement

Ratings Differed by 1 Point

Ratings Differed by 2 Points

Cohen’s kappa

Unit Plan 5 Categories

27/40 = 68% 11/40 = 27% 2/40 = 5% K = 0.66 p = .513

Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees reviewed the 3 years of data and decided upon benchmarks for candidate performance in subsequent years of data. In Fall 2018, faculty will compare the findings from 2017-18 to the approved benchmark. The benchmark for this assessment, based on initial review of overall means for the initial licensure programs, is a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that do not meet the benchmark in the 2018 analyses will need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual action plans and present this data in an aggregated form in the annual assessment report. The SOE Assessment Coordinatorwill aggregate these assessment analyses and action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review.

3.2 Unit Plan TCED iLab G (2) Standard 1: CONTENT AND PEDAGOGICAL KNOWLEDGE The provider ensures that candidates develop a deep understanding of the critical concepts and principles of their discipline and, by completion, are able to use discipline-specific practices flexibly to advance the learning of all students toward attainment of college- and career-readiness standards. The assessment is used as evidence for the following INTASC Standards for Candidate Knowledge, Skills, and

31

Professional Dispositions: Standard #1: Learner Development. The teacher understands how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences. Standard #2: Learning Differences. The teacher uses understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. Standard #3: Learning Environments. The teacher works with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation. Standard # 4 The teacher understands the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches and creates learning experiences that make these aspects of the discipline accessible and meaningful for learners to assure mastery of the content. Standard #5: Application of Content The teacher understands how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues. Standard #6: Assessment The teacher understands and uses multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher’s and learner’s decision making. Standard #7: Planning for Instruction The teacher plans instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context. Standard #8: Instructional Strategies The teacher understands and uses a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways. 1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Learner Development INTASC St 1

Demonstrates minimal or no understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences.

Demonstrates limited understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences.

Demonstrates an emerging understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences.

Demonstrates a comprehensive understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Learning Differences INTASC St 2

Demonstrates minimal or no understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable

Demonstrates limited understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable

Demonstrates an emerging understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable

Demonstrates a comprehensive understanding of individual differences and diverse cultures and communities to ensure inclusive learning

32

each learner to meet high standards.

each learner to meet high standards.

each learner to meet high standards.

environments that enable each learner to meet high standards.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Learning Environment INTASC St 3

Demonstrates minimal or no ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation

Demonstrates limited ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation

Demonstrates emerging ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation

Demonstrates comprehensive ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Content Knowledge INTASC St 4

Demonstrates minimal or no understanding of the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches and creates learning experiences that make these aspects of the discipline accessible and meaningful for learners to assure mastery of the content.

Demonstrates limited understanding of the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches and creates learning experiences that make these aspects of the discipline accessible and meaningful for learners to assure mastery of the content.

Demonstrates an emerging understanding of the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches and creates learning experiences that make these aspects of the discipline accessible and meaningful for learners to assure mastery of the content.

Demonstrates a comprehensive understanding of the central concepts, tools of inquiry, and structures of the discipline(s) he or she teaches and creates learning experiences that make these aspects of the discipline accessible and meaningful for learners to assure mastery of the content.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Application of Content INTASC St 5

Demonstrates minimal or no understanding of how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues.

Demonstrates limited understanding of how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to authentic local and global issues.

Demonstrates emerging understanding of how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related

Demonstrates comprehensive understanding of how to connect concepts and use differing perspectives to engage learners in critical thinking, creativity, and collaborative problem solving related to

33

to authentic local and global issues.

authentic local and global issues.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Assessment INTASC St 6

Demonstrates minimal or no ability to understand and use multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

Demonstrates limited ability to understand and use multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

Demonstrates emerging ability to understand and use multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

Demonstrates comprehensive ability to understand and use multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Planning for Instruction INTASC St 7

Demonstrates minimal or no ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

Demonstrates limited ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

Demonstrates emerging ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

Demonstrates coprehensive ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Instructional Strategies INTASC St 8

Demonstrates minimal or no understanding of and ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build

Demonstrates limited understanding of and ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways.

Demonstrates emerging understanding of and ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build

Demonstrates comprehensive understanding of and ability to use a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build

34

skills to apply knowledge in meaningful ways.

skills to apply knowledge in meaningful ways.

skills to apply knowledge in meaningful ways.

Unit Plan Initial Licensure Program Degree Unsatisfactory

1 Basic

2 Proficient

3 Distinguished

4 N Mean SD

EDUC MED:

2016-17 MED

Learner Development 10 10 4.0 0.00

Learning Differences 10 10 4.0 0.00

Learning Environment 10 10 4.0 0.00

Content Knowledge 10 10 4.0 0.00

Application of Content 10 10 4.0 0.00

Assessment 10 10 4.0 0.00

Planning for Instruction 10 10 4.0 0.00

Instructional Strategies 10 10 4.0 0.00

2015-16 MED

Learner Development 7 4 11 3.36 0.50

Learning Differences 6 5 11 3.45 0.52

Learning Environment 6 5 11 3.45 0.52

Content Knowledge 6 5 11 3.45 0.52

Application of Content 1 6 4 11 3.55 1.04

Assessment 1 6 4 11 3.55 1.04

Planning for Instruction 7 4 11 3.36 0.50

Instructional Strategies 1 6 4 11 3.55 1.04

2014-15 MED

35

Initial Licensure Program Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

Learner Development 1 5 18 24 3.71 0.55

Learning Differences 1 4 19 24 3.75 0.53

Learning Environment 6 18 24 3.75 0.44

Content Knowledge 6 18 24 3.75 0.44

Application of Content 3 3 18 24 3.62 0.71

Assessment 2 5 17 24 3.63 0.65

Planning for Instruction 6 18 24 3.75 0.44

Instructional Strategies 2 4 18 24 3.67 0.64

36

S1_1.6_Dispositions Although dispositions are assessed several times across the programs, this specific iLab Dispositions assessment is administered at the exit level in the final teaching or internship experience and it is part of the AR-TESS observation instrument. Specifically, candidate dispositions are assessed based on their observed performance by university supervisors in the teaching or internship site using the following subdomains from the AR-TESS observation instrument: Component 2a: Creating an Environment of Respect and Rapport; Component 2b: Establishing a Culture for Learning ; Component 3e: Demonstrating Flexibility and Responsiveness ; Component 4c: Communicating with Families ; Component 4f: Showing Professionalism. Below is the common iLab scoring rubric for the Dispositions assessment extracted from the larger AR-TESS instrument. The data tables follow and provide data by program for those program completers who graduated in the 2014-15, 2015-16, and 2016-17 academic years. Review of these tables suggests that a majority of our completers of the EDUC MED initial programs created a positive environment of respect and their interactions with students, and the interactions of their students with each other, are polite and respectful, reflecting general warmth and caring, and are appropriate to the cultural and developmental differences among groups of students. They also created a classroom culture characterized by high expectations for most, if not all, students and genuine commitment to the subject by both teacher and students, with students demonstrating pride in their work. The completers of the initial licensure programs also promoted the successful learning of all students, adjusting as needed to instruction plans, and accommodating student questions, needs, and interests. Most completers also communicated frequently with families and successfully engaged them in the instructional program. Information to families about individual students was conveyed in a culturally appropriate manner. Finally, our candidates exhibited a high level of ethics and professionalism in dealings with both students and colleagues and complied fully and voluntarily with school and district regulations. However, a closer review of the data suggests the need to further investigate the relatively lower scores for a relatively substantial minority of our candidates in communicating with families. Specifically, a relatively substantial minority of our candidates appeared to display a basic understanding of communicating with families. While it appears that they were adhering to school procedures for communicating with families, they may have made only modest attempts to engage families in the instructional program. Further, some of the communications may not have always been appropriate to the cultures of those families. In discussions among faculty when analyzing this data, several possible reasons for these findings emerged, but the one most mentioned was that some of our candidates do simply not get the opportunity to communicate with parents and other family members. The rubric allows for observers to mark levels with a “not applicable” or “not observed,” which is one reason for varying numbers of scores in the data below. It could be that some raters gave lower ratings on this level when it was not observed and should have marked that level as “na.”. Other informal observations during the discussions when analyzing the data did suggest that some candidates may have scored lower in their observations based on their performance. Some faculty discussed a general informal observation that some candidates need more

37

knowledge and skill in collaborating and communicating with adults in the educational setting, including in communicating with parents. One faculty member who teaches several education and special education courses has added class sessions, curriculum, and weblinks in Blackboard on creating positive relationships with parents, especially difficult parents, because of his informal observations. In terms of the validity of the measure, it is important to note that the assessment rubric is taken from the AR-TESS observation instrument used in the state of Arkansas to evaluate the performance of teachers in schools (http://www.arkansased.gov/divisions/human-resources-educator-effectiveness-and-licensure/office-of-educator-effectiveness/teacher-evaluation-system ). This system is based on Danielson’s framework for teaching (https://www.danielsongroup.org/framework/ ), a system that many of EPP faculty have used for several years because our candidates used to have to successfully pass the Praxis 3- Pathwise assessment for licensure, which was also based on Danielson’s framework. Our candidates are assessed on the AR-TESS rubric across multiple field and clinical experiences by university supervisors. Most graduate candidates are not observed by Cooperating Teachers because they are Teachers of Record, teaching their own class on a provisional licensure. Faculty have been trained on the AR-TESS and we have also calibrated our ratings at least two times across the last few years. All Cooperating Teachers must have received ADE training on the AR-TESS as well. Validity and Reliability Studies: Because we have taken these rubrics directly from an external assessment, we do not plan on performing validity studies on this instrument. We do plan to do inter-rater reliability studies of the TESS Observation instrument in 2018. Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees reviewed the 3 years of data and decided upon benchmarks for candidate performance in subsequent years of data. In Fall 2018, faculty will assess the data from the 2017-18 AY related to the new benchmarks. The benchmark for this assessment, based on initial review of overall means for the initial licensure programs, is a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that do not meet the benchmark will need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual action plans and present this data in an aggregated form in the annual assessment report. The SOE Assessment Coordinator will aggregate these assessment analyses and action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review.

38

2.4 Dispositions TCED iLab UG & Graduate The assessment meets part of the requirements for CAEP Standard 1: Standard 1: CONTENT AND PEDAGOGICAL KNOWLEDGE The provider ensures that candidates develop a deep understanding of the critical concepts and principles of their discipline and, by completion, use discipline-specific practices flexibly to advance the learning of all students toward attainment of college- and career-readiness standards. The assessment is used as evidence for the following INTASC Standards Candidate Knowledge, Skills, and Professional Dispositions: Standard #1: Learner Development. The teacher understands how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences. Standard #2: Learning Differences. The teacher uses understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. Standard #3: Learning Environments. The teacher works with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation.

DOMAIN 2: THE CLASSROOM

ENVIRONMENT

UNSATISFACTORY 1

BASIC 2

PROFICIENT 3

DISTINGUISHED 4

Component 2a: Creating an Environment of Respect and Rapport INTASC St 2, 3

Classroom interactions, both between the teacher and students and among students, are negative, inappropriate, or insensitive to students' cultural backgrounds and are characterized by sarcasm, put-downs, or conflict.

Classroom interactions, both between the teacher and students and among students, are generally appropriate and free from conflict, but may be characterized by occasional displays of insensitivity or lack of responsiveness to cultural or developmental differences among students.

Classroom interactions between the teacher and students and among students are polite and respectful, reflecting general warmth and caring, and are appropriate to the cultural and developmental differences among groups of students.

Classroom interactions between the teacher and individual students are highly respectful, reflecting genuine warmth and caring and sensitivity to students' cultures and levels of development. Students themselves ensure high levels of civility among members of the class.

Component 2b: Establishing a Culture for Learning INTASC St 3

The classroom environment conveys a negative culture for learning, characterized by low teacher commitment to the subject, low expectations for student achievement, and little or no student pride in work.

The teacher's attempt to create a culture for learning is partially successful, with little teacher commitment to the subject, modest expectations for student achievement, and little student pride in work. Both the teacher and students appear to be only "going through the motions."

The classroom culture is characterized by high expectations for most students and genuine commitment to the subject by both teacher and students, with students demonstrating pride in their work.

High levels of student energy and teacher passion for the subject create a culture for learning in which everyone shares a belief in the importance of the subject and all students hold themselves to high standards of performance--for example, by initiating improvements to their work.

DOMAIN 3: INSTRUCTION UNSATISFACTORY

1 BASIC

2 PROFICIENT

3 DISTINGUISHED

4 Component 3e: Demonstrating Flexibility and Responsiveness INTASC St 8

The teacher adheres to the instruction plan, even when a change would improve the lesson or address students' lack of interest. The teacher brushes aside student questions; when

The teacher attempts to modify the lesson when needed and to respond to student questions, with moderate success. The teacher accepts responsibility for student success, but has only a limited

The teacher promotes the successful learning of all students, making adjustments as needed to instruction plans and accommodating student questions, needs, and interests.

The teacher seizes an opportunity to enhance learning, building on a spontaneous event or student interests. The teacher ensures the success of all students, using an

39

students experience difficulty, the teacher blames the students or their home environment.

repertoire of strategies to draw upon.

extensive repertoire of instructional strategies.

DOMAIN 4: PROFESSIONAL RESPONSIBILITES

UNSATISFACTORY 1

BASIC 2

PROFICIENT 3

DISTINGUISHED 4

Component 4c: Communicating with Families INTASC St 10

The teacher's communication with families about the instructional program or about individual students is sporadic or culturally inappropriate. The teacher makes no attempt to engage families in the instructional program.

The teacher adheres to school procedures for communicating with families and makes modest attempts to engage families in the instructional program. But communications are not always appropriate to the cultures of those families.

The teacher communicates frequently with families and successfully engages them in the instructional program. Information to families about individual students is conveyed in a culturally appropriate manner.

The teacher's communication with families is frequent and sensitive to cultural traditions; students participate in the communication. The teacher successfully engages families in the instructional program, as appropriate.

Component 4f: Showing Professionalism INTASC St 9, 10

The teacher has little sense of ethics and professionalism and contributes to practices that are self-serving or harmful to students. The teacher fails to comply with school and district regulations and time lines.

The teacher is honest and well intentioned in serving students and contributing to decisions in the school, but the teacher's attempts to serve students are limited. The teacher complies minimally with school and district regulations, doing just enough to get by.

The teacher displays a high level of ethics and professionalism in dealings with both students and colleagues and complies fully and voluntarily with school and district regulations.

The teacher is proactive and assumes a leadership role in making sure that school practices and procedures ensure that all students, particularly those traditionally underserved, are honored in the school. The teacher displays the highest standards of ethical conduct and takes a leadership role in seeing that colleagues comply with school and district regulations.

Dispositions – University Supervisor

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4

N Mean SD

2016-17 MED

Component 2a: Creating an Environment of Respect and Rapport 3 7 10 3.70 0.48

Component 2b: Establishing a Culture for Learning 3 7 10 3.70 0.48

Component 3e: Demonstrating Flexibility and Responsiveness 3 7 10 3.70 0.48

Component 4c: Communicating with Families 1 2 0 7 10 3.30 1.16

Component 4f: Showing Professionalism 3 7 10 3.70 0.48

40

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

2015-16 MED

Component 2a: Creating an Environment of Respect and Rapport 1 4 4 9 3.33 0.71

Component 2b: Establishing a Culture for Learning 5 4 9 3.44 0.53

Component 3e: Demonstrating Flexibility and Responsiveness 1 4 4 9 3.33 0.71

Component 4c: Communicating with Families 1 4 4 9 3.33 0.71

Component 4f: Showing Professionalism 5 4 9 3.44 0.53

2014-15 MED

Component 2a: Creating an Environment of Respect and Rapport 7 15 22 3.68 0.48

Component 2b: Establishing a Culture for Learning 8 14 22 3.64 0.49

Component 3e: Demonstrating Flexibility and Responsiveness 1 6 15 22 3.63 0.58

Component 4c: Communicating with Families 1 6 15 22 3.63 0.58

Component 4f: Showing Professionalism 7 15 22 3.68 0.48

41

S1_1.7_Technology Although the various technology projects may vary across licensure areas, they all involve the planning and teaching of a lesson or lesson in a unit using various educational technology, most often in the classroom or a field site. The technology project is an assignment typically administered at the intermediate level in the candidates’ classroom or in the internship experience for those candidates who are not teachers of record. Below is the common iLab scoring rubric used across programs by all faculty to score the assignment and the aggregated data across the licensure areas. The data tables follow and provide data for the EDUC MED program for those program completers who graduated in the 2014-15, 2015-16, and 2016-17 academic years. Review of these tables suggests that most of our completers of the EDUC MED program demonstrate an above standard to exemplary knowledge and understanding of the use of technology in their teaching. But, we do not believe that we can provide more interpretation of our candidates’ technology skills used in instruction and management due to the general nature of the assessment rubric. Frankly, we do not believe that the assessment captures much of the technology knowledge and skills our candidates must develop and display across their program. Almost every single course they take involves the use of online technology through Blackboard course shells as well as using the internet to develop and support a wide range of assignments, many of which are lesson and unit plans that require descriptions of the use of technology in instruction. The use of technology in teaching students in field and clinical experiences is also included in the AR-TESS observation assessment (Component 2e: Organizing Physical Space). In terms of the validity of the measure, it is important to note that unlike the other iLab assessments of performance, the common rubric below is not based on an assessment of candidate performance that is a key program assessment. Thus, it has not been vetted and approved by specific professional associations and the Arkansas Department of Education. However, this iLab lesson plan assessment has been in place for several years and was evaluated by the previous NCATE Board of Examiners. In terms of reliability, it is important to note that most of the faculty who are assessing our candidates on the measure have been using this key assessment, both at the program level and at the EPP level for iLab for several years. Validity & Reliability: Validity and reliability studies were performed by an expert panel in the Spring 2017 semester. Please see Standard 5.1 and 5.2 for a description of the extensive analyses of the validity and reliability studies of this measure. As a result of review of the alignment tables and the panel review, the EPPAC has determined that the current assignment parameters and scoring rubric for the Technology Assessment are insufficient. Moreover, the assessment does not adequately address EPP Goal 3 (to prepare K-12 candidates who demonstrate innovative best practices including technology). The EPPAC committee has already begun identification of specific ISTE Standards with which to align a revised Technology Assessment to be completed during the 2017 – 2018 academic term. It should also be noted that although there was 100% agreement on the Technology Assessment, the quality of

42

the assessment is less than desired and will be revised. The revised Technology Assessment will then undergo review for evidence of reliability and validity.

Results from Interrater Reliability Analyses

Agreement

Ratings Differed by 1 Point

Ratings Differed by 2 Points

Cohen’s kappa

Technology 1 Category

5/5 = 100% 0/5 = 0% 0/5 = 0% NA

Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees will review the 3 years of data and will decide upon benchmarks for candidate performance in subsequent years of data. An example of a benchmark for this assessment, based on initial review of overall means for the initial licensure programs, would be a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that did not meet the benchmark would need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual action plans and present this data in an aggregated form in the annual assessment report. The SOE Director of Assessment will aggregate these assessment analyses and action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review.

3.1 TCED Technology iLab UG & Grad The assessment is used as evidence for the following INTASC Standards for Candidate Knowledge, Skills, and Professional Dispositions: Standard #7: Planning for Instruction The teacher plans instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context. Standard #8: Instructional Strategies The teacher understands and uses a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways. 1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

43

Instructional Strategy INTASC St 7, 8 1.0

Project demonstrates limited knowledge and understanding of the use of technology in teaching.

Project demonstrates knowledge and understanding of the use of technology in teaching.

Project demonstrates above standard knowledge and understanding of the use of technology in teaching.

Project demonstrates exemplary knowledge and understanding of the use of technology in teaching.

Table S1_1.7_Technology: Use of Technology in Teaching by Program

2016-17 Use of Technology in Teaching Initial Licensure Program Degree Unsatisfactory

1 Basic

2 Proficient

3 Distinguished

4 N Mean SD

Early Childhood Education BSE 39% 61% 31 3.60 0.49

Middle Childhood Education BSE 100% 4 3.00 0.00

Middle Childhood Education MED 100% 1 3.00 0.00

Special Education BSE 67% 33% 3 3.33 0.58

Secondary Education Minor By UG Content Major:

Foreign Language BA 100% 1 4.0 0.00

Math BS 100% 2 3.0 0.00

Social Studies (History) BA 100% 2 4.0 0.00

Secondary Education Graduate By Content/Licensure Area:

Art MED 100% 1 4.0 0.00

Drama/Speech MED 100% 2 4.0 0.00

English MED 100% 3 4.0 0.00

Math MED 100% 1 4.0 0.00

Music MED 100% 2 4.0 0.00

Science MED 100% 2 4.0 0.00

Commented [BS1]: Need ELED data, need to unpack sced med by program; no sced ug data by program; but we are showing an N of 3 by assessment instrument

44

2015-16 Use of Technology in Teaching

Initial Licensure Program Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

Early Childhood Education BSE 36% 64% 22 3.6 0.5

Middle Childhood Education BSE 80% 20% 10 3.2 0.40

Middle Childhood Education MED 43% 43% 14% 7 2.7 0.70

Secondary Education Minor By UG Content Major:

PE/Health Sciences BS 25% 75% 4 3.8 0.43

Science BS 100% 1 4.0 0

Social Studies (History) BA 20% 80% 4 3.8 0.43

Secondary Education Graduate By Content/Licensure Area:

Drama/Speech MED 100% 1 4.0 0

English MED 100% 4 4.0 0

Music MED 100% 1 3.0 0

PE/Health MED 100% 1 3.0 0

Science MED 100% 2 4.0 0

Social Studies MED 100% 1 4.0 0

2014-15 Use of Technology in Teaching

Initial Licensure Program Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

Early Childhood Education BSE 18% 82% 34 3.8 0.5

45

Initial Licensure Program Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

Middle Childhood Education BSE 100% 6 4.0 0

Middle Childhood Education MED 100% 1 4.0 0

Secondary Education Minor By UG Content Major:

Art BA 100% 2 3.0 0

English BA 100% 1 3.0 0

PE/Health Sciences BS 33% 67% 3 3.7 0.5

Science BS 100% 1 4.0 0

Social Studies BA 100% 2 4.0 0

Secondary Education Graduate By Content/Licensure Area:

Art MED 100% 4 4.0 0

Business Technology MED 100% 5 4.0 0

Drama/Speech MED 100% 1 4.0 0

English MED 100% 5 4.0 0

Foreign Language (Spanish) MED 100% 1 3.0 0

Math MED 100% 1 3.0 0

Music MED 100% 1 4.0 0

PE/Health MED 100% 2 4.0 0

Science MED 100% 3 4.0 0

Social Studies MED 100% 3 4.0 0

46

S1_1.8_Teacher Work Sample Although the various teacher work sample assignments may vary across licensure areas, they all involve a sequence of the use of assessment in assisting the candidate in a clinical site in planning instruction, the teaching of the plan, using formative assessment to guide instruction, a summative evaluation of instruction of student learning, and an analysis of the data to guide reflection. The teacher work sample is an assignment typically administered at the intermediate level in the classroom, field experiences or at exit in internship for those candidates who are not yet teachers of record. Below is the common iLab scoring rubric used across programs by all faculty to score the assignment and the aggregated data by program. The data tables follow and provide data by program for those program completers who graduated in the 2014-15, 2015-16, and 2016-17 academic years. Review of these tables suggests a majority of our EDUC MED completers demonstrate an emerging to comprehensive understanding of the use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making. Additionally, candidates show emerging to comprehensive ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context. Finally, they demonstrate an emerging to comprehensive understanding and use of a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways. It is important to note that our candidates submit a number of similar assignments involving the use of assessments across a teaching experience to guide planning and instruction and evaluate the success of that instruction for assessment across their programs (such as the case study, the unit plan, and AR-TESS observations during field experiences and internship). In terms of the validity of the measure, it is important to note that the iLab assessment of performance using the common rubric below is based on an assessment of candidate performance in a key assessment (Assessment #5) in the field site from each initial program that has been vetted and approved by specific professional associations and the Arkansas Department of Education. Further, the program assessments are also aligned with INTASC, the specific Arkansas Curriculum Frameworks (most based on Common Core), and the Arkansas Teaching Standards (that use INTASC Standards and extend them). These assessments were originally developed by faculty and subsequently revised by a collection of experts such as EPP faculty, P-12 teachers, P-12 administrators, and EPP alumni. This iLab lesson plan assessment has been in place for several years and was evaluated by the previous NCATE Board of Examiners. Since the last NCATE review, the iLab rubric has been realigned with the new CAEP and INTASC standards. However, faculty analyses of the data suggest that the common iLab teacher sample rubric does not allow for a more fine-grained analysis of the specific performances required. There are many levels involved that the instrument does not effectively capture. How the candidate assesses, analyzes assessment data, reflects on the experience using the data analyses, and adjusts plans and instruction is a sequence that needs to be more adequately assessed to allow for more specific and targeted feedback to assist EPP development. We need to

47

differentiate the skill levels on the rubric on this EPP assessment. It is not sensitive enough or operationalized enough to capture the complex skills we are trying to assess. In terms of reliability, it is important to note that most of the faculty who are assessing our candidates on the measure have been using this key assessment, both at the program level and at the EPP level for iLab for several years. Validity & Reliability: Validity and reliability studies were performed by an expert panel in the Spring 2017 semester. In terms of content validity, there is a concern about the range of the alignment of the assessment with INTASC standards. Comments from the Expert Panel also suggested that while there are essential elements in the measure, some other essential elements are missing. Thus, in the future, we hope to expand the iLab rubric as described above to be able to capture the essential and sequenced assessment, evaluation, planning, instruction, and reflection skills across all programs. Nevertheless, the findings below appear to be valid and measure essential aspects of a teacher work sample. An inter-rater reliability study was also conducted and the results are presented below. Cohen’s kappa results were statistically not significant (K = 1.72, p = .086) for ratings on the Teacher Work Sample; however, 80% perfect agreement and only 20% disagreement by a single point so there is a need for some improvement in scoring this rubric but the rating differences are not substantial.

Results from Interrater Reliability Analyses

Agreement

Ratings Differed by 1 Point

Ratings Differed by 2 Points

Cohen’s kappa

Teacher Work Sample 3 Categories

12/15 = 80% 3/15 = 20% 0/15 = 0% K = 1.72 p = .086

Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees reviewed the 3 years of data and decided upon benchmarks for candidate performance in subsequent years of data. In Fall 2018, faculty will analyze the 2017=18 data using these benchmarks. The benchmark for this assessment, based on initial review of overall means for the initial licensure programs, is a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that do not meet the benchmark will need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual action plans and present this data in an aggregated form in the annual assessment report. The SOE Assessment Coordinator will aggregate these assessment analyses and

48

action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review.

4.1 Teacher work Sample TCED iLab UG and Graduate Standard #6: Assessment The teacher understands and uses multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making; Standard #7: Planning for Instruction The teacher plans instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, an pedagogy, as well as knowledge of learners and the community context; Assessment #8 Instructional Strategies The teacher understands and uses a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways. 1.0Unacceptable 2.0Basic 3.0Proficient 4.0Distinguished

Assessment INTASC St 6

Demonstrates minimal or no understanding and use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

Demonstrates limited understanding and use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

Demonstrates emerging understanding and use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

Demonstrates comprehensive understanding and use of multiple methods of assessment to engage learners in their own growth, to monitor learner progress, and to guide the teacher's and learner's decision making.

1.0Unacceptable 2.0Basic 3.0Proficient 4.0Distinguished

Planning for Instruction INTASC St 7

Demonstrates minimal or no ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

Demonstrates limited ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

Demonstrates emerging ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

Demonstrates comprehensive ability to plan instruction that supports every student in meeting rigorous learning goals by drawing upon knowledge of content areas, curriculum, cross-disciplinary skills, and pedagogy, as well as knowledge of learners and the community context.

1.0Unacceptable 2.0Basic 3.0Proficient 4.0Distinguished

Instructional Strategies Demonstrates minimal or no understanding and use of a

Demonstrates limited understanding and use of

Demonstrates emerging understanding and use of

Demonstrates comprehensive

49

INTASC St 8 variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways.

a variety of instructional strategies to encourage learners to devlop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways.

a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways.

understanding and use of a variety of instructional strategies to encourage learners to develop deep understanding of content areas and their connections, and to build skills to apply knowledge in meaningful ways.

Table 1.5 Teacher Work Sample

Teacher Work Sample EDUC MED Degree Unsatisfactory

1 Basic

2 Proficient

3 Distinguished

4 N Mean SD

2016-17 MED

Assessment 10 10 4.0 0.00

Planning for Instruction 10 10 4.0 0.00

Instructional Strategies 10 10 4.0 0.00

2015-16 MED

Assessment 3 8 11 3.73 0.47

Planning for Instruction 3 8 11 3.73 0.47

Instructional Strategies 3 8 11 3.73 0.47

2014-15 MED

Assessment 4 19 23 3.83 0.39

Planning for Instruction 4 19 23 3.83 0.39

Instructional Strategies 4 19 23 3.83 0.39

50

Table S1_1.9_Teaching Philosophy All EDUC MED initial licensure candidates demonstrate that they are developing professional dispositions related to fairness and the belief that all students can learn through a common assessment across all programs, the Philosophy of Teaching and Learning, which has four major domains: (1) Learner Development involves the development of a coherent philosophy of teaching and learning that has emerged from course and field experiences and is related to professional theories that all students can learn but may require accommodations to individual differences based on the understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas and designs and implements; (2) Learning Differences refers to a general and specific theoretical understanding individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards; (3) Learning Environments refers to a general and specific theoretical understanding of and the ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation; and (4) Professional Learning and Ethical Practice refers to a general and specific theoretical understanding that leads to a demonstration of ongoing professional learning and using evidence to continually evaluate his/her practice, particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community), and adapts practice to meet the needs of each learner. The Philosophy of Teaching and Learning is an assessment administered at the intermediate or exit level. The assessment is a written philosophy statement. Several programs administer this assessment twice to attempt to assess growth across a candidate’s program while some programs administer the assessment once and often at the intermediate level prior to internship. Below is the common i-Lab scoring rubric used across programs by all faculty to score the assignment and the aggregated data by program by each domain of the assessment. The data tables follow and provide data by program for those program completers who graduated in the 2014-15, 2015-16, and 2016-17 academic years. Review of these tables suggests that a majority of our EDUC MED completers demonstrate an emerging to comprehensive understanding that all students can learn but may require accommodations to individual differences based on the understanding of how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas and designs and implements. They also appear to have developed an emerging to comprehensive understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. A majority of our completers also appear to have developed ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self-motivation. Finally, they appear to have developed a general and specific theoretical understanding of the need for ongoing professional learning and using evidence to continually evaluate his/her practice,

51

particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community), and adapts practice to meet the needs of each learner. In terms of the validity of the measure, it is important to note that unlike the other iLab assessments of performance, the common rubric below is not based on an assessment of candidate performance that is a key program assessment. Thus, it has not been vetted and approved by specific professional associations and the Arkansas Department of Education. However, this iLab teaching philosophy assessment has been in place for several years and was evaluated by the previous NCATE Board of Examiners. Further, the program assessments are also aligned with INTASC, the specific Arkansas Curriculum Frameworks (most based on Common Core), and the Arkansas Teaching Standards (that use INTASC Standards and extend them). Since the last NCATE review, the iLab rubric has been realigned with the new CAEP and INTASC standards and we redesigned the rubric with four separate levels from that of a three-level rubric, thus allowing for four separate scores for the four dimensions of lesson planning in the current iLab rubric (from Common Professional Knowledge, Sensitivity to Individual Differences, and Understanding of Student-Centered Learning to Learner Development, Learning Differences, Learning Environments, and Professional Learning and Ethical Practice). In terms of reliability, it is important to note that most of the faculty who are assessing our candidates on the measure have been using this assessment, both at the program level and at the EPP level for iLab for several years. Validity & Reliability: Validity and reliability studies were performed by an expert panel in the Spring 2017 semester. In terms of content validity, there is a concern about the range of the alignment of the assessment with INTASC standards. Comments from the Expert Panel also suggested that while there are essential elements in the measure, some other essential elements may be missing. Thus, in the future, we hope to expand the iLab rubric as described above to be able to capture some of these essential elements mentioned by the panel, such as expanding the descriptors of considering learning differences to explicitly include students with disabilities, and to include philosophy statements from the beginning of the program to exit to display growth and development in teaching philosophies. Nevertheless, the findings below appear to be valid and measure essential aspects of a professional teaching philosophy. An inter-rater reliability study was also conducted and the results are presented below. Cohen’s kappa results were statistically not significant for ratings on the Teaching Philosophy; however, 70% perfect agreement and 15% disagreement by a single point and 15% disagreement by 2 points suggest that there is a need for improvement in scoring this rubric but the rating differences are not substantial.

Results from Interrater Reliability Analyses

Agreement

Ratings Differed by 1 Point

Ratings Differed by 2 Points

Cohen’s kappa

Teaching Philosophy 14/20 = 70% 3/20 = 15% 3/20 = 15% K = 0.93

52

4 Categories p = .353 Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees reviewed the 3 years of data and decided upon benchmarks for candidate performance in subsequent years of data. In Fall 2018, faculty will analyze and interpret the 2017-18 data using this benchmark. The benchmark, based on initial review of overall means for the initial licensure programs, is a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that did not meet the benchmark would need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual action plans and present this data in an aggregated form in the annual assessment report. The SOE Assessment Coordinator will aggregate these assessment analyses and action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review.

4.2 Teaching Philosophy TCED iLab UG (2) The assessment meets part of the requirements for CAEP Standard 1: Standard 1: CONTENT AND PEDAGOGICAL KNOWLEDGE The provider ensures that candidates develop a deep understanding of the critical concepts and principles of their discipline and, by completion, are able to use discipline-specific practices flexibly to advance the learning of all students toward attainment of college- and career-readiness standards. The assessment is used as evidence for the following INTASC Standard Candidate Knowledge, Skills, and Professional Dispositions: Standard #1: Learner Development. The teacher understands how learners grow and develop, recognizing that patterns of learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas, and designs and implements developmentally appropriate and challenging learning experiences. Standard #2: Learning Differences. The teacher uses understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards. Standard #3: Learning Environments. The teacher works with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation. 1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Learner Development INTASC St 1

Demonstrates minimal or no understanding of how learners grow and develop, recognizing that patterns of learning and development

Demonstrates limited understanding of how learners grow and develop, recognizing that patterns of learning and

Demonstrates emerging understanding of how learners grow and develop, recognizing that patterns of learning and

Demonstrates comprehensive understanding of how learners grow and develop, recognizing that patterns of

53

vary individually within and across the cognitive, linguistic, social, emotional, and physical areas and designs and implements developmentally appropriate and challenging learning experiences.

development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas and designs and implements developmentally appropriate and challenging learning experiences.

development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas and designs and implements developmentally appropriate and challenging learning experiences.

learning and development vary individually within and across the cognitive, linguistic, social, emotional, and physical areas and designs and implements developmentally appropriate and challenging learning experiences.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Learning Differences INTASC St 2

Demonstrates minimal or no use of understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards.

Demonstrates limited use of understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards.

Demonstrates emerging use of understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards.

Demonstrates comprehensive use of understanding of individual differences and diverse cultures and communities to ensure inclusive learning environments that enable each learner to meet high standards.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Learning Environments INTASC St 3

Displays minimal or irrelevant knowledge of student differences (i.e. development, interests, culture, abilities/disabilities) that may affect learning.

Demonstrates limited ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation.

Demonstrates emerging ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation.

Demonstrates comprehensive ability to work with others to create environments that support individual and collaborative learning, and that encourage positive social interaction, active engagement in learning, and self motivation.

1.0Unsatisfactory 2.0Basic 3.0Proficient 4.0Distinguished

Professional Learning and Ethical Practice INTASC St 9

Demonstrates ongoing professional learning and uses evidence at a minimal or no level to continually evaluate his/her practice,

Demonstrates ongoing professional learning and uses evidence at limited level to continually evaluate his/her practice,

Demonstrates ongoing professional learning and uses evidence at emerging level to continually evaluate

Demonstrates ongoing professional learning and uses evidence at comprehensive level to continually evaluate his/her

54

particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community:, and adapts practice to meet the needs of each learner.

particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community:, and adapts practice to meet the needs of each learner.

his/her practice, particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community:, and adapts practice to meet the needs of each learner.

practice, particularly the effects of his/her choices and actions on others (learners, families, other professionals, and the community:, and adapts practice to meet the needs of each learner.

Table S1_1.9_Teaching_Philosophy

Teaching Philosophy EDUC MED Degree Unsatisfactory

1 Basic

2 Proficient

3 Distinguished

4 N Mean SD

2016-17 MED

Learner Development 2 6 8 3.75 0.46

Learning Differences 2 6 8 3.75 0.46

Learning Environment 1 7 8 3.88 0.35

Professional Learning and Ethical Practices 2 6 8 3.75 0.46

2015-16 MED

Learner Development 11 11 4.00 0.00

Learning Differences 11 11 4.00 0.00

Learning Environment 11 11 4.00 0.00

Professional Learning and Ethical Practices 11 11 4.00 0.00

2014-15 MED

Learner Development 5 23 28 3.82 0.39

Learning Differences 5 23 28 3.82 0.39

Learning Environment 5 23 28 3.82 0.39

Professional Learning and Ethical Practices 5 23 28 3.82 0.39

55

S1_1.10_TESS Internship Observation Although the number and kind of observations may vary by program across field and clinical experiences, this assessment involves the use of the same observational instrument for all initial licensure candidates by trained university observers that is based on the Arkansas Teacher Excellence and Support System (AR-TESS) Instrument (http://www.arkansased.gov/divisions/human-resources-educator-effectiveness-and-licensure/office-of-educator-effectiveness/teacher-evaluation-system ). AR-TESS is used in Arkansas schools to evaluate teacher performance. This system is based on Danielson’s framework for teaching (https://www.danielsongroup.org/framework/ ). The observer assesses and analyzes the teaching in twenty-two specific criteria within four broad domains of effective teaching: Planning and Preparation; The Classroom Environment; Instruction; and Professional Responsibilities. The TESS-based instrument in this assessment maintains the domains and components or subdomains of the TESS instrument as well as the descriptors of each level across the components/subdomains, and all observers have successfully completed Arkansas Department of Education approved TESS training. The iLab TESS Internship Observation assessment offered below involves the final observation by the university supervisor (US) because that is the one observation that all initial candidates have in common, since most of the graduate initial licensure candidates are teachers of record on a provisional license and do not also have cooperating teachers observing them like the undergraduate initial candidates. Below is the common iLab scoring rubric (the AR-TESS scoring rubric) used across programs by all university supervisors to score the assessment. This rubric is aligned to the INTASC Standards using The Danielson Group’s “Correlation between the Danielson Framework for Teaching and the… InTASC Standards” table available for download at http://www.danielsongroup.org/download/?download=448 as a guide. The data tables follow and provide data by program for those program completers who graduated in the 2014-15, 2015-16, and 2016-17 academic years. Review of these tables suggests that a vast majority of our EDUC MED completers were observed as being proficient to distinguished across all four broad domains and all 22 components or subdomains of the TESS Observation Scoring Rubric. While there is some variation, in general, our initial licensure candidates:

• plans and practice reflect solid knowledge of the content, prerequisite relationships between important concepts, and the instructional practices specific to that discipline. • actively seeks knowledge of students' backgrounds, cultures, skills, language proficiency, interests, and special needs, and attains this knowledge for groups of students. • Instructional outcomes are stated as goals reflecting high-level learning and curriculum standards. They are suitable for most students in the class, represent different types of learning,

and can be assessed. The outcomes reflect opportunities for coordination. • Are fully aware of the resources available through the school or district to enhance own knowledge, to use in teaching, or for students who need them. • coordinate knowledge of content, of students, and of resources to design a series of learning experiences aligned to instructional outcomes and suitable for groups of students. The lesson

or unit has a clear structure and is likely to engage students in significant learning. • plans for student assessment are aligned with the instructional outcomes, uses clear criteria, and is appropriate to the needs of students. The candidate intends to use assessment results

to plan for future instruction for groups of students.

56

• Classroom interactions between the teacher and students and among students are polite and respectful, reflecting general warmth and caring, and are appropriate to the cultural and developmental differences among groups of students.

• The classroom culture is characterized by high expectations for most students and genuine commitment to the subject by both teacher and students, with students demonstrating pride in their work.

• Little instructional time is lost because of classroom routines and procedures for transitions, handling of supplies, and performance of non-instructional duties, which occur smoothly. • Standards of conduct appear to be clear to students, and the teacher monitors student behavior against those standards. The candidate’s response to student misbehavior is appropriate

and respects the students' dignity. • The classroom is safe, and learning is accessible to all students; the teacher ensures that the physical arrangement is appropriate to the learning activities. The candidate makes effective

use of physical resources, including computer technology. • Expectations for learning, directions and procedures, and explanations of content are clear to students. Communications are appropriate for students' cultures and levels of development. • Most of the candidate's questions elicit a thoughtful response, and the teacher allows sufficient time for students to answer. All students participate in the discussion, with the teacher

stepping aside when appropriate. • Activities and assignments, materials, and groupings of students are fully appropriate for the instructional outcomes and students' cultures and levels of understanding. All students are

engaged in work of a high level of rigor. The lesson's structure is coherent, with appropriate pace. • Assessment is regularly used in instruction, through self-assessment by students, monitoring of progress of learning by the candidate and/or students, and high-quality feedback to

students. Students are fully aware of the assessment criteria used to evaluate their work. • promotes the successful learning of all students, making adjustments as needed to instruction plans and accommodating student questions, needs, and interests. • provides an accurate and objective description of the lesson, citing specific evidence. The candidate makes some specific suggestions as to how the lesson might be improved. • systems for maintaining both instructional and non-instructional records are accurate, efficient, and effective. • communicates frequently with families and successfully engages them in the instructional program. Information to families about individual students is conveyed in a culturally

appropriate manner. • participates actively in the professional community and in school and district events and projects, and maintains positive and productive relationships with colleagues. • seeks out opportunities for professional development based on an individual assessment of need and actively shares expertise with others. The candidate welcomes feedback from

supervisors and colleagues. • displays a high level of ethics and professionalism in dealings with both students and colleagues and complies fully and voluntarily with school and district regulations.

In terms of the validity of the measure, it is important to note that the assessment rubric is taken from the AR-TESS observation instrument used in the state of Arkansas to evaluate the performance of teachers in schools (http://www.arkansased.gov/divisions/human-resources-educator-effectiveness-and-licensure/office-of-educator-effectiveness/teacher-evaluation-system ). This system is based on Danielson’s framework for teaching (https://www.danielsongroup.org/framework/ ), a system that many of EPP faculty have used for several years because our candidates used to have to successfully pass the Praxis 3- Pathwise assessment for licensure, which was also based on Danielson’s framework.

57

Our candidates are assessed on the AR-TESS rubric across multiple field and clinical experiences by university supervisors. Most graduate candidates are not observed by Cooperating Teachers because they are Teachers of Record, teaching their own class on a provisional licensure. Faculty have been trained on the AR-TESS and we have also calibrated our ratings at least two times across the last few years. All Cooperating Teachers must have received ADE training on the AR-TESS as well. However, what we have not done is any formal inter-rater reliability studies of our use of the AR-TESS measure. We have already started a reliability study of the agreement between University Supervisors and Cooperating Teachers for those candidates who had both ratings in their teaching performance in field and clinical experiences. Results of this study will be available in Fall 2018. The assessment team that analyzed the data for this self-study has also suggested a validity study to investigate the extent to which our disposition assessments could have identified those few candidates who had concerns. Perhaps our biggest concern with reliability of the instrument is a matter that we have debated for some time across the EPP. This instrument was created for assessing teachers in schools for their evaluation by administrators. It is generally considered best practice that beginning teachers should develop up through the scale from basic to proficient and few teachers, presumably veteran teachers, should score at the distinguished level. But, we are assessing interns and are concerned that sticking with these “guidelines” for schools would basically provide us with a 2-scale measure (unsatisfactory or basic). So, some observers use the entire range (that is, including proficient and distinguished for interns) of the scale while others appear to be using the school rating scale with relatively low ratings as the “norm” for beginning educators. This is a matter that we hope to resolve in the upcoming validity and reliability studies in spring 2018. Validity and Reliability Studies: Because we have taken these rubrics directly from an external assessment, we do not plan on performing validity studies on this instrument. We do plan to do inter-rater reliability studies of the TESS Observation instrument in 2018. Goal Benchmarking: In Fall 2017, EPPAC, faculty, and the advisory committees reviewed the 3 years of data and decided upon benchmarks for candidate performance in subsequent years of data. In Fall 2018, faculty will analyze and interpret 2017-18 data based on these benchmarks. The benchmark for this assessment, based on initial review of overall means for the initial licensure programs, is a program mean of at least 3.33 on all performance levels with at least 67% of all candidates performing at the “Proficient” level on all levels of performance. Further, programs that do not meet the benchmark will need to review existing program curriculum and course activities tied to the assessment and provide a detailed discussion of the analysis and possible action plans for meeting the benchmark in the future in the annual program assessment report. Finally, program faculty will also provide individual development plans for individuals who score “Basic” or below on any of the levels of performance. The program faculty will aggregate these individual

58

action plans and present this data in an aggregated form in the annual assessment report. The SOE Assessment Coordinator will aggregate these assessment analyses and action plans and the candidate development plans across programs for inclusion in the annual EPP assessment report in the goal benchmarking section of the specific EPP-wide assessment review. TESS Observation Scoring Rubric

DOMAIN 1: PLANNING AND PREPARATION

UNSATISFACTORY 1

BASIC 2

PROFICIENT 3

DISTINGUISHED 4

Component 1a: Demonstrating Knowledge of Content and Pedagogy INTASC St 4

The teacher's plans and practice display little knowledge of the content, prerequisite relationships between different aspects of the content, or the instructional practices specific to that discipline.

The teacher's plans and practice reflect some awareness of the important concepts in the discipline, prerequisite relationships between them, and the instructional practices specific to that discipline.

The teacher's plans and practice reflect solid knowledge of the content, prerequisite relationships between important concepts, and the instructional practices specific to that discipline.

The teacher's plans and practice reflect extensive knowledge of the content and the structure of the discipline. The teacher actively builds on knowledge of prerequisites and misconceptions when describing instruction or seeking causes for student misunderstanding.

Component 1b: Demonstrating Knowledge of Students INTASC St 1, 2, 7

The teacher demonstrates little or no knowledge of students' backgrounds, cultures, skills, language proficiency, interests, and special needs, and does not seek such understanding

The teacher indicates the importance of understanding students' backgrounds, cultures, skills, language proficiency, interests, and special needs, and attains this knowledge for the class as a whole.

The teacher actively seeks knowledge of students' backgrounds, cultures, skills, language proficiency, interests, and special needs, and attains this knowledge for groups of students.

The teacher actively seeks knowledge of students' backgrounds, cultures, skills, language proficiency, interests, and special needs from a variety of sources, and attains this knowledge for individual students.

Component 1c: Setting Instructional Outcomes INTASC St 1, 7

Instructional outcomes are unsuitable for students, represent trivial or low-level learning, or are stated only as activities. They do not permit viable methods of assessment.

Instructional outcomes are of moderate rigor and are suitable for some students, but consist of a combination of activities and goals, some of which permit viable methods of assessment. They reflect more than one type of learning, but the teacher makes no attempt at coordination or integration.

Instructional outcomes are stated as goals reflecting high-level learning and curriculum standards. They are suitable for most students in the class, represent different types of learning, and can be assessed. The outcomes reflect opportunities for coordination.

Instructional outcomes are stated as goals that can be assessed, reflecting rigorous learning and curriculum standards. They represent different types of content, offer opportunities for both coordination and integration, and take the needs of individual students into account.

Component 1d: Demonstrating Knowledge of Resources INTASC St 3, 7

The teacher demonstrates little or no familiarity with resources to enhance own knowledge, to use in teaching, or for students who need them. The teacher does not seek such knowledge.

The teacher demonstrates some familiarity with resources available through the school or district to enhance own knowledge, to use in teaching, or for students who need them. The teacher does not seek to extend such knowledge.

The teacher is fully aware of the resources available through the school or district to enhance own knowledge, to use in teaching, or for students who need them.

The teacher seeks out resources in and beyond the school or district in professional organizations, on the Internet, and in the community to enhance own knowledge, to use in teaching, and for students who need them.

Component 1e: Designing Coherent Instruction INTASC St 1, 4, 7

The series of learning experiences is poorly aligned with the instructional outcomes and does not represent a coherent structure. The experiences are suitable for only some students.

The series of learning experiences demonstrates partial alignment with instructional outcomes, and some of the experiences are likely to engage students in significant learning. The lesson or unit has a

The teacher coordinates knowledge of content, of students, and of resources to design a series of learning experiences aligned to instructional outcomes and suitable for groups of students. The lesson or

The teacher coordinates knowledge of content, of students, and of resources, to design a series of learning experiences aligned to instructional outcomes, differentiated where appropriate to make them suitable to

59

recognizable structure and reflects partial knowledge of students and resources.

unit has a clear structure and is likely to engage students in significant learning.

all students and likely to engage them in significant learning. The lesson or unit structure is clear and allows for different pathways according to student needs.

Component 1f: Designing Student Assessments INTASC St 6

The teacher's plan for assessing student learning contains no clear criteria or standards, is poorly aligned with the instructional outcomes, or is inappropriate for many students. The results of assessment have minimal impact on the design of future instruction.

The teacher's plan for student assessment is partially aligned with the instructional outcomes, without clear criteria, and inappropriate for at least some students. The teacher intends to use assessment results to plan for future instruction for the class as a whole.

The teacher's plan for student assessment is aligned with the instructional outcomes, uses clear criteria, and is appropriate to the needs of students. The teacher intends to use assessment results to plan for future instruction for groups of students.

The teacher's plan for student assessment is fully aligned with the instructional outcomes, with clear criteria and standards that show evidence of student contribution to their development. Assessment methodologies may have been adapted for individuals, and the teacher intends to use assessment results to plan future instruction for individual students.

DOMAIN 2: THE CLASSROOM

ENVIRONMENT

UNSATISFACTORY 1

BASIC 2

PROFICIENT 3

DISTINGUISHED 4

Component 2a: Creating an Environment of Respect and Rapport INTASC St 2, 3

Classroom interactions, both between the teacher and students and among students, are negative, inappropriate, or insensitive to students' cultural backgrounds and are characterized by sarcasm, put-downs, or conflict.

Classroom interactions, both between the teacher and students and among students, are generally appropriate and free from conflict, but may be characterized by occasional displays of insensitivity or lack of responsiveness to cultural or developmental differences among students.

Classroom interactions between the teacher and students and among students are polite and respectful, reflecting general warmth and caring, and are appropriate to the cultural and developmental differences among groups of students.

Classroom interactions between the teacher and individual students are highly respectful, reflecting genuine warmth and caring and sensitivity to students' cultures and levels of development. Students themselves ensure high levels of civility among members of the class.

Component 2b: Establishing a Culture for Learning INTASC St 3

The classroom environment conveys a negative culture for learning, characterized by low teacher commitment to the subject, low expectations for student achievement, and little or no student pride in work.

The teacher's attempt to create a culture for learning is partially successful, with little teacher commitment to the subject, modest expectations for student achievement, and little student pride in work. Both the teacher and students appear to be only "going through the motions."

The classroom culture is characterized by high expectations for most students and genuine commitment to the subject by both teacher and students, with students demonstrating pride in their work.

High levels of student energy and teacher passion for the subject create a culture for learning in which everyone shares a belief in the importance of the subject and all students hold themselves to high standards of performance--for example, by initiating improvements to their work.

Component 2c: Managing Classroom Procedures INTASC St 3

Much instructional time is lost because of inefficient classroom routines and procedures for transitions, handling of supplies, and performance of noninstructional duties.

Some instructional time is lost because classroom routines and procedures for transitions, handling of supplies, and performance of noninstructional duties are only partially effective.

Little instructional time is lost because of classroom routines and procedures for transitions, handling of supplies, and performance of noninstructional duties, which occur smoothly.

Students contribute to the seamless operation of classroom routines and procedures for transitions, handling of supplies, and performance of noninstructional duties.

Component 2d: Managing Student Behavior INTASC St 3

There is no evidence that standards of conduct have been established and little or no teacher monitoring of student behavior.

It appears that the teacher has made an effort to establish standards of conduct for students. The teacher tries, with uneven

Standards of conduct appear to be clear to students, and the teacher monitors student behavior against those standards. The teacher's

Standards of conduct are clear, with evidence of student participation in setting them. The teacher's monitoring of student behavior is subtle and

60

Response to student misbehavior is repressive or disrespectful of student dignity.

results, to monitor student behavior and respond to student misbehavior.

response to student misbehavior is appropriate and respects the students' dignity.

preventive, and the teacher's response to student misbehavior is sensitive to individual student needs. Students take an active role in monitoring the standards of behavior.

Component 2e: Organizing Physical Space INTASC St 3

The physical environment is unsafe, or some students don't have access to learning. Alignment between the physical arrangement and the lesson activities is poor.

The classroom is safe, and essential learning is accessible to most students; the teacher's use of physical resources, including computer technology, is moderately effective. The teacher may attempt to modify the physical arrangement to suit learning activities, with partial success.

The classroom is safe, and learning is accessible to all students; the teacher ensures that the physical arrangement is appropriate to the learning activities. The teacher makes effective use of physical resources, including computer technology.

The classroom is safe, and the physical environment ensures the learning of all students, including those with special needs. Students contribute to the use or adaptation of the physical environment to advance learning. Technology is used skillfully, as appropriate to the lesson.

DOMAIN 3: INSTRUCTION UNSATISFACTORY

1 BASIC

2 PROFICIENT

3 DISTINGUISHED

4 Component 3a: Communicating with Students INTASC St 5

Expectations for learning, directions and procedures, and explanations of content are unclear or confusing to students. The teacher's use of language contains errors or is inappropriate for students' cultures or levels of development.

Expectations for learning, directions and procedures, and explanations of content are clarified after initial confusion; the teacher's use of language is correct but may not be completely appropriate for students' cultures or levels of development.

Expectations for learning, directions and procedures, and explanations of content are clear to students. Communications are appropriate for students' cultures and levels of development.

Expectations for learning, directions and procedures, and explanations of content are clear to students. The teacher's oral and written communication is clear and expressive, appropriate for students' cultures and levels of development, and anticipates possible student misconceptions.

Component 3b: Using Questioning and Discussion Techniques INTASC St 8

The teacher's questions are low-level or inappropriate, eliciting limited student participation and recitation rather than discussion.

Some of the teacher's questions elicit a thoughtful response, but most are low-level, posed in rapid succession. The teacher's attempts to engage all students in the discussion are only partially successful.

Most of the teacher's questions elicit a thoughtful response, and the teacher allows sufficient time for students to answer. All students participate in the discussion, with the teacher stepping aside when appropriate.

Questions reflect high expectations and are culturally and developmentally appropriate. Students formulate many of the high-level questions and ensure that all voices are heard.

Component 3c: Engaging Students in Learning INTASC St 1, 3, 4, 5, 8

Activities and assignments, materials, and groupings of students are inappropriate for the instructional outcomes or students' cultures or levels of understanding, resulting in little intellectual engagement. The lesson has no structure or is poorly paced.

Activities and assignments, materials, and groupings of students are partially appropriate to the instructional outcomes or students' cultures or levels of understanding, resulting in moderate intellectual engagement. The lesson has a recognizable structure, but that structure is not fully maintained.

Activities and assignments, materials, and groupings of students are fully appropriate for the instructional outcomes and students' cultures and levels of understanding. All students are engaged in work of a high level of rigor. The lesson's structure is coherent, with appropriate pace.

Students, throughout the lesson, are highly intellectually engaged in significant learning, and make material contributions to the activities, student groupings, and materials. The lesson is adapted as necessary to the needs of individuals, and the structure and pacing allow for student reflection and closure.

Component 3d: Using Assessment in Instruction INTASC St 6

Assessment is not used in instruction, either through monitoring of progress by the teacher or students, or through feedback to students. Students are

Assessment is occasionally used in instruction, through some monitoring of progress of learning by the teacher and/or students. Feedback to students is uneven,

Assessment is regularly used in instruction, through self-assessment by students, monitoring of progress of learning by the teacher and/or students, and high-quality feedback

Assessment is used in a sophisticated manner in instruction, through student involvement in establishing the assessment criteria, self-assessment by students, monitoring of progress by

61

unaware of the assessment criteria used to evaluate their work.

and students are aware of only some of the assessment criteria used to evaluate their work.

to students. Students are fully aware of the assessment criteria used to evaluate their work.

both students and teacher, and high-quality feedback to students from a variety of sources.

Component 3e: Demonstrating Flexibility and Responsiveness INTASC St 8

The teacher adheres to the instruction plan, even when a change would improve the lesson or address students' lack of interest. The teacher brushes aside student questions; when students experience difficulty, the teacher blames the students or their home environment.

The teacher attempts to modify the lesson when needed and to respond to student questions, with moderate success. The teacher accepts responsibility for student success, but has only a limited repertoire of strategies to draw upon.

The teacher promotes the successful learning of all students, making adjustments as needed to instruction plans and accommodating student questions, needs, and interests.

The teacher seizes an opportunity to enhance learning, building on a spontaneous event or student interests. The teacher ensures the success of all students, using an extensive repertoire of instructional strategies.

DOMAIN 4: PROFESSIONAL RESPONSIBILITES

UNSATISFACTORY 1

BASIC 2

PROFICIENT 3

DISTINGUISHED 4

Component 4a: Reflecting on Teaching INTASC St 9

The teacher does not accurately assess the effectiveness of the lesson and has no ideas about how the lesson could be improved.

The teacher provides a partially accurate and objective description of the lesson but does not cite specific evidence. The teacher makes only general suggestions as to how the lesson might be improved.

The teacher provides an accurate and objective description of the lesson, citing specific evidence. The teacher makes some specific suggestions as to how the lesson might be improved.

The teacher's reflection on the lesson is thoughtful and accurate, citing specific evidence. The teacher draws on an extensive repertoire to suggest alternative strategies and predicts the likely success of each.

Component 4b: Maintaining Accurate Records INTASC St 9

The teacher's systems for maintaining both instructional and noninstructional records are either nonexistent or in disarray, resulting in errors and confusion.

The teacher's systems for maintaining both instructional and noninstructional records are rudimentary and only partially effective.

The teacher's systems for maintaining both instructional and noninstructional records are accurate, efficient, and effective.

The teacher's systems for maintaining both instructional and noninstructional records are accurate, efficient, and effective, and students contribute to its maintenance.

Component 4c: Communicating with Families INTASC St 10

The teacher's communication with families about the instructional program or about individual students is sporadic or culturally inappropriate. The teacher makes no attempt to engage families in the instructional program.

The teacher adheres to school procedures for communicating with families and makes modest attempts to engage families in the instructional program. But communications are not always appropriate to the cultures of those families.

The teacher communicates frequently with families and successfully engages them in the instructional program. Information to families about individual students is conveyed in a culturally appropriate manner.

The teacher's communication with families is frequent and sensitive to cultural traditions; students participate in the communication. The teacher successfully engages families in the instructional program, as appropriate.

Component 4d: Participating in a Professional Community INTASC St 10

The teacher avoids participating in a professional community or in school and district events and projects; relationships with colleagues are negative or self-serving.

The teacher becomes involved in the professional community and in school and district events and projects when specifically asked; relationships with colleagues are cordial.

The teacher participates actively in the professional community and in school and district events and projects, and maintains positive and productive relationships with colleagues.

The teacher makes a substantial contribution to the professional community and to school and district events and projects, and assumes a leadership role among the faculty.

Component 4e: Growing and Developing Professionally INTASC St 9

The teacher does not participate in professional development activities and makes no effort to share knowledge with colleagues. The teacher is resistant to feedback from supervisors or colleagues.

The teacher participates in professional development activities that are convenient or are required, and makes limited contributions to the profession. The teacher accepts, with some

The teacher seeks out opportunities for professional development based on an individual assessment of need and actively shares expertise with others. The teacher welcomes feedback from supervisors and colleagues.

The teacher actively pursues professional development opportunities and initiates activities to contribute to the profession. In addition, the teacher seeks feedback from supervisors and colleagues.

62

reluctance, feedback from supervisors and colleagues.

Component 4f: Showing Professionalism INTASC St 9, 10

The teacher has little sense of ethics and professionalism and contributes to practices that are self-serving or harmful to students. The teacher fails to comply with school and district regulations and time lines.

The teacher is honest and well intentioned in serving students and contributing to decisions in the school, but the teacher's attempts to serve students are limited. The teacher complies minimally with school and district regulations, doing just enough to get by.

The teacher displays a high level of ethics and professionalism in dealings with both students and colleagues and complies fully and voluntarily with school and district regulations.

The teacher is proactive and assumes a leadership role in making sure that school practices and procedures ensure that all students, particularly those traditionally underserved, are honored in the school. The teacher displays the highest standards of ethical conduct and takes a leadership role in seeing that colleagues comply with school and district regulations.

Table 1.10 Internship Observation by Program

2016-17 Internship Observation – University Supervisor

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

MED

DOMAIN 1: PLANNING AND PREPARATION

Component 1a: Demonstrating Knowledge of Content and Pedagogy 2 7 9 3.78 0.44

Component 1b: Demonstrating Knowledge of Students 3 6 9 3.67 0.50

Component 1c: Setting Instructional Outcomes 3 6 9 3.67 0.50

Domain 1d: Demonstrating Knowledge of Resources 3 6 9 3.67 0.50

Domain 1e: Designing Coherent Instruction 3 6 9 3.67 0.50

Domain 1f: Designing Student Assessments 2 7 9 3.78 0.44

DOMAIN 2: THE CLASSROOM ENVIRONMENT

Component 2a: Creating an Environment of Respect and Rapport 3 6 9 3.67 0.50

63

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

Component 2b: Establishing a Culture for Learning 3 6 9 3.67 0.50

Component 2c: Managing Classroom Procedures 3 6 9 3.67 0.50

Component 2d: Managing Student Behavior 1 2 6 9

Component 2e: Organizing Physical Space 3 6 9 3.67 0.50

DOMAIN 3: INSTRUCTION

Component 3a: Communicating with Students 2 7 9 3.78 0.44

Component 3b: Using Questioning and Discussion Techniques 1 2 6 9 3.56 0.73

Component 3c: Engaging Students in Learning 3 6 9 3.67 0.50

Component 3d: Using Assessment in Instruction 3 6 9 3.67 0.50

Component 3e: Demonstrating Flexibility and Responsiveness 3 6 9 3.67 0.50

DOMAIN 4: PROFESSIONAL RESPONSIBILITES

Component 4a: Reflecting on Teaching 1 2 6 9 3.56 0.73

Component 4b: Maintaining Accurate Records 2 1 6 9 3.44 0.88

Component 4c: Communicating with Families 1 2 6 9 3.56 0.73

Component 4d: Participating in a Professional Community 1 2 6 9 3.56 0.73

Component 4e: Growing and Developing Professionally 1 2 6 9 3.56 0.73

Component 4f: Showing Professionalism 1 2 6 9 3.56 0.73

2015-16 Internship Observation – University Supervisor

64

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

MED

DOMAIN 1: PLANNING AND PREPARATION

Component 1a: Demonstrating Knowledge of Content and Pedagogy 6 4 10 3.40 0.52

Component 1b: Demonstrating Knowledge of Students 1 6 3 10 3.20 0.63

Component 1c: Setting Instructional Outcomes 1 6 3 10 3.20 0.63

Domain 1d: Demonstrating Knowledge of Resources 1 4 5 10 3.40 0.70

Domain 1e: Designing Coherent Instruction 1 6 3 10 3.20 0.63

Domain 1f: Designing Student Assessments 1 6 3 10 3.20 0.63

DOMAIN 2: THE CLASSROOM ENVIRONMENT

Component 2a: Creating an Environment of Respect and Rapport 1 6 3 10 3.20 0.63

Component 2b: Establishing a Culture for Learning 6 4 10 3.40 0.52

Component 2c: Managing Classroom Procedures 7 3 10 3.30 0.48

Component 2d: Managing Student Behavior 1 6 3 10 3.20 0.63

Component 2e: Organizing Physical Space 1 6 3 10 3.20 0.63

DOMAIN 3: INSTRUCTION

Component 3a: Communicating with Students 6 4 10 3.40 0.52

Component 3b: Using Questioning and Discussion Techniques 1 6 3 10 3.20 0.63

Component 3c: Engaging Students in Learning 6 4 10 3.40 0.52

Component 3d: Using Assessment in Instruction 1 5 4 10 3.30 0.67

Component 3e: Demonstrating Flexibility and Responsiveness 1 5 4 10 3.30 0.67

DOMAIN 4: PROFESSIONAL RESPONSIBILITES

65

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

Component 4a: Reflecting on Teaching 1 6 3 10 3.20 0.63

Component 4b: Maintaining Accurate Records 1 6 3 10 3.20 0.63

Component 4c: Communicating with Families 1 6 3 10 3.20 0.63

Component 4d: Participating in a Professional Community 1 5 4 10 3.30 0.67

Component 4e: Growing and Developing Professionally 1 6 3 10 3.20 0.63

Component 4f: Showing Professionalism 7 3 10 3.30 0.48

2014-15 Internship Observation – University Supervisor

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

MED

DOMAIN 1: PLANNING AND PREPARATION

Component 1a: Demonstrating Knowledge of Content and Pedagogy 5 19 24 3.79 0.41

Component 1b: Demonstrating Knowledge of Students 1 4 19 24 3.75 0.53

Component 1c: Setting Instructional Outcomes 2 3 19 24 3.71 0.62

Domain 1d: Demonstrating Knowledge of Resources 3 2 19 24 3.67 0.70

Domain 1e: Designing Coherent Instruction 5 19 24 3.79 0.41

Domain 1f: Designing Student Assessments 5 19 24 3.79 0.41

DOMAIN 2: THE CLASSROOM ENVIRONMENT Component 2a: Creating an Environment of

Respect and Rapport 1 4 19 24 3.75 0.53

Component 2b: Establishing a Culture for Learning 5 19 24 3.79 0.41

66

EDUC MED Degree Unsatisfactory 1

Basic 2

Proficient 3

Distinguished 4 N Mean SD

Component 2c: Managing Classroom Procedures 1 4 19 24 3.75 0.53

Component 2d: Managing Student Behavior 2 3 19 24 3.71 0.62

Component 2e: Organizing Physical Space 3 2 19 24 3.67 0.70

DOMAIN 3: INSTRUCTION Component 3a: Communicating with Students 5 19 24 3.79 0.41

Component 3b: Using Questioning and Discussion Techniques 1 5 18 24 3.71 0.55

Component 3c: Engaging Students in Learning 5 19 24 3.79 0.41

Component 3d: Using Assessment in Instruction 6 18 24 3.75 0.44

Component 3e: Demonstrating Flexibility and Responsiveness 2 4 18 24 3.67 0.64

DOMAIN 4: PROFESSIONAL RESPONSIBILITES

Component 4a: Reflecting on Teaching 4 3 17 24 3.54 0.78 Component 4b: Maintaining Accurate Records 4 4 16 24 3.50 0.78

Component 4c: Communicating with Families 5 2 17 24 3.50 0.83

Component 4d: Participating in a Professional Community 3 4 17 24 3.58 0.72

Component 4e: Growing and Developing Professionally 1 6 17 24 3.67 0.56

Component 4f: Showing Professionalism 1 6 17 24 3.67 0.56

67