edTPA Evidence 3.A 3.G EPP - fs26. · PDF fileedTPA Evidence . The edTPA evidence ... EPP core...

edTPA Evidence The edTPA evidence begins with the CIP matrix and timeline followed up with both fall 2013 and spring 2014 summaries and data reports. Continuous Improvement Plan

3.A Yearly Goals &

Objectives

3.G Implementation

Year

Candidate Requirements

Review Process Candidate Accountability

3.D EPP

Activities/Initiatives

3.F Human and Capital Resources

Continuous Improvement Data

Goal 1: Phase out of 6 NCDPI electronic evidences.

2012-2013

Objective 1: Complete review of pipeline candidate evidences

Fall 2012 All candidates were required to submit 6

evidences.

EPP Faculty reviewed and approved by program.

High Stakes: Candidates required to successfully complete evidences to be recommended for a teaching license.

Awaited state-level process for submission and program review

Human: Time required to conduct reviews by EPP faculty, KMA, OPE Capital: None

Our EPP was asked to house and analyze the state-level electronic evidence program reviews. OPE was one of 3 official trainers for the state-level pilot. Pilot review by made it clear that the capacity to review all candidate evidence throughout NC did not exist. While our candidates were successful, inconsistencies in training and evaluation made cross-program (Unit) analysis difficult and impacted reliability and validity.

Objective 2: Complete transition plan for remaining candidates

Spring 2013 All candidates were required to submit 6

evidences.

EPP Faculty reviewed and approved by program.

High Stakes: Candidates required to successfully complete to be recommended for a license.

EPP engaged in initial edTPA consortium meetings and training. KMA and OPE worked with programs to determine which programs and candidates would begin

Human: EPP developed a core edTPA group to begin transition planning EPP core group attended consortium training

The Core edTPA team created a transition document to share with EPP programs and faculty. An edTPA Moodle site was created to house the transition documents in addition to training materials and edTPA handbooks for

edTPA implementation in Fall 2013 Elementary MAT volunteered for pre-pilot

Financial: The EPP redesigned SAGE to accommodate edTPA submission (approximately $25,000)

faculty. (See edTPA Evidence)

Goal 2: Develop a plan for full edTPA implementation

2013 - 2016

Objective 1: Provide additional support for faculty implementation of edTPA. Objective 2: Provide support for candidate use of edTPA.

Fall 2013 Programs volunteered to participate in the pilot edTPA administration (n=45). All other programs completed 6

evidences.

EPP faculty, University Supervisors, and Doctoral Students reviewed edTPA artifacts developed by candidates. Each pilot edTPA portfolio was scored by 2 independent raters on the SCALE 3-point local evaluation rubrics.

Low Stakes: Candidates were required to complete edTPA artifacts and successfully student teach.

Introduced and trained faculty on the edTPA process related to the transition plan. Hired an edTPA manager to support faculty and candidates. Trained faculty and staff on SCALE local evaluation rubric. Provided Candidate support for edTPA through workshops, website, and online resources.

Scored each fall completer’s edTPA portfolio with 2 independent reviewers.

Human & Financial: EPP committed resources for an edTPA manager. ($16,000 plus tuition waiver) edTPA state consortium committed resources for faculty and staff to attend local and national edTPA training. (Approximately $10,000) Financial: EPP provided resources to purchase 100 digital cameras for candidates. (Approximately $30,000) Human: EPP provided 2 half day SCALE local scoring rubric trainings for EPP faculty and University Supervisors EPP faculty and University Supervisors volunteered to score edTPA portfolios

Data on the pilot was collected and analyzed. (See edTPA pilot evidence) The edTPA manager conducted 4 candidate training sessions. The edTPA manager created a faculty electronic warehouse (moodle) to store all relevant documents, templates, and meeting information. (accessible on site) The edTPA manager created a candidate google site to store all handbooks, templates, webinars, and training information. (See edTPA evidence) EPP analyzed fall data and created program reports based on rubrics. (See edTPA evidence)

Objective 3: Implement and

Spring 2014 All Spring completers

Our EPP used current P-12

Low Stakes: Candidates were

Trained P-12 teachers and doctoral students

Human:

EPP provided rubric analysis by program in SnapShot

Evaluate edTPA artifacts across all programs.

required to complete edTPA artifacts. (n=220)

teachers and doctoral students with teaching experience for edTPA pilot portfolio review. Each pilot portfolio was scored on the SCALE 3-point local evaluation rubrics.

required to complete edTPA and successfully student teach.

on SCALE local evaluation rubric. Provided Candidate support for edTPA through workshops, edTPA Google site, and online resources.

Scored each Spring completer’s edTPA portfolio.

EPP provided 1 half day SCALE local scoring rubric training for P-12 teachers, doctoral students, and University Supervisors Human & Financial: EPP provided resources to pay P-12 teachers and doctoral students to score spring edTPA portfolios ($25K)

Objective 4: Transition from local scored (3 point) to Pearson scored (5 point) edTPA portfolios.

Fall 2014 All completers will submit edTPA portfolios to Pearson for scoring.

All completer edTPA portfolios will be scored by Pearson.

Moderate Stakes: Candidates are required to complete, higher threshold for quality; Resubmission is required for candidates not meeting program specified criteria.

Provide Candidate support for edTPA through workshops, website, and online resources.

Financial: EPP will provide funding for the Pearson Scoring (Approximately $12k)

EPP will provide results from Pearson to programs through the SnapShot

Objective 5: Conduct review of 2014-15 data to determine EPP cut score.

Spring 2015 All completers will submit edTPA portfolios to Pearson for scoring.

All completer edTPA portfolios will be scored by Pearson. EPP will conduct an analysis of completer scores along with national data to determine an appropriate cut score.

Moderate Stakes: Candidates are required to complete, higher threshold for quality; Resubmission is required for candidates not meeting program specified criteria.

Provide Candidate support for edTPA through workshops, website, and online resources.

Financial: EPP will provide funding for the Pearson Scoring ($60k) Human: KMA and OPE will work with programs to create cut score policy KMA and OPE will work with programs to determine candidate remediation procedures

EPP will provide results from Pearson to programs through the SnapShot

Fall2013PilotSummary

The Unit implemented edTPA for all graduating teacher candidates in Fall 2013.

Forty-two candidates were required to complete an edTPA portfolio in lieu of state

evidences for this pilot. Each candidates’ edTPA portfolio was assigned two raters and

scored according to the three point Local Rubric provided by SCALE. Each rater was

required to complete a day long rubric training in order to be qualified to review. Most

fall reviewers were faculty and university supervisors. An inter-rater reliability analysis

using Cohen’s Kappa was performed to determine consistency across raters. Two

reliability tests were conducted, and absolute score and a pass/fail score. The absolute

Kappa statistic determined the degree to which raters agreed on ratings for each of the 3

levels of scores. The pass/fail Kappa statistic determined the degree to which raters

agreed on passing (achieving level 2 or 3 for a standard), or failing (achieving level 1 for

a standard). Both calculations showed no greater than chance reliabilities for both the

pass/fail as well as the overall Kappa. The small sample size may be a factor in the

analysis. Therefore, percentages were derived from the total number of reviewers on

each rubric score to determine the overall decision of quality given by reviewers. The

following is a summary of these percentages for each department and each rubric.

Appendix A has a Unit summary table as well as an example of a program detail report.

Overall, Secondary Social Studies portfolios scored the highest percentage rate

for all 15 of the edTPA rubrics. Within these portfolios teacher candidates scored a 100%

on Rubrics 4, 6, 9, and 11. The lowest scoring rubric within this department were rubrics

12, and 15. The Special Education department candidates achieved an 81% pass rate for

all rubrics combined. Candidates achieved a 100% on rubric 11. Achievement was lowest

in the Special Education department for rubrics 9, and 15. Elementary Education

candidates achieved an 80% for their edTPA portfolios as a whole. Achievement for

rubric 7 was the highest at 96% with rubrics 2 and 13 having the lowest achievement of

67%. English as an Additional Language department achieved a 78% on their edTPA

portfolios as a whole. Rubrics 4, and 15 constituted the highest achievement at 100%.

English as an Additional Language portfolios showed achievement difficulties in rubrics

6 (50%), 7 (50%), 8 (67%), 9 (42%), and 10 (58%). Many of these rubrics scored video

excerpts that were unavailable, incorrectly uploaded, or unrelated to the rubric prompt for

this department. Finally, the Secondary English department portfolio (2 raters for 1

portfolio), showed an achievement of 77% across all rubrics. A 100% achievement rating

was given for rubrics 1, 2, 3, 4, 5, 6, 8, 10, and 15.

Each rubric was given an overall percentage to determine the highest achievement

areas and the lowest achievement areas as a college. According to raters, Rubric 4 was

achieved by the most portfolios (92%). Rubrics 1, and 3 showed achievement of 87%.

Raters determined that the portfolios were weakest on achieving the goals of rubrics 7

(71%), 9 (69%), 13 (66%), and 14 (71%).

Each program was provided with a detailed summary of their rubric scores.

Included in the report were percent at each level of each rubric as well as what specific

“look for” items were included or omitted from the portfolio. The Unit compiled and

included this information in the program reports to aid programs in program

improvement.

Overall, the Unit saw three areas where support is needed. First, candidates

struggled with some of the technology requirements. Several video clips were not able to

be viewed, had corrupt links, or were of poor quality. The Unit is providing addition

technology training in digital media, compressing video files, and camera use. In

addition, an assignment was added to the ED 312 (assessment course) for students to

practice recording and uploading small segments of instruction. The Unit has purchased

120 small video camera for check out in METRC. Candidates can check out equipment

free of charge. Second, area of support is use of academic language. Rubric data

revealed candidates struggle with academic language portions of the rubric. The Unit has

added more discussion and practice with academic language in the ED 204 course. This

course is required of all candidates. The spring 2014 sections of ED 204 included the

additional support so we should see an improvement in this area for the 2016 cohort. The

third area of improvement focuses around timing issues and logistics. The Unit provided

a timeline for programs to have students complete and submit edTPA documents. The

2013-2014 academic year was a low stakes year for candidates. Candidates were

required to submit edTPA portfolios but results were not used for graduation or licensure

purposes. Scoring for 2013-2014 was done using the local scoring rubric and candidates

were not provided rubric scores. Moving forward, academic year 2014-15, all edTPA

portfolios will be submitted to Pearson for full scoring. Although this academic year is

still low stakes, candidates must submit documents earlier than previous years. The Unit

is working with a subgroup of program coordinators to draft a semester timeline for

candidates and programs so all materials are submitted before the unit deadline. Program

requested more help with timeline creation and implementation. The edTPA timeline

actually is impacted by the student teaching year-long placement process so both

timelines are being done simultaneously to ensure a smooth transition.

The following table is the Program Summary by Rubric for the Fall 2013 administration.

Ag Ed Bus Ed ESL MAT: Elem Ed Math Mid LA

S1: Passing 93% 60% 87% 90% 87% 100%Level 1 7% 40% 13% 11% 13% 0%Level 2 29% 0% 60% 28% 31% 54%Level 3 64% 60% 27% 62% 56% 46%









Spring 2014 Program Summary of Pass Rates by Rubric

Ag Ed Bus Ed ELSMAT: Elem Ed Math Mid LA



S12: Passing 86% 100% 74% 86% 62% 85%Level 1 14% 15%Level 2 43% 80% 7% 14% 6% 62%Level 3 43% 20% 67% 49% 56% 23%

27% 37% 38%S13: Passing 79% 100% 87% 54% 88% 77%Level 1 21% 0% 13% 30% 13% 23%Level 2 50% 80% 60% 31% 69% 54%Level 3 29% 20% 27% 23% 19% 23%



Pass Rate: 85.333333 82.666667 75.333333 85.4 79.333333 84.6

Total Students Scored 14 5 15 47 16 13

Mid Sci Mid SS Sec Eng Sec Sci Sec SS Spec Ed Tech and Eng

91% 90% 90% 0% 94% 80% 85%9% 10% 10% 5% 20% 14%

26% 35% 42% 68% 50% 21%65% 55% 48% 26% 30% 64%

91% 90% 90% 0% 83% 80% 86%9% 10% 10% 17% 20% 14%

39% 60% 53% 50% 40% 36%52% 30% 37% 33% 40% 50%

87% 75% 70% 0% 94% 90% 78%13% 25% 30% 6% `0 21%57% 55% 43% 61% 70% 64%30% 20% 27% 33% 20% 14%

82% 65% 83% 0% 95% 80% 72%17% 35% 17% 6% 20% 29%65% 50% 53% 56% 40% 43%17% 15% 30% 39% 40% 29%

82% 90% 80% 0% 89% 70% 86%17% 10% 20% 11% 30% 14%17% 45% 47% 56% 30% 36%65% 45% 33% 33% 40% 50%

105% 95% 90% 0% 94% 90% 93%4% 5% 10% 6% 10% 7%

79% 60% 63% 50% 50% 43%26% 35% 27% 44% 40% 50%

87% 85% 87% 0% 88% 80% 93%13% 15% 13% 11% 20% 7%48% 60% 57% 44% 50% 79%39% 25% 30% 44% 30% 14%

78% 85% 84% 0% 89% 90% 86%22% 15% 17% 11% 10% 14%39% 60% 57% 56% 70% 57%39% 25% 27% 33% 20% 29%

87% 80% 80% 0% 89% 70% 92%13% 20% 20% 11% 30% 7%48% 55% 47% 72% 20% 21%39% 25% 33% 17% 50% 71%

Mid Sci Mid SS Sec Eng Sec Sci Sec SS Spec EdTech and Eng

69% 85% 70% 0% 95% 80% 64%30% 15% 30% 6% 20% 36%43% 50% 50% 67% 50% 50%26% 35% 20% 28% 30% 14%

91% 95% 86% 0% 100% 100% 86%9% 5% 13% 0% 0% 14%

52% 55% 43% 56% 80% 43%39% 40% 43% 44% 20% 43%

65% 90% 84% 0% 89% 90% 72%35% 10% 17% 11% 10% 29%43% 60% 57% 56% 60% 43%22% 30% 27% 33% 30% 29%

69% 55% 54% 0% 83% 90% 72%30% 45% 47% 17% 10% 29%39% 40% 37% 61% 60% 43%30% 15% 17% 22% 30% 29%

91% 65% 70% 0% 89% 80% 64%9% 30% 30% 11% 20% 36%

52% 50% 50% 39% 60% 14%39% 15% 20% 50% 20% 50%

65% 85% 83% 0% 89% 70% 78%35% 15% 17% 11% 30% 21%48% 55% 60% 50% 40% 57%17% 30% 23% 39% 30% 21%

82.666667 82 80.066667 0 90.666667 82.666667 80.466667

23 20 31 19 10 14

UG Elem World Lang

88% 100%13% 0%53% 33%35% 67%

76% 100%24% 0%60% 33%16% 67%

74% 100%25% 0%56% 33%18% 67%

85% 100%15% 0%67% 50%18% 50%

78% 100%22% 0%58% 67%20% 33%

91% 67%9% 33%

67% 50%24% 17%

97% 66%4% 33%

82% 33%15% 33%

77% 67%24% 33%64% 17%13% 50%

89% 83%11% 17%58% 50%31% 33%

UG Elem World Lang

76% 100%24% 0%56% 33%20% 67%

89% 83%11% 17%53% 50%36% 33%

80% 100%20% 0%64% 83%16% 17%

71% 83%29% 17%53% 50%18% 33%

78% N/A22%56%22%

85% N/A15%65%20%

82.266667 88.384615

55 6

S3: Using Knowledge of

Students to Inform Teaching

and Learning‐ How does the

candidate use knowledge of

his/her students to justify

instructional plans?

89.00%

superficial descriptions of classroom students' prior

learning

80%

Level 1

included:

superficial descriptions of classroom students' lived

experiences.60%

Emerging

Performance

pervasive negative portrayals of students'

backgrounds, educational experiences or

family/community characteristics.

0

n = 5 Level 1 did

NOT include

from Level

2:

concrete and specific connections between tasks

and prior learning. 80%

at least surface level of discussion of theory or

research.80%

Level 2

included:

concrete and specific connections between tasks

and prior learning.95%

Proficient

Performance

at least surface level of discussion of theory or

research.70%

n = 20

Level 2 did

NOT include

from Level 3

concrete, specific connections between tasks and

prior learning.10%

grounded discussion of theory or research.100%

S6: Learning Environment‐ How

does the candidate

demonstrate a positive literacy

learning environment that

supports students' engagement

in learning?

84.00%Emerging

Performance

disrespectful interactions. 14%

Level 1

included: disruptive behaviors. 29%

n = 7 controlling or directive environment. 14%

minimal support for learning goals. 29%

Level 1 did

NOT include

from Level

2:the majority of assessments providing evidence of

subject specific understandings. 71%

IEP/504 requirements for adaptations/modifications

were addressed. 86%

Level 2

included:

Proficient

Performancethe majority of assessments providing evidence of

subject specific understandings. 100%

IEP/504 requirements for adaptations/modifications

were addressed. 26%

n = 19

assessments that provided evidence of the full

range of subject specific understandings. 26%

Level 2 did

NOT include

from Level 3

assessments that were used in each lesson. 5%

S10: Analyzing Teaching

Effectiveness‐ How does the

candidate use evidence to

evlauate and change teaching

practice to meet students'

varied learning needs?

89.00%Emerging

Performance

Level 1

included:

proposed changes addressed problems with

classroom student behavior and how to "fix" it. 14%

n = 5

proposed changes addressed gaps in the whole of

class learning/understanding. 80%

Level 1 did

NOT include

from Level

2:proposed changes re‐reengaged students in new

revised or additional task. 60%proposed changes included surface level discussion

of research or theory. 80%

Level 2

included:

proposed changes addressed gaps in the whole of

class learning/understanding. 75%

Proficient

Performanceproposed changes re‐engaged students in new

revised or additional task. 67%

proposed changes included surface level discussion

of research or theory. 71%

n = 24

Level 2 did

NOT include

from Level 3 proposed changes that were concrete, specific and

elaborated. 30%

proposed changes addressed gaps in student

learning for different students in different ways. 54%

edTPA Evidence 3.A 3.G EPP - fs26. · PDF fileedTPA Evidence . The edTPA evidence ... EPP core...

Documents

Transcript of edTPA Evidence 3.A 3.G EPP - fs26. · PDF fileedTPA Evidence . The edTPA evidence ... EPP core...