Secondary Student Data Self Analysis Secondary Student Assessment...

Secondary Student Data Self Analysis

Secondary Student Assessment Data Self Analysis

ADDIE Instructional Unit

Kevin Spurgin

Instructional Design

EDTC 6321-01

July 27, 2013

Secondary Student Data Self Analysis 2

Analysis Summary

Performance Assessment

Dr. Rodriguez, a professor at UTB, requested an instructional unit of my choosing. This

proposal is in response to that request. Students at A+ Academy, a public charter school in

southeast Dallas, are struggling to achieve mastery both on local and state assessments. After

conducting research, one identified strategy for obtaining mastery is the student’s ability to

analyze his or her own assessment data in euphoria! to identify strengths and weaknesses and

work toward mastery of the content in each area of weakness. A sample population of students

was interviewed and it was found that less than 20% were able to successfully analyze his or her

assessment data. The goal is to increase the percentage of students who can successfully analyze

assessment data from less than 20% to 100%. This gap in performance is due primarily to the

fact that students have not been properly trained in the analysis of assessment data. Results of the

interviews along with other relevant data are presented in the following Performance Assessment

chart.


Actual

Student

Performance

Desired

Student

Performance

Primary

Cause

% of Total

Performance Gap

43% were able to

identify and locate

assessment data

Locate both state and

local assessment data

in each content area

1. References to assessment data are very limited in the classroom

2. Lack of clarity regarding the difference between local and state assessments

11%

21% were able to

interpret data by

matching assessment

questions to

prescribed objectives

Interpret data by

matching questions to


1. Rarely is data

analysis conducted 14%

16% analyzed data by

determining which

objectives had not yet

been mastered

Analyze data to

determine which

objectives have not

yet been mastered

1. Complete list of objectives is lengthy and specific objectives are difficult to locate

2. Many Assessment questions are coded for more than one objective

39%


10% used assessment

data analysis to set

personal goals for

achieving mastery

Use assessment data

to set personal goals

for achieving mastery

of prescribed

objectives

1. Lack of understanding regarding how to set achievable goals

2. Failure to prioritize

36%

Purpose

Statement

The purpose of this training is to provide secondary students with the

skills to self evaluate and analyze assessment data in eudphoria! to

achieve mastery in each prescribed objective.

Instructional Goals

1. Locate and identify both local and state assessment data in eduphoria!

2. Interpret data by matching assessment questions to prescribed objectives.

3. Analyze assessment data to determine which objectives have not yet been mastered.

4. Utilize assessment data to create personal goals for achieving mastery of prescribed

objectives.

Learner Analysis

Group Identifications

The learners in the group will be secondary students at A+ Academy, a charter school in

southeast Dallas.


General Characteristics

The average age of these students is 15 and the age range is 12 – 18. Most students in this learner

group learn best visually and when working with computers. Also, these students tend to work

best individually. Students from the intended learner group are seventh - twelfth graders at a

Title One charter school in southeast Dallas. Over 92% of the students were born in Texas but

Spanish is the primary language spoken in the home. The parents were born and raised in

Mexico and speak some English but often this student is responsible for translating important

information for the parents.

Hard work (manual labor) and family are two of the core values instilled in these

students. Education is important but not a top priority. Higher education is not a serious

consideration.

Vocabulary is a consistent struggle for these students, especially academic and technical

vocabulary. A reading level assessment for these students, prior to beginning instruction, would

be very helpful in tailoring the instruction.

Number of Students/Trainees

There will be 510 students in the learner group. This is based on 85 students per grade in

grades 7-12. The number of learners per class is 20-25 and the number of learner groups is also

20-25.

Location of Students/Trainees

All training will occur in classrooms at A+ Academy. Students will receive computer-

based training during their regularly scheduled classes with classroom teachers serving as

monitors.


Experience Levels

The students are tech savvy and have a strong command of computer skills. These

students may need help with some vocabulary but the definitions of troublesome terms could be

found online during the training. Each student will have at least completed the sixth grade.

Attitudes

Motivation is an issue for the intended learner. Making good grades and long-term goals

(i.e. top 10% of high school graduating class, college acceptance, earning scholarships, preparing

for SATs and ACTs, and career aspirations) do not motivate the student in the classroom. The

student’s primary motivation for coming to school is socialization, not learning. The student

realizes he or she is in a school of choice and values that to a certain degree but is primarily in

the school because his or her parents want a safe environment for their child.

Immediate rewards are one effective method of motivation. Something as simple as

granting a privilege such as listening to music during class or giving candy for a correct response

can be helpful tools to encourage this student. Outside of school, the student is very interested in

soccer and plays on a select team. References to soccer, including famous professional players,

fitness, skill, determination, and even projects that incorporate research in the field all entice this

student to engage and learn. Another outside interest of this student is video games. When

possible, this student plays with gamers around the world. Anything gaming related immediately

peaks the interest of this student.

Prerequisites

Students should have a strong command of basic computer skills. Students should also

be able to read simple graphs and tables.


Required Resources

Content Resources

1. Data from state assessments for each student is available in Eduphoria!

2. Data from local assessments for each student is available in Eduphoria!

3. Sample data analysis summaries are available for each student

4. Lead4ward documents to simplify data analysis are available for each student

5. A list of prescribed student objectives (TEKS) for the appropriate content area is

available for each student

Technology Resources

1. A computer and internet connection for each student

2. Note that coordination and scheduling will be required so teachers can share the 3 class

sets of laptop computers and the two computer labs

Instructional Facilities

1. Classrooms with adequate amenities (including technology) have been identified

2. As many as 25 learners can be trained in each of the identified classrooms

3. These classrooms are available during the students regular class schedule

Human Resources

1. Instructional design team members (see list below) have been identified to assist with

course development

2. Classroom teachers for each of the 20-25 learner groups have been identified to serve as

monitors


Potential Delivery Systems

Computer based training

Estimated time: One 50-minute session

Cost Estimate: the cost for this training will be zero because employees of A+ Academy will

develop and implement this instructional unit and use facilities, equipment, and materials

provided by the school. The employees involved will implement this instructional unit as part of

their assigned contractual duties during the regular school day.

Project Management Plan

The Instructional Design Members

1. Project Manager: “Oversees the full life cycle of the project, interfaces between internal

client and eLearning team, schedules deliverables, ensures the team has the information it

needs to get the job done” (Malamed)

2. Instructional Designer/Writer: “Uses instructional systems design and adult learning

theory to organize, design and write the course” (Malamed)

3. Editor: “Improves writing, proofreading” (Malamed)

4. Graphic Designer: “Creates graphics and animations, designs the look and feel of the

course, enhances stock photos to fit project needs” (Malamed)

5. Media Specialist: “Produces and edits audio and video” (Malamed)

6. Authoring Specialist: “Assembles all the elements into a running course, adds

interactivity, ensures the course can interface with a Learning Management System if

required” (Malamed)


7. Tester: “Runs Quality Assurance checks by testing the course from a technical

perspective and ensuring it matches the storyboard” (Malamed)

8. Subject Matter Expert: Provides technical information for the training material and

ensures that all information is accurate and valid

Note: The Subject Matter Expert is a partner to the designer of the instructional unit.

Kevin Spurgin will serve as both the Instructional Designer and the Subject Matter

Expert.

Significant Constraints

This instructional unit will provide:

1. Computer based training module to accomplish agreed upon goals

2. An instructor for one 2 hour session to train classroom teachers how to use this computer

based module with their students

3. Handouts or links to handouts that correlate with and are necessary for successful

completion of the module

4. Assessment to determine if the module is successful

This instructional unit will not provide:

1. Facilities or equipment necessary for the implementation of the module

2. Personnel to monitor student groups during training

3. Ongoing support for the use of this module


Schedule Tasks

2013

June July August September October November December

All Projects

Analyze

Design

Develop

Implement

Evaluate


Design Brief

Task Inventory

Purpose Statement Instructional goal(s) The purpose of this training is to provide

secondary students with the skills to self

evaluate and analyze assessment data in

eduphoria! to achieve mastery in each

prescribed objective.

5. Locate and identify both local

and state assessment data in

eduphoria!

6. Interpret data by matching

assessment questions to

prescribed objectives.

7. Analyze assessment data to

determine which objectives

have not yet been mastered.

8. Utilize assessment data to

create personal goals for

achieving mastery of

prescribed objectives.


Purpose: This course will provide secondary students with the skills to self evaluate and analyze

assessment data in eduphoria! to achieve mastery in each prescribed objective.

NOTE: Prerequisite Knowledge and Skills are listed below the broken line.

Locate and identify both local and state

assessment data in eduphoria!

Click “Submit Report” and wait for

download

Click on the appropriate student group and student

assessment

Click on the Reports tab and select the

“Student Test Summary Report”

Log in to eduphoria! and click on the "Aware" icon

Utilize eduphoria! login and password and Basic computer

skills

Interpret data by matching assessment

questions to prescribed objectives

Record the selected assessment data in

the SLR

Download the “Student Learning Report” (SLR) at

http://lead4ward.com/

resources/#learning

Basic math skills for calculating percentages

Analyze assessment data to determine which

objectives have not yet been mastered

Answer the questions aligned

with the appropriate assessment

objectives in the SLR “Where are

my strengths?” and “Where can I

improve?”

Ability to identify strengths and

weaknesses based on data

Utilize assessment data to create personal goals for achieving mastery of


Answer the question aligned

with the appropriate assessment

objectives in the SLR “What actions do I need to take?”

Set an appropriate goal in the “My

Goal” column of the SLR

How to write an appropriate

achievable goal


Performance Objectives

1. Using eduphoria! the learner will locate both local and state assessment data for each

assessment taken.

2. Using the Lead4ward document titled “Student Learning Report” the learner will

match and record assessment data to prescribed objectives with no errors.

3. Given the appropriate assessment data the learner will analyze the data to determine

which prescribed objectives are strengths and which prescribed objectives are areas

that need improvement.

4. Utilizing the interpreted assessment data the learner will set personal goals and action

steps for achieving mastery of at least 80% of each prescribed objective.


Testing Methods

Task Objective Test Item Log in to eduphoria! and click on the "Aware" icon

Using eduphoria! the learner will locate both local and state assessment data for each assessment taken.

Have the student login to eduphoria! and access the “Aware” page

Click on the Reports tab and select the “Student Test Summary Report”


Have the student find and select the “Student Summary Report”

Click on the appropriate student group and student assessment


Ask the student to find her or his own assessment data for a given assessment

Click “Submit Report” and wait for download


Ask the student to download and print the “Student Test Summary Report”

Download the “Student Learning Report” (SLR) at http://lead4ward.com/resources/#learning

Using the Lead4ward document titled “Student Learning Report” the learner will match and record assessment data to prescribed objectives with no errors.

Have the student print the appropriate SLR for a given assessment

Record the selected assessment data in the SLR

Using the Lead4ward document titled “Student Learning Report” the learner will match and record assessment data to prescribed objectives with no errors.

Provide a sample set of assessment data for the learner and have the learner record the correct percentages in the correct blanks on the SLR

Answer the questions aligned with the appropriate assessment objectives in the SLR “Where are my strengths?” and “Where can I improve?”

Given the appropriate assessment data the learner will analyze the data to determine which prescribed objectives are strengths and which prescribed objectives are areas that need improvement.

Have student check to make sure he/she has answered one of the given questions for each prescribed objective

Set an appropriate goal in the “My Goal” column of the SLR

Utilizing the interpreted assessment data the learner will set personal goals and action steps for achieving mastery of at least 80% of each prescribed objective.

One-on-one teacher/student discussion will occur to ensure students are writing appropriate goals

Answer the question aligned with the appropriate assessment objectives in the SLR “What actions do I need to take?”

Utilizing the interpreted assessment data the learner will set personal goals and action steps for achieving mastery of at least 80% of each prescribed objective.

One-on-one teacher/student discussion will occur to ensure students are developing adequate plans of action


Testing Strategy

There are two keys to the testing strategy I would implement to determine if the instructional unit

is effective.

1. Are the students able to successfully complete the “Student Learning Report” (SLR)?

Because this document is the key to analyzing their own assessment data, students must

complete the SLR accurately for each given assessment. Giving sample data to students

and having them complete the SLR could assess this skill. If there are consistent errors

among students in a certain area, that would indicate that the instructional unit may need

editing to make sure students are properly trained in that area.

2. Are the goals and action plans written in a way that enable the students to effectively

measure their progress toward mastery of each objective. While the goal writing itself is

prerequisite knowledge and not part of the instructional unit, this area is critical to student

success. To measure this criterion, apply the S.M.A.R.T. goal model. Is the goal

Specific, Measurable, Attainable, Relevant, and Time Specific? In addition, the action

plans must be practical and realistic. If these two components are completed correctly

then students have a much greater chance of successfully mastering objectives that are

areas of weakness for them.

In determining if students have learned from the Instructional Unit, there is really only one

criteria to measure: “Are students progressively moving toward the mastery of prescribed

objectives over time?” As this practice of student data self analysis becomes a habit and is

conducted consistently, the data will show whether or not students are moving toward mastery -

over time. This is the best method of determining student learning for this instructional unit.


Learning Resources

Selected Media

This instructional unit includes the use of the online program eduphoria! and is therefore

computer based. As there is no other mode of accessing the assessment data produced by

eduphoria! the use of an internet connected computer is required. Another aspect of the

instruction is the use of Lead4ward’s “Student Learning Report”. This tool can be accessed only

through Lead4ward’s website and therefore necessitates the use of an Internet connected

computer as well.

An electronic device that may be required by some students is a calculator for calculating

percentages when assessing mastery of objectives during data analysis. The math required is

fairly simple but the use of a calculator will not only aid students who may have difficulty with

calculations, it will also ensure accuracy.

Lesson Plan

In the planning of the lesson for this instructional unit a 5E approach is taken. The 5 E’s

are Engagement, Exploration, Explanation, Elaboration, and Evaluation. These 5E’s encompass

all of Gagne’s Nine Events of Instruction and therefore should provide all the necessary tools for

learners to be successful.

This instructional unit focuses on secondary student assessment data self analysis and, of

course, the lesson plan leads learners through that process. In the engagement phase learners are

motivated to use data via their favorite national football (soccer) team. The exploration segment

guides students through a discussion and data activity surrounding a Ted Talk. Learners can


choose from two different Ted Talks and then create a visual data display from data they find in

every day life.

The explanation phase goes to the core of the instruction and gives students the

opportunity to both understand and practice using assessment data from eduphoria! At the

conclusion of this phase, learners actually perform the initial analysis of their data, which will

begin to bring meaning and value to the data.

During the elaboration phase learners analyze their data by answering a series of

questions that are designed to help students reflect on, evaluate and gauge their progress in

relation to the successful mastery of the prescribed content objectives.

Although learners are assessed throughout the instructional unit, the evaluation phase

includes the component that defines student mastery of the performance objectives. The

independent successful completion of the “Student Learning Report” is key to learner success.

The other components necessary for mastery are the writing of appropriate goals and action plans

as well as the progression of mastery over time in relation to the content area objectives being

evaluated.

All these components will provide learners with the tools and skills to effectively analyze

and utilize assessment data. Ultimately, this will enable learners to continuously, intentionally

and consciously move toward mastery in each subject content area.

Engagement

Ask students to name their favorite national football (soccer) team and follow up with

asking why they like that team.


Project the following objectives and select a student to read each one aloud. Pick a

second group of students to put these objectives in their own words.

5. Using eduphoria! the learner will locate both local and state assessment data for each

assessment taken.

6. Using the Lead4ward document titled “Student Learning Report” the learner will

match and record assessment data to prescribed objectives with no errors.

7. Given the appropriate assessment data the learner will analyze the data to determine

which prescribed objectives are strengths and which prescribed objectives are areas

that need improvement.

8. Utilizing the interpreted assessment data the learner will set personal goals and action

steps for achieving mastery of at least 80% of each prescribed objective.

Have students go to www.fifa.com and find their favorite national football team (men’s

or women’s). Ask students to list their chosen team and at least 5 data points about that team

including:

1. Based on the “Ranking History” column, the team’s current FIFA ranking

2. Based on the “Ranking Evolution Graph”, the year the team had its highest ranking

3. Based on the “Ranking Evolution Graph”, the year the team had its lowest ranking

4. Student choice

5. Student choice

Ask student to name at least 2 other instances in which data is used in their everyday

lives. If students have trouble thinking of examples prompt them with some of the


following ideas: grades, Google search “learning” your preferences, GPS monitoring

traffic patterns, sports, science experiments, and weather apps.

Ask students to discuss the following question with a partner: “Why might it be important

to recognize and utilize the data around you?” After discussing ask at least 3 students to

talk about what they discussed with the rest of the group. Provide feedback to help

students determine whether or not they are on the right track.

Exploration

Have students choose and watch either of the Ted Talks linked below in pairs. Both talks

emphasize the value and humanization of data in everyday life.

http://www.ted.com/talks/jer_thorp_make_data_more_human.html

http://www.ted.com/talks/david_mccandless_the_beauty_of_data_visualization.html

After viewing one of the talks, students should work on the following 2 items. These

items are application activities that address innovative data displays.

1. With your partner, create a visual data display from your everyday life (refer to

the ideas you generated earlier). The display you create should be imaginative.

Do not use a simple bar or line graph. Instead, use one of the visual data display

ideas from the Ted Talk you watched or a display you develop yourself.

2. Each pair should present and explain their data display to another pair. Ask

students to give meaningful feedback to their partner pair including if the visual

data display effectively communicated the information.


Explanation

Clarify to the learners that they will now use eduphoria! to locate and analyze assessment

data. Students should already have a password and login. Use projector/smartboard to walk

students through each step of the following process. Have them work each step on their own

computer as you demonstrate it on the screen. Monitor student progress by periodically moving

around the room to check screens for accuracy and answer questions. This presentation could

also be done ahead of time using screen casting software/video and students could watch it at

their own pace and back up when necessary.

1. Log in to eduphoria! at

https://aplus.schoolobjects.com/eduphoria_webcontrols/Login.aspx?ReturnUrl=%2fedup

horia_webcontrols%2fApplications.aspx and click on the "Aware" icon.

2. Click on the Reports tab and select the “Student Test Summary Report”.

3. Click on the appropriate student group and student assessment. For purposes of this

example/assignment, select the “Science Biology Unit 08: Genetics and Heredity 2012-

2013” data.

4. Click “Submit Report” and wait for download (this may take a few minutes).

5. Download and print the “Student Learning Report” (SLR) at

http://lead4ward.com/resources/#learning. In the “Snapshot Table” under the column

titled “Standard Recording Sheets” click on “Biology”. Again, selecting Biology is for

the purposes of this example/assignment and, of course, any grade level/content area can

be selected for additional data analysis. Please note that the SLR’s are specific to each

grade level/content area and are NOT interchangeable.


6. Record the selected assessment data into the SLR. In order to do this properly, learners

need to locate the “Student Expectations” section on their “Student Test Summary

Report”. Next, learners match each objective (i.e. Bio.6(A)) to the appropriate category

on the SLR. Before recording, the data must be converted to a percentage. This is

accomplished by converting the number correct verses the number of questions for that

objective to a percentage. That percentage will then be recorded in the “Test 1” column.

The columns labeled “Test 2” and “Test 3” will be used in subsequent data analysis.

Elaboration

Leaners need to make meaning of the data they have just recorded. In order for this data

to be valuable, it must be analyzed appropriately and personally. To accomplish this, students

will now address the following items regarding their recorded data.

1. Direct students to answer the questions aligned with the appropriate assessment

objectives in the SLR “Where are my strengths?” and “Where can I improve?”

2. Students should set an appropriate goal in the “My Goal” column of the SLR for each

objective. The goal to be set should be relative to the student’s current level of mastery.

For example, if a student is at 75% mastery then the goal should be set at about 95%.

However, if the student’s current level of mastery is 50% or lower then the goal should

be set around 70-75%. No goal should be set lower than 70%. Ultimately, every student

should achieve a minimum of 80% mastery for each assessed objective.

3. Refer students to the prior knowledge about S.M.A.R.T. goals and remind them that these

goals are Specific, Measurable, Attainable, Relevant, and Time Specific. Have students

write a S.M.A.R.T. goal for each applicable Reporting Category.


4. Have students answer the question aligned with the appropriate assessment objectives in

the SLR “What actions do I need to take?”

In order to better retain the information presented in this lesson, lead students to analyze the data

from a different assessment. Students can follow the same steps as prescribed in the

“Explanation” phase above with a new set of data. This will give them the opportunity to

practice and solidify the learning.

Evaluation

In assessing the learner’s mastery of the performance objectives at least three areas need

to be assessed including the SLR, goals with action plans, and a progression over time toward

mastery of the content objectives.

A correctly completed SLR is the primary indicator of the student’s performance. It is

best to check the SLR from the student’s independent practice instead of just the one used in the

example. The former will give the best indication of whether or not the learner is able to

successfully complete the performance objectives.

Check each student’s goals to ensure that they are Specific, Measurable, Attainable,

Relevant, and Time Specific. Also, conference with students to ascertain whether or not the

action plans are appropriate, practical and realistic.

Over time, the data will show whether or not students are moving toward mastery. This is

the best method of determining student learning for this instructional unit. If the student is

progressing toward mastery then the instruction has been successful.


Guidance for the Facilitator

As facilitator of this instructional unit, your job is to: a) oversee the overall flow of the unit; b)

guide and monitor the learners’ interaction with the media; c) conduct one-on-ones with students

to check for appropriate goals and action plans.

Description of the training process: This instructional unit focuses on secondary student assessment data self-analysis. Both local and

state assessment data will be accessed in eduphoria! Learners will be guided by both the

facilitator and their own analysis toward achieving the instructional goals and objectives of the

training consisting of the following four sections.

Locate and Identify Assessment Data: After studying data in contexts relevant to students,

learners will understand the value of data in their everyday lives. Learners will be guided through

eduphoria! to find their own assessment data and print appropriate reports for analysis.

Interpreting Data: Students will interact with the Lead4ward website in order to locate and

print content appropriate Student Learning Reports. For the selected content area, students will

record their own data verses the TEKS objective assessed by each question on a given

assessment instrument.

Analyze Data: In this section students begin to reflect and self assess by answering questions

about their data. The answers to these questions will enable students to appropriately begin to

analyze their data and determine areas of strength and weakness in relation to the TEKS

objectives.


Utilize Data: Learners will write specific goals and action plans in this section. These goals and

action plans will be approved by the facilitator and, over time, will guide the students toward

mastery of each objective.

Guidance for the Learner

Dear Student:

Welcome to this instructional unit on data self-analysis. My guess is that your own assessment

data is not your top priority but I’m sure that high school graduation and success thereafter is.

This unit will help you get there! After an activity that will help you see the value of data in

everyday life, you will begin analyzing your own test data. Why? To help you achieve mastery

of each objective for which you are tested. All of this is to help you attain those goals of high

school graduation and success thereafter. Listed below are the major topics you will encounter

during this training.

Locate and Identify Assessment Data: You will study some football (soccer) data in order to

understand the value of data in your everyday life. You will then be guided through eduphoria!

to find your own assessment data and print appropriate reports for analysis.

Interpreting Data: Here you will interact with the Lead4ward website in order to locate and

print content appropriate Student Learning Reports. For the selected content area, you will record

your own data versus the TEKS objective assessed by each question on a given test.

Analyze Data: In this section you will begin reflecting and self-assessing by answering

questions about your data. The answers to these questions will enable you to appropriately begin


to analyze your data and determine areas of strength and weakness in relation to the TEKS

objectives.

Utilize Data: At this point you will write specific goals and action plans. These goals and action

plans will be approved by the facilitator and, over time, will guide you toward mastery of each

objective.

Formative Revisions of the Instructional Unit (Formative Evaluation)

One-to-one Trial

During the one-to-one trial phase of the ADDIE process the instructional designer,

myself, will sit down with individuals of varying ability levels as each of them works

through the instructional unit with a facilitator. This trial will occur in a classroom similar

to the one the majority of students participating in the instruction will use. The computer

will also be similar. I will be looking for and soliciting feedback on the following:

1. Learner attitudes toward the instruction and its value

2. Any errors in the design

3. Whether or not the instructional unit can adequately accommodate learners of varying

ability levels

4. The amount of time needed for each section and activity

5. Mastery of performance objectives


Small Group Trial

In the small group trial I plan to collect data from a group of 20 students participating in

the instructional unit with a facilitator. Revisions to the unit will be based on data

collected from a series of questions including the following:

1. Is the amount of time for each section and activity adequate? (Intulogy)

2. Are learners engaged in the material? (Intulogy)

3. Is material too easy or too difficult? (Intulogy)

4. Do learners understand the instructions given for each activity? (Intulogy)

5. How well have learners achieved the course’s objectives?

6. Are the assessment tools adequate? (Intulogy)

7. What do learners think about the course?

• Field Trial

The field trial will be conducted with a class of students at A+ Academy with a

facilitator. This field trial is the final formative evaluation and will be conducted to

determine the following:

1. Can the instruction be utilized in the intended context? (Branch 2009)

2. Is the student’s performance adequate to move to the implementation phase? (Branch

2009)

3. Is the delivery system planned for the instruction strategies feasible? (Branch 2009)


Pilot Test Conduction

Data Collection

In this phase data are collected from the following sources:

1. Pilot instructor debriefings (Branch 2009)

2. Observation (Branch 2009)

3. Participant interviews (Branch 2009)

4. Embedded test scores (Branch 2009)

5. Post-test scores (Branch 2009)

Data analysis

In this stage data will be summarized and compiled into graphs that enable reviewers to

determine which objectives are being met and which are not. (Branch 2009)

Formative Revisions

Revisions determined necessary after data analysis must be made prior to implanting the

course. (Branch 2009)

Evaluation Team

The evaluation team including the instructional development group, an external review

panel, and a representative from management will generate a pilot test report. (Branch

2009)

Role of Management

Management will make the final decision regarding whether or not the course is

implemented. (Branch 2009)


Pilot Test Plan

Participant Description

The number of participants in this pilot test is 1 group of 25. All students in the pilot test

will be 9th grade students at A+ Academy.

Prerequisites

The learners should come to the pilot test with an eduphoria! login and password and

basic computer skills. The learners will also need to be able to calculate percentages,

identify strengths and weaknesses, and write appropriate S.M.A.R.T. goals.

Location: A+ Academy

Date: August 30, 2013

Time: 9:30 AM

Learning Environment

Technology Resources include a computer and Internet connection for each student.

Institutional resources include a classroom with adequate amenities (including

technology) and seating for 25 learners.

Facilitator Qualifications

Certified classroom teachers on staff at A+ Academy will serve as facilitators.

Measurement Plan

The following items will be measured during and after the pilot test:

1. Measure the amount of time learners need for each module and activity. (Intulogy)

A timekeeper will clock the amount of time each student spends on each activity.

2. Check learners' engagement with the material. (Intulogy)

Timekeepers will clock the amount of time students are off task.


3. Detect points where material may be too easy/too difficult. (Intulogy)

Note when students hurry through a section or are pushing to move at a faster pace.

Also note when students ask questions and review material more than once before

moving on.

4. Confirm that learners understand the instructions for activities and exercises.

(Intulogy)

Note when students ask questions about instructions or have difficulty following

directions.

5. Test how well learners achieve the course's stated learning objectives by the end of

the course. (Intulogy)

Review assessment data to determine whether or not students mastered the objectives.

6. Collect feedback from learners about the course. (Intulogy)

Ask students the following questions: Is this course valuable to you and if so, how?

What would you change about the course?

What did you like most about the course?

7. Locate points where the course should be revised (Intulogy)

Summarize all data collected and with the review team make this determination.

Evaluation Team Description

The evaluation team includes:

Instructional developer (Kevin Spurgin): Along with evaluation and finalizing the pilot

test report, this person will make the changes deemed necessary by the evaluation team

prior to implementation.


External review panel (Three teachers from various schools and content areas): These

three individuals will provide fresh perspective from outside the project. Valuable

objective feedback from this panel will be included in the pilot test report.

Representative from management (administrator): this person will give input for the pilot

test report as well as make the final decision about whether or not the instructional unit is

ready for implementation.

Revisions will need to be made based on the findings of the evaluation team.


Implementation Strategy

Prepare the Teacher

Identification

Facilitators of this instructional unit are:

a) Secondary teachers

b) Texas certified teachers

c) Employed by A+ Academy in Dallas, TX

d) Eduphoria! Aware certified from Region 10 Education Service Center

e) Experienced in using the Lead4ward Student Learning Report

f) Committed to data driven instruction

Schedule

A 90-minute training session for facilitators will take place in the cafeteria at A+ Academy in

Dallas, TX on Monday July 29, 2013 at 1:30PM (CDT).

The following materials/equipment will be available in the training room:

a) Computer with internet access for each participant

b) LCD projector

c) Projection screen

d) Student Learning Report handouts for each participant

Pilot Test Date: July 19, 2013

Initial Course Start Date: August 26, 2013

Train the Trainer

Trainers will receive the following 90-minute training prior to facilitating the course:


Introduction and the Performance Gap (5 minutes)

Students at A+ Academy, a public charter school in southeast Dallas, are struggling to

achieve mastery both on local and state assessments. The goal is to increase the percentage of

students who can successfully analyze assessment data from less than 20% to 100%. This gap in

performance is due primarily to the fact that students have not been properly trained in the

analysis of assessment data.

Acquiring Additional Expertise (55 minutes)

Facilitators will participate as students as the instructional designer leads the course. This

will enable facilitators to experience the material from the perspective of the students, ask

questions, personalize the material, and make recommendations for improvement.

“The instructional designer will describe the learner profile, discuss relevant material

from the Analyze Phase, and conduct a Q&A session where the teachers discuss strategies for

facilitating the newly developed instruction. Teachers review the Guidance for the Teacher

materials.” (Branch, 2009)

Practice Facilitating and Using Resources (30 minutes)

Each participant will be placed in a group with three other participants. Each participant

will take one of the four objectives to prepare a micro lesson. This micro lesson will be presented

to the rest of the group for feedback and critique. The feedback should include potential

challenges and how to manage those effectively.


Prepare the Student/Trainee

Identification

The learners in the group will be secondary students at A+ Academy, a charter school in

southeast Dallas.

General Characteristics

The average age of these students is 15 and the age range is 12 – 18. Most students in this

learner group learn best visually and when working with computers. Also, these students tend to

work best individually. Students from the intended learner group are seventh - twelfth graders at

a Title One charter school in southeast Dallas. Over 92% of the students were born in Texas but

Spanish is the primary language spoken in the home. The parents were born and raised in

Mexico and speak some English but often this student is responsible for translating important

information for the parents.

Hard work (manual labor) and family are two of the core values instilled in these

students. Education is important but not a top priority. Higher education is not a serious

consideration.

Vocabulary is a consistent struggle for these students, especially academic and technical

vocabulary. A reading level assessment for these students, prior to beginning instruction, would

be very helpful in tailoring the instruction.

Experience Levels

The students are tech savvy and have a strong command of computer skills. These

students may need help with some vocabulary but the definitions of troublesome terms could be

found online during the training. Each student will have at least completed the sixth grade.


Attitudes

Motivation is an issue for the intended learner. Making good grades and long-term goals

(i.e. top 10% of high school graduating class, college acceptance, earning scholarships, preparing

for SATs and ACTs, and career aspirations) do not motivate the student in the classroom. The

student’s primary motivation for coming to school is socialization, not learning. The student

realizes he or she is in a school of choice and values that to a certain degree but is primarily in

the school because his or her parents want a safe environment for their child.

Immediate rewards are one effective method of motivation. Something as simple as

granting a privilege such as listening to music during class or giving candy for a correct response

can be helpful tools to encourage this student. Outside of school, the student is very interested in

soccer and plays on a select team. References to soccer, including famous professional players,

fitness, skill, determination, and even projects that incorporate research in the field all entice this

student to engage and learn. Another outside interest of this student is video games. When

possible, this student plays with gamers around the world. Anything gaming related immediately

peaks the interest of this student.

Prerequisites

Students should have a strong command of basic computer skills. Students should also

be able to read simple graphs and tables.


Schedule

Number of Students/Trainees

There will be 510 students in the learner group. This is based on 85 students per grade in

grades 7-12. The number of learners per class is 20-25 and the number of learner groups is also

20-25.

Location of Students/Trainees

All training will occur in math classrooms at A+ Academy. Students will receive

computer-based training during their regularly scheduled classes with classroom teachers serving

as monitors.

Class Lists

Each secondary math teacher at A+ Academy will train their own classes so class lists

will consist of all secondary math class rosters.

Pre-course Communication

Math teachers will notify all of their students and parents both verbally and in writing one week

prior to the training. The notification will include the location, time, performance objectives, and

purpose of the training.

Tracking

Each student will complete the Student Learning Report (SLR) for each assessment taken. The

SLR is the method for tracking student performance. It will be maintained over the course of the

school year for determining whether or not students are mastering objectives.


Evaluation

Summary Outlining the Purpose of the Evaluation

With every instructional unit evaluation is crucial in determining whether or not students

have mastered the objectives set forth. This unit is no different and therefore multiple levels of

evaluation are built in to the training.

Student reactions are measured to determine the quality of the materials. The

effectiveness of the instructor is also measured during this phase of the evaluation process.

The students’ ability to perform tasks both in and out of the training environment is

measured to determine if the training paid off. In other words, are the students able to complete

the learned tasks in a “real world” environment? This transfer is critical in determining the

success of the training.

A final level of evaluation remains in place for a period of months from the conclusion of

the course to the end of the semester. This enables teachers and administrators to measure the

long term success of student mastery of objectives in each content area. So not only will students

be able to measure their individual performance using meaningful data but the school and district

will also be able to measure student performance holistically. The school and district should see

great benefits not only in the quality of individual student performance but also in higher

STAAR scores in very content area.

Data Collection Tools

Student perceptions will be measured using a questionnaire. Knowledge and skill acquisition will

be measured using practice and simulations. Actual learning transfer will be measured by

authentic work tasks, supervisor assessments, and observations.


Timing

Student perceptions will be measured immediately at the conclusion of the course. Knowledge

and skill acquisition will also be measured immediately at the conclusion of the course. Actual

learning transfer will be measured for a time period ranging from immediately after the student

returns to other content classes until the end of the semester.

People Responsible for Conducting each Evaluation Level

Student Perception (Level 1)

The teacher will administer the student perception evaluation.

Student Learning (Level 2)

The teacher will conduct the evaluation to measure the students’ ability to perform tasks.

Performance (Level 3)

A teacher in a different content area (other than math) will administer the evaluation in the

classroom environment to measure how well the students apply the knowledge and skills

acquired from the training.


Set of Evaluation Tools

Summary Chart Evaluation Plan

Who What When Where Why How Level 1 Perception

Admin-istered by the Teacher

Measure student perception

Immediately at the conclusion of the course

Within the learning space (class-room)

• Determine degree of satisfac-tion with the content

• Determine degree of satisfac-tion with the teacher

Question-naire

Level 2 Learning

Admin-istered by the teacher

Measure knowledge and skill acquisition

Immediately at the completion of the course

Learning space (class-room)

• Determine student potential to perform in each content area

• Determine quality of the learning resources

• Practice • Simula-

tions

Level 3 Performance

Admin-isterd by teacher in a different content area

Measure actual learning transfer

Immediately after student returns to other content classes until the end of the semester

In content area class-rooms

• Determine student’s ability to perform task with genuine consequences

• Judge whether the perfor-mance gap has been closed

• Authen-tic work tasks

• Super-visor assess-ments

• Observa-tions


References

Branch, R. (2009). Instructional design: The addie approach. New York, NY: Springer Science

Business Media.

Intulogy. (n.d.). Retrieved from http://www.intulogy.com/addie/pilot-testing.html

Malamed, C. (n.d.). Retrieved from http://theelearningcoach.com/elearning_design/starting-an

elearning-department/


Appendices

Interview Questions used with Students

1. Do you know where to find your own assessment (test) data for state and local

assessments?

2. Upon locating your own assessment (test) data, can you interpret that data by matching

assessment questions to prescribed content objectives (TEKS)?

3. Do you understand how to analyze assessment (test) data to determine which objectives

have not been mastered?

4. Can you use your assessment (test) data to set personal goals for achieving mastery of the

prescribed content objectives (TEKS)?

Secondary Student Data Self Analysis Secondary Student Assessment...

Documents

Transcript of Secondary Student Data Self Analysis Secondary Student Assessment...