The following strategies are based on a theoretical approach to standardized test deconstruction still in its design phase. This PowerPoint is not finished. Some slides may be incomplete.

Cracking the Code

A Strategy for Student Achievement by Deconstructing Standardized Tests

from the Inside-Out!

Curt Von Wooster M.A., M.A., Ed.D., NBCT

Central Theory of Deconstruction

“If you can't measure it,

you can't manage it.”

- Peter Drucker

(1909–2005) - a writer, management consultant, and self-described “social ecologist.”

…….In 1959, Drucker coined the term “knowledge worker.”

Knowledge Worker Defined

Knowledge Worker (also referred to as intellectual worker or brain worker) is someone who is employed due to his or her knowledge of a subject matter, rather than their ability to perform manual labor.

We educators are all… Knowledge Workers!

As such, we all need to think about testing in a more holistic way.

We need to work smarter when confronting We need to work smarter when confronting testing and not hardertesting and not harder!!

Why do our students not do well on tests? And the answer is…….

They do not READ on a regular basis.

As we all know, studies show that students who read regularly have increased test scores.

If we can improve their reading and vocabulary comprehension of the actual test, they will do

better on the test.

I am convinced……

….that our low test score problem is NOT due to lax or poor instruction on our part as educators.

Instead, I believe that the our low scores are due to poor comprehension of the material and of the tests themselves.

If students can’t read the tests, they will do poorly on the tests!

We need an improved school wide strategy!

Deconstruct the Tests!

The truth is……..

We can and do effectively teach students all day long- all year, but proper instruction alone will not solve the problem.

If there was an easier way or perhaps shortcuts to helping kids retain the information thereby increasing test scores, wouldn't we be foolish not to use it.

If our school funding is so tied to test scores and we risk losing it if we do not increase our AYP/API standing, then why not try an additional approach to test preparation…….!

Strategy-Based Achievement

Previously the school focused on Goals and a Vision for achieving higher scores on tests.(ex: Do your best on the test!)

Simple strategies were sometimes focused school-wide. Often too little and too late. (ex: Re-teaching and reviews)

Past Professional Development days were geared toward success on the test - usually only a month before the actual May assessments.

Post vs. Pre Analytic Strategy

Post-Analytic Strategies were focused on doing well on the test by getting students to learn how to take the test (the external approach) based on past test scores from previous years. –analyzing after the test?

Pre-Analytic Strategies focus on what comprises the test and how it was designed (the internal approach). This is where deconstructing the tests is crucial. We need to know where the testing problems are before they occur! – analyze before the test!

We need to use BOTH effectively in unison!

We need a three fold approach…

Post-Assessment

Data Analysis

Pre-Assessment Preparation

– DECONSTRUCTING THE TEST

Effective Teaching &

Classroom Test Preparation

Metrics defined ….

Test design relies on metrics….Metrics/metrical design- set of ways of

quantitatively and periodically measuring, assessing, controlling or selecting a person, process, event, or institution, along with the procedures….. to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments.

Wordy, huh!? Instead let’s simplify it….

Psychometric Design….

Standardized tests do not design themselves. They are not teacher designed.

Each state directs the contracted testing companies with their individual needs, and requests that tests be designed accordingly. The testing company then goes to work designing.

Each test is designed by teams of Psychometricians. Psychometricians are highly trained individuals who are skilled

in mathematics, statistical analysis, educational theory and psychology.

If we take even a small amount of time to understand the test If we take even a small amount of time to understand the test design process and psychometrics, then we will be ahead of the design process and psychometrics, then we will be ahead of the game.game.

Think of the test as the enemy! Understanding the enemy is our goal, in order that we can learn

its weaknesses and formulate a more thoughtful approach to attacking the test.

To understand the test we must understand who makes it….

The same company who makes the state tests – Educational Testing Service or ETS licenses some of their questions to be used on LAUSD periodic assessments.

Just as any product that emanates from its manufacturer has certain attributes and inherent design functions, ETS tests have a certain rhythm and pattern to them. We need to review every aspect of these tests to identify certain ETS design considerations.

The same goes for other testing companies.

Design Philosophy….defined…. Example – Ford makes

automotives. All Fords have certain

design similarities. As a result, we can make

certain assumptions about what is important to Ford’s design philosophy.

- Powerful engines- Sleek designs- Fuel efficiency- Strong- Dependable- A certain ‘Ford attitude’

meant to make consumers want to drive their vehicles

We can look at the state standardized tests and LAUSD periodic assessments the same way!

Believe it or not!?

Nothing on any standardized assessment is by chance. Modern metrical test design is viewed as both science and an art.

Nothing, from the layout and language to the graphics, italicization and bolding of certain words or letters is by accident.

Not even the use of certain syllables and the frequency of certain words, themes or answers is an accident. Even the number of questions on a page is by design.

Review released questions from past tests and see for yourself.

Playing the game…..

There are only three ways to play any game –

1. Play to Win

2. Play to Lose

3. Play to break even or to a draw

Vegas makes money because they understand these rules.

We need to more effectively play to WIN

rather than talk and try to win!

In short…These tests are a game!

Any game can be won with the right strategies. All games include inherent design flaws or

weaknesses. If we choose to view them as a game we can

beat them! Whether we choose to admit it or not, we tend

to teach to the test. We might as well try to beat the test!

We can WIN the testing game with the right strategies and approach!

Beautiful minds? Game theory and test design…. Remember this movie. It is

the story of psychiatric patient and mathematics professor John Nash who helped form the science of Game Theory.

By using simple game theory principles (predicting the probability of certain outcomes) we can predict the subject matter on the test and also student responses.

Not a mathematician… or insane, no problem.

But more on that later…

And now on with the show!!…….

Now that you have learned some crucial information about standardized testing and their design, let’s put this information into practice.

Remember, this school should continue to do all of the things we already do to properly instruct!!!!!!!!!!!!

We just need to add to our arsenal of instructional weapons, some enhanced strategies that you will see in the following slides!

An Example of Deconstructing the Test using the 7th Grade Periodic History Assessment

Think about this…… On several standardized history tests, every question or answer related to the history of Ancient Islam that involved the name “Muhammad”, corresponded to a question or answer that included the word “prophet”.

By looking at the tests, it becomes apparent that the concept that MUHAMMAD is viewed as a PROPHET by the followers of Islam, is important to the state of California and the test makers. (part of state standard 7.2.2)

In other words………

If you teach a student that knows nothing about the history of Islam, that when they see a question that includes the name Muhammad….., they should look for an answer that includes prophet. (keyed words)

While this theory will not prove correct EVERY time, it does give students a leg up on answering questions that they may be unsure about.

This is a better analytic strategy than simply choosing/guessing “C”, if they do not know the answer.

When in doubt choose ‘C’?...Not so fast…..

Statistically speaking choosing the letter ‘C’ is not a better choice to use in standardized tests if one does not know the answer.

This a MYTH! While there may have been a time in testing history that this was true, it is not true today.

Modern metrical test design would never allow for and does not allow for such a repetition in an answering pattern.

However, it is possible that at least any one test might be weighted towards certain answers – A,B,C or D

Sadly, it allows students to take the easy way out and not think or read the questions and can, in fact, hurt their overall scores.

Still don’t believe deconstruction works….…This question is straight from a 7th grade

periodic history assessment. ---- According to Islam, Muhammad wasA – the first to understand the meaning of the

Christian BibleB – an angel sent to earth by God to help the

poorC – an Islamic scholar who worked to preserve

Greek LearningD – the prophet through whom God revealed

his final message

Let’s try another a different way…

When you see a question involving Ancient African civilizations of Ghana or Mali, you should think about salt and gold trading.

Ghana/Mali = salt/gold trade ---(keyed words) Now answer this question using the keyed

word strategy. This is an actual question from a recent

periodic assessment.

Use the keyed word strategy….

The economy of Ghana was dependent upon the trade of ….

A – grain and salt

B – salt and gold

C – gold and oil

D – grain and slaves

And the answer is B – salt and gold

Yes, it is a real question from a recent history periodic assessment.

More keyed word answers….

On several recent periodic assessments and by looking at released questions if can be seen that…

Many 7th grade science questions that have to do with LIGHT are keyed to a response that includes the answer REFRACTION. Light = Refraction

Many 8th grade history tests that include a question about CHANGES to the CONSTITUTION are keyed to a response that has to do with the answer AMMENDMENTS. Constitutional changes = amendments

7th grade questions about the development of CALENDARS are keyed to Mesoamerican cultures (AZTEC, MAYA, etc.) Calendar = Maya or Aztec

Get kids to understand KEYED WORDS!!!

To illustrate this deconstruction further let’s suppose that……perhaps on a test, the questions that most

students did poorly on were analyzed and it was

found that those questions had – More difficult vocabulary words (more use of academic

words)

These words contained three or more syllables These questions/answers contained longer

sentence structures (maybe six or more words) These may be examples, but each may lead to an understanding of

why some questions appear more difficult, even though we teachers hammer certain key concepts, only to have students get the answers wrong.

Visual Prompts in Testing

Look at the WORLD MAPS above. Compare them. Think about how a struggling learner might see these two

maps differently. Can you tell what the MAJOR difference is in the maps?

Color or Shade Inversion

Many of the tests include maps, political cartoons, or diagrams that invert shades or colors based on differing questions.

This type of test methodology is not an accident – Test makers use this as a metric when designing the tests.

These very subtle test design changes can cause major problems for the visually impaired, special education students and the wider test taking population as a whole.

These shade inversions are actually studied by test makers and become part of the metrical design of future tests.

Brainstorming Deconstruction We need to look at every element

of the test and find out why they are having difficulty with the questions.

No detail is to be considered a waste of time.

We need to be open to even the most minute trends, patterns, and themes.

Do not assume the test taker will know certain basic words and phrases.

Just because they can say it verbally does not mean they can read or write it!

Again, NOTHING in the test design process is an accident. If we find even one new strategy per test by breaking down each assessment, we can improve overall statistical results for the whole campus.

Assume nothing!!!!!!

Let’s look at – Syllable counts Sentence length Word use frequencies Picture/chart/drawing use Color/shade inversion Use of italicization Use of boldfacing certain words or

terms Use of underlining Number of questions per test Number of questions per page Location/placement of certain

questions on a test

..anything that MIGHT help even if it turns out to be false.

Vocabulary Review

If students cannot read and comprehend the questions, then they cannot choose correct answers.

Focus on Brick and Mortar words – those more basic ‘mortar’ words that connect academic words ‘bricks’. (Kate Kinsella)

Need to focus on non-academic words and phrases that affect how students read questions - (occupied, scholar, through whom, preserve, etc.) These type of words actually trip up students based on surveys of my classes.

Non-Academic Vocabulary – History words we should never assume students already know

alliances

geographic (location)

initial refusal

unrelated

related

‘believe that’

inspiration

faith(s)

believed

direct descendant(s) of,

successful

scholar preserve ‘through whom’ monotheistic complex accurate developed economic (growth) spreading increasing defeat

Distracters – defined….

n.   One of the incorrect answers presented as a choice in a multiple-choice test.

Some distracters are obviously wrong, while many are possibly correct or half-right.

A test takers challenge is to weed out the BEST Possible Answer from the distracters.

Our goal – implement strategies and reinforce them in the classroom that teach students how to tackle these distracters.

We need better strategies for dealing with distracters when test prepping students.

‘List’ Questions and Distracters

Most questions involving lists of three or more concepts will use a ‘negative’ (not, except, etc.) when written on a test.

Example: Which of the following is not considered to be one of the five

pillars of Islam:A - Pray five times a dayB - Maintaining a special religious dietC - Pilgrimage to MeccaD - Give alms to the poor By previewing these types of questions and using the distracter

format when teaching, we can give students an added boost in their thinking process.

But this needs to become a conscious part of the teaching process!

Trick questions -

Students should be aware of questions using ‘negative’ words/terms such as –

‘which is…except’ which is not ‘all are....except’ except for none of the above all of the above

-------------------------------------------------------- On average about a third of questions on any

standardized tests use these ‘except’ type questions.

New strategies?

We have committees for…

Technology Discipline School site council Safe school issues Shared decision making Pearson/other learning curriculums….etc.

But we DO NOT yet have an INSTRUCTIONAL committee to look at tests and test data

You would think that this would have been implemented previously.. A no brainer!?

Instructional Committee?!

Proposal – a campus wide committee of one science, math, history, English, elective and physical education, and special education teacher along with one administrator should be formed to review all of the tests from all disciplines. Total of eight people maximum.

And/or we can use the departmental approach and have each look at their own tests !

Goal - This committee’s sole purpose should be to deconstruct the tests of each content subject area and find major themes, patterns and trends in each test that could be taught to the students before they take the assessments.

Mission – This committee should report back all findings to the staff during team, department and professional developments.

Instructional Committee Processes

The entire group would look at each test (example: all eight review the history test) and each person, regardless of whether they know anything about that subject area, will give their input about how they view the test (fresh eyes possibly looking at unfamiliar material).

Example: A science teacher might view the history test differently and point out something that a history teacher might have taken for granted and overlooked.

Remember this is primarily about deconstructing the tests and not looking at only test data after the fact.

Instructional Committee Results

After this process has been completed, each person from their specific disciplines will write a brief summary (bullet points) of their findings that the committee reviewed for their content area.

The information will be given out to all faculty to use in their classrooms during everyday instruction and also during review periods for the different tests.

Warning!.......

These strategies that result from the committee will NOT alone solve the testing issues we face.

We must continue to use effective teaching methods, regular reviews and re-teaching of key concepts throughout the year.

This process of deconstructing the tests is to be used as a support to regular instruction and should be used in conjunction with all post-test statistics.

Implementation…….What do we do next?Use a three-fold strategy –1. Deconstruct the tests to find vocabulary and

terms that need to be re-taught to our students. Create subject area vocabulary lists to use as study sheets.

2. Deconstruct the test design to find any themes, patterns or rhythms to the tests. Create lists of keyed words.

3. Form and implement an Instructional Committee to evaluate the tests from the outside and deconstruct from the inside.

7th History Formative Assessment

The following items have been evaluated based on their relevance and importance to this assessment.

Generic breakdown of actual questions – Actual numbers indexed to actual STAR test percentages in May.

Thematic breakdown – based on state standards - (contributions, cause/effect, core beliefs, etc.)

Keyed Language – words that should be taught together and used to link one to the other

Islam Questions4 Total -

Generic Themes

Muhammad as prophet.

Islam as monotheistic

religion.

Core Beliefs -Five Pillars of

Islam

Contributions of Islam -

Art & Science

Rome Questions3 Total –

Generic Themes

Causes/EffectsOf the Fall of

Rome

Contributions of Rome

Architecture, Art,& Government

Rise Of Christianity –

Byzantine Empire

Rome Questions - Breakdown

Rome Questions3 Total –

Generic Themes

Causes/EffectsOf the Fall of

Rome

Contributions of Rome

Architecture, Art,& Government

Rise Of Christianity –

Byzantine Empire

11. Map based question –Area of empire was too large to

defend. Map does not show any physical

features except area of empire.

Keyed words – Rome and “difficulty defending”

7. Use of “all or except”type of questioning

Wrong answer consists of Greek “Olympic Games”

or non-Roman response

12. Keyed words –Rome and Christianity

Establishment of Constantinople led to shift in power to East

Islam Questions – Breakdown

Islam Questions4 Total -

Generic Themes

Muhammad as prophet.

Islam as monotheistic

religion.

Core Beliefs -Five Pillars of

Islam

Contributions of Islam -

Art & Science

1. Keyed Words – Muhammad and

Prophet

False Positives –Use of words like..Bible, Christianity,

angel, poor, scholar, Greek.

5. Keyed Words – Christianity, Judaism, Islam,

Monotheistic – belief in one god

Keyed words – “Related to” and “all are”

But I need proof this works?!...

On the first 7th grade periodic assessment given in December ’08, the questions that were evaluated in which I used these strategies showed a 92% or above score. Only one class had a slightly lower percentage in relation to one question and it was still at a respectable 88%.

Other History teachers also found a higher success rate using these strategies.

Even special education students showed an increase in correct answers on those questions using the keyed word strategies.

Don’t believe it… You are welcome to look at the my test results and see for yourself!

The future….?

And now we can choose to take this further or not.

The choices we make shortly could affect the outcome of our school funding.

I do NOT profess to have all the answers or to claim that my theories and strategies will alone solve the problem. These ideas are but a suggested part of the test-prep whole.

Thanks for your time!