Panel 3 Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing

Panel 3

Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing

Panelists

Bob Wilson: Wilson, Morton & Downs, LLCTisha S. Edwards: Baltimore City Public Schools

Steve Ferrara: PearsonLou Fabrizio: North Carolina Department of Public Instruction

Robert (Bob) E. WilsonAttorney

Wilson, Morton & Downs, LLC

One of the Special Investigators Appointed by the Governor of Georgia to Investigate

the Atlanta Public School System

Atlanta Investigation

• Investigative Team – over 60 state agents, attorneys and paralegals• 10 month investigation• 2200 interviews• 56 schools investigated• Cheating found in 44 schools• In 30 schools, 82 principals and teachers confessed• Over 180 educators involved• 38 principals involved

• For System: time consuming/off mission

Atlanta

Created Targets Beyond NCLB+

Imposed Pressure to Achieve(Both Positive and Negative)

=

DISASTER

You have three years to meet targets or I will find someone who will.

There will be no exceptions and no excuses.

Warnings

• Significant increases in scores• Significant decreases in scores• Unusual answer patterns• High wrong-to-right erasures• Report of suspicious conduct• Report of cheating

Report To

• School Leadership• System Leadership• System Legal Counsel• State Education Authorities• State Licensing Authority• Resource Officer/Law Enforcement

• Ignore nothing – all concerns are important• Vigorously inquire• Interview everyone who might know anything:

- Administrators- Teachers- Staff- Parents- Students

• Consult experts• Re-Interviews• Polygraphs

Fundamentals

• The test should be about children, not adults.• Image is not more important than children.• Nothing is more important than creating a

culture of integrity with an honor system to report suspicious activity.

nces.ed.gov

Testing Integrity Symposium

Baltimore City Public Schools

Andrés A. Alonso, Ed.D.CEO, Baltimore City Public Schools

February 28, 2012 P U B L I C

S C H O O L SB A LT I M O R E C I T Y

nces.ed.gov

Objectives

• Share a lesson learned from Baltimore City Public Schools’ Test Monitoring and Investigation Process

• Provide Baltimore City Schools’ best practices on its Test Monitoring and Investigation Process

• Share recommendations for restoring credibility

P U B L I C S C H O O L S

B A LT I M O R E C I T Y

nces.ed.gov

Best Practice Story• A Baltimore City School was alleged to have altered its scores from the Maryland State Test in order

to qualify for Blue Ribbon Status. • There had been previous allegations concerning this school that were found to be unsubstantiated.

• Duration of case: 18 months • City Schools experienced challenges establishing culpability as a result of: people not talking; there

was over-sensitivity in the reporting of allegations; there was a back tracking of allegations once staff and parents were interviewed.

• By conducting multiple analyses (communication analyses, erasure analyses, etc) City Schools established a very clear case that there was likely systematic cheating occurring at the school.

• Analysis is never enough to establish culpability. • Unless comprehensive investigations occur, the credibility of districts and schools suffer. While the

organization under investigation will take an initial hit by the investigation, the price is worth it• Strengthening all aspects of our internal controls. • Conducting multiple types of analyses in order to establish culpability with certainty. • Continuing to adhere to the Code of Maryland Statutes for conducting testing investigations.

Issue:

Result:



Lessons Learned

nces.ed.gov

City Schools’ Test Monitoring and Investigation Processes

• PREVENTING POTENTIAL VIOLATIONS• DETECTING POTENTIAL VIOLATIONS• INVESTIGATING POTENTIAL VIOLATION• PREPARE INVESTIGATIVE REPORT• MAINTAINING CREDIBILITY



nces.ed.gov

PREVENTING POTENTIAL VIOLATIONS• Establish and implement internal controls that will signal potential problems

• Complete statistical forensic analyses that will pick up unusual patterns or trends in the data. City Schools analyses include:• Growth Analysis to examine annual results to identify significant gains/decreases • Benchmark Analysis to examine student performance on Quarterly Benchmarks in comparison to their performance on State

Assessments• Entrant and Leaver Analysis to examine how students perform as they transfer in or out of schools• Cohort Analysis to examine the results of a particular group of students over a period of time

• Develop a detailed monitoring plan and monitor all high stakes assessments annually• In SY 2009-10 City Schools dispatched 120 central office administrators to monitor all aspects (distribution and

collection of materials, classroom observations, security protocols, secure location etc.) of the MSA test administration to 13 schools

• In SY 2010-11 City Schools hired over 200 temporary employees to monitor all aspects of the MSA test administration to 157 schools

• Independent monitors assigned to each school increases the integrity of the results and the public’s trust of the results• Enhance the education and awareness for all stakeholders

• Provide a mandatory training for all persons participating in the administration • Disseminate video presentation from the CEO stressing the importance of implementing the assessments with fidelity

and consequences for violators • All personnel participations in the administration sign a Non-Disclosure form • Communicate monitoring plan to all stakeholders in collaboration with the state



nces.ed.gov

DETECTING POTENTIAL VIOLATIONS

• Establish well defined protocols to detect potential concerns• Conduct forensic data analysis after each administration to identify potential concerns

• Flag potential concerns with support from stakeholders • Work in collaboration with the state and other stakeholders to flag potential concerns. • For example, the state provides City Schools with Scoring Alerts which identify potential concerns for

City Schools to review. City Schools also collaborates with the state to complete Erasure Analysis where necessary.



nces.ed.gov

INVESTIGATING POTENTIAL VIOLATIONS

• Establish lines of communication between City Schools and the State • Primary responsibility for conducting test security investigations belongs to City Schools • City Schools works in collaboration with the state to complete our investigations • Coordinator of Security Investigations in City Schools assigned to directly collaborate with the state’s Test

Security Officer on all testing investigations.

• Establish well defined protocols for conducting investigations in accordance with state statutes • Adhere to the state statures for conducting security investigations • Increase investigative protocols by hiring an external law firm to review and codify protocols

• Implement a process to ensure that ALL allegations are investigated • Plan for the investigation• Have a plan in place if a test security violation or test administration procedural deviation occurs

• Collect strong qualitative evidence• A barrier is establishing open communication and getting individuals to talk openly about the allegations. • City Schools has protocols in place to ensure confidentiality of all individuals interviewed as a part of the

investigation. • All staff are interviewed to protect the source of the information. • All person interviewed are provided with a summation of the district’s anti-retaliation policy.



nces.ed.gov

PREPARE INVESTIGATIVE REPORT



• Create a strong investigative report • Conclusions are based on solid data utilizing both quantitative and qualitative • Where possible, utilize methods to triangulate the data • Evidence supports findings • Present findings to proper internal authorities

• Plan for litigation• Throughout the investigation, the Coordinator of Security Investigations works with:

• City Schools’ Legal Office on potential findings and to prepare for case hearings• City Schools’ Labor Relations in preparation for personnel discipline hearings

nces.ed.gov

MAINTAINING CREDIBILITY

How do you maintain credibility in the face of accusations?

• Administrators should address the allegations publicly.

• Administrators should also buffer schools during the investigation, making sure that they do not suffer a decline in credibility.

• Effective communication is very important as it is difficult to establish causation in terms of cheating, especially if schools are not open to discussing the allegation. • City Schools’ CEO actively takes the lead on communicating potential issues with the board, media,

politicians etc.) • In the best practice story shared, City Schools’ CEO met with the staff, parents, and the newly

assigned principal to share his expectations and how the school was going to move forward.• The Coordinator of Security Investigations works in conjunction with City Schools’ Office of Parent

and Community Engagement to meet with community organizations and media to share the Testing Integrity Plan.



nces.ed.gov

RESTORING CREDIBILITYHow do you restore credibility of a school system tarnished by alleged or actual wrong doing?

• Fortunately for City Schools, our reputation was not tarnished as a result of testing investigations.

• A reason for this was CEO’s proactive position to openly and honestly communicate the districts’ efforts to eliminate these issues. • For example, the CEO held a press conference with the State Superintendent to publically announce

that based on City Schools’ investigations and internal controls, he was referring several schools to the state for violating testing protocols.

• Recommendations to School Districts: • Maintain open and honest communication acknowledging the problem and how it is being

addressed. • Firmly communicate that these types of actions will not be accepted as they undermined the great

work being done in the district. • Provide a general overview of what the district is doing to prevent potential violations from

reoccurring.



Investigation and Response: Experiences and Reflections of a Former State Assessment

Director

Steve FerraraPearson Center for Performance Assessment

NCES Testing Integrity SymposiumPanel III: Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing

February 28, 2012

Overview

• Background regarding investigations– Testing integrity is broader than test security– Mutually supportive test security system and culture of

professional ethics• How the system and culture can support investigations• Investigation vignettes• Investigation considerations

Testing integrity is broader than test security

• Conceptual distinction• Test security

– Security of test content and materials– Appropriateness of test administrations

• Testing integrity– Requires test security– Requires professional ethical behavior– Yields trustworthy data– Avoids inappropriate messages to children

22

Mutually supporting test security system and culture of professional ethics

• Security system– Rules, guidelines, requirements, and procedures—including training—to

protect security, avoid breeches, and enforce security

• Culture1 of professional, ethical thinking and behavior– You don’t have to persuade people to buy into the culture; you do have

to get them to abide by it.

1 The set of shared attitudes, values, goals, and practices that characterizes an institution, organization, or group (from http://en.wikipedia.org/wiki/Culture)

How the system and culture can support investigations

• Three groups of people ()• Culture of professional ethics

– Encourages (supports, exerts pressure) professionally ethical thinking and behavior

– Encourages reporting of irregularities and cooperation with investigations when irregularities may have occurred

Three (hypothetical) groups of people

Buy into the culture, act accordingly,

may be vocal about

it

May not buy in fully but act

appropriately, and may

support the culture and

cooperate in investigations

Other

Brief vignettes

• What often happens after an investigation of a school?– Behavior typically changes, typically in a positive direction– Concern: misbehavior becomes less brazen, goes further

underground

• State Teacher of the Year in a state’s Blue Ribbon School

• Teacher dictating responses to a testing group– During task pilot testing– In an operational administration

Considerations for investigations

• Typically, test development and delivery contractors are not involved in investigation, are involved in protecting security, may be involved in detection

• Investigation principles and procedures– Must be fair – Must withstand potential challenges in administrative law courts– Manuals, training programs, and experts in virtually every field– How do you get that expertise into schools, LEAs, and states?

• Cheating risks with new devices– Currently, we can lock down online testing; tablets have task

swapping and screen shot capabilities– Investigation challenges if personally owned devices have been

allowed• It is necessary to investigate, enforce, and sanction

rigorously and consistently

Thanks!

Steve FerraraPearson Center for Performance [email protected]

mailto:[email protected]

Responses to Alleged Testing Irregularities

Lou Fabrizio, Ph.D.Director of Data, Research

and Federal PolicyNC Department of Public Instruction

State Board of Education(Rules & Regulations/Administrative Code)

• Testing Code of Ethics – (16 NCAC 06D .0306)

• License Suspension and Revocation– (16 NCAC 06C .0312)

• Standards of Professional Conduct– (16 NCAC 06C .0602)

30

Record & Monitor Irregularities

• NC uses an online system• Suspected and actual irregularities must be

reported• Local investigation required• Assistance, as to process, is provided by

Regional Accountability Coordinators and others from the SEA

• Serious irregularities trigger required check-lists

31

Steps in Serious Allegations/Actual Irregularities

• Once LEA completes a local report, it is reviewed by Accountability Services staff in the SEA

• Reports of serious irregularities are then forwarded to the State Board of Education (SBE) Attorney

• In collaboration with the State Attorney General’s (AG) office, there can be a further investigation by the SBE and/or AG-assigned attorney

32

Barriers

• Limited resources – There are no staff with sole assignment for investigations and no specific budget line item for investigations

• Follow-up investigations can take longer than desired because of the above

• Some investigations come before the State Superintendent’s Ethics Advisory Committee for Teacher Licensure

• Some investigations result in civil action and/or loss of license

33

Other Thoughts

• Serious allegations or actual instances of irregularities should be communicated to appropriate leadership in the SEA, including the Communications & Information Division

• Prevention is the best solution• Need to advocate for a culture of honesty

34

Panel 3 Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing

Documents

Transcript of Panel 3 Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing