Post on 04-Jan-2016
1
ATEAM
ATEAMKenneth Barbour
Faye Atkinson, Rhonda Carlyle, Lori Ivey, Phyllis Price, Jim Rhew, Kevin Skelton, George Stubblefield
2
Our Purpose
• Support the design Assessments of Student Achievement and implementation of Assessment
• Collect and Analyze Achievement Data• Report Student Achievement• Design, develop and implement a system to analyze
student academic performance outcome data for purposes of professional development, and technical assistance geared to improve instruction and student performance
3
Our Customers• Accountability Services
– Test Development– Testing Policy and Operations
• LEA Test Coordinators• Regional Accountability Coordinators (RACs)• Regional Computing Consultants (RCCs)• State Board of Education• Public• Local Education Agencies (LEAs)• Schools• Researchers• Department of Justice• Federal Reporting
4
Data Consumer List
• State Board of Education
• Legislative / Judicial Requests
• Federal Submissions
• Researchers
• LEAs / Schools
• Title I & III
• Public
5
Our Partners
• Portfolio Managers in other DPI Areas• Accountability Services Division• Technical Outreach for Public Schools (TOPS)• Regional Accountability Coordinators
– Regional Computing Consultants• Test and Accountability Coordinators/Directors
6
Our Governance• Accountability Services Division• Data Management Group (DMG)• US Department of Education • No Child Left Behind (NCLB)
– Adequate Yearly Progress (AYP)• Accountability Basics Control (ABCs) of Public
Education• Accountability Configuration and Control Board
(CCB) - Advisory• State Board of Education (SBE)• Legislature
7
Services of ATEAM
• Applications
• Analyses
• Reports
• Accountability Database (ADB)
• Technical Support– Customer Support (RITS - External)– Customer Support (RTS - Internal)
• Training
8
Products List• WinScan• ABCTools• Request Tracking System (RTS)• Regional Issue Tracking System (RITS)• NC WISE Test Upload System• Online Testing Irregularity Submission System (OTISS)• Individual Student Report (ISR)• Data Collection, Verification and Reporting (DCVR)• Collection Date Submission System • Accountability Database (ADB)• Writing Instruction System (WIS) - TOPS• NC Registration - TOPS• Online Testing Systems - TOPS • Online Ordering System - TOPS• Testing News Network (TNN) – TOPS• Request Tracking System - TOPS
9
ATEAM
Other DPI Authoritative Data Sources
NCLB
Public
ABCs
NC WISESchools
RACs/RCCs
ATEAM INTERFACES
Test Development
Pearson and Other Contractors
TOPS
Testing PolicyAnd Operations
District Accountability /
Testing Coordinator
80% of theReport Card
10
Accountability Services Structure• Accountability Division
– Lou Fabrizio, Director of Accountability, Policy and Communications
– Gary Williamson, Director of Accountability Operations– Tammy Howard, Chief, Test Development Section– Sarah McManus, Chief, Testing Policy and Operations
Section• ATEAM• Contractors
– TOPS (Technical Outreach for Public Schools) – Pearson– Other Contractors
11
Application: WinScan• Windows application used by the Central Office
of LEAs
• WinScan is a software program that is used to scan, score and process statewide tests
• WinScan identifies and corrects errors
• WinScan produces local reports– Class Rosters– Goal Summaries– Label Reports– Many Others
12
Application: ABCTools• Windows application used by the Central Office of LEAs• Data file loading for MYSQL database using managed file transfer• Manages accountability data for NCLB and ABCs with MYSQL data
warehousing system• Provides predefined SQL statements to assist users with queries• Provides student level audit of data elements used for NCLB and ABCs• Provides reports to assist school staff in identifying students at risk as
result of proficiency levels• Provides classroom scale score averages• Provides access SQL Query Tool• Provides precoding for answer sheets used for ABCs and NCLB• Provides verification reports and processes for test data submitted to
NCDPI for NCLB and ABCs • Provides links to accountability information
13
Application: Online Testing Irregularity Submission System (OTISS)
• A web based APEX Application
• Allows for the collection, approval workflow, tracking and archiving of testing irregularities
• Includes school level access
14
Application: Individual Student Report
• Windows application used by the Central Office of LEAs
• Prints parent/teacher reports• Subject to peer review by USED• NCLB regulates the reports• Includes scale scores, percentiles, exit
standards, Lexile measures, Quantile measures, achievement level descriptors, and sub score reporting
15
Application: Data Collection, Verification and Reporting (DCVR) andAccountability Database (ADB)
• Manages file transfer, data validation and reporting • Manages the collection of data sets from schools and LEAs,
validates and merges the data into an accountability database
• Provides interactive error reporting and data collection • Provides for the data cleaning and aggregation of data into
the accountability database (ADB) • Insures a unique ID across all the student information
collected from various data sources, including NC WISE • Encompasses all business logic to create the source data for
the ABCs and AYP reports • Provides the source data for all accountability data collected
for the EdFacts requirement
16
ADB / DCVR OVERVIEW
File Submission
And Validation
ETL Processes
ABCTools WinScan Other
Status Reports
DPIAuthoritative
Sources
Static Data And Configuration
Business Rule
Processes
Table HierarchyFollows BusinessRule Progression
To Enable Auditing And Analysis
ADB
Reports Analysis Data Requests LEA Exports
17
Application: Collection Date Submission System
• A web based APEX application• Allows the LEAs to enter their start of
school, 20th day, First Day of Fall testing, First Day of Spring testing, and last day of school dates
• Allows the ATEAM to use these dates to pull the enrollment and historical test data for the schools
• Decreases the load on the NC WISE servers and the LEA Testing Coordinators
18
Application: NC WISE Test Upload System
• Provides ability to import student test score information into NC WISE
• SCO file (from WinScan) is processed during a nightly batch job
• Summary / error reports are available through a web interface
• Also includes capabilities for import of SAT and CTE test data
19
Analyses:• Ad hoc analyses for customers• Growth Model development and validation• Reconciliation of reporting anomalies • Assist standardized test data certification• Apply linking tables to scales• Supply DPI Business Units with analyses results
– District School Transformation – Charter Schools– NC Virtual Public Schools– Title I & III– Others
20
Reports: Major Report Systems
• NCLB/AYP
• ABCs
• Disag
• NC School Report Card (SRC)
• Greenbook - TOPS
• EdFacts – (via EDEN)
• Considated State Performance Report (CSPR) (EdFacts)
21
Reports: Reports ListList of Reports for ABCs/AYP Release• Executive Summary• Low Performing Schools• Secondary Analyses
– Number and Percentage of School Level Targets Met by Subgroup
– Number and Percent of Schools by Percent of Targets Missed
– Number and Percent of Schools by Number of Targets Missed
– Number of Schools That Met AYP Using at Least One of the Following Special Flexibilities: Safe Harbor, Confidence Interval, Growth (Schools may be counted on more than one row)
– Number of Schools, Number Meeting AYP and Percent Meeting AYP by Number of Targets
– Number of Schools that did not make AYP: Number due to one subgroup
– Number of Schools that missed AYP due to two subgroups
22
Reports: Reports List Continued -
• Title I School Improvement List• LEA Improvement Status• LEAs in Corrective Action• Impact on Performance Using Retesting Results• 4-Year Cohort Graduation Rate (Current Year)• 4-Year Cohort Graduation Rate (Previous Year)• 5-Year Cohort Graduation Rate (Previous Year)• Evolution of the ABCs• AYP Schools Under Review• Possible ABCs Violations of the 95% Rule• Possible Violations of AYP 1% Cap
23
Report Tables List• Preliminary End-of-Grade General Test Multiple-Choice Test Results
Statewide Performance Summary in Reading and Mathematics Grades 3-8
• Preliminary End-of-Grade NCCLAS Test Results Statewide Performance Summary in Reading and Mathematics Grades 3-8
• Preliminary End-of-Grade NCEXTEND2 Test Results Statewide Performance Summary in Reading and Mathematics Grades 3-8
• Preliminary End-of-Grade NCEXTEND1 Test Results Statewide Performance Summary in Reading and Mathematics Grades 3-8
• End-of-Grade General Test Multiple-Choice Test Results, Statewide Characteristics and Performance of Grade 3-8 Students
• Preliminary End-of-Grade General Test Multiple-Choice Test Results Statewide Performance Summary in Science Grades 5 and 8
• 1985-86 to 2007-08 End-of-Course General Test Multiple-Choice Test Results, Statewide Performance Summary
• Preliminary End-of-Course NCCLAS Test Results Statewide Performance Summary
• Preliminary End-of-Course NCEXTEND2 Occupational Course of Study Test Results Statewide Performance Summary
24
Technical Support: Request Tracking System (RTS)
• Internal Request Tracking is an integral part of the total IT responsibility of the ATEAM
• It serves as the primary method to communicate changes requested to the “owner” of the work with applications and systems and the corresponding resolutions back to the customer
• It maintains consistency in logging and reporting requests, utilizing standardized procedures
25
Technical Support: Regional Issue Tracking System (RITS)
• External Request Tracking is an integral part of the total IT responsibility of the ATEAM
• It serves as the primary method to communicate regional changes requested to the “owner” of the work with applications and systems and the corresponding resolutions back to the customer
• It maintains consistency in logging and reporting requests, utilizing standardized procedures
26
Authoritative Sources
27
Technical Outreach for Public Schools (TOPS)• EOC On-line Testing• Paper and Pencil Testing• NCExtend1• Writing Instruction System (WIS)• Data Analysis Group• NC Registration• On-line Ordering System• Test Item Development• Test Item Bank• Testing News Network (TNN)• Logistics
– McKimmon Center– Blue Ridge Location– Warehouse (Other)
28
Writing Instruction System (WIS) - TOPS • Application resulting from the Grade 7 Online Writing Project • A pilot using the Moodle freeware system configured and hosted
at North Carolina State University (The pilot ended 6/30/09) • Consists of 3 online Professional Development modules for
Teachers, 2 on demand writing assignments and 2 content specific writing assignments for students in grades 4 and 7 – 7th graders were allowed to develop and upload the writing
assignments to an online portfolio – Teachers were then able to access and grade the assignments
online • Based on the results, the State Board of Education approved
the DPI recommendation to extend the pilot for an additional year and to open participation to additional LEAs. The project is extended through June 2010.
29
NC Registration - TOPS
• Schools and school systems use the system to indicate students: – who are eligible for testing– to schedule test administrations within
predefined testing sessions for groups of students
30
Online Testing System - TOPS
• Provide same assessment as paper and pencil online format
• Used for the eight EOC tests
• Browser based
• Need Adobe Flash to use
31
Online Ordering System - TOPS
• System developed that allows the LEA Test Coordinator to place, review, and update orders for testing materials
• System includes materials such as:– Braille– Read Aloud – One Item per Page – Large Print Test
32
Testing News Network (TNN) - TOPS
• TNN is a closed information and collaboration service sponsored by the Accountability Services Division and hosted by TOPS
• Users are testing and accountability staff of LEAs in NC Public Schools
• Provides secure information dissemination of local school system test coordinators
• Provides indexing of critical events, policies, reports, communications, process details and software upgrades for the testing program
33
Training System - TOPS
• NC Testing Program Online Training– Item Development Basics– Developing EOC Mathematics– Grades 3-5 Items– Grades 6-8 Items– Developing EOC Items– EOC English 10 Items– Additional Courses will be added
34
NC FALCON - TOPS Training
• North Carolina Formative Assessment Learning Communities Online Network
• Opening as a pilot in January, 2010 to train teachers, educators and administrators on how to do formative assessment
35
Archive System - TOPS
• Electronic archive of test materials for the NC Testing Program– EOGs – EOCs – NCEXTEND1 – NCEXTEND2 – Grade 10 Writing Assessment
36
Help Desk System - TOPS
• Provides HelpDesk Support for LEA / School Staff for all TOPS systems
37
Hardware Required for Standardized Testing and Accountability at the LEA Level
• In 1992, the NC General Assembly revised the NC Public School Testing Program legislation authorizing NCDPI to implement the end-of-grade and the end-of-course tests (reference Public School Laws of NC, Article 10A, Part 2, 115C-174.11)
• A key component to the successful implementation of the NC Testing Program is the timely scanning and scoring of the students' answer documents and the printing of the students' reports
• To accomplish these tasks, the NCDPI purchases, upgrades, maintains hardware, and develops software for computers, scanners, and printers
38
Software Required for Standardized Testing and Accountability at the LEA Level• ABCTools
• Adobe Acrobat
• Individual Student Reports
• Microsoft Office Professional
• Antivirus
• SFTP Secure Shell Client
• WinScan
• WinZip
39
EQUIPMENT ROLES EQUIPMENT RESPONSIBILITIES
Accountability IT Team: Equipment specificationsEquipment and/or software compatibilityGhost imagesEquipment set up assistance
Accountability Operations Director: Approve purchases
Accountability Purchasing Agent: Purchase orders
DPI Fixed Asset Clerk: Provide asset tagsMaintain official inventory (indicating where equipment is permanently housed)Provide inventory list for annual auditState Surplus contact
Accountability Equipment Acquisitions, Maintenance and Inventory Management:
Maintain internal inventory (indicating where equipment is temporarily housed)Assist with surplus/transfer formsLiaison between Accountability and Fixed Asset ClerkEnsure surplus/transfer forms completed and submitted as neededApprove temporary transfer of equipmentFacilitate annual audit
Lead RAC: Liaison between RACs and Accountability Inventory ManagementProvide RAC input on equipment issues
RCC: Equipment set up assistance (LEAs)Equipment maintenance assistance
RAC: Maintain regional equipment inventoryReconcile regional and fixed asset inventory annuallyRequest equipment transfers
LEA Test Coordinator: Day to day equipment maintenance and upkeepFamiliarity with equipment and its proper useReport damages/breakdowns to RAC/RCC
TOPS: Provide shipping of equipmentMaintain equipment poolRepair equipment
40
Training for: ABCTools • Monthly training for registered participants held at DPI • Demonstration of loading data into ABCTools, if
needed. ATEAM may have processes to move data into a set of folders for transformation.
• Demonstration of where to find data on web (Disag and others) and what the reports provide
• Demonstration on published public reports• Demonstration on Data Security• Demonstration of other data transformation use for
school improvement
41
Training for: Precode Plus
• Precodes answer sheets for EOC/EOG tests
• Designed to assist Testing Coordinators with specialized precoding needs
• Allows user to precode from roster listing, precode from a pick list of students or precode using an external file for a specific list of students
42
Training for: Reports • Historical Audit Rosters (Provides Scale Scores and Levels)• Membership/Course Codes (Provides membership rosters by subject)• Student Profile Report• Exit Standards Roster• Graduation Reports• ABC Summary• AYP Reports• Growth Summary Report (Classroom Scale Score Averages)• Students Repeating Courses Report• Student Rosters with Demographics from NC WISE• Classroom Scale Score Averages (Growth Summary from ABCTools)• Rerostering of WinScan Records for EOG/EOC • Goal Summary Reports in WinScan• Growth Target Reports (Why and why not to use these reports) • Detail AYP Reports for Web• Detail ABCs Reports for Web
43
ATEAM (Extra)
• Revised 01/21/10 RMC