SBAC RFP #7 Item Authoring and Item Pool Application 7_Proposal_Pacific...A. PROJECT...

SBAC RFP #7 Item Authoring and Item Pool Application

TABLE OF CONTENTS

Section 1: Letter of Submittal Letter of Submittal Certifications and Assurances (Exhibit A)

Informational Attachment

Section 2: Technical Proposal A. Project Approach/Methodology ................................................................................................ 2-1 Overview .............................................................................................................. 2-4

1.0. Application Requirements ........................................................................ 2-8 2.0. Application Development and Implementation ....................................... 2-12 3.0. Ongoing Application Housing and Maintenance .................................... 2-25 4.0. Item Authoring and Item Pool Application Software ............................... 2-32 5.0. Records of Decision Making .................................................................. 2-44 6.0. Annual Contractor Meetings .................................................................. 2-46

B. Work Plan ...................................................................................................... 2-46 C. Project Schedule ........................................................................................... 2-52 D. Deliverables................................................................................................... 2-58 E. Outcomes and Performance Measurement ................................................... 2-61 F. Risks .............................................................................................................. 2-62

Section 3: Management Proposal A. Project Management ....................................................................................... 3-1

A1. Project Team Structure/Internal Controls ................................................. 3-1 A2. Project Management Deliverables ........................................................... 3-8 A3. Staff Qualifications/Experience .............................................................. 3-22

B. Experience of the Consultant ........................................................................ 3-31 C. References .................................................................................................... 3-73 D. Contractor Intake Form ................................................................................. 3-77

Section 4: Cost Proposal A. Identification of Costs ...................................................................................... 4-1

Appendices

Appendix A: Functional and System Requirements Item Authoring and Item Pool Application Appendix B: Compliance Item Authoring and Item Pool Application Appendix C: Resumes


2-1

2. TECHNICAL PROPOSAL

A. PROJECT APPROACH/METHODOLOGY Introduction Pacific Metrics is pleased to submit this proposal to customize its off-the-shelf (COTS) web-based item authoring and item pool application (IAIP) to support the SMARTER Balanced Consortia’s item banking, item authoring, and development activities. Our proposal outlines our ORCA™ Development environment and the customizations that will be made to satisfy the SBAC functional and system requirements. ORCA Development is an item management platform that allows for scalable online assessment content editing and management and is provided through an intuitive Web-based interface. It allows test developers to construct content that is aligned to content standards. ORCA™ Development provides a versatile, flexible, and secure means of developing, managing, and exporting test content to the test delivery platform.

The ORCA™ Development application was a passion project for Pacific Metrics, and exemplifies our mission to bring lasting improvements to the assessment and learning environment through the thoughtful use of technology. The application was initially developed as an internally-focused product with the simple goal of streamlining the migration of content to online test delivery systems.

Years of internal collaboration and innovation between our assessment specialists and our software engineers resulted in a comprehensive IAIP application that now provides extensive assessment development support, including:

• Seamless integration with test delivery platforms • Robust import/export of content • Improved quality and consistency of content • Support various item types • Customizable workflow • Provide management reporting and tools – increased productivity

Pacific Metrics understands the demands placed on a technology system meant to support a wide variety of user roles and user types across a broad geographic area. We’ve provided OTS or COTS implementations of our ORCA Development application to leading companies in the assessment industry, all of whom had unique requirements related to system functionality.

Throughout the technical proposal, we refer to SBAC’s customized version of ORCA Development application as SBAC’s IAIP application.

Company Overview Pacific Metrics was founded over a decade ago with the mission to bring lasting improvements to educational practices and problems through the introduction of new technologies. We are essentially an educational R&D company with specialization in psychometric and assessment software, focusing solely within K-12 assessment. The company is based in the U.S., and all our staff (including engineers and software developers) are also U.S. based. Our software


2-2

engineers and program staff have developed technical assessment systems for classrooms, students working from home, and centrally administered secure summative tests. In the past decade, working with educators and IT professionals from the school level to the state department of education, we have architected, developed, deployed and operated large scale assessment platforms for the first statewide practice tests, statewide formative assessment, high stakes summative assessments, and experimental research programs. Our practice tests included the first Spanish audio capability in 2002. These and our summative tests include automated scoring of student responses to ELA, math and science items. Now in their third generation, our platforms have shown the flexibility and extensibility to deliver graduation tests to students displaced to a dozen states following Hurricane Katrina, deliver the highly innovative and interactive science and algebra ONPAR field tests, and successfully incorporate innovations in accessibility, item content, and automated scoring, working with four states in an Enhanced Assessment Grant (EAG).

We have routinely administered millions of assessments to hundreds of thousands of students in classroom-based and summative assessment environments, giving us experience across the entire assessment landscape from item development and content portability, to online student registration systems, to fast-turnaround scoring and reporting systems that blend human and machine scoring. We have written and deployed an online content development environment, designed, deployed and continue to maintain a state summative item bank accessed by multiple vendors, written interfaces and organizational designs for hand and machine scoring systems, maintained the official accountability reporting site for the State of Louisiana, and written the official state online testing decision tool for Council of Chief State Officers (CCSSO): http://www.ccsso.org/Resources/Digital_Resources/Online_Computer-Based_(Testing)_Decision_Making_Tool.html

Through this considerable breadth of programmatic experience and success in the assessment field, Pacific Metrics has learned a great deal about the integration of technology-based assessment systems with psychometrics and educational policy, including accountability. We believe to be unmatched in this skill set by any other organization or vendor. These accomplishments have only been possible due to the excellence of our research and engineering staff and the strong organization around our development processes imposed by project management practices. Pacific Metrics IT development practices emphasize the need for information systems to function in human organizations at multiple levels, incorporate requirements from multiple sources, and operate in tandem with other technologies.

As mentioned in the introduction, Pacific Metrics proposes to develop a COTS version of its ORCA Development content authoring environment to fulfill SBAC’s needs as outlined in the Technical description. This solution will be consortium-owned, and throughout the proposal we will refer to it as SBAC’s IAIP application.

Online assessment system development is an area of robust growth within the larger educational industry. Pacific Metrics is focused on developing innovative, leading-edge technologies to support the demands of the assessment world. Acknowledging that one of SBAC’s key technology priorities is the utilization of leading edge technologies, the system Pacific Metrics will customize for SBAC’s IAIP application employs dynamic, cutting-edge functionalities such as the creation of Technology Enabled Item content, support for dozens of world languages, and seamless publication to a variety of delivery systems while also remaining flexible enough to adapt to the changes inherent within the assessment industry.

http://www.ccsso.org/Resources/Digital_Resources/Online_Computer-Based_(Testing)_Decision_Making_Tool.html

http://www.ccsso.org/Resources/Digital_Resources/Online_Computer-Based_(Testing)_Decision_Making_Tool.html


2-3

The SBAC IAIP provides a scalable online assessment content authoring, editing, and management system provided through an intuitive web-based interface. It includes a user friendly HTML editor for authoring and editing test content, along with a fully functional equation editor and other flexible data and content display tools. The SBAC IAIP supports XHTML and also features real-time reporting and customizable printing capabilities to support content development needs across multiple states and organizations. Users are able to create multiple-choice, constructed response, and essay items with easy-to-use HTML editing tools in an end-to-end assessment process in a fully online environment that includes the development, publication, administration, reporting, and analysis of tests and test results.

SBAC’s IAIP application allows test developers to construct content that can be delivered through multiple test delivery systems. It provides a versatile, flexible and secure means of developing, managing, importing and exporting test content with many benefits including:

• Configurable process integration and workflow • Automation of time consuming and potentially error prone processes • Standardization and quality management of test development processes • Editorial processes for approvals • Management and end-to-end publication of test content

Pacific Metrics will provide the following services and deliverables for the complete and successful implementation of SBAC’s IAIP application.

Table 2-1: Components of the Project

Services Description

1 Application Requirements Functional Requirements Technical Systems Requirements

2 Application Development and Implementation Application Design Application Development Quality Assurance and Testing Data Portability – Item/Task, Stimuli, and Multimedia Import and export modules Application Deployment Training Materials

3 Ongoing Application Hosting and Maintenance Maintenance and Support Application Hosting Knowledge Transfer, Transition, and Turnover

Products Description

4 Item Authoring and Item Pool Application Software with full ownership by SMARTER Balanced


2-4

Overview of Services and Products: General Approach

Services 1

Application Requirements (Functional and Technical) Pacific Metrics will begin the customization process by first gathering and refining the Functional and Technical Requirements.

Functional Requirements serve dual purposes by providing both high-level business requirements and detailed software requirements. We plan to utilize an Agile iterative development process to customize SBAC’s IAIP application, and as such requirements are developed and refined as part of a specific iteration. Pacific Metrics will begin the functional requirements gathering process with the initial system requirements outlined in Appendix A of the RFP, and will work closely with SBAC and other stakeholders to ensure all requirements are clearly identified.

The Functional Requirements document will remain the main artifact of the requirements development process and will be maintained as iterative development is completed.

Pacific Metrics will also deliver a detailed Technical Systems Requirements document as part of the overall requirements effort. This document will include information relating to the following areas:

• Infrastructure and Client Requirements • Location and Interoperability Requirements • Security Requirements • Performance and Scalability Requirements • Availability, Reliability, and Redundancy Requirements

Services 2

Application Design, Development, and Implementation As noted above, Pacific Metrics plan to implement the Agile method for customizing SBAC’s IAIP application. This approach offers several benefits. Agile software development practices utilize a process of continuous planning and feedback to ensure that value is always maximized throughout development. Iterative planning and feedback allows for continuous review of and alignment to the system requirements, and ensures that changes are easily accommodated. By measuring and evaluating status based on working software, much more accurate visibility into the actual progress of projects is available. Finally, as a result of following an agile process, at the conclusion of a project is a software system that most effectively addresses SBAC’s needs.

The design of SBAC’s IAIP will be defined predominantly through interviewing key stakeholders of the system and receiving feedback and direction from the designated SBAC work group. Given the iterative process, any concerns or issues are found quickly in the process and can be addressed directly. Typical areas of consideration surrounding system design might include hardware and software architecture, specific technologies implemented to address key functional or technical requirements, and coding methodologies.


2-5

Quality Assurance and Testing Pacific Metrics has an experienced Quality Assurance (QA) team, offering a substantial depth and breadth of knowledge of diverse software systems. QA team members work closely with all members of the software development team to ensure a thorough understanding of not just the SBAC’s IAIP application system functionality, but also of its intended uses and the variety of user types and roles the system will need to support.

Our software quality assurance process is supported and enforced through the highest quality tools available to ensure consistent, reliable, and repeatable software releases. The QA team participates in a workflow-based problem tracking system that automatically assigns issues to the designated software engineers as they are submitted. This significantly increases the efficiency with which issues are identified, reviewed, and resolved.

Pacific Metrics will develop and submit a Quality Assurance Plan that provides insight into the processes used to review SBAC’s IAIP application. The plan will include information on processes such as testing methodology, unit testing, system testing, and User Acceptance Testing.

User Acceptance Testing (UAT) is a significant component of both Pacific Metrics’ testing methodology as well as the Agile software development process. Throughout the development lifecycle of SBAC’s IAIP application, SBAC staff will engage in UAT at the conclusion of each iteration. This process signals the final stage of testing performed on a specific part of the application prior to releasing it to a live environment. This testing is a critical component of development, as it will allow SBAC to ensure the application meets the requirements.

Data Portability Pacific Metrics understands that SBAC’s IAIP application must have flexible, robust capabilities with regards to data portability. To that end, our COTS solution offers seamless import from and export to multiple sources, which ensures that users from all member states can easily perform this function. Our technology team brings considerable experience in the area of batch content import, having worked on multiple large-scale assessment projects involving the import of thousands of items and their accompanying stimuli, graphics, metadata, and statistical information directly into the COTS application from a third party source. This system, customized for SBAC, will support the import of all content and associated information and stimuli.

The data import process is streamlined and efficient, and offers users “smart feedback” to identify any items or attributes that may be outside a valid range, or may be labeled incorrectly. This tool effectively minimizes risk to content validity and ensures the import process returns the desired and expected results.

Training Pacific Metrics understands that training is a critical component of the success of software systems. SBAC faces additional challenges due to the number of potential system users, their geographic diversity, and their varying levels of technology proficiency. By providing a variety of training materials, tools, and options, Pacific Metrics will ensure that users from all member states have the knowledge they need to use SBAC’s IAIP application efficiently and effectively.

We propose utilizing several different training strategies to support the transition to SBAC’s IAIP application. These include:


2-6

• User Guide—provides an overview of the application, including structure and technical specifications.

• Training Guides—provide in-depth descriptions of specific system functionalities, as well as “best practices” for use.

• Online Demonstration Site—provides a separate version of SBAC’s IAIP application that mimics the functionality of the live system.

• Self-Learning Tutorials—provides online step-by-step walk-through of system functionality covered in the Training Guides, with audio voice-over.

Application Deployment Pacific Metrics understands that we will need to provide all necessary hardware, software and applications according to the schedule outlined in the RFP. At the conclusion of this application deployment schedule, Pacific Metrics will deliver to SBAC an IAIP application ready to support mass data import, and end-user access to all required system functionalities, including all necessary hardware and software.

Services 3

Ongoing Application Hosting and Maintenance/Support Pacific Metrics recognizes that the SBAC’s IAIP application provides crucial support for all activities related to content authoring and maintenance for SBAC’s assessments. As such, we will address the need for ongoing maintenance and support of the system in the following areas.

Help Desk and production application support Pacific Metrics will provide three distinct levels of escalation for technical issues related to the production system, so that all issues are resolved as quickly as possible.

Production fixes for issues (“bugs”) identified with completed features For SBAC’s IAIP application, Pacific Metrics will support production fixes through a patch release process, prioritizing based on severity.

Release of application enhancements after the initial software release A formal process will be used for releasing full versions of SBAC’s IAIP application for both the initial release, as well as for any subsequent releases that may be required throughout the course of the contract.

Monitoring and system maintenance of the application within the hosting environment As a component of hosting services, Pacific Metrics will provide regular monitoring and application maintenance. A sophisticated alert system and a skilled team of system maintenance engineers ensure expeditious responses to any system issues.

Document updates related to production fixes and additional features. Pacific Metrics places the highest priority on communication with the client regarding any software fixes or upgrades that may occur throughout the course of a contract. We will provide detailed documentation to SBAC regarding updates made to the production version of the IAIP application.


2-7

Application Hosting Given the high-states nature of the data to be stored in the IAIP application, it is essential to utilize a Tier 1 solution. The COTS system Pacific Metrics will customize for SBAC was architected to address security issues related to the system and the data it manages. We utilize a robust IT infrastructure and world class Tier 1 hosting facility that will house SBAC’s IAIP application. At this facility, all customer hardware is placed in specifically designated cages to which only authorized staff members have access. The data center provider enforces several levels of security measures, including palm scans, verification of government issued identification, and access cards.

Knowledge Transfer, Transition, and Turnover As a leading software provider that specializes in application for the assessment industry, Pacific Metrics has extensive knowledge and experience in the area of management and maintenance of production software. This includes the transition of these responsibilities to an identified third party at the conclusion of a program or contract.

To ensure the smooth transfer of SBAC’s IAIP application to SBAC at the conclusion of the contract, Pacific Metrics will develop a thorough Transition Plan that outlines the specific activities involved in this transition. A Technical Systems Manual, addressing issues related to the managing and maintenance of the IAIP application, is another key component of this transfer that Pacific Metrics will provide. Using these two documents as a foundation, Pacific Metrics will also provide targeted, hands-on training to identified staff to impart a concrete knowledge of the system’s technical functionalities.

Recognizing that some support will be required after the initial transition of SBAC’s IAIP application, arrangements can be made for Pacific Metrics to provide Level 3 professional support provided by phone or through online meetings.

Products 4

Item Authoring and Item Pool Application Software Since its inception in 2000, Pacific Metrics has garnered a reputation for developing intuitive, creative, flexible software solutions for the assessment industry. The COTS product Pacific Metrics proposes for the SBAC was initially created as an internal solution to address the need to develop, store, and deliver digital content seamlessly to online testing platforms. Through an ongoing collaboration between content specialists, psychometricians, and software engineers, the IAIP application grew from a straightforward content authoring system to a full-scale assessment development application used by multiple companies and hundreds of individual users. Because the system is designed by assessment specialists, for assessment specialists, the functionality it offers demonstrates attentiveness and deference to the unique needs of this specific industry.

The IAIP application supports the development, review, storage, import, and export of both traditional and non-traditional item types and formats developed for a variety of purposes, across all grades and content areas. This includes all related stimuli and various graphic types. Content is stored and managed in distinct banks, allowing specific user permissions to be set for selected content and projects.


2-8

EXPECTED WORK STATEMENT Here we provide a complete description of our approach and methodology for each of the components identified in the RFP.

1. APPLICATION REQUIREMENTS

1.A Functional Requirements Pacific Metrics will construct a functional requirements document that provides detailed specifications for the Item Authoring and Pool system. As is highlighted in Section 2 of the Technology Proposal (Application Development and Implementation), we plan to use an Agile iterative process to develop the application, in which requirements are refined as part of each iteration. Recognizing this fact, we will develop an initial requirements document prior to beginning iterative development, but will refine the document as features (user stories) are completed as part of each iteration. The Functional Requirements document will remain the main artifact of the requirements development process and will be maintained as iterative development is completed.

The functional requirements document defines both high level business requirements and detailed software application requirements for the software system. The requirements are functional in nature, defining observable system behaviors. The product of this effort is a functional requirements document that helps drive the Agile development process. It helps to define design, implementation, validation, documentation, and deployment activities that are completed as part of each software development iteration.

Details of the Functional Requirements Document As part of a software development effort, Pacific Metrics works closely with a customer to establish and define the features and functions needed for a software application. Pacific Metrics captures and defines these requirements in a Functional Requirements Document for the software application. Major components of the requirements document include the following:

1. Introduction

a. Purpose of the document b. Scope of the requirements definition c. Definition of requirements vocabulary d. Reference materials e. Terms and acronyms

2. Overview - High level overview of the application being developed

3. Assumptions and Dependencies - Assumptions made in the requirements definition and dependencies of the requirements on external factors

4. Requirements Sections - A functional decomposition of the software application into one or more levels, in which the lowest level constitutes functional requirements for a given feature. A feature could be akin to a user story in Agile development or a Use Case in use case analysis. Feature requirements can include the following:

a. User interface requirements


2-9

b. Screen Mockups c. Business logic requirements d. Use case & flow diagrams e. Technical input and output requirements related to the feature, or references to these

types of requirements defined in the Technical Systems Requirements document. f. Non-functional and performance requirements related to the feature, or references to

these types of requirements defined in the Technical Systems Requirements document.

Requirements Development Process As highlighted earlier, since an Agile development process will be employed to develop the Item Authoring and Pool Application, functional requirements will be refined as part of the iterative development process. This means that requirements development is not fully completed prior to the beginning of software development, rather requirements are defined at a higher level up front and elaborated as part of the iterative process. We anticipate that the requirements will be refined as part of the Agile development, but that the Functional Requirements Document will reflect these changes.

During initial requirements development and during the iterative development process, several activities are expected to be completed related to requirements development. These activities include the following:

1. Requirements Discovery - Working with relevant stakeholders to discover requirements for the given application or feature.

2. Requirements Definition - Defining these requirements in the form of functional requirements and possible associated user stories and acceptance criteria.

3. Requirements Review - Reviewing requirements with relevant stakeholders to ensure they reflect the needed functionality.

1.B Technical Systems Requirements Pacific Metrics, using the SMARTER Balanced IT Systems Architecture and its guidelines, will deliver a detailed Technical Systems Requirements document as items are discovered through the Requirements Gathering Phase. The document will cover, but is not limited to, the following areas:

• Infrastructure and Client Requirements • Location and Interoperability Requirements • Security Requirements • Performance and Scalability Requirements • Availability, Reliability, and Redundancy Requirements

1.B.1 Infrastructure and Client Requirements Pacific Metrics will define infrastructure and client requirements that define the minimum supported hosting and client machine environment to successfully run the application.

This can include, but is not limited to:

• Hardware: CPU, memory, disk space • Software: operating system, library/framework and language dependencies


2-10

• Network: bandwidth, protocols, ports • Database: database management system type, version, and configuration

1.B.2 Location and Interoperability Requirements Pacific Metrics will define location and interoperability requirements that define the ability of the system to overcome problems brought about by the location of its elements and the distances between them.

This can include, but is not limited to: • Public interfaces to the system • Open communication methods (web services, etc.) • Standard data interchange formats (APIP, SIF)

1.B.3 Security Requirements We understand that one of SBAC’s key technology priorities is system and data recoverability. As such, our customized IAIP application for SBAC is designed with maximum data recoverability in mind. The system has been engineered to enable multiple methods of backup recovery. The system employs the ability to recover from a point in time backup and has the flexibility to be backed up manually as well as automatically to fulfill user requirements.

Pacific Metrics will define security requirements that define the ability of the system to reliably control, monitor, and audit what entity can perform what actions on resources and the ability to detect and recover from failures in security mechanisms.


• Authentication • Authorization • Monitoring • Auditing • Encryption • On-site security

1.B.4 Performance and Scalability Requirements Pacific Metrics will define performance and scalability requirements that define the ability of the system to predictably execute within its mandated performance profile and to handle increased processing volumes.


• Response times • Throughput • Peak load behavior • Distributed processing


2-11

1.B.5 Availability, Reliability, and Redundancy Requirements Pacific Metrics will define availability, reliability, and redundancy requirements that define the ability of the system to be fully or partially operational as and when required and to effectively handle failures that could affect system availability.


• Planned/unplanned downtime • Clustering • Load balancing • Disaster recovery

We acknowledge that open licensing is a fundamental requirement of SBAC’s IAIP application as indicated in the key technology priorities. In the development of the system, we recognize that the classes of technologies which are incorporated will depend on those that have appropriate Open License models. While some fundamental components of the system may require Closed or Commercial solutions, there must be the ability to trade off technologies in the architecture such that Open Sourcing of the IAIP application can eventually be achieved.

2. APPLICATION DEVELOPMENT AND IMPLEMENTATION Agile Development Approach The efforts and process Pacific Metrics plan to incorporate for the successful delivery of SBAC’s requirements will involve the practices around Agile software development and the Agile Manifesto.

Twelve principles underlie the Agile Manifesto, including:

• Customer satisfaction by rapid delivery of useful software • Welcome changing requirements, even late in development • Working software is delivered frequently (weeks rather than months) • Working software is the principal measure of progress • Sustainable development, able to maintain a constant pace • Close, daily co-operation between business staff and developers • Face-to-face conversation is the best form of communication (co-location), however

tools such as video conferencing and GoToMeeting® will be leveraged for distributed teams

• Projects are built around motivated individuals, who should be trusted • Continuous attention to technical excellence and good design • Simplicity • Self-organizing teams • Regular adaptation to changing circumstances


2-12

Figure 2-1: General Overview of the Agile Development Process

The Pacific Metrics method of Agile will break tasks into small increments with less formal planning, and do not directly involve long-term planning. Iterations are short time frames (timeboxes) that will last four weeks. Each iteration will involve a team working through a full software development cycle including planning, requirements analysis, design, coding, unit testing, documentation and acceptance testing when a working product is demonstrated to stakeholders. This minimizes overall risk and allows the project to adapt to changes quickly. An iteration may not add enough functionality to warrant a market release, however the goal is to have an available release (with minimal defects) at the end of each iteration. Multiple iterations may be required to release new features.

Team composition will be cross-functional and self-organizing without consideration for any existing corporate hierarchy or the corporate roles of team members. This means that team members take responsibility for tasks that deliver the functionality an iteration requires, and they decide individually how to meet an iteration's requirements.

Recognizing the formal documentation required by the project, Pacific Metrics will still emphasize regular and consistent communication over written documents. The Pacific Metrics Project Manager will need a dedicated SBAC representative for the life of the program by acting as a single point of contact for managing and directing customer decisions needing to be made and provided to Pacific Metrics. This person must be determined by SBAC prior to project kick-off to act on SBAC’s behalf and must make a personal commitment to being available for the Pacific Metrics team to answer such things as mid-iteration problem-domain questions. At the end of each iteration, stakeholders and the SBAC representative will schedule a time to review progress and re-evaluate the priorities with a view to optimizing SBAC’s return on investment (ROI) and ensuring alignment with customer needs and company goals.

If change to the scope is requested by SBAC it will go through a formal Change Request process requiring sign-off. Furthermore, the Pacific Metrics Program Manager will deliver bi-weekly reports to SBAC’s point of contact that will give clear visibility to status of hours, the progress of scope completed, and projected timelines. As the process of Pacific Metrics accepts


2-13

change throughout the lifecycle of SBAC’s project, this bi-weekly report will be a living/breathing document and will therefore change over time to accommodate any change provided by SBAC’s dedicated point person.

Lastly, Pacific Metrics will use a routine and formal daily communication among team members called a Scrum meeting. This specifically includes the customer representative and any interested stakeholders as observers. In a brief session, team members report to each other what they did the previous day, what they intend to do today, and what their roadblocks are. This level of communication exposes problems as they arise. Pacific Metrics emphasizes working software as the primary measure of progress. This, combined with the preference for regular communication, produces less written documentation than other methods. This method encourages stakeholders to prioritize wants with other iteration outcomes based exclusively on business value perceived at the beginning of the iteration (also known as value-driven).

2.A Application Design Because Pacific Metrics follows Agile Design practices, our software designs usually do not go through a large and resource-intensive design phase that may, or likely not, meet all of the stakeholder’s requirements. Pacific Metrics development process focuses on delivering working, useful software in iterations, so design is mainly emergent and evolves with the software (because of the nature of Agile development). However, this allows the system design to emerge over time, permitting it to take advantage of new technologies and fulfill new requirements.

There are a range of Agile Design practices that Pacific Metrics follows, done in a collaborative nature with stakeholders of the system. This is detailed in the following figure:

Figure 2-2: Agile Design Practices

• Architecture Modeling: modeling (usually light) at the beginning of a project to identify critical architectural issues

• Iteration Modeling: modeling at the beginning of each iteration to identify a strategy for that iteration


2-14

• Solution Modeling: modeling to think through an aspect of the solution • Test-first Design: write a test and enough domain code to fulfill the test • Refactoring: small changes to a part of a solution, improving quality without changing

the semantics • Continuous Integration: automatically compile, test, and validate components of the

solution whenever a component changes

Whenever design work is being completed the Architect or Engineer uses the appropriate tools to accomplish the job without hindering the effort. Whiteboards, index cards, and diagramming tools can all be used to work out and communicate the solution to stakeholders. They also consider system qualities (non-functional or technical system requirements) concerns such as hardware and software architecture, technologies used to meet defined functional and technical system requirements, coding methodologies, programming languages, and component/object-based decomposition of the system. Pacific Metrics will deliver software design documents and technical specifications, where appropriate (e.g., interoperability) that affect the software implementation as driven by the architecture.

The design documents can include, but are not limited to:

• User Stories • Domain models • Flow diagrams • Enterprise integration models • Entity relationship models • Class diagrams • Logical and Physical Data models

We understand that system flexibility is one of SBAC’s key technology priorities. Pacific Metrics recognizes the flexibility required in a system meant to be utilized across states and by users with a variety of proficiency levels and technology resources. The SBAC IAIP application will be customized in a flexible and modularized fashion which will benefit SBAC by minimizing the programming required to make future modifications or customizations.

2.B Application Development As detailed in Section 2, Agile Development Approach, Pacific Metrics creates the software implementation iteratively going through each phase of a typical system development lifecycle. This method constructs the system as fully functional software that satisfies the business and technology requirements in phases while adhering to the defined software design in a flexible manner.

Pacific Metrics has deep knowledge in software engineering practices, thus our development approach has a variety of tasks that engineers are responsible for:

• Acquiring and installing systems environments • Creating and testing databases • Preparing test files • Coding • Compiling, building and deployment • Unit-testing


2-15

• Load-testing

Acquiring and Installing Systems Environments Pacific Metrics software engineers have the knowledge and expertise to acquire, install, and configure a variety of system environments, application servers, etc. We have expertise in industry standard environments, and use some of the most widely used web-based languages, frameworks, and systems, such as:

• Apache HTTPD server • Java SE and Java EE technologies (EJB, JPA, JMS, etc) • Apache Tomcat • JBoss Application Server • Glassfish Application Server • Web Services (SOAP and REST) • PHP • Perl

Creating and Testing Databases Our engineers have significant expertise in creating highly-available and resilient database solutions with both MySQL and Oracle database management systems, clustering, redundancy, and data integrity.

Preparing Test Files Pacific Metrics software engineers build and create test files during development for consistent and reliable feature development and testing. Test files are included as part of integration work and becomes part of the project and build process.

Coding Engineers follow coding methodologies and standards in order to keep a healthy, readable, and maintainable code base. Engineers also use a set of standard development tools as part of their efforts, including (but not limited to) the Eclipse IDE with Java EE and Web Application Development support, Subversion code management client, and Test Track Pro issue management client.

Compiling, Building and Deployment Our software engineers follow best practices and tools for compiling, building and deploying software. This includes using industry-standard tools like Ant and Maven to build and distribute software artifacts. This facilitates the use of Continuous Integration software, particularly Jenkins, so that the software remains in a stable state and issues are discovered quickly. In addition, our engineers have the knowledge and experience setting up repositories for sharing deployable java artifacts like JAR, WAR, and EAR files, and the expertise in creating linux-based software repositories for distributing software like RPM (Red Hat) and DEB (Debian and Ubuntu) files.

Unit-Testing Pacific Metrics engineers employ unit testing as part of their development efforts, particularly the XUnit testing frameworks, and specifically JUnit for Java development. This allows us to continually refactor our software while maintaining consistent behavior.


2-16

Load-Testing Apache JMeter is used regularly after software builds in order to assess software and application performance and capacity. It also allows us to check our systems and catch performance issues that may have been introduced during the development process.

The software implementation deliverables will include, but is not limited to:

• Software product • Development tools • Support tools • Data migration software • Integration software • Installation software

2.C Quality Assurance and Testing Pacific Metrics has a strong Quality Assurance department that ensures the quality of all products is of the highest standards. System Quality Assurance falls into three categories:

1. Functional requirements and specifications (i.e., does the system do what it was designed to do and under what conditions?)

2. Usability requirements (i.e., is the system easy to use and are instructions and guides clear and concise?)

3. Adverse scenario testing (i.e., how does the system react to external, non-controllable events such as a Web browser freeze up, or power loss to the local computer?)

Procedures for System/Software Quality Assurance Quality Assurance for the functional aspects of the software is best accomplished outside the technology team to assure a system with appropriate checks and balances. As such, Pacific Metrics corporate organization structures the reporting for this function through our Project Management group rather than through Technology.

Software Quality Assurance Tools Our software quality assurance process is enforced through the use of tools that contribute to consistent, repeatable software releases. Software configuration management tools, a problem-tracking system, and test case repository tools ensure that Pacific Metrics products can be built reliably. Our software repository is integrated with our error-tracking system and our communications systems so that development information and software check-ins are always available to the technology team and project leaders. Tools that will be utilized for the ORCA Development application include the following:

• Subversion repository system • TestTrack Pro bug tracking and reporting system • Rally Project Management software

Rally Test and Defect Management delivers integrations with test and defect tracking applications that allow your team members to synchronize the testing and QA progress made during iterations with details managed by departments using specialized tracking tools. Rally provides agile project management, including story and defect tracking, to assist customers in


2-17

managing the rapidly changing flow of requirements and issues that are common in Agile development processes.

A workflow-based problem tracking system automatically assigns issues to developers as they are submitted, eliminating any lag time related to manual review and assignment of bugs. Because our problem tracking system is based on workflow, we ensure that verification and validation is completed before each issue is closed. Test cases, stored in a repository, are assigned based on project lifecycle and methodology. The test case repository is used to generate detailed reports.

Software Quality Assurance: Quality Requirements Pacific Metrics notes that proper functioning against specifications is an important aspect of software design and quality assurance. However, it represents only one dimension of what is required. Below, we note the four principles that will guide development and testing of the

• Functionality: the ability of the system to do the work for which it was intended, demonstrated through functional testing methodology

• Performance: the response time, utilization, and throughput behavior of the system • Availability: the measure of time that the system is up and running without system issues • Usability: the ease of use for the end users of the system, including “usability”, efficiency,

helpfulness, and control

Particularly with respect to the final bullet, we note that the overall success of the system requires that users, who may have little training on software, will find the system easy to use.

Software Quality Assurance Plan Pacific Metrics will develop and submit a Quality Assurance Plan that details the test plan for the Item Authoring and Pool Application, including unit testing, system testing, user acceptance testing, functional testing, performance testing, and operational testing. The manual will contain the following sections:

• Test plan and procedures for quality assurance activities at each phase of the project • Definition of how testing activities integrate into the Agile development process. • Sections that outline each type of testing: unit, system, user acceptance, functional,

performance and operational (more details provided below). • Requirements traceability matrix related to functional and technical requirements of the

system. • Test case specifications.

Testing - Software Quality Assurance: Test Methodology Test methodology refers to systematized, repeatable procedures for exercising software features and functions. It is important for testing to follow a lifecycle-appropriate methodology. The idea here is that at early stages of the product lifecycle, it makes the most sense to test individual units in isolation—checking internal logic and behavior, whereas during later stages, it makes sense to check that all individual units fit together and operate correctly as an integrated whole.

The following tests will be performed by Pacific Metrics on the system:

• Unit Tests: checks isolated code for internal logic


2-18

• Functional Tests: confirms the functionality of project components against requirements, including:

o User inputs and destructive testing o Use case and scenarios o Data flow through integrated components o Report generation

• Requirements Tests: validates successful completion of the software requirements for the project

• Regression Tests: demonstrates that bug fixes were successful and did not destabilize the system

• System Tests: establishes that components are integrated correctly and that the system behaves as specified

• Performance tests: demonstrates load-handling capabilities of the system • Documentation verification and validation: assures that project documentation is

accurate and appropriate for the project’s use and audience

User Acceptance Testing As part of Pacific Metrics’ Agile development process, SBAC staff will participate in User Acceptance Testing (UAT) at the conclusion of each iteration. UAT is a final stage of testing performed on specified parts of the system, defined within each iteration, prior to releasing it to a user environment.

Acceptance testing allows customers to ensure that the system meets their requirements and expectations as each acceptance test represents some expected result from the system. Acceptance tests are generally performed as "black box" tests where a user tests specified inputs into the system and verifies that the resulting outputs are correct, without knowledge of the system's internal workings. These inputs are made up of multiple tests or test cases that are defined in the beginning of each iteration with the team.

Furthermore, test cases consist of a sequence of steps to perform that emulate the use case that is being tested; it also contains input data (if required) as well as the expected output. The result of a test case is either a pass or a fail.


2-19

Table 2-2: Examples of Test Case Fields

Test Case Field Description Test case ID Record for the test case Unit to test What is to be verified

Assumptions Assumptions related to

feature functionality Test data Variables and their values

Steps to be executed List of necessary steps in

validation process

Expected result Result specified in

requirements document

Actual result What happened during

testing

Pass/Fail Indicates if system met

Expected Result

Comments Additional information

provided by tester

Acceptance tests are created from user stories, which will in turn be derived from the technology requirements SBAC and Pacific Metrics developed over the past several months. During an iteration, the user stories selected during the iteration planning meeting will be translated into acceptance tests. A user story can have one or more acceptance tests; the central goal is to ensure the functionality works and is ready for a product level environment.

A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created with each iteration. SBAC staff will be responsible for verifying the correctness of the acceptance tests developed, and determining when a test is accepted.

Sample Acceptance Test statement:

A [named user role] can [select/operate] [feature/function] so that [output] is [visible/complete/etc.] and leads to a Yes/No decision for user and developer

After UAT is completed, the SBAC either accepts the section of the system or identifies further changes that are required. After these subsequent changes are completed, either the entire test is performed again or just those portions in question are retested. Pacific Metrics will leverage Automated Tests where possible. In some cases, portions of the system may be accepted while others are sent back for more work. The details of this process will be contained in the Acceptance Test Plan.

Acceptance Test Plan Pacific Metrics proposes that a formal kick-off meeting for the configuration work be held. At this meeting, Pacific Metrics will submit an Acceptance Test Plan containing a formal, documented testing process to be followed by both teams. Defining this process will help the teams understand what is expected of them throughout the system configuration and validation process.

http://www.extremeprogramming.org/rules/iterationplanning.html


2-20

The Acceptance Test Plan will include: • SBAC responsibilities: the activities SBAC is responsible for and what others are

responsible for. • Acceptance criteria: what criteria will be used to decide whether the system is

acceptable or not. • Acceptance Test Workplan: the activities associated with the acceptance test, when

they will begin and end, who is responsible, and so on. In other words, you define the workplan for the test.

Further, Pacific Metrics proposes that a more detailed Acceptance Test Plan be submitted to SBAC at the beginning of each iteration. This plan will include the specific activities, unique to the iteration, that both Pacific Metrics and SBAC staff are responsible for completing.

2.D Data Portability – Item/Task, Stimuli, and Multimedia Import Module Data Portability is recognized as an imperative requirement of the Item Authoring and Pool Application. Pacific Metric’s ORCA Development system utilizes a Item/Task, Stimuli, and Multimedia Import Module in order to provide that support.

All item content and metadata and associated stimuli will be imported into the Item Authoring and Pool Application using 2 basic functions:

1. Item Content Upload (i.e. Item Data Upload) 2. Item Metadata Upload (i.e. Batch Data Upload)

Item Content Upload Item Content is uploaded in the QTI/APIP format as defined by the SBAC Assessment System Architecture. Per the QTI specification, the item package format is a ZIP file. The item package also contains a manifest file which specifies the particular items/tasks and stimuli that are to be uploaded from that package. Each item/task and stimuli is represented in its own XML file which contains the content and references to any associated graphics or multimedia components (e.g. flash files).

After the item package is uploaded, the user receives a feedback report. This report indicates if all item/task and stimuli content was parsed successfully, some high-level metadata regarding each item/task and stimuli that was uploaded, and whether each item/task or stimuli was created new or updated an existing item/task or stimuli.

Item Metadata Upload Item Metadata is uploaded in a CSV file format. The upload page contains specific instructions on which specific metadata are supported and the data types of each field. For example, if a field is coded to a discrete set of values, such as a grade being one of K-12, the relevant values are shown in the instructions. Once the metadata file is uploaded, a report is displayed indicating each item which has metadata in the update file, the value of each update field, and also indicates any values that are out of range for coded or constrained fields. If there are any errors, none of the metadata is uploaded, and the option to save the changes is not available. Once the user has reviewed the prospective updates, and if there are no errors in the metadata upload file, the user may save the changes.


2-21

2.E Application Deployment Pacific Metrics understands that as a deliverable of this contract, all necessary hardware, software, and applications must be installed according to the following schedule:

Date Application Deployment Activity

August, 2012

Deployment of the application to support item development for the Pilot Test. This will include efforts to load items/tasks and their associated stimuli, multimedia, interoperability tagging (e.g. text-to-speech tags) and translations (Braille, languages); conduct SMARTER Balanced review processes; and edit items/tasks, stimuli, and all associated assets and attributes. Test items/tasks will be initially developed and stored using external application platform(s) and must be loaded into the new Item Authoring and Item Pool Application starting This will also include functionality to support item development within the item bank system (item writing, multimedia fulfillment, item tagging, item review, expert and panel review) for the Field Test. Deployment of the application to support item writing for the Field Test.

February, 2013 Execution of the Pilot Test requiring data exchanges with the Test Delivery Application for computer based testing.

March, 2014 Execution of the Field Test requiring data exchanges with the Test Delivery Application for computer based testing.

At the conclusion of this application deployment schedule, Pacific Metrics will deliver to SBAC an IAIP application ready to support mass data import, and end-user access to all required system functionalities, including all necessary hardware and software.

2.F Training Pacific Metrics understands the need for a variety of training resources to be provided for the item authoring application so that all types of users can receive the information needed to successfully use the system.

The IAIP application will serve as a mission-critical application supporting SBAC assessment functions. As such, prior to its implementation, identified users will require an understanding of the system as a whole as well as specific training on those aspects of the system that directly pertain to their work area. In addition, staff may need access to ongoing support as they transition to this work environment.

Our recommendations for training and support include the provision of documentation/user guides, training, and technical support as follows.


2-22

Proposed Trainings

Training Type Notes Overview Training for SBAC stakeholders on the General features of the IAIP application

4 hour web training conducted by Pacific Metrics in conjunction with the general release of the application and prior to the initial import of legacy content.

Training/Demonstration for SBAC’s identified system users on import processes/retrospective conversion of items, art, and statistics

4 hour web training conducted by Pacific Metrics on the processes and workflows for retrospective conversion.

Repeat of the Overview Training for SBAC’s identified users on the General features of the IAIP application.

We recommend repeating this 4 hour onsite training after all the configurations have been completed, and prior to the SBAC staff undertaking the specific training and utilizing the IAIP in their daily work. Specific Training Guides for Targeted Areas

(for example) Item Creation and Submission Item Review and Submission Form Creation Reports and Batch Data Upload Art Creation and Submission

Targeted Training Guides for each area provided to the appropriate group of SBAC staff. Exact training guides to be determined.

Online Demonstration Site Pacific Metrics will develop a demonstration site that can be accessed by authorized SBAC Trainers and used on an ongoing basis as they train other authorized personnel to use the IAIP application. The site will be populated with sample content to allow users to explore all areas of the system.

Access to this site will be through a link on the SBAC homepage. Users can access this site with the appropriate credentials and login information. Users will be able to complete all tasks that would normally be performed in the IAIP application. For example, a content developer from any member state could create a test blueprint, and an item administrator could search for items with specific data characteristics.

User Guide Pacific Metrics will develop a User Guide designed to help users at all levels learn the functions of SBAC’s IAIP application. The guide will feature a glossary of terms, icons and system screen shots, and will encompass multiple sections to address the following areas:

• overview of the system • information about the system’s navigation tools • structure of the application • technical specifications • Help Desk information • suggestions for optimal efficiency

In conjunction with the iterative software release schedule, Pacific Metrics will draft and submit to SBAC an updated version of the User Guide. Pacific Metrics will work closely with SBAC to


2-23

further define the format and content specifications for each section of the guide. The User Guide will go through our internal editorial review process prior to being submitted to SBAC. Consistent with established review procedures, we will build three rounds of SBAC review into the schedule for this task, assuming that SBAC will want to review the draft document.

Round 1: This round of review is intended to finalize the basic structure of the document and verify that the overall content of the document is correct and consistent with SBAC requirements.

Round 2: This round of review is intended to ensure that all edits from Round 1 are incorporated properly, and to confirm that all information and instructions provided are accurate.

Round 3: This final round of review is intended to ensure that all previous requested changes are incorporated properly and that no errors were introduced in the process. Also in this round, final cross references will be made throughout the document, including table of contents, header titles, general information, and page references.

To ensure accuracy in the process, all SBAC edits for each round of review should be consolidated and provided to Pacific Metrics in a single electronic file. The User Guide will be posted to the SBAC website for easy access to all users.

Training Guides In addition to the User Guide, Pacific Metrics recommends the development of targeted Training Guides to address specific user roles or tasks in the application. These Training Guides move a step beyond the User Guide in that they provide both the physical processes the application offers, as well as best practices that ensure the most efficient and effective methods are used to accomplish tasks. Examples of targeted Training Guides Pacific Metrics has created are listed below:

• Item Creation and Submission • Item Review and Submission • Form Creation • Reports and Batch Data Upload • Art Creation and Submission


2-24

Figure 2-3: Sample from Training Guide

Self-Learning Tutorial Software Pacific Metrics is experienced in creating self-learning tutorials and using Camtasia® Studio software. Pacific Metrics will design a tutorial module for each of the different users – for example, item writer, artist, administrator, etc. – accessible through their menu page. The tutorial will provide a step-by-step walk-through of each person’s responsibilities and how that person can address these responsibilities through the system. A voiceover explanation of each process step will also be included.

A detailed plan will be provided to SBAC for review and approval prior to script creation and production approval. The plan will include a timeline and a description of activities that will be conducted at each stage of the process. As part of this plan, four reviews will be included; an initial review of the script, finalization of the script, and two online reviews. The script will go through a copy edit prior to the first submission. The name and resume of the personnel responsible for developing the training modules will be included in the plan.


2-25

3. ONGOING APPLICATION HOSTING AND MAINTENANCE

3.A Application Maintenance and Support As defined in the technical work plan, we will plan an initial software release of the Item Authoring and Pool application in August of 2012. At this point, the application will be considered in production and will require ongoing support. Support of the application will include the following activities, and will be provided for the remainder of the contract period, August, 2012 – September, 2014:

1. Help Desk and production application support 2. Production fixes for issues (“bugs”) identified with completed features 3. Release of application enhancements after the initial software release 4. Monitoring and system maintenance of the application within the hosting environment 5. Document updates related to production fixes and additional features.

These areas are described in greater detail below.

Help Desk and Production Application Support Pacific Metrics recognizes the need to support our existing technology products in a production environment and employs a well-defined process for escalating and managing technical issues identified on its production systems. The process defines three levels of escalation for technical issues such that the issues are resolved in a timely fashion by the appropriate staff. Pacific Metrics will provide all three levels of Help Desk support.

The first level typically consists of the customer service and help desk function that responds to general issues. We anticipate that many of the SBAC Level 1 questions will relate to user-questions for SBAC’s IAIP. We will ensure there is trained Level 1 staff that is able to answer these questions.

Technical issues that cannot be resolved by the Level 1 Application help desk are escalated to Level 2, which consists of experienced product support engineers at Pacific Metrics. These engineers understand all functional and non-functional aspects of the system and can resolve more complex issues related to use of the system. Technical issues that cannot be resolved by Level 2 are escalated to Level 3, which consist of software engineers responsible for system development and maintenance. Level 3 issues are typically ones that require a system modification to resolve. Response times and resolution times are defined for each level of escalation, and these times depend on both the priority and severity of the issue being managed. As defined in greater detail below, Pacific Metrics’ production support process allows for the quick deployment of system fixes outside of a regularly scheduled product release for those technical issues that require one.

Production Fixes A complex online application normally requires production fixes for issues identified with the application. The purpose of a production fix is to correct issues identified with existing features of the application, to ensure the application functions as designed and at an expected level of performance. A production fix is not meant to add additional features to the application, which would rather be managed through a full release of the application software. Pacific Metrics typically releases production fixes in a patch release of the application; a software release that is incremental in nature and is limited to production fixes. Pacific Metrics will support production


2-26

fixes through a patch release process for the Item Authoring and Pool Application after the first production release in August of 2012. Fixes will be limited to application issues as defined above.

The patch release process will prioritize application issues based on both application severity and customer priority. These attributes will define the timeframe surrounding when an issue is resolved and delivered in a patch release. Pacific metrics will work with SBAC to define the periodicity surrounding patch releases, associated with different priority/severity levels. For instance, an issue that impacts most users in a significant fashion, like an application outage, would likely require a patch release as soon as possible.

Release of Application Enhancements Pacific Metrics will us a formal software release process for releasing full versions of the Item Authoring and Pool Application to the production environment. The full software release process will be employed for both the initial software release in August 2012 and in any subsequent releases that are required. A full software release contains significant features and functionality enhancements over a prior version of the software application. The release process is strongly tied to the application development process, and hence includes all components of that process such as analysis, design, implementation, validation, user acceptance testing, etc. The work plan outlines the anticipated releases that will be needed for the customized SBAC IAIP application. Pacific metrics will work with SBAC to ensure that the full software releases as planned integrate into larger activities related to the Item Authoring and Pool Application.

The type of enhancements we expect to be released after the initial release in August of 2012 are enhancements planned in original scope of work but not implemented in the initial release. If SBAC identifies additional application enhancements that are required as part of this effort but not covered by the original scope of work, the defined change control process will be used to scope, cost and manage these enhancements.

Monitoring and Application Maintenance within the Hosting Environment Pacific Metrics will provide regular monitoring and application maintenance as part of hosting the Item Authoring and Pool Application. We describe each function in greater detail below.

Application and System Monitoring: All applications and hardware components within Pacific Metrics’ hosting environment are monitored using SNMP manager probes and all software systems are monitored by additional health checking capabilities that are specific to the application. All alert types have associated distribution lists and notification occurs across multiple channels including but not limited to: 1) email, 2) SMS, 3) monitoring management consoles. The monitoring system has components that are located both within and outside of the production environment, providing a comprehensive capability to detect system and application issues. The system also contains capabilities to generate and escalate system alerts based on predefined rules and monitoring triggers.

Pacific Metrics has production staff that is responsible for observing and managing the data and alerts produced by the monitoring system. This staff in responsible for initiating required maintenance and/or escalation procedures related to the reported data and alerts. Escalation is the responsibility of IT staff who generally contact customer service representatives that act as liaisons to the customers that are potentially affected. Customers are kept apprised of the situation until it has been resolved. Subsequent to resolution, the customer is provided with a detailed post mortem of what caused the problem and how it was resolved.


2-27

Application Maintenance: Pacific Metrics production staff is responsible for all regular maintenance related to its production applications. This includes but is not limited to the following activities:

1. Regular maintenance and servicing of all production hardware components including servers, switches, firewalls, load balancers, etc.

2. Regular management and patching of production systems such as servers, operating systems, virtualization environments, network topology, monitoring and reporting systems.

3. Provisioning and installation of additional IT infrastructure to support increased application capacity.

4. Remediation and correction of production system failures. 5. Security management and monitoring.

As described in the next section, this proposal assumes Pacific Metrics will be hosting the Item Authoring and Pool Application.

3.B Application Hosting The ORCA family of assessment products has been designed to address concerns with respect to the security of the computing systems and data managed by those systems, the high availability of the applications and the requirements surrounding the use of the applications in a production environment.

Pacific Metrics employs a robust IT infrastructure and a Tier 1, world-class hosting facility to host and deliver its production applications. This facility has the capacity to support the consortium’s needs during the scope of this engagement, thereby reducing the hosting costs for member states which we understand is one of SBAC’s key technology priorities.

This section highlights these capabilities in greater detail and how they will help to meet the needs of the Item Authoring and Pool Application.

Access to ORCA applications and Information Security Recognizing the importance of maintaining the integrity of highly sensitive information as one of SBAC’s key technology priorities, Pacific Metrics’ solution provides data integrity as an inherent quality of the system. All system and data access is role-based and password protected to ensure that only authorized access is allowed. Using our RDBMS in the backend minimizes data duplication and further enforces data security.

Pacific Metrics employs multiple techniques to protect the systems and data associated with its assessment products, including the COTS application that would be used for SBAC’s IAIP Application. At our hosting facility, all customer hardware is segregated into separate cages that only authorized staff have access to. Authorized staff includes Pacific Metrics and third party managed hosting personnel that have been assigned to a specific customer. Pacific Metrics’ data center provider enforces several physical security procedures to gain access to the facility, the raised floor room, and lastly the cage. These procedures include:

• Palm scans • Verification of government issued identification • Access cards


2-28

All customer cages have dedicated network connections. All customer data is stored in databases installed on hardware in the segregated cages. All sensitive data (i.e., passwords, SSNs, etc.) is stored in the database encrypted. Accounts and passwords to the database and servers comply with enterprise recommendations for length and content and are regularly changed. Database and server security logs are monitored and any attempt at breaches results in alerts being escalated to Pacific Metrics and our contracted IT service provider who delivers managed services for our technology solutions, including email, hosting servers for applications and data management, and other aspects such as load balancing and firewalls.

Information security, privacy and production access are also enforced by Pacific Metrics IT staff. Policies include:

• Only IT staff (Pacific Metrics and IT service provider) has access to the production environment

• Only IT staff assigned to specific customers have physical access to those production cages

• Only IT staff assigned to specific customers are given credentials to hardware residing in those production cages

• Database and server credentials must satisfy enterprise recommendations for length and content

• Database and server credentials must be changed at regular intervals • Database storage of sensitive information must be encrypted • All database and servers must log security breaches • All security logs must be audited by monitors at regular intervals • Deployment of applications, tools, and etc. into the production environment must be

performed by IT staff

Pacific Metrics understands the increasing security needs for content development and item banking systems are one of SBAC’s key technology priorities. Our system has been designed with security enhancements in mind and provides excellent security on multiple levels:

1. Application security – System access is role-based and password protected. Password control is active.

2. Network security – All web communications are encrypted via HTTPS. 3. Site Security – The system is hosted in a tier one facility with full physical security

measures including the enforcement of physical access to the site. Security policies are reviewed quarterly and updated accordingly.

IT Infrastructure Capabilities With respect to IT infrastructure, Pacific Metrics offers value in terms of the significant investment the company has made in architecting and implementing a robust production hardware stack that will scale to meet application needs and will ensure the optimal performance of deployed production systems.

Pacific Metrics typically deploys applications in a hosted manner, and our production infrastructure has been architected as a high availability environment designed to provide 24/7/365 uptime (99.7%). Our ORCA platform, which includes SBAC’s IAIP application, has been load tested using industry adopted tools (JMeter) to validate that both the production hardware stack and the applications can handle tens of thousands of simultaneous users. All hardware and applications are monitored using SNMP clients that generate real time health alerts if problems are detected and provide daily load reporting that is utilized by operations staff


2-29

to constantly assess the state of the system. The ORCA platform is deployed onto server clusters that utilize server virtualization to allow for easier scaling of the production applications. These applications are also load balanced using commercial hardware and can be horizontally scaled by adding additional virtual and physical servers.

The Pacific Metrics production infrastructure is also designed to be high available (HA). The network infrastructure is redundant and consists of hardware that is capable of automatic failover. All hardware appliances (i.e., firewall) are redundant and operate in a high availability mode eliminating any possibility of a single point of failure. The network circuits are redundant, served up by multiple carriers, and can burst to GBit speeds. As described above, the production applications are deployed on a cluster of servers that are load balanced. If one of the servers fail the sessions currently being served by it are redistributed to other servers.

The production infrastructure employs comprehensive, automated backup and restoration capabilities. All data is backed up based on a defined schedule, to both onsite and offsite archive data stores. In the event of data loss, defined procedures are used to restore the latest archived data. Backup schedules and techniques are defined to allow for the efficient use of archive resources while ensuring data is archived on a schedule necessary based on the priority of the data.

The value of these capabilities will be realized in the ease of deployment and quality of performance delivered by our proposed solution, and in the level and consistency of delivery that users will experience.

IT Infrastructure Scalability Pacific Metrics develops all of our IT systems to be flexible and scalable to support large populations of users. As noted in SBAC’s key technology priorities, the customized IAIP solution for SBAC fulfills the goal of being a highly available and scalable system. The application was designed on a flexible systems architecture that enables it to be both highly available and scalable by emphasizing a modular design. Greater availability and scalability are easily achieved by simply adding more web and database servers to the system, without overhauling the system architecture.

Pacific Metrics uses server clusters to administer our production applications. Each server cluster consists of Web servers and database servers. Both Web and database servers are defined in a redundant configuration to ensure no single point of failure in the assessment system. Additional servers can be added whenever the user population warrants more capacity. Pacific Metrics also has access to significant bandwidth resources to ensure our servers operate at peak performance to adequately support the size of the user population.

We understand that two of SBAC’s key technology priorities are the minimal impact on state and local systems and the varying levels of state and local technology capabilities. The impact that the IAIP application has on state and local systems strongly depends on the state and local technology capabilities. Users within the consortium states are likely to have varying levels of technology infrastructure and capabilities, and the assessment system must provide a solution that can work across all states. The differing technologies include internet bandwidth, type and age of computing systems, network infrastructure, sever infrastructure and hosting facilities. The IAIP application will address variability by centralizing infrastructure needed for the application where possible, reducing the need for state and district infrastructure to support it.

SBAC’s IAIP application will minimize bandwidth requirements through the use of caching systems and efficient content representation. The architecture will consider the use of lower-


2-30

cost though scalable computing resources (i.e. “cloud resources”) to help reduce impacts to state and local systems.

ORCA Application Requirements and Performance The ORCA family of products has the following general hardware and software requirements for operation.

Hardware:

• Enterprise class x86 servers for virtualization servers and database servers • Enterprise class firewall and switching equipment • Hardware load balancer components

Software:

• VMWare Virtualization Software • Red Hat Enterprise Linux • JBOSS or Weblogic Application Server • Enterprise Mysql or Oracle Database Server • Apache Foundation Components (httpd, Tomcat, etc) • RealObjects EditOnPro Web-based Editor

3.C Knowledge Transfer, Transition, and Turnover Pacific Metrics has extensive prior experience with the management and maintenance of production applications, including the transfer of these responsibilities to a third party at the end of a program. For many contracts, application transition and knowledge transfer have been required deliverables of the programs. Pacific Metrics will employ this past experience to ensure a smooth knowledge transfer, application transition and turnover of the Item Authoring and Pool Application to SBAC as part of completing the program.

Below we provide more detail on the components of the Knowledge Transfer, Transmission, and Turnover deliverable.

Transition Plan We will develop a transition plan beginning approximately one year prior to the completion of the program in September, 2014. The Transition Plan will detail work plan components specific to transitioning the Item Authoring and Pool Application including those described in this proposal. Planning will be completed for the following activities.

• Construction of a technical systems manual • Transitioning application hardware and software to SBAC • Administrative training • Development of application transition checklist • Professional support after transition

The work plan developed a the beginning of the program is expected to cover these activities, but the Transition Plan will cover them in greater detail, leveraging experience working with SBAC, State DOEs and other system vendors to ensure a smooth transition of the application at the end of the program.


2-31

Technical Systems Manual As part of this deliverable, Pacific Metrics will produce a Technical Systems Manual. The intended primary audience for the Technical Systems Manual is the staff responsible for managing and maintaining the Item Authoring and Pool Application after transition. The purpose of the Technical Systems Manual is to document technical system specifications, descriptions of the components of the application system and processes for managing and maintaining the application. The manual will include but is not limited to the following technical information:

a. Application components b. Application operational requirements

i. Server Hardware ii. Software iii. Networking Infrastructure (bandwidth, load balancing, switching, etc.) iv. Build and deployment environment

c. Application build process d. Application deployment process e. Application configuration procedures f. Application operational procedures g. Database maintenance and tuning procedures h. Required data migration

Costs for this manual assume that it is an internal document for use by SBAC, Pacific Metrics, and any SBAC designee, and not developed for public release. We have budgeted to produce a document of approximately 100 pages.

Knowledge Transfer and Training As part of its larger training effort for this program, Pacific Metrics will complete training specifically related to the management, maintenance and enhancement of the Item Authoring and Pool Application. This training will be conducted by Pacific Metrics staff that have deep knowledge of the application being customized, and it may include staff originally responsible for developing and customizing the application. The content of the training will be similar to what is covered in the Technical Systems Manual outlined above.

Professional Support after Transition We anticipate that Pacific Metrics will need to provide some level of support to SBAC or its designee after the Item Authoring and Pool application has been transitioned. We have allocated two weeks of Level 3 professional support time for this activity. Needed support will be provided by phone and online meeting. Support will be provided via the defined support

Cost Option

Knowledge Transfer and Training Pacific Metrics will provide onsite trainings specifically related to the management, maintenance and enhancement of the Item Authoring and Pool Application to ensure seamless handoff of the IAIP application to SBAC at the conclusion of the contract. Costs for this training would assume three consecutive training sessions conducted at SBAC-designated facility in which two Pacific Metrics Trainers conduct the training and five or less SBAC staff members are involved with each session.


2-32

channel, with Pacific Metrics staff knowledgeable with the management and maintenance of the application providing Level 3 support.

In order for this support to be most effective, transition of the application must be completed by September 30, 2014. If it is desired by SBAC, Pacific Metrics can develop a cost option for additional post-transition support beyond September 30, 2014.

4. ITEM AUTHORING AND ITEM POOL APPLICATION SOFTWARE The COTS application proposed by Pacific Metrics and customized for SBAC is designed to efficiently and accurately meet the requirements identified within the RFP. Because the system was designed by and specifically for Pacific Metrics’ assessment specialists working on multiple state contracts, it offers an inherent flexibility that is typically lacking in systems built for a single state’s purposes and specific needs. The IAIP application was architected from a broader perspective, to allow support for assessment development across multiple states, programs, and vendors.

This proposal outlines how the consortium-owned SBAC IAIP application supports the four main functional aspects enumerated in Appendix A:

1. item/task/stimuli development 2. test form development 3. psychometric information 4. dynamic reporting/querying

We provide a high-level overview of each feature below, and then provide specific details related to each of these functional aspects in the table in Appendix A of this proposal.

Item Development: The SBAC IAIP application is a scalable online assessment content editing and management system provided through an intuitive Web-based interface. It allows test developers to construct content that can be delivered seamlessly to test delivery systems. Alignment to content standards is an inherent feature of the system, as is the creation, alignment, and storage of stimuli associated with item content. The application provides a versatile, flexible and secure means of developing, managing, importing and exporting test content with many benefits including:

• Configurable process integration and workflow • Automation of time consuming and potentially error prone processes • Standardization and quality management of test development processes • Editorial processes for approvals • Management and end-to-end publication of test content • Management of Users and Permissions

The SBAC IAIP application supports high-quality item development, providing the front end for item creation, management and review functions. Figure 4 shows the integration of the application with other assessment components.


2-33

Authoring/Editing Interface The SBAC IAIP application leverages an intuitive and comprehensive HTML editor to support the creation and revision of test content. While users typically interact with the editor in WYSIWYG mode, XHTML content is simultaneously generated to support downstream Web applications (i.e., online test delivery). The display below shows the item stem presented in the editor interface.

Figure 2-5: Item Stem in WYSIWYG Mode

The editing interface includes a robust toolbar that supports the various formatting elements and structures common to test items. The toolbar buttons and their icons are readily familiar to users of the most common word processing applications (e.g., Microsoft Word). This version of the HTML editor is provided through a third party source. Pacific Metrics understands that one of

SBAC IAIP Application

Mass import of legacy content

Export to assessment

delivery platform

Deliver tests online

Print/.pdf Accomm.

Import content and data to IAIP Bank

Reporting

Synchronize data

Figure 2-4 : Integration of the IAIP application with Other Assessment Components.


2-34

SBAC’s key technology priorities is the utilization of open-source systems. As such, Pacific Metrics will implement a similar open-source HTML editor for SBAC’s IAIP.

As noted in Appendix A, all item-level changes initiated within the Item Editor interface are recorded in the system, including information regarding the date, time, user, and change made to the item.

Accessibility

Pacific Metrics recognizes that one of SBAC’s key technology priorities is to ensure all consortium systems accommodate accessibility and universal design principles. The IAIP application will provide a set of accessibility requirements that will be amalgamated from several sources including but not limited to:

• Pacific Metrics accessibility guidelines. • ISO/TS 16071 • WCAG 2.0 • Section 508 standards

The accessibility requirements will be applied based on the component context. The system architecture will also define how component accessibility compliance must be validated.

It is recommended that item accessibility be facilitated through an interoperability standard and not a proprietary format.

The two (2) interoperability standards that will be given serious consideration are:

• Accessible Portable Item Profile (APIP) • Question and Test Interoperability (QTI)

Technology Enabled Item Development As a leading provider of innovative technology solutions in the assessment industry, Pacific Metrics recognizes that the development and administration of innovative item types will be critical to the success of the common assessments. As such, the SBAC IAIP will offer a streamlined solution for the development of these item types.

The SBAC IAIP application supports the creation of content for Technology Enabled Items (TEIs) through an identified workflow and integration with Pacific Metrics’ ORCA™ Connect test delivery platform.

Integrated Workflow Pacific Metrics recognizes the need for customizable, flexible workflows to support the diverse needs driving content development projects. We recognize that as part of SBAC’s key technology priorities, innovative and creative solutions are necessary to improve the assessment workflow. The SBAC IAIP system architecture was designed with an understanding that each development project may require a separate workflow, and that steps within each workflow may need to be repeated or bypassed. The application can easily be configured to allow for multiple workflows of this type, and this results in increased efficiency and productivity for users.

The default workflow in the SBAC IAIP application assumes a standardized review and approval workflow that all content follows. For example, an item submitted by a writer may be routed to two content reviewers and a copy editor, in that order. A comprehensive review and quality


2-35

check is applied to the content developed and different forms of content can be subject to different levels of review. This practice ensures that all test content undergoes an appropriate level of rigorous review prior to publication.

Management of Item Content An advanced administrator toolkit supports the item development and management needs of the system and includes the following features:

• Moving or copying items between banks • Deleting items from a bank • Uploading batch-level item metadata • Searches and filters for accessing or restricting content • One-click editorial or review access to content, independent of workflow state • Re-assigning of item, rubric, or passage IDs • Assigning test content to a specific author (e.g., item or passage writer) • Changing the workflow state of an item or passage • Changing the publication status of an item • Changing the export status of an item • Changing the read-only status of an item • Unlocking content that has been locked for editing

Content is stored within SBAC’s IAIP application in XML format. The application supports input/output modules that allow the item and form data to be easily exported or imported using other XML formats, such as CIS. All content can also be exported from the system in a print-ready PDF format.

Pacific Metrics understands that one of SBAC’s key technology priorities is the need for item interoperability. The customized IAIP application that Pacific Metrics will build for SBAC will support item interoperability in a number of ways, including:

1. The ability to mass import items from multiple sources 2. The use of APIP as a standardized item format 3. The use of a web service interface for interaction with the system

Supporting this interoperability facilitates the importation of items from multiple state users developing items in different formats. This interoperability will reduce associated software and technology costs to member states.

All access to the SBAC IAIP application is restricted using a built-in authentication /authorization mechanism. Users are assigned a password which is automatically generated and never known to administrators (although administrators can have a new password generated for users who forget it). Each user may be restricted to normal, admin, or super-admin functions per item bank, allowing for fine-grained access to all content. Each user with the ability to edit item data is assigned a particular role (such as item writer, content reviewer, artist) which restricts which system-level functions they are able to access. The user types, as well as the permissions associated with them, can be configured based on SBAC’s specific requirements.

Test Form Development The SBAC IAIP application supports all test form creation processes with its Forms Manager interface. Authorized users are able to create specific test blueprints. Once items have been


2-36

accepted for use on a test form, sophisticated item search and filtering capabilities aid users in finding items that meet specific statistical requirements, and exclude items that do not from the available item pool. Form-related functions include creation/selection, editing, and export, and can be utilized for an entire form, or for test form part or session. Test map creation, modification, and export are components of the test form development process as well.

Users are able to save tests for future use. Tests are identified by their name, an optional description, and test type (Field Test, Operational, Equating, etc.).

Figure 2-6 : Test Form Creation

Washington


2-37

Figure 2-7: Creation of Test Form Session

Pacific Metrics recognizes the need for SBAC’s IAIP application to support traditional item attributes as well as cutting-edge, multimedia assets. The application currently supports the association of Word documents, PDF files, and websites at the item level. The application also has a comprehensive audio file management system that allows users to associate MP3 files with all parts of an item (stem, answer choices, directions, etc.) and also to passages and stimulus materials. The IAIP application offers a user interface specifically designed for the creation and export of item scripts. This scripting functionality serves as a foundation for the development of multiple types of accommodations, including Braille and Read-Aloud. Users are given a special toolkit designed specifically around this process, which includes the automatic insertion of frequently used words/phrases, the ability to configure and export scripts in specific formats, and access to helpful external resources.

Figures 2-8 and 2-9 demonstrate the Audio Scripting interface.


2-38

Figure 2-8: Audio Script Main Menu

Figure 2-9: Script Editing Window

Psychometric Information and Item Banking The SBAC IAIP application stores all test items, directions, reference materials, manipulatives, and other electronic tools provided to students during the test, and the association between test items and online assessment tools are made within the item editing application.

The Item Banking component is essential to support the storage of test item data and statistics related to their usage and is embedded within the IAIP application. There are two main components of the bank—the database (DB) and the Web-based standard user interface. The retrieved data views can be stored for further user needs in different formats, including standard format, such as Excel. Basic functions supported through the Item Banking component include the following:

• Storage and display of items, item statistics, item usage history, passages, graphics and stimuli

• Easy and convenient access to items and item information


2-39

• Ability to sort/select items based on testing program, test administration, content specification, etc.

• Ability to record and retrieve comments or notes specific to an item • Synchronizing item data in the database

The item bank area acts as the post-administration repository of relevant data and details generated from an assessment event, i.e., a test administration. After an assessment event and after analyses steps have been completed and reviewed, then quantitative results which are instructive toward downstream administrations (e.g. classical statistics and IRT item parameters) can be archived into the item bank as part of the descriptive history of the recent assessment event. These data elements document how an item performed against a particular target audience. Certain performance indicators associated with past administrations (e.g. item usage history and performance metrics) can then be updated to the content development system from the item banking system to instruct future content creation efforts and form(s) selection activities. The statistics related to an item’s publication history display based the most recent administration of that item. Successive records can be accessed from within a single user interface by advancing the records as needed.

Figure 2-10: Example of Item Publication History

Day to Day Updates/Corrections The process for updating item statistics in the SBAC IAIP application can take two paths. For day-to-day updates and corrections, the application allows authorized users to make adjustments to information manually within the items themselves.

Post-Administration Updates For post-administration item updates—where several hundred items require a simultaneous update across a variety of metadata—a batch upload process should be used. During this process, a designated Item Administrator works to ensure data is accurate and is assigned to items correctly. All relevant statistics and item metadata are documented in an Excel spreadsheet. This spreadsheet is saved as a CSV file and then uploaded into the SBAC IAIP application for the immediate update of item content.

Record 1 of 2


2-40

Dynamic Reporting/Querying

Real-Time Content Reporting The SBAC IAIP provides several default reports related to the content creation and management activities. The application can be easily configured to support additional default reports, and/or offer the ability for users to create custom reports.

All reports are generated in real-time, reflecting the most current status of all system content and data.

The IAIP application supports dynamic querying, and was designed to support the ability for authorized users to execute strategic searches from multiple parts of the application.

The application was designed with a fundamental understanding of the needs and processes of content developers. The streamlined user interface allows users to quickly access the content they want, without having to apply extraneous filter fields. Strategic filtering and search capabilities allow users to search for content (items, stimuli, graphics) based on numerous attributes in the following categories: content standards/alignment, item metadata, and statistical data. In addition, key word and phrase searching is also supported. The following screens show examples of the querying interface.

Figure 2-11: Example of Standards filter


2-41

Figure 2-12: Example of Metadata filter

Figure 2-13: Example of Statistical filter

5. RECORDS OF DECISION MAKING Pacific Metrics personnel assigned to this project team will work closely with SBAC and other involved vendors to ensure the successful operation and implementation of its Item Authoring and Item Pool application. We will follow state-of-the-industry measures for assuring quality and error-free production and operation by staffing the project with the highest caliber staff, adhering strictly to comprehensive quality control checklists, and using processes that are both efficient and minimally prone to human error.

The responsibilities of the project team will include:

• Working with SBAC to plan and schedule all activities • Receiving written approval from SBAC for any changes to the scope of work • Monitoring and reporting on the progress of the project • Recording the results of discussions and clarifying issues in the meeting minutes • Ensuring all deliverables are on schedule. • Ensuring that all processing steps are completed on time with 100% accuracy.

We note that many of the activities that will be planned and scheduled involve input from SBAC staff at the designated times. To ensure timeliness and accuracy in gathering this input, Pacific


2-42

Metrics will request that SBAC’s input on any activity or document be consolidated prior to being provided to Pacific Metrics.

The following table outlines the documentation Pacific Metrics proposes to use to ensure accurate records of decisions made throughout the project are maintained, and that requested changes to the system will be documented.

Table 2-3: Recommended Documentation

Potential Document Description Purpose

Design decisions

A summary of critical decisions pertaining to design and architecture that the team makes throughout development

The goal of this effort is to avoid refactoring in the future or rehashing a previously made decision.

Operations documentation

An indication of the dependencies the system is involved with; the nature of its interaction with other systems, databases, and files; references to backup procedures; a list of contact points for the system and how to reach them; a summary of the availability/reliability requirements for the system; an indication of the expected load profile of the system; and troubleshooting guidelines.

The goal of this document is to provide an overview of the system, its dependencies, and staffing. Any changes made to the Operations component of the system will be reflected in this document.

Project overview

A summary of critical information such as the vision for the system, primary user contacts, technologies and tools used to build the system, and the critical operating processes (some applicable to development. Also provides references to critical project artifacts.

This document serves as a starting point for anyone new to the team, including maintenance developers, because it answers fundamental questions that they are likely to have. Any changes affecting the project will be reflected in this document.

Requirements document

This document defines what the system does, summarizing or composed of system requirements.

This document identifies the specific system requirements for the software and ensures that necessary functionality is identified at the onset of a project. If requirements change over the course of the project, this document will be updated to reflect these changes.


2-43

User documentation

Specific user training and user support documentation are referenced in section 2.F: Training Materials and include User Guides, Training Guides, and online tutorials.

These documents provide guidance and information regarding system functionality. Any changes made to the system throughout the development process will be reflected here.

Communication Pacific Metrics will support efficient lines of communication and information exchange throughout the contract. We recognize the importance of ongoing communication between the project team, SBAC, and member states during all stages of the contract to ensure its effective operation. Conference calls, facsimile transmissions, mail, e-mail, web meetings and overnight delivery services will be utilized as appropriate. We have a toll-free number and a voice-mail message system that can be accessed 24 hours a day. With the exception of holidays, our office hours are from 8:30 A.M. until 5:00 P.M., Monday through Friday. Key personnel from Pacific Metrics will be reachable by telephone during standard business hours, with cell phone numbers available for emergencies outside of standard business hours. Given their technical expertise and key roles on the project, each of them will be empowered to solve problems directly with a minimum of bureaucratic or technical hurdles to clear. As part of our planning process, we will develop a communications plan that will be incorporated into the overall program plan. The communications plan will contain contact information for our senior personnel and primary program contacts (the Program Manager, Project Coordinator, and other key staff). In addition, the plan will specify delivery methods for various communications, including any materials that need to be delivered by overnight courier service. As a general rule, we prefer that the Program Manager be the primary point of contact between the SBAC and our project team. However, we will facilitate communication between other key staff that best addresses specific tasks as needed. The specific processes around communication are further outlined in the Management Proposal.

6. ANNUAL CONTRACTOR MEETINGS Annual Contractor Meetings Pacific Metrics understands the importance of cooperation and coordination with SBAC and its member states, as well as with other involved vendors and contractors. As such, there will be four annual meetings, held to support project-specific and cross-consortium contractor engagements.

Pacific Metrics will cover the costs for our staff members to attend the four annual meetings. Representatives from our Executive Team will attend as appropriate. Final decisions on which staff will be in attendance at any meeting will depend on the SBAC IAIP application project activities to be addressed on the agenda and their areas of expertise. It may be that some staff will simply be conferenced into parts of the meeting by telephone to maximize their input, while also minimizing the cost of their participation.


2-44

Document Meetings

Pacific Metrics is committed to providing excellent documentation for all meetings that take place between the project team and SBAC. Agendas will be sent out electronically at least 24 hours in advance of meetings. All meeting minutes will provide clear guidance as to the discussion held, decisions made, and next steps. We follow the Project Management Institute’s best practices, which dictate that we maintain a high level of project documentation and provide it in a timely fashion. SBAC will be provided with archival records of the project including status reports; decisions reached via e-mail, telephone, or fax and any other supplemental documentation that will help facilitate our discussions and planning.

B. WORK PLAN In this section we describe the work plan to complete the scope of services and provide the defined deliverables on the project. The Work Plan is structured around the outlined components of the project, both products and services, and it separates unique development activities from ongoing operational activities.

The project work plan has a component of application development that creates a majority of the project deliverables and also encompasses many services such as requirements development, application design and development, quality assurance testing, data portability application deployment, training development and infrastructure development. It also has an operational component that encompasses ongoing application hosting/maintenance, maintenance and support. Finally, it has a project management and communication component that covers project management processes and formal communications activities. The remainder of this section describes the plan in greater detail.

Application Development The application development component of the work plan encompasses all product and service deliverables necessary to deliver a fully functional and supportable Item Authoring and Item Pool application. The following specific deliverables are satisfied in this work plan component, and as further detailed in Section D, Deliverables:

• 1.A Application Requirements Services • 1.B Technical Systems Requirements Services • 2.A Application Design Services • 2.B Application Development Services • 2.C Quality Assurance Testing Services • 2.D Data Portability-Item/Test, Stimuli, and Multimedia Import module • 2.E Application Deployment Services • 2.F Training Services • 4 Item Authoring and Item Pool Application Software


2-45

Application development is structured around application release cycles that are aligned with major milestones as defined in the RFP. We envision that the customizations to meet the requirements of the Item Authoring and Bank Application will be developed in a set of two software releases. The first release will be aligned with the beginning of item development in August, 2012. The requirements that will be satisfied by application customizations in the first release include the following high level functional requirements:

• Ability to load all item and associated assets into the Item Authoring and Bank Application. This includes item, stimuli, multimedia and accessibility tagging.

• Ability to complete all development, management and workflow functions related to items, stimuli, and assets such as multimedia, including required review processes and committee reviews.

• Basic reporting and querying. • Backup and Recovery. • A production system prepared to support item development starting in August, 2012.

A second software release will be completed prior to the February, 2013 pilot test and will include customizations that address all remaining functional requirements. These include the following areas:

• Ability to complete all development, management and workflow functions related to blueprints, test form templates and test forms, including required review processes.

• Ability to import, manage and report on Psychometric information. • Dynamic reporting and querying. • Security, access and password controls • A production system that can exchange data with the Test Delivery Application.

Each software release will contain all lifecycle activities for constructing a full release of the application. As highlighted in Section A, we will employ an Agile development process to construct the software, which further decomposes a software release into iterations that also contains all lifecycle activities. An iteration will include the following activities:

• Requirements development/refinement • Software Design • Software Development • Documentation Development • Training Materials Development • Software Testing (Unit, Integration, Functional, Performance) • Software Deployment

To account for the project requirement to develop formal requirements documents as part of Requirements Development Services, we will modify our traditional Agile process to allow for a requirements development iteration. This iteration will focus on defining initial versions of the Functional Requirements Document and Technical Requirements Document as highlighted in section 1.A. These documents will then be refined as part of each software iteration. It is important to re-emphasize the goal of Agile development to enhance the communication process on the team and focus less of formal documentation.

In addition, we anticipate we will only complete full application deployment and user acceptance testing at the end of each software release, so we will define a final Acceptance/Deployment iteration that focuses on these activities.


2-46

Hosting and Maintenance The Hosting and Maintenance component of the work plan encompasses all product and service deliverables necessary to manage and maintain the Item Authoring and Bank application in addition to transitioning the application at the end of the project. The following specific deliverables are satisfied in this work plan component, and as further detailed in Section D, Deliverables:

• 3.A Application Maintenance and Support Services • 3.B Application Hosting Services • 3.C Knowledge Transfer, Transition, and Turnover

Most of these activities are on-going in nature once they are initiated through the end of the project. We expect that Application Hosting Services to begin approximately two months after contract award. This milestone timing allows production, development and QA environments to be in place prior to the first production software release. Once the environments are constructed, they will be fully managed and maintained according to Application Hosting Services deliverable.

Application maintenance and support services will begin on the go-live date of August, 2012. At this milestone, the help desk will be up and running, support processes and production bug tracking will be in place, and the support organization will be prepared to provide end-user application support. Prior to the millstone, there will be non-recurring activities to prepare these support components.

Knowledge Transfer, Transition and Turnover activities will begin approximately one year prior to contract end and as detailed in Section 3.C. The primary work activities will be completed in that timeframe, and as detailed in the project schedule. The focus will be on planning activities earlier within the effort (development of a transition plan and application transition checklist) and executing the activities later in the effort. Pacific Metrics must also work with SBAC to define when formal transition is executed such that post-transition support can be provided. The current work plan and schedule assumes that these activities are all completed prior to the contract end date of September 14, 2014.

Project Management and Communications This work plan component encompasses all activities and deliverables to effectively manage the project in addition to all formal project communications. The activities and deliverables include the following, which are further detailed in the management plan:

Project Management Activities Project Management Deliverables • Project Planning and Scope Management • Project Tracking and Control • Communications Planning and Project

Communications • Work Plan Management • Risk Planning and Management • Issues Management • Escalation Planning and Management • Performance Management

• Work Plan • Deliverable Matrix • Status Reports • Issue Tracking & Management Log • Communications Plan • Collaboration Approach • Web Conferencing Service


2-47

A majority of the project management deliverables will be constructed within the first month of the project effort. Ongoing project management activities will be conducted on a continuing basis, with periodic meetings related to several of them. As noted earlier in the proposal, Pacific Metrics proposes using the Agile software development process for the IAIP. The following table highlights meetings recommended by Pacific Metrics to help facilitate successful delivery of software that meets SBAC’s identified requirements.

Meeting Recommended Location Purpose Length Needed SBAC

Participants Schedule / Periodicity

Project Kickoff Virtual— this would be one of the 4 annual contractor meetings

Orient relevant staff to the project effort and establish initials parameters for planning and executing the effort.

4 hours Cross-functional team including staff with project management, technology, publishing and research experience.

Once, during the first month of the program

Project Planning Meeting

Virtual— this would be one of the 4 annual contractor meetings

Report project status, adjust large project plan based on progress, and address issues identified in the project effort.

4 hours Cross-functional team including staff with project management, technology, publishing and research experience.

Approximately once per year, beginning one year after kickoff meeting.

Requirements Definition and Backlog Planning Meeting

Virtual—via conference call and web meeting

Initial meeting to kick off requirements development for a given software release, in addition to establish and prioritize Product Backlog Items

2 Days Selected subject matter experts in functional areas being defined. Project Management support.

One meeting per software release, two total for two software releases, both in Year 1 of the project.


2-48

Meeting Recommended Location Purpose Length Needed SBAC

Participants Schedule / Periodicity

Sprint Review Meeting

Via conference call and web meeting

Review the software products of a completed sprint

1 Hour SBAC staff member that can act as a product sponsor and any other SBAC staff that wish to attend

Once per sprint (two weeks) as part of the Agile iterative process

Daily Scrum Via conference call and web meeting

Discuss sprint progress and blockers to continued progress

15 minutes

SBAC staff member that can act as a product sponsor and any other SBAC staff that wish to attend

Once per working day as part of the Agile iterative process (Daily Scrum meeting not called out in Work Plan Schedule)

Project Status Meeting

Via conference call and web meeting

Discuss project progress, upcoming work, and project issues

1 Hour Project management and selected subject matter experts

Once per week

Pacific Metrics believes that SBAC and its member states should be active participants in the software development process. As highlighted in the table above, SBAC is expected to participate in several meetings related to the project. We expect that SBAC and its constituents will be instrumental in defining effective customization requirements for the Item Authoring and Bank application as conveyed through these meetings. SBAC will also be able to provide guidance and direction with respect to the integration of the application into the larger Assessment System Architecture. As noted earlier, we are recommending that some of these meetings occur in person at SBAC facilities. However, recognizing the workload borne by SBAC members and workgroups, as well as their geographic diversity, we are open to conducting these meetings remotely if that is SBAC’s preference. As highlighted in the Agile overview, we expect that SBAC will play a Product Owner role with respect to the iterative development process. While there is a time investment in playing this role, it will allow SBAC to have direct effect on the application development while it is in progress. With respect to ongoing project management, we expect that SBAC will play a role in monitoring the project, providing feedback with respect to the timeliness and quality of deliverables, and overall guidance of how this project integrates into the larger consortium efforts. Finally, we expect that SBAC or its designee will participate in project transition activities such that the application can be effectively transferred before the end of the effort.


2-49

C. PROJECT SCHEDULE


2-50


2-51


2-52


2-53


2-54

2-55

D. DELIVERABLES This section iterates all deliverables of the proposed project effort.

Deliverable Area Deliverable Description 1.A Application Requirements Services

Development of detailed functional requirements document for the Item Authoring and Bank application with the primary artifact being a Functional Requirements Document.

This document will be delivered as part of each software release, refined to include additional requirements and modifications to existing requirements associated with each release.

1.B Technical Systems Requirements Services

Development of detailed technical system requirements document for the Item Authoring and Bank application with the primary artifact being a Technical Requirements Document.

This document will be delivered as part of each software release, refined to include additional requirements and modifications to existing requirements associated with each release.

2.A Application Design Services

Development of architecture and detailed system design document for the Item Authoring and Bank Application with the primary artifact being a detailed Architecture and Design document. The document will address all functional and technical requirements and will include the following components at a minimum:

• User Stories • Domain models • Flow diagrams • Enterprise integration models • Entity relationship models • Class diagrams • Logical and Physical Data models

This document will be delivered as part of each software release, refined to include additional functionality being provided with each release.

2.B Application Development Services

Service activities surrounding the system development for the Item Authoring and Bank Application. This activity includes all effort to construct the application, including preparing the hosting environment for production application use and preparing a demonstration site. The primary deliverables of this activity are encompassed in Product Deliverable 4, Item Authoring and Item Pool Application Software.

2.C Quality Assurance and Testing Services

Development of a test plan and testing specifications for the Item Authoring and Bank application. The specific artifacts from this service activity are as follows:

1) Test Plan and Specifications document encompassing: • Unit Testing • System Testing • User Acceptance Testing • Performance Testing • Operations Testing

2-56

Deliverable Area Deliverable Description

2) Test results for testing that has been completed. Both the test plan and test results will be delivered as part of each software release, refined to include additional testing components, test cases and test results for functionality being provided with each release.

2.D Data Portability-Item/Test, Stimuli, and Multimedia Import module

Data mapping requirements to support mass import process, software programs, verification procedures, and error reports. This deliverable will be provided as part of the Technical Requirements document. If necessary it can be published as a separate document.

2.E. Application Deployment Services

A Production environment prepared to support required system functionalities, including all necessary hardware and software. The Work Plan outlined in Section B of the proposal describes how this deliverable will be met.

2.F. Training Services Development of training materials for the Item Authoring and Bank application. The specific artifacts produced from this service activity are as follows:

1) Application Users Guide 2) Training Guides 3) Self Learning Tutorial

These artifacts will be delivered as part of each software release, refined to include additional functionality being provided with each release.

3.A Application Maintenance and Support Services

Detailed application maintenance and support for the period of time the application is in production. This include the following recurring activities:

1) Help Desk and production application support (Level 1, 2 & 3) 2) Production fixes for issues (“bugs”) identified with completed

features 3) Release of application enhancements after the initial software

release 4) Monitoring and system maintenance of the application within the

hosting environment 5) Document updates related to production fixes and additional

features. Services will be provided on a continuous basis starting August, 2012.

3.B Application Hosting Services

Hosting of the Item Authoring and Bank Application, to include the following services:

1) Hosting facility for hosting the application. 2) Managed services to ensure the application and associated

infrastructure are maintained and fully operational. 3) Security capabilities to ensure the application is fully protected in

the hosting environment. 4) Disaster recovery capabilities to ensure application availability. 5) Scalability capabilities to allow for easy application growth.

2-57

Deliverable Area Deliverable Description Services will be provided on a continuous basis starting approximately two months after contract award.

3.C Knowledge Transfer, Transition, and Turnover

Transition of the Item Authoring and Bank application to SBAC or its designee, to include the following services:

1) Construction of a technical systems manual 2) Transitioning application hardware and software to SBAC 3) Administrative training 4) Development of application transition checklist 5) Professional support after transition

Several artifacts will also be generated as part of this activity:

1) Transition Plan 2) Application Transition Checklist 3) Technical Systems Manual

These services will begin approximately one year prior to contract end with the Transition Plan and Technical Systems Manual being delivered as defined in the Work Plan.

4 Item Authoring and Item Pool Application Software

This is the primary product deliverable of the project, encompassing the following areas:

1) The Item Authoring and Bank Application running in a production hosted environment, meeting the defined project requirements, and released according to the release schedule defined in the project work plan.

2) The demonstration application that demonstrates system functionality.

3) All support applications needed to maintain and operate the primary application.

4) All technology infrastructure needed to support the application.

Components of this deliverable will be delivered as defined in the project work plan.

2-58

E. OUTCOMES AND PERFORMANCE MEASUREMENT Pacific Metrics’ employs Project Management Institute (PMI) based techniques for overall project management. To confirm that all project goals, performance tasks and deliverables are completed in a timely and high quality manner, we utilize formal processes for the following: scope management plan, communications plan, work plan, risk management plan, issues management plan, change management plan, and a performance plan. Implementing project management best practices in each of these categories helps ensure that the project will be developed and implemented successfully.

Pacific Metrics employs a detailed program plan that is submitted to the client for approval at the beginning of a project. The Program Plan serves as the focal point of all program management, communication and documentation. To that end, the Program Plan includes the following key components:

• Scope of Work: A description of the scope of the work for each phase of a project, recognizing that the plan will be updated as needed to provide details as appropriate for future phases, as well documenting relevant changes in scope

• Project Schedule: A high-level project schedule showing all anticipated tasks, meetings and other activities

• Communications Plan: This details the communications plan that will be in place to between Pacific Metrics’ and client project managers

• Risk Overview: An overview of any perceived risks or issues, and a mitigation plan for addressing potential risks and issues as they arise

• Budget Overview: Budget assumptions and level of staffing assigned to the program

Project Management Institute best practices advise maintaining a high level of project documentation and making it available in a timely fashion. Pacific Metrics follows this practice and provides the client access as needed to the project‘s Communications Register, which is used by the project manager to track the release of formal communications such as: regular project status reports, meeting/conference call minutes, project plan releases, decisions reached via e-mail, telephone, or fax, and other supplemental documentation.

As a component of the documentation created through our Program Plan, Status Reports are created with the purpose of measuring, monitoring, and reporting the expected outcomes related to the project.

Status reporting will be based upon the parameters set forth in the Work Plan created for this project. Status reports will include the following items:

• Progress on milestones • Summary of completed and upcoming deliverables • A summary of the work status referenced to the work plan • Key project accomplishments since last report • Discussion of upcoming project activities • Review of current issues – open, closed, and anticipated

Additional information regarding status reporting can be found in Section 2 of the Management Proposal.

2-59

F. RISKS As part of our work plan, Pacific Metrics incorporates a risk mitigation plan. Through this we identify risks, assess their impact, develop plans to handle the risks, monitor risks to determine if the level has changed, and document our overall strategy.

Our risk management strategy for large-scale projects is to identify critical areas and events that might cause risks (i.e., threats to the overall objectives) and take necessary action to handle them before they can become problems and cause serious cost, schedule, or performance impacts. Essentially, we focus on the risk management techniques of acceptance, avoidance, and control of risk.

For the Item Authoring and Item Banking Application system, our goal is to identify any critical areas that might cause threats to the overall project objectives. The persons most closely involved in the implementation of the risk management are the program manager project advisors; however those employees working closely with the development and delivery of the assessment have valuable input in terms of identifying areas of risk. The risk management plan will list those processes and products that are critical to meeting the objectives (e.g., integration of existing content into the item bank,). Once identified, risk ratings of Low, Medium, or High will be assigned based on the following criteria:

Low—has little or no potential for increase in cost, disruption of schedule, or degradation of performance.

Actions within the scope of the planned program and normal management attention should result in controlling acceptable risk.

Medium—-may cause some increase in cost, disruption of schedule, or degradation of performance. Special action and management attention may be required to control acceptable risk

High—-likely to cause significant increase in cost, disruption of schedule, or degradation of performance.

Significant additional action and high priority management attention will be required to control acceptable risk.

We will then determine the consequence of the risks by looking at the potential effect on technical performance, schedule, cost, and impact on deliverables. The most significant risks will be listed and monitored on a regular basis to determine if a plan of action needs to be implemented.

In summary, Pacific Metrics risk mitigation plan will:

• Provide risk information for decisions; • Allow monitoring of activities as they proceed; • Serve as a basis for identifying alternatives to achieve cost, schedule and performance

goals; and • Assist in making schedule and budget decisions.

Although we will develop the risk mitigation plan, we will seek SBAC approval before implementing any action that will be visible to SBAC’s stakeholders.

2-60

Risk Comments

10% or more potential requirements increase

Pacific Metrics acknowledges that, as noted in the Objective and Scope of Work section of the RFP, there’s the possibility of an approximate 10% increase in the requirements as already defined. If this estimation is low by any meaningful measure, it could pose a risk to the project timeline.

SBAC Technology Architecture Incomplete

Pacific Metrics acknowledges that the Technology Architecture is not fully defined yet. Since the Architecture is the primary driver of the technologies that compose the system and its components, significant changes could have an impact on the scope of work.

Technology Enhanced Item Support

Pacific Metrics acknowledges that Technology Enhanced items and the ability to support them is a requirement of the system. However, due to the broad nature of the terminology (Technology Enhanced), more detailed explanation of what a Technology Enhanced item could entail and the scope of the support will need to be defined in order to estimate an accurate level of effort.

3rd Party Dependencies Pacific Metrics acknowledges that there are requirements to work with outside parties (e.g. Psychometric Services vendor). Delay in the selection of these 3rd party dependencies could pose a risk to the project timeline.

SBAC RFP #7 Item Authoring and Item Pool Application 7_Proposal_Pacific...A. PROJECT...

Documents

Transcript of SBAC RFP #7 Item Authoring and Item Pool Application 7_Proposal_Pacific...A. PROJECT...