City of Santa Monica 2018 Outcomes Measurement System ...

City of Santa MonicaHuman Services DivisionRecommendations for an

Outcomes Measurement System

Understand the measurable.Quantify the meaningful.20

18

Santa Monica Human Services Division Recommendations 2

IntroductionThe City of Santa Monica Human Services Division has invested in a robust portfolio of projects and programs to support all segments of the community. From infants, youth, and families, to seniors, the disabled and homeless households, the services provided by the City through projects like VAP, PAL, and CREST support the entire community. Through recent initiatives such as Cradle to Career (C2C), the Wellbeing Project, and SaMoStat, the City of Santa Monica’s performance management program, the City has started to use data to measure how these and other services benefit the community. As these initiatives generate and inspire more rigorous analysis methods, the City is increasingly poised for an ambitious and transformative evolution in its approach to delivering human services. Starting with the FY17-19 budget cycle, the City implemented the Framework for a Sustainable City of Wellbeing. This approach to budgeting maps key department activities and projects with six outcome areas based on Civic Wellbeing, the Sustainable City Plan, and the principles of good governance. Wellbeing, for example, takes a data-driven approach to articulating and measuring the goals of City residents in areas such as life satisfaction or daily servings of fruits and vegetables. The project brings into focus the wide diversity and income inequality in Santa Monica; households struggling to maintain stable employment and housing will be unable to achieve many Wellbeing goals until their basic needs are met. Initiatives like Learn & Thrive and Inclusive Diverse Community have begun to connect the visionary efforts of the Wellbeing Project with the full scope of needs within the entire community. Many HSD projects target these complex households, connecting resources to efforts that most effectively meet the needs of Santa Monica’s most vulnerable populations. Consistent with communities throughout the United States, the City of Santa Monica endeavors to create a modern, data-driven system to accurately measure the effectiveness, success, and true impact of this work. As demonstrated by the recent hiring of a Chief Performance Officer, the City is committed to the development and implementation of performance management with clearly defined goals and metrics for all departments.

The City’s increasing focus on performance management through its recently established SaMoStat program is intended to set City-wide outcomes and goals while increasing accountability. Currently, the City is working to map work of all departments towards these outcome areas, including the development of input and output metrics for Framework activities that tie back to the outcomes. HSD joins these efforts with their intention to create a data-driven system to benefit and improve their projects.

This report proposes a concise recommendation to design and build an outcome measurement system for HSD projects made up of a technology component and a library of outcomes, enhance staff capacity to effectively manage it, and hire an experienced Data Science Administrator to lead the project going forward. Typically, a project of this scope would take two to four years, but that timeline can be accelerated based on how long it takes the City to accomplish each phase. The City starts this process with a deep knowledge around the delivery of human services and will only expand its expertise by enhancing data collection and better analyzing project outcomes. A successful effort will significantly improve the City’s approach to service delivery, leading to better lives for Santa Monica’s most vulnerable populations.

Canavan Associates86 Sherman Street,

Cambridge, MA 02140

Phone: 617.492.0562

http://www.santamonicacradletocareer.org/

http://wellbeing.smgov.net/

https://beta.smgov.net/samostat

http://wellbeing.smgov.net/findings/outlook

http://wellbeing.smgov.net/findings/health

https://beta.smgov.net/strategic-goals/learn-thrive

https://beta.smgov.net/strategic-goals/inclusive-diverse-community

https://beta.smgov.net/strategic-goals/inclusive-diverse-community

https://beta.smgov.net/samostat#framework

Santa Monica Human Services Division Recommendations3

Defining the Problem

HSD’s current approach to assessing services integrates physical site visits to service providers along with semi-annual project and fiscal status reports. These reports, from both City-funded and City-managed providers, contain clear descriptions of services offered along with demographic breakouts of participants served and a quantitative measurement of each service supplied. Though HSD currently collects some outcomes, without a standardized approach comparisons across the portfolio are not possible. While a large amount of project data is collected, the data remains on paper and is not stored in a flexible format. This in turn makes it difficult for the City and HSD to utilize and understand the data collected and to know how best to assist their projects for the greatest benefit of project participants.

The City of Santa Monica is invested in ensuring quality services and outcomes for their most vulnerable community members. To do so, they need to collect more meaningful data, implement a system that clearly analyzes the outcomes of their projects, and incorporate modern innovative technology. A successful data collection system will include consistent data systems, universal data elements (UDE), and standardized outcomes based on UDE’s. All of these components will assist HSD and HSD’s projects to align with the City’s other performance management activities and expand the way they serve their community.

Although technology and data are key pieces of an effective approach, they alone are insufficient to improve outcome measurement or generate meaningful analysis. The most critical component is an accurate assessment of what services are connected to success and how that success is measured. This requires broad agreement by policy-makers, service providers, and the community. Each HSD project must be measured by its larger impact on other community resources in order to understand if clients are truly achieving their desired outcomes. Building consensus within the community around the proper balance of privacy and utility of data will require intentional engagement with all stakeholders. Fortunately, excellent tools to implement a layered approach are available; these are discussed further under Action 9 Privacy.

Communities across the country have only just begun the challenging process of combining data, project effectiveness, and technology to deliver the best possible human services. While consensus on the best method does not yet exist, the City of Santa Monica and several other communities are leading the effort to set data-driven expectations for their projects to improve outcomes and better help their communities. Each attempt builds on previous efforts to expand the knowledge base and establish best practices, and incredible potential for success remains. To that end, the City of Santa Monica boldly embraces this challenge and joins other municipalities, such as Santa Clara County and the City of Boston, in seeking an appropriate strategy to engage and partner with regional, state, and national service providers.

A successful effort will result in increased accountability for the City and its grantees and better outcomes for participants. This creates an ideal environment for benefits to accrue in a variety of areas such as finance (cost savings from lower arrest and detention rates), productivity (increased employment), and social support (more stable family units and relationships).

The US Department of Housing and Urban Development’s (HUD) approach to data regarding homelessness provides a valuable example of a successful data system for human services. Over the past decade, HUD has invested over $10 million to create universal data standards about homeless households and the effectiveness of projects. Its investment in collaborative systems, academic research, and quantitative reporting now allows homeless serving projects to connect with leaders in finance and government to make new investments that achieve better outcomes for both the household and the larger community.

https://economicrt.org/publication/home-not-found

https://blogs.microsoft.com/newengland/2017/07/07/bostons-coordinated-access-system-technology-to-help-homeless-individuals-find-their-way-home


The Process

In order to identify a practical solution, the City of Santa Monica contracted Canavan Associates (CA), a firm based in Cambridge, MA with nearly 15 years of experience working at the local, state, federal, and international level of program management and design. The CA team previously worked with HSD to implement the City’s Homeless Management Information System (HMIS), a HUD-supported platform that collects and analyzes information about the effectiveness of homeless services provided.

The City of Santa Monica tasked CA with making specific recommendations to support HSD with the design and implementation of a new outcome measurement system. The proposed system is comprised of two main elements, a conceptual framework constructed by experienced human services practitioners and a technology component designed and supported by information technology experts. These new tools will improve project monitoring and increase provider accountability through uniform tracking of outcomes. This, in turn, will result in more effective services leading to better lives for project participants.

CA began by identifying and gathering necessary sources of information. In order to gain a comprehensive understanding of current Human Services projects, CA reviewed 2015-16 project plans and outcomes, analyzed funding sources documented in project budgets, and administered a survey instrument to HSGP grantees. The survey sought to better understand grantees’ current data collection methods and to identify any potential benefits and concerns associated with a new outcome measurement system. The survey results (Appendix A) illustrated that service providers believed a new system could increase collaboration with other service providers, improve communication, and expand understanding of client needs.

CA and HSD followed the survey with onsite interviews with a range of HSD projects to discuss topics including operations, project design, and administration. CA also conducted interviews with a broad cross section of City of Santa Monica staff, including staff from HSD; Community & Cultural Services (CCS); the Information Systems Department (ISD); the Office of Civic Wellbeing; and the City Manager’s Office. Review of project materials and follow-up interviews with project directors at the VAP, PAL, and CREST projects provided excellent sources for understanding the work of the many professionals committed to programmatic excellence.

From these interviews, it became clear that while the City requires projects to report on specific achievements and client progress towards goals, many targets tend to measure outputs rather than outcomes, and measures can vary widely across project types and populations. While valuable insight is provided by the data currently collected, measuring success based on outputs as opposed to outcomes does not fully determine how effectively a service is resolving the problem.

An output details what your organization does, whereas an outcome defines changes that have taken place because of your organization’s work.

- The Non-Profit Times The distinction between ‘outputs’ and ‘outcomes’ can seem purely semantic at first glance, but the difference is critical. In the provision of human services, outputs are the units of services provided. In a hypothetical illustration, an output for a project that provides emergency overnight shelter is the number of bednights provided. A “bednight” is defined as a quantity of nights an individual utilized a bed in the project. In this example, a project that has a maximum capacity of 20 beds, but only shelters 15 people each night of the week, provided 105 (15 people in beds X 7 nights) bednights at the end of the week.

Continuing with the hypothetical of the shelter, if the shelter remained at 75% capacity (15/20 occupied beds) the full year (364 nights), it will have provided 5,460 bednights in that year. Without additional

Privacy and confidentiality are two critically important features of any successfully outcome measurement system. To learn more about the perspective recommended here, jump to Action 9: Privacy.

http://www.thenonprofittimes.com/news-articles/grants-outputs-versus-outcomes


information, the community can perform very little further analysis of this project’s effectiveness. Were the five beds out of service for the entire year? Is the project in a small town or a large city? Is the community in a temperate climate or a seasonal one? While all of these details are helpful to understanding the context of the project and the level of utilization, none of them answers the fundamental, outcome-focused question asked by policy makers: “How is this project ending homelessness?”

When creating a system to measure project effectiveness, it is important to have clear definitions of both success and its counterpart, failure. Before creating a centralized database, the City must review its outcomes and performance measures for clarity and accuracy. Without clearly defined outcomes, no technology would be sufficient to measure a project’s success, and prematurely implementing an expensive software system that cannot meet the City’s long-term needs would be a failure. Once outcomes and performance measures are clearly defined, an appropriate technology component can be identified. In the interim, CA recommends using simple, inexpensive technology as a temporary solution to automate the current paper reporting process while a more robust system is designed, piloted, and implemented.

With this new understanding, CA and HSD decided to focus on laying a strong foundation that would support a variety of technologies, service providers, and funders rather than immediately selecting software that replicates current barriers.


Partnering with the local school district to better use data in evaluating program effectiveness may be one of the priorities of the new outcome measurement system. For example, it could improve understanding on an aggregate level of how students who participate in out-of-school time (OST) direct services see improvements in reading proficiency and other metrics as part of their engagement with service providers.

The Recommendation

The City of Santa Monica seeks a superior method for collecting and analyzing meaningful data to improve their projects and support their most vulnerable community members. Based on the interviews and surveys completed by Canavan Associates, it is clear that HSD needs to define clear outcomes to measure their projects’ success before they can implement a technology component. CA recommends HSD undertake a three-phase process to design, test, and implement an outcome measurement system for all services provided.

The outcome measurement system incorporates the rigor of current models of best practice while preserving the flexibility and responsiveness required to meet emergent community needs. Time and expertise to properly define outcomes and the constituent metrics are required. This new programmatic infrastructure is at the cutting edge of human services delivery and represents a groundbreaking piece of work The process will result in the creation of a set of standardized outcomes across projects, an interim technology solution to get reports off paper and into a digital format, and an eventual integrated technology solution for centralized data. The comprehensive outcome measurement system consists of two components: one piece is the conceptual connections between service provided, data collected, and successful accomplishment of goals and the other is the actual technology or tools that will be utilized by HSD to implement those interconnections. The initial phase of the project is focused on the development of the conceptual layer while latter phases target identifying the correct tools or technology to implement it.

The following is a brief overview of the three-phase recommendation with further details specified later in the report.

In PHASE ONE, the City will choose an implementation consultant and hire a Data Science Administrator (DSA) to lead the effort to align outcomes, data, and service providers into a rational system. The City will engage with the community, and Academic and Subject Matter Experts (SME) to begin to determine how specific services impact the community and to define a framework to organize the different HSD projects. The City will also construct a plan for a pilot project to test new outcome measurement techniques by choosing existing projects to model and outcomes to measure. Meanwhile, the DSA will develop and implement an interim solution to increase the efficiency and effectiveness of current project reporting until the outcome measurement system is finalized.

In PHASE TWO, the City will continue to engage the community and SME to determine a library of outcomes, outputs, and other details necessary for the new system. The new DSA will identify the data the City already possesses and determine the best approach to acquire additional data. An essential step will be mapping both community values and current law around privacy expectations for Santa Monica residents. At this point, the DSA will be able to articulate the functional requirements of any potential technology platforms. Once the outcome measurement system has been designed and a plan for implementation has been determined, the City will meet with internal City partners and external community partners to review the proposal.

In PHASE THREE, the City will implement the outcome measurement system using a phased rollout. The new system will inform decisions for allocating HSD staff resources in order to better serve the community. In addition, HSD will design a schedule of escalating programmatic support for service providers to encourage participation and assist projects.

The City of Santa Monica joins communities across the nation as they seek to improve performance management and implement data-driven systems to better analyze their services. By collecting more meaningful data, analyzing outcomes rather than outputs, and implementing a technology component, HSD will better understand their projects. Using a three-phase recommendation created in conjunction with Canavan Associates, HSD will create an outcome measurement system with an accompanying technology component that enhances the City’s capacity to deliver human services to their most vulnerable populations.


Project DesignThe City of Santa Monica is poised for an ambitious and transformative overhaul of the way it uses data to evaluate its human services projects. The successful realization of this three-phase endeavor will require significant resources, both current and new, along with careful navigation of the delicate balance between privacy concerns and the benefits of data collection. One of the City of Santa Monica’s most valuable resource is the years of expertise possessed by staff and service providers. HSD’s staff, in particular, provides invaluable experience in the administration of projects that serve the City’s most vulnerable populations. Additionally, HSD can leverage assets like ISD’s technical skill, Wellbeing’s expertise in data analysis, and the Chief Performance Officer’s experience with developing metrics.

Another existing resource is HSD’s network of service providers, both HSGP grantees and the City’s direct services staff, who have developed strong long-term relationships within the community. Several projects have senior leaders who were beneficiaries of City services when they were children or adolescents. These deep connections to Santa Monica’s diverse communities are a unique asset that will be critical to the success of this process.

HSD needs to add significant technical capacity to their staff by hiring a Data Science Administrator (DSA). The DSA will develop and maintain a modern system of data collection and analysis and provide ongoing leadership and guidance to staff and partners. The hiring of a DSA equips HSD with the long-term leadership required to build on the principles and guidelines established during the initial implementation. In addition, HSD should identify external resources, including experts from academia or other sectors, that neither receive City funds nor are residents of Santa Monica. These resources can provide a valuable source of unbiased, objective information.

Privacy concerns understandably arise when collecting sensitive data about residents. Though the City is already committed to ensuring the privacy of its clients, and current processes guarantee that sensitive information is not shared, changes in data collection processes require that City staff be mindful so that privacy is maintained. The crux of any privacy discussion is consent; the City should identify and address any benefits which are contingent on the sharing of potentially sensitive, personally identifiable information, such as religious affiliation or immigration status. The City should also recognize other related concerns, such as how data is collected, how data will be used, and with whom it will be shared. Privacy will be discussed in greater detail in Action 9, but it is necessary to remember the importance of privacy at every step.

The execution of this recommendation will require the City’s clear focus, commitment, and careful management of resources. The following section of the report breaks the recommendation into key action items, separated by phase, which the City must accomplish to develop a new outcome measurement system.


Phase One: Engage During Phase One, HSD will identify an implementation consultant and prepare City staff for the upcoming process. They will hire a Data Science Administrator with expertise in data analytics. Engagement with the community and a network of Subject Matter Experts will provide the public with updates and elicit crucial insight from the community. A pilot project will test outcome measurement techniques. In addition, an interim solution will be established to augment the effectiveness of current project reporting. By the end of Phase One, the city will be prepared to design a robust outcome measurement system.

ACTION 1: Project Launch

Prepare City staff for the design and implementation of the outcome measurement system.

It is important that the City quickly identifies an implementation consultant to ensure successful project kick-off. This role can be filled by a City staff member or by an independent contractor. The consultant will work closely with HSD staff and coordinate with the Office of Wellbeing, the Chief Performance Officer, and external subject matter experts to ensure alignment with strategic initiatives and stakeholder priorities. The implementation consultant will also need to build and sustain the consensus of senior City leaders, service providers, and the community throughout the process to deliver a comprehensive outcome measurement system that remains aligned with City goals.

The ideal consultant will immediately establish a firm foundation to successfully support each phase of the design and implementation of the outcome measurement system. The consultant will focus on designing the conceptual framework, convening and facilitating initial groups of experts, and developing the skills of HSD staff.

The consultant must be well versed in community engagement, outcome measurement, project leadership, and building a network of experts, and will mentor HSD staff to increase their skills and knowledge in these key areas. HSD staff should convene regularly with project coordinators and key service providers to review targets and metrics in each area. Use of an internal team structure within HSD will ensure that any turnover among staff does not derail progress.

ACTION 2: Hire a Data Science Administrator

Hire a new staff member to support implementation and maintain the new outcome measurement system.

HSD requires an expert in human service analytics to provide ongoing project direction and maintain alignment with other City departments and external stakeholder interests. Unlike the implementation consultant, whose role is finite and will end once the outcome measurement system is designed and implemented, the Data Science Administrator will serve to expand HSD’s resources by providing ongoing leadership and guidance to HSD staff. In addition, the DSA will act as the custodian of the system, ensuring that it continues to grow and operate optimally.

In order to hire the best candidate, HSD must search widely for the right combination of skills and experience. Interviews should take a practical, skill-based approach and should utilize appropriate skill-testing instruments that incorporate sample datasets. Expertise in data analytics will bolster the knowledge

1


of existing staff, and enable HSD to develop and maintain a robust system based on best practices. This individual should have a strong background in data analytics along with relevant experience in human services administration. In addition, the ideal candidate will demonstrate the capacity to implement formal procedures for building and interpreting meaningful datasets.

The Data Science Administrator will work alongside the implementation consultant and will be key to the ongoing success of the outcome measurement system. The DSA will ensure continued alignment with best practices and current academic research, and that data resources actually deliver meaningful information to decision makers. As such, it is essential that the correct candidate be hired as quickly as possible.

ACTION 3: Community Engagement

Increase community engagement to determine how specific services impact the community and define project success.

While each project has developed strong relationships with the constituents they serve, establishing a group to review HSD’s portfolio of projects will require additional input from the community. Engagement should seek to utilize existing resources like Virginia Avenue Park’s parent groups, which structure community engagement through affinity groups. HSD will also engage the City’s advisory bodies, including boards and commissions, for essential feedback. Philanthropic organizations are vital partners to a successful collaboration and should be incorporated in any structured outreach. Initial tasks include establishing relationships with community leaders, engaging with members of the community impacted by HSD-funded projects, and convening regular community meetings.

Community members who depend on services provided by HSD are some of the best sources of information about the effectiveness of those services. While City staff and consultants can draw on lessons learned from other communities, the residents of Santa Monica themselves know intimately how well a project is working for them and if it is not, why not. Engaging with Santa Monica residents will allow City staff to discover what outcomes are most important to the community and how best to measure project success.

During the first phase, HSD staff will design a strategy for community engagement through regularly held meetings. Meetings should be scheduled to account for factors and strategies that inhibit or encourage participation by diverse households. For instance, in order to ensure that meetings are accessible, it would be helpful to hold the meetings outside of normal business hours. Meetings scheduled during times when childcare would be needed may limit participation by some households. Making childcare available may encourage households depending on the services delivered to attend.

Prior to convening any meetings, HSD will need to determine the metrics that define a successful engagement. Potential indicators include total attendance, audience diversity, documentation of needs, and discussion of successful interventions. Good initial discussion topics would include identifying the real-world change HSD-funded projects seek to accomplish, ascertaining how progress towards these goals is perceived, and generating additional project targets where appropriate. The final outcome measurement system will be greatly improved by community engagement and feedback.

A successful system provides as much utility to service providers and clients as any other stakeholder. Designing a system that makes possible reporting to many funders adds to the benefits for service providers. In addition to government funders, adding philanthropic funders and foundations to early discussions will support the broadest utility of the system.

https://www.smgov.net/Departments/CCS/content.aspx?id=33619


ACTION 4: Build and Expand Network of Experts

Identify and engage a network of experts to define best practices and explore current outcome research within different HSD service disciplines.

HSD staff will establish a network of local subject matter experts and refine HSD’s process for reviewing and integrating academic literature. Developing a knowledgebase of current research related to HSD’s services will ensure a solid foundation for building projects that are consistent with best practices. Utilizing expertise to align partners towards common goals will serve to further increase the impact of invested resources.

HSD provides a wide range of services that require different criteria for defining success. Luckily, the City has a wealth of relevant expertise available via current residents, local colleges and universities, and world-class consulting practices. While experts that reside or are employed in the City of Santa Monica are a valuable resource, they are also members of the community. It is important that City staff have access to confidential, expert advice that is not influenced by external factors such as future contracts or personal household considerations.

In order to expand their current supporting network, the City should seek formal relationships with nearby experts, academics, industry leaders, and potential partner organizations. Building a network of local subject matter experts who can provide key insight and recommendations will ensure that sufficient rigor is embedded into the design and administration of services as well as the outcome measurement system.

ACTION 5: Outcome Pilot

Use a sample of existing HSD projects to organize a pilot project that re-engineers outcomes aligned with Council strategic initiatives, metrics, and action plans.

During the first phase, the City will work with community partners to select a sample of existing HSD projects to pilot new outcome measurement techniques. The project pilots the union of the newly defined outcome measures and existing data collection and reporting capacity. The Data Science Administrator will manage this project and use this pilot to identify opportunities and gaps that create barriers to full scale implementation. During the pilot project, a subset of projects willing to collaborate on this new endeavor will be identified to test the process of implementation. Using a pilot to test procedures minimizes disruptions and avoidable mistakes. In addition, building momentum through early success will ease the adoption of new technologies and procedures by encouraging participants and relieving anxieties.

It is important that the projects selected for the pilot project utilize service providers that have previously demonstrated highly effective staff, excellent administrative capacity, and a willingness to partner with the City on new projects. This will enable swift identification of problems and minimize other variables that increase the risk of failure.

Through this facilitated process, the HSD team focused on the pilot will need to reach consensus on a number of procedural methods, including:

• Review and redefine the intended outcomes of each pilot project;

• Determine what specific data each project will collect to measure achievement of expectations; and

• Decide the reasonable burden (such as number of hours) that service providers should be expected to spend collecting such data.

During the pilot phase, the HSD team must create methods for evaluating their projects and implement


needed changes if the pilot is not delivering the required information. Once the pilot establishes enough proof of concept so that the city can apply a consistent strategy, HSD staff will be well positioned to define the goals and processes of the outcome measurement system and discuss them with local partners and other communities defining human services outcomes.

As discussed, several similar efforts are underway in other communities around the country. Connecting to cities or counties engaged in these efforts will provide valuable insight into potential pitfalls and effective solutions. In addition, communication will help City staff establish new connections for continued collaboration and support. Concrete questions developed at the conclusion of the pilot will ensure all discussions produce actionable information and the necessary input to guide Santa Monica’s efforts.

ACTION 6: Implement Interim Reporting Solution

The implementation consultant and Data Science Administrator will develop a simple online process for increasing the efficiency and effectiveness of current project reporting.

Working with appropriate internal and external partners, the DSA will lead the design of an interim reporting solution that can be deployed immediately and that takes advantage of the City’s existing software and data. A comprehensive and optimal outcome measurement system—the goal of this recommendation—will take time to develop, so an intermediate reporting solution will allow HSD to improve management of data currently being collected. This process side steps privacy related issues and other concerns by using only aggregate level data, which avoids the inclusion of personal identifying information and instead generates a cumulative number.

In order to design the interim reporting solution, the implementation consultant and the Data Science Administrator will utilize a simple process inventorying the data currently collected by providers. This process will not require new data collection procedures at existing grantee sites. Rather, instead of submitting a paper report, as is the current practice, the same or similar information would be entered into an online form or other virtual tool.


Phase Two: Design

During Phase Two, the City will design the outcome measurement system and develop a plan for its implementation. Before the system can be designed, HSD staff will need to assemble a library of outcomes that fits their projects. Once the library is complete, HSD can proceed with system design and decide on a technology component that supports service delivery. HSD must pay careful attention to privacy issues when designing the system to make sure that participant data is properly safeguarded. Once the system is designed, HSD will receive and respond to feedback from external and internal partners. By the end of Phase Two, the City will have a clear plan for the implementation of an outcome measurement system.

ACTION 7: Library of Outcomes

HSD staff design a library of outcomes.

At the beginning of Phase Two, the implementation consultant and Data Science Administrator will collaborate with HSD staff, the Chief Performance Officer, external partners, and City staff to assemble a comprehensive library of updated and refined outcomes and metrics for all human services projects. Outcomes will be aligned with City goals and initiatives, and the library will contain all information necessary to deploy the proposed outcome measurement system across HSD service providers.

The Data Science Administrator will work closely with HSD staff to incorporate key lessons learned from the pilot project and community engagement. With a completed library of outcomes, HSD will increase its capacity to document and evaluate project operations and to identify the best-fit outcome measures for each service provider.

ACTION 8: System Design

Analyze data needs, design the outcome measurement system, and determine a coherent program for its implementation.

At this stage, the Data Science Administrator is ready to design the outcome measurement system, identify a technological component, and create an implementation plan. The DSA must determine what data the City already possesses, what data the system would need to acquire, and the anticipated cost of developing any additional required data. The DSA must carefully consider which sources of data will best suit their system design, taking privacy into account. The privacy element is discussed in greater detail in Action 9.

Once it is clear what data is necessary, the Data Science Administrator will work with the implementation consultant to decide on the technology required to implement the outcome measurement system. The purpose of the technology component is to receive, integrate, and organize data from multiple sources into a central location or database. Once centralized, the data can be used to create dynamic, ad-hoc reporting to inform more meaningful project analysis. This new technology will improve data collection methods by taking into account how partners already collect and utilize their data to inform policymaking and improving upon those methods for increased efficiency. Successful implementation of new technology will make HSD and their partners more effective and better able to help the people in their community.

Different technologies offer distinct benefits. Matching the right technology to the right project is part of a process known as requirements gathering. In order to identify which technology to use, the DSA will

2


lead a requirements gathering process, which will run concurrently with the establishment of the library of outcomes. At the conclusion of the requirements gathering process, HSD will have documented the needs of each project and determined the degree to which any technology meets those needs. Specific tools may easily collect the necessary information without further burdening service providers with data collection. Soliciting the perspective of information system developers on formal process mapping can support the identification of efficiencies and how to leverage them effectively. Discussions may include:

• Physical layout of entry and exit points,

• Reasonable data driven assumptions that can be programmed to application specifications, and

• Opportunities to utilize existing data in new ways, for example:

○ Use existing data for data quality checks; if staff enter a date of birth that indicates the client is a young child and they also indicate that the client is a veteran, then the staff would be prompted to resolve the conflicting data,

○ Refine access to ensure the most valuable data for decision making is easily and quickly available, and

○ Ensure ad hoc reporting can occur quickly and easily to improve ownership and utility in decision-making.

This process must result in a complete set of functional requirements that the City can use to evaluate any technology in the event that it decides to release an RFP for a commercial off-the-shelf system.

In order to identify the appropriate technology to pair with the outcome measurement system, the DSA may consult with internal City experts. ISD would be an ideal partner, as they have extensive experience with the practical evaluation of technology products. Many potential products struggle with import and export of data – two functions critical to the success of any system. ISD may be able to share existing internal or subscribed software that may meet HSD’s needs. Alternately, other appropriate City departments or external partners may be consulted to achieve a similar result. In any case, all parties must give careful consideration to the total cost of ownership, system and data control, and goals of any software used.

Total cost of ownership includes software licensing fees, support fees, hardware purchases, and technical support costs. Some communities have adopted open source software that requires significant investment and expertise and data analysis, while other communities have chosen fee based software. Costs vary as a function of the number of users, the nature of the projects involved, the volume of records entered, and the complexity of reports required. During the initial year, costs will be higher to cover purchase of hardware, data integration, and report configuration. An estimate of year one costs would be no more than $350,000. This does not include potential staffing costs. In years two and three, costs will likely decrease to less than $150,000 annually.

Using the requirements gathering process to examine technology the City of Santa Monica already owns may reveal that the right long term solution already exists. While the development of the long-term solution is under way, significant gains in efficiency and effective reporting can be made by simply moving the existing reporting process online. As discussed in Phase One, any interim solution should be selected and implemented with the long-term vision in mind. At the conclusion of this process, the Data Science Administrator will complete a detailed implementation plan for the outcome measurement system and the overlaying technology solution.

ACTION 9: Privacy

Data acquisition, issues of privacy, data use, and disclosure must be addressed.

In discussing the best approach for acquiring data, issues of privacy must also be explored. Looking at the overall outcome library and understanding that some data that is desirable from an analytic perspective may also be intrusive or inappropriate from a civil society standard is an important step in the process.

It is important to use aggregate, unduplicated, and anonymous data whenever possible as this improves


the flexibility and usability for analysts. Statistical and cryptographic procedures should be employed to ensure that agencies can reliably aggregate individual records across systems without losing the unique identity of the individual. Maintaining this utility without disclosing the identity of the individual is also possible. In the design of datasets, this property is known as k-anonymity. Developed by Dr. Latanya Sweeney, director of the Data Privacy Lab at Harvard University, this concept allows analysts to maintain the maximum utility of data without compromising the privacy of individuals within the dataset.

Regardless of the technology chosen, both the City and its non-profit partners must be clear and transparent about their intended use of the data at the time it is collected. To do so, they will need to build specific use and disclosure commitments into contracts, end-user agreements, and individual intake forms. Improper use of sensitive data can have a significant negative effect on an individual and must be prevented. Depending on the nature of the technology involved, the hardware, and the context in which data is collected (at a project site, through street outreach, in a participant’s home), specific controls and procedures may be required. Review of City policy and other resources should result in a clear set of expectations for the City, service providers, end-users, and service participants.

An excellent resource for designing a process that supports transparent decision-making and accountability for all stakeholders in the system was developed by the Federal Trade Commission (FTC) and is known as Fair Information Practice Principles, or FIPPs, and could assist the process. The core principles of privacy addressed by FIPPs are notice/awareness, choice/consent, access/participation, integrity/security, and enforcement/redress. FIPPs is outlined in greater detail in Appendix E. Having a well-documented process for registering and resolving privacy complaints, along with an external appeals process, will also help support the fair collection and use of data. In fact, formal mechanisms and documentation for sign-off by the City, the community, and service providers will assist in ensuring transparency to decision-making in the event that future changes become necessary

Clear authority regarding who – within the City – owns the system and is authorized to make changes, as well as how they will communicate changes, should be included in any documentation. External providers and end-users should execute commitments to adhere to the established standards. Documents should transparently identify specific consequences including repayment of HSD funds, termination of grants and/or employment, and - in the most severe circumstance – criminal or civil prosecution.

https://epic.org/privacy/reidentification/Sweeney_Article.pdf

http://dataprivacylab.org/people/sweeney/

http://dataprivacylab.org/people/sweeney/

https://en.wikipedia.org/wiki/FTC_Fair_Information_Practice


ACTION 10: Continuous Internal and External Feedback

Acquire continuous feedback from external and internal city partners to remain in alignment.

Throughout Phase Two, HSD will continue to provide progress updates to the community at regularly heldpublic meetings. At these meetings, residents can evaluate the success of the new system within theircommunity and provide valuable feedback which will be responded to through a formal process. Using the community engagement process, the Data Science Administrator should ensure Santa Monica residents fully understand the proposed outcome measurement system and implementation plan.

At the same time, HSD should seek buy-in and feedback from service providers, domain specific boards, commissions, and advisory bodies. Concerns around privacy, stigma, and easy access to life-sustaining services should be addressed. Without sacrificing required functionality, HSD should incorporate feedback from external HSD partners into the outcome measurement design.

As important as external stakeholder buy-in is, consensus from internal HSD partners is also essential. The varying missions of different departments may come into conflict during this process and the Data Science Administrator will need to meet with internal partners—including interdepartmental partners and community stakeholders—to collect feedback. It is important to meet with the internal City users of the data to determine if the new data will introduce new constraints. For example, if a performance management goal or SamoStat metric is dependent upon access to a HSD data element, then the team will want to understand the potential impact before making any changes to that particular element.

The Community Engagement Process


Phase Three: Implement

During the final phase of the recommendation, the City will begin a phased implementation of the outcome measurement system. HSD will incorporate the new system into the decision-making process for allocating staff resources. In addition, HSD will design a structure of escalating support to help service providers improve their projects. By the end of Phase Three, the City of Santa Monica will have achieved its goal of improving data collection and project outcome analysis to enhance the lives of project participants.

ACTION 11: Implement the Outcome Measurement System

Implement the outcome measurement system over an 18-month period.

The optimal method for implementation is a phased approach. Any new procedure requires real-world testing to identify errors and inefficiencies. Using a subset of pre-selected projects to assist with this task will minimize disruptions and negative experiences with the new process. Building momentum through early success will support the adoption of the new technology and procedures. Finding service providers that have previously demonstrated highly effective staff, excellent administrative capacity, and a willingness to partner with the City on new projects will enable swift identification of actual problems and minimize risks that may contribute to project failure.

ACTION 12: Incorporate System into Decision-Making

Incorporate the outcome measurement system into the decision-making process for allocating HSD staff resources.

As the phased rollout progresses, HSD must integrate the resulting data into the decision-making process. Grant agreements should require participation in the new outcome measurement system and technology solution, and funded staff positions should incorporate timely data entry as a part of annual evaluations. In addition, HSD’s process for awarding new and renewal grants should incorporate how HSD projects meet expectations related to the outcome measurement system. Once a majority of service providers use the outcome measurement system, HSD can begin to understand new baseline requirements and goals for both the delivery of units of service and the effectiveness of each project. Community partners, service providers, and other stakeholders should review these requirements and goals to ensure buy-in to the process and the achievability of new targets.

ACTION 13: Project Support

Design a schedule of escalating programmatic support for service providers.

Over time, these new performance expectations and metrics should be incorporated into the regular management of service providers and grantees. HSD can encourage participation and assist struggling

3


projects as it gains insight into the operational capacity of projects. If projects are struggling, HSD can offer programmatic support for those service providers.

HSD can support projects by helping them carve out distinct achievable goals, understand progress towards those goals, and encourage commitment to creative new solutions. HSD should have a set of resources available to assist providers with different challenges that emerge. Returning to resources developed in Phase One, the engagement of subject matter experts will assist HSD in targeting the correct resources. Modes of support likely include monitoring for contract compliance, providing opportunities for project staff professional development, ensuring access to subject matter experts, and supplying technical assistance.

Additional HSD recommendations may include coaching, administrative support, technical assistance, or corrective actions. HSD can provide access to the network of experts, or professional development opportunities, improve their assessment instruments or data entry capabilities, or make structural or project design changes. In this way, HSD can help transform the effectiveness and success of struggling projects. Many projects will likely welcome access to these opportunities and a structured schedule of escalating interventions ensures all providers and HSD maintain shared expectations throughout the grant period.

ConclusionAfter completing the three-phase recommendation, the City will have implemented a new outcome measurement system, integrated a technology platform, enhanced staff capacity to measure project outcomes, and hired a leader in the Data Science Administrator to evolve the system. The City’s ability to measure project success will be enriched by a greater understanding of outcomes and new comprehensive techniques for measuring and processing the data collected. The system will allow the City and HSD to improve their decision-making process, manage human service projects, and assist struggling projects. These decisions will lead to stronger choices that benefit the entire Santa Monica community as they receive needed services.

Moreover, the City of Santa Monica will join communities around the nation as a pioneer of data-driven policymaking. The City will become a model of how to successfully combine data, project effectiveness, and technology to proactively deliver the best possible human services. The path forward is clear as the City uses this opportunity to improve human service delivery, enhance their decision-making process, and transform their approach to system design.


City of Santa Monica Service Provider Survey

In order to comprehend the current state of data collection and use, an online survey was sent to the City of Santa Monica’s HSGP grantees in February 2017. The survey was sent to seventeen providers and fifteen responded. It looked at agency composition, agency management, data collection, and the technology that these agencies currently use. It also sought to understand how a community wide client management system could support each agency’s needs and to ensure availability of the resources needed for successful operations.

The first section of the survey examined agency composition. Providers were asked about their target populations, the types of services they provide, their clients, and how they track the total number of services provided. The survey found that the respondent agencies are diverse in who they serve and can be dramatically different in the volume of services offered. For instance, while one agency served 25,000 people in a year, another agency served 64 clients.

The second section of the survey concentrated on agency management. Providers were asked how they update their policies and procedures and at what level they engage in policy decisions. Results demonstrated that the majority of agencies (66%) reviewed and updated their policies and procedures annually. Furthermore, agency involvement in City, state, and federal policy decisions varied, with agencies more engaged in collaborations with local service providers than federal policy decisions.

The third section of the survey centered on data collection and usage. Providers were asked whether they utilized an electronic data collection tool, what specific software solutions they use, the main benefits they hope to gain from using a City supported platform, and potential barriers to implementing a community wide data system at their agency. Respondents indicated a range of data collection tools, from the widely accessible Microsoft Excel to highly configured systems like the Homeless Management Information System (HMIS), along with numerous funder specified systems like CliniTrak. The possible benefits they identified were increased collaboration with other service providers, better communication, and better understanding of client needs.

The fourth and final section of the survey focused on each agency’s technology status. Providers were asked who in their agency was responsible for keeping data secure, what type of IT staff they employ, and the name of their leading IT staff person. The results found that most agencies have at least part-time IT staff, with two agencies reporting that they have no dedicated IT staff.

The greatest takeaway from the survey was that the needs and resources of the agencies vary significantly. The surveys were followed by in person interviews with a sample of grantees. Based on the information gathered, Canavan Associates and HSD were able to proceed with an improved understanding of the current state of HSD-funded providers and create a recommendation that better suited these agencies.

Appendix A


Appendix BCity of Santa Monica Staff Discussion Guide

Describe how you understand the benefits and structure of single platform, participant level data. Discuss possible drawbacks, open discussion to gauge understanding of underlying concepts and capacity.

Discuss data usage. What is the purpose of program data? What potential pitfalls and opportunities do you see with using participant level data for reporting? Daily/weekly/monthly/annual opportunities to use data? What policies/initiatives (fed/state/local) will benefit the most from new data driven insights?

How well does your portfolio articulate success and failure of project participants? How do you define success/failure of the grantee/project? Are these measurable? How so/not? Any recommendations for what data should drive those determinations?

What do you perceive as the commitment by agencies to demonstrate their program outcomes? Defined measurable outputs? Articulated goals? Measurable outcomes?

What is your confidence level in the agencies measuring their program performance accurately? Discuss concerns and how a data system could improve the quality and usefulness of the data. Discuss challenges of how the agencies measure program capacity (number beds/slots).

What’s going to be the biggest challenge to projects in your portfolio implementing this system? What support can the City structure/make available to assist in meeting those challenges? What resources would you like to have for yourself to assist in ensuring a successful implementation?

Is there anything you wanted to talk about that we did not get a chance to cover? Any key people in the agencies who you think might be helpful to launching and supporting this project? Can we reach out to them at some point?

1

2

3

4

5

6

7


Santa Monica Service Provider Discussion Guide

Interviewer will conduct interview with survey in hand and ground questions in survey responses.

Describe benefits and structure of single platform, participant level data as an abstract concept. Discuss possible drawbacks, open discussion to gauge engagement and understanding of underlying concepts.

Discuss mission of organization. Demonstrate knowledge of operations and/or goals. Do these tie into larger national goals or overarching schemas? Engage to determine how rigorous agency’s commitment is to outcomes. Defined outputs? Correct formulation of outcomes (change accomplished)? How does the agency actually measure that change?

How does the agency understand success and failure of project participants?

Diving deeper into discussion of outputs and outcomes. How long has the agency used the framework (if using)? What analysis is done on the data and how rigorous is this analysis? Has the agency updated outcomes or measures based on experience?

How does the agency’s mission and outcomes align/not align with the City’s Cradle to Career Initiative? Opportunity Youth? Other significant City of Santa Monica initiatives?

Review the population selections that the agency made on the survey. Discuss population specific goals. Review service provided selections from survey; discuss service outcomes and measures.

Review scale of services (number beds/slots) discuss how the agency measures projects at capacity, below capacity, and how service demand informs project scale decisions.

Based on scale, what kinds of special considerations to timing of data collection (scan cards vs. interviews, sampling plan vs. every client, live data entry vs. delayed data entry).

1

2

3

4

5

6

7

8

Appendix C


Appendix DSteps to a Standardized Reporting Platform

The City of Santa Monica currently employs an offline process to collect the various data elements included in the project and fiscal status reports required by City-funded agencies. This data is aggregated at the project level, duplicated within projects, and is not longitudinal. Even with these constraints, moving this process to a simple web-enabled database will immediately allow HSD to more swiftly analyze data in response to grant applications, public inquiries, and service provider requests.

The requisite level of skill to produce such a system currently resides within ISD. The privacy implications for this data are minimal as no personally identifiable data and very little sensitive data are being collected. Further, the level of effort to design, implement, and sustain this system should be minimal. Electronic data collection and reporting tools must articulate the need or business requirements of the system users. The primary goals are:

City staff receives aggregate project data in an electronic format;

City staff can export the aggregate data to review performance of individual projects or across multiple projects; and

Agency staff have a simple way to submit aggregate data to HSD.

To achieve these goals, the following describes the functional requirements, testing process, and implementation steps.

THESE FUNCTIONAL REQUIREMENTS DESCRIBE THE WAY IN WHICH THE STANDARDIZED REPORTING PLATFORM SHOULD BEHAVE OR PERFORM:

a. General:

i. Low cost hosting either internally by the City of Santa Monica or through an existing provider that offers hosting solutions to the City;

ii. Web-based interface for end users, compatible with low bandwidth devices;

iii. Easy to use digital forms that are consistently available online and require minimal training;

iv. Web-based interface for end users;

v. Allows for interface workflow or sequence of data collection questions to be adjusted;

vi. Customization of questions and response values by City staff;

vii. Easily set-up collection forms with minimal training;

viii. Automatically assign a unique record number for each submission;

ix. Create automated log of when data is submitted and who submitted the data;

x. Display automatic flags for potential data quality issues (e.g. blank fields);

xi. Allow end user to correct data entry issues.

1

1

2

3

GOALA uniform electronic submission process for Human Services project status reports, permitting staff to receive agency data in a format that allows for easier analysis and portfolio-wide reporting.


Appendix D

DESIGN REFERENCE FOR THE STANDARDIZED REPORTING PLATFORM: a. The goal is a low-cost form similar to Google Forms or Survey Monkey;

b. Design a simple back end database and front end user interface for data entry and for running reports that can be exported in common file formats;

c. Set up reporting form with common questions and response categories;

d. Develop a simple data entry and report guide with sample report form.

HSD FORMS DEPLOYED IN THE STANDARDIZED REPORTING PLATFORM: a. Existing reporting structures can be replicated in the new online forms;

b. Forms will use existing reporting requirements with minimal additions.

TEST THE NEW STANDARDIZED REPORTING PLATFORM: a. Select two or three projects that will represent the array of agencies currently reporting to

the City of Santa Monica;

b. Train agency/project staff to use the new reporting tool;

c. Train selected City staff to review and export project data from the tool;

d. Have projects make submissions for three consecutive months;

e. Assess solution:

i. Does data entry process work as expected for both City of Santa Monica staff and end users at the agencies?

ii. Does the data export as anticipated?

iii. Are exports usable and do they meet the goals of the reporting solution?

f. Make necessary adjustments to selected software solution and retest.

IMPLEMENT THE NEW STANDARDIZED REPORTING PLATFORM: a. Create instructions for collecting and submitting uniform data based on final iteration;

b. Provide training and support to City staff and end users at agencies;

c. On selected “go live” date, all agencies will transition to the new reporting solution and will be required to submit aggregate data through the new reporting process.

2

3

4

5


Appendix EFTC Fair Information Practice Principles (FIPPs)

There are common privacy principles adopted and promoted by the Federal Government that are used across the country. These basic principles can be used to help ground and inform discussions on fair data collection and usage activities in Santa Monica. An updated version of these principles can be found in the 2011, Fair Information Practice Principle (FIPPS). The principles are a foundation for understanding privacy and developing policies specific to the data collection and usage. The following is an excerpt from this update.

a. Access and Amendment. Agencies should provide individuals with appropriate access to PII and appropriate opportunity to correct or amend Personal Identifiable Information (PII).

b. Accountability. Agencies should be accountable for complying with these principles and applicable privacy requirements, and should appropriately monitor, audit, and document compliance. Agencies should also clearly define the roles and responsibilities with respect to PII for all employees and contractors, and should provide appropriate training to all employees and contractors who have access to PII.

c. Authority. Agencies should only create, collect, use, process, store, maintain, disseminate, or disclose PII if they have authority to do so, and should identify this authority in the appropriate notice.

d. Minimization. Agencies should only create, collect, use, process, store, maintain, disseminate, or disclose PII that is directly relevant and necessary to accomplish a legally authorized purpose, and should only maintain PII for as long as is necessary to accomplish the purpose.

e. Quality and Integrity. Agencies should create, collect, use, process, store, maintain, disseminate, or disclose PII with such accuracy, relevance, timeliness, and completeness as is reasonably necessary to ensure fairness to the individual.

f. Individual Participation. Agencies should involve the individual in the process of using PII and, to the extent practicable, seek individual consent for the creation, collection, use, processing, storage, maintenance, dissemination, or disclosure of PII. Agencies should also establish procedures to receive and address individuals’ privacy-related complaints and inquiries.

g. Purpose Specification and Use Limitation. Agencies should provide notice of the specific purpose for which PII is collected and should only use, process, store, maintain, disseminate, or disclose PII for a purpose that is explained in the notice and is compatible with the purpose for which the PII was collected, or that is otherwise legally authorized.

h. Security. Agencies should establish administrative, technical, and physical safeguards to protect PII commensurate with the risk and magnitude of the harm that would result from its unauthorized access, use, modification, loss, destruction, dissemination, or disclosure.

i. Transparency. Agencies should be transparent about information policies and practices with respect to PII, and should provide clear and accessible notice regarding creation, collection, use, processing, storage, maintenance, dissemination, and disclosure of PII.¹

¹ National Strategy for Trusted Identities in Cyberspace, Enhancing Online Choice, Efficiency, Security, and Privacy at Appendix A

(2011), https://www.hsdl.org/?view&did=7010

https://www.hsdl.org/?view&did=7010

Tasks Task OwnerDuration (Days)

Level of Effort in Hours

Y1M1 Y1M2 Y1M3 Y1M4 Y1M5 Y1M6 Y1M7 Y1M8 Y1M9 Y1M10 Y1M11 Y1M12

Task ID Project Design Phase 1

1.1

Act

ion

1 - P

roje

ct La

unch

Articulate key attributes and skills required of an implementation consultant along with proposed deliverablesHuman Services Division

(HSD)30 15

1.2 Implement a procurement process and hire an implementation consultant City of Santa Monica 45 3

1.3Enter into a contract with an implementation consultant (please note that if it takes longer than anticipated for the City to execute a contract, the timeframe for other tasks may need to be adjusted)

City of Santa Monica 30 3

2.1

Act

ion

2 - H

ire a

Dat

a Sc

ienc

e Ad

min

istra

tor

Announce open position for Data Science Administrator with business expertise in human services and exper-tise in data analytics

HSD 30 5

2.2 Use practical, skill-based interviews to ensure hiring the correct candidate HSD 30 80

2.3 Select and hire the top candidate HSD 14 15

2.4Orient the new Data Science Administrator to city practices and the outcome measurement system implementation plan

HSD 14 40

3.1

Act

ion

3 - C

omm

unity

En

gage

men

t

Identify clear goals for engagement HSD 10 3

3.2Determine the time and location of community meetings and decide on metrics that define a successful meeting

HSD 10 5

3.3Hold monthly community meetings on the scheduled dates to share knowledge about project design and project success

HSD Ongoing 67.5

4.1

Act

ion

4 - B

uild

and

Exp

and

Net

wor

k of

Exp

erts

Build a network of local subject matter experts (SMEs) to provide insight on the design and administration of services provided, and on design and measuring services

HSD 90 60

4.2Provide SMEs and Academics with access to data sources or study opportunities that can foster a long-term relationship

HSD Ongoing 92

5.1

Act

ion

5 - O

utco

me

Pilo

t

Work with the implementation consultant to form an HSD staffed workgroup and begin facilitating weekly meetings

HSD 7 5

5.2Come to consensus on which three existing HSD grantee projects to measure and how to execute and evaluate the pilot project

HSD Workgroup 60 25

5.3Define the intended outcomes for each project, determine specific data to collect, decide the reasonable burden expected of service providers to collect data, and design a method for evaluating the project

HSD Workgroup 30 35

6.1

Act

ion

6 - I

mpl

emen

t Int

erim

Re

porti

ng S

olut

ion

Utilize a simple process to inventory data currently collected by providers and submitted to the CityData Science

Administrator (DSA)30 20

6.2Design an interim reporting solution to deploy immediately, taking advantage of the City’s existing software and data

DSA 30 30

6.3 Implement the interim reporting solution DSA 90 30

Project Design Phase 1 - Engage





7.1

Act

ion

7 - L

ibra

ry

of O

utco

mes

Incorporate lessons learned from the pilot project and community engagement to determine a thorough library of outcomes

DSA 60 90

7.2 Document project operations and identify the best-fit outcome measure for each service provider HSD 30 40

8.1

Act

ion

8 - S

yste

m D

esig

n

Analyze the current capacity of City data resources to populate specific outcome metrics HSD 60 30

8.2 Identify any metrics unsupported by current data resources HSD 60 15

8.3 Determine ideal sources for unsupported metrics HSD 60 20

8.4 Identify the anticipated cost of developing missing data HSD 14 5

8.5If ideal sources include sensitive, personally identifiable information, work with stakeholders to reduce or eliminate privacy concerns

DSA 90 15

8.6 Assemble a complete set of functional and technical requirements that can be implemented and tested against DSA 60 25

8.7Identify efficiencies to be leveraged (i.e. adjusting workflow, program assumptions into application specifications, utilizing existing data in new ways)

DSA 30 20

8.8 Identify opportunities for automation and technology to relieve service providers of data collection burdens DSA 30 10

8.9Make a determination about building, acquiring, or leveraging existing technology to deliver the required functionality

DSA and HSD 30 30

8.10 Evaluate prospective products to ensure they will meet functional and technological requirements DSA and HSD 30 60

8.11Synthesize information from requirements gathering process into a coherent program that can be systematized and implemented within 18 months

DSA 30 25

9.1

Act

ion

9 - P

rivac

y

Ensure the proposed solution uses aggregate, unduplicated, and anonymous data as well as statistical and cryptographic procedures

DSA 15 10

9.2Review City rules, FIPPS, and other resources to set clear use and disclosure expectations, for the City, service providers, end-users, and service participants

DSA 180 20

9.3 Determine what policies and procedures will be developed prior to public engagement DSA 90 15

9.4 Facilitate discussions with the public on privacy and policies that require external stakeholder buy-in DSA 60 30

9.5Document policies and procedures covering privacy, data use, grievance procedures with stakeholders and SME’s

DSA 60 15

9.6Determine and document clear authority regarding who within the City owns the system, who is authorized to make changes, and how changes will be communicated

DSA 60 10

9.7 Ensure the proposed solution meets privacy standards DSA 30 15

10.1

Act

ion

10 -

Con

tinuo

us In

tern

al a

nd E

xter

nal F

eedb

ack

Over the series of regular community meetings (continued from Phase 1), provide regular progress updates to the community

DSA Ongoing 18

10.2Provide the community with documents that articulate clear expectations about its role in the development of the outcome measurement system

DSA 5 20

10.3Meet with external partners to ensure it is understood, assess level of buy-in, and determine if the new data introduces new constraints

DSA 90 40

10.4Meet with internal City partners to ensure it is understood, assess buy-in, and determine if the new data introduces new constraints

DSA 14 5

10.5 Identify and implement steps to mitigate constraints DSA 90 50

10.6Resolve any re-use issues (e.g. privacy, stigma, and easy access to life-sustaining services) among external and internal partners that may be a result of using data in a way not originally intended

DSA 180 50

10.7 Set internal policy that ISD and HSD review the system on a regular schedule DSA 30 10

10.8 Set policy to incorporate new projects into the system as they are developed DSA 30 15

10.9Provide HSD a formal project plan, milestones, phased roll-out of the outcome measurement system and any procurement process for the an appropriate solution

DSA 27 50

10.10 Determine if mobile applications or other modality is necessary for data acquisition DSA and HSD 549 40

Project Design Phase 2 - Design





11.1

Act

ion

11 -

Impl

emen

t the

Out

com

e M

easu

rem

ent S

yste

m

Develop a phased implementation schedule across the portfolio of projects DSA 30 25

11.2Select a sub-set of projects with highly-effective staff, administrative capacity and willingness to partner in implementing and testing the system

DSA 30 10

11.3 Prepare, train, and implement the new outcome measurement system with the pilot Phase 1 projects DSA 60 120

11.4 Test for errors and inefficiencies among projects in the first phase of the implementation DSA 60 60

11.5 Prepare, train, and implement the new outcome measurement system with the next phase of projects DSA 30 70

11.6 Test for errors and inefficiencies among projects in the second phase of the implementation DSA 60 40

12.1

Act

ion

12 -

Inco

rpor

ate

Syste

m in

to

Dec

ision

Mak

ing

Once projects consistently and successfully participate in the new system, integrate resulting data into HSD process to embed system use at the institutional level

DSA N/A 40

12.2Ensure grant agreements require agency participation and that the agency implements policy that staff posi-tions incorporate timely data entry as an element of annual evaluations

DSA 546 20

12.3Ensure the process for awarding new and renewal grants from HSD include meeting expectations related to the outcome measurement system

DSA N/A 15

12.4Set baseline requirements and goals for both the delivery of units of service and the effectiveness of the project once there is sufficient coverage of a majority of service providers

DSA 79 40

12.5Facilitate review of baseline requirements and goals for unit and service delivery by community partners, high functioning service providers, and other stakeholders to ensure buy-in to the process and assess the reason-ableness of new targets prior to formal implementation

DSA 79 30

13.1

Act

ion

13 -

Proj

ect S

uppo

rt

Implement a set of resources (i.e. technical assistance, funding, capacity building materials) to assist providers with different challenges that emerge.

HSD Ongoing Ongoing

13.2 Engage subject matter experts to assist HSD in targeting the correct resources HSD Ongoing Ongoing

13.3Monitor for contract compliance, providing opportunities for project staff professional development, ensuring access to subject matter experts, and supplying technical assistance

HSD Ongoing Ongoing

Project Design Phase 3 - Implement

City of Santa Monica 2018 Outcomes Measurement System ...

Documents

Transcript of City of Santa Monica 2018 Outcomes Measurement System ...