DataShare
description
Transcript of DataShare
DataShare
Michael Fiorito, Kate Fitta NYC Mayor’s Office of the Criminal Justice Coordinator
New York City’s
Enterprise
Integration
Platform
Kris Carpenter, Jon NatarajanNYC Department of Information Technology and Telecommunications
Agenda
DataShare Overview – Progress to Date
Michael Fiorito
NYC Mayor’s Office of
the Criminal Justice Coordinator (CJC)
DataShare Overview – Progress to Date
Rewind to 2003…
Criminal justice agency CIO’s, CJC, and DoITT convened a workgroup to discuss the possibility of introducing new technology to meet their business needs.
The group produced a report that summarized their research and outlined possibilities for new technology.
The group also recommended hiring a small team of dedicated CJC and DoITT staff to coordinate the overall effort (i.e., a project team) and build a sustainable integration program.
DataShare Overview
Why not replace an existing mainframe system utilized by criminal justice agencies (DataShare 1.0) with a scalable enterprise solution?
DataShare was designed for citywide integration, but deployed initially for 17 agencies (city, state, not-for-profits).
The Public Safety Portal, which is the user-facing dimension of the City’s integration solution for criminal justice agencies, includes both administrative and business user functions.
The initial phase concluded in July 2006 with the deployment of the Public Safety Portal and a set of GJXDM exchanges.
This undertaking was a Mayoral initiative, but agencies did not have a mandate to participate
Phase 1 – Facts & Figures
12 months for requirements gathering, solution analysis, systems integration procurement and solution selection
18 months to complete implementation – 2 major deployments
17 city, state and nonprofit agencies integrated via hub-and-spoke architecture
35-40 FTEs comprised the dedicated project team – City staff and consultants combined
60+ agency-side resources brought to bear during 1st phase
Approximately $12M spent on professional services, HW, SW
100% of post prod support provided by City of New York upon Phase 1 completion. No ongoing costs.
Juvenile Case Management System Law Department Dept. of Probation Dept. of Juvenile Justice (DJJ)
Richmond County DA
Public Safety PortalProviders
Special Narcotics Prosecutor
Queens County DA
Kings County DA
Bronx County DA
Legal Aid Society
Manhattan DA
Criminal Justice Agency
Dept. of Corrections
NYPD
18B Web (Assigned Counsel)
Office Court Administrator
DataShare – Current Landscape
ReceiversPublic Safety Portal
DataShare
Data Exchanges/Services
Agencies connected to new DataShare infrastructure and providing or receiving at least one data exchange via the hub or portal
Criminal justice data that is shared between and among participating agencies via the hub and portal
The user-facing dimension of the City’s integrated justice solution hosted by DoITT on CityShare
Agency Participation
Public Safety Portal Functionality and Links to Applications
Criminal Justice Integration Platform (Hub)
Phase 1 Providers• NYPD• NYC DOC• Criminal Justice Agency (CJA)• OCA
Phase 1 Receivers• NYPD• Department of Probation• Department of Correction• District Attorneys (5)• Special Narcotics Prosecutor• CJA• Law Department• Dept. of Juvenile Justice (DJJ)• OCA• Legal Aid Society• Assigned Counsel Plan• HRA• DoITT (on behalf of CJIS)
The behind-the-scenes, criminal justice segment of the Citywide integration environment hosted by DoITT
Core Hub Features & Functionality• Push functionality• Pull functionality• Query application• Notification application
Phase 1 Business Deliverables• NYPD arrest data push (legacy & GJXDM)• NYPD arrest notification (GJXDM)• CJA ROR interview push and pull (GJXDM)• OCA supreme and criminal court actions push (legacy & GJXDM)• DOC inmate information (legacy & GJXDM)• DOC release notification (GJXDM)• DOC jail status query (GJXDM)• CJA NYSID – SSN match (legacy & GJXDM)
Core Portal Features & Functionality• Functionality for agencies to subscribe to notifications • Functionality for agencies to perform queries • Functionality for agencies to pull data• System administration for agencies • Reduced sign-on (RSO) for DNA HITS application• e911 application link • webCRIMS application link from OCA• e-Justice portal link from DCJS
Phase 1 – Scope and Deliverables
• Decisions about DataShare architecture needed to consider the autonomy of agencies
• DataShare had to be architected and designed in such a way that it could be scaled for citywide integration in future phases
• DataShare was intended to serve as a pass-through system, with minimal data stored centrally
• The deployment of DataShare 2.0 and decommissioning of DataShare 1.0 should have minimal impact on agencies and result in no loss of functionality for agencies.
• The City decided to adopt GJXDM as its transport standard for criminal justice services
Guiding Principles & Strategic Decisions
GJXDM Strategy & Approach
• Global Justice XML Data Model (GJXDM) is is intended to be a data reference model for the exchange of information within the justice and public safety communities. The Global JXDM is sponsored by the US Department of Justice (DOJ) Office of Justice Programs (OJP), with development supported by the Global Justice Information Sharing Initiative's (Global) XML Structure Task Force (GXSTF).
• City decided to implement both legacy and GJXDM versions of existing exchanges via new platform to facilitate agency transition and minimize risk
• GJXDM SME consultants worked with provider and receiver agencies to develop schemas and leverage national model as appropriate.
• GJXDM workgroup has formed among NYC stakeholder agencies to drive strategy going forward, especially collaboration with state and national initiatives; DOP, NYPD, DOC to play key leadership roles.
Portal Functions Available to Business Users
Applications Business Functions
Pull/Query Engine• User can execute distributed queries via the portal • Pull/Query engine processes the portal requests and provides
the results back to the user via the portal.
• Results can be viewed in multiple pre-defined formats.
• CJA Interview Pull
• DOC Jail Status Query
Notification Engine• Users can subscribe to various notifications via the portal.• The subscriptions are processed by the notification engine for
possible matches.• The resulting notifications will be delivered via email or via
the MQ infrastructure.
• NYPD Arrest Subscription
• DOC Release Subscription
Portal Administrative Functions
Applications Services/Functions
Administration - Allows the DoITT and
Agency Administrators to manage services. A service is any available function.
Provision pre-existing services to new receivers
Add new services upon completion of backend and adapter components.
Modify field filtering settings for pre-existing services.
•Provided Services - Add/Remove Field Level Permissions, Add New Service, Enable/Disable Services•Received Services - Review Received Services including what field level permissions receiving agencies have per exchange, group level settings for services
Operational Reports - Allows authorized
agency administrators to run operational reports. •Reports will be generated based on transactional meta-data.•Reports contain pre-defined content related to the transitions brokered by the DataShare hub.
•Volume Reports -Provide information surrounding the number of data services that flow through the system.•Service Detail Reports - Provide details regarding specific service transactions•Administrative Reports - Provide information surrounding administrative metadata and global reference data.
Public Safety Portal Services
DOC Jail Status Query – This function enables users to enter a NYSID, book and case number or a combination of first name, last name and date-of-birth via the Public Safety Portal and retrieve the incarceration status and history for the individual(s) that match the criteria. Jail status data includes information about the inmate's current incarceration status, incarceration history, warrant history and DOC housing history from 1999 - present.
CJA Interview Pull – This function enables users to enter a docket number or arrest number via the Public Safety Portal and retrieve the details of the corresponding interview the Criminal Justice Agency conducted with the defendant prior to their criminal court arraignment. Interview data includes information about the defendant's employment, residence, education and criminal history and is only available for interviews conducted from 2005 - present.
Subscriptions for NYPD Arrest Notification (Basic) - This function enables business users to enter a NYSID, name information or location information via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the individual or location specified by the criteria.
Subscriptions for NYPD Arrest Notification (Advanced) - This function enables administrators to enter criteria corresponding to up to 150 arrest fields via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the specified criteria.
Subscriptions for DOC Release Notification - This function enables users to enter a NYSID or arrest number or book & case number via the Public Safety Portal and subscribe to be notified via email when the corresponding inmate is released from or admitted to City correction custody.
Links to Applications - DNA HITS, e911, WebCRIMS
Public Safety Portal Services
Orchestrating services
Growing the community of participants
Key to Success: Ongoing partnership between DoITT, CJC and participating agencies
More to Come…
DataShare Architecture
Kris Carpenter
NYC Department of
Technology & Telecommunications (DoITT)
Information Sharing Capabilities
The DataShare & Public Safety Portal solution facilitates the sharing of information across city agencies providing business process integration for the participating agencies.
DataShare will provide the following services related to enterprise applications integration (EAI)– Business process routing (via publish/subscribe)– Data standardization– Web-based portal that can offer truly distributed business
applications – Business process auditing capabilities– Field level filtering for messages – Security
– 24/7 Monitoring
Existing Systems Installed Base
MVS VAX/VMS LINUX UNIX Microsoft FTP SMTP CICS Flat Files
Existing Systems
Oracle
Unix/Linux
File share
SQL Server
Web Services
LDAP
VAX/VMS
BEA WebLogic/Vignette
Microsoft
J2EE
MVS
DB2 DB2 ADABASE DATACOM VSAM ADMINS/VAX RDB CACHE MUMPS SQL SERVER ACCESS ORACLE EXCEL
Technical Architecture Requirements
Security – Authentication, Authorization, Encryption, LDAP
Performance – Handle 100’s of messages/second
Scalability – Able to add new data exchanges easily
Availability – 99.9% availability as ‘crime takes no holiday’
Extensibility – Expose functionality via different methods like Web services
Standards – J2EE, LDAP, GJXDM
Standards-based (J2EE, LDAP, SQL)
Enterprise Service Bus (WebSphere MQ and Broker)
Global JXDM
Public Safety Portal (Vingette & BEA WebLogic )
Involve the agencies that know the data
Involve the agencies that know the processes
Use existing City application implementation standards
Meeting The Needs – New Environment
DataShare: Message Oriented Middleware
DataShare
NYPD
DOP
DOC
OCA
CJA
Et all...
Hub and Spoke Architecture
DataShare: The Solution
The Solution consists of 6 key components
Criminal Justice End Users
Vignette Application Portal
J2EE Applications
BEA Weblogic Application Server
NotificationsWebSphere Everyplace
Access Server
WebSphereMessage Broker
(Enterprise Service Bus)
JMS-MQ
MQ
DataShare Data Base
RDBMS File Drop
Agency Adapter Platform
WebSphere MQ
iWay Adapter
MQ
RDBMS File Drop
Agency Adapter Platform
WebSphere MQ
iWay Adapter
MQ
Agency A Agency B
Mail ServerSMTP
Novell eDirectory
SSL
SSL SSL
SSL
Architecture & Components
Agency Adapter Platform Agency adapter programs and WebSphere MQ that is utilized as a gateway
between the criminal justice agencies and the enterprise service bus hosted at DoITT.
Enterprise Service Bus (Message Broker) Integration layer that provides message routing and integration business services
for citywide agencies. Novell eDirectory (DoITT’s enterprise directory server)
Enterprise directory server that stores citywide agency users and groups. It’s used to provide services such as distributed provisioning, authentication and authorization.
Database Storage of application and transactional metadata.
Public Safety Portal Web front end that provides access to web applications (such as subscriptions and
queries), user administration, and reports. Notification Services
Services provided to the Portal users, and the backend systems of criminal justice agencies, to subscribe to key transactions traversing the Enterprise Service Bus.
• Adapters key to joining systems:– Used in EAI to be non invasive to systems– Concept used in SOA as a Wrapper
• Allow existing formats to be utilized:– Suppliers convert to a standard– Receivers convert standard to legacy format for consumption
• Access supplied and/or received messages in an existing fashion:– File System– Queue– Database– Other Storage mechanism
Adapter Approaches
Enhancements Underway Currently (4-6 months)
Low hanging fruit
Federated query – giving users the ability to perform searches on NYPD, CJA, and DOC and retrieve a consolidated view of a person or event by entering a single set of criteria. The federated query capability will be greatly enhanced as more services are deployed through DataShare.
Service orchestration – when a user performs a DOC jail status query, they are given the option to initiate a DOC release or admit subscription from the jail status query output screen
Geocoding – exposing existing GIS services to the criminal justice participants via Datashare – incorporating a global address system.
Phase 2 Gameplan (12 – 18 months)
Grow the list of agency participants
We have some new agencies coming on as providers: DCJS and DOI
Some current receivers will become providers: DOP, BXDA, DANY, QCDA, LAW, DJJ, SNP, LAS
Some agencies/entities will become new receivers: ACS, HIDTA, OCFS
Integrate services deployed through DataShare into applications: DNA HITS, eArraignment
Deploy 15 – 20 new services/exchanges through Datashare
Working on delivering a host of new data exchanges with existing and new agencies
Goal is to build on what’s been deployed
Building the Highway
Highway has been built
Datashare is an enterprise system built to grow with the addition of new services and new providers and incorporate diverse environments
Now it gets really interesting
Contacts
Non-Criminal Justice Agencies
Jon Natarajan, DoITT ([email protected])
Kris Carpenter, DoITT ([email protected])
Criminal Justice Agencies
Michael Fiorito, Mayor’s Office (CJC) ([email protected])
Kate Fitta, Mayor’s Office (CJC) ([email protected])