Case Study : Ireland
description
Transcript of Case Study : Ireland
ITSIP
Case Study : Ireland
METIS Workshop, 4-6 July 2007
Data Management System (DMS)
2
ITSIPPresentation Agenda
Introduction and Overview
Statistical Metadata Systems and the Statistical Cycle
Statistical Metadata in each phase of the cycle
Systems and Design Issues
Organisation and Cultural Issues
3
ITSIP
Introduction & Overview: Project Governance
Project BoardCSO Directorate, CMOD Representative
Project Management Team
Margaret McLoughlin
Quality Assurance
Bob Heaton
Test Team
Edward LambeCathy Kelly
Rory NaughtonPaul O’Connor
Karl Stone
Business Analyst Team
Dave JenningsJohn Hayes
Marian McCannMairead Coughlan
Jennifer Banim
Executive Support
Cathy Kelly
Project Steering Committee
IT Director; Margaret McLoughlin, Dave Jennings; Cognizant Personnel
Project ManagerMargaret McLoughlin
Programme ManagerDave Jennings
4
ITSIP
Introduction & Overview:
Overall Strategy
Main drivers
EU requirement to move to Open Systems
Storage of all CSO data in a RDBMS
DMS to be business led with metadata driven processes
DMS to require use of common classifications (CARS)
DMS to require use of common dissemination database
CSO produced
IT Strategy for 1999-2002 & beyond (April 1999)
Data Warehouse / Data Management Strategy (November 1999)
CGEY (10 week contract) produced an implementation plan for CSO’s IT & Data Management Strategies (first quarter 2001)
5
ITSIP
Introduction & Overview: Project Objectives
ITSIP - Information Technology Strategic Implementation Programme
Deliver set of applications to meet the survey processing and dissemination needs of CSO
Migrate existing DEC Alpha-based applications to client server environment
Implement the new applications within the CSO Corporate Data Model
Interface these applications with the existing client server and Sybase systems
6
ITSIP
Introduction & Overview:Project Goals Obtained
Legacy Situation
Stove type approach to survey processing
150 systems written & maintained centrally
250 end-user applications written & maintained locally
SAS V6.12 & PC SAS V8.02, Excel, Access
Data Management System (DMS)
Consolidates legacy processes into a suite of survey processing system
Nine corporate applications reside on a corporate database storing all data and metadata required in the survey-processing lifecycle.
Promotes consistency and reuse across the various survey areas
7
ITSIP
Introduction & Overview:
Project Background
Stage A contract (6 Months) awarded to Accenture who compiled the Requirements Specification & High Level Architectural Design (April 2003)
Stage B contract (30 Months) awarded to Cognizant Technology Solutions Ltd., Chennai, India.
Project currently at performance testing phase
CSO first Irish Government office to use onshore/offshore outsourcing model
Cognizant staff onsite have ranged from 2-17 depending on need
Offshore team has ranged from 20-50 depending on Project phase
CSO ITSIP team ~ 15 staff
All above contracts were fixed price contracts
8
ITSIP
Introduction & Overview: Stage A Review (Accenture)
Requirements Analysis PhaseBottom up approach through 20+ workshops with over 100 CSO business users
Production of:
51 ‘As Is’ process descriptions with 61 process maps
Data Model (Swedish Data Model for Aggregation & Dissemination)
44 ‘To Be’ process descriptions
Consolidation of existing processes into 9 survey processing applications plus a security application
Design phase
High Level Architectural Requirements
High Level Architectural Design
High Level Performance Model
High Level Interface Requirements and Design Specification
Web Enablement Specification
9
ITSIP
Introduction & Overview: Stage B Review (CTS)
Stage B involved:
Further validation of Stage A system design for baseline DMS
Building DMS
Migration of historic data and integrity metadata from legacy systems
UAT
Migration of metadata from the UAT environment to the production environment
10
ITSIP
Introduction & Overview: Stage B Review (CTS)
Original Schedule 3 Nov 2003 - 2 May 2006
Latest Schedule 3 Nov 2003 - 29 August 2007
Delay of 16 months arose because of
delay in initial increment deliveries due to new requirements
delay in CSO testing due to underestimation of time required
extra functionality in the DMS
change in design needed for better performance
change from Windows to Unix for Sybase to cope with production load
Reworking of Java code to meet QA standards
11
ITSIP
Introduction & Overview: Data Migration Approach
Business areas identified for ~ 100 surveysminimum data and integrity metadata required to support normal survey processing
all required back versions of data including all historic data required to be migrated (back to 1939 in some cases)
any additional data which should be migrated
Cognizant produced required ETL scripts using Informatica (data restructured into cube format to use classifications)
ETL scripts run to move data to UAT environment
Same scripts will move all data to Production environment (including latest processed periods)
Minimum integrity metadata migrated to all relevant databases because of application dependancy on same metadata
12
ITSIP
Introduction & Overview: Process Metadata Migration Approach
Business areas should and would only enter process metadata once (in UAT environment)
Business areas identified process metadata entered during UATFor Survey Instance specific modules (SS, SM, DC & IMP): UAT Survey instance and Production Start Survey instance
For Survey specific modules (Reg. M., Agg. & Diss.): list of Registers, Aggregate, Weight & Disseminate Tables to be available in Production
Cognizant produced required ETL scripts to move this process metadata from the UAT to the Production environment
Comparison reports of metadata residing in UAT & Production will be used to validate migration process
Ultimate check will be another parallel run in the Production environment to ensure that all migrated metadata (process & mimimal integrity metadata ) is consistent and correct
13
ITSIP
Introduction & Overview: Recommendations for others
Consider carefully the organisation’s capacity for insourcing / outsourcing development work
Consider the time scale for implementation of the solution
Manage the change process well
Understand the complexity of the solution and in procurement stage reject very low bids
Assume contractor has no knowledge of your business
Ensure adequate in house skills in IT Design so IT Partner’s assumptions can be validated
Ensure adequate in-house skills in IT Partner’s development tools and proposed application infrastructure
Don’t accept IT Partner’s project plan lightly where your office’s resources are concerned
14
ITSIP
Introduction & Overview: Recommendations for others
Don’t under estimate the resources needed to (1) manage the project and (2) keep abreast of all project documentation
Consider carefully the items that are for sign-off, review and for information by you - these will have financial implications later
QA is more important than just ticking boxes but throughout the software development lifecycle should include:
reviewing the decisions taken to obtain technical solutions
examining the underlying deliverable
adherence to agreed standards
Allocate adequate time to reviewing the test process and test cases
Managing the contract requires high-level expert resources with project management, statistical and IT skills
Organisational support and commitment from top management critical
15
ITSIP
Introduction & Overview: Future Challenges
DMS is to Go Live in Sept 2007
Six month gradual implementation
New SAS environment as we move from SAS V6, on the VAX, and
PCSAS V8.02
New IT Strategy is required
16
ITSIP
Register Management
Sample SelectionCentral Business Register
Survey Management
Data Capture SPROCeT
Aggregation
Dissemination
BoPFACTS
DMS
Respondent Management
Imputation
SeasonalAdjustment
ExternalData CaptureApplication e.g. Blaise, Scanning
CARS
Sec
uri
tyS
ecu
rity
SA
SStatistical Metadata Systems:
Process Model
17
ITSIP
Statistical Metadata Systems: DMS Applications &
Metadata
Register Management - Create Register
- Define Register Variables
- Set-up Register Coding
Sample Selection - Set-up Sample Selection Criteria
- Define Stratification Groups
Data Capture - Create Data Capture Form
- Define Variable Characteristics
- Set-up Coding Rules
- Set-up Import Details
- Set-up Edit Rules and Validations
- Version control of data
18
ITSIP
Statistical Metadata Systems: DMS Applications & Metadata
Imputation - Set-up Imputation Groups
- Set-up Imputation Rules
Aggregation - Define Groups, Data Columns, Tables
- Create Weights and Weight Tables
- Macro edits and Confidentiality Rules
Dissemination - Create disseminate tables
- Define Additional Data Column attributes
Seasonal Adjustment - Set-up Seasonal Adjustment Rules
Survey Management - Set-up Post Out details
19
ITSIP
Statistical Metadata Systems: Existing Systems
CBR Central repository for all enterprises engaged economic activity
CARS Database containing all classifications and concordances
SPROCET Re-usable survey processing template used by the Industrial surveys in the CSO
BoPFACTS Data processing and survey management system used by the Balance of Payments section
SAS SAS V6.12 and PC SAS V8.02
External Data Capture Applications - Blaise, Scanning
20
ITSIP
Statistical Metadata Systems: Mapping the DMS to the CMF Life Cycle
Register Management Survey Preparation (2)
Sample Selection Survey Plan & Design (1)
Survey Management Survey Preparation (2)
Data Capture Data Collection (3)
Input Processing (4)
Derivation (5)
Imputation Estimation (5)
Aggregation Aggregation (5)
Dissemination Dissemination (7)
Respondant Management Post Survey Evaluation (8)
The DMS is a processing and not an analysis tool, therefore CMF LifeCycle Model “(6) Analysis” cannot be linked to the DMS.
21
ITSIP
Statistical Metadata in the Statistical Cycle: Input Metadata Examples
Group Description Examples Source
Register NameLists the Register Names stored in the DMS
R_RSI: The register for the Retail Sales survey. DMS Table: Register
Survey Name Lists the survey in the DMS RSI: The Retail Sales Survey DMS Table: Survey
DMSCommonCodes
Holds a list of metadata which may be reused across surveys
Data Type: String, Int, Boolean, Varchar, Numeric.
DMS Table: DMSCommonCodes
ClassificationsClassifications used to code variables in RM. NACE Rev 1.1 CARS
Survey Periodicity Monthly 2007M06
DMS Table: SurveyInstance
Sample Method Census DMS Table: Sample
Unit of Measure Euro
DMS Table:DMSCommonCodes
PostOut Item Type Questionnaire or Reminder
DMS Table: DMSCommonCodes
22
ITSIP
Statistical Metadata in the Statistical Cycle: Output Metadata Example
Table Name Creation type
Column Name DataType Relationship Description
M_RegisterVariable Static This table contains all the Register Variables information for a Register.VariableID_PK int Primary Key Variable IDRegisterID_FK int Register ID of the Register in which the Variable is presentVariableNameText varchar(30) Variable Name LabelText varchar(40) Variable Label
FormatID int
Format type ID
0 - Default1 - Herd Number2 - PPS Number3 - MVSerNo
DataTypeID smallint
Data type ID
1 - Numeric2 - String3 - Date4 - Boolean
ApplicableInd bit Flag to indicate if the variable is applicable in the Register.SearchVarInd bit Flag to indicate if the variable is Search applicable in the Register.CommonInd bit Flag to indicate if the Variable is a generic Variable.MandatoryInd bit Flag to indicate if the Variable is Mandatory.UpdateUser varchar(20) The login name of the user who last updated the row.UpdateDate datetime This is the date and time at when the user last updated the row.
CodedInd bitFlag to indicate if the Variable is coded.
HoldsCodedValueInd bit
Flag to indicate if the Variable holds codes directly.
0 - Does not Holds the coded Values1 - Holds the coded Values
Length int Length of the VariablePrec int Precision of the VariableCapsOnInd bit Flag to indicate if the CAPS ON property is turned on for the Variable.
23
ITSIP
Statistical Metadata in the Statistical Cycle: Output Metadata Example
24
ITSIP
Systems and Design Issues: Technical Starting Point
Sybase
In-house knowledge in Sybase (ASE) Technologies
SAS access for complex analysis
(SAS did not bid for tender)
Link to Classifications and Related Standards (CARS) system
All disseminated data groups must link to a CARS classification
Windows platform
(Not possible due to performance issues identified with Sybase transactions, hence move to Solaris)
25
ITSIP
Systems and Design Issues: Technical Overview
AS
E
(fai
lov
er)
AS
E
IQ
(fai
lov
er)
IQ
Win: WebLogic Cluster
PC / ClientIE6
JRE1.4.2_05
SAS
Filestore
CARS
SSA Names3CBR
Unix: Sybase
T3 (RMI) JDBC
26
ITSIP
Systems and Design Issues: Database Layer
The CSO has now established in-house skills in both Sybase ASE & IQ Technologies
High Level Technical Architecture:
Data Capture, Imputation : Sybase ASE
Aggregation, Dissemination : Sybase IQ
Two types of table:
Core DMS Table (Survey Metadata)
Survey Specific Table (Data)
All complex numerical processing is performed within the database layer through the use of stored procedures
User of Veritas Clustering software on database layer to facilitate database failover
27
ITSIP
Systems and Design Issues: Weblogic / J2EE MidTier
J2EE Application Server (Weblogic)
Stateless Session Beans
JMS Queues
JDBC Connection to ASE / IQ Databases
Application Security
Users validated against corporate Active Directory Service
Within DMS Database validated users will have assigned roles / privileges
28
ITSIP
Systems and Design Issues: Client Layer
The DMS is a complex GUI interface
‘Fat Client’ using Java Swing technology
The client is deployed using Java Web Start Technology
Centrally managed releases
Quick deployment to client desktop
Client uses Java RMI to communicate with the J2EE server
(Currently using WebLogic T3 protocol)
29
ITSIP
Systems and Design Issues: Other Components
Filestore
Shared network drive onto which data to be Imported / Exported to the DMS resides
SAS
Required for Seasonal Adjustment
Required for Import / Export of SAS Datasets to/from DMS
CARS (Classifications) [Statistics New Zealand]
All data to be dissemintated must use a CARS classification
CBR (Central Business Register) [Statistics New Zealand]
Hierarchical database
SSA Names3
Duplicate matching / searching of registers
30
ITSIP
Organisational & Cultural Issues: Roles within the CSO
DMS Administrator (I.T.)
Highest level of access to the DMS
Supports the DMS
Manages the day to day interaction with the DMS
Survey Administrator (Statistician)
Defines the survey
Runs the survey
Assigns staff survey access and privileges
DMS Administrator
Survey AdministratorSurvey Administrator
User
User
User
User
User
User
User
User
31
ITSIP
Organisational & Cultural Issues: DMS Maintenance
In the future the DMS will be supported by:
Cognizant Technology Solutions Ltd
1 year maintenance contract
provision for a 5 year support contract
CSO Java Development Team
CSO Weblogic Team
32
ITSIPThank You for Your Attention