Methods and Techniques for Integration of Small Datasets September 13-14, 2005 St. Louis, Missouri...

Methods and Techniques for Integration of Small Datasets

September 13-14, 2005St. Louis, Missouri

Sponsored by the U.S. Department of Housing and Urban Development

Deb Little, MISI

Loren Hoffmann, Wisconsin Dept. of Commerce

September 13-14, 2005 St. Louis, MissouriSponsored by the U.S. Department of Housing and Urban Development

2

The Landscape

• The Ideal• All agencies in a continuum use the same HMIS

• The Reality• Some agencies have invested substantial resources in

their own system, which finally works “perfectly”

• Some agencies have multiple funders, some of which may require the use of another system or the reporting of data not collected by the HMIS of the CoC


3

Issues to Consider

What are the Benefits?• For the Community• For the Participating Agencies• For the Client

What is the Goal of Integration?• Unduplicated Count• Service Coordination/Gaps Analysis• Comprehensive Client Information


4

Where is the Problem?

• Why can’t I “just merge” the data?


5

Data Issues to Consider

• What data is included?• Where is it to be stored?• Who has access?• Who controls the data?• Who owns the data?• Who maintains or keeps it in synch?


6

Data Issues

• What Data? Each field has its issues.• Type - e.g., Name

• How many fields? Fname, Lname, Mname, suffix• What about punctuation such as hyphens

• How is the data stored• DOB vs. Age of client

• Picklist variations• Education - 11 years of education vs. “some High School”

• How is it stored/coded• Male vs. M vs. 1, etc


7

Where Is It Stored?

• Options for storing shared data:• On the same server as the HMIS• A neutral machine


8

Who Has Access?

• Who establishes the rules about the data?• The integrated data has more value than the

parts separately• Who determines what is published?• Who determines what is “cleaned” data vs. “raw”

data?• Who is able to release the data to another party?


9

Maintenance

• As the data needs/rules change over time, what effort is to be made to update historic data?

• Are all data elements collected to be maintained?


10

Options

• Centralized HMIS - used by everyone

• Dual Entry - agency enters data into multiple systems• Those who have achieved either of these don’t need this

session

• Integration• Multiple systems maintained• Data merged from multiple system• (this session)


11

Issues to Consider

Data elements necessary to achieve the goal?• Only the Universal elements?• Locally defined data elements?

Frequency of data integration?• Data Warehousing vs. Data Integration

What are the political considerations?• Identifying data vs. de-identified• Who can view data?• Who can release data?


12

Issues to Consider (Con't)

• Who will be involved in the planning and decision making process?

• Where will the data be merged and stored?


13

Resources to Consider

• Technical expertise• Get them involved early

• Adequate budget• Initial setup and on-going

• Available time• Auditing tools

• How will we know how good the data are?


14

How to Make It Work

Identify the appropriate model• One way integration – independent agencies

send data file for inclusion in central database• Two way integration – agencies send data to

and receive data from central system• Data Warehousing

*pros and cons to each method


15

Merge - Integration Options

• Real time• Data transferred between two or more

systems; each system is refreshed with data from the other

• Pro: Real time transfer between systems.• Con: Very expensive; probably cost prohibitive

for most CoCs.


16

Periodic Integration

• Periodic transfer of data between two or more systems

• Pro: Periodic updates.• Con: Time consuming; not real time; could be

cost prohibitive for many CoCs.


17

Periodic Merging for Analysis Only

• Aggregation of data from various system for analytical purposes only.

•Pro: Least expensive option; can generate unduplicated count.

•Con: No real time access to data; data is time limited.


18

Rules Needed

• Define needed “integration rules”• File format required (CSV, XML, other)• Data auditing procedures• How to handle “bad data”


19

More Rules

• Value Mapping Determining when values in the database

mean the same thing. Database A: Male=1, Female=2,

Transgender=3. Database B: Female=1, Male =2. Database C: Female = F, Male = M


20

And More Rules

• Record matching• Determine whether a record in two systems are the

same• Do I add a second record representing the same client?• What is the “rule” for determining that the client in two

different records is the same client?• Is this service already there?

• Record matching approach will vary depending on reason for merging and what is merged.


21

Overall Process - Summary

• Data Conversion: Create data standard: Per implementation. Data mapping: Per system.

• Data Merging: Export. Validate. Transport. Validate Again Import.

• Analysis: As needed for reporting.


22

Analysis

• Integration is not complete when all the data is merged into a single database. Additional tasks: “Unduplication.”

Use data from multiple agencies to fill in blanks Handle conflicting information

Extract data. Distribute data.


23

Baltimore Integration

Currently: • Large metropolitan implementation• 90 agencies, 120+ programs using designated HMIS• 7 agencies utilize the same third party software• 1 agency utilizes product developed in-house• 1 agency in development of integration “bridge”

Issues:• Avoid duplicate data entry• Establish single reporting tool for “Homeless Stats”


24

Wisconsin Integration

• Currently:• Statewide implementation• 130 agencies• 2-3 agencies need to participate due to funding

requirements

• Issues:• Don’t want to do double entry• Client confidentiality


25

Baltimore (Con’t)

• Results:• Data integration file defined; includes universal,

program and continuum defined data elements • One way integration model – agencies not using

designated HMIS send data in regular intervals to central database. No data returned

• Data passing quality controls is processed against live database

• Auditing tools utilized regularly to detect and eliminate duplicate files


26

What Is Collected?

• Only HUD required data is currently collected.• Agency provides one client record per client• Agency provides one service record per service

event (3 services = 3 records) - link on UniqueID field

• Unduplication based on HMIS algorithm


27

Wisconsin Integration

Results:• Data will be collected periodically for reporting

purposes only• Data will be housed in a separate database,

accessibly only to state system staff• System staff will generate reports, distributing

only de-identified information


28

Summary

• Integration:• Is complicated• Will cost $$• Is an ongoing cost (time and money)• Can be done• If managed properly, can be very cost effective

and beneficial


29

The Moral of the Story...

Anything is Easy,

Once You Know

How!

Methods and Techniques for Integration of Small Datasets September 13-14, 2005 St. Louis, Missouri...

Documents

Transcript of Methods and Techniques for Integration of Small Datasets September 13-14, 2005 St. Louis, Missouri...