CUPUM 2007 – Foz do Iguaçu, Brazil NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT:
description
Transcript of CUPUM 2007 – Foz do Iguaçu, Brazil NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT:
CUPUM 2007 – Foz do Iguaçu, Brazil
NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT:
Intelligent Middleware for Understanding Neighborhood Markets
Prof. Joseph Ferreira, Jr. and Dr. Raj Singh
MIT, Urban Information Systems
Department of Urban Studies & Planning
July 12, 2007
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 2
NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT
Applaud recent interest in communtiy data centers, mashups, and PPGIS But, current practice cannot meet high expectations
Problems Data maintenance and currency Limited and inflexible modeling & analysis
Solution – more IT ??!! Modular service chaining - (geo)web services, SOA Intelligent middleware for modeling and analysis Lessons (even without adding new IT)
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 3
Trends in IT for Community Development
Neighborhood Information Systems Local Data Centers serving NGOs & sharing data PPGIS (public participation GIS - beyond census tables) Public/private/non-profit partnerships Examples: NNIP website: http://www.urban.org/nnip
Mashups and Web 2.0 Internet GIS: ArcIMS, MapServer, GeoServer… Map ‘mashups’ on Google Earth/Maps, Virtual Earth,
Yahoo maps… User-generated content: Flickr photos, social networking…
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 4
GIS Office for State of MassachusettsMassGIS: http://www.mass.gov/mgis/
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 5
MassGIS:
Example of very good online
mapping and
download site for
typical US State
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 6
Google Earth Mashup: Nanaimo, Canada
City website: http://earth.nanaimo.ca/data.html NanaimoMaster and NanaimoVirtuallyDowntown using Google Earth+MapGuide
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 7
National Neighborhood Indicators Partnership: http://www.urban.org/nnip
Non-governmental organizations focused on needs of particular
communities
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 8
The Providence
Plan
Providence Rhode Island
USA
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 9
Neighborhood Knowledge Los Angeleshttp://nkla.sppsr.ucla.edu
Land Opportunities Tracking Systemhttp://lots.ucla.edu
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 10
Limitations of Neighborhood Information Systems
Keeping data up to date No problem for decennial census data Big problem for land use/ownership (cadastre)
and permitting More difficult as more parties try to share data
Interpreting & Analyzing ‘official’ data Must clean/re-interpret data for local use Hard to include ‘local knowledge’ Customized analyses are useful but non-trivial
Intelligent Middleware for Understanding Neighborhood Markets
Urban Markets Initiative Pilot Project
A collaborative effort amongMassachusetts Institute of Technology (MIT), Boston Metropolitan Area Planning Council (MAPC)Boston’s Department of Neighborhood DevelopmentThe Boston Foundation (TBF)Open Geospatial Consortium (OGC)
with support from the Urban Markets Initiative of The Brookings Institution and participation of local CDCs.
People Working on the ‘Intelligent Middleware’ Project
Principal Investigator: Joe Ferreira, Professor, MIT
Principal Collaborators: Holly St. Clair, Director, MAPC Data Center Larry Braman, Manager, Mapping and Data Services, Boston DND Charlotte Kahn, Director, Boston Community Indicators, TBF
MIT Research Team: Raj Singh, system architect (now at OGC) Jiawen Yang, database services (now Assist. Prof. Georgia Tech) Myounggu Kang, mapping services (now Assist. Prof. Seoul U.) Mi Diao, Xiongjiu Liao, Weifeng Li (PhD candidates)
Sponsored by: The Brookings Institution, Urban Markets Initiative
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 13
‘Intelligent Middleware’ ProjectGoals and Concepts
Facilitate data sharing within metropolitan areas Without everyone needing their own ‘data center’
Because everyone needs to change their copy of the data
Allow reinterpretation of official data Based on local knowledge
Without explicitly changing the original data
Codify local knowledge as rules to interpret official data on the fly
Deliver results into familiar desktop environment Tables into spreadsheet or browser; maps as mashups
Examples: Identify key landowners, targeted land uses, …
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 14
‘Smart’ Middleware AlternativeIntelligent Middleware for Understanding Neighborhood Markets
Environmental portal
Housing portal
Read-only, ‘Official’ data repositories
Intelligent data intermediaries
Normalize, trim, merge, transform, overlay, cookie-cut, extrapolate, qualify, …
Processing
Service
Service
Police Dept. Assessing Dept.
HealthDept.
HousingDept.
Data Sets Data Sets Data Sets Data Sets
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 15
One Example:Land use & Ownership Patterns
Want: REPORT & MAP of largest landowners: Top-10 Owners Keep map/report accurate and up to date Allow flexibility in definition of ‘neighborhood’ Allow flexibility in which land uses to consider (residential,
open space…) Problem: Parcel map & Assessing database not enough
Spelling problems: many different spellings of city agencies and other large landowners
Updating: old snapshot of parcels is out of date
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 16
Desired Output:
Table, Chart, and Map in Familiar Desktop Environment
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 17
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 18
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 19
Middleware Example: Prototype NGO Webpage (for DSNI)
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 20
Results BEFORE Standardizing Owner Names
For Dudley Street Neighborhood Initiative (DSNI) Neighborhood:
OWNER NAME (Boston parcels, FY2005) parcels acres
CITY OF BOSTON 182 37.85
CITY OF BOSTON BY FCL 123 9.76
DUDLEY NEIGHBORS INC 121 12.14
COTTAGE BROOK HOUSING LP 48 2.38
KEPNES EDWARD N TS 44 4.02
ALEXANDER MAGNOLIA LP 39 3.21
BOSTON HOUSING AUTHORITY 36 2.27
CITY OF BOSTON FCL 27 1.82
DUDLEY_NEIGHBORS_INC 21 1.68
RAIMONDI ROBERT A SR TS 19 0.51
Total Parcels/Acres owned by largest owners 660 75.6
…but Top-10 Landowner List has Problems:
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 21
For Dudley Street Neighborhood Initiative (DSNI) Neighborhood
OWNER NAME (Boston parcels, FY2005) parcels acres
CITY OF BOSTON 339 50.12
DUDLEY NEIGHBORS INC 145 14.06
COTTAGE BROOK HOUSING LP 48 2.38
KEPNES EDWARD N TS 44 4.02
ALEXANDER MAGNOLIA LP 39 3.21
BOSTON HOUSING AUTHORITY 36 2.27
RAIMONDI ROBERT A SR TS 19 0.51
ORCHARD PK REVITALIZATION 18 0.81
HOWARD DACIA LLC 16 1.43
ORCHARD PARK REVITALIZATION 16 2.92
Total Parcels/Acres owned by largest owners 720 81.7
Results AFTER Standardizing Some Owner Names
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 22
Solving the Spelling Problem
Problem: many different spellings of city agencies and large landowners in parcel data
Solution choices: Top-down: standardize spellings in official
datasets Bottom-up: fix spellings in copy of dataset Middle-out: accumulate spelling changes
separately and apply on-the-fly as needed…build ‘middleware’ for middle-out strategy
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 23
Example of RULE to Fix Spelling Error•Standardize OWNER names
•Create ‘update’ RULE to fix spellings•Apply on-the-fly to Boston’s parcel data (BASETABLE) to produce temporary tables (FACADES) with spelling corrections•Use temporary table for REPORTS and THEMATIC MAPS
•Example of one RULE for City of BostonUPDATE fas_parcel_bos05 SET OWNER_FY05='CITY OF BOSTON' where OWNER_FY05 like '%CITY%' and ((OWNER_FY05 like '%BOSTON%' or OWNER_FY05 like %BSOTON%') and OWNER_FY05 not like '%CITYWIDE%' and OWNER_FY05 not like '%SUITES%' and OWNER_FY05 not like '%LIGHT%‘ and OWNER_FY05 NOT LIKE '%PROPERTIES%' and OWNER_FY05 NOT LIKE '%LLC%' and OWNER_FY05 NOT LIKE '%L L C%' and OWNER_FY05 NOT LIKE '%CREDIT UNION%)';
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 24
‘Intelligent Middleware’ ProjectGoals and Concepts
Facilitate data sharing within metropolitan areas
Allow reinterpretation of official data based on local knowledge Deliver results into familiar desktop environment
HOW?
Modular design: interoperable parts, distributed access, secure access control Data intermediary manages ‘virtual’ data center middleware
Middleware uses web services and open standards
Middleware is hidden from end-user applications
User/group logins and SSL provide data sharing control
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 25
‘Smart’ Middleware AlternativeIntelligent Middleware for Understanding Neighborhood Markets
Environmental portal
Housing portal
Read-only, ‘Official’ data repositories
Intelligent data intermediaries
Normalize, trim, merge, transform, overlay, cookie-cut, extrapolate, qualify, …
Processing
Service
Service
Police Dept. Assessing Dept.
HealthDept.
HousingDept.
Data Sets Data Sets Data Sets Data Sets
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 26
Middleware Components
Concept Example DeliveryThematic MAP Map of land owned by largest
landownersGoogle Maps pinmap showing parcels owned
REPORT Table listing largest landowners in neighborhood
Excel spreadsheet showing acreage of Top-10 owners
FACADE (Virtual) parcel records with
corrected spellings Manage façade rules and sharing via browser
RULES SQL query to standardize spelling of owner names
Apply SQL rules that fix parcel data on-the-fly
Study Area Service area of Dudley Street Neighborhood Initiative
Reports are for user-defined study areas
BASETABLE Parcel data from Boston’s Assessing Office
Use read-only copy of city data
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 27
Tables are not static but result from ‘web query’:http://uis.mit.edu/umi/service/report.php?Request=QueryReport&name=top_own_parcels_a_jf1&tablename=own05_mixed_after&studyarea=dsni&xsl=http://uis.mit.edu/umi/xl/report.xsl
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 28
Same URL in Browser:http://uis.mit.edu/umi/service/report.php?Request=QueryReport&name=top_own_parcels_a_jf1&tablename=own05_mixed_after&studyarea=dsni&xsl=http://uis.mit.edu/umi/xl/report.xsl
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 29
Thematic Maps are also web services:http://uis.mit.edu/umi/mapservice/umimap.php?request=GetMap&layers=top_own_parcels_map1&WIDTH=500&HEIGHT=500&SRS=EPSG:4326&TRANSPARENT=true
Map requests conform to Web Mapping Standard (WMS) of Open Geospatial Consortium (OGC)
•Many applications now support WMS•E.g., Excel, Google, ArcGIS, …
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 30
A WMS call to the middleware services yields a thematic map top-10 landowners for all of Boston
The URL:
http://uis.mit.edu/umi/mapservice/umimap.php?request=GetMap&layers=testjf0004
The same URL works from inside of ArcMap (see
next slide)
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 31
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 32
Middleware Components
Prototype on RHEL4 Linux PC running open source applications
Database server: Postgres with Postgis spatial extensions
Mapping server: MapServer (or GeoServer) Web server: Apache2 (with Tomcat) Web services programmed in PHP Additional middleware management tools in
AJAX (asymmetric javascript and XML)
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 33
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 34
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 35
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 36
Middleware Use: More than Spelling Fixes…
Understand Land Use and Ownership Patterns: Identify “top-10” landowners in neighborhood Trace effective ownership: who controls ABC Corp. Track (un)desired land use, vacant land suitable for
reuse… Study trends: gentrification impacts of condo
conversion, predatory lending practices, …
Measure urban performance: Model affordability, accessibility, … Estimate retail ‘gaps,’ development potential, …
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 37
Conclusions: (1) Key Concepts
Sharing data within a metropolitan area Avoid ‘data center’ limitations as small geographies and
large/changing datasets explode data needs Provide mechanism for accumulating, utilizing, and
sharing local knowledge about land use, ownership, and market potential (independently of official datasets)
Use web services approach, with interoperability standards, to build ‘lego blocks’ for many management and end-user applications.
Examine the skill sets and institutional relationships that can take advantage of ‘service oriented architectures’.
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 38
Conclusions:(2) Community Development Implications
As volume and detail of ‘relevant’ data grows, traditional sharing of data and tools becomes more problematic Corrolary: Google Earth is great but can’t meet expectations
Web services and ‘middleware’ offer promising avenues for deeper sharing, but Require different mix of skills and institutional arrangements Won’t be offered soon by vendors (unless we push!) Suggests new direction and short-term strategies
More complex urban models and performance indicators have limited use unless built as plug-and-play modules that provide tunable web services.
CUPUM07: Intelligent Middleware - Joe Ferreira - July 12, 2007 39
Thanks…
Joe Ferreira, [email protected]
Raj Singh, [email protected]
‘Middleware’ project website: http://uis.mit.edu