Use of Open Data in Hong Kong
-
Upload
sammy-fung -
Category
Technology
-
view
1.401 -
download
4
description
Transcript of Use of Open Data in Hong Kong
![Page 1: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/1.jpg)
Use of Open Data in Hong Kong
Sammy Fung
sammy.hk
Incu-Lab ICE in StartMeUpHK - Open Data Initiative Gathering
2013/12/04
http://slidesha.re/1cleS2y
![Page 2: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/2.jpg)
We want a better life with public data.
![Page 3: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/3.jpg)
We want a easier way to access the public data.
![Page 4: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/4.jpg)
Agenda
● What is Open Data ?● Use of Open Source Software in web crawling.● Starting new Open Source project hk0weather
to create Open Weather Data.
![Page 5: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/5.jpg)
Sammy Fung
● Software Developer – to use and develop open source sofware.– Perl → PHP → Python.– interests on Data Mining / Web Crawling.– own a startup of web and mobile technology.
![Page 6: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/6.jpg)
Sammy Fung
● 15+ years in Open Source Communities.– Founding Chairman, Hong Kong Linux User Group.– Founding Chairman, Open Source Hong Kong.– Member, GNOME Asia committee.– Mozilla Representative– Member, program committee at COSCUP
● Conference for Open Source Coders, Users and Developers.● Largest open source conference in Taiwan.
![Page 7: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/7.jpg)
What is Open Data ?
![Page 8: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/8.jpg)
Open Data
Three Laws of Open Government Data by David Eaves.
1.If it can't be spidered or indexed, it doesn't exist.
2.If it isn't available in open and machine readable format, it can't engage.
3.If a legal framework doesn't allow it to be repurposed, it doesn't empower.
http://eaves.ca/2009/09/30/three-law-of-open-government-data/
![Page 9: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/9.jpg)
Open Data
● Tim Berners-Lee, the inventor of the Web.– 5stardata.info– 5 star deployment scheme of Open Data.
![Page 10: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/10.jpg)
* One Star - Open Data
1.make your stuff available on the Web (whatever format) under an open license.
2.make it available as structured data (e.g., Excel instead of image scan of a table)
3.use non-proprietary formats (e.g., CSV instead of Excel)
4.use URIs to denote things, so that people can point at your stuff.
5.link your data to other data to provide context.
5stardata.info by Tim Berners-Lee, the inventor of the Web.
![Page 11: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/11.jpg)
** Two Star - Open Data
1.make your stuff available on the Web (whatever format) under an open license.
2.make it available as structured data (e.g., Excel instead of image scan of a table)
3.use non-proprietary formats (e.g., CSV instead of Excel)
4.use URIs to denote things, so that people can point at your stuff.
5.link your data to other data to provide context.
5stardata.info by Tim Berners-Lee, the inventor of the Web.
![Page 12: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/12.jpg)
*** Three Star - Open Data
1.make your stuff available on the Web (whatever format) under an open license.
2.make it available as structured data (e.g., Excel instead of image scan of a table)
3.use non-proprietary formats (e.g., CSV instead of Excel)
4.use URIs to denote things, so that people can point at your stuff.
5.link your data to other data to provide context.
5stardata.info by Tim Berners-Lee, the inventor of the Web.
![Page 13: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/13.jpg)
**** Four Star - Open Data
1.make your stuff available on the Web (whatever format) under an open license.
2.make it available as structured data (e.g., Excel instead of image scan of a table)
3.use non-proprietary formats (e.g., CSV instead of Excel)
4.use URIs to denote things, so that people can point at your stuff.
5.link your data to other data to provide context.
5stardata.info by Tim Berners-Lee, the inventor of the Web.
![Page 14: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/14.jpg)
***** Five Star - Open Data
1.make your stuff available on the Web (whatever format) under an open license.
2.make it available as structured data (e.g., Excel instead of image scan of a table)
3.use non-proprietary formats (e.g., CSV instead of Excel)
4.use URIs to denote things, so that people can point at your stuff.
5.link your data to other data to provide context.
5stardata.info by Tim Berners-Lee, the inventor of the Web.
![Page 15: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/15.jpg)
Open Data in Hong Kong
![Page 16: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/16.jpg)
Open Data in Hong Kong
● Data.One– http://www.gov.hk/en/theme/psi– released on 2011/3/31.– First App Competition on Data.One
● Call for Submission now till 2014/02/28.
![Page 17: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/17.jpg)
Weather Information in Hong Kong
● Hong Kong Observatory– Hourly Hong Kong Weather Report– Regional Weather in Hong Kong (10 min updates)– Weather Forecast and Weekly Weather Forecast– Typhoon Report and Forecast
![Page 18: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/18.jpg)
Hong Kong Observatory RSS
![Page 19: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/19.jpg)
Hong Kong Observatory RSS
![Page 20: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/20.jpg)
Weather at Data.One
● I posted a blog 'Progress of Open Government Data in Hong Kong' on 2013/01/17.
● Weather at Data.One provides 7 dataset URLs, returns RSS (XML) format (Eng/TChi/SChi)– One word: Useless.– Data.One dataset (RSS) is completely different
with HKO own paid service (XML).
![Page 21: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/21.jpg)
Weather at Data.One
● Example - Current local weather report: ● Plain text report in RSS.● Difference to quote report content:
– Website: a pair of HTML tags, eg. <PRE>....</PRE>.– Data.One: a pair of RSS description tags,
<description>....</description>.
● Other weather data is missing, eg. Regional temperture updates per each 12 mins.
![Page 22: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/22.jpg)
Weather at Data.One
● Weather at Data.One is 'report' but not 'data'.● Weather RSS is already released by HKO
before launch of Data.One.● Technically, json/xml format is better
readable by computer programs.
![Page 23: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/23.jpg)
Data.One
● In November 2013, 43 datasets are available.– JSON/XML = 18– RSS = 10– XLS = 6– CSV = 4– JPG/PNG = 3– HTML/MDB = 2
![Page 24: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/24.jpg)
Data.One
● JSON/XML (18 datasets)– Air Pollution.
● Past 24-hour Air Pollution Index from stations.
– Approved Charitable Fund-raising Activities– Restaurant and Food Licences.– Details of facility locations.– Reward Notices from Police Force.– Marine Traffic (Arrival/Departure).– Traffic Speed and special news.– EventHK information.
![Page 25: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/25.jpg)
Data.One
● RSS (10 datasets)– Weather Information (7 datasets) – Beach Water Quality (1 datasets)– Current Air Pollution Index range and forecase (2
datasets)
![Page 26: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/26.jpg)
Data.One
● JPG/PNG (3 datasets)– Exhibition gallery of government building
projects.– Speed map panels.– Traffic snapshot images.
![Page 27: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/27.jpg)
Data.One
● CSV– Past Record of Air Pollution Index– Locations of Public Facility and GovWifi– Marine Shipping directory of HK
● HTML– HTML version of Marine Traffic.
● XLS, MDB– 2011 Population Census.– Property Market Statistics.– Monthly Digested Stats and Registers of Auth Persons from Building Dept.– Routes and fares of public transport.
![Page 28: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/28.jpg)
Data.One
● Many departments does not release their useful data, and release current information available on their website.– Few of them keep available open data in their own.
● Most of them does not understand what is 'real' open data.– Data insteads of Information.– Open data format insteads of proprietary data format.– Useful of data.
● Some departments should manage their open data in better data structure.
![Page 29: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/29.jpg)
Legco Meeting Minutes and Voting Results
![Page 30: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/30.jpg)
Legco Meeting Minutes and Voting Results
![Page 31: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/31.jpg)
Legco Meeting Minutes and Voting Results
● In October 2013, LegCo start to publish voting results of House Committe in XML.
● It is not a part of Data.One project.● My open source software on LegCo vote
result XML:– http://github.com/sammyfung/legcovotes
![Page 32: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/32.jpg)
Digital21 Strategy Public Consultation Document
(G) Public Sector Information (PSI) as Default
"34. Through different channels (like press releases, publications, websites, etc.), the Government releases a lot of information in different areas. However, most of such information can only be read but cannot be used. In view of the immense benefits of widening access to PSI for free and easy re-use, we propose to make all Government information released for public consumption machine-readable by default. Where appropriate, datasets will be released with application programming interfaces (APIs), providing predefined functions to make their retrieval easier."
(G) 廣泛提供公共資料
"34. 政府透過不同途徑 (例如新聞稿、出版物、網站等 )發放大量不同範疇的資料。然而 ,這些資料大都只可供閱讀而不能使用。有見開放公共資料以供免費再用可帶來巨大效益 ,我們建議所有開放予公眾使用的政府資料都須以數碼格式編製。在適用情況下 ,資料發布時會同時推出應用程式界面,以便提供預設功能 , 讓公眾輕易地檢索資料。 "
![Page 33: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/33.jpg)
Digital21 Strategy Public Consultation Document
"33. PSI datasets can be used and meshed together to create innovative new applications, as demonstrated by the creative and useful products and services developed from PSI in Hong Kong and around the world. For example, using PSI datasets on traffic snapshot images, a number of mobile apps have been developed to provide real-time traffic situation for users to avoid traffic jams in planning their traffic routes. Experience from other developed economies shows that widening access to PSI datasets can open up lucrative business opportunities and bring social benefits. By tapping the creativity of the community and entrepreneurs, the use of PSI can lead to positive social outcomes. For instance, in some cities in the United States, application of PSI on hygiene inspections has led to a significant drop in food poisoning incidents."
![Page 34: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/34.jpg)
Digital21 Strategy Public Consultation Document
"33. 由本港及世界各地利用公共資料所開發的實用創意產品及服務所見 , 公共資料可個別及混合使用 , 以開發創新的應用程式。例如 , 現時已有多個利用交通情況快拍圖像的公共資料開發的流動應用程式 , 以提供實時交通情況資料 , 讓使用者計劃行車路線 , 從而避開交通擠塞情況。根據其他經濟體系的經驗 , 開放公共資料 , 供大眾廣為使用 , 可開拓有利可圖的商機 , 並為社會帶來禆益。我們可藉着開放公共資料 , 借助市民及企業家的創意來造福社會。舉例來說 , 在美國一些城市 , 有關衞生檢查的公共資料在開放使用後 ,食物中毒事故宗數大幅減少。 "
![Page 35: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/35.jpg)
Digital21 Strategy Public Consultation Document
"35. Apart from Government data, there are vast amounts of PSI handled, collected and disseminated by public organisations, which are equally useful for the development of innovative services and products. Therefore, we propose to encourage public organisations (e.g. public utilities and transport operators) to release data owned by them in machine-readable format."
"35. 除了政府資料外 , 本港亦備有大量經公共機構處理、收集及發放的公共資料 ,這些資料對開發創新服務及產品同樣有用。因此 , 我們建議鼓勵公共機構 ( 例如公用事業及運輸機構 ) 發放以數碼格式編製的資料。 "
![Page 36: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/36.jpg)
Open Data is important to citizens.
![Page 37: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/37.jpg)
User of Open Source Software in web
crawling
![Page 38: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/38.jpg)
Web Scraping
● a computer software technique of extracting
information from websites. (Wikipedia)● for business, hobbies, research purposes.
![Page 39: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/39.jpg)
Web Scraping
● Look for right URLs to scrap.● Look for right content from webpages.● Saving data into data store.● When to run the web scraping program ?
![Page 40: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/40.jpg)
Use of Open Source Software in Web Crawling
● Use Open Source Tools to collect useful and meaningful machine-readable data.
● Doesn't need to wait provider to release data in machine-readable format.
![Page 41: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/41.jpg)
Open Source Tools
● Python programming lanugage● with Regular Expression library● Scrapy web crawling framework
![Page 42: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/42.jpg)
Why python + scrapy ?
● python: my current favourite programming language for few years.
● scrapy: web crawling framework written in Python.
![Page 43: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/43.jpg)
What is Scrapy ?
● An open source web scraping framework for Python.
● Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
![Page 44: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/44.jpg)
Scrapy Features
● define data you want to scrapy● write spider to extract data● Built-in: selecting and extracting data from HTML
and XML● Built-in: JSON, CSV, XML output● Interactive shell console● Built-in: web service, telnet console, logging● Others
![Page 45: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/45.jpg)
Programme List of Paid TVs in 2004
![Page 46: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/46.jpg)
Programme List of Paid TVs in 2004
● I want to know live football match was showing on which channel.
● Paid TV web site = M$ + IIS + ASP + Flash● Slow....... Very Slow...... Extremely Slow!● Couldn't connect at any peak hours!● Wrote my first web crawler in PHP in 2004.
![Page 47: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/47.jpg)
Public Transportation in 2006-2010
● Kowloon Motor Bus (KMB)– No map view for a bus route
● Public Transportation Enquiry System (PTES)– Exteremly Poor, Ugly (or much worse) map UI on
PTES.
![Page 48: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/48.jpg)
HK Observatory and Joint TyphoonWarning Center
● Any typhoon is coming to Hong Kong ? And When will it come ?
● No easy data exchange format.● No RSS nor ATOM.● We aren't check websites everyday.
![Page 49: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/49.jpg)
My Products
● WeatherHK ← ← ← ● TCTrack
![Page 50: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/50.jpg)
WeatherHK● http://twitter.com/weatherhk● hourly current weather report● weather forecast report● tropical signal warning
![Page 51: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/51.jpg)
WeatherHK
● Backend: Python + Scrapy + Database + Twitter + NNTP......
● Frontend: Twitter + Newsgroup
![Page 52: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/52.jpg)
WeatherHK
● http://twitter.com/weatherhk● Interview by MetroPop in 2009.
![Page 53: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/53.jpg)
My Products
● WeatherHK● TCTrack ← ← ←
![Page 54: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/54.jpg)
TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php● Plot TC current and forecast tracks over
Google Map.● Source:
– JTWC– HKO
![Page 55: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/55.jpg)
TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php● Probably first tctrack map in HK using
GoogleMap● Use of GMap: TCTrack -> Weather
Underground Hong Kong -> HKO
![Page 56: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/56.jpg)
TCTrack
● http://twitter.com/tctrack● Tweet JTWC updates for Northwest Pacific.
![Page 57: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/57.jpg)
Releases information to citizens in a better presentation.
![Page 58: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/58.jpg)
Starting new Open Source project
hk0weather to create Open Weather Data.
![Page 59: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/59.jpg)
Starting new Open Source projects to create Open Data
● Develop a open source project.● Release data in standard machine-readable
data format.
![Page 60: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/60.jpg)
hk0weather
● https://github.com/sammyfung/hk0weather● Open Source Hong Kong Weather Project.● convert to JSON data from HKO webpages.● python + scrapy● 1st version: from current weather report,
extracting temperture and humidity from 20+ weather stations, export in json format.
![Page 61: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/61.jpg)
hk0weather
● https://github.com/sammyfung/hk0weather● $ virtualenv hk0weatherenv● $ source hk0weatherenv/bin/activate● $ pip install scrapy● $ git clone
https://github.com/sammyfung/hk0weather.git● $ cd hk0weather● $ scrapy crawl currwx -t json -o testresult
![Page 62: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/62.jpg)
hk0weather
● Python– import re
● Scrapy– web crawling framework written in Python.– HtmlXPathSelector.– built-in JSON, CSV, XML output.
![Page 63: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/63.jpg)
hk0weather[{"humidity": 80, "station": "hko", "temperture": 17, "time": 1360785720},{"station": "kingspark", "temperture": 16, "time": 1360785720},{"station": "wongchukhang", "temperture": 17, "time": 1360785720},{"station": "takwuling", "temperture": 16, "time": 1360785720},{"station": "laufaushan", "temperture": 15, "time": 1360785720},{"station": "taipo", "temperture": 16, "time": 1360785720},{"station": "shatin", "temperture": 17, "time": 1360785720},{"station": "tuenmun", "temperture": 17, "time": 1360785720},{"station": "tseungkwano", "temperture": 16, "time": 1360785720},{"station": "saikung", "temperture": 16, "time": 1360785720},{"station": "cheungchau", "temperture": 17, "time": 1360785720},{"station": "cheungchau", "temperture": 17, "time": 1360785720},
{"station": "tsingyi", "temperture": 17, "time": 1360785720},
{"station": "shekkong", "temperture": 15, "time": 1360785720},
{"station": "tsuenwanhokoon", "temperture": 15, "time": 1360785720},
{"station": "tsuenwanshingmunvalley", "temperture": 17, "time": 1360785720},
{"station": "hongkongpark", "temperture": 17, "time": 1360785720},
{"station": "shaukeiwan", "temperture": 16, "time": 1360785720},
{"station": "kowlooncity", "temperture": 16, "time": 1360785720},
{"station": "happyvalley", "temperture": 18, "time": 1360785720},
{"station": "wongtaisin", "temperture": 17, "time": 1360785720},
{"station": "stanley", "temperture": 16, "time": 1360785720},
{"station": "kwuntong", "temperture": 15, "time": 1360785720},
{"station": "shamshuipo", "temperture": 17, "time": 1360785720}]
![Page 64: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/64.jpg)
Items.py
class Hk0WeatherItem(Item):
time = Field()
station = Field()
temperture = Field()
humidity = Field()
![Page 65: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/65.jpg)
Currwx.py
start_urls = (
'http://www.weather.gov.hk/wxinfo/currwx/currentc.htm',
)
![Page 66: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/66.jpg)
Currwx.py
def parse(self, response):
laststation = ''
temperture = int()
stations = []
hxs = HtmlXPathSelector(response)
report = hxs.select('//div[@id="ming"]')
![Page 67: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/67.jpg)
libhk0
class hk0:
stations = [
(u' 天 文 台 ', 'hko'),
(u' 京 士 柏 ', 'kingspark'),
(u' 黃 竹 坑 ', 'wongchukhang'),
(u' 打 鼓 嶺 ', 'takwuling'),
(u' 流 浮 山 ', 'laufaushan'),
![Page 68: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/68.jpg)
libhk0
class hk0:
def gettime(self, report):
…
def hk0current(self, report):
…
![Page 69: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/69.jpg)
Agenda
● What is Open Data ?● Use of Open Source Software in web crawling.● Starting new Open Source project hk0weather
to create Open Weather Data.
![Page 70: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/70.jpg)
We want a easier way to access the public data.
![Page 71: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/71.jpg)
We want a better life with public data.
![Page 72: Use of Open Data in Hong Kong](https://reader034.fdocuments.in/reader034/viewer/2022052523/5552c203b4c90581158b4841/html5/thumbnails/72.jpg)
Thank You!sammy.hk
http://slidesha.re/1cleS2y