Web scraping in German Price Statistics - … scraping in German Price Statistics Dorothee Blang,...
1
Web scraping in German Price Statistics Dorothee Blang, Federal Statistical Office of Germany Federal Statistical Office of Germany, 30th Voorburg Group meeting, Sydney, 21th – 25th september 2015 Aims of the feasibility study Is it possible to automize price surveys via internet by imitation of manual collection? Is this an efficient survey method? What are the advantages / disadvantages? Results Yes, for a lot of products it‘s a feasible solution It‘s efficient and can help to increase number of price observations Development requires profound programming skills Website changes occur irregularly, thus – Service has to be available at any time – Work load for service cannot be predicted Conclusions Implementation in daily production is intended Allocation of stuff resources for support is an essential precondition iMacros – recording scripts, form filling iMacros – code example for flights iMacros – data extraction IT-infrastructure and applied tools MySQL database: Output Data MySQL Database: Input Data for iMakros Data extraction
Transcript of Web scraping in German Price Statistics - … scraping in German Price Statistics Dorothee Blang,...
Web scraping in German Price StatisticsDorothee Blang, Federal Statistical Office of Germany
Federal Statistical Office of Germany, 30th Voorburg Group meeting, Sydney, 21th – 25th september 2015
Aims of the feasibility study Is it possible to automize price surveys via internet by imitation of manual collection? Is this an efficient survey method? What are the advantages / disadvantages?
Results Yes, for a lot of products it‘s a feasible solution It‘s efficient and can help to increase number of price observations Development requires profound programming skills Website changes occur irregularly, thus – Service has to be available at any time – Work load for service cannot be predicted
Conclusions Implementation in daily production is intended Allocation of stuff resources for support is an essential precondition
iMacros – recording scripts, form filling iMacros – code example for flights
iMacros – data extraction
IT-infrastructure and applied tools
MySQL database: Output Data
MySQL Database: Input Data for iMakros Data extraction
