03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for...
Transcript of 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for...
![Page 1: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/1.jpg)
Roadmap to Mobile Phone
DataBuilding preparedness towards use of mobile data
Siim Esko
Positium
Estonia
@positium
www.positium.com
30/04/2019 Mobile Phone Data Training – Bogota, Colombia
![Page 2: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/2.jpg)
The Puzzle Pieces – What do you need
Data
Mobile
operators
Infrastructure
Data centre
Skills
Staff,
consultants,
universities
Legal
Privacy
preservation
Legal
supervision
Technology
Mature big data
tech
Funding
Low-hanging
fruit
International co-
funding
![Page 3: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/3.jpg)
Processing Location
Depends on:
• Legal basis -> What is possible?
• Method -> Is centralized processing necessary for validity?
• Potential Future Scope -> What is efficient?
![Page 4: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/4.jpg)
Data extraction Raw data Transfer dataData importer and
formatter
Prepared data Core processes Core data modelDomain-specific
processes
Aggregated dataReference data &
calibrationExtrapolating to
general populationDomain specific
resulting indicators
Dissemination (API)
Domain aggregated data
API / Applications
Components of the system – open source
![Page 5: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/5.jpg)
• Modular architecture
• Transitional databases (for data mining and QA)
• Metadata generation throughout the process
• Parallel processing (Hadoop?) according to processing time
requirements
• API for dissemination
• Data transmission standards (SDMX, JSON, XML, …)
• White box / Grey box / Black box
• Maintenance agreements (SLA)
Desired aspects of the system
![Page 6: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/6.jpg)
Required Skills
• Tools for running processing algorithms (e.g. Java, Python, R)
• Supporting tools for QA of the data and visual analyses (e.g. GIS tools such as Esri, MapInfo, QGIS, R, and Excel, Tableau, Qlik, etc.).
• Skills of developing software for the parallel processing of big data – data processing software, ETL (Extract, Transform and Load) developer with Python, Java, PIG, Hive, Spark, SQL, etc.
• Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.
• System architect and system operators
• Statistical data processing methodologists with big data experience.
30/04/2019 Mobile Phone Data Training – Bogota, Colombia 6
![Page 7: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/7.jpg)
Desired Timeline
Interest:
seminarsFeasibility Decision Implement
Continuous
updates
6-12 months 6-12 months 6-12 months
![Page 8: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/8.jpg)
Seminars, workshops
+
Pilot Project / feasibility assessment
+
Implementation cost
+
Maintenance (SLA)
Cost of the system
![Page 9: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/9.jpg)
Cost of the system
![Page 10: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/10.jpg)
Cost of the system
![Page 11: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/11.jpg)
Funding for Mobile Data Analytics comes from Value
• Population statistics
• Everyday commuting
• Transportation planning
• Urban planning
• De facto population
• Tourism statistics
• Epidemiology
• Economics
• Natural disasters, proximity of population to risks
• Safety and Security
• ICT statistics
• Spatial marketing
• Scientific research in many areas
![Page 12: 03 Mobile Phone Data - Roadmap · Python, Java, PIG, Hive, Spark, SQL, etc. •Skillset for PostgreSQL/PostGIS, Oracle or any other database system (including GIS data) is required.](https://reader035.fdocuments.in/reader035/viewer/2022081513/5f01b8987e708231d400b7cd/html5/thumbnails/12.jpg)
Homework
• What are the first priority projects in your country for the use of mobile phone data?
• Personal opinion – Gathered at the workshop
• Consolidated NSO opinion – To do after returning back to office
30/04/2019 Mobile Phone Data Training – Bogota, Colombia 12