ANGELO FURNO INRIA UrbaNet, INSA de Lyon – France

angelo.furno@inria.fr 4-8 July 2016

QGIS A PRACTICAL INTRODUCTION

Orange Labs

Paris

Angelo

Furno 30/05/2016

A little bit about Myself…

• Current Positions • Since Nov. ‘14: Inria postdoctoral researcher

• Inria UrbaNet Team, CITI-Lab INSA-Lyon • Inria CORDIS fellowship - ABCD ANR Project

• Team Responsible: Hervé RIVANO

• Advisors: Razvan STANICA (INRIA/INSA), Marco FIORE (INRIA/CNR-Italy)

• Département Télécommunications Services et Usages, l’INSA-Lyon

• Previous Positions and Formation • July 2014: PhD in Information Engineering

• University of Sannio, Benevento, Italy

• Thesis: Scalable Service Composition in Autonomic Computing

• Experience with QGIS during the ABCD project • NOT AN EXPERT BUT SOMEONE WHO FOUND HIMSELF WORKING WITH QGIS!

• Profiling of Users’ Aggregate Mobile Traffic Data • Temporal Profiles of users’ traffic demand

• Spatial Profiles and land use classification

4

Angelo

Furno 30/05/2016

Outline

• Part 1

• GIS and QGIS basics

• Practical Session 1: Visualizing data through maps

• Part 2

• Basic spatial analysis

• Practical Session 2: Searching data based on geographical variables

• Part 3

• Visualizing and Analyzing spatio-temporal data

• Practical Session 3: Analyses with mobile phone data

• Advanced tools and processing with QGIS

Angelo

Furno

BASIC INTRODUCTION GIS, Data Layers, Coordinate Systems

1

Angelo

Furno 30/05/2016

Geographical Information Systems

• Geographic Information System • system designed to capture, store, manipulate, analyze, manage, and present

geographically referenced data

• ask questions of data related to maps, search for patterns and distributions and investigate the underlying relationships between different sets of spatial data

• Applications • Municipalities, forestry, military, commercial businesses

• What do you use them for?

• Some typical usages • Measure distances and areas

• Draw maps that are easy to update

• Study phenomena by geographical proximity

• Stacy Maple • hunter-gatherer tribes were located in places with panoramic views where they could see the surrounding land

• John Snow (1854) • created a map that showed the relationship between cholera and the public sources of water in Soho England

http://www.iota-es.de/light_pollution

Light Pollution in France

Angelo

Furno 30/05/2016

The GIS Workflow

1. Data acquisition

2. Data integration and management • storage, metadata, indexing, projection

3. Spatial Analysis • querying, selecting, filtering, joining, proximity analysis, spatial statistics, etc.

4. Visualization and Cartography

Output

• Maps • Printed

• Digital (PDF, JPEG, PNG)

• Other data

• Spreadsheets

• Databases

• Files

• Conclusion about studied phenomena

Input: 2 main kinds of data • Vector Data

• Raster Data

see next slide

Angelo

Furno 30/05/2016

Vector Data Layers

• Vector data is used to show discrete spatial features • Geo-referenced by means of one or more 2-D (X-Y) coordinates

• Point feature

• trees, base stations

• Polyline feature: connected points not forming an enclosed shape

• highways and rivers

• Polygon feature: connected points that form an enclosed shape

• school boundaries

• Vector Attribute table • Non-spatial information associated to the vector feature

• Some Vector Data Formats • Shape Files by Environmental Systems Research Institute (ESRI),

standard in QGIS

• CSV (Comma Separated Values)

• Keyhole Markup Language (KML)

• XML-based language for managing the display of 3D geospatial data.

• WFS (Web Feature Service)

• Protocol to stream data directly from a server to the GIS

Angelo

Furno 30/05/2016

Raster Data Layers

• Image-based data • Matrix of cells (pixels) organized into rows and columns (a grid)

where each cell contains a z-value • representing information such as elevation, temperature

• Appropriate for continuous data where discrete boundaries are not necessary

• e.g., digital aerial photographs, imagery from satellites, digital pictures, or even scanned maps

• Pixels have to be georeferenced, and cell size must be known • Need a helper file or coordinates are saved within the file itself

• Coordinate system should be known

• Uses? • Base map for other (vector) layers

• Elevation maps, terrain, bathymetry

• Be used to analyze/combining multiple rasters

• Formats • DEM (Digital Elevation Model)

• Color-coded based on the elevation by default • in meters from the sea level,

• File formats: USGS DEM, GEOTIFF (Tagged Image File Format)

• ECW (Enhanced Compressed Wavelet) • Typical size is much smaller than GIF or JPEG

• WMS (Web Mapping Service) / WMTS (Web Mapping Tiles Service)

Angelo

Furno 30/05/2016

Geographic Coordinate Systems

• GCS uses 3D models to define

locations on the earth

• Sphere, Ellipsoid, Geoid

• Latitude & Longitude

• Degrees, minutes, seconds

• Latitude: from 0° at the Equator to 90° at the North Pole and -90° at the South Pole

• Specifies the north–south position of a point on the Earth's surface

• Longitude: from 0° at the Prime Meridian to 180° eastward and -180° westward

• Specifies the east-west position of a point on the Earth's surface

• Most Popular GCS: World Geodetic System 84 (WGS 84 - EPSG: 4326)

• The coordinate origin of WGS 84 is meant to be located at the Earth's center of mass

• The error is believed to be less than 2 cm

• Uses the EGM96 (Earth Gravitational Model 1996) geoid, revised in 2004

• Standard for use in cartography, geodesy, navigation

Angelo

Furno 30/05/2016

Representing the earth on a 2D surface

• Projected Coordinate Systems

• earth is flattened on a 2D surface

• Naive approach

• Longitude as X and Latitude as Y

• Longitude and latitude can locate exact positions on the surface of the

globe, but they are not uniform units of measure

• Only along the equator does the distance represented by one degree of longitude

approximate the distance represented by one degree of latitude

• because the equator is the only parallel as large as a meridian

• E.g.: one degree of longitude at the equator equals 111.321 km, while at 60° latitude it is only

55.802 km.

• You can’t measure distances or areas accurately or display the data

easily on a flat map or computer screen

Angelo

Furno 30/05/2016

Popular Projected Coordinate Systems

• A map projection is a systematic transformation of the latitudes and longitudes of locations on the surface of a sphere or an ellipsoid into locations on a plane

• different organizations use different projected coordinate systems

• always introduce error and distortion

• Distortion may be minimized in one or more of the following properties • Shape conformal

• Shape is preserved for small areas, but for larger areas (such as a continent)

• Map projections preserve angles locally

• Transverse Mercator (UTM zones), Lambert Conformal Conic

• Area equal area • Lambert azimuthal equal-area

• Albers equal-area

• Distance equidistant

• Azimuthal equidistant

• Direction true direction

• Direction to a fixed location B (the bearing at the starting location A of the shortest route) corresponds to the direction on the map from A to B:

• Littrow

Angelo

Furno 30/05/2016

Open GIS – Formats, Apps, Data

• Open Standards / Formats – Open Geospatial Consortium (OGC)

• GML; CityGML(3D); KML (GE); GeoJSON (JavaScript)

• Moving Features

• NetCDF

• WMS; WMTS; WFS

• Open Source Applications – Open Source GeoSpatial Foundation (OSGeo) • QGIS/GRASS (QGIS plugins; extend with Python)

• gvSIG/uDig (former extensible/mobile; latter Java/Eclipse based)

• RDBMS+Spatial Extensions – e.g. Postgres with PostGIS extensions (also pgRouting)

• Open Data

• OS OpenData (since 2010); Also extra OS/other data avail to HE via EDINA Digimap

• OpenStreetMap (linked to likes of OpenLayers/Leaflet etc.) – VGI (Volunteered GI)

• Also traditionally ‘open’ providers: Natural Earth, USGS

Angelo

Furno

INTRODUCTION TO QGIS

1.1

Angelo

Furno 30/05/2016

What is QGIS?

• Quantum GIS

• Free and Open Source GIS Software

• Official Project of QSGEO

• Runs on Unix, Mac OSX, Windows, and Android (!)

• Version 1.0 was first released in January of 2009

• development began back in 2002 by Gary Sherman

• Fully capable, supports vector (points, lines, polygons) and raster

• Many Supported input data formats

• Esri Shapefiles, PostgreSQL/PostGIS geodatabases, GRASS based vector and

raster files, as well as GeoTIFFs

• Standalone Desktop QGIS: two versions currently available

• Long-term Stable build of QGIS 2.8.9 (Wien)

• supported with backported bug fixes for one year

• Latest stable release 2.14.3 (Essen) provides access to latest plugins and features

Angelo

Furno 30/05/2016

QGIS Installation

• You should have QGIS already installed on your machine • https://www.qgis.org/en/site/forusers/download.html

• For Windows users, you can download the standalone installer (64/32 bit)

• Also installs python 2.7 that runs inside of QGIS

• For Mac OS users, go to KyngChaos QGIS download page

• For Debian/Ubuntu users, edit your list of alternative software repositories, /etc/apt/sources.list file, by adding one of the deb-lines below

Angelo

Furno 30/05/2016

Let’s Start Playing

with QGIS!

…Just click on QGIS Desktop 2.14.3 icon…

Angelo

Furno 30/05/2016

Change QGIS Language?

Angelo

Furno 30/05/2016

Getting familiar with the UI

Angelo

Furno 30/05/2016

Map Canvas

Angelo

Furno 30/05/2016

Map Navigation Toolbar

Angelo

Furno 30/05/2016

File Toolbar

Angelo

Furno 30/05/2016

Manage Layers Toolbar

Angelo

Furno 30/05/2016

Attribute Toolbar

Angelo

Furno 30/05/2016

Layers Panel

Angelo

Furno 30/05/2016

Browser Panel

Angelo

Furno 30/05/2016

Missing a Panel? Use the View Settings

Angelo

Furno 30/05/2016

Some Useful Plugins to Install

1

2

3

INSTALL

1. TimeManager

2. OpenLayers

3. QuickMapServices Plugin

Angelo

Furno

EXERCISE MODULE A:

GETTING FAMILIAR WITH QGIS Layers and Maps

2

Angelo

Furno 30/05/2016

Loading Vector Data Layers

• We are going to work on heterogeneous vector data • each corresponding to different layers in QGIS

• from different data providers

• possibly, different Coordinate Systems

• Our focus will be the Grand Lyon region • An area of approximately 530 Km^2

• 67 communes (including 9 Lyon arrondissment)

• Most of the data is already available as shapefiles or csv files

• Shapefile is a specific file format to store GIS data in group of files • Each layer consists of several files with the same name, but different file types.

• Shapefiles are easy to send back and forth, and most GIS software can read them.

• QGIS files are simply TEXT FILES

Exercise A.1: Adding vector layers to your first QGIS project • Open QGIS. You will have a new, blank map (or the recent project list).

• Click on the Add Vector Layer button:

• In the following dialog browse for the shapefile /lyon_qgis/boundaries/grand_lyon_administrative_boundaries.shp

Angelo

Furno 30/05/2016

Playing with Coordinate Systems (1)

• You have just loaded a multypoligon layer containing the shapes of the Communes

in Grand Lyon. What’s the Coordinate System of the loaded layer? • Click on the Attribute Table button: . What do you see? • Explore the buttons of the Attribute Toolbar

• If you cannot properly see strings, you probably are on a Windows machine and

you may want to use the UTF-8 System Encoding, by right clicking on the layer,

Properties -> General -> Data Source Encoding

Remember to save your QGIS project OFTEN

!

Use the button to view

attribute values by clicking on features

on the map

!

Angelo

Furno 30/05/2016

Playing with Coordinate Systems (2)

Exercise A.2: exploring layers with different CRS

• Add a second shapefile containing the same features as the previously loaded

layer, but in EPGS: 32631 (UTM Zone 31). Load the layer: boundaries/grand_lyon_administrative_boundaries_utm_zone_31N.shp

• Uncheck the two layers one per time in the layer panel

• What just happened? Try to explain!

• Go to Project Properties -> CRS • Change the status of the CRS option on the fly transformation

• Right click on one of the loaded layers, Refresh and use the Zoom to layer multiple times to find your layers in the map canvas

• Re-enable the CRS option on the fly transformation

• By right clicking on the layers, you can rename the two layers with simpler names,

e.g. Lyon_4326 and Lyon_32631

• To better organize your layers, you can re-order the layers by moving them up and

down in the layer panel. Top layers are rendered over the bottom layers

• Largely use the option Group Selected on multiple related layers (CTRL+left click to select more than one layer) and rename accordingly the newly created layer

Angelo

Furno 30/05/2016

On the fly Coordinate Reprojection

Check Settings Options CRS

The option «on the fly reprojection» should be checked by default

Allows layers having different CRSs and being correctly visualized.

The different layers will however keep the different CRSs

Angelo

Furno 30/05/2016

Adding attributes to a vector layer

Exercise A.3: adding attributes, computing shape areas w different CRS

• Check that the «on the fly» option is disabled for the current project

• Select the layer Lyon_4326

• Click on the button Open Field Calculator • Create a new decimal field called «area», with a precision of 3 digits

• Use a Geometry Function in the Expression field to compute the area for each

shape feature (the value will be in degree^2)

• What happened to our layer?

• Check the attribute table again and save the layer

• Do the same for the layer Lyon_32631

• Compare the two areas for the same feature. Are they the same?

Know that Lyon 1er has an area of approx 1.52 Km^2, Lyon 2eme of 3.40 Km^2,

Lyon 3eme 6.35 Km^2 (from Wikipedia.fr), …

• You can compute the sum of the area field for the two layers by using the Show Statistical Summary button that will open the Statistics Panel. Use the panel to

compute the total area and see the big difference!

• You can measure distances, areas and angles directly on the map canvas

Angelo

Furno 30/05/2016

Adding attributes to a vector layer (2)

Keep in mind the importance of the CRS

when computing distances or areas!

! All units will be calculated in the units of the projection, so you

probably want to project it to an “equal area” projection that uses feet or meters before doing that,

rather than lat/lon.

!

You can use any mathematical operation to specify your area, e.g. you

can divide $area by 10^6 to have the area in squared

meters instead kilometers

! You can re-project a layer into a new CRS by right-clicking on the layer ->

save-as -> CRS (select the proper one). You will generate a brand new layer in the new CRS

!

Angelo

Furno 30/05/2016

Selecting features based on attribute values

Exercise A.4: selecting communes with smaller area

• Uncheck or remove the layer lyon_4326. We will not use it anymore

• Select your lyon_32631 layer and click on the select by expression button

• Write an expression to select only communes with an area smaller than 5 Km^2

• Right click on the layer and use the save layer option to save only the selected features in a new layer called lyon_small_communes. The layer will be automatically

loaded in your panel.

Exercise A.5: selecting communes by name

• Use again the select by expression on the new layer to select only the communes

corresponding to lyon arrondissements. Hint: look at the operators or the string

expression

• Use the select by expression to find all the non-Lyonnaise communes whose area

is smaller than 5 Km^2

• Use the invert selection button to invert your selection

Angelo

Furno 30/05/2016

Adding attributes to a vector layer

Exercise A.6: joining layers on common attributes

• Add the layer population_data/population_legales_2013_rhone_alpes.csv, containing

INSEE census data for Grand Lyon. To correclty interpretate field types, use Add Delimited Text Layer instead of Add Vector Data.

• Right click on Lyon_32631 -> Layer Properties -> Join • Select the population_legales_2013_rhone_alpes as your Join layer, and code_commune

as your Join Field and insee_id as your Target Field • Select only the interesting fields you want to join from the join layer.

• What happened to your layer? Hint: Look at its attribute table!

Angelo

Furno 30/05/2016

Styling up your layers

Exercise A.7: change layer symbology

• In Layer Properties -> Style change the default style color, transparency, symbols,

etc. used for rendering. You can also save the style for later use.

• Use the previously joined fields (e.g. Pupulation Totale) from the census data and

the Graduated style to present with red colors the areas with larger population

and with blue colors those with smaller population

• Remove borders from the colored shapes to make the map less cluttered

• Use the same approach to present areas according to their population density.

population density

Angelo

Furno 30/05/2016

Creating maps Exercise A.8: creating a university map of the Grand Lyon area

• Load the multipolygon layer university/university.shp

• The file contains polygons representing university buildings (in WSG:84)

• Re-order the layers in the layer panel if the university layer is not visible

• We want to generate a new point-feature layer in which each point correspond to

the centroid of a university building

• Use the Vector -> Geometry Tools -> Polygon Centroids tool to compute

the new layer • Save it in a separate shapefile (e.g. university_centroids.shp)

• In the attribute table, enter the Edit Mode, select all the features without a

name (by a select expression) and remove them.

• If you want to further refine your data, remove all the features whose name

does not contain the word «université» or «iut» (case unsensitive)

• Change the symbology of the university layer to show a school-related symbol

instead of the polygons

• In Style, change the Simple Marker with a symbol from SVG Marker you like

• Change the font size if needed (e.g., 7) and the label placement (around point, with an offset of 2 for x and y)

• Add a Label to show the name of the university

• Modify the label to show a buffer area around the name (use a value of 2)

• To avoid cluttering, you can enact the option Scale Dependant Visibility in

the layer general tab. The layer will be not rendered above the specified scale

Angelo

Furno 30/05/2016

Exercise 8: Maps

Angelo

Furno 30/05/2016

Creating maps (2)

Exercise A.9: using QuickMapServices /OpenStreetMap Layers

• Load the Open Street Map by selecting Web -> QuickMapServices -> OSM -> OSM Mapnik

• The Plugin will automatically download a new layer (in the layer panel)

containing a Map (only available as EPSG:3857 -WGS 84 Pseudo Mercator) for

the currently active area

• In the Style tab, change the transparency to 50% for the Lyon_32631 layer

To properly work with the

OpenStreetMap layer, your project will be rendered according to the EPSG:3857 CRS

!

Angelo

Furno 30/05/2016

Creating maps (3)

Exercise A.10: using the Map Composer to generate detailed maps

• You can easily export your current map canvas to image files (.png, .jpg, .etc) by

Project -> Save as Image. This will produce nice pictures, but your map will be

very basic (no legend, no additional labels, no control on dpi resolution, etc.)

• Go to Project -> New Print Composer to create a new composer • Give the composer a name, e.g., GrandLyonComposer

• By default, the output will be on an A4 page, landscape orientation, with a 300 dpi

resolution and 0 cm margins. You can easily fit this settings to your specific needs

• Click on the button to add a new map to the page (every layer currently active

in your map canvas).

• Draw a rectangle on the page. Your map will be shown in that rectangle and a new

item (Map) will appear in the top-right items panel

• Use the button to move the map and zoom (using the mouse wheel, with CTRL

pressed for finer movements) at your commodity. Use the refresh button

• When your map looks ok, you can lock the item on the canvas in the items panel

• Add a legend to your map by pressing on the button

• You may want to reduce the number of items in the legend

• In the items panel, select your legend and go to Item Properties in the panel

just below. You can change the properties, uncheck the Auto Update option

and hide/remove all the item you don’t want to show

• You can add a scalebar, shapes, arrows and text labels

Angelo

Furno 30/05/2016

You can save the PrintComposer for later re-

usage (thus storing your settings for the page, dpi, etc.)

!

Export your map to pdf or other

formats by using one of the buttons

!

You can save the map you created (the items and their properties) by saving a Composer Template (.qpt

file). You will be able to reload them in a map composer

!

Angelo

Furno 30/05/2016

Creating maps (4) Exercise 1.11: can you reproduce this map?

Angelo

Furno

EXERCISE MODULE B:

SPATIAL ANALYSIS Working with multiple layers to answer

basic spatial research questions

3

Angelo

Furno 30/05/2016

Vector Data Spatial Analysis

• The GIS Process

• Before we start, it would be useful to recall the process that can be used to solve any

GIS problems

• The way to go about it is:

• State the Problem

• Get the Data

• Analyse the Problem

• Present the Results

Exercise B.1: (solution on the next page, try it yourself) • Load the residential landuse OSM ground truth (file

residential/residential_areas.shp)

• Retrieve all the residential areas (polygons with the associated OSM information)

within the communes of Grand Lyon with more than 40,000 inhabitants

Angelo

Furno 30/05/2016

Vector Data Spatial Analysis: select by

location and spatial queries Solution B.1 (first approach)

• You can use Vector -> Research Tools -> Select by Location • This tool selects all the features of the input layer satisfying one of the

available spatial property (e.g., intersect, touch, overlap, within)

• You can restrict the search on a subset (expressed as a selection) of the

target layer

• ATTENTION: input and target layers must have the same CRS • Project the population layer into a new one having the same CRS as the

residential layer • use the Lyon_32631 layer, which contains the right areas and has been already

joined to the population layer

• the joined fields will be saved as part of the new shapefile

• if you want to edit their names, install the Table Manager plugin

(then, Vector -> Table manager)

• Select the features having the required population in the projected layer

• Apply the Select by location tool using the within property • You have to check the option Only Selected Features

• Input layer = residential_areas / Output Layer = Lyon_32631_projected • Check the difference when using the other properties (e.g. intersection, etc.)

• Save the selection to a new layer residential_areas_in_populated_communes.shp

• You can achieve the same result using Vector -> Spatial Query with within and

selected geometries option

Angelo

Furno 30/05/2016

Vector Data Spatial Analysis: clipping

Solution B.1 (second approach)

• You can also use Vector -> Geoprocessing Tools -> Clip • This tool creates a new layer by keeping only the features that fall within (or

overlap the borders) the target layer

• Polygon features that are partially covered by features of the target layer are

kept, but only with the shape portion covered by the target layer (see picture)

• Only the attributes from the source layer are kept in the resulting layer

• You can restrict the search on a subset (a selection) of the target layer

• ATTENTION: input and target layers must have the same CRS

• Can you spot any difference between this solution and the previous one?

Clipping the brown layer

to the yellow one

This is still one feature

despite crossing different

yellow shapes

Angelo

Furno 30/05/2016

Vector Data Spatial Analysis: intersection

Solution B.1 (third approach)

• You can also use Vector -> Geoprocessing Tools -> Intersect • Similarly to Clip, this tool creates a new layer by keeping only the features

that fall within (or overlap the borders) the target layer

• Polygon features that are partially covered by features of the target layer are

kept, but only with the shape portion covered by the target layer

• Both the attribute fields from the source layer and those from the target layer

are kept in the resulting layer. Therefore, if a feature from the input layer

overlaps multiple target layers, it will be splitted in multiple features having

different values for the attribute fields of the target layer

• You can restrict the search on a subset (a selection) of the target layer

• ATTENTION: input and target layers must have the same CRS • Can you spot any difference between this solution and the previous one?

Clipping Intersecting

You can also apply the geoprocessing tools to

point/line feature layers (e.g., you might

want to know the commune information

for each university)

!

Angelo

Furno 30/05/2016

Output

Layer

Clip

• Creates a new version of the input layer where the only features

present are those overlapped by the Clip layer

• Can be used on point, line, polygon, or raster files

Angelo

Furno 30/05/2016

Output

Layer

Intersect

• Creates a new layer where the only features present are those

overlapped by the input layers.

• Can be used on point, line, polygon, or raster files

Angelo

Furno 30/05/2016

Dissolve

• Simplifies (and reduces) polygons in a vector file

• With respect to a “dissolve” field

• All features having the same value on the dissolve field are merged together

• Single-layer operation

Output

Layer

Angelo

Furno 30/05/2016

Output

Layer

Buffer

• Creates a copy of each feature of the input layer into a new layer that has the border extended out a defined distance • Single-layer operation

• Can be used on point, line, polygon, or raster files

• An optional dissolve can be performed to remove buffer feature overlap

Buffered point layer

Angelo

Furno 30/05/2016

Output

Layer

Union

• Creates a new layer where all features from the input layers are

completely preserved, but all overlaps result in new unique polygons

(having field values from both the overlapping layers)

• Can be used on point, line, or polygon files

Angelo

Furno 30/05/2016

Output

Layer

Difference

• Creates a new layer containing only the features from the input layer

that are not overlapped by those of the difference layer

• Can be used on point, line, or polygon files

Angelo

Furno 30/05/2016

Output

Layer

Symmetrical Difference

• Creates a new layer containing both the features from the input layer

not overlapped by those of the difference layer and vice-versa

• Can be used on point, line, or polygon files

Angelo

Furno 30/05/2016

Output Layer

Merge multiple shapefiles

• Creates a new layer where all features from the Merge layer and the Input layer are preserved, keeping all features intact (but merging the attribute fields) • Can be used on line or polygon files

• Accessible in Vector -> Data Management Tools -> Merge multiple shapefiles to one

• If you want to merge multiple polygons in one single polygon use Vector -> Geometry Tools -> Singleparts to Multipart

The two features will keep

their original shape

Angelo

Furno 30/05/2016

Other useful spatial analysis tools

• Points in Vector

• When solving our spatial analysis problem, you might need to combine a point vector

layer X and a polygon vector layer S

• You might like to include in the polygon layer S a new attribute counting how many points

from X fall within each feature in S

• The analysis tool Vector -> Analysis Tools -> Points in Vector can be used for that

• For each polygon shape s, containing n points from X, the tool will add a counter value

PNTCNT and allows for aggregating (sum, mean, min, max) the other fields from X

• To properly use the tool, the two layers should be in the same CRS

• Spatial Join

• Vector -> Data Management Tools -> Join attributes by location allows for transferring

attributes from another layer to the target one based on their spatial relationship

• E.g. you have one layer for Lyon’s communes and another layer reporting on one-day traffic per city

base-stations (i.e., pairs of lat-lon points with the total traffic in one day). You might want to transfer

on the Lyon communes the total trafic (sum of all the base stations falling in one commune).

• For more options use the SpatialJoin plugin (requires refFunctions plugin)

Angelo

Furno 30/05/2016

Vector Data Spatial Analysis

Exercise B.2: Imagine you are an estate agent looking for a residential property in Lyon for a

customer with the following criteria:

• The building should be larger than 300m^2

• It needs to be in a highly-populated commune of Grand Lyon (more than 3500 inhabitants per

Km^2).

• It must be within a walking distance of a school (say 300m).

• Within a 1000m distance from important roads of the city (primary highways)

• Farther than 100m to a restaurant or fast-foods (we don’t like noise during evening/night).

• It should also be in an area with more than 2 bicycle stations closer than 500m

• because our clients love biking and wants to be sure to have a close place to leave it

when coming back home

• A green area (forest, park or orchard) should be on a cycling route not far than 300m from the

building

Assumptions and hints

• You have all the grand truth information you might need in the provided project files

• To simplify our analysis, assume we refer to areas related to the whole building!

• Assume all requirements are strict, so the order in which you satisfy them does not matter

• Remember to compute the areas according to a proper equal area projection (see next slides)

• Remember about the spatial operations (buffer, intersection, clip, difference, union, etc.)

• Remember that the input layers have to be in the same CRS

Angelo

Furno 30/05/2016

Some more detailed hints

• Geometry Transformation allows for projecting on-the-fly • In the field calculator, you have the geometry function transform allowing for

transforming “on-the-fly” shape features from one CRS to another one.

• The area function (without $) takes a geometry as an input and compute the area in

the units of the CRS

• Therefore, you can compute the area in an equal area projection, thus avoiding

duplicating your data:

• area(transform($geometry, 'EPSG:4326','EPSG:32631'))

• Divide by 10^6 for km^2

• Remember to check geometry validity

• Ground truth data often have topology problems

• As a consequence, area values can be completely wrong (e.g., very large/small values)

• Use the Vector -> Geometry Tools -> Check Geometry Validity to spot errors

• The tool will produce a log of retrieved errors

• You can click on the log entries in order to have the map canvas centered above the bad

feature (zooming before doing that can be very useful)

• You can correct the feature using the Edit Panel -> Node Tool

Angelo

Furno 30/05/2016

Solution Solution: • Load residential_buildings.shp

• Correct its geometry (1 shape only should be needed to modify) • Compute buildings area: area(transform($geometry, 'EPSG:4326','EPSG:32631'))

• Select residential buildings larger than 300m^2 • Load grand_lyon_administrative_boundaries_correct_areas.shp (from previous exercise) and select high-density areas

• Use spatial query (with the “selected” option checked on both layers and create a new virtual layer using the tool) • Save the virtual layer to EPSG:32631, residential_buildings_in_highly_populated_communes.shp

• Load school.shp file (it’s in EPSG:4326)

• Attention: buffer unit is relative to the system. So, you will need to convert schools from EPSG:4326 to EPSG:32631

• Use a 300 m buffer distance with 5 segments (number of sides for the surrounding polygon), use the dissolve option since

we are only interested to the shape (not the attributes) • Save the buffer to schools_32631_buffer_300m.shp

• Load the highways.shp file, select highway = ‘primary’, then save as an EPSG:32631 layer

• Compute a 1km buffer layer around primary_highways_32631.shp (remember to dissolve)

• Compute the intersection between school_32631_buffer_300m.shp and primary_highways_32631_buffer_1km.shp and

save to schools_and_primary_highways_buffer.shp

• Load restaurants_and_fast_foods, perform conversion to EPSG:32631 and buffering to 100m • Compute the difference between school_150m_and_highways_500m.shp and

restaurant_and_fast_foods_buffer_100m.shp. Save to schools_and_primary_highways_far_from_restaurants_buffer.shp

• Load the layer cycling_stations jc_decaux_cycling_stations.shp, convert it to EPSG:32631

• Buffer WITHOUT DISSOLVING to 500m the layer buildings_in_highly_populated_communes.shp

• Count Points in Polygon for buildings_in_highly_populated_communes_500m.shp wrt

jc_decaux_cycling_stations_32631.shp, adding a BIKECNT field

• Save residential_buildings_highly_populated_communes_32631_buffer_500m_not_dissolved_bikes.shp and select

features with more than 2 bike station ("BIKECNT" > 2)

• Add all the green layers to your project, compute their union (or merge) , save to EPSG:32631

• Load cycling_roads.shp, convert to EPSG:32631 and compute the intersection with the union green layer in EPSG:32631

• Buffer the intersection to 300m (with dissolve) and intersect it with the previous selection performed on the

buildings_highly_populated_area_with_bike_stations.shp

• CLIP the previous result with the schools_and_primary_highways_far_from_restaurants_buffer.shp • (intersection won’t work because of the high complexity of the layers. As an alternative use buffer 0m and dissolve)

• YOUR SOLUTION WILL BE GIVEN BY ALL THE BUILDINGS FALLING IN THE PREVIOUS INTERSECTION (do a spatial query)

Angelo

Furno 30/05/2016

The Solution

Angelo

Furno 30/05/2016

Falling in the denser communes

Building area

larger than

300 m^2

Angelo

Furno 30/05/2016

300m from schools 1km from major roads 100m or more away

from restaurants

More

than 2

bike

stations

in a

500m

range

Close

to bike

routes

traversi

ng

green

areas

Angelo

Furno 30/05/2016

Visualizing Spatio-temporal Data

Exercise B.3: Using the Time Manager plugin to visualize mobile traffic data

• You will need the TimeManager plugin installed in your QGIS • In the folder mobile_phone_data, you can find csv files for one week of CDRs in France and Orange’s base

stations lon-lat position (nidt_france.csv)

• Load as many csv files as you want using the option “Add as Delimited Text Layers” (they have no geometry). Also, load the file nidt_france.csv. Pay attention to the presence of the header in the nidt CSV and use a

proper CRS

• Load the Grand Lyon boundary layer (in EPSG:4326) from previous exercises and select base stations in the Grand Lyon area, by saving them to a new shapefile nidt_lyon.shp

• To properly work with the TimeManager plugin, you need to add lon-lat coordinates in the traffic record layers. To this purpose, join each hourly traffic csv with the nidt_lyon.shp

• Select just the records referring to Lyon (how do you do that?). Then, save them to a new csv file. Do not add the new layer to canvas. You will need to load it explicitly as delimited text layer, by specifying the X-Y fields containing the lon-lat positions

• Know that each record contains the following information in the CDRs files : timestamp(date and hour); tower id; nb of French SIMs, nb of non-French SIMs (external mobiles in roaming), nb of communications (calls + SMS) of French users, nb of communications (calls + SMS) of non-French users

• In the layer style dialog, click on the Simple Marker to define a rule-based size for each CDR-related

shapefile you just saved. The rule should visualize each base station as a marker with size depending on one

traffic measure you want to show (e.g. you can compute the total number of communications from French and non-French users and use a size = 1/1000 * communication of French users). You can copy and paste

the style from one layer to another one. Try other rules!

• Configure the TimeManager Plugin to show the layers and play! • Use this format for the time field in time settings: %Y-%m-%d %H • Set the time frame size to 1 hour

Angelo

Furno 30/05/2016

Voronoi Polygons to Represent

Base Station Coverage Areas

• In spatial analysis with mobile phone data, we often need to search for spatial patterns in the areas that surround each mobile base stations (BSs)

• Given a set of points (called seeds, sites, or generators) on a 2D plane, its Voronoi diagram partitions the plane into a set of convex polygons such that: • All points inside a polygon are closer to one seed

only than to any other

• The points on polygon edges are equidistant from nearby seeds

Base station

Mobile users

Voronoi cell

• It has been proved that the cell area of each BS forms a Voronoi tessellation • So we can use Voronoi diagrams to study the spatial properties

of cell areas

• Related to Voronoi is Delaunay Triangulation • for a set P of points in a plane it is a triangulation DT(P) such that no

point in P is inside the circumcircle of any triangle in DT(P) Delaunay

Voronoi cell

Angelo

Furno 30/05/2016

Computing Properties for Voronoi Cells Exercise B.4: We want to group ground truth (residential, green areas, business, etc.) data in for each mobile base station in Lyon. This ground truth will be used later for comparison with a clustering algorithm. Specifically, we want to associate the following information to Voronoi cell of each base station • Number of bike stations in the area

• Number of train station in the area

• Number of university buildings in the area

• Percentage of the base station area covered by green

• Percentage of the base station area covered by commercial landuse

• Percentage of the base station area covered by leisure areas

• Percentage of the base station area covered by residential areas

• An estimation of the population leaving in that area

Hint 1: use Vector -> Analysis -> Points in Polygon to count the number of points falling in another polygon.

Hint 2: use Vector -> Data Management -> Join Attributes by Location to count the number of polygons falling

in another polygon.

Hint 3: to compute the percentage of a Voronoi cell covered by a certain land use, merge the multiple polygons

associated to the land use in one single polygon use Vector -> Geometry Tools -> Singleparts to Multipart.

Then, intersect your Voronoi tessellation with the merged layer and compute the fraction of the intersected area to

the total land use area. Another approach consists in intersecting directly the layer with the Voronoi tessellation

and then dissolve on the NIDT field.

Hint 4: to estimate the population living in a Voronoi cell, use the census data available for Grand Lyon communes.

Also, consider as “livable area” the difference between the Voronoi cell and any green area in the cell. Finally,

assign a fraction of the population to each Voronoi cell. Some Voronoi cell that are at the intersection of different communes will receive different amount of population depending on the commune population density.

Angelo

Furno 30/05/2016

Land Use Detection

signature

clustering

signature

similarities ?

A

A

B

D

mobile traffic

signatures

Cluster 67: St Peter’s square

Cluster 4: Paris

Subway Stations

Spatial Profiles

(France and Italy)

Pearson correlation coefficient

Angelo

Furno 30/05/2016

Comparing GT with a Clustering Algorithm Exercise B.5: By exploiting our previous outcomes (ex. B.4), we want to compare the output of the time-series clustering algorithm with our collected ground truth information. The final goal is to find correlations between the clusters and different kinds of ground truth data. As an example: does the areas with train stations map to one specific cluster?

• In the folder clustering_results\grand_lyon_sms_call, you will find the results of our clustering algorithm

for land use classification from mobile traffic time-series. The clustering_results.dat csv file contains the

clustering output from a median week extracted from sms + calls related to the period from Tuesday 2014-08-

12 to Sunday 2014-11-30 and from Thursday 2015-03-05 to Wednesday 2015-03-25.

• Load as «Delimited Text Layer» the file into your QGIS project (The same used for Ex. B.4)

• Join your Voronoi representation with the previously loaded file, keeping the field containing the cluster_id (cluster)

• Use a Categorized Style with random colors on the joined field cluster_id to color each different cluster

• Use a label to better distinguish the cluster

NO SPECIFIC RULE IS AVAILABLE TO VERIFY CLUSTERING OUTPUT WITH GROUND TRUTH. SOME POSSIBLE APPROACHES: • Visual inspection:

• From our previous exercise, you have one layer containing a lot of ground truth information and another

layer with Voronoi cells classified by clustering.

• Can you spot correlations between clusters and areas dominated by certain kinds of ground truth?

• Duplicate the ground-truth layer you built, and use different styling to ease your search (you might want

to select those areas which are mostly residential, with high number of train stations, business, etc (you

can use the graduated style or rule-based styling)

• Intersection

• You can try to look for the maximum intersection between the clustering-classified Voronoi cells and one

specific kind of ground-truth layer

• Metrics

• You can compute metrics by leveraging your per-Voronoi-cell ground truth data • See next slides

Angelo

Furno 30/05/2016

Metrics for comparing detection techniques

• Density of ground truth elements of type per cluster

• Type entropy for the clusterset

randomness of the clusterset wrt ground truth data (lower entropy means higher cluster specificity)

• Clusterset coverage of type ground truth elements

• Type F-measure combines entropy and coverage for the analyzed clusterset

71 28/08/2015

with

set of geo-localized ground truth elements of class and = 1 if belongs to c

0 otherwise

by only those clusters having a type density higher than the average number of type ground

truth elements over the analyzed area

with representing normalized Shannon entropy

higher is better, with 1 indicating the best performance achievable by the given cluster set wrt

Angelo

Furno 30/05/2016

Milan, SMS + Call, Nov. 2013 – Clusters Clusterset and density with MWS-daily-euclidean

Angelo

Furno 30/05/2016

Milan, SMS + Call, Nov. 2013 – Clusters Clusterset and density with MWS-daily-euclidean

Angelo

Furno 30/05/2016

Main Outcomes (2)

Metro station

signature

Angelo

Furno 30/05/2016

Milan, SMS + Call, Nov. 2013 - Entropy

Takeaways 1: • Cici is an improvement

with respect to both Soto-

10 and Soto-5

• MWS entropy is lower

than Soto and Cici’s

Angelo

Furno 30/05/2016

Milan, SMS + Call, Nov. 2013 - Coverage

Takeaways 2: • The entropy gain granted by

MWS is associated to an

increase in coverage

Angelo

Furno 30/05/2016

Milan, SMS + Call, Nov. 2013 – F1 score

Takeaways 3: • Higher effectiveness of the

median-week signature for

synthetizing mobile traffic at

unit areas and support

urban fabric detection

Angelo

Furno 30/05/2016

Turin, SMS + Call, Mar. 2015 - F1 score

Takeaways 4: • MWS outperforms Cici

and Soto also in the case

of Turin

• Lower accuracy than the

Milan-2013 case, because

of the coarser spatial

granularity of cells

Angelo

Furno

PYTHON SUPPORT IN QGIS

4

Angelo

Furno 30/05/2016

Python Support in QGIS

• QGIS supports the popular Python scripting language

• Starting from 0.9 release of QGIS

• Creating custom functions within the QGIS application framework using Python

• Extends the functionality of QGIS

• PyQt Python Library provides access to QGIS UI and Core

• There are several ways how to use Python bindings in QGIS desktop

1. Automatically run Python code when QGIS starts • You can run Python code just before QGIS initialization completes by setting the PYQGIS_STARTUP

environment variable to the path of an existing Python file

2. Issue commands in the Integrated Python console within QGIS

• It can be opened from menu: Plugins -> Python Console

3. Create and use plugins in Python

4. Create custom applications based on QGIS API

Angelo

Furno 30/05/2016

Python Console

• You can run this code in your console for a previous project…

• What output do you get?

• For interaction with QGIS environment, there is a iface variable in the package qgis.utils which is an instance of QgsInterface

• allows access to the map canvas, menus, toolbars and other parts of the QGIS application

• The following statements are always executed when the console is started • from qgis.core import *

• import qgis.utils

Angelo

Furno 30/05/2016

Adding Vector Layers to the Layer Panel

• Creates a new layer and adds it to the map layer registry making it

appear in the layer list in one step

• Using the OGR library (related to shapefiles and many other file formats)

• The addVectorLayer function returns the layer instance or None if the

layer couldn’t be loaded

• Try the code above by loading the following shapefile:

layer = iface.addVectorLayer("G:/lyon_qgis/boundaries/grand_lyon_administrative_boundaries.shp",

"Grand Lyon", "ogr")

layer = iface.addVectorLayer("/path/to/shapefile/file.shp", "layer name

you like", "ogr")

if not layer or not layer.isValid():

print "Layer failed to load!"

Angelo

Furno 30/05/2016

Adding Delimited Text Layers to the Layer

Panel

• Creates a new layer and adds it to the map layer registry making it

appear in the layer list in one step

• The text above is for CSV or other delimited text files with a

semicolon as a delimiter, with field “x” for x-coordinate and field “y”

with y-coordinate

• Try the code above loading the following shapefile (use the proper

delimiter, and “x” - “y” fields

uri =

"file:///G:/lyon_qgis/mobile_phone_data/nidt_france.csv?delimiter=%s&xField=%s&yField=%

s" % (",", "lon", "lat")

vlayer = QgsVectorLayer(uri, "nidt", "delimitedtext")

uri = “file:///some_path/file.csv?delimiter=%s&xField=%s&yField=%s" % (";",

"x", "y")

vlayer = QgsVectorLayer(uri, "layer name you like", "delimitedtext")

Angelo

Furno 30/05/2016

Map Layer Registry

• If you want to use the opened layers for rendering, do not forget to add them to map layer registry • E.g., in the previous example with delimited text layers

• The map layer registry takes ownership of layers and they can be later accessed from any part of the application by their unique ID • when the layer is removed from map layer registry, it gets deleted, too

• Adding a layer to the registry: • QgsMapLayerRegistry.instance().addMapLayer(layer)

• Layers are destroyed automatically on exit, however if you want to delete the layer explicitly, use:

• QgsMapLayerRegistry.instance().removeMapLayer(layer_id)

• For a list of loaded layers and layer ids, use: • QgsMapLayerRegistry.instance().mapLayers()

• You can retrieve information about the fields associated with a vector layer by calling pendingFields() on a QgsVectorLayer instance:

• for field in layer.pendingFields():

• print field.name(), field.typeName()

Angelo

Furno 30/05/2016

Operations on layer features

• To select features for a given layer, you can call • setSelectedFeatures()

• passing the list of features IDs: • # Get the active layer (must be a vector layer)

• layer = iface.activeLayer()

• # Get the first feature from the layer

• feature = layer.getFeatures().next()

• # Add this features to the selected list

• layer.setSelectedFeatures([feature.id()])

• To clear the selection, just pass an empty list: • layer.setSelectedFeatures([])

• You can retrieve an iterator to explore features: • iter = layer.getFeatures()

• You can retrieve properties from features: • geom = feature.geometry()

• print "Feature ID %d: " % feature.id()

• # fetch attributes

• attrs = feature.attributes()

Angelo

Furno 30/05/2016

Operations on layer features (2)

• Attributes can be referred to by their name. • print feature[‘name’]

• Alternatively, attributes can be referred to by index. For example, to get the first attribute:

• print feature[0]

• If you only need selected features, you can use • selection = layer.selectedFeatures()

• print len(selection)

• for feature in selection:

• # do whatever you need with the feature

• Another option is the Processing features() method: • import processing

• features = processing.features(layer)

• for feature in features:

• # do whatever you need with the feature

• By default, this will iterate over all the features in the layer, in case there is no selection, or over the selected features otherwise

Angelo

Furno 30/05/2016

Rendering Vector Layers

• The appearance of the data in a layer is defined by the renderer and

the symbols associated with it

• Symbols are classes in charge of drawing visual representation of features

• Renderers determine how symbols will be used to represent a particular feature

• The renderer for a given layer can obtained as shown below: • renderer = layer.rendererV2()

• Several renderer types are available in QGIS core library: • print QgsRendererV2Registry.instance().renderersList()

Type Class Description

singleSymbol QgsSingleSymbolRendererV2 Renders all features with

the same symbol

categorizedSymbol QgsCategorizedSymbolRendererV2 Renders features with a

different symbol for each

category

graduatedSymbol QgsGraduatedSymbolRendererV2 Renders features with a

different symbol for each

range of values