GeoNode Integration with GIS and Data Processing...

GeoNode Integration with GIS and Data Processing workflows

Salvador Bayarri sbayarrigmailcom

World Bank Consultant

Contents (1)

bull GeoServer components

bull Direct access to GeoServer

ndash Styling with SLD

ndash Adding data from external databases

bull Direct access to GeoNetwork

ndash Searching displaying and editing metadata catalog

bull Direct access to the Geonode spatial database

ndash Examples of DB analysis processes

ndash Projecting data in DB

Contents (2)

bull Access from desktop GIS ndash Connecting and displaying map and feature services

ndash Editing the DB

ndash Projecting data in GIS

ndash Access to metadata catalogs

ndash Exporting styles through SLD

ndash Georeferencing data

ndash Uploading mobile data

ndash Verifying and correcting topology

GeoNode stack

Application layer

Service layer

Data layer

GeoNetwork Web interface

GeoServer Web interface

Direct access to GeoServer bull GeoServer is the component that provides WMS WFS and WCS services

bull It can be used to upload and style data using its own Web front-end

bull However your GeoNode user wonrsquot be recognized by GeoServer you need to use a GeoServer user normally with administration privileges

Connect with your Web browser httpgeonodeURLgeoserver

GeoServer styling with SLD

bull It is easier to work with SLD styles directly within GeoServer

bull Click on the lsquoStylesrsquo link in the left-side menu and select the style you want to edit or add a new one


In the style editor you can bull Change the name of the style

bull Copy the SLD from an existing style

bull Change the code of the SLD

bull Upload an SLD file

bull Validate the SLD to detect errors

GeoServer new styles

bull If you create a new style you need to associate it with a layer

bull To do so select the desired layer in the Layers page


bull Select the lsquoPublishingrsquo tab in the layer properties

In the WMS Settings section you can choose the default style and add more style choices


bull The styling changes will show up in GeoNode (you may need to refresh the page)


bull Exercise

ndash Modify and add new styles for some of your vector layers

ndash Modify the style for the elevation layers

ndash See the changes in GeoNode

GeoServer adding data

bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options

bull Data sources are referred in GeoServer as stores

GeoServer vector data stores

bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo

We can access any data from the PostGIS DB using the same store

GeoServer geoTIFF stores

bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store


GeoServer adding external data

bull We can add other data sources for GeoNode by creating GeoServer stores

More raster formats


bull GeoServer has extensions to create stores from other database vendors

bull More information in

httpdocsgeoserverorglatestenuserwebadmindatastoreshtml


IMPORTANT NOTE

If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode

Unfortunately this updating does yet not work for cascaded WMS layers as we will see

geonode updatelayers


bull Exercise ndash Add a new store for the GTOPO30

elevation dataset named W200N40dem

ndash After the store is added click on Publish and accept the default

ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode

Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management

and CSW catalog search service

bull It can be used to search and display the GeoNode metadata

bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial

metadata and harvest metadata from other CSW servers

Connect with your Web browser httpgeonodeURLgeonetwork

Direct access to GeoNetwork

bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo

ndash Go to the Administration New metadata menu

ndash Select the Vector data ISO19139 template


bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo

ndash Scroll down and type a Web address as online resource

ndash Click on lsquoSave and Closersquo


bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo

ndash Set access privileges for All and Submit

ndash Go back and search for your new lsquodatarsquo link in GeoNode

Direct access to the GeoNode DB

bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata

bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)


bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB

Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy


Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools


bull We can create reports like a data dictionary report


bull We can view the data records and filter them


bull We can execute SQL queries to select insert or delete records

We can apply operators to perform calculations like average count standard deviation summation etc


bull Exercise ndash Use simple SQL queries to extract columns from some layers

SELECT some_columns

FROM some_data_source

WHERE some_condition

Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo

ndash Use Sum() and Count() operators with SELECT to aggregate results

For instance find out how many different polygons make each District and what is their total areas


There are also spatial operators that perform calculations on geometries

bull ST_GeometryType(geometry) returns the type of the geometry

bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry

bull ST_X(point) returns the X ordinate

bull ST_Y(point) returns the Y ordinate

bull ST_Length(linestring) returns the length of the linestring

bull ST_StartPoint(geometry) returns the first coordinate as a point

bull ST_EndPoint(geometry) returns the last coordinate as a point

bull ST_NPoints(geometry) returns the number of coordinates in the linestring

bull ST_Area(geometry) returns the area of the polygons

bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)

bull ST_ExteriorRing(polygon) returns the outer ring as a linestring

bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring

bull ST_Perimeter(geometry) returns the length of all the rings


And operators that generate a text description of the geometries in different formats or vice versa

bull ST_AsText(geometry) returns WKT text

bull ST_AsEWKT(geometry) returns EWKT text

bull ST_GeomFromWKB(bytea) returns geometry

bull ST_AsBinary(geometry) returns WKB bytea

bull ST_AsEWKB(geometry) returns EWKB bytea

bull ST_GeomFromGML(text) returns geometry

bull ST_AsGML(geometry) returns GML text

bull ST_GeomFromKML(text) returns geometry

bull ST_AsKML(geometry) returns KML text

bull ST_AsGeoJSON(geometry) returns JSON text

bull ST_AsSVG(geometry) returns SVG text

Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations

Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation

PostGIS provides the following geometric operators

bull ST_Contains(geometry A geometry B)

bull ST_Crosses(geometry A geometry B)

bull ST_Disjoint(geometry A geometry B)

bull ST_Distance(geometry A geometry B)

bull ST_DWithin(geometry A geometry B radius)

bull ST_Equals(geometry A geometry B)

bull ST_Intersects(geometry A geometry B)

bull ST_Overlaps(geometry A geometry B)

bull ST_Touches(geometry A geometry B)

bull ST_Within(geometry A geometry B)

Analysis with GeoNode DB spatial joins

The SQL JOIN operator allows us to add fields to a table by performing spatial tests

For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to

We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm


We can also create new attributes by aggregating the information of all the features in the joined dataset

Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)

For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset

GIS Analysis with GeoNode DB

bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in

which district or geological zone is an industry)

ndash Find out which biological corridors are crossed by a certain river

ndash You can get more ideas from

httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml

Saving your results

bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet

bull We can also use the results of a query to create a new table

bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values

CREATE TABLE table_name AS

SELECT hellip FROM hellip WHERE hellip

ALTER TABLE table_name ADD PRIMARY KEY (fid)

CREATE VIEW view_name AS


Saving your results

bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial

SELECT

Populate_Geometry_Columns(lsquotable_namersquoregclass)

Saving your results

bull Exercise ndash Create a spatial table by copying an existing one and adding some

columns

ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store

ndash Run the lsquoupdatelayersrsquo command

ndash Display the new layer in a map and use the new column to style it

CREATE TABLE districts_area AS

SELECT fid the_geom ST_Area(the_geom)

FROM districts

-- Define a primary key

ALTER TABLE districts_area ADD PRIMARY KEY (fid)

-- Spatially enable it

SELECT

Populate_Geometry_Columns(districts_arearegclass)

Saving your results

Projecting data with GeoNode DB

bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg

Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format

each SRID

bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)


bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests

ndash GIS clients can project data to the view SRID on-the-fly

bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)

bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)


bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as

bull Wersquoll see we can do the same as easily in a GIS

SELECT ST_Transform(the_geom DestinationSRID)

FROM original_table


bull With a query we can find all SRIDs based on UTM zone 16N


bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)

bull In this case differences are about 2 meters


bull To learn more about working with PostGIS

httpworkshopsopengeoorgpostgis-intro

httppostgisrefractionsnetdocumentation

manual-14ch04html

Direct access to GeoNode from GIS

bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode

bull From a GIS we can connect to GeoNodersquos

ndash PostGIS DB

ndash Map Service (WMS) published by GeoServer

ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer

ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet


GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end

(Editing geoprocessing)

File download

File upload


bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows

Access to Map Service

bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection


bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be

displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo


bull The selected layer will be displayed using images from the map server


bull We can identify features in the map (internally uses GetFeatureInfo request)

Access to Map Service from ArcGIS

bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool


bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw

bull However the Identify feature does not seem to work well

Access to Feature Service bull By accessing WFS services we can access data with full geometry

and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some

GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future

projects in the SettingsOptions dialog

Access to Feature Service

bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs


bull Click on lsquoConnectrsquo

bull The layers published in the GeoNode feature service will be displayed

bull Select one layer (belize_land_degradation)

bull Click on lsquoApplyrsquo and then lsquoClosersquo

Note that we didnrsquot change the Coordinate Reference System from the original

which is different from the current qGIS project view


bull The selected layer will be displayed using features downloaded from the server

bull We can style them and use in GIS analysis


bull We can also edit the WFS feature layer

bull Use the lsquoLayerToggle Editingrsquo command to enable editing


bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values


bull We can also edit attribute values in the table view and add or remove columns


bull If the GIS or Web editor supports WFS-T you can save changes directly to the server

bull Otherwise you can save the results to a file or to the PostGIS database

Access to Map and Feature Services

bull Exercise

Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)

Save the project

Access to Feature Service from ArcGIS

bull It is necessary to have the Data Interoperability extension installed (not a free add-on)

bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL


bull Click the lsquoParametersrsquo button and open the Feature Types selector

bull A list of the layers from the WFS server should appear () and you can select to import one or more

() Does not seem to work with the Geoserver WFS try the URL showed here


bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap


bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class

Access to PostGIS DB from GIS

bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source

bull We create a connection just like we did from pgAdmin

bull We need to use the password in etcgeonodelocal_settingspy

Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of

available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers

and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild

queryrsquo (see next slide)


bull With the query we select only features complying with some condition

Editing PostGIS DB from GIS

bull We can toggle editing and use the same tools as with WFS or other feature layers

bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected


bull Exercise

Upload in qGIS some feature layers from the PostGIS DB

Edit them

Save the results to the DB

Go to GeoNode and see that the layers and maps reflect the changes

Projecting data in GIS

In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS

In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data


Feature and raster projection is a common feature in other GIS like ArcGIS

Projecting data in GIS datum shift

bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo

bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations

bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters

bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)

() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL


bull Once you have the grid shift file in ArcGIS you can define a custom transformation

bull This will later be selected when projecting data


bull Most open-source software tools use the proj4 library which also supports shift grids

bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)

bull qGIS as well as PostGIS keep the proj4 configuration in a database table

More details in httpprojmaptoolsorggen_parmshtml

Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use

the NADCON utility from the USA National Geodetic Survey ()

bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP

bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be

26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON

() However after testing it seems the NADCON tool does not include the grids for the Caribbean region

Accessing PostGIS DB from ArcGIS

bull ArcGIS support PostGIS spatial databases via ArcSDE

bull It is necessary to install the ArcSDE PostgreSQL AddOn

bull In theory this will allow us to access the GeoNode spatial data from ArcGIS

Access to Catalog Service from GIS

bull The qGIS CSW Client plugin has been installed for the workshop

bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog

bull Create a new connection to the Geonode URL

httpgeonodeURLgeonetworksrvencsw

bull Click on lsquoServer Inforsquo to verify the connection


bull Now we can search for metadata records in the catalog

bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata

In the administration module we will see how to add the WMS linkage in the metadata

Access to Catalog Service from ArcGIS

bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from

httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122

bull New catalog connections can be created using the Geonetwork profiles


bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)

bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog

Using SLD styles from GIS

bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode

bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs

bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file

ndash Go to layer properties with the right-click menu

ndash Select the ldquoCategorizedrdquo type

ndash Create a color ramp

ndash Classify


bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file

bull Open the sld file in a text editor like gedit

bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences

bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported

Using SLD styles from GIS bull The GeoServer SLD uses a different

namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)


bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax

bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)


bull Exercise

Open the GeoServer style editor for the elevation data

Create a different elevation color ramp for the same node in qGIS

Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp


bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)

httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm

ndash Arc2Earth (Free Community Edition)

httpwwwarc2earthcomcommunityedition

Processing data in GIS

bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles

or original DB

ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data

bull Some typical processes are

ndash Importing data point with X-Y values in a table


ndash Limit the data to an area of interest

ndash Table and spatial joins

ndash Cleaning topology

bull Most of these can be done directly with PostGIS and SQL using spatial operators

Importing points from XY table

bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml


bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589

Limiting the data area

bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip

bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district

bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool


bull A new layer with the clipped features is now created

Georeferencing data table join

bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)

bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents

bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors

Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf

bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer


bull Exercise

Join two shapefiles by attribute

Load the first shapefile as usual

Load the DBF from the second shapefile

Join the tables by using a common attribute as explained


Raster georeferencing

bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference

bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis

bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data

bull Then select the Georeferencer tool in the raster menu


bull We upload a raster to the Georeferencer which displays it in its own window

bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data

Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS

Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)


When you click on the image a dialog will appear to enter the reference coordinates for that point

Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo

Enter at least other 3 reference points in the same way better if they are spread in different directions

You can remove points from the GPS table later on


bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options


bull A new GeoTIFF file with the CRS and coordinate information will be generated

bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct

More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf

httpglaikitorg20110327image-georeferencing-with-qgis


bull Exercise

Capture a view in Google Maps or any other map source

Save the capture to an image file

Georeference the image and upload it in GeoNode

Using GPS data

bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs

bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices

bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum

bull It is widely supported in many systems and software including the Open Street Map project

Using GPS data

bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device

bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog

Using GPS data

bull Exercise

Import some of the sample GPX files which may contain only points or tracks

Do not forget to enable the lsquoon the fly projectionrsquo

Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)

Using GPS data

bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before

bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode

bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project

Topology

bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)

bull These problems affect display and analysis results that some times may turn invalid or incorrect

bull qGIS provides some tools to

detect problems and change the offending geometries

Topology

bull Before or after we upload a vector dataset to GeoNode we can detect topology problems

bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer

bull It will detect some cases of line self-intersection and duplicated nodes

bull These can be corrected by editing

Topology

bull We can do the same for polygon data (eg land_degradation) where we find more serious problems

bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology

Topology with qGISGRASS

bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS

bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data

(Enable the GRASS plugin in lsquoPluginsManagersquo)


bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin


bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard

bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent

bull Then create a new Mapset that will be you personal workspace


bull After loading in qGIS the vector layer you want to clean up

bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool


bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)

bull Write a name for the lsquocleaned uprsquo version

bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)

bull Then click on this button to see in qGIS the imported vector data

bull Some problems may have already been fixed by the import

bull For instance intersecting polygons rings are separated into non-intersecting rings

Before

After


bull Once imported we can use additional GRASS topology clean up tools


bull Exercise

Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS

ndash Display cleaned Roads in qGIS

ndash Run the vcleanbreak tool on the imported version

ndash Display the result in qGIS and Save As a new shapefile

ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added


Reference documentation

httpqgisspatialthoughtscom201201

setting-up-working-grass-environment-inhtml

httpdocumentationqgisorghtmlituser_manual

grass_integrationgrass_integrationhtml

GRASS has many other analysis tools worth trying

It also has tools to upload GRASS data directly into PostGIS


References bull OpenGeo education center

httpworkshopsopengeoorg

bull Introduction to GIS using qGIS

httpwwwbaruchcunyedugeoportalpracticum

bull Quantum GIS documentation

httpwwwqgisorgendocumentationhtml

bull PostGIS documentation from its makers Refractions Research


Contents (1)

bull GeoServer components

bull Direct access to GeoServer

ndash Styling with SLD

ndash Adding data from external databases

bull Direct access to GeoNetwork

ndash Searching displaying and editing metadata catalog

bull Direct access to the Geonode spatial database

ndash Examples of DB analysis processes

ndash Projecting data in DB

Contents (2)









GeoNode stack

Application layer

Service layer

Data layer

























bull Exercise















More raster formats






IMPORTANT NOTE












































SELECT some_columns





























































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise























Contents (2)









GeoNode stack

Application layer

Service layer

Data layer

























bull Exercise















More raster formats






IMPORTANT NOTE












































SELECT some_columns





























































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise























GeoNode stack

Application layer

Service layer

Data layer

























bull Exercise















More raster formats






IMPORTANT NOTE












































SELECT some_columns





























































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise













































bull Exercise















More raster formats






IMPORTANT NOTE












































SELECT some_columns





























































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise


































More raster formats






IMPORTANT NOTE












































SELECT some_columns





























































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise




























IMPORTANT NOTE












































SELECT some_columns





























































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise































































SELECT some_columns





























































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise














































































Saving your results









Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise






























Saving your results


SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise
























SELECT


Saving your results


columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise
























columns






FROM districts




SELECT


Saving your results




each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise


























each SRID











FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise
































FROM original_table










manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise
































manual-14ch04html




ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise


























ndash PostGIS DB





GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise
























GeoServer

GeoNetwork

PostGIS DB

GeoNode

GIS

CSW

WFS WFS WCS

SLD

(Editing) WFS-T

Web front end


File download

File upload












































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise


































































bull Exercise


Save the project


























bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise
















































bull Exercise


Edit them


























26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise














































26716 (UTM 16N + NAD27)

4267 (GCS NAD27)

4152 (GCS NAD83)

26916 (UTM 16N + NAD83)

Project Project

NADCON






























ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise



















































ndash Classify












bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise


































bull Exercise











or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise






























or original DB


























bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise








































bull Exercise





























bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise














































bull Exercise




Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise























Using GPS data





Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise























Using GPS data



Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise























Using GPS data

bull Exercise




Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise























Using GPS data




Topology





Topology





Topology























Before

After




bull Exercise























Topology





Topology





Topology























Before

After




bull Exercise























Topology





Topology























Before

After




bull Exercise























Topology























Before

After




bull Exercise











































Before

After




bull Exercise

























bull Exercise























GeoNode Integration with GIS and Data Processing...

Documents

Transcript of GeoNode Integration with GIS and Data Processing...