Data managing and Data Exchange: Geodatabases

and Interoperability Jaime Nieves Isaac Medel Esri Spain

European User Conference Madrid 2011 esri

Session highlights

• An Overview of the Esri Geodatabase - The basics about databases in ArcGIS - Data model and object types

• Geodatabase Workflows & Editing - Geodatabase levels - Versioning, replication and historics

• Database Administration - Database creation and managing

• Data storage and connection - Connection approaches: Clients and databases - Spatial types and interoperability

Geodatabase basic concepts

Geodatabase

• Geographic Data Store - Scalable, multi-user

• Core ArcGIS Data model

• Transactional model for

GIS workflows

• COM Components for data accesibility

Dimensions

Parcels

Networks Survey

Addresses

Annotations Vectors

3D Objects

Topology

CAD Raster

Terrain

Attributes

Cartography

Data management approach

• Geodatabase built on a table structure - Spatial Data Types & SQL - Relational Integrity - Scalability, Reliability, Flexibility - Support for continous, large datasets

• Based on the simple feature model - Points, Lines, Polygons - Open Access

Data on Geodatabases

• Esri System Tables - Advanced Business Logic

• Consolidated at 10.0 in 6 single tables. - Improved stability & fast data access - Uses XML columns - Same structure for all GDBs

• Sharing of data model - Schema & Data to XML

XML SQL type

System

Tables

User D

ata

Features and Feature Classes

• Builds on the Relational Model

• A feature is a spatial object

• A feature is an instance of a feature class

• Extended the relational model - Geometry attribute types

A feature class is a table of rows, where each row has a geographic column

Additional geodatabase elements Parcel fabrics Terrain datasets Representations Locators

Feature Classes

Behavior Attribute defaults Connectivity rules Attribute domains Relationship rules Split/merge policy Topology rules

Feature dataset Spatial reference

Relationship classes

Topology

Geometric networks

Raster Datasets

Toolboxes Tool Model Script

Network datasets

Tables

Polygon Route Line Dimension Point Annotation

Geodatabase Elements

Complex data in the Geodatabase

• Multi-part features - Many parts, one record in the table

• 3D Objects - Vertical lines

- Multipatches

• Measurements - M-Coordinate

- Dimensions

• Cartographic representations

• Geodatabase attachments

Topanga Canyon

Sacramento

Placerville

1 3/8"

Complex data in the Geodatabase

• Elevation data - Dynamic TINs (Terrain Datasets).

- LiDAR datasets

• Raster data - Imagery and continuous data

- Mosaic Datasets - Image processing & filtering

- Raster catalogs - Raster files inside a table field

Complex behavior in the Geodatabase

• Validation Rules - Attribute, connectivity and

relationships

• Domains - Ranged or coded values

• Subtypes - Defined as the class level

• Relationship classes - Association between objects

on different classes

Complex behavior in the Geodatabase

• Geometric Networks - Connectivity rules

• Network datasets - Multi-modal networks - Turns, restrictions, costs

• Topology - Set of rules that define spatial behavior - Ensure data integrity

• Parcel fabrics

Behavior: Summary

• You can: - Control the default value and acceptable values for any attribute

(Domains)

- Partition the objects into like groups (Subtypes)

- Instantiate classes with predefined behavior (Dimensions and Annotation)

- Control the general and network relationships in which an object can participate (Relationship Classes)

• Out of the Box in ArcGIS! - Configurable, no programming required

Geodatabase workflows Geodatabase editing

Geodatabase levels

Microsoft Access

2 GB

Windows

Single User

Files

1 to 256 TB per table

Multi-platform

Single User

RDBMS

Variable

Variable

Multi-user

ArcGIS Desktop

File Personal

ArcGIS Server

Workgroup Enterprise

Desktop

Capacity Number of Users

ArcSDE

- Storage

- Capacity

- Platform

- Users editing

Geographic data management

ArcGIS Server

Workgroup Enterprise

Desktop

Capacity Number of users

ArcSDE

Esri technology for managing and accessing geographic data on Relational Database Systems

Geographic data management

ArcGIS Server

Workgroup Enterprise

Desktop

ArcSDE

SQL Server Express

10 Gb Max

Windows

4 / 1

SQL Server Express

10 Gb Max

Windows

10 / 10

DB2, Informix, Netezza Oracle, PostgreSQL & SQL Server

Unlimited

Multi-platform

Unlimited

RDBMS

Capacity

Platorm

Users/Editing

Capacity Number of users

Editing Geodatabases

• ArcGIS datasets in the geodatabase are editable - Modify building footprints in parcel management - Add water mains to a water network - Update land owners information stored in a table - Etc…

• Transaction model for editing in ArcGIS - Edits are performed in an edit session

- Open session – edit – save edits / don’t save edits

- A series of edit operations constitutes a transaction - The transaction is either committed or rolled back

Editing Geodatabases…

• Personal Geodatabases - Mainly single user editing on small datasets - Multiple readers - Editing locks at geodatabase level

- Two editors cannot edit within the same geodatabase at same time

• File Geodatabase - Mainly single user editing small to very large datasets - Multiple readers - Editing locks at the dataset level

- Multiple editors cannot edit the same table or stand-alone feature class at the same time

- Multiple editors cannot edit feature classes in the same feature dataset at the same time

Editing Geodatabases…

• Enterprise Geodatabases - Extend the transaction model with Versions

- Multiuser editing without locking

- Unique isolated view of the geodatabase

• Benefits of versioned editing - Multiple editors, editing over long periods of time

- Undo / Redo

- Archiving

- Replication

Versioning

• Technology that allows multiple users to edit and view data at the same time

- Appears to users as if they have their own copy of a table - Does not apply locks or duplicate data

Version1 Version2

Default

Editor 1 Editor 2

Readers

Workflow in a multi-user editing environment

Version creation

Connect to the version

Editing

Reconcile: ¿conflicts?

Conflict resolution

Remove & Compress

Post

Replicas

• Geodatabase replication extends the concept of versioning to multiple databases.

DFLT

VR1 VR2

DFLT

V1 V2

DFLT

ArcSDE GDB

ArcSDE GDB

File GDB

Two way

One way

Parent Child

ArcSDE GDB

ArcSDE GDB

DFLT

VR1 VR2

DFLT

V1 V2 File GDB

check out check in

Parent

Child

ArcSDE GDB

- Versions in parent and child geodatabases are used as replica versions.

- Changes in this replica versions are transferred during synchronization

Replication types

• Checkout/Check-in replication: - Edits in child replica can be syncronized one single time with

parent replica.

- Uses: Disconected editing; File GDB as a requisite.

• One-way Replication: - Allows data changes to be sent multiple times in a single

direction (parent-to-child or child-to-parent)

- Data in origin GDB are editable; data in destination GDB are read-only.

- Changes in destination GDB are overwritten during syncronization if edits in origin GDB exist.

- Uses: development-staging-production models; data publishing

• Two-way replication: - Allows data changes to be sent multiple times, both ways.

- If the same row is edited in both replica geodatabases, it is detected as a conflict when the replicas are syncronized.

Parent Child

One time

Edit Read

Production Development

Edit A Edit B

Office A Office B

Conflict resolution

SDE SDE

PGDB FGDB

SDE SDE (child-to-parent)

PGDB FGDB

Persist

Persist

SDE SDE

New workflows

• New replication scenarios: - File GDBs can be used as one-way and

check-in/check-out replicas.

- One-way child-to-parent replicas - Multiple sendings from child to parent - Editable data on child GDB; read-only on parent GDB.

- One-way replicas using archiving

- Improved performance and easier management compared to versioning.

SDE FGDB

Resolving syncronization conflicts

• Conflict resolving using Replica Manager - Enables user to decide the geometry and attributes to keep

in case of a conflict. - Uses a visual environment - Merge geometries if there is a conflict on different areas on the same Shape.

Historical versions

• Geodatabase archiving. - Built on Geodatabase versions. - Enables recording and accessing changes made to a subset of data. - No need for storing snapshots for the entire database. - Archiving lives as long as the Version is alive. - Introduces a historical version. - Creation of historical markers possible.

Geodatabase compress

• Maintain database performance and health. - Delete all states that do not participate within a version's lineage.

- Collapse any candidate lineage of states into one state.

- Move rows from the delta tables into the business tables.

DEFAULT

0

v2

v1

v4

Before

0

v1

v2

DEFAULT

v4

After

Database administration

Geodatabase Data Management

• Schema is defined in ArcCatalog - Define feature classes, datasets, relationships, etc - Catalog window in ArcMap @ 10

• Import and convert data from other formats - Shapefile - Coverage - CAD - Raster

• Copy and Paste

• ArcGIS.com

- Import basemaps and layers from galleries, groups…

• Use an esri Data Model or Downloadable Template - Industry specific data models available - Download a geodatabase template from the resource centers

http://resources.arcgis.com

Create an enterprise geodatabase

• Enabling the advanced capabilities of the geodatabase in: - IBM DB2 - Informix - Microsoft SQL Server - Oracle - PostgreSQL

• Enable Enterprise Geodatabase Tool. - Adds the geodatabase to an existing

database and authorizes it

• Create Enterprise Geodatabase Tool. - Creates database and geodatabase

in PostgreSQL and SQL Server

• Python script

Geodatabase administration in 10.1

• New Geodatabase administration dialog box - Manage versions - View and remove connections - View and manage locks - Block new connections

• For system tables: - Rebuild indexes - Update statistics

Managing data in ArcGIS 10.1

• View and manage privileges • View locks on a version

or specific dataset • Rebuild indexes on multiple datasets • Update database statistics

on multiple datasets • Enable editor tracking • One-click spatial table registering

Isaac Medel

Database administration in 10.1

Database storage and connection

From Esri Geodatabases To Esri Clients

ArcSDE ArcGIS Desktop

Direct connect the database

• Client-DBMS connection with an SDE service in the way.

• Process load on the client side, not the server side - Improved CPU & Memory usage on the server. - Higher scalability. - Backwards compatibility. - No need for ArcSDE service

administrattion. - Enables operating system authentication

for your ArcGIS Connections in Oracle geodatabases.

Geodatabase

AppServer

GIS client

Application Server (three-tier)

Direct connect driver

GIS client

Direct Connect (two-tier)

Database compatibility

Geodata

• Geodatabase access through a web connection.

• Exposes remote capabilities for: - Replication.

- Copy (data extraction).

- Query.

Local GDB

Geodata Service

2-way

Internet

LAN

File GDB

SQL Server Express

SQL Server Express

File GDB

Enterprise GDB Enterprise

GDB

Forward compatibility in 10.1

• Updates in 10.1 datasets: - Python scripting with annotation & dimensions - Editor tracking in tables & feature classes - Netwok datasets improvements - Improved mosaic datasets

• Creation of new datasets in 10.0 geodatabases - No upgrade required - Still accesible by 10.0 clients. - 10.0 SP2 recommended

10.0 10.1

Query Layers

• A layer or stand alone table defined by a SQL query

• Provides direct, read-only access to spatial data independent of where it is stored

• Data does not need to be registered with the geodatabase

• Full power of SQL to create feature classes and Tables: - Joins, Spatial Types, Aggregation, Field Derivation

• New tool in 10.1: - Make Query Layer

SELECT Name, City, ShapePoint FROM Hospitals

Isaac Medel

Query Layers

From Any Database To Esri client

Anywhere ArcGIS Desktop

Work with databases in ArcGIS 10.1

• Database connection from Desktop - Work with databases without enterprise geodatabases - Copy & paste between databases and geodatabases - Create & query tables and features in a database - Create views in a database - Publish data to ArcGIS Server

• Database connection dialog box simplified

- Less inputs required - Sharing connection files - Create database connection

geoprocessing tool

Isaac Medel

Connection to databases

From Any Database To Any Client

Anywhere Anything

Feature Services

• Web access to feature geometry • RESTful capabilities:

- Geometry & attribute query - Database editing - Access related records and non-spatial tables

Feature Service Mobile

Desktop

Web

Desktop / Database ArcGIS Server

URL

Web

Spatial Data Server (SDS)

• Low-memory web server. • Exposes geometries, attributes, symbols and templates

from vector data in a database or geodatabase. • Uses feature services to access spatial data. • Available for IIS andJava

Spatial Data

Server

Database

Isaac Medel

Spatial Data Server

From Esri Geodatabases To Any client

ArcSDE Anything

Third-party clients concurrent editing

• Non-versioned Multi-user editing - Table-to-table edition - Short transactions - Simple features

• Versioned and non-versioned editing on the same data.

- “Move edits to base tables” option - Edit from ArcGIS and third-party

applications concurrently. - DEFAULT version stores changes in base tables.

SDE FGDB

Database accessing from third-party clients

GDB APIs

C# Java

ArcSDE GDB Client

GDB developing environments

• Geodatabase ArcObjects: - Application-level logic. - Geometry behavior and integrity. - Programming: C++, Java, .Net (C#, Java), Python

• ArcSDE SDK: - A set of independent libraries for database accessing

from third-party clients. - Programming: C, Java

• Features: - Connections and transactions.

- Geometry operations

- Table management.

- Raster management.

- Change or create the Schema.

- SQL access to Esri’s spatial type.

- Versioning.

- XML storage and search.

File Geodatabase API

• No ArcObjects (no licence).

• Access to File GDB from third-party clients.

• C++ API which enables developers to: - Create, access and delete FGDBs. - Read FGDB Schema - Create Schemas for simple objects. - Read/write data. - Perform alphanumeric and spatial queries - Navigate through relationships. - Work with geodatabase attachments.

http://resources.arcgis.com/content/geodatabases/10.0/file-gdb-api

GDB APIs

C# Java

ArcSDE GDB Client

Database accessing from third-party clients

SQL

SFS (OGC) ISO 19125

ST_GEOMETRY

About ST_Geometry

• UDT available for storage in all databases.

• OGC * and ISO** definition: • SFS (Binary Geometry) 1.1, SFS (Types and Functions) 1.1

• Implemented as OGC Compliant in ArcGIS since 2006.

• Install the ST_Geometry storage type in Oracle or PostgreSQL

• Create Spatial Type tool

* OpenGIS Implementation Specification for Geographic information - Simple feature access - Part 2: SQL option ** ISO/IEC 13249-3 SQL multimedia and application packages - Part 3: Spatial ISO 19125-2:2004 Geographic information -- Simple feature access -- Part 2: SQL option

OGC and Standards

OGC

OGC Client

Non-GIS Client

OGC support in ArcGIS

Category Standards

Map WMS 1.0, WMS 1.1.1, WMS 1.3.0, WMtS 1.0.0, KML 2.1.0, KML 2.2.0, GML 2.1.2, GML 3.0

Feature WFS 1.0.0, WFS 1.1.0, GMLsf 1.1.0, WFS(T) 1.0.0

Metadata CAT CS/W 2.0.1, CAT 1.0, CAT 1.1.1, CAT 2.0.2, GeoRSS 1.0.0, CAT2 AP eRIM 1.0.0, CAT2 ISO AP 1.0.0, CAT2 ebRIM part2 1.0.0

SQL SFS(BG) 1.1, SFS(TF) 1.1

Raster WCS 1.0.0, WCS 1.1.0, WCS 1.1.1

Symbology SLD 1.0

Processing WPS 1.0.0

Encoding Filter 1.0, Filter 1.1

OGC

http://www.esri.com/library/whitepapers/pdfs/supported-ogc-iso-standards.pdf

Any client

OGC support in ArcGIS

ArcGIS supports IT standards

OGC Client

REST / HTTP

Javascript Flash Silverlight

Android iOS WP7 C++

.Net (C#, VB) Java Python

XML UNICODE SQL IPv6

Geocoding Editing Routing Querying

Network analysis Image processing

Spatial Analysis Spatial Statistics Windows

UNIX Linux

WMWare MS Hyper-V Amazon EC2 MS Azure

Isaac Medel

Data for everyone

Thank You