If You Want to Have your Database Sing on Key, Then You Need to Get It Tuned Bryan Dickerson,...
-
Upload
elizabeth-norton -
Category
Documents
-
view
225 -
download
1
Transcript of If You Want to Have your Database Sing on Key, Then You Need to Get It Tuned Bryan Dickerson,...
If You Want to Have your Database Sing on Key, Then You Need to Get It Tuned
Bryan Dickerson, Woolpert
Jim Moening, ESRI
Database Tuning Water/Wastewater DB Design Conference February 2005
Agenda
How Does ArcSDE Work?
Database Tuning (Oracle and SQL)
Best Practices for Geodatabase Design
ArcSDE Management Roles & Responsibilities
Database Tuning Water/Wastewater DB Design Conference February 2005
How Does ArcSDE Work?
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Data Storage
A feature class (layer) is comprised of 3 tables in an ArcSDE database
Business table
Feature table
Spatial index table
Database Tuning Water/Wastewater DB Design Conference February 2005
Business Table
Contains attributes and a spatial column
Spatial column is a key used to relate to the feature and spatial index tables
SHAPESHAPE OBJECTIDOBJECTID NAMENAME ……
11 11 Main StMain St ……
FIDFID …… AREAAREA LENLEN POINTSPOINTS
11 …… 00 926.45926.45
SP_FIDSP_FID GXGX GYGY EMINXEMINX EMINYEMINY EMAXXEMAXX EMAXYEMAXY
11 77 1212 786786 840840 339339 374374
Business table Business table (Roads)(Roads)
Feature table Feature table (F1)(F1)
Spatial IndexSpatial Indextabletable(S1)(S1)
Database Tuning Water/Wastewater DB Design Conference February 2005
Feature Table
Stores geometry including annotation and CAD featuresAlso stores attributes describing the geometry such as Area and Length
SHAPESHAPE OBJECTIDOBJECTID NAMENAME ……
11 11 Main StMain St ……
FIDFID …… AREAAREA LENLEN POINTSPOINTS
11 …… 00 926.45926.45
SP_FIDSP_FID GXGX GYGY EMINXEMINX EMINYEMINY EMAXXEMAXX EMAXYEMAXY
11 77 1212 786786 840840 339339 374374
Business table Business table (Roads)(Roads)
Feature table Feature table (F1)(F1)
Spatial IndexSpatial Indextabletable(S1)(S1)
Database Tuning Water/Wastewater DB Design Conference February 2005
Spatial Index Table
Defines the grid range for each feature
Defines the shape envelope for each feature
SHAPESHAPE OBJECTIDOBJECTID NAMENAME ……
11 11 Main StMain St ……
FIDFID …… AREAAREA LENLEN POINTSPOINTS
11 …… 00 926.45926.45
SP_FIDSP_FID GXGX GYGY EMINXEMINX EMINYEMINY EMAXXEMAXX EMAXYEMAXY
11 77 1212 786786 840840 339339 374374
Business table Business table (Roads)(Roads)
Feature table Feature table (F1)(F1)
Spatial IndexSpatial Indextabletable(S1)(S1)
Database Tuning Water/Wastewater DB Design Conference February 2005
FID POINTS
CoordinateCoordinateInformationInformation
1
2
3
. . . . . xy,xy,xy
. . . . .
. . . . .
xy,xy,xy
xy,xy,xy
ArcSDE Storage Model
23
1
CENSUS_TRACTSCENSUS_TRACTS
SHAPE
SpatialSpatialcolumncolumn
1
Business tableBusiness table
2
3
Feature table stores geometryFeature table stores geometry
Spatial index table stores gridSpatial index table stores grid tiles and envelopes tiles and envelopes
Tables join on integer FeatureTables join on integer Feature ID column ID column
S34S34
SP_FID
2
1
GX GY
11
1 1
21
1
12
1 13
21
12
2
3
2
2
1
21
F34F34
Database Tuning Water/Wastewater DB Design Conference February 2005
Spatial index
Spatial index is used to improve performance of data retrieval
IBM DB2, Oracle, and SQL Server use grid tilesGoal
Optimize the size of the grid tilesBalance
Number of features referenced by each tile (features per tiles)Number of tiles referencing each feature (tiles per feature)
Informix and Oracle Spatial use R-tree indexing R-tree index constructs the index using the feature envelope
Do not need to add any parameters for building R-tree (built automatically)
Database Tuning Water/Wastewater DB Design Conference February 2005
Reducing Work With a Spatial Index
Clients use a spatial filter to reduce query results and speed up data retrieval
Example: Only fetch visible features for display
ArcSDE uses spatial index to reduce I/OEliminates need for full table scanSame principle as RDBMS column index
Spatial filter
Database Tuning Water/Wastewater DB Design Conference February 2005
Spatial Index
Spatial index uses the envelope (bounding box) of the features for fast searching
Envelopes for area and line features use min x,y and max x,y Envelopes for area and line features use min x,y and max x,y coordinates. The envelope for a point is the actual x,y coordinate.coordinates. The envelope for a point is the actual x,y coordinate.
49,25
53,28
49.5,53.6
38,68
32,64
Database Tuning Water/Wastewater DB Design Conference February 2005
Grid tilesGrid tilesTwo numbers in indexed columnsTwo numbers in indexed columns
Shape envelopeShape envelopeTwo coordinates (four numbers) in indexed columnsTwo coordinates (four numbers) in indexed columns
ShapeShapeMany coordinates in binary columnMany coordinates in binary column
Grid Tile Spatial Index Components
7373 7474 7575 7676 7777
4848
ParcelsParcelsExampleExample
4949
A
Database Tuning Water/Wastewater DB Design Conference February 2005
How the Grid Tile Spatial Index Works
1.
Goal: Avoid reading shapes for entire layer (full table scan)Four stages of elimination
2.
3.
4.
FILTER (SELECTION SHAPE) LAYER
tile tile
envelope envelope
shape envelope
shape shape
RDBMSRDBMS
ArcSDE server ArcSDE server or clientor client
Database Tuning Water/Wastewater DB Design Conference February 2005
Selecting a Spatial Index Size
Size tiles based on average feature envelope
Many rows take longer to searchMany rows take longer to search
Many tiles referencing one feature inefficientMany tiles referencing one feature inefficient
Balance tiles per feature against feature per tile
Grid tile size too smallGrid tile size too small S-table is too bigS-table is too big Many tiles referencingMany tiles referencing one feature one feature
Grid 3x feature envelopeGrid 3x feature envelope S-table smallerS-table smaller Fewer tiles referencing oneFewer tiles referencing one feature feature
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Command Line Tools Are Your Friend
sdelayer -o si_stats -l Parcels,shape -i esri_sde -s jayhawk -D demo
ArcSDE 8.2 Build 967 Thu Feb 28 22:31:11 PST 2002Layer Administration Utility-----------------------------------------------------Layer 6 Spatial Index Statistics:Level 1, Grid Size 1326.14|-------------------------------------------------------------------|| Grid Records: 6438 || Feature Records: 5099 || Grids/Feature Ratio: 1.26 || Avg. Features per Grid: 55.03 || Max. Features per Grid: 181 || % of Features Wholly Inside 1 Grid: 81.64 ||-------------------------------------------------------------------|| Spatial Index Record Count By Group || Grids: <=4 >4 >10 >25 >50 >100 >250 >500 ||---------- ------ ------ ------ ------ ------ ------ ------ ------ || Features: 5073 26 8 0 0 0 0 0 || % Total: 99% 1% 0% 0% 0% 0% 0% 0%||-------------------------------------------------------------------|
Database Tuning Water/Wastewater DB Design Conference February 2005
Spatial Index Tile Size Considerations
Optimal size varies by average feature envelopeRecalculate index if feature class changes
Most feature envelopes should not exceed grid size
S-table becomes too largeFeatures are referenced by many tiles which is inefficient
Use multiple resolutions (multiple grids) for feature classes with highly-variable feature sizes
Database Tuning Water/Wastewater DB Design Conference February 2005
Making Data Editable - Register As Versioned
Performed by data owner in ArcCatalog
Two tables are added to feature classAdditions tableDeletions table
Database Tuning Water/Wastewater DB Design Conference February 2005
How Tables React to Edits
Business tableBusiness table
SHAPE ID Name
12
RomaPalm
Deletes tableDeletes tableAdds tableAdds table
SHAPE NameID
Citrus3IDID
InsertingInserting
Adds tableAdds table
SHAPE NameID
Citrus3
Deletes tableDeletes table
ID2
DeletingDeleting
UpdatingUpdatingAdds tableAdds table
SHAPE NameID
Citrus31
Deletes tableDeletes table
ID2
Roma 1
Database Tuning Water/Wastewater DB Design Conference February 2005
Raster Data Storage
Raster data is stored in a similar fashion to vector data
4 tables are used to reference a raster feature classBusiness tableRaster bands table (stores bands for each image)Raster auxiliary table (stores metadata for each raster band)Raster blocks table (stores the actual raster in a tiled format)
Blocks table is most important as it is by far the largest
Database Tuning Water/Wastewater DB Design Conference February 2005
Pyramids
Pyramids can be built for a raster layer when loaded into ArcSDEMultiple resolutions of the raster layer are stored in the databaseAt a zoomed out scale the lower resolution is displayedAs the user zooms in to the highest level of detail, the raster as it was originally loaded (its highest resolution) is shownThis significantly helps to speed display of raster layers in ArcGIS and ArcIMS
Database Tuning Water/Wastewater DB Design Conference February 2005
Raster Blocks Table (SDE_BLK_1)
Raster is divided into tiles containing many pixels and each tile is a row in the table
Each tile at each pyramid level has a row in the table
This table WILL get very large!
RASTERBAND_ID RRD_FACTOR ROW_NBR COL_NBR BLOCK_DATA
1 0 1 2
Database Tuning Water/Wastewater DB Design Conference February 2005
Database Tuning for Oracle and SQL
Database Tuning Water/Wastewater DB Design Conference February 2005
Query Response Time
When the client submits a query, the server must:
Analyze query statementFetch dataTransfer results to client over network
2 biggest performance factors are network and disk I/O
Network I/O is database independentScanning large tables increases disk I/OFetching and transmitting large results increases disk and network I/O
Database Tuning Water/Wastewater DB Design Conference February 2005
Network I/O
T1 line connecting a remote office to an ArcSDE database in a headquarters office can be tied up with one ArcGIS desktop performing normal operations
Solution to this is terminal emulation software such as Citrix or Terminal Server
Database Tuning Water/Wastewater DB Design Conference February 2005
SQL Server Basics
Terminology is different than that for OracleA SQL Server instance is made up of many databasesSQL Server databases are a logical container for objects such as:
UsersRolesTablesEtc…
Database Tuning Water/Wastewater DB Design Conference February 2005
SQL Server Database
A SQL Server database is comprised of data files and log files
A database has, at minimum, one data file and one log file
DatabaseDatabase
DataDatafilefile
LogLogfilefile
Database Tuning Water/Wastewater DB Design Conference February 2005
File groupFile group
SQL Server File Groups
There can be multiple data files and log files and these can be stored in different places
A group of data files and/or log files is called a file group
DatabaseDatabase
DataDatafilefile
LogLogfilefile
LogLogfilefile
DataDatafilefile
Database Tuning Water/Wastewater DB Design Conference February 2005
Oracle Basics
Terminology is different than that for SQL ServerAn Oracle database is made up of many table spacesA table space can be made up of many data files. Also need to be aware of redo log files and control filesTable spaces are logical containers for objects such as:
TablesStored procedures
Roles and users are stored at the database level and not the table space level
Database Tuning Water/Wastewater DB Design Conference February 2005
DBTune File/Table
Actually exists as a table in the SDE database or table spaceContains storage parameters for tables and feature classesWhenever a new feature class is created and data is loaded, the ArcSDE server process reads the DBTune table for information on how to store these new objectsThere can be different parameters for different layers or data types and these are specified via keywords
Database Tuning Water/Wastewater DB Design Conference February 2005
Storing Layers in the SDE Database (SQL) or Table Space (Oracle)
Do not store layers in the SDE database or table space
Leave the SDE database or table space for ArcSDE application use (managing what layers are where, versioning, etc.)
Otherwise this can significantly slow down data retrieval as this database or table space will become large
Database Tuning Water/Wastewater DB Design Conference February 2005
Multiple Database Logins
Do not have all your editors in your organization use the same loginEach database user has a log table in the SDE database or table space that tracks a user’s actionsIf all users use one login, all their actions will write to one table and this table will grow very large very fastDoing this also prohibits from restricting user access on a layer by layer basis
Database Tuning Water/Wastewater DB Design Conference February 2005
Disk I/O Contention
This is the number one way to slow down an ArcSDE database
As users view or query data remember that, at minimum, ArcSDE is working through the business tables, feature tables, and spatial index tables for each layer in the map
If the layers are versioned, ArcSDE is also working through the adds and deletes tables
Database Tuning Water/Wastewater DB Design Conference February 2005
Preventing Disk I/O Contention
Use many disks and spread the tables and logs across those disks
In SQL Server, make use of file groups and spread the data and log files across different physical disks
In Oracle, make use of different table spaces and store the different tables (business, feature, and index) in the different table spaces on the different disks
Database Tuning Water/Wastewater DB Design Conference February 2005
Example SQL Server Configuration
Operating SystemOperating SystemPagefilePagefile
ArcSDE SoftwareArcSDE Software
Feature table file groupFeature table file group
SQL Server SoftwareSQL Server SoftwareSystem databaseSystem database
SDE databaseSDE databaseBusiness table file groupBusiness table file group
SDE Transaction logSDE Transaction logRaster table file groupRaster table file groupSpatial index file groupSpatial index file group
Disk 1
Disk 2
Disk 3
Disk 4
Database Tuning Water/Wastewater DB Design Conference February 2005
Example Oracle Configuration
Operating SystemOperating SystemPagefile/Swap spacePagefile/Swap space
ArcSDE SoftwareArcSDE Software
Feature table spaceFeature table space
Oracle SoftwareOracle SoftwareBusiness table spaceBusiness table spacePrimary control filesPrimary control files
Redo logRedo logRaster table spaceRaster table space
Spatial index table spaceSpatial index table spaceSecondary control filesSecondary control files
Disk 1
Disk 2
Disk 3
Disk 4
Database Tuning Water/Wastewater DB Design Conference February 2005
Disk I/O - RAID
RAID = Redundant Array of Independent (or Inexpensive) Disks
Different levels of RAIDRAID 0 = Striping. Data is striped across drives. Offers good performance but no fault tolerance.
RAID 1 = Mirroring. Data is written or mirrored to a second disk. Simple fault tolerance and performs better than a single disk.
RAID 5 = Striping w/ parity. Data is striped and parity calculations are distributed among disks. Provides good READ performance and excellent fault tolerance.
RAID 0 + 1 = Mirroring and striping. Excellent read/write performance and good fault tolerance.
Database Tuning Water/Wastewater DB Design Conference February 2005
Other Things You Can Do…
Reduce amount of data processed for faster queries
Provide a filter to limit number of rows
Index tables and layers to reduce searchingIf you know users will search or query on particular fields, index those fields ahead of time.
ArcSDE is fastest for small, indexed queries!!!
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Performance Tuning
Optimize Workflow•Data Loading•Data Maintenance •Reconcile/Post/Compress
Collect Performance Stats•Establish baseline•Use for performance validation
Database Design•Cost of complex data models
•While Editing•Recursive Relationships•Feature linked annotation•Types of labels
Index Management•Regularly rebuild indexes
Monitor Table Statistics•Adds and Deletes tables•SDE.state_lineages, states
Database Configuration•Keep it simple•Increase default cache sizes
Editing Operations•Use of edit cache is CRITICAL
Versioning Methodology•Reconcile performance costs•Using versioning to model history or alternatives•Reconcile/post/compress procedures
A good DBA is ESSENTIAL!
Database Tuning Water/Wastewater DB Design Conference February 2005
Tools for Managing ArcSDE Databases
Database vendor provided toolsSQL Server Enterprise ManagerSQL Server Query AnalyzerOracle DBA Studio
ESRI provided toolsArcCatalogArcSDE command line tools
Other toolsStorageVisitron (available from ArcScripts)Version Manager ArcObjects sample (Developer Kit)
Database Tuning Water/Wastewater DB Design Conference February 2005
Excellent Instructor-led Classes
ArcSDE Administration for SQL Server
ArcSDE Administration for Oracle
System Architecture Design for GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices for Geodatabase Design
Custom features
Geodatabase
**ArcSDE Geodatabase ArcSDE Geodatabase (Enterprise)(Enterprise)
Seamless Data Structure
Disconnected Editing*
Versioning*
Scalability*Multi-User Editing*
Integrity: Domains, Subtypes,
Relationship classes
Topology
Database Tuning Water/Wastewater DB Design Conference February 2005
support.esri.com – Project Center
Database Tuning Water/Wastewater DB Design Conference February 2005
support.esri.com – Project Center (cont.)
Database Tuning Water/Wastewater DB Design Conference February 2005
Database Design
Database Tuning Water/Wastewater DB Design Conference February 2005
Data Design Planning
Database Tuning Water/Wastewater DB Design Conference February 2005
Geodatabase
Database Tuning Water/Wastewater DB Design Conference February 2005
Data Models Tips and Tricks
Database Tuning Water/Wastewater DB Design Conference February 2005
Steps to designing geodatabases
Conceptual Design
Identity the information products to be produced with your GIS.
Identify the key thematic layers based on your information requirements.
Specify the scale ranges and spatial representations for each thematic layer.
Group representations into datasets.
Database Tuning Water/Wastewater DB Design Conference February 2005
Steps to designing geodatabases
Logical Design
Define the tabular database structure and behavior for descriptive attributes.
Define the spatial properties of your datasets.
Propose a geodatabase design.
Database Tuning Water/Wastewater DB Design Conference February 2005
Steps to designing geodatabases
Physical Design
Implement, prototype, review, and refine your design.
Design work flows for building and maintaining each layer.
Document your design using appropriate methods.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices
The schema design phase of your geodatabase implementation is critical to the ultimate performance of your GIS system.
Geodatabases can contain feature datasets, feature classes, relationship classes, geometric networks, topologies, feature-linked annotation, and other specialized abstractions.
Careful consideration of the cost, limitations, and true purpose of each of these abstractions is required to avoid creating a geodatabase that may not perform well in either a single- or multi-user environment.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Feature Datasets
A feature dataset is a container for feature classes that share the same spatial reference, along with relationship classes, geometric networks, and topologies.
Feature classes edited together in ArcMap can be grouped into feature datasets
Feature classes not typically edited together should be segregated into separate feature datasets or geodatabases
Feature classes that participate in a geometric network or topology must be grouped in the same feature dataset.
Feature classes can participate in no more than one topology OR geometric network.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Subtypes
Subtypes let you apply a classification system within a feature class and apply behavior through rules. Subtypes are essential to good design because they help reduce the number of feature classes and improve performance.
Let you control every configurable behavior:Attribute rules
Relationship rules
Network Connectivity Rules
Topology Rules
Reduce the number of feature classes by creating collections of features within a single feature class
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Relationship Classes
Use relationship classes for referential integrity persisted in the geodatabase, on-the-fly relates for editing performance, and joins for labeling and symbology.
Messaging occurs between related objects when querying or editing these objects. This incurs overhead in the application. Composite relationship require more messaging, and thus more overhead, than simple relationships.
Relationships can only be defined between objects in the same geodatabase.
Still requires a join for SQL query, labeling, or symbology.
When modeling spatial relationships, you will gain efficiency in your data creation and maintenance procedures by first thinking about how to use geodatabase topologies, geometric networks, shared-edge editing, and geospatial operators in your data model before resorting to relationship classes.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Geometric Networks
A geometric network is a system of edges (lines) and junctions (points) that convey objects. They are a mechanism to represent connectivity between edges and junctions where they connect.
Geometric networks provide fast network tracing capabilities and maintain connectivity during editing.
The maintenance of the logical network adds overhead to editing and version management.
The greater number of feature classes participating in a network, the greater amount of overhead will be incurred in maintaining the logical network and connectivity during editing.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Topologies
A geodatabase topology lets you accurately model geometric relationships between features. Topology can be considered a special type of relationship among features. Your data model will be more efficient if you can associate features by topology rather than with a relationship class. But remember that a feature class cannot participate in a topology and a geometric network at the same time.
Topologies manage a set of rules and errors associated with the violation of the rules.
A geodatabase topology can incur editing overhead and work flow considerations.
The greater number of feature classes and rules in a topology, then greater amount of overhead for the management of the rules and errors within a topology.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Annotation and Labeling
Choosing whether to cartographically edit or automatically place text is an important decision. It depends on the quality of source data, how much effort is to be expended, and the desired map quality.
Feature-linked annotation Simple annotation Labels
Definition Annotation class joined in composite relationship with feature class. Standalone annotation class.
Text properties, such as font and size, and a labeling field, defined in an ArcMap
layer.
Framework Geodatabase data model Geodatabase data model ArcMap document or layer property
Typical uses Fine cartographic placement of annotation with attribute-based text.
Fine cartographic placement of annotation with fixed text.
When text values are attribute values that can be placed automatically on the
map.
Composite objects Yes No No
Referential integrity Yes No No
ProsThe composite relationship lets you update,
move, or delete annotation when features are edited.
Simple annotation lets you place fixed text with a high degree of control.
Labeling features is a quick and easy way to add descriptive text to your map.
Cons Adds editing overhead; requires more work than labeling.
Requires more work than labeling; text values cannot be updated
automatically if attributes change.
While labels will not collide with each other, cartographic placement is rough.
RecommendationsUse feature-linked annotation for text, such
as street names, building numbers, and gages.
Use simple annotation for static text or text describing large or indeterminate
areas.
Use labeling for quickly placing descriptive text, such as addresses or
measurements.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Design Tips
Build on your existing GIS designsMost existing designs are suitable for moving forward. You can build on what has worked in the past and find new capabilities than can improve efficiency.
Use generic geodatabase types when feasibleCombining generic data structures with rich GIS tools provide the best solutions to scale and support multiple users and applications.
Leverage the ArcGIS software logic as much as possible and only use customized GIS data structures as a last resort.
Integrate independent feature classes using topologyMany users will find opportunities to use topologies in the geodatabase. This can create significant increases in efficiency for customization, data maintenance, and user productivity.
Database Tuning Water/Wastewater DB Design Conference February 2005
Best Practices – Design Tips
Combine GIS design concepts with traditional relational database design methods
Both RDBMS and GIS design methodologies are critical for good GIS design. One is not sufficient without the other. Learn to use and apply both techniques.
Prototype and pilot your geodatabase designPrototyping a design using personal geodatabases, ArcMap, and ArcCatalog is a very effective option for creating an efficient design process.
During the final stages of design, you will want to test scalability and work flows that represent the work your organization will perform. Use this activity to make final adjustments to your design as necessary.
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Management Roles & Responsibilities
RDBMSRDBMS
ArcSDEArcSDEArcSDEArcSDE
Database Tuning Water/Wastewater DB Design Conference February 2005
support.esri.com - Project Center
Database Tuning Water/Wastewater DB Design Conference February 2005
support.esri.com - Project Center (cont.)
Database Tuning Water/Wastewater DB Design Conference February 2005
Project Center - Skill Development
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Management Roles & Responsibilities
The following discussion outlines the major roles and responsibilities involved in deploying your spatial data and GIS applications throughout your enterprise.
Two Broad Categories for ArcSDE Roles & Responsibilities:
Administration
Development
Database Tuning Water/Wastewater DB Design Conference February 2005
Administration Tasks
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Service Management
Responsibility: Manage the ArcSDE service including:
Starting and stopping the service
Establishing a recovery process, if the service stops
Establishing a troubleshooting process
Remedy an ArcSDE service that will not restart
Fail-over
Requirements to do the Job:Must have Administrative or Power User rights on the service host
Must have access to the Windows Event Log
Must have Administrative access to the DBMS to diagnose database changes or problems
Must have knowledge of IT policies and methodologies
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Server Load Management
Responsibilities: Monitor and remedy server load
Monitor server load or automate the monitoring of the server load
Create framework for alerts
Diagnose which processes and operations are overloading the server
Remedy unnecessary load, like a runaway process
Build plan(s) to overcome overloaded server
Potentially build fail-over infrastructure of mission critical server
Requirements to do the Job: Access to system monitoring tools (e.g. PerfMon, Task Manager)
Must have Administrative or Power User rights on the host
Must have knowledge of IT policies and methodologies
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Service Anomaly Management
Responsibility: Manage the ArcSDE service if unforeseen troubles occur
Requirements to do the Job:Must have Administrative or Power User access to the service hostMust have access to the Windows Event Log to diagnose problemsMust have access to a technical support group to diagnose and remedy problemsMust have Administrative access to the DBMS to diagnose database problemsMust have knowledge of IT policies and methodologiesMust be able to troubleshoot ArcSDE service anomaliesMust be able to understand ArcSDE error logs to help resolve problems
Owner: IT or GISNotes:
Timely support required and is part of the criteria for choosing the staff
Database Tuning Water/Wastewater DB Design Conference February 2005
Spatial Database Access
Responsibilities: Manage access to the spatial database in ArcSDE, including:
Create new users to a spatial databaseMaintain administration userCreate a procedure to support ArcSDE/RDBMS standard security (i.e. ArcSDE does not support integrated security) Prepare and enforce procedure to manage ArcSDE clients (e.g. individual logins/roles)Lock down command line administration tools
Requirements to do the Job:Must have administrative access to RDBMS to create loginsMust have administrative access to RDBMS spatial databasesMust have the ability to control the use of the ArcSDE admin tools (e.g. sdemon)Must know how to enforce RDBMS data accessMust have knowledge of IT policies and methodologies
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Spatial Database Recovery
Responsibility: Establish recovery policy and procedures including:
Frequency (incremental, differential)Archive locationVersioningRecovery medium (i.e. backup vs. replication)
Enforce recovery policy and proceduresAutomate recovery procedure where practical
Requirements to do the Job:Must have administrative access to SQL Server to create and maintain recovery procedureMust know how to create and enforce SQL Server recovery proceduresMust have knowledge of ArcSDE layer schema to recover correct database objectsMust have knowledge of IT policies and methodologies
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Replicate Spatial Data
Responsibilities: Configure ArcSDE replication facilities (e.g. f and s tables)
Establish appropriate replication for business needs (e.g. transactional, snapshot, merge)
Implement replication needs
Requirements to do the Job:Must have administrative access to RDBMS to create and maintain replication configurations
Must know how to create and enforce RDBMS replication
Must have knowledge of ArcSDE layer schema to replicate correct database objects
Must have knowledge of IT policies and methodologies
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Tune ArcSDE Layers
Responsibilities: Configure a spatial index for a given layerMeasure and monitor the performance of the spatial searches critical to the business needs of the organization (i.e. data maintenance, custom products, general products)Maintain performance and Quality of Service as new spatial applications arrive, data and maintenance grows, and the user base increases
Requirements to do the Job:Must know how to adjust a spatial index of an ArcSDE layerMust know how to measure the performance of a spatial search in a given applicationMust know Quality of Service requirements
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Tune Server and Network Infrastructure
Responsibilities: Configure the ArcSDE instance parameters for the ArcSDE server
Measure and monitor the performance of the ArcSDE server instance, and the anticipated needs (i.e. data maintenance, custom products, general products, usage growth)
maintain performance and quality of service as new users and applications arrive, data and maintenance grows, and the user base increases
Requirements to do the Job:Must know Quality of Service requirements
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
ArcSDE Upgrade Management
Responsibilities: Ensure ArcSDE service version is up to date, and supports existing and new application functionality Ensure ArcSDE compatibility with applications (i.e. MapObjects, ArcIMS, ArcGIS)Install and configure upgrades to ArcSDE service and its application clientsSupport and remedy incompatibility issuesImplement ArcSDE upgrades (service packs, sdeupgrade)Install and configure upgrades of ArcSDE dependencies (e.g. DBMS, access libraries, operating system)
Requirements to do the Job:Must be able to monitor and maintain the versions of ArcSDE and the applications it supportsMust be able to monitor and maintain the versions of ArcSDE dependent technologies (e.g. operating system, DBMS, etc.)Must have administrative privileges on ArcSDE serverMust have administrative privileges on ArcSDE DBMSMust have knowledge of IT policies and methodologies
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Research Improvements in Data Management
Responsibilities: Continually investigate how to improve spatial data managementContinually test and scrutinize new spatial data management tools, techniques, and technologiesInvestigate new data management techniques (e.g. DBMS abstract data types)Investigate new spatial analysis to improve overall the organizations effectiveness
Requirements to do the Job:Must have knowledge of common GIS tools, techniques, and modern technologiesMust have knowledge of common spatial data maintenance issues and requirements Must have knowledge of common spatial data access issues and requirementsMust have knowledge of GIS services and practicesMust have a willingness to try new things
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Development Tasks
Database Tuning Water/Wastewater DB Design Conference February 2005
Develop Tools and Workflow to Maintain Changes to Spatial Data
Responsibilities: Translate related changes in business data to changes in spatial data (e.g. city annexed 2 new parcels)Use the right GIS tool and approach to maintain the spatial dataAutomate data maintenance where practicalSupport automation (tools or workflow)Integrate new GIS tools and techniques to maintain the spatial data
Requirements to do the Job:Must have knowledge of the functional capabilities of ESRI client applicationsMust have knowledge of current software engineering policies and methodologiesMust have knowledge of GIS policies and methodologiesMust have knowledge of GIS discipline (e.g. coordinate systems, topology)Must have knowledge of spatial editing issues (e.g. topology, spatial processing)
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Develop Custom GIS Solutions
Responsibilities: Understand and translate specific business need to GIS analysis perspectiveIdentify the right tool and spatial data to deliver that business needDevelop spatial solution to satisfy that business needTest spatial solution Deliver or deploy spatial solution
Requirements to do the Job:Must have the ability to translate general business needs to spatial processing requirementsMust have knowledge of the functional capabilities of ESRI client applicationsMust have knowledge of current software engineering policies and methodologiesMust have knowledge of GIS policies and methodologiesMust have knowledge of GIS discipline (e.g. coordinate systems, topology)
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Develop General GIS Applications
Responsibilities: Develop GIS application for general business needs (e.g. “show all the hydrants inspected this year on a map”)Test GIS application for general business needsDeploy GIS application for general business needsMaintain GIS application for general business needs
Requirements to do the Job:Must have knowledge of the functional abilities and limitations in ESRI client applications (e.g. ArcObjects)Must have knowledge of current software engineering policies and methodologiesMust have knowledge of GIS policies and methodologiesMust have knowledge of GIS discipline (e.g. coordinate systems, topology)Must have knowledge of IT policies and methodologiesMust know how to deploy applications
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Support Deployed GIS Applications
Responsibilities: Support users with technical issues about GIS applications (e.g. “help desk”)
Maintain the data sources for GIS applications
Troubleshoot functional issues of a GIS application
Support web based spatial applications (if web based mapping services apply)
Requirements to do the Job:Must have the ability to install and uninstall applications and their dependencies
Must have knowledge of IT policies and methodologies
Owner: IT or GIS
Database Tuning Water/Wastewater DB Design Conference February 2005
Spatial Data Design
Responsibility: Determine the appropriate layout of spatial data layers, including:
Organize by entity type (e.g. points, lines, polygons)Organize by functional analysis (e.g. SQL queries)Organize by normalization principlesOrganize by maintenance (i.e. minimize maintenance effort)Organize by spatial data vendors (e.g. Assessor changes, CAD structure)
Requirements to do the Job:Must know spatial data maintenance workflow in detailMust know how the spatial data will be used (i.e. map functionality and products)Must know basic principles of normalization and database objectsMust know ArcSDE layer schema
Owner: IT or GIS
Thank you!
Questions?