Post on 29-May-2020
UAV Flight Operations andData Accuracy
Precision. Accuracy. Reliability
Part One: MappingWhy is Mapping different?Flight Operations and PlanningIntegration of Ground Control
Part Two: AccuracyNSSDA and ASPRS standards Translation to NMA Accuracy Assessments
AGENDA
Part 1MAPPING WITH DRONES
Precision. Accuracy. Reliability
OVERVIEW
Get Legal:Section 333 or Section 107, Insurance
OVERVIEW
Get Legal:Section 333 or Section 107, Insurance
Get Equipment: Rotocopter or Fixed wing?RTK? Camera? LiDAR?
OVERVIEW
Get Legal:Section 333 or Section 107, Insurance
Get Equipment: Rotocopter or Fixed wing?RTK? Camera? LiDAR?
Get Proficient:Takeoff and Land and…Crash
OVERVIEW
Get Legal:Section 333 or Section 107, Insurance
Get Equipment: Rotocopter or Fixed wing?RTK? Camera? LiDAR?
Get Proficient:Takeoff and Land and…Crash
Get Tools:Plan, Control, ProcessAutonomous flight
Sensors and Platforms
“ITS THE DATA, STUPID”
• Most vendors focus on the promise of automation.
“ITS THE DATA, STUPID”
• Most vendors focus on the promise of automation.
“ITS THE DATA, STUPID”
THERE IS NO EASY BUTTON
https://www.altavian.com/2017/01/09/one-button-survey-fallacy/
PURPOSE
PURPOSE
Imagery
Video
Orthos
Point Cloud*
DSM
DEM
Planimetrics
Analytics
3D Objects
Point Cloud*
Obliques
PURPOSE
Imagery
Video
Orthos
Point Cloud*
DSM
DEM
Planimetrics
Analytics
3D Objects
Point Cloud*
Obliques
What is this?
ERROR
ERROR
ERROR
ERROR
ERROR
ERROR
• Horizontal and Vertical Error translate to
VOLUMETRIC ERROR- even when within specs
• Change detection can only be done within the
accuracy limits of the sensor.
• Fly the same site twice in the same day with
the same control. Check the difference.
DATA QUALITY TRIANGLE
DATA QUALITY
Ensure data quality by adapting existing
standards and best practices
PRECISION
PRECISION
PRECISION
QUALITY
ACCURACY
ACCURACY
ACCURACY
ACCURACY
ACCURACY
ACCURACY
ACCURACY
REPEATABLITY
REPEATABLITY
REPEATABLITY
Flight Planning & Operations
FLIGHT PLANNING
FLIGHT PLANNING
Flight Lines to the edge of the AOI
Flight Lines beyond the AOI
FLIGHT PLANNING
FLIGHT PLANNING
FLIGHT PLANNING
Control Points
FLIGHT PLANNING
Check Points
FLIGHT PLANNING
GCP Planning
• No matter how you fly, GCPs control the accuracy
• 5-10 points per site
• Re-use existing known control when possible
• The GCP layout and the flight plan done together.
• Put GCPs in overlap areas and isolated areas
• GCP persistence-Adapt your plan over time
• Augment in areas of low confidence or obscured
GCP Planning
• Use an appropriate target
• Include KMZ/ASCII/Shape
• Need to be visible, recoverable, numbered.
• Be Practical!
Photo. Scale Thickness of Leg (T) Length of Legs (L)
1:1800 6 Inches(150mm) 3 Feet(0.9m)
1:2400 6 Inches(150mm) 3 Feet(0.9m)
1:3000 6 Inches(150mm) 4 Feet(1.2m)
1:3600 6 Inches(150mm) 4 Feet(1.2m)
1:4200 6 Inches(150mm) 5 Feet(1.5m)
Data Processing
ProcessProcess
Collect
Deliver
UAV MAPPING OVERVIEW
Imagery
Video
Orthos
Point Cloud*
DSM
DEM
Planimetrics
Analytics
3D Objects
Point Cloud*
Obliques
Level 0 Level 1 Level 2 Level 3 Level 4
UAV MAPPING OVERVIEW
Level 0 Level 1 Level 2 Level 3 Level 4
UAV MAPPING OVERVIEW
Raw ImagesGPS LogCamera Orientation
Level 0 Level 1 Level 2 Level 3 Level 4
UAV MAPPING OVERVIEW
Raw ImagesGPS LogCamera Orientation
Automated ProcessingQC ChecksFlight Validation
Level 0 Level 1 Level 2 Level 3 Level 4
UAV MAPPING OVERVIEW
Raw ImagesGPS LogCamera Orientation
Automated ProcessingQC ChecksFlight Validation
Ground ControlOrtho- controlRTK -in air
Level 0 Level 1 Level 2 Level 3 Level 4
UAV MAPPING OVERVIEW
Raw ImagesGPS LogCamera Orientation
Automated ProcessingQC ChecksFlight Validation
Ground ControlOrtho- controlRTK -in air
Mapping Products
Level 0 Level 1 Level 2 Level 3 Level 4
UAV MAPPING OVERVIEW
Raw ImagesGPS LogCamera Orientation
Automated ProcessingQC ChecksFlight Validation
Ground ControlOrtho- controlRTK -in air
Mapping Products
Analysis Products
Level Description What Do you Get? Data Quality TriangleLevel 0 Flight Operations
Raw DataGPS Log FileFlight Line LogOutline of AreaGeotag Imagery
Repeatability
Level 1 Automated Processing(cloud based services)
Image mosaicPoint cloudInitial Surface Model (DSM)3D Mesh
Precision
Level 2 Ground Controlled Survey GradeIntegrated control with known accuracy
Accuracy
Level 3 Topographic Data Point Cloud ClassificationBare Earth Elevation modelsSurface Constraints/BreaklinesDSM/DEM
Stitching multiple Flights Manual Tie PointsQuality Control
Level 4 Analytics/Derivatives ContoursPlanimetricsVolumesChange detectionHabitat mapping DrainageLand Cover/Vegetation/ImperviousFeature Extraction
WHAT YOU DON’T WANT TO SEE
PART 3- The Results
• Screen Shots of Tepper Quad Case Study
WHAT YOU DO WANT TO SEE
Part 2ACCURACY
Precision. Accuracy. Reliability
NMAS and the NSSDANMAS- 1947
Paper Map produced at a certain scale Not appropriate to use for digital data “2 foot accuracy” and “1 foot accuracy” terms no
longer apply, but are still WIDELY used
Standards
NSSDA and NMASNMAS- 1947
Paper Map produced at a certain scale Not appropriate to use for digital data “2 foot accuracy” and “1 foot accuracy” terms
are still WIDELY used
NSSDA – 1998 Applies to raster, point or vector data Must compare your data against a “reference” Reference must be 3x more accurate RMSE vs 95% Accuracy
Standards
RMSE vs 95% Accuracy in NSSDARMSE is a measure of the data itself- the ERROR measured
in the datai.e., RMSEz = 0.312 ft
95% confidence is a measure of selected data points against a target, or “tested to meet” standardi.e., 95% of the measurements will be within a certain threshold
of a “reference” or “true” valueNote: 5% of the measurements may not meet the target
threshold
Accuracy(x) = 1.7308 * RMSE(z)
Accuracy(z) = 1.9600 * RMSE(z)
Standards
“Produced to meet…”
Vs
“Tested to meet…”
2010- FEMA Policy MemorandumAligns FEMA Appendix A with USGS Lidar Guidelines and
Base Specifications v 13
Defines terms DEM, DSM, DTM
Calls out NSSDA as the standard for Accuracy Reporting, supplemented by NDEP 2004 and ASPRS 2004 for vertical accuracy.
“All of these standards, designed for digital elevation data, replace the National Map Accuracy Standard (NMAS)”
Standards
Horizontal Standards
Horizontal Standards
UAV
Converting from NMAS to ASPRS 2014 Standards:
Given a map or orthoimagery with a horizontal accuracy of RMSEx = RMSEy = 15 cmaccording to the new 2014 standard, what is the equivalent accuracy andmap scale according to the legacy National Map Accuracy Standard (NMAS)?
Exercise Problems
Horizontal Standards
Converting from NMAS to ASPRS 2014 Standards:
Given a map or orthoimagery with a horizontal accuracy of RMSEx = RMSEy = 15 cmaccording to the new 2014 standard, what is the equivalent accuracy andmap scale according to the legacy National Map Accuracy Standard (NMAS)?
1:380 or 1”=32’
Exercise Problems
Converting from NMAS to ASPRS 2014 Standards:
Given mapping requirements for creating planimetric data at1:400 scale:
A) What is the lowest resolution imagery that can be used?
B) What is the horizontal accuracy requirement for the orthoimagery?
Exercise Problems
Horizontal Standards
Converting from NMAS to ASPRS 2014 Standards:
Given mapping requirements for creating planimetric data at1:400 scale:
A) What is the lowest resolution imagery that can be used?
15cm or 6”B) What is the horizontal accuracy requirement for the orthoimagery?
-same-
Exercise Problems
Converting from NMAS to ASPRS 2014 Standards:
Given mapping requirements for creating planimetric data at1:400 scale:
A) What is the lowest resolution imagery that can be used?
15cm or 6”B) What is the horizontal accuracy requirement for the orthoimagery?
-same-What if you get 6” imagery but the accuracy reported is +/-1ft?
Exercise Problems
Surface ModelingVertical Standards
Surface ModelingVertical Standards
Surface ModelingVertical Standards
Surface Modeling FEMA- NPS should be equal to or less than the DEM post spacing
(resolution) required
1-meter DEM for 1ft contours 2-meter DEM for 2ft contours 5-meter DEM for 5ft contours
0.7m NPS -> 2ppsm -> 1m DEM -> 1ft contours (QL 2) 1.4m NPS -> 0.5ppsm -> 2m DEM -> 2 ft contours (QL 3)
Note: Higher density data may still fail vertical RMSE testLower density data may meet higher accuracy tests for
discrete points.Accuracy is tested from a sample of points against the surface model,
regardless of the point density
Vertical Standards
Vertical Standards
Vertical Standards
Converting from NMAS to ASPRS 2014 Standards:
Given an elevation dataset with a vertical accuracy of RMSEz = 10 cmaccording to the new 2014 standard, what is the National Map Accuracy Standard (NMAS)? equivalent contour interval?
Vertical Standards
Converting from NMAS to ASPRS 2014 Standards:
Given an elevation dataset with a vertical accuracy of RMSEz = 10 cmaccording to the new 2014 standard, what is the equivalent contour intervalaccording to the legacy National Map Accuracy Standard (NMAS)?
32.9cm or 1ft
Converting from NMAS to ASPRS 2014 Standards:
Given the requirement for 6” contour elevation data, what is the minimum point spacing and/or point density?
Vertical Standards
Vertical Standards
Converting from NMAS to ASPRS 2014 Standards:
Given the requirement for 6” contour elevation data, what is the minimum point spacing and/or point density?
NPS = 35cm or ~1ftNPD = 8 ppsm
Measuring Accuracy
I have drone data that is 1” pixel resolution ortho and 6” contours.
How accurate is it?
Measuring Accuracy
I have drone data that is 1” pixel resolution ortho and 6” contours.
How accurate is it?
Who knows?....
Measuring Accuracy
“Produced to meet…”
Vs
“Tested to meet…”
“In no case shall orthoimagery or planimetric accuracy be based on less than 20 checkpoints.”
Vertical checkpoints on slopes <10% Checkpoints at intervals of at least 10% of the diagonal At least 20% of the points are located in each quadrant
>20%
Measuring Accuracy
NSSDA: minimum of 20 points NSSDA: 95% Accuracy means 1 point can fail FEMA and ASPRS: 20 per land cover type USGS: use land cover types >10% of area LiDAR Projects: Typically 100 points
Reference must be 3X target accuracy
For UAS collections- MINIMUM of 20 POINTS !!!
Meeting accuracy specs does not necessarily equate to having quality data
Measuring Accuracy
If I use ground control with +/- 0.1ft accuracy for each point on a UAV survey, And use the same to collect checkpoints across the project area, how do I report the resulting accuracy of the planimetrics and topo?
Measuring Accuracy
Horizontal Standards
If I use ground control with +/- 0.1ft accuracy for each point on a UAV survey, And use the same to collect checkpoints across the project area, how do I report the resulting accuracy of the planimetrics and topo?
If UAV imagery is ~1inch 2.5cm Accuracy Class
0.1ft 3cm
3cm Checkpoints = 3x target accuracy = 9cm Accuracy Class
10cm Accuracy Class = 1:400 Planimetrics, 1ft contours
TESTED TO MEET Accuracy is only as good as the checkpoints used to TEST.
If you want to prove 0.1ft Accuracy (3cm)then you need 1cm Accurate Checkpoints
Measuring Accuracy
“ITS THE DATA, STUPID”
THERE IS NO EASY BUTTON