Tony Rees and Glenelg SmithDivisional Data Centre + Remote Sensing Facility

CSIRO Marine Research, Australia

(Tony.Rees@csiro.au)

Application of c-squares spatial indexing to an archive of

remotely sensed data

The starting point …

• Archive of c.60,000 satellite AVHRR scenes, 1986-current, accumulating at another 15+ per day

• Region covered varies for each scene

• Metadata (satellite, orbit, time parameters) and computed polygon (e.g. 7 points per side) available for each scene

• Determining which scenes include any given point, on-the-fly (to answer user’s request) is a standard requirement, but not trivial – available methods may be slow, or inaccurate, or both...

example in more detail ...

• Example – may need to query which out of these four scenes includes point “x” (in reality, stop/start points of scenes also vary, also other scenes cross at different angle/s)

x

Options for spatial indexing

Bounding polygons

• Require a GIS system (or similar) for spatial queries (polygon overlays); can be slow to test many thousands of scenes

• Can be problems with footprints which cross the date line, or include a pole

Bounding rectangles

• Computationally cheap, rapid to query, no GIS required

• Typically a poor fit to actual data footprints (typically these are not square, or even nearly so) – many “false positives”

• Can be problems with footprints which cross the date line, or include a pole

Gridded (tile based) representation

• Potentially simple to query, no GIS required

• May be poor fit to actual data footprints if tile size is large (smaller tiles best, however index gets larger – tradeoff required here)

• Compromise approach – slightly inexact compared with polygon overlays, but should be better performance for real-time queries (most computation has been done in advance)

• Problems with footprints which cross the date line, or include a pole, can be pre-handled

C-squares basics

• Based on a tiled (gridded) representation of the earth’s surface, at choice of 10 x 10, 5 x 5, 1 x 1, 0.5 x 0.5 degree tiles, etc. (0.5 x 0.5 degree tiles are used in this example)

• Every tile has a unique, hierarchical alphanumeric ID (incorporates the ID of all parent, grandparent, etc. in every child tile ID)

• Dataset (=scene) extents are represented by a list of all the tiles included in, or intersected by the footprint

• Spatial search comprises looking for one or more tile IDs in the set associated with any dataset (= simple text search).

(more details – see www.marine.csiro.au/csquares/ )

C-squares nomenclature – ten degree squares

(follows WMO 10 x 10º square nomenclature)

Adaptive square size – 10 x 10, 5 x 5, 1 x 1, 0.5 x 0.5 degrees (etc.) – produces quadtree-like efficiencies (“compaction”)

(this example shows detail to 1-

degree square level)

10 degree square: 3213:***

5 degree square: 3214:1**

1 degree square: 3315:100

- can reduce storage requirement by up to

90%, cf. constant-resolution encoding

A real-world exampleNOAA-12 .. 21 Jun 2003 06:23

10 x 10 degree squares (28)(base level of hierarchy, cannot compact)5 x 5 degree squares (99)

= 36 after “compaction”

1 x 1 degree squares (1,982) = 515 after comp.

0.5 x 0.5 degree squares (7,691) = 704 after comp.

0.5 x 0.5 degree squares - detail

Metadata table concept – 2 “search” tables

only

Scene details (1 row per scene)

Scene <> c-square pairs (600-1200 rows per scene)

Clickable map

interface

Example search result

SQL for spatial search(example for 0.5 degree search square)

select distinct A.scene_id, B.satellite, B.scene_date_time, B.image_location from satdata_csq_all A, scene_info B where (

(sqrsize = 0.5 and (A.csquare = search_csq -- e.g. 3414:100:1 (0.5 degree square)

or A.csquare = substr(search_csq,1,8)|| ':*' -- 1 level of compression

or A.csquare = substr(search_csq,1,6)|| '**:*' -- 2 levels of compression

or A.csquare = substr(search_csq,1,4)||':***:*') -- 3 levels of compression

) -- (other supported search square size options inserted here) ) and (startdate is null or B.scene_date_time >= startdate) and (enddate is null or B.scene_date_time <= enddate) and (sat = 'any' or B.satellite = sat) and A.scene_id = B.scene_id order by B.scene_date_time, B.satellite;

Summary of steps involved

• (1) Encoding – using custom polygon-to-c-squares algorithm (first attempt: 3+ mins per scene; second attempt: 30 seconds per scene to encode)

• (2) Storage – first attempt: all scene/c-square pairs in one large table, second attempt: partitioned by 10 x 10º “bins” (100 smaller tables, for faster searching)

• (3) Querying – using standard SQL and text matches

• (4) (optional) Visualising – generating footprints, using web call to multi-purpose c-squares mapper

More information: c-squares website - http://www.marine.csiro.au/csquares/

CMR Satellite Data spatial search:http://www.marine.csiro.au/remotesensing/csq-chooser.htm