Developing standards and automated production for Sentinel...
Transcript of Developing standards and automated production for Sentinel...
Page 1 of 56
CUSTOMER
DEFRA
Developing standards and automated production for Sentinel-2 Analysis
Ready Data
PROJECT
EODIP CCDP5 Sentinel-2 ARD
DATE
18/10/2017
STATUS
Final
VERSION NUMBER
V0.1
CLASSIFICATION
PREPARED BY OPEN
Tom Jones, Earth Observation Specialist (Catapult)
Dan Wicks, Senior Earth Observation Specialist (Catapult)
Simon Agass, Head of Geospatial Systems & Solutions
(Catapult)
Dr. Pete Bunting, Reader (Aberystwyth University)
REVIEWED BY
Mark Jarman, Head of Earth Observation (Catapult)
Richard Hilton, Head of Technology Solutions (Catapult)
Joana Kamenova, Space Innovation Facilitator (Catapult)
APPROVED BY
Dr Gwawr Jones, Earth Observation Specialist (JNCC) Satellite Applications Catapult Electron Building Fermi Avenue Harwell Didcot Oxfordshire OX11 0QR Tel: +44 (0) 1235 567999 Email: [email protected]
THIS DOCUMENT IS THE PROPERTY OF SATELLITE APPLICATIONS CATAPULT AND MUST NOT BE COPIED OR USED FOR ANY PURPOSE OTHER THAN THAT FOR WHICH IT HAS BEEN SUPPLIED.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 2 of 56
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 3 of 56
TABLE OF CONTENTS
TABLE OF CONTENTS ........................................................................................................................ 3
GLOSSARY ........................................................................................................................................ 5
FOREWARD ...................................................................................................................................... 8
PART 1: SENTINEL-2 ANALYSIS READY DATA PROJECT ........................................................................ 9
Introduction ............................................................................................................................................. 9
Background .............................................................................................................................................. 9
Sentinel-2 Context ................................................................................................................................. 11
Sentinel-2 ARD Project ........................................................................................................................... 12
Summary ................................................................................................................................................ 13
Development of a UK ARD and Data Cube Strategy........................................................................... 14
Sentinel-2 ARD - next steps for the project and beyond .................................................................... 15
Requirements for UK routine production of Analysis Ready Data ..................................................... 16
PART 2: SUPPORT FOR ESA SENTINEL-2 DATA WITHIN ARCSI ........................................................... 21
Key Points: ............................................................................................................................................. 21
Introduction ........................................................................................................................................... 21
Data Analysis .......................................................................................................................................... 22
Parsing Sentinel-2 Headers................................................................................................................. 22
At-sensor (top of atmosphere) radiance ............................................................................................ 23
At-sensor (top of atmosphere) reflectance ........................................................................................ 23
Surface Reflectance (SREF) ................................................................................................................. 23
Standardised Surface Reflectance ...................................................................................................... 24
Cloud Masking .................................................................................................................................... 25
Clear-Sky Masks .................................................................................................................................. 25
Topographic Shadow Masks ............................................................................................................... 26
Image Band Sharpening ...................................................................................................................... 27
Processing Multiple Granules ............................................................................................................. 27
Access to Sentinel-2 Imagery ............................................................................................................. 28
Results .................................................................................................................................................... 28
Quality of the Cloud Masking ............................................................................................................. 28
Quality of the Surface Reflectance Spectra ........................................................................................ 29
Image Pixel Comparison to ESA ARD Product ........................................................................................ 29
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 4 of 56
Standardised Reflectance ................................................................................................................... 34
Imagery Sharpening ............................................................................................................................ 34
ARCSI Recommendations & Conclusions ............................................................................................... 36
Recommendations.............................................................................................................................. 36
Conclusions ......................................................................................................................................... 37
APPENDIX A - SENTINEL-2 CHARACTERISTICS AND PROCESSING CONSIDERATIONS .......................... 38
APPENDIX B - SENTINEL-2 ARD INTERNATIONAL CONTEXT ............................................................... 46
APPENDIX C - A SUMMARY OF STANDARDS .................................................................................... 52
REFERENCES ................................................................................................................................... 54
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 5 of 56
GLOSSARY
Term Definition
AC Atmospheric correction
ACIX Atmospheric Correction Inter-Comparison Exercise
AGDC Australian Geoscience Data Cube
AOD Aerosol Optical Depth
AOT Aerosol Optical Thickness
API Application Programming Interface
ARCSI Atmospheric and Radiometric Correction of Satellite Imagery
ARD Analysis Ready Data
AWS Amazon Web Services
BEIS Department for Business, Energy and Industrial Strategy
BoA Bottom of Atmosphere
BRDF Bidirectional Scattering Distribution Function
CARD4L Committee on Earth Observation Satellites Analysis Ready Data for Land
Catapult Satellite Applications Catapult
CEDA Centre for Environmental Data Analysis
CEMS Climate, Environment and Monitoring from Space
CEOS Committee on Earth Observation Satellites
CESBIO Centre for the Study of the Biosphere from Space
CNES French National Centre for Space Studies
DC Data Cube
Defra Department for Environment, Food & Rural Affairs
DEM Digital Elevation Model
DfID Department for International Development
DIAS Data and Information Services
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 6 of 56
DOS Dark Object Subtraction
ECMWF European Centre for Medium-Range Weather Forecasts
EO Earth Observation
EO CoE Earth Observation Centre of Excellence
EODIP Earth Observation Data Integration Pilot
ESA European Space Agency
FCO Foreign and Commonwealth Office
FLAASH Fast Line-of sight Atmospheric Analysis of Spectral Hypercubes
GDP Gross Domestic Product
GEO Group on Earth Observations
GEOSS Group on Earth Observations (GEO) and Global Earth Observation System of Systems
GFZ German Research Centre for Geosciences
HMA Heterogeneous Missions Accessibility
HMG Her Majesty's Government
HPC High Performance Computing
IT Information Technology
JASMIN Joint Analysis System
JAXA Japan Aerospace Exploration Agency
LSI-VC Land Surface Imaging Virtual Constellation
MACCS Multi-sensor Atmospheric Correction and Cloud Screening
MOD Ministry of Defence
MODIS Moderate Resolution Imaging Spectroradiometer
MODTRAN MODerate resolution TRANsmission
MPI Message Passing Interface
NCEO National Centre for Earth Observation
NDVI Normalized Difference Vegetation Index
NERC Natural Environment Research Council
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 7 of 56
NIR Near Infrared
NSA United States National Security Agency
ODC Open Data Cube
OGC Open Geospatial Consortium
RBINS Royal Belgian Institute of Natural Sciences
SAFE Standard Archive Format for Europe
SEDAS Sentinel Data Access Service
SGP Space Growth Partnership
SREF Surface Reflectance
SSGP Space for Smarter Government Programme
STFC Science and Technology Facilities Council
SWIR Shortwave Infrared
THEIA French Land Data Centre
TIR Thermal Infrared
ToA Top of Atmosphere
UKSA UK Space Agency
USGS United States Geological Survey
UTM Universal Transverse Mercator
WGCV Working Group on Calibration and Validation
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Draft
Version: v0.1
Date: 18/10/2017
Page 8 of 56
FOREWARD
This report provides information about using the Atmospheric and Radiometric Correction of Satellite
Imagery (ARCSI) software to produce Sentinel-2 Analysis Ready Data (ARD). It further provides context
regarding the rationale for ARD and wider international activities relating to ARD.
The report consists of:
PART 1: Sentinel-2 ARD project. This section provides a summary of project rationale, an
overview of international ARD activities, commentary on the development of a UK ARD and Data
Cube (DC) strategy and requirements for UK routine production of ARD.
PART 2: Support for ESA Sentinel-2 data within ARCSI. This section demonstrates how the ARCSI
software can be used to produce ARD as a topographically corrected surface reflectance product
for Sentinel-2.
Appendices: This sectional provides supporting materials on Sentinel-2 characteristics and
processing considerations, Sentinel-2 ARD international context and a summary of standards.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 9 of 56
PART 1: SENTINEL-2 ANALYSIS READY DATA PROJECT
Introduction
The launch of the various Sentinel satellites by the European Space Agency (ESA) as part of the European
Commission’s Copernicus Programme1 is generating ever-increasing interest and therefore user
requirements across government and different market sectors. However, there remain barriers to
realising the potential use of this data as a routine input to value added service and product generation.
Technical complexities associated with the handling and manipulation of the data remain, alongside a
lack of standardisation of the form in which the data should be made available. To allow immediate
analysis of the data without additional significant user effort, these barriers need to be addressed.
Through a collaboration between the Satellite Applications Catapult (Catapult), Aberystwyth University,
and the Defra Earth Observation Centre of Excellence (EO CoE) with support from the UK Space Agency’s
(UKSA) Space for Smarter Government Programme (SSGP), the Sentinel-2 Analysis Ready Data (ARD)
project has created a series of recommendations designed to ensure that UK users across government
and industry can maximise routine exploitation of Sentinel-2 data (See Appendix A for detailed Sentinel-
2 characteristics and processing considerations). These recommendations focus on the requirements for
the processing of Sentinel-2 data to a new standard termed ARD, the creation of an open toolset for
implementing the generation of Sentinel-2 ARD products, and the generation of a Sentinel-2 ARD data
set for the UK covering the full Sentinel-2 acquisition campaign up to June 2017. This project ensures
continued UK contribution to wider international community efforts to develop standards and methods
of Earth Observation (EO) data access.
Background
The UK firmly recognises that the space industry must adapt to meet the ever-growing interest and user
requirements for EO data products and services. When considering the exponential growth of EO data
availability and wide range of potential users, it becomes quickly apparent that it is simply not technically
feasible or financially affordable to consider traditional methods of storing, handling and manipulating
EO data. Local processing and data distribution methods currently exploited by industry and government
are not suitable to address the challenge of scalability, increases in the size of data volumes, and the
growing complexities in preparation, handling, storage, and analysis required to meet user requirements.
To access the valuable information contained within EO data, users are required to undertake a series of
complex pre-processing steps to turn the data from a ‘raw’ unprocessed format into a state that can be
1 http://www.copernicus.eu/
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 10 of 56
analysed. Unless the user has the expertise, software and infrastructure to handle and process this
information, efficient exploitation of the data is not realised.
A solution to this, in part, is through the systematic and regular provision of ARD. This is defined as
“satellite data that have been processed to a minimum set of requirements and organised into a form
that allows immediate analysis without additional user effort and interoperability with other datasets
both through time and space”2.
The main steps in the production of ARD, include:
• Geometric correction (it is assumed that the Level-1 data provided by ESA is to a desirable
standard).
• Atmospheric correction
• Radiometric correction
The potential for regular provision of ARD is achievable given developments and advancements in
contemporary computing infrastructures, technologies, and data architectures that overcome data
management and analyses challenges that have arisen from increasing data volumes. One such example
is Data Cube (DC) technology, which considers a scalable architecture of spatially aligned time series stack
of pixels for ease of data interrogation and analysis. Such a solution has great potential to streamline
data distribution and management for providers while simultaneously lowering the technical barriers for
users to access and exploit EO data.
It is a critical time for the UK to contribute to development around ARD technologies given the work that
is actively being undertaken by several key organisations on the international stage (see further details
in Appendix B). The main protagonist is the Committee on Earth Observation Satellites (CEOS), which is
working to coordinate this activity, bringing together entities such as Geoscience Australia, ESA, Centre
National D’Etudes Spatiales (CNES), the National Aeronautics and Space Administration (NASA), the
United States Geological Survey (USGS) and the Japan Aerospace Exploration Agency (JAXA). CEOS is
developing a series of standards for ARD, branded ‘CEOS Analysis Ready Data for Land (CARD4L)’, through
an initiative primed by the CEOS Land Surface Imaging Virtual Constellation (LSI-VC). Directly supporting
this work is the CEOS Working Group on Calibration and Validation (WGCV) that is investigating issues of
data interoperability between ARD produced from different EO sensors. More broadly feeding into
generation of the standards and assessment of best practice is an international collaborative initiative to
inter-compare a set of atmospheric correction (AC) processors, led jointly by ESA and NASA. Both
organisations have a vested interest in progressing consolidation of an ARD standard due to a desire to
publish Sentinel and Landsat ARD retrospectively as part of their core data dissemination activities.
Despite limited UK involvement thus far on the international stage within the aforementioned groups,
there is significant activity being undertaken by UK government, academia, industry and the Catapult to
2https://www.google.co.uk/search?q=ceos+card4l&oq=ceos+card4l&aqs=chrome..69i57.3367j0j7&sourceid=chrome&ie=UTF-8#
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 11 of 56
consolidate requirements for ARD standards and investigate future EO data architecture approaches.
Most notable is the work being undertaken by the Defra EOCoE within their EO Data Integration Pilot
(EODIP) Programme. This includes ARD production for government users built upon initial work carried
out by the Welsh government and their data access hub, which makes available pre-processed Landsat
data and the work of the Catapult, contributing to the definition of ARD standards for Sentinel data. To
maintain the UK’s strong leadership position in space technology, UKSA, The National Centre for Earth
Observation (NCEO), Defra and the Catapult are working together to feed this activity into international
discussions and provide a stronger steer on future development activity.
Sentinel-2 Context
While the concept of ARD is directly applicable to all EO data, initial efforts have been focused on
increasing the exploitation of openly available free to access data sets given the increased user interest
driven by the launch of the Sentinel satellites. The Sentinel-2 dataset is prioritised due to its high
temporality, rich multi-spectral characteristics, effective continuation of the extensive Landsat record,
and familiarity to end users given the optical nature of data (see further details in Appendix A).
In terms of routine access, ESA has been providing a Top of Atmosphere (ToA) reflectance Sentinel-2
product since October 2015. To realise a consistent data set for exploitation, corrections for variation
resulting from atmosphere, including cloud cover masking, and terrain illumination must be implemented
on the end user side. To account for this requirement, ESA made available the Sen2Cor3 processor to
allow users to generate Sentinel-2 Bottom of Atmosphere (BoA) surface reflectance products.
Despite its open source nature, Sen2Cor still requires significant EO and IT expertise to use, and remains
a barrier to non-expert users. Furthermore, it is temperamental in terms of its installation and corrections
implemented are known to be limited and poorly geographically transferable. A variety of proprietary
alternatives exist; however, their Sentinel-2 correction capabilities remain publicly unproven through
wide-scale comparison exercises, provide only partial solutions, or become impractical for routine
implementation through automation.
Given the investments that ESA has made in relation to the Sentinel satellites, it has realised that to
enable mass exploitation and to remain relevant in an increasingly commercial industry, it must
disseminate datasets of greater accessibility. Since May 2017, ESA has been piloting the provision of
Level-2A (BoA) products over Europe. However, many unanswered questions remain regarding its
suitability for routine use, such as:
• The finality of the algorithm and its transferability
• Service levels associated with data provision, most notably relating to coverage and latency
3 http://step.esa.int/main/third-party-plugins-2/sen2cor/
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 12 of 56
• The responsibility for data dissemination, whether it is through the core Copernicus hub or part
of future services such as the ESA Data Information and Access Service (DIAS).
There is also uncertainty as to how ESA will position dissemination of this data against parallel services
being developed by organisations such as Amazon Web Services (AWS) and Google.
Sentinel-2 ARD Project
If the anticipated industry growth in the UK is to be realised, the community must quickly establish access
to reliable, operational data services for ARD. Demand from users familiar with the use and exploitation
of EO data is clear, driven by requirements for efficient, operative, scalable and commercial solutions.
However, more importantly there is increasing interest from organisations and markets outside of the
space sector who are keen to realise the benefit of EO data in new digital services and applications. As a
sector, we will be failing significantly if we do not embrace this demand. This project sought to support
the UK moving towards this ideal, although with much future discussion required as to where the
responsibility for hosting and financing a solution might lie.
Currently, a suitable tool for producing Sentinel-2 ARD is not openly available, nor a standard associated
with this product type. Catapult and Defra’s EO CoE identified an opportunity to fund a piece of work to
help facilitate UK exploitation of Sentinel-2, while simultaneously raising the profile of EO and UK
expertise in this field. The project leverages UK capability in processing and exploiting EO datasets to
develop a method for generating Sentinel-2 ARD products that meets the standards required by the UK
geospatial and EO community, both public and private, and is also well aligned with the proposed CEOS
CARD4L standards.
The project has made available an open source and readily accessible tool that delivers state-of-the-art
functionality in correcting optical satellite imagery; including atmospheric correction, terrain illumination
mitigation, novel band-sharpening correction, and neighbouring multi-image processing capabilities to
deliver a Sentinel-2 BoA surface reflectance product. This tool is an expansion of the Atmospheric and
Radiometric Correction of Satellite Imagery (ARCSI)4 software developed via leading research at
Aberystwyth University and provides an accessible means of atmospherically correcting optical satellite
imagery that aids automation (see further details in Part 2).
4 http://rsgislib.org/arcsi/
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 13 of 56
Identified advantages of using the ARCSI tool over ESA’s Sen2Cor processor include:
• The image products produced by ARCSI provide a comparable result to that currently being
piloted by ESA but without artefacts creating within the topographic correction.
• The innovative image band sharpening approach implemented with ARCSI provides a significant
improvement over traditional approaches of image resampling when using the 20-metre image
bands within a 10-metre resolution image stack.
• The ability to process multiple image granules together, thus avoiding boundary effects between
the image granules.
The support for Sentinel-2 within ARCSI provides a step forward towards realising Sentinel-2 ARD
provision, although further work is recommended, particularly on the approaches to cloud masking and
correcting for bidirectional reflectance.
Further to the processor development, an extensive UK Sentinel-2 ARD dataset has been made openly
available for users to test the suitability of and to explore the added value in exploiting Sentinel-2 data
sets in a format that is analysis ready. Dissemination of this ARD will be made through the Catapult’s
Sentinel Data Access Service (SEDAS)5 data portal, the user-facing UK Copernicus Collaborative Ground
Segment. All acquisitions over the UK have been processed that comprise more than 25% useable
imagery (not obscured by cloud) since commencement of operational acquisition in October 2015. If
further funding is identified, a more complete dataset can be made available.
Summary
Funding of the current project has developed a highly enthusiastic UK community around Sentinel-2 and
ARD, with 60 representatives from across Government, industry and academia attending two workshops.
Over 90 participants have also joined the online discussion via the project's LinkedIn group and this
includes international participation from Australia, France, India, Italy, Spain, the Netherlands, and the
United States.
It is generally accepted by the community that the requirement to have access to an ARD product sooner,
outweighs the need to achieve perfection in the definition of the product, although the provision of
suitable metrics for quality are considered of high importance.
The project has not only helped address standards questions related to Sentinel-2 ARD, but has allowed
the UK to constructively contribute to wider conversations in the international ARD sphere, positioning
the UK as a leading authority in EO exploitation. An open source tool for generating Sentinel-2 ARD is
now readily accessible, capturing UK satellite expertise, as well as being very well aligned with the latest
international ARD standards definition (CARD4L). An extensive dataset of Sentinel-2 ARD has been
generated for the UK and made readily available to the UK community. This dataset shall further
5 https://geobrowser.satapps.org/
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 14 of 56
demonstrate the value offering of EO, as well as amplifying UK conversations around the necessary
routine provision of ARD suitable for UK end users; no such provision currently exists.
The rationale for an ARD data service is clear and the user demand strong. The contribution of space and
space-enabled services to UK GDP continues to grow at a significantly higher rate than the rest of the
economy6. ARD would only further this growth by accelerating the development of new applications and
services through improved routine access to EO data. It would enable the delivery of solutions with
greater efficiency, at reduced cost and allow effective embedding of EO within new markets.
Furthermore, government increasingly recognises the value of EO, evidenced by Defra EO CoE’s
commitment to integrate “EO into at least five Defra policy areas to ensure cheaper, more effective, more
customer-focused delivery by 2020”7. Despite these positive steps, the UK has some decisions to make
about how to champion these concepts in the long-term and whether it should take more of a leadership
role on the international stage.
Development of a UK ARD and Data Cube Strategy
Successful implementation of ARD and Data Cube (DC) technologies within the UK geospatial ecosystem
will only be achieved with clear direction, ownership and alignment with wider UK and international
programmes. In the very short-term, a UK lead needs to be established to drive work in this area and to
define a coherent, implementable strategy that reflects the interests of stakeholders across government,
academia and industry. This strategy should tell a strong user story, consolidating evidence collected
through programmes such SSGP8 and the Space Growth Partnership (SGP)9. It is recommended that the
leading entity should ideally come from a government agency such as the UKSA or from the Catapult,
either of which can exploit their neutral trusted status to appropriately represent the interests of all
parties. With clear governance and strategy, the UK EO community will be well positioned to deliver
services and products to both new and existing users at increased efficiency and value.
The UK needs to firmly establish its engagement with the wider international community, notably
organisations such as CEOS and initiatives such as the Open Data Cube (ODC). With an active presence,
the UK can not only draw on the experience of others, but also influence the future direction of travel for
major EO data developments. Through nominated UK leads from the UKSA, the Catapult and the UK
Science community, the UK can contribute to critical working groups such as LSI-VC and WGCV.
Engagement has started with these groups. However, the UK needs to guarantee resource availability to
ensure continued active involvement within key conversations and meetings that are rapidly shaping
developments around ARD and DC technologies. Through this engagement, the UK has an exciting
6 https://sa.catapult.org.uk/wp-content/uploads/2016/04/C222628-VISION-2030-Design-Stg8.pdf
7 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/488133/defra-earth-obs-roadmap-2015.pdf
8 http://www.spaceforsmartergovernment.uk/
9 http://spaceigs.org/
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 15 of 56
opportunity to develop partnerships, increase its credibility and develop an exportable model for ARD
generation and exploitation.
To consolidate its position, the UK community needs to address the following fundamental questions:
• What are the major technical requirements for UK ARD?
• What needs to exist to maximize the exploitation of ARD in the UK?
• How does the UK need to structure itself to best take advantage of ARD?
It is the consensus of stakeholders engaged through this project that the Sentinel-2 ARD data
demonstrated meets requirements for a minimum viable product. There is also overwhelming support
for tailored UK ARD data making use of, for example, high resolution Digital Elevation Models (DEMs)
that are available. The highest prioritised technical requirement is for transparency and ease of
repeatability of processing method. Regarding exploitation, stakeholders agreed that the key
requirement was for ease of access. Stakeholders strongly advocated for free access to ARD products as
an entry threshold but that higher level products such as mosaics, cloud free composites and spectral
indices (e.g. NDVI – Normalised Different Vegetation Index) products are also of high priority. It is
generally perceived that UK Government and public funding should anchor the costs for long term
provision but that private sector co-investment is also encouraged. A UK ARD standard supported by UK
Government gives confidence to industry and ensures sustainability of the technical solution.
Sentinel-2 ARD - next steps for the project and beyond
With regards to this project, several follow-on activities are required:
• Collaborative work with the UK and wider community on the technical challenges associated
with realising a truly interoperable BoA Sentinel-2 dataset that is comparable in space and time
with itself, notably harmonisation across neighbouring image granules and robust correction for
bidirectional reflectance.
• Work with the UK and wider community on the technical challenges associated with realising a
truly interoperable BoA Sentinel-2 dataset that is comparable in space and time with other
sensors. This ideal for ARD presents enormous technical challenges that are actively being
addressed by the scientific community. Harmonisation across sensors would allow users to work
with any EO data from similar sensors interoperably e.g. Sentinel-2 and Landsat-8.
• Move towards the routine delivery of a Sentinel-2 ARD dataset to the UK and to explore export
models. This will require significant work to explore possible routes of access, governance
structure and business models. For example, the relative merits and demerits of consumption
of ARD from a third-party service provider, versus construction of a national UK capability,
championed by government or achieved through public-private partnership, versus provision
through wholly commercial enterprise(s).
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 16 of 56
• Further develop the definition of Sentinel-1 ARD, building on previous work undertaken10.
Sentinel-1 is an important dataset for enabling services and applications, particularly in a mid-
latitude environment with cloud, such as the UK.
• Investigate geometric elements associated with generated Sentinel-2 ARD products.
• Investigate ARD possibilities for future Sentinel and commercial missions.
• Investigate the value and use cases for exploiting DC technologies.
In conclusion, the UK is in a strong position, in many ways leading Europe and certainly in the top-tier
globally. Our exploitation focus is a great advantage and we must not lose our position as others progress.
The community is positive and overall there is a strong desire to make ARD a success.
Requirements for UK routine production of Analysis Ready Data
The batch processing of the UK Sentinel-2 data was completed on the Catapult’s infrastructure during
August and September 2017. The process injected the Sentinel-2 archive for the UK, covering a time
window that spanned from 2015 to early September 2017. To accomplish the task, Catapult engineers
developed a data pipeline flow based on open source software. The main technology used is an open
source data flow system called Apache NiFi11. Apache NiFi is a software project from the Apache Software
Foundation which enables the automation of data flow between systems. It can be described as data
logistics. The project was created by the United States National Security Agency (NSA) and originally
named Niagarafiles. In 2014 the NSA released it as open source software, via its technology transfer
programme.
The developed pipeline included custom processes, developed ad-hoc for the batch processing. The
entire pipeline was composed of 99 processors. Figure 1 shows the high level architecture of the solution.
For the UK batch processing the following hardware was used:
• 3 x 380 GB RAM, 24 CPU, 12 TB storage
The observed performance and statistics of the re-processing campaign were as follows:
• Total input: 543 GB
• Total output: 1.02 TB
• 16 parallel processes per machine, with a total parallelization of 48 processes for the cluster
• User time ~ 120hours (48 parallel processes)
• Average time of the process: 2.5 hours
10 http://starhub.sa-catapult.co.uk/wordpress/wp-content/uploads/2016/05/C223766-IPSP-Cat-Australia-4WS.pdf
11 https://nifi.apache.org/
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 17 of 56
Figure 1 - High level diagram of the pipeline data flow
The development of this pipeline has opened up multiple scenarios, where ARD production would
become a straightforward process. The Catapult is currently looking to produce a more scalable and ready
to use pipeline that can be shared and used in the CEMS (Climate, Environment and Monitoring from
Space (CEMS) facility) environment. The main components of the pipeline under development are:
• Easy to use SEDAS API query for Sentinel-2 data: a simplified query interface is under
development, to locate, copy locally and process the data. End users will need to state their
area of interest and the time window that are interested.
• Scalable pipeline: an automate pipeline process to be deployed over multiple servers to
consume high volume of data.
• Landsat plugin to create ARD products.
• Enabling the end users to generate ARD themselves for an area of interest through providing
them the necessary toolset. For this purpose the entire data pipeline will be built into a CEMS
template. When a new data re-process would be required, a VM(s) can spin up with the correct
settings and the all required software.
In parallel, the Catapult has been investigating the use of DC technology through completing a DC
prototype for Wales. This contains all the Sentinel-2 ARD for Wales to date and will be made open to the
community to test. The enhancement of the DC prototype with further data and its sustainability will
depend on budget availability.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 18 of 56
Table 1 provides a high level overview of different options and associated costs of ARD exploitation within
a DC. The table is to be read in accordance with the below assumptions:
• ARD volume in this particular project is 2 x input ESA data for Sentinel-2. This may vary depending
on the requirement.
• ARD volume is 5 x input ESA data for Sentinel-1. This is untested for other Sentinels and will need
validation.
• No new buildings / server rooms are needed.
• Public cloud pricing models will persist into the future.
• Bespoke service wrappers would be in addition to the options presented; these could be low-
cost or expensive depending on service levels / complexity.
• No assumptions for service-legal agreements have been made
• Formal quotes would be needed as figures presented here will change depending on who
implements any solution
• Largest cost-driver is storage
• Savings realised - today’s prices assumed and worst case scenario
• Preference for Open / Closed / Hybrid solution not clear
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Draft
Version: v0.1
Date: 18/10/2017
Page 19 of 56
Table 1 – High level overview of options for routine ARD exploitation within a Data Cube
Pros Cons Scale Cost
Do Nothing No additional
funding needed
Lost
opportunity to
improve
efficiency
across
Government
and the UK
industry
Lost
opportunity to
place the UK in
a leading role
in these
technologies
and data
exploitation
N/A – Users
manage with
what they need /
can do locally
N/A
Do the minimum Very low cost
Improves current
working situation
Multiple
versions of
data across
different
departments
Specialist
uptake likely
No economies
of scale
Ad Hoc – a
selection of areas
available on
“request”
Covered by
existing budgets
UK coverage Central store of ARD
products for
multiple
departmental use
Significant reduction
in replicated work,
and hence cost
More powerful data
analytics capability
Commercial
clouds could
be expensive
for information
retrieval or
data download
May not cover
all government
user needs
(e.g. FCO, DfID,
MOD, BEIS etc)
UK-wide (~32 S1
scenes per
coverage) –
overseas
territories would
be extra
£250-450k / year
depending on
number or
missions
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 20 of 56
Could take
advantage of public
cloud offers
EU+ coverage Opportunity to lead
the way in European
service provision
Opportunity to offer
this capability into
Copernicus
programme and
attract Copernicus
operational funding
Opportunity to
meaningfully engage
with Global Network
of Data Cubes
Data and capabilities
available across
government with
corresponding
efficiencies
BREXIT and
hence the UK’s
position in
Copernicus
unclear –
access to
budgets
May not cover
all government
user needs
(e.g. FCO, DfID,
MOD, BEIS etc)
European
countries plus N.
Africa. Russia?
(estimated at
~7PB data per
year – ESA + ARD
for all Sentinels
+ Landsat)
~£5m for initial
setup and 1st
year of S1
operations,
£2.5m-3m per
year for each
mission
Global coverage Opportunity to lead
the way in Global
service provision
Opportunity to offer
this capability into
Copernicus
programme and
attract Copernicus
operational funding
Opportunity to
meaningfully engage
with Global Network
of Data Cubes
Data and capabilities
available across
government with
corresponding
efficiencies
BREXIT and
hence the UK’s
position in
Copernicus
unclear –
access to
budgets
Cost to HMG –
initial and
ongoing
Global - ~25PB
data per year
(ESA data + ARD
products – for all
Sentinels +
Landsat)
~£12m for initial
setup and 1st
year of S1
operations, £7m-
10m per year for
each mission
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 21 of 56
PART 2: SUPPORT FOR ESA SENTINEL-2 DATA WITHIN
ARCSI
Key Points:
1) The ARCSI software can now be used to produce analysis ready data (ARD) as a topographically
corrected surface reflectance product for Sentinel-2.
2) The quality of the product is comparable and in some ways an improvement (i.e., image band
sharpening and reliable topographic correction) of the ARD surface reflectance currently (July
2017) provided by ESA through the Copernicus Open Access Hub.
3) Further improvements are required to classify cloud and cloud shadows for Sentinel-2.
4) ARCSI is a flexible, open source and free to use software system which supports many satellite
sensors and can be further developed and improved by the wider community.
5) Methods and datasets for validating the surface reflectance product are required these are
currently unavailable and would require resources to generate and then valid the product
generated through ARCSI processing chain.
6) It is recommended that ARCSI should be tested and approved to provide ARD data which meets
the CEOS CARD4L standard.
Introduction
The aim of this project was to provide functionality with the ARCSI software
(http://www.rsgislib.org/arcsi) to enable the production of analysis ready data (ARD) for the Sentinel-2
sensor. ARCSI already provides this functionality for several sensors including:
• Landsat (1-5, 7 & 8)
• WorldView 2 & 3
• Pleiades
• RapidEye
• SPOT 5, 6 & 7
Functionality varies for different sensors depending on the wavelengths and metadata provided for a
given sensor. For Sentinel-2, the aim is to provide functional equivalence to Landsat TM, ETM+ and OLS.
However, Sentinel-2 does not have any thermal wavelengths, which makes consistent and reliable cloud
masking more difficult.
To install ARCSI, binaries are provided through the conda (https://conda.io) package management
system. Once conda is set up, to install the stable release the following command is used:
conda install -c conda-forge arcsi
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 22 of 56
The development channel, through which development versions of ARCSI are made available, is au-eoed.
The development version can be installed using the following command:
conda install -c au-eoed -c conda-forge arcsi
The source code for ARCSI is available online through the bitbucket platform:
https://bitbucket.org/petebunting/arcsi
Data Analysis
ARCSI provides a number of product options, those relevant to Sentinel-2 are listed in Table 2.
Table 2 - ARCSI Processing Products
ARCSI Product Definition
RAD At sensor radiance image.
SHARP Sharpen the 20 m image bands to 10 m using a local correlation filter.
SATURATE Mask of saturated pixels
TOA At sensor reflectance (top of atmosphere; TOA)
DOS Perform a dark object subtraction to retrieve an estimate of surface reflectance.
DEM Extract and reproject the inputted DEM for the image.
TOPOSHADOW Produce a mask for topographic shadow using the DEM.
CLOUDS Perform a classification to mask cloud and cloud shadow.
CLEARSKY Identify large continuous areas without cloud, providing an image mask
DOSAOTSGL Estimate a single value of Aerosol Optical Thickness (AOT) for the scene.
SREF Using the 6S atmospheric model calculate the surface reflectance pixel value for the scene.
STDSREF Calculate standardised reflectance, normalising for the sun angle and topography (i.e.,
topographic correction)
FOOTPRINT Extract the image data footprint as a shapefile.
METADATA Create a metadata file with key information from the input image and information calculated
during the analysis.
The user can also choose whether the outputs are at 10 m or 20 m image resolution and the resampling
method. If the sharpening option is select then the output image resolution needs to be at 10 m and a
nearest neighbour resampling needs to be selected. Additionally, the user can also select an output
coordinate system and projection that is different to input image projection and coordinate system.
Parsing Sentinel-2 Headers
ARCSI can parse all the known ESA SAFE header formats extracting the relevant data for the Sentinel-2
data. However, for the whole scenes (i.e., a number of granules within a single SAFE file, as disseminated
before September 26th, 2016) these need to be split before processing within ARCSI, which operates on
a per granules basis. A command arcsisplitsen2granules.py has been provided for splitting these files into
individual granules.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 23 of 56
At-sensor (top of atmosphere) radiance
Sentinel-2 Level-1c data is provided as top of atmosphere reflectance. To retrieve at-sensor radiance
from these data the equation for deriving top of atmosphere reflectance needs to be rearranged for at-
sensor radiance:
𝐿𝜆 =ρλ ∙ ESUNλ ∙ cos θs
π ∙ d2
where λ is the wavelength, ρλ is the spectral (planetary or TOA) reflectance for wavelength λ, Lλ is the
spectral radiance (W m-2 sr-1 μm-1) d is the Earth-Sun distance in astronomical units, ESUNλis the mean
solar exoatmospheric irradiance in units of W m-2 μm-1, and cosθs is the solar zenith angle.
At-sensor (top of atmosphere) reflectance
At sensor or TOA reflectance is the ratio of the at-sensor radiance and the incoming radiance energy from
the sun (ESUN values). However, Sentinel-2 is provided as with pixel values as TOA reflectance, multiplied
by 10,000. ARCSI has a user selectable multiplier (default 1000) and therefore if different from 10,000
the TOA reflectance data as provided by ESA is rescaled.
Surface Reflectance (SREF)
Surface reflectance also called bottom of atmosphere reflectance is the ratio of incoming radiance (i.e.,
from the sun) with the radiance that is reflected from the ground surface. To derive this measurement
the effect of the atmosphere needs to be removed from the top of atmosphere radiance measured at
the sensor. The Second Simulation of the Satellite Signal in the Solar Spectrum (6S) atmospheric model is
used within the ARCSI software to generate a modelled atmosphere for the scene being corrected. It is
one of the most widely used, rigorously validated, and heavily documented RT codes in the scientific
remote-sensing community (Vermote et al., 2006). By using 6S, the main atmospheric effects (gaseous
absorption by water vapour, carbon dioxide, oxygen, and ozone, and scattering by molecules and
aerosols) are considered. The primary inputs relate to geometrical conditions, the atmospheric model for
gaseous components, the aerosol model (type and concentration), spectral condition, and ground
reflectance (type and spectral variation) (Tripathi et al., 2005).
Outputs from the model include:
• Direct and diffuse irradiance onto a horizontal surface, referred to as Ehdir and Eh
diff respectively.
• Radiance at the satellite level.
• Apparent surface bi-directional reflectance (ρh) for a horizontal surface.
The key parameters for achieving a good atmospheric correction, requiring a good atmosphere
representation within 6S, are the Aerosol Optical Thickness (AOT) also referred to as Aerosol Optical
Depth (AOD) and the atmospheric water vapour (Table 3). AOT most strongly affects the visible
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 24 of 56
components of the spectrum, particularly the blue. Water vapour most strongly affects the shortwave
infrared (SWIR) components of the spectrum.
Table 3 - The relative percentage of contribution for each input parameter to 6S (Powell et al., 2017).
AOT Altitude Water Vapor Ozone View Angle
Landsat 5 78.74 % 4.51 % 16.74 % 0.0 % 1.79 %
Landsat 7 80.03 % 4.80 % 15.45 % 0.0 % 1.68 %
Landsat-8 87.98 % 2.30 % 10.52 % 0.0 % 0.05 %
Sentinel-2 86.67 % 2.35 % 10.96 % 0.0 % 1.85 %
AOT has the most effect on the overall atmospheric correction accuracy, however it also varies on short
spatial and temporal baselines. Therefore, the estimation of AOT needs to be undertaken on a per scene
basis. The method applied is based on the application of a dark object subtraction (DOS) applied to the
at-sensor reflectance for the blue wavelength. For the darkest feature within the scene, the resulting
surface reflectance estimation from the DOS is then used to estimate an AOT (at 550nm) for the scene.
This is performed by iteratively testing AOT values (at steps of 0.05) until a value is identified that most
closely corresponds to the surface reflectance estimated via the DOS. This has been found to be robust
in identifying a single global value for the scene. However, it should be noted that variation of AOT is
often identified within the scene and approaches have been proposed which identified a AOT surface for
the scene. Future work within ARCSI is to develop and include such methods.
Standardised Surface Reflectance
The method of Shepherd & Dymond (2003) was implemented to provide a standardised reflectance
product. Standardised reflectance refers to a product which is normalised for the solar and sensor view
angles and in this case topography. In terms of providing a topographic correction, this method only
works for images where the solar elevation is between 50 and 70 degrees (i.e., from late spring to early
autumn) but does not produce artefacts outside of this range. Standardised reflectance is defined using
the following equation
𝜌ℎ𝑑ir =
𝜋𝐿
𝐸dir𝛾⁄ + 𝛽𝐸dif
where 𝜌ℎ𝑑ir is the direct reflectance for a horizontal surface, L is the radiance at the bottom of the
atmosphere, 𝐸dir is the direct component of the irradiance and 𝐸dif is the diffuse component, the ratio 𝛽
is evaluated using a bidirectional reflectance distribution model but in the study a constant of 1 was used
(Shepherd & Dymond, 2003):
𝛾 =cos 𝑖 +cos 𝑒
cos 𝑖ℎ +cos 𝑒ℎ
where 𝑖 and 𝑒 are the incidence, and exitance angles on an inclined surface, and 𝑖ℎ and 𝑒ℎ are the
incidence and exitance angles on a horizontal surface (Figure 2).
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 25 of 56
Figure 2 - Standardised reflectance normalised for topographic and view angle (adapted and redrawn from Shepherd &
Dymond 2003)
Cloud Masking
The FMASK algorithm of Zhu and Woodcock (2012), implemented within the python-fmask
(http://pythonfmask.org/) library has been used to classify cloud and cloud shadow features within the
scene. Unfortunately, Sentinel-2 does not have a thermal image band (unlike Landsat) and therefore the
quality of the result cloud mask is of poorer quality than that produced for Landsat imagery. This is an
area where further work is required to derive an ARD Sentinel-2 product.
Clear-Sky Masks
Following cloud and cloud shadow masking it is common to get many small selections/pieces of imagery
between the clouds left over (e.g. Figure 3). In many cases, those small regions of imagery are either not
useful for a given application and/or atmospheric correction of the regions is of poor quality due to
adjacency effects from the clouds and atmosphere associated with a cloudy environment.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 26 of 56
Figure 3 - There are many small pieces of imagery between the clouds following cloud masking.
A solution to this problem is to select only the areas of the scene above a particular size threshold as
being ‘useful', discarding the rest, within ARCSI this product is called ‘Clear Sky'. A two-step process was
deployed where first a buffer of 30 km was taken from all clouds. Within those regions, only those with
a size greater than 3000 pixels were selected and then they were grown, so they were not within 10 km
of a cloud object.
It should be noted, that for Sentinel-2 the number of false positives within the cloud masking is quite high
and therefore, until the cloud masking is improved, the clear sky product can remove a large amount of
useful imagery.
Topographic Shadow Masks
A ray tracing algorithm (Figure 4), like the process implemented for the cloud shadow masking, has been
implemented. The algorithm produces a binary mask for each pixel within the scene identifying pixels
that are within regions of direct shadow and therefore are not going to provide useful and reliable data
for automatic analysis. A binary mask is provided as output.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 27 of 56
Figure 4 - Ray tracing for identifying regions of topographic shadow.
Image Band Sharpening
The sharpening product to enhance the 20 m image bands using the 10 m bands to produce a sharper
stacked 10 m product works through the application of local linear regression models. The method was
first proposed by Dymond & Shepherd (2004) for pan-sharpening Landsat-7 imagery and has
subsequently been extended for Sentinel-2. Within ARCSI it is applied to the radiance image where the
20 m bands have been oversampled to 10 m resolution image using a nearest neighbour interpolation. A
7x7 pixel filter is applied to the 10-m image stack where within the 7x7 window a linear regression is
performed independently for each lower resolution band to each higher resolution band. The linear
model for each lower resolution image band with the best fit, above 0.5, is then used to predict the image
pixel value for the band. If not fit above 0.5 is identified, then the image pixel value is not altered.
Processing Multiple Granules
One of the issues with many approaches of atmospheric correction is that individual images (e.g.,
Sentinel-2 granules) are independently processed creating boundary effects and artificial changes in
reflectance between scenes due to the atmospheric correction parameterisation. To mitigate those
issues within ARCSI a new option has been implemented through this project which allows a list of
adjacent image header files to be passed to the ARCSI command and for them to be processed together
such that the parameters are harmonised to avoid tile boundary effects. Currently, within ARCSI the
inverted AOT values are only considered for harmonisation and in this first implementation a mean AOT
value is derived from the values inverted from each of the input scenes. The mean value is then used to
correct all of those scenes. While this approach has a number of limitations and can be developed further,
this functionality provides a starting point for further harmonisation as ARCSI develops in the future and
is applicable to all the sensors (e.g., Landsat) which ARCSI supports. For multi-granule processing ARCSI
provides support for multi-core processing allowing all available computational cores to be used,
speeding up processing. Two multi-core solutions have been implemented, firstly via the python multi-
processing module which supports multi-threading on a shared memory machine (e.g., single desktop
computer or virtual machine such as AWS). Secondly, via the Message Passing Interface (MPI) which
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 28 of 56
supports processing on computational clusters, where memory is not shared. Examples of such machines
include NERC’s JASMIN cluster and the Supercomputing Wales cluster used for testing in this project.
Access to Sentinel-2 Imagery
For downloading the Sentinel-2 imagery commands have been provided within ARCSI to a) download
database of all the Sentinel-2 scenes globally (arcsisetupsen2db.py) and b) query the database and
retrieve a list URLs to download the scenes using the Google cloud tools (gsutils; arcsigensen2downlst.py)
from the Google Sentinel-2 archive: https://cloud.google.com/storage/docs/public-datasets/sentinel-2
The Google repository of Sentinel-2 imagery (note, Landsat imagery is also available; ARCSI has equivalent
commands for searching and downloading Landsat imagery) has advantages in that the whole global
achieve of imagery is available and the bandwidth of the Google service is significant allowing imagery to
be quickly and efficiently downloaded. For example, 11,000 Landsat scenes were downloaded from the
Google service in only 48 hours on to the Supercomputing Wales HPC system on the UK Janet academic
network.
Results
Quality of the Cloud Masking
To assess the accuracy of the FMASK cloud masking approach, 6000 random points (2000 in each class)
were manually interpreted against the input imagery. The accuracy of the classification is shown in Table
4, where the overall accuracy is 73 %, with a kappa of 0.6. However, the primary source of the error is an
over estimation of the clear sky area, which is reflected in the low producers accuracy (58 %) and users
accuracy for the cloud (67 %) and cloud shadow (69 %). The most common errors are associated with
urban areas and ploughed or recently cut fields where they are classified as clouds as they are spectrally
similar to some cloud regions within the scene. As a consequence of over estimation in cloud cover there
is also an over estimation in the extent of cloud shadow as for each cloud feature identified a search is
performed for a corresponding cloud shadow. This effectively doubles the error for false clouds.
Table 4 - Accuracy Assessment of the cloud masking. Overall accuracy is 73% with a kappa of 0.6.
Clear Cloud Shadow User (%)
Clear 1693 223 84 84.65
Cloud 649 1333 18 66.65
Shadow 578 48 1374 68.7
Producer (%) 57.98 83.1 93.09 73.33
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 29 of 56
Quality of the Surface Reflectance Spectra
Unfortunately, there are no ground spectra available which are coincident with the Sentinel-2
acquisitions. However, a visual inspection of the spectra for common land covers with expert knowledge
on the expected shape and magnitude of the values can be undertaken (Figure 5). Spectra can also be
compared to other sensors (e.g., Landsat 5; Figure 6).
The spectra illustrated in Figure 5 were selected from a set of regions of interest for a Sentinel-2 scene
acquired on April 8th 2017 while the spectra illustrated in Figure 6 were from a Landsat 5 scene over a
similar area from April 2011. There is a strong correspondence in the shape and magnitude of the
reflectance for the land covers illustrated and these also correspond with the expected shapes and
magnitudes of those land covers from our expert knowledge.
Image Pixel Comparison to ESA ARD Product
To compare the surface reflectance product produced from ARCSI to the ESA ARD product released
through the Copernicus Open Access Hub a number of scenes have been downloaded, visually compared
(Figure 7) and a histogram based comparison of the images undertaken (Figure 8). Through the visual
analysis, a number of artefacts (Figure 7b) were identified in the ESA ARD SREF (July 2017). The artefacts
appear to be associated with the topographic correction routine, where slopes have been over corrected.
However, the ARCSI topographic correction routine has successful flattened the topography (Figure 7a).
To directly compare the ESA ARD and ARCSI ARD surface reflectance products on a per pixel level images
were plotted against one another (e.g., Figure 8). The blue image band had an r2 higher than 0.8 while all
other image bands had an r2 higher than 0.9. The spread from the one-to-one line is in part due to the
ESA ARD image artefacts, particularly within the visible channels. The single AOT values used within ARCSI
verses the AOT surface used within the ESA product will also contribute to the variance between the two
products. However, the overall correspondence between the two products is very close and comparable.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 30 of 56
Figure 5 - Example spectra examples from Sentinel-2. A) Conifer Forest, B) Arable Grass, C) Water, D) Urban, E) Non-
Photosynthetic Vegetation and F) Sand.
a) b)
c) d)
e) f)
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 31 of 56
Figure 6 - Example spectra examples from Landsat 5. A) Conifer Forest, B) Arable Grass, C) Water, D) Urban, E) Non-
Photosynthetic Vegetation and F) San
a)
f)
b)
e)
c) d)
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Draft
Version: v0.1
Date: 18/10/2017
Page 32 of 56
Figure 7. a) ARCSI ARD Surface reflectance b) ESA ARD Surface reflectance product (July 2017) which has a number of artefacts where the topography has been over corrected.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Draft
Version: v0.1
Date: 18/10/2017
Page 33 of 56
Figure 8 - Per-pixel comparison of the ESA ARD and ARCSI ARC SREF product for 17th June 2017 for tile 30UVD. All bands
other than the blue have an r2 of above 0.9 demonstrating a strong correspondence.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 34 of 56
Standardised Reflectance
Producing standardised reflectance normalises the surface reflectance with respect to the solar angle
and topography. Therefore, a topographic correction is applied; Figure 9 illustrates this correction. The
application of a topographic correction has been shown in several studies to increase the accuracy of the
resulting derived products (e.g., land cover classification).
Figure 9 - An example of a Sentinel-2 scene with a) Surface Reflectance and b) Standardised Surface Reflectance which
performs a topographic correction.
Imagery Sharpening
Figure 10 shows a region of a Sentinel-2 image where the 20 m image bands have been resampled to 10
m using a cubic convolution interpolation (Figure 10a) versus the same area which has been processed
using the image sharpening option (Figure 10b). The image sharpening makes the spatial detail much
sharper maintaining the 10-m resolution while using the 20 m image bands.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Draft
Version: v0.1
Date: 18/10/2017
Page 35 of 56
Figure 10 - An example of a) an image without sharpening and b) an image which has been sharpened
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Draft
Version: v0.1
Date: 18/10/2017
Page 36 of 56
ARCSI Recommendations & Conclusions
Recommendations
The report has presented the updated version of ARCSI to support the production of ARD Sentinel-2
imagery. ARCSI provides a command line user interface which is easy to batch process and automate with
scripting. In terms of further development of ARCSI for the production of Sentinel-2 ARD data the
following should be considered:
• The over estimation of cloud and cloud shadow with the cloud classification approach reduces the
amount of available imagery. Further work is required to improve the overall accuracy of the cloud
masking approach. For a monitoring system over a specific area (e.g., the UK) a time series
approach would be recommended as the whole image archive can be maintained and utilised.
However, for a system where single or a small number of images are required for a given area a
time series approach is limited by the need to download and process a time series for the area of
interest. Therefore, it is recommended that both time series and single scene approaches are
explored and developed as options within ARCSI.
• Improvements in the recovery of AOT such that a surface across the image can be reliably retrieved
even with large areas of missing data due to cloud cover. Issues were found with reliably recovering
an AOT surface for scenes with significant gaps due to cloud cover, therefore for consistency a
single value was recovered from each scene and used for the whole scene. Further research on the
recovery of AOT is needed to improve this aspect of the atmospheric correction.
• Only a simple implementation of an approach to harmonise between neighbour’s image granules
has currently been provided within ARCSI (i.e., taking a mean AOT). This is providing a starting point
from which further developments can be made and considered, particularly when considering the
generation of AOT surfaces within and between image granules.
• Further work on correcting for bidirectional reflectance would be beneficial. Specifically, reviewing
more closely the degree to which the standardised reflectance product has corrected for changes
in sun angle between scenes. The method of Shepherd & Dymond (2003) has been applied
throughout providing a significant increase in the quality of the scenes from late spring to early
autumn. However, methods such as Flood et al., (2013) has been presented and may improve the
quality of the output product, but would require parameterisation which not trivial.
• A formal test and validation of the ARD surface reflectance product further datasets are required.
During this project, a basic visual check to ensure the spectra produced are of the right shape, and
a comparison to the ESA ARD product were carried out. However, these are insufficient and
provide no information on the uncertainty associated with the derived spectra. To perform a
formal validation, third party datasets of ground spectra that are temporally coincident with the
satellite acquisition are required for sites which are spectrally homogeneous at the appropriate
resolution for the sensor. Alternatively or alongside, spectrally invariant features can be used but
these are difficult to source for a range of spectral targets. Methods and appropriates have been
proposed for lower resolution instruments, such as MODIS, and similar work is therefore required
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 37 of 56
to validate these Sentinel-2 ARD products. In undertaking this work significant progress in
demonstrating ARCSI meets the CARD4L standard (Point 6) would have also been carried out.
• CEOS have proposed a standard for ARD data under CARD4L standard. The details are this are still
be finalised but it is expected that a peer-review process will be undertaken to approve a given
system produces data which meets the CARD4L standard. It would be desirable to have ARCSI
tested ensuring that ARCSI produces data which meets the CARD4L standard.
Conclusions
The project has provided an open source tool for providing an ARD surface reflectance product for the
ESA Sentinel-2 instruments. As demonstrated, the product produced by ARCSI provides a comparable
result to the product currently being released by ESA through the Copernicus Open Access Hub but
without artefacts within the topographic correction. The image band sharpening approach implemented
with ARCSI provides a significant improvement over traditional approaches of image resampling when
using the 20 m image bands within a 10 m resolution image stack. The ability to process multiple image
granules together, avoiding boundary effects between the image granules is also a significant step and
advantage of ARCSI over other similar software systems. While further work is recommended,
particularly on the approaches to cloud masking and product validation, the support for Sentinel-2 within
ARCSI provides a good base by providing a similar ARD product as to that which has been demonstrated
in the Landsat archive for Wales. The outputs are also comparable to the ARD product provided by ESA
and ready to be used with a production environment to deliver an ARD service for Sentinel-2.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 38 of 56
APPENDIX A - SENTINEL-2 CHARACTERISTICS AND PROCESSING CONSIDERATIONS
Mission Rationale
The objective of Copernicus is to use multi-source data to get timely and quality information, services
and knowledge, and to provide autonomous and independent access to information in relation to the
environment and security on a global level. The twin satellites of Sentinel-2 provide continuity of Landsat-
type image data, through providing multispectral observations at high temporal and spatial resolutions
that are unprecedented within a fully open access data policy. This data benefits services and applications
such as land management, agriculture and forestry, disaster control, humanitarian relief operations, risk
mapping and security [1]. Most importantly, unprecedented access to this type of data shall enable
services to be developed and insights attained that were not previously possible.
Satellite Characteristics
The Sentinel-2 constellation comprises twin polar-orbiting satellites in the same orbit, phased at 180°
apart. A sun-synchronous orbit at an average altitude of 786 km ensures a consistent angle of sunlight
upon the Earth’s surface is maintained. A resulting mean local solar time of 10.30 am is comparable to
that of Landsat and almost identical to SPOT-6, enhancing sensor time-series and spatial compatibility
and continuity within monitoring applications.
Sentinel-2A launched June 2015 and Sentinel-2B launched March 2017. While lifespan of the
constellation is designated as 7.25 years, core components can accommodate up to 12 years of operation
(up to 2027). Continuity of the Copernicus Sentinel missions is ensured with launches of Sentinels-2C and
D to complement current Sentinel-2A and B.
The optical multi-spectral imaging (MSI) telescope payload design enables a 290-km wide swath and,
consequentially high frequency revisit of 5 days at the equator and revisit of 2-3 days at mid-latitudes.
Systematic acquisition is completed over land and coastal areas between latitudes 56° South and 83°
North as shown in Figure 11.
A summary of the 13 spectral bands with respects to their radiometric configuration and spatial
resolution up to 10 m is provided in Table 5. These are contrasted to the Landsat-8 OLI sensor and SPOT
6/7 sensor characteristics in Figure 12.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 39 of 56
Figure 11 - Sentinel-2 geographical coverage (green) [3].
Table 5 - Summary of Sentinel-2 MSI band radiometric and spatial characteristics [4].
WaveBand Central Wavelength (μm) Spatial Resolution (m)
Band 1 - Coastal Aerosol 0.443 60
Band 2 - Blue 0.490 10
Band 3 - Green 0.560 10
Band 4 - Red 0.665 10
Band 5 - Vegetation Red Edge 0.705 20
Band 6 - Vegetation Red Edge 0.740 20
Band 7 - Vegetation Red Edge 0.783 20
Band 8 - NIR 0.842 10
Band 8A - Vegetation Red Edge 0.865 20
Band 9 - Water Vapour 0.945 60
Band 10 - SWIR - Cirrus 1.375 60
Band 11 - SWIR 1.610 20
Band 12 - SWIR 2.190 20
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 40 of 56
Figure 12 - Comparison of Sentinel-2 MSI radiometric and spatial characteristics against Landsat-8 OLI and SPOT 6/7 [2].
Product Levels Available
Sentinel-2 products are systematically made available to users at processing Level-1C: top-of-atmosphere
reflectances in fixed cartographic geometry (combined UTM projection and WGS84 geodetic system)
along with cloud, land and water masks. Level-1C images are a set of tiles of 100 km2 and approximately
500 MB. This product contains applied radiometric and geometric corrections; including
orthorectification and spatial registration to sub-pixel accuracy [5]. Scene products comprising
Land/Water, Cloud Masks and ECMWF data are also provided alongside the multi-spectral bands.
For robust exploitation of Sentinel-2 products alongside other sensors, data is required in surface
reflectance that is spatially and temporally consistent and comparable. Responsibility for generation of
this product currently falls on the user side, with identical processing from Level-1C to Level-2A
duplicated for the same product by different users. To implement this processing step ESA made the
Sen2Cor atmospheric correction tool available. This Level-2A tool, and others, currently have very limited
accessibility, documentation or proven implementation.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 41 of 56
Existing Data Dissemination
Open dissemination of all Sentinel mission data is provided through the Copernicus Programme via a
combination of Core and Collaborative Ground Segments.
Key challenges associated with the dissemination of Sentinel-2 and other Sentinel datasets include:
• Storage required
• Timely data provision
• Accessibility of data
• Engaged community
Figure 13 - Overview of Sentinel Ground Segment infrastructure [6].
Sentinel Core Ground Segment
The Sentinel Core Ground Segment comprises a European distributed infrastructure that coordinates to
monitor and control the Sentinel satellites and routinely download, process and disseminate quality
controlled data products to users.
Fundamentally, the Core Ground Segment is responsible for ensuring systematic and timely processing
of all Sentinel data to their designated product levels and to serve these products through a single virtual
access point (Sci Data Hub) [7].
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 42 of 56
Sentinel-2 Level-1C data products are routinely available within 3-24 hours of being sensed by the
satellite, with this latency largely related to the delay following data acquisition to overpassing a ground
station.
Sentinel Collaborative Ground Segment
The Sentinel Collaborative Ground Segment complements the exploitation of the Sentinel missions in
many areas, such as:
• Data acquisition and (near-) real-time production. Local ground stations are configured to receive
Sentinel data as the satellite passes overhead.
• Complementary products and algorithm definitions – these ‘collaborative data products’ may be
tailored for regional coverage or specific applications. These types of products may extend the
Sentinel core product chains.
• Data dissemination and access, supporting redistribution of Sentinel core products by establishing
additional pick-up points.
• Development of innovative tools and applications.
• Complementary support to calibration/validation activities.
The Sentinel Collaborative Ground Segment is funded by third parties. It provides the basis for
international cooperation, and is supported where necessary by generic interfaces and/or Copernicus
Ground Segment interfaces.
The UK collaborative ground segment for Copernicus, SEDAS, comprises Airbus DS Geo-intelligence who
operate the Processing and Archiving facilities for Sentinel-1 and 2; Satellite Applications Catapult who
are responsible for managing and disseminating the data to UK commercial users and STFC-CEDA who
are responsible for managing and disseminating the data to UK academic users.
Informal Data Dissemination
An increasing number of organisations collect all or a subset of Sentinel data products from the Scientific
Data Hub (or collaborative ground segment equivalent) and make these openly available alongside other
geospatial datasets (e.g. AWS, Google, Planet). Such data dissemination results from the recognised value
of exploiting satellite datasets directly alongside additional geospatial (and ancillary) datasets as well as
close to dynamically available compute resources. While such informal data dissemination is largely
open, providing contemporary satellite datasets in this type of environment is an attractive prospect for
users.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 43 of 56
Image processing and correction considerations
Optical Earth Observation satellite imaging sensors, such as Sentinel-2, passively measure radiation
reflected from the Earth’s surface. Radiance measured directly by a sensor therefore includes
contributions from Rayleigh and aerosol scattering, gas absorptions of the atmosphere, surface
bidirectional reflectance distribution function (BRDF) effects of anisotropic surfaces and topographic
illumination effects resulting in shadowing. Optical imagers are also particularly vulnerable to cloud
cover, a key pre-Sentinel-2 limitation to satellite image exploitation over the UK.
Robust processing of optical satellite datasets is required to mitigate or remove these effects to obtain
spatially and temporally comparable measurements of surface reflectance that are truly representative
of Earth surface characteristics. Corrections to a processing level that is ‘analysis ready’ are increasingly
required in an operational environment to enable full exploitation of the unprecedented volume of
satellite Earth observation data from the full variety of sensors available.
This section briefly summarises methods for implementing corrections required to process Sentinel-2
Level-1C products to Level-2A: (i) atmospheric correction, (ii) BRDF correction, (iii) topographic
correction, and (iv) cloud masking.
Atmospheric Correction
Atmospheric correction is the process of correcting from a TOA radiance product to a BOA surface
reflectance product. This retrieval of surface reflectance requires mitigating effects predominately
associated with Mie, Rayleigh and particle (aerosol) scattering and absorption of atmospheric gases (inc.
water vapour, CO2, etc.) [11], [12]. With techniques for performing these corrections very well established,
implementation of physics-based as opposed to empirical models is feasible, and methods for VIR (visible
and infrared) bands particularly mature [13].
All techniques entail the use of a radiative transfer model, the parameterisation of which (aerosol optical
depth, ozone, water vapour etc.) is directly related to the estimation quality of atmospheric parameters [14], hence quality of corrected surface reflectance product. Atmospheric parameters being estimated by
a radiative transfer model include ratio of diffuse to total irradiance for both sun and sensor directions,
atmospheric albedo, path radiance and transmittance for sun and sensor directions.
MODerate resolution TRANsmission (MODTRAN) [15] and the Second Simulation of a Satellite Signal in
the Solar Spectrum (6S) [16] are two of the most commonly used radiative transfer models [14]. MODTRAN
is complex yet highly flexible, whereas 6S is relatively of lesser complexity. Accuracy of atmospheric
parameter estimates are, however, associated much more with the quality of input data than choice of
model. A variety of interfaces exist to enable parameterisation of these models, within both proprietary
and open software. For example, Fast Line-of sight Atmospheric Analysis of Spectral Hypercubes
(FLAASH) [17] associated with MODTRAN, and Py6S and (ARCSI [18] associated with 6S.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 44 of 56
BRDF Correction
Mitigation of surface BRDF is an important step to correct view and illumination angle effects and to
normalize surface reflectance both in a single image and across multiple images. Due to different view
and solar angles and anisotropic surfaces, observed surface reflectance is different even if the surface
cover is the same [19]. This occurs for a single scene with different view and solar angles and different
scenes sensed at different seasons and geographical regions due to the solar angle variation [13], [19].
Like correcting for atmospheric effects, the quality of BRDF mitigation is directly related to the quality of
parameterisation. If these parameters may be derived then BRDF and atmospheric correction may be
coupled together into a single correction [13], [20].
For lower spatial resolution, finer temporal resolution satellite imagers (i.e. MODIS), the frequent revisit
(up to twice a day with MODIS) enables BRDF parameters to be derived from the data itself [21]. For finer
spatial resolution datasets, and typically limited temporality, BRDF parameters must be obtained from
other sources (i.e. MODIS data) [12].
Topographic Illumination Correction
Terrain illumination (topographic) correction is a processing step particularly required for areas of
elevated terrain. On inclined surfaces irradiation measured by a satellite sensor is regularly modified such
that slopes facing toward the sun receive more solar irradiance and appear brighter in satellite images
than those facing away from the sun [22]. This effect combined with those associated with BRDF result in
hill shade across a satellite image.
Use of a good quality DEM that is very well aligned with the satellite data is necessary for derivation of
accurate terrain parameter computation; including slope, aspect, incident and exiting angles, relative
azimuth angles, etc. Traditionally empirical models have been utilised for applying the terrain correction
and within a separate process to atmospheric and BRDF correction stages.
Cloud Masking
A fundamental difficulty in exploiting the full acquisition capacity of optical satellite imagery is the
prevalence of cloud and associated shadowed regions, with global annual mean cloud cover estimated
at approximately 66% [23]. Improved temporality of Sentinel-2 over Landsat missions greatly improves the
probability for frequent cloud free coverage of an area of interest. It remains important to reliably
identify these clear skies areas within an ARD product to fully exploit available datasets. Significant
research has therefore been associated with extracting cloud free areas from legacy Landsat data in an
automated way, with mature techniques that are being transferred to Sentinel-2 datasets.
The most effective cloud masking techniques exploit multiple Thermal Infrared (TIR) bands for masking
clouds, and others Cirrus bands specifically designed for effectively mitigating the effects of semi-
transparent clouds, whose signal comprises both clouds and surface underneath.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 45 of 56
Absent of a TIR band, cloud masking of Sentinel-2 datasets can leverage information within Aerosol,
Water Vapour and Cirrus bands. Sentinel-2 Level-1C products contain an associated vector cloud mask [24], however this is often of insufficient quality, with most users requiring over-identification of cloud to
ensure effective clear-skies regions.
Masking cloud shadow has traditionally been achieved using spectral masking techniques, however these
unsurprisingly also result in masking of spectrally similar non-shadowed land covers. State of the art
methods therefore exploit cloud masks alongside sensor geometry to infer regions of probable shadow
prior to implementing spectral masking techniques [25].
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 46 of 56
APPENDIX B - SENTINEL-2 ARD INTERNATIONAL CONTEXT
Background
Satellite image product generation as ARD and the growing need to lower the barrier to entry for
exploitation of these datasets is not a new concept. Many organisations globally have been working
towards this aim for several years. From defining standards for accessibility of geospatial data to space
agencies understanding the need to justify from a business perspective the cost of Earth observing space
craft, through making associated data readily exploitable. This has led to an environment in which work
is increasingly being undertaken on how these missions and data can be accessed and exploited by non-
expert users.
This section acts as a consolidated summary of international activities that embed the context of the
heritage of ARD and organisations that are acting in the space of or directly relevant to the uptake of a
Sentinel-2 ARD product standard.
The process of Calibration and Validation could be considered the precursor to wide-spread production
of ARD. The Calibration and Validation of satellite data ensures that when satellite data is processed to a
higher-level product it is directly comparable to data from the same satellite but from a different
acquisition or in some cases to enable the comparison of data from different satellites. Calibration may
be defined as the process of quantitatively defining a system’s responses to known, controlled signal
inputs. Contrastingly, Validation is the process of assessing, by independent means, the quality of the
data products derived from those system outputs [26].
The volume of data acquired by Earth observing satellites is increasing at an exponential rate and
providing unprecedented insights into the dynamics of our planet. Due to the nature of satellites often
monitoring within different regions of the electromagnetic spectrum, at different spatial resolutions,
from varying geometries, and processed using different methodologies, it can be difficult to compare one
satellite dataset with another, making it essential to have globally recognised guidelines for the
Calibration and Validation process. By having a calibration process the data can be quantitatively defined
to controlled system inputs, the validation process in turn independently assessed by independent means
the quality of the data derived from the system outputs.
Organisation such as the Group on Earth Observations (GEO) and CEOS support the development of these
processes by facilitating participation in workshops to define such things as Calibration/Validation site
characterisation and classification, satellite and in site data access, the methodologies and guidelines for
the Calibration/Validation and Harmonisation and quality information. Five Calibration and Validation
domains are generally considered: radiometric, spectral, spatial, temporal, polarization and inter-
calibration of instruments against a common reference instrument allows achieving consistency across
satellite measurements at a given point in time and space.
Calibration and Validation activities, while varying across sensors, do not produce fully prepared and
consistent data through space and time that may be incorporated directly into analyses. In addition to
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 47 of 56
costs associated with generating, storing and disseminating additional higher level products, this is due
to variation in user application requirements; hence final corrections, have traditionally been
implemented independently by users. Such bespoke processing stages are essential in the exploitation
of satellite EO datasets for rigorous and specific scientific analyses. There will therefore always be a
requirement for the widespread distribution of lower level products to enable these activities and hence
an increasingly evident trend that definition of standards for these lower level products should sit within
the remit of calibration and validation.
International Community
A growing international community has developed over the last 20 years that specifically deals with the
sharing of geospatial information and the interoperability between different systems, both traditional
geospatial systems and satellite based. These organisations largely deal with the development of
standards. The standards these organisations deal with enable data to be curated in a way that not only
deals with the inherent hostility associated with what are traditionally scientific data formats but lower
the barrier to entry to the use of these data and their parent systems. Some of the key stakeholders are
identified below.
Heterogeneous Missions Accessibility (HMA)
The HMA initiative addresses issues of data publishing, discovery and access by engaging with national
space agencies, satellite or mission owners and operators, and industry, providing a forum for promoting
standaridisation.
Open Geospatial Consortium (OGC)
The OGC is an international industry consortium of companies, government agencies and universities
participating in a consensus process to develop publicly available interface standards.
Group on Earth Observations (GEO) and Global Earth Observation System of Systems (GEOSS)
GEO promotes the exploitation of Earth observation data to address global challenges. A fundamental
part of GEO’s Mission is to build GEOSS, a set of coordinated, independent Earth observation, information
and processing systems that interact and provide access to diverse information for a broad range of users
in both public and private sectors. GEOSS links these systems to strengthen the monitoring of the state
of the Earth and facilitates the sharing of and access to data.
Committee on Earth Observing Satellites (CEOS)
CEOS has long recognised a need for data processing infrastructure to support Earth science objectives.
Through its network of global connections, CEOS has determined that global users share many common
needs, such as data access, preparation, and efficient analyses to support applications. CEOS recognise
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 48 of 56
that many of these can be addressed through the Open Data Cube (ODC) initiative, a data architecture
solution that enables proper connection between data, applications and users. ODC is particularly
focused on supporting nations that lack the in-country expertise and computational infrastructure to
utilise EO data. There are currently deployments in Colombia, Vietnam and Kenya. CEOS too have been
working to develop a recognised ARD ‘brand’, CARD4L, which they define as - satellite data that have
been processed to a minimum set of requirements and organized into a form that allows immediate
analysis with a minimum of additional user effort and interoperability both through time and with other
datasets. As part of this work they have been instrumental in facilitation of the Atmospheric Correction
Australian Geoscience Data Cube (AGDC)
The Australian Geoscience Data Cube (AGDC) is a collaboration between Geoscience Australia, CSIRO and
NCI Australia and has been a key contributor to the ODC. The AGDC is a series of structures and tools that
calibrate and standardise datasets, enabling the application of time series and the rapid development of
quantitative information products. It uses open source standards and is designed to help both
government and private industry make informed decisions.
United States Geological Survey (USGS)
The United States Geological Survey (USGS) is a scientific agency of the United States government. A core
responsibility of the USGS is to produce Higher-Level Science data products, data which are processed
consistently to create a continuous record of the effects of climate change on Earth’s surface. This
includes the production of a Landsat ARD product. The USGS is a key contributor to the ODC.
European Space Agency (ESA)
ESA has a long history of publicly funded Earth observing missions and data publishing. With the new
availability of data through the Copernicus programmes and the associated volumes of data, the
European Space Agency is core to the global movement towards standardised data access and
processing. Various activities and work streams by ESA have influenced and driven the production and
dissemination of ARD as well as the emerging ARD standards. Below is a non-exhaustive list of the most
prominent initiates. ESA is a key contributor to the ODC.
CNES / THEIA
The Theia Land Data Centre is a national inter-agency organization designed to foster the use of images
coming from the space observation of land surfaces. Theia offers a broad range of images at different
scales, methods and services related to the land surface observation from space. For Sentinel-2
THEIAproduces and distributes level 2A data, corrected for atmospheric effects and provided with a good
cloud and cloud shadow mask. The method used to produce these data was developed in CESBIO and
implemented in the operational processor MACCS developed for CNES by CS-SI company.
Copernicus
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 49 of 56
Copernicus is the world's largest single Earth observation programme, directed by the European
Commission in partnership with the European Space Agency (ESA). It delivers core services that provide
standardised multi-purpose information common to a broad range of application areas relevant to EU
policies. Feeding these services are a series of Sentinel satellite missions (Figure 14). Each Sentinel
mission is based on a constellation of two satellites to fulfil revisit and coverage requirements, providing
robust datasets for Copernicus Services. A summary is seen in the figure below.
Figure 14 - Overview of Copernicus Sentinel satellites
• Copernicus Masters
The Copernicus Masters is an international competition, launched by AZO in 2011 on behalf of ESA, and
it awards prizes to innovative solutions for business and society based on Earth observation data.
• Sentinel Hub
In 2016 the Sentinel Hub, a cloud-based web service that allows non-expert user to browse, process and
download EO data, was the overall Copernicus Masters’ winner. The Sentinel Hub significantly simplifies
the way users can access and process satellite data. With applications like these, the competition has
again clearly demonstrated its potential to drive the innovative use of Earth observation data and make
the Copernicus programme accessible to new user groups. The Sentinel Hub is one of many portals
disseminating EO data.
Thematic Exploitation Platforms
In context of this growing volume of environmental data from space, enabled by the Sentinel missions,
along with the Copernicus Contributing Missions as well as Earth Explorers and other, Third Party
missions, in 2014 ESA started the EO Exploitation Platforms (EPs) initiative, a set of R&D activities that in
the first phase (up to 2017) aims to create an ecosystem of interconnected Thematic Exploitation
Platforms (TEPs) on European footing, addressing:
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 50 of 56
• Coastal,
• Forestry,
• Hydrology,
• Geohazards,
• Polar,
• Urban themes; and
• Food Security (under definition).
In short, an EO exploitation platform is a collaborative, virtual work environment providing access to EO
data and the tools, processors, and Information and Communication Technology resources required to
work with them, through one coherent interface. As such the EP, may be a new ground segments
operations approach, complementary to the traditional operations concept.
Data and Information Services (DIAS)
With the Copernicus Data Policy of open and free access to Sentinel products for any user in the current
operations context, it is projected that the volume of Sentinel data available to users will increase at a
rate of approximately 10PB per year once the Sentinels missions will have reached their full operational
capacity. While open distribution of the Copernicus data and information is actively promoted, the
European Commission (COM) and ESA aim at offering to users the capability to exploit Copernicus data
and information without the complexity and the cost associated to its transfer and storage on user side.
In this respect, the European Commission envisages that through the creation of DIAS, Copernicus data
and information will be routinely available to users through a cloud-based architecture.
CEOS-WGCV Atmospheric Correction Inter-Comparison Exercise (ACIX)
ACIX is an international collaborative initiative to inter-compare a set of atmospheric correction (AC)
processors for high-spatial resolution optical sensors, coordinated by ESA and NASA. The exercise will
focus on Landsat-8 and Sentinel-2 imagery over a set of test areas. The inter-comparison of the derived
BOA products is expected to contribute to the understanding of the different uncertainty contributors
and help in improving eventual ARD production. Participating algorithms/organisations are summarised
in Table 6.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 51 of 56
Table 6 - ACIX summary
Pilot production of Sentinel-2 Level-2A
In May 2017 ESA started the pilot production of Sentinel-2 Level-2A products over Europe. Level-2A
products are nominally published 48-60 hours after the publication of corresponding L1C products, on
the Copernicus Open Access Hub. Any potential gap in the production will be recovered automatically
within 4-5 days [28]. The pre-operational production of Sentinel-2 Level-2A atmospherically corrected data
over Europe (EEA-39) is done using Sen2Cor processor (Version 2.3.1). Sen2Cor is one of the algorithms
explored under ACIX and ESA’s next steps in relation to ACIX and the move from feasibility to global
production include [29]:
• Q2/2017: 2nd workshop ACIX for comparison metrics across 14 algorithms
• Q3/2017: Results from feasibility study L2A best European algorithm
• Q1/2018: Validated Global Reference Image (GRI): best geometric quality globally (<0.3 pixels @3
sigma multi-temporal)
• 2018: start of GRI-based production for S2A and S2B
• 2018: move from SEN2COR towards operational L2A processor for global production
• 2018: start major reprocessing exercise of all S2A/S2B archived data with GRI and operational L2A
processor (archive size ~5PB)
Organisation/
companyAC Processor
Input
DataOutput Format
Eric Vermote
Jean-Claude Roger
2 Jermoe Lours Telespazio Sen2Cor S2 JPEG2000 (*.jpg2)
3 Pflug Bringdred DLR Sen2Cor, ATCOR L8 & S2
4 Andre Hollstein GFZ SCAPE-M S2 JPEG2000 (*.jpg2)
5 Olg Dubovik LOA GRASP N/A
6 Steve Adler-Golden Spectral Sciences, Inc. FLAASH L8 & S2 N/A
7 Larry Leigh SDSU SMACAA L8 & S2 N/A
Erwin Wolters
Sterckx Sindy
Quinten Vanhelllmont
Kevin Ruddick
10 Antoine Mangin ACRI-HE LAC L8 & S2 N/A
11 Olivier Hagolle CNES MACCS L8 & S2 N/A
Fuqin Li
Lan-Wei Wang
13 David Frantz Trier University Ind-prepro L8 & S2 ENVI (*.hdr)
14 Grit Kirches Brockmann Consulting TBD L8 & S2 GeoTiff (*.tif)
9 RBINS ACOLITE L8 & S2 NetCDF, GeoTiff (*.nc, *.tif)
12 Geoscience Australia GA-PABT ENVI (*.hdr)
VITO OPERA L8 & S2 GeoTiff (*.tif)8
Data Format
Name
1 NASA/ UMd L8SR, S2SR L8 & S2 HDF (*.hdf)
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 52 of 56
APPENDIX C - A SUMMARY OF STANDARDS
As discussed throughout the present document, assigning an appropriate standard to a data product is a
reasonably straightforward process so long as it complies with relevant internationally-defined criteria.
International standards associated with EO products are very well established across the organisations,
however have not yet been defined within the international community for Sentinel data products. Many
activities are ongoing within this space, with an informal contrast between users that require a standard
to be unequivocally scientifically rigorous and would therefore be very difficult to comply with, compared
to the clear majority of users that require a standard that predominantly ensures consistency in
generated products that may be incorporated into value added services. It is the latter group of users
that will play the lead role in exploiting contemporary satellite datasets to offer unprecedented services
across applications and markets.
A summary of relevant standards that a UK Sentinel-2 ARD product should therefore adhere to is
provided below. It is by no means an exclusive list and includes those that are both formal and informal,
nationally and internationally.
It is clear, however, that a Sentinel-2 ARD standard product must comprise a minimum of two key
elements: (i) a standard associated with the accuracy and precision of corrected data comprising the
product, and (ii) a standard associated with the output product format, including associated metadata.
Therefore, individual standards that would be of relevance to these two elements are identified.
Through the current project, both accessible open source tool and open source datasets made available
to the community adhere to the key geospatial standards and are therefore appropriate for incorporation
into typical remote sensing and GIS systems. Part 2 of this report covers in detail the technical standard
to which the developed tool adheres, the published UK Sentinel-2 ARD dataset.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 53 of 56
Table 7 – Summary of Standards
Organisation Standard Reference
ISO ISO/TS 19159-1:2014 Geographic
information — Calibration and validation
of remote sensing imagery sensors and
data — Part 1: Optical sensors
https://www.iso.org/obp/ui/#iso:std:iso:ts:19159:-1:ed-
1:v1:en
ISO ISO/IEC 18026:2009 Information
technology — Spatial Reference Model
(SRM)
https://www.iso.org/obp/ui/#iso:std:iso-iec:18026:ed-
2:v1:en
ISO ISO 19115-2:2009 Geographic
information — Metadata — Part 2:
Extensions for imagery and gridded data
https://www.iso.org/obp/ui/#iso:std:iso-iec:18026:ed-
2:v1:en
ISO ISO/DIS 19115-2 Geographic information
— Metadata — Part 2: Extensions for
acquisition and processing
https://www.iso.org/obp/ui/#iso:std:iso:19115:-2:dis:ed-
2:v1:en
ISO ISO/TR 19120:2001 Geographic
information — Functional standards
https://www.iso.org/obp/ui/#iso:std:iso:tr:19120:ed-
1:v1:en
ISO ISO 19115-1:2014 Geographic
information — Metadata — Part 1:
Fundamentals
https://www.iso.org/obp/ui/#iso:std:iso:19115:-1:ed-
1:v1:en
ISO ISO/TS 19115-3:2016 Geographic
information — Metadata — Part 3: XML
schema implementation for fundamental
concepts
https://www.iso.org/obp/ui/#iso:std:iso:ts:19115:-3:ed-
1:v1:en
INSPIRE Orthoimagery Thematic Area http://inspire.ec.europa.eu/Themes/124/2892
INSPIRE Coordinate Reference Systems Thematic
Area
http://inspire.ec.europa.eu/Themes/130/2892
INSPIRE Geographical Grid http://inspire.ec.europa.eu/Themes/131/2892
OGC Earth Observation Metadata profile of
Observations & Measurements
http://docs.opengeospatial.org/is/10-157r4/10-
157r4.html#1
OGC OpenSearch Extension for Earth
Observation
http://docs.opengeospatial.org/is/13-026r8/13-
026r8.html
OGC Ordering Services Framework for Earth
Observation Products Interface Standard
https://portal.opengeospatial.org/files/?artifact_id=4392
8
UK GEMINI Metadata Standards http://www.agi.org.uk/agi-group/standards-
committee/uk-gemini
CEOS CARD4L – OSR TBD
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 54 of 56
REFERENCES
[1] “Sentinel-2 Overview,” [Online]. Available:
https://sentinel.esa.int/web/sentinel/missions/sentinel-2/overview.
[2] “Sentinel-2 Revist & Coverage,” [Online]. Available: https://sentinel.esa.int/web/sentinel/user-
guides/sentinel-2-msi/revisit-coverage.
[3] “Sentinel-2 Resolutions,” [Online]. Available: https://earth.esa.int/web/sentinel/user-
guides/sentinel-2-msi/resolutions.
[4] “Sentinel-2 Heritage,” [Online].
[5] “Sentinel-2 Product Types,” [Online]. Available: https://earth.esa.int/web/sentinel/user-
guides/sentinel-2-msi/product-types.
[6] “Copernicus Ground Segment Infrastructure,” [Online]. Available:
http://www.esa.int/Our_Activities/Observing_the_Earth/Copernicus/Ground_Segment_overview
[7] “Copernicus Open Access Hub,” [Online]. Available: https://scihub.copernicus.eu/dhus/#/home.
[8] C. Song, C. E. Woodcock, K. C. Seto and M. P. Lenney, “Classification and Change Detection Using
Landsat TM Data. When and How to Correct Atmospheric Effects?,” Remote Sensing of
Environment, 2001.
[9] N. Flood, T. Danaher, T. Gill and S. Gillingham, “An operational scheme for deriving standardised
surface reflectance from Landsat TM/ETM+ and SPOT HRG imagery for eastern Australia,” Remote
Sensing, pp. 83-109, 2013.
[10] T. Malthus and F. Li, “A technical handbook supporting calibration and validation activities of
remotely sensed data products,” 2015.
[11] A. S. Mahiny and B. J. Turner, “A Comparison of Four Common Atmospheric Correction Methods,”
Photogrammetric Engineering & Remote Sensing, 2007.
[12] A. Berk, G. P. Angerson, L. S. Bernstein, P. K. Acharya, H. Dothe, M. W. Matthew, S. M. Adler-
Golden, J. H. Chetwd, Jr., S. C. Richtmeier, B. Pukall, C. L. Allred, L. S. Jeong and M. L. Hoke,
“MODTRAN4 Radiative Transfer Modeling for Atmospheric Correction,” SPIE's International
Symposium on Optical Science, Engineering, and Instrumentation. International Society for Optics
and Photonics, 1999.
[13] D. Tanre, T. L. Deuze, M. Herman and J. Morcrette, “Second Simulation of the Satellite Signal in
the Solar Spectrum, 6S: An Overview,” IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE
SENSING, 1997.
[14] S. Adler-Golden and et al., “FLAASH, a MODTRAN4 atmospheric correction package for
hyperspectral data retrievals and simulations,” Proc. 7th Ann. JPL Airborne Earth Science
Workshop, 1998.
[15] P. Bunting and D. Clewley, “Atmospheric and Radiometric Correction of Satellite Imagery (ARCSI),”
[Online]. Available: http://rsgislib.org/arcsi/index.html.
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 55 of 56
[16] J. Roujean, M. Leroy and P. Deschamps, “A bidirectional reflectance model of the Earth's surface
for the correction of remote sensing data,” Journal of Geophysical Research: Atmospheres, 1992.
[17] F. e. a. Li, “An evaluation of the use of atmospheric and BRDF correction to standardize Landsat
data,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2010.
[18] A. H. e. a. Strahler, “MODIS BRDF/albedo product: algorithm theoretical basis document version,”
MODIS documentation.
[19] R. Richter, “Correction of satellite imagery over mountainous terrain,” Applied Optics, 1998.
[20] Zhang, “Calculation of radiative fluxes from the surface to top of atmosphere based on ISCCP and
other global data sets: Refinements of the radiative transfer model and the input data,” J.
Geophys. Res, 2004.
[21] “Sentine-2 L1C Cloud Masks,” [Online]. Available: https://sentinel.esa.int/web/sentinel/technical-
guides/sentinel-2-msi/level-1c/cloud-masks.
[22] Z. Zhu and C. E. Woodcock, “Object-based cloud and cloud shadow detection in Landsat imagery,”
Remote Sensing of Environment, 2012.
[23] “CEOS Working Group on Calibration & Validation,” [Online]. Available:
http://ceos.org/ourwork/workinggroups/wgcv/.
[24] “ESA's Sentinels Online,” April 2017. [Online]. Available:
https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-2/news/-/article/pilot-
production-of-sentinel-2-l2a-products-over-europe.
[25] “Sentinel-2 status as relevant to CARD4L discussion,” ESA, March 2017. [Online]. Available:
http://ceos.org/meetings/lsi-vc-3/.
[26] “Amazon Web Services & Sentinel-2,” [Online]. Available: http://sentinel-pds.s3-website.eu-
central-1.amazonaws.com/.
[27] “Google Earth & Sentinel-2 & Landsat Availability,” [Online].
[28] “Sentinel-2 & Landsat-8 Data Available on Planet Platform,” [Online].
[29] “ESA's Heterogeneous Missions Accessibility (HMA) work yields EO Satellite Ground Segment
Interoperability Manual,” [Online]. Available: http://www.opengeospatial.org/blog/1662.
[30] Bunting, P., Clewley, D., Lucas, R. M., & Gillingham, S. (2014). The Remote Sensing and GIS
Software Library (RSGISLib). Computers and Geosciences, 62, 216–226.
[31] Chander, G., Markham, B. L., & Helder, D. L. (2009). Summary of current radiometric calibration
coefficients for Landsat MSS, TM, ETM+, and EO-1 ALI sensors. Remote Sensing of Environment,
113(5), 893–903.
[32] Dymond, J. R., & Shepherd, J. D. (2004). The spatial distribution of indigenous forest and its
composition in the Wellington region, New Zealand, from ETM+ satellite imagery. Remote Sensing
of Environment, 90(1), 116–125.
[33] Flood, N., Danaher, T., Gill, T., & Gillingham, S. (2013). An Operational Scheme for Deriving
Standardised Surface Reflectance from Landsat TM/ETM+ and SPOT HRG Imagery for Eastern
Australia. Remote Sensing, 5(1), 83–109. http://doi.org/10.3390/rs5010083
Developing standards and automated production for Sentinel-2 Analysis Ready Data
Status: Final
Version: v0.1
Date: 18/10/2017
Page 56 of 56
[34] Shepherd, J. D., & Dymond, J. R. (2003). Correcting satellite imagery for the variance of
reflectance and illumination with topography. International Journal of Remote Sensing, 24(17),
3503–3514.
[35] Tripathi, S. N., Dey, S., Chandel, A., & Srivastava, S. (2005). Comparison of MODIS and AERONET
derived aerosol optical depth over the Ganga Basin, India. Annales Geophysicae, 1093–1101.
[36] Vermote, E. F., Saleous, N. Z., & Privette, J. L. (2006). Surface reflectance Earth system data
record/climate data record White paper. Nasa.
[37] Zhu, Z., & Woodcock, C. E. (2012). Object-based cloud and cloud shadow detection in Landsat
imagery. Remote Sensing of Environment, 118, 83–94.