Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high...
Transcript of Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high...
![Page 1: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/1.jpg)
Issue/Revision: 1.0
Reference: Gaia Archive – VO inside
Status: Issued
ESA UNCLASSIFIED - Releasable to the Public
Gaia Archive operations for DR1
J. González-Núñez, J. Salgado, R. Gutiérrez-Sánchez, J.C. Segovia, J. Durán, C. Arviset, R. Alvarez, R. Gil, E. Anglada, J. Bakkers, U. Lammers ESA Science Data Centre 23/10/2016
![Page 2: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/2.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 2
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
14th September 2016
![Page 3: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/3.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 3
ESA UNCLASSIFIED - Releasable to the Public
The AS Role
![Page 4: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/4.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 4
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
High level scalability measures Ø Replication of data with Associate & Affiliate data centres
Ø AIP, ARI, ASDC, CDS, IRSA, GAVO
Ø Bulk download infrastructure based on a Content Delivery Network
![Page 5: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/5.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 5
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
![Page 6: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/6.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 6
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
RDBMS
Distributed Data Repository
science archive
TAP+
Browser GUI
TAP+, VOSpace
Programmatic
VO Apps
TAP+
VO s
ervi
ces
SAMP
Euro-VO Registry
![Page 7: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/7.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 7
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Scaling up S*AP vs scaling up TAP Ø Potentially large processing time for requests Ø Requests can be quite CPU intensive, both front-end and DB sides
Scaling up TAP+ Ø Persistent uploads Ø Server-side crossmatches
![Page 8: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/8.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 8
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Cloud vs dedicated HW Ø Dedicated HW can be selected and configured in ways Cloud of virtualized
infrastructures can’t Ø But it sets limits to scablability
Infrastructure dimensioning Ø Conservative engineering considering worst case scenarios Ø High performance hardware Ø Measures to ensure graceful service degradation if capacity is exceeded
![Page 9: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/9.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 9
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Stress test campaign Ø Stress test plan defined covering all server-side elements of the Gaia
Archive developed at ESAC (GACS), ie. TAP+ interface Ø Built with support of IT support team (SITU) & vendors engineering
teams support. Ø Executed in 4 iterations (IT1, IT2, IT3, IT4) with an increasing level of
systems monitoring Ø Full system performance analysis at each Iti Ø Incremental system infrastructure implemented to IT(i+1)
Ø Tested high data volume scenarios, with good performance up to network limits Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation factor)
![Page 10: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/10.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 10
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Resource limits
Sync rows
Async rows
Sync time
Async time
DB space
Job results
Anonymous 100K 100K 1min 30min none N/A (np)
Registered unlimited unlimited 1min* 30min* 1GB* 1GB*
(np) not persistent * Can be upgraded on demand
![Page 11: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/11.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 11
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Job Scheduling
![Page 12: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/12.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 12
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
![Page 13: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/13.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 13
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
Archive UI usage: First 24 hours
• Usage sessions: 12,005 • # of users: 10,959
UI Usage
![Page 14: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/14.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 14
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
File CDN download: First 15 days
• Total volume downloaded: 73 TB • # download requests: 9 millions
File download
![Page 15: Gaia Archive ops DR1 - IVOAwiki.ivoa.net/internal/IVOA/InteropOct2016Ops/Gaia... · Ø Tested high CPU consumption scenarios, with efficient use of archive resources (up to 80% occupation](https://reader033.fdocuments.in/reader033/viewer/2022050203/5f570f85814c477c64141ce9/html5/thumbnails/15.jpg)
ADASS XXVI - Trieste | Gaia Archive DR1 Operations | | 23/10/2016 | Slide 15
ESA UNCLASSIFIED - Releasable to the Public
Ops for Gaia DR1
• Number of queries • Synchronous: 174,807 • Asynchronous: 95,090
TAP Interface
TAP Interface: First 15 days