WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre...
description
Transcript of WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre...
![Page 1: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/1.jpg)
IT-SDC : Support for Distributed Computing
WLCG Transfers Dashboard A unified monitoring tool for
heterogeneous data transfers.
Alexandre Beche<[email protected]>
![Page 2: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/2.jpg)
2IT-SDC
Outlines Mission History of data transfer monitoring tools Data volume and retention policy Main improvements Future work Application usage
12 February 2014Alexandre Beche – GDB
![Page 3: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/3.jpg)
3IT-SDC
ATLASFTS + XRootD
CMSFTS + XRootD
LHCbFTS
ALICEXRootD
Mission
12 February 2014Alexandre Beche – GDB
![Page 4: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/4.jpg)
4IT-SDC
ATLASFTS + XRootD
CMSFTS + XRootD
LHCbFTS
ALICEXRootD
Mission
12 February 2014Alexandre Beche – GDB
![Page 5: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/5.jpg)
5IT-SDC
WLCG Transfers Dashboard User Interface
12 February 2014Alexandre Beche – GDB
![Page 6: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/6.jpg)
6IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WLCG
FTS
WLCGTransfers
Dashboard
12 February 2014Alexandre Beche – GDB
![Page 7: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/7.jpg)
7IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WLCG
FAX
AAA
FTS
WLCGTransfers
Dashboard
XRootDIntegration
12 February 2014Alexandre Beche – GDB
![Page 8: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/8.jpg)
8IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
WLCG FAX AAA
FAX
AAA
FTS
WLCGTransfers
Dashboard
XRootDIntegration
XRootDDetailed
Dashboard
12 February 2014Alexandre Beche – GDB
![Page 9: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/9.jpg)
9IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
WLCG FAX AAA
FAX
EOS
AAA
EOS
FTS
WLCGTransfers
Dashboard
XRootDIntegration
XRootDDetailed
Dashboard
EOSIntegration
12 February 2014Alexandre Beche – GDB
![Page 10: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/10.jpg)
10IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
WLCG FAX AAA
FAX
EOS
AAA
EOS
FTS
ALIC
EWLCG
TransfersDashboard
XRootDIntegration
XRootDDetailed
Dashboard
EOSIntegration
ALICEIntegration
12 February 2014Alexandre Beche – GDB
![Page 11: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/11.jpg)
11IT-SDC
WLCG Transfers DashboardFederated approach
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
FTS FAX AAA
FAX
EOS
AAA
EOS
FTS
ALIC
E
WEBAPI
ALICE
12 February 2014Alexandre Beche – GDB
![Page 12: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/12.jpg)
12IT-SDC
WLCG Transfers DashboardFederated approach
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
FTS FAX AAA
FAX
EOS
AAA
EOS
FTS
ALIC
E
WEBAPI
ALICE
WLCG Transfers DashboardAPI / UI
12 February 2014Alexandre Beche – GDB
![Page 13: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/13.jpg)
13IT-SDC
Data Volumesince June 2012
Data are split into 3 schemas FTS, XRootD (AAA and FAX) High optimization (Partition, no Global indexes, …)
12 February 2014Alexandre Beche – GDB
![Page 14: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/14.jpg)
14IT-SDC
Data Volumesince June 2012
# ROWS (million) Space (GB)FTS 648 (+7.2/day) 274 (+3/day)
XRootD (incl. local accesses) 2,283 (+4.2/day) 636 ( +1.2/day)
Data are split into 3 schemas FTS, XRootD (AAA and FAX) High optimization (Partition, no Global indexes, …)
12 February 2014Alexandre Beche – GDB
![Page 15: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/15.jpg)
15IT-SDC
Data Volumesince June 2012
# ROWS (million) Space (GB)FTS 648 (+7.2/day) 274 (+3/day)
XRootD (incl. local accesses) 2,283 (+4.2/day) 636 ( +1.2/day)
Data are split into 3 schemas FTS, XRootD (AAA and FAX) High optimization (Partition, no Global indexes, …)
Snapshot on Tuesday 11 Feb 2014 Shows the usage of the DB at this specific time No conclusion can be drawn on technology usage
See next slide on data retention policy
12 February 2014Alexandre Beche – GDB
![Page 16: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/16.jpg)
16IT-SDC
Data retention policy Two kind of data:
Raw: Kept for 3 months for FTS Kept indefinitely for AAA and FAX
Statistics: Different granularity (10 minutes, daily) Less than 50GB of data Kept indefinitely
12 February 2014Alexandre Beche – GDB
![Page 17: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/17.jpg)
17IT-SDC
Main improvements
ALICE traffic is now included No more duplication at the raw data
level WLCG view is an aggregation over APIs
FTS Dashboard is much faster Does not have the XRootD overhead
Long time period can be queried See next slide
12 February 2014Alexandre Beche – GDB
![Page 18: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/18.jpg)
Alexandre Beche 18IT-SDC 27 May 2013
Activity since June 2012
Impossible with the old architecture
![Page 19: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/19.jpg)
19IT-SDC
Future work Improve the integration of
technology specific dashboards
12 February 2014Alexandre Beche – GDB
![Page 20: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/20.jpg)
20IT-SDC
Future work Integration of the map view
Already available for AAA, FAX and FTS
12 February 2014Alexandre Beche – GDB
![Page 21: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/21.jpg)
21IT-SDC
Application usageincluding WLCG Home page
12 February 2014Alexandre Beche – GDB
![Page 22: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/22.jpg)
22IT-SDC
Summary Completely new architecture
Greater flexibility Easier integration of heterogeneous data
Optimized technology-specific dashboards Aggregated together inside a top level UI
Thanks for your attention
12 February 2014Alexandre Beche – GDB
![Page 23: WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < Alexandre.beche@cern.ch>](https://reader035.fdocuments.in/reader035/viewer/2022062521/568168d5550346895ddfc6d5/html5/thumbnails/23.jpg)
23IT-SDC
Useful links
AAA Dashboard http://dashb-cms-xrootd-transfers.cern.ch
FAX Dashboard: http://dashb-atlas-xrootd-transfers.cern.ch
FTS Dashboard: http://dashb-fts-transfers.cern.ch
WLCG Transfers Dashboard: http://dashb-wlcg-transfers.cern.ch
12 February 2014Alexandre Beche – GDB