ATLAS Rucio DB characteristics
Transcript of ATLAS Rucio DB characteristics
![Page 1: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/1.jpg)
ATLASRucioDBcharacteristics(basedontheOracleRDBMS)
Ruciocommunityworkshop,Oslo,Feb-March2019GanchoDimitrov(CERN)
![Page 2: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/2.jpg)
Aboutme
1
• MemberoftheATLASdatabasegroupsince2006
• Acquired(some)DBrelatedknowledgethroughoutthelast15+years
• Mainfocuson:Ø datamanagementØ databaseschemadesignØ databaseperformancetuning
• CertifiedinOracleRDBMS
• RegularspeakeratOracleUserGroupcommunityconferencesinUK,BulgariaandGermany
RucioCommunityWorkshop,Oslo,Feb2019
![Page 3: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/3.jpg)
Rucioarchitecture
2RucioCommunityWorkshop,Oslo,Feb2019
![Page 4: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/4.jpg)
![Page 5: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/5.jpg)
Ruciodatabaseschema
4
Designedin2013
RucioCommunityWorkshop,Oslo,Feb2019
![Page 6: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/6.jpg)
Ruciodatabaseschema
5
61tables
RucioCommunityWorkshop,Oslo,Feb2019
![Page 7: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/7.jpg)
Ruciodatabaseschema
6
74indices
RucioCommunityWorkshop,Oslo,Feb2019
![Page 8: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/8.jpg)
Ruciodatabaseschema
7
5triggers
RucioCommunityWorkshop,Oslo,Feb2019
![Page 9: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/9.jpg)
Ruciodatabaseschema
8
3functions
RucioCommunityWorkshop,Oslo,Feb2019
![Page 10: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/10.jpg)
Ruciodatabaseschema
9
12procedures
RucioCommunityWorkshop,Oslo,Feb2019
![Page 11: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/11.jpg)
Ruciodatabaseschema
10
17jobs
RucioCommunityWorkshop,Oslo,Feb2019
![Page 12: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/12.jpg)
Tablesegments
11
Physicallyorganised basedonRucio’s scopeunit
scopedata=tablepartition**separatesegment(providesflexibilityinsetofoperations)
RucioCommunityWorkshop,Oslo,Feb2019
![Page 13: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/13.jpg)
Rucioscopegeneration
12
In2014:5140scopes(systemstartup)
2015-2018=~400peryear
RucioCommunityWorkshop,Oslo,Feb2019
![Page 14: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/14.jpg)
Currently
13
About7000scopesaremappedto34500table
partitions
RucioCommunityWorkshop,Oslo,Feb2019
![Page 15: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/15.jpg)
DIDS(DataIDentifierS)*
14
1014million(AVGrowsize:220bytes)
*Theprimaryaddressableunitofdata(files,datasets,containers)
RucioCommunityWorkshop,Oslo,Feb2019
![Page 16: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/16.jpg)
DIDSreplicasontheGrid
15
1106million(AVGrowsize:150bytes)
RucioCommunityWorkshop,Oslo,Feb2019
![Page 17: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/17.jpg)
Rucio<-->DBinteraction
16
SQLcommandsgeneratedby
SQLAlchemy ORM*
*Object-RelationalMapperRucioCommunityWorkshop,Oslo,Feb2019
![Page 18: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/18.jpg)
DBsessions
17
1100-1400DBsessions15-25activesessions
RucioCommunityWorkshop,Oslo,Feb2019
![Page 19: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/19.jpg)
Rucioinsert/update/deleterate
18
~2-4Kexec/sec
RucioCommunityWorkshop,Oslo,Feb2019
![Page 20: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/20.jpg)
RucioDBqueriesrate
19
~4-8Kexec/sec2Mblockreads(15GB)/sec
RucioCommunityWorkshop,Oslo,Feb2019
![Page 21: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/21.jpg)
DBserverCPUusage
20
~30-40%
RucioCommunityWorkshop,Oslo,Feb2019
![Page 22: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/22.jpg)
RucioDBvolumegrowth2016-2019
21RucioCommunityWorkshop,Oslo,Feb2019
![Page 23: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/23.jpg)
April-June2019
22
Excitingperiod:
TestandvalidationonOracle18c
DBserverupgradefrom11gto18c
RucioCommunityWorkshop,Oslo,Feb2019
![Page 24: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/24.jpg)
Spring2020
23
ExpectedtohavenewDBhardware
RucioCommunityWorkshop,Oslo,Feb2019
![Page 25: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/25.jpg)
RucioDBschemaextension
24
EnrichRucioDIDSwithmetadatastoredinJSON*
*CERNcomputingseminaronDBbuilt-inJSONfeaturesplannedfor15th May2019
RucioCommunityWorkshop,Oslo,Feb2019
![Page 26: ATLAS Rucio DB characteristics](https://reader031.fdocuments.in/reader031/viewer/2022011913/61d76bb35d2d5d252f617164/html5/thumbnails/26.jpg)
Take-homemessages• RucioDBsetuponOracleRDBMSkeepsonperformingwellseveralconsecutiveyears.•WorkforsmoothDBserverupgradeisnecessary• PlannedextensionofthecurrentDBschematosupportDIDSmetadatainJSON
SpecialthankstotheRucioteamforthecollaborativeworkintheyears.
WelltunedDB=goodnight’ssleepforall!25RucioCommunityWorkshop,Oslo,Feb2019