Aziende veronesi in ritardo nell'uso del web marketing 19-05-2011
Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene...
-
Upload
amie-matthews -
Category
Documents
-
view
212 -
download
0
Transcript of Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene...
Recent improvements in HLRmon, an accounting portal suitable for
national Grids
Enrico Fattibene (speaker), Andrea Cristofori,Luciano Gaido, Paolo Veronesi
INFN - Italyhlrmon<at>lists.infn.it
WORLDCOMP GCA’11 – Las Vegas, July 18th, 2011
Outline
• gLite Grid infrastructure
• Grid accounting issues
• Accounting layout in the Italian Grid Infrastructure (IGI)
• HLRmon overview
• HLRmon main report section
• Storage accounting prototype
• Outlook and conclusions
2HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
gLite Grid infrastructure
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 3
CECE
WMSWMS
BDIIBDII
Accounting ServiceAccounting Service
Grid site A
SESE
CECE
Grid site B
SESE
• The Computing Element (CE) is the service representing a computing resource• The Storage Element (SE) is the storage unit at Grid site level• CEs and SEs send information about the available resources and their status to the Berkley
Database Information Index (BDII)• Grid user can submit jobs:
through the Workload Management System (WMS) that retrieves information on the available resources from the BDII and sends the job to the appropriate CE
directly to the CE
• The accounting service is responsible for the collection of computing and storage information
Grid user
Grid accounting issues
• Resource usage accounting in Grid is challenging due to the distributed nature of the infrastructure and
the heterogeneous nature of the resources accessed necessary for its stakeholders, such as site administrators,
Virtual Organization (VO) managers, Grid managers, etc.
• Users need to access accounting information at different levels of aggregation an exhaustive and easy to use graphical interface is
fundamental many views are needed for different stakeholders
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 4
Accounting layout in IGI
5HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
• Distributed Grid Accounting System (DGAS) sensors collect accounting information at Grid site level
• Site data are sent to site or multisite Home Location Registers (HLRs)• A top-level HLR receives Usage Records (URs) from all the Grid HLRs • HLRmon retrieves data from the top level HLR and presents them through a web interface.
Aggregated data are also sent to the European Grid Infrastructure (EGI) Accounting Portal
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 6
HLRmon introduction
• Developed and funded in the framework of the INFN Grid project
• Deployed in the Italian production Grid since 2007• Choosen as the accounting portal by gLite-based
National Grid Infrastructures (NGIs) and projects IGI (Italy) HellasGrid (Greece) D-Grid (Germany) W-eNMR project Cybersar project
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 7
HLRmon structure• HLRmon collector retrieves URs from the desired DGAS HLR server
• Data are aggregated, stored in a local MySQL database and presented through a php-based web portal charts highly customizable by the users tabular reports available also in xls format
• A x.509 personal certificate is needed to access the portal
• The web interface includes: a main report section providing usage information aggregated in many
different ways a section reporting on resource usage by VOs, aggregated by the
corresponding scientific disciplines a section dedicated to the Worldwide LHC Computing Grid (WLCG)
sites and VOs (alice, atlas, cms, lhcb) a restricted section providing accounting per-user information, visible
only by registered and authorized users
8
Main section• Different sets of charts
Number of executed jobs CPU Time (CPT) and Wall Clock Time (WCT) row and normalized, expressed in
time (days) o the normalization is based on the KiloSpecInt2000 benchmark
Job efficiency in terms of (CPT / WCT) ratio
• Different keys for data aggregation and selection Grid site VO, group and role
o groups and roles can be granted to perform specific tasks within a VO Certification Authority (CA) and Registration Authority (RA)
o that released the x.509 personal certificate used by VO member to authenticate him/herself to the Grid
Job type (Grid or local)
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
Main section overview
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 9
10
Main section charts
Normalized CPTexpressed in KSI2K-dayper VO and per VO per day
Job efficiency (CPT/WCT) per site per day and per VO per day
Normalized CPTexpressed in KSI2K-dayper site and per site per day
10HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
VO / groups / roles menuVOComplete list of VOs with at least 1 job into the HLRmon DB
VO groupsFor each VO, the VO groups with at least 1 job are listed
VO rolesFor each VO group, the VO roles with at least 1 job are listed
11HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
VO groups / roles charts
Job number per VO groupJob number per VO group per day
Job number per VO roleJob number per VO role per day
12HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
CA / RA menu
CAThe CA list depends on the HLRmon server configuration.
RAThe IGI HLRmon server is configured to aggregate data per RA. For the INFN CA, the RA information is extracted from the L= field of the DN.
13HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
CA / RA charts
Job number per CAJob number per CA per day
Job number per RAJob number per RA per day
14HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
Storage accounting approach• Storage accounting system integrated in DGAS under development• Non standard schema has been used to generate storage URs
no standard schema exists we join the Open Grid Forum (OGF) working group on the storage UR
definition
• The information on the used space is collected from the BDII once a day specific sensors can be developed to retrieve more accurate data
o making the system independent from the BDII
• Storage URs are stored on the HLR server as the computing URs• HLRmon is able to retrieve storage information directly from the
desired HLR
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 15
Storage report
• HLRmon shows the daily disk consumption in terms of used space
• Data are aggregated per site, SE and Storage Area (SA)• Used space will be compared to the total available space (at
SA, SE and site level)• The plotting of these data allowed to detect wrong values
published in the BDII sites have been notified in order to correct the values
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 16
Storage charts
Used space per siteUsed space per site per day
Used space per SAUsed per SA per day
17HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011
Outlook
• The storage accounting prototype will evolve in order to adopt a standard UR when available we will follow this discussion into the OGF UR working group
• HLRmon will be made independent from the underlying accounting system accounting services adopted in EGI are evolving to transport their
data through ActiveMQ messaging system HLRmon data collector will be improved to retrieve accounting
data from a common interface, where EGI accounting services will publish their URs
HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 18
Conclusions• HLRmon
is developed to meet requirements from different kind of users is successfully deployed as accounting portal for IGI and other Grid
projects/NGIs provides a complete section on computing accounting reports offers a fine granularity of data aggregation is close to provide a complete storage accounting section can be adopted to visualize data taken from other accounting systems
• Documents, instance list, installation guide, video tour, etc.:http://www.italiangrid.org/grid_operations/tools/accounting_portal/HLRmon
19HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011