The National Grid Service User Accounting System
Katie Weeks
Science and Technology Facilities Council
Katie Weeks – [email protected] 13th September 2007
The National Grid Service
• The NGS is the UK’s grid for academics• Free and easy to apply online• Providing a number of heterogeneous compute and data
resources on the grid• Providing help and support to those using the grid • The NGS is led and coordinated by the STFC in
collaboration with the University of Manchester, the University of Oxford, the University of Edinburgh and the White Rose Grid at the University of Leeds
Katie Weeks – [email protected] 13th September 2007
The National Grid Service
• Over 590 registered users
• 4 core sites, 6 partner sites and 3 affiliate sites
• Entered production in Autumn 2004
• Now in phase 2
• Predominantly focussed on compute and storage at present
Katie Weeks – [email protected] 13th September 2007
Grid Accounting
• Accounting for any production grid is an important part of the monitoring process– Pricing policies may be introduced to grids in the future– To uphold policies relating to grid use and allocated hours– To monitor systems – particularly important for funding and
future planning– To have an overview of the system – how much are we
allocating? How much is being used? How much spare capacity do we have? How much are our biggest users using?
• JISC has recently funded a review of accounting and usage monitoring
• It’s an issue many grids now face
Katie Weeks – [email protected] 13th September 2007
Grid policing
• Users are allocated limited resources• Important to know how much of those resources
have been consumed• Users tend to go over quota even when
monitored• Need to ‘lock-out’ users who go over quota• There is an important distinction between
accounting and policing• Retain integrity of application and peer-review
process
Katie Weeks – [email protected] 13th September 2007
Our problem
• Situation at beginning of 2006:– Applications under go light-weight peer review
process– Ability to monitor individual usage– Policies in place for usage– No tools to use the monitoring data to enforce the
policies– Needed a system to enable us to manage users and
enforce allocation policies
Katie Weeks – [email protected] 13th September 2007
Other solutions
• APEL (EGEE)– Accounting Processor for Event Logs– Accounting information such as CPU time, Wall Time and DN– Virtual Organisation emphasis with no individual policing
• TeraGrid– Account monitoring available via command line– No architecture in place for automatic policing
• OGF– Resource Usage Service (RUS)– Stores monitoring information that can be queried– NGS has had a RUS instance since early 2006
No current solution met all the needs of the NGS – needed our own solution
Katie Weeks – [email protected] 13th September 2007
Our Solution
• Resource usage data obtained by querying the Resource Usage Service (RUS) at Manchester
• RUS already in place and had functionality the NGS required• Accounting system based on an Oracle database• Interface to the Oracle database was created using Oracle
Application Express (Apex)• Information collected
– User details– Application information– Account status– Resource allocation– Resource usage
• Usage and allocation is collated over all core sites• Records historical information• Records changes in Distinguished Name (DN)
Katie Weeks – [email protected] 13th September 2007
Katie Weeks – [email protected] 13th September 2007
Policing the NGS
• User Accounting System (UAS) queries the RUS every day for total CPU and disk space for every user
• A warning email is sent out when you reach 90% of your CPU allocation
• The account is automatically locked and an email sent when you reach 100% of your CPU allocation
Katie Weeks – [email protected] 13th September 2007
Policing the NGS (2)
• The Lightweight Directory Access Protocol (LDAP) queries the User Accounting System every hour
• It’s only populated with users whose account has ‘active’ status
• Users whose accounts are locked or pending are not included in the LDAP
• Maximum of hour before accounts are active again once they’ve been unlocked
Katie Weeks – [email protected] 13th September 2007
Policing the NGS (3)
• When an account is locked, you can apply for more resources– Via application form– Via your account details
• When your application is successful, your account is automatically updated with your new allocation and account is ‘active’ again
• An email is sent to you letting you know you’re back within your limits
• Your account will be active within the hour once the LDAP has queried the UAS
Katie Weeks – [email protected] 13th September 2007
Integration with other systems
• NGS helpdesk is run using the commercial application Footprints
• Application process went through Footprints• Reviewers were used to Footprints• UAS sends applications directly to Footprints as well as
putting them in the database• System unchanged from reviewers point of view• Systems need to be synchronised
– Approved in Footprints so ticket can be closed– Approved in UAS so account is active
• Far from ideal situation
Katie Weeks – [email protected] 13th September 2007
Using Oracle and Apex
• Oracle is robust, scalable and efficient• Oracle is used for hosting projects on the NGS
e.g. CPOSS data• Expertise, experience and Oracle support
already available to us• Apex allows web forms and reporting to be done
very quickly• Apex also allows graphs to be produced
dynamically and data queried in a variety of ways
Katie Weeks – [email protected] 13th September 2007
Accessing your details
• Users wanted to know how much of their allocation they had used
• https://www.ngs.ac.uk/useraccountinfo.php • Certificate access to account details
– Not supported by Oracle Apex– Needed a workaround to take certificate details from
browser• Also provides ability to change contact details • Renewals can be done through their own
account
Katie Weeks – [email protected] 13th September 2007
Katie Weeks – [email protected] 13th September 2007
Katie Weeks – [email protected] 13th September 2007
NGSUser
PK NGSUserID
SRBUserName OracleUserName DN Email Name AddressFK1 ProjectID AccountStatusStartDateFK2 AccountStatusTypeID Institution Telephone Funding InfoSource
NGSUserApplication
PK NGSUserApplicationID
FK1 NGSUserID ApplicationDate Case
ResourceApplication
PK ResourceApplicationID
FK1 ResourceTypeIDFK2 NGSUserApplicationID Limit
ResourceType
PK ResourceTypeID
Description
ResourceAllocation
PK ResourceAllocationID
FK2 ResourceTypeIDFK1 NGSUserID Limit StartDateFK3 ResourceApplicationID
ResourceAllocationHistory
PK ResourceAllocationHistoryID
FK1 ResourceTypeIDFK2 NGSUserID Limit StartDate EndDateFK3 ResourceApplicationID
AccountStatus
PK AccountStatusID
Description
NGSUsageRecord
PK NGSUsageRecordID
FK1 NGSUserIDFK2 ResourceTypeID UsageDate CumulativeValue
ResourcePolicing
PK ResourcePolicingID
FK2 NGSUserIDFK1 PolicingStateTypeIDFK3 ResourceTypeID StartDate
PolicingStatus
PK PolicingStatusID
Description
ResourcePolicingHistory
PK ResourcePolicingHistoryID
FK1 NGSUserIDFK3 PolicingStateTypeIDFK2 ResourceTypeID StartDate EndDate
NGSUserAccountStatusHistory
PK NGSUserAccountStatusHistoryID
FK1 NGSUserIDFK2 AccountStatusTypeID StartDate EndDate
Project
PK ProjectID
DescriptionFK1 NGSUserID
NGSUsageRecordHistory
PK NGSUsageRecordHistoryID
FK1 NGSUserID StartDate EndDateFK2 ResourceTypeID CumulativeValue
NGSApplicationReviewComments
NGSApplicationID CommentDate NGSReviewer ReviewCommentsFK1 NGSUserApplicationID
NGSUserCertificate
PK NGSUserCertificateID
FK1 NGSUserID DN
X509_authorisation
PK session_id
FK1 ngsuserid ipaddress time
Katie Weeks – [email protected] 13th September 2007
Problems
• Limitation on field length restricted case entries and caused some problems when users wrote more than 2200 characters
• Oracle Apex doesn’t support X509 certificate access – our own workaround had to be implemented for the user to access their account details
Katie Weeks – [email protected] 13th September 2007
Results
• Entered production in October 2006• Begun locking out users at 10,000% over quota • Gradually reduced to 100% of quota• 64 accounts have been locked• 73% have successfully reapplied• Over 150 new users have applied using the
system
Katie Weeks – [email protected] 13th September 2007
Future Work
• The UAS is continuously developing• Peer-review process is being integrated into the
Oracle system• Incorporation of external system that creates the
SRB accounts and subscribes users to JISCmail.
• Inclusion of accounting of storage usage by users
• Extension to support Virtual Organisations
Katie Weeks – [email protected] 13th September 2007
Summary
• Oracle and Apex offered the best solution for the NGS UAS
• Monitoring and policing accounts is all done automatically now
• Application process is handled automatically
• Room for expansion of the system
Katie Weeks – [email protected] 13th September 2007
Further information
• Ask a question now!
• Talk to me after the presentation
• Visit the NGS stand (booth 20)
• Contact the NGS support centre [email protected]
• Visit the NGS website www.ngs.ac.uk