Engineering at Illinois IT Governance: Research Working Group

Final Report On The Committee’s Activities During The 2016-2017 Academic Year

Members:

Narayana Aluru (MechSE) Jennifer Bernhard (Engineering Administration) Daniel Bodony (AE), Chair Davide Curreli (NPRE) Jim Hurst (Engineering Administration), ex officio Mike Insana (BioE) Andreas Kloeckner (CS) Kim Nguyen-Jahiel (Engineering Administration) Andre Schleife (MatSE)

Executive Summary The Engineering IT Governance, Research Working Group reviewed the current state of ENGR-IT as it supports research across the CoE as well as identifying challenges. Through interviews, presentations, and dialog with ENGR-IT and CoE members we find that the status of computational research support provided to CoE members needs emphasis in two areas: (1) software training and (2) available computational resources provided to all CoE members. A software training plan is proposed where Computational Science and Engineering (CSE) becomes the lead unit to provide training throughout the CoE. CSE is encouraged to bundle training whenever feasible and generate content when required to improve the CoE computing landscape. CSE and the College are recommended to negotiate a long-term vision for CSE and firmly define CSE’s mission. Concomitant with CSE’s role in software training, the loss of the CSE-enabled Taub portion of the Illinois Campus Cluster Program (aka “Campus Cluster”) meant the loss of CoE-wide free computing resources larger than single workstation. We propose that the CoE become an investor in the Campus Cluster and establish different resource minimums for different groups of CoE members. Estimates are provided on how large of an investment should be made and how the CoE might choose to pay for the investment.

1

1. Introduction At the request of Associate Dean Roy Campbell, a committee representing CoE faculty, CoE 1

administrators, and ENGR-IT was formed to conduct a strategic-focused assessment of the 2

CoE’s research computing needs during the 2016-2017 academic year. (A copy of the Associate Dean’s charge letter appears in Appendix A.) From the charge letter: “Your recommendations and comments are advisory to the Associate Dean for Information Technology, with primary responsibilities for:

● Reviewing policies concerning the sustainability and provision of any needed research IT that enhances the College's research mission, both at the Engineering College level and at the campus level.

● Considering IT support for new research pursuits in our College, anticipating new opportunities and responding to interests and initiatives from our units.

● Monitoring, assist with prioritizing, and provide feedback on the implementation of any proposals concerning research IT.

● Assist with monitoring, providing feedback, and reporting on the progress of work to align Engineering IT services with campus-level IT services, in particular with respect to research.

● Coordinating and communicating with the other IT working groups over overlapping interests. You will also be asked to provide an assessment at the end of the academic year on the current state of IT support for research activities.”

The Research WG (RWG) committee was directed to report to, and be in frequent communication with, the ENGR-IT Governance Executive Advisory Committee, chaired by Associate Dean Roy Campbell. On May 24, 2017, Associate Dean Campbell directed the RWG to consider the following topics in its final report:

● “Assessment or report of progress on campus Research IT initiatives and value to Engineering

○ (Whatever Towns has managed to accomplish out of his research IT proposal with no funding)

● “Recommendations or findings related to the proposal for computational literacy training

1 College of Engineering 2 Engineering IT Shared Services

2

● “Recommendations or findings related to the provision of high-performance computing resources for the College (Taub replacement)

● “Summary assessment of current state of IT services for research” This report summarizes the committee’s actions, findings, and recommendations.

3

2. Actions of the Committee A summary of the RWG’s activities is below:

● Face-to-face meetings were held on: October 10, 2016; October 24, 2016 (hosted Dr. Neal Davis (CS) to discuss software training); November 14, 2016; December 12, 2016; February 10, 2017 (joint with NCSA; hosted Mr. John Towns (Deputy CIO, NCSA ) and 3

Mr. Tim Boerner (NCSA, ICCP )) 4

● Collaboration app Slack used for virtual meetings from March 6, 2017, through

submission date of this report.

● The committee hosted the following guests:

○ Dr. Neal Davis, Teaching Assistant Professor (CS), on October 24, 2016. ○ Mr. John Towns, Executive Director for Science and Technology, NCSA and

Principal Investigator of the NSF XSEDE Project , and Deputy CIO for Research 5

IT for Technology Services, on February 10, 2017. ○ Mr. Tim Boerner, Center Network Engineering Lead for NCSA and Operations

Manager for the ICCP In addition, the committee requested information directly from ENGR-IT from Mr. Jim Hurst (Director, ENGR-IT).

3 National Center for Supercomputing Applications 4 Illinois Campus Cluster Program, a.k.a. “Campus Cluster” 5 National Science Foundation Extreme Science and Engineering Discovery Environment

4

3. Findings of the Committee At the request of Associate Dean Campbell, the RWG originally focused on the following three items:

○ Training and high-throughput computing ○ Getting ENGR-IT people in the right place, especially technology facilitator ○ Getting ahead of needs or, at least, learning what needs are envisioned at proposal

submission These topics are are discussed below.

3.1 Training and high-throughput computing

3.1.1 Software Training Software training on campus is not centrally organized with difficult-to-find entry points. A CoE member seeking more information on how to effectively use computational tools for research is currently responsible for searching several sources, including:

● ENGR-IT (http://it.engineering.illinois.edu). Training page links to: ○ CBT Nuggets, ○ CSE, ○ FAST3 Workshops, ○ Lynda.com Tutorials, ○ Technology Services Training

● CSE (http://cse.illinois.edu/training). Training page links to: ○ Technical Computing Workshops

■ CSE Training Workshop Series—Python ■ CSE Training Workshop Series—MATLAB & Data Analytics ■ CSE Training Workshop Series—High-Performance Computing ■ CSE Training Workshop Series—Abaqus FEA ■ CSE Training Workshop Series—Fluent CFD ■ Archive of past training workshops hosted on git-hub

● NSF’s XSEDE Program (https://www.xsede.org/training1, some items only available if the CoE member is an XSEDE user)

● Technology Services Training (https://techservices.illinois.edu/training). Training page links to:

○ FAST3 Workshops, ○ Lynda.com,

5

○ Classroom Technology Training, ○ Tech Services Youtube Channel (covers e-mail, smartphone setup, etc.)

● FAST3 Workshops (https://webstore.illinois.edu/shop/category.aspx?zcid=127). Training opportunities cover Word, Excel, Illustrator, and other productivity applications

● ICCP Training (https://campuscluster.illinois.edu/training.html). Training page links to: ○ Cyberinfrastructure tutor (CI-Tutor, https://www.citutor.org/browse.php), ○ XSEDE Training, ○ CSE Training, ○ Linux Cluster Institute Workshops (joint venture between NCSA, University of

New Mexico, and the IBM Watson Research Center) From the list two observations are apparent: (1) except for CSE, the CoE does not generate it’s own training material, and (2) except for CSE, the material covered by the training opportunities deals with low-level high-performance computing (HPC) topics, including C/C++/Fortran, MPI, OpenMP, etc., or high-level desktop applications (Word, Excel, Illustrator), etc. The October 24, 2016, presentation by Dr. Neal Davis, formerly CSE’s Software Training Coordinator and now Assistant Teaching Professor in CS, focused on what material CSE created for software training sessions and how that material was created. Appendix B contains a copy of the slides presented by Dr. Davis where software categories are identified and how CSE uses the “Software Carpentry” infrastructure for already-existing content and for a skeleton when creating new content.

3.1.2 High-throughput computing On February 10, 2017, the RWG hosted a meeting with Messrs John Towns and Mr. Tim Boerner, both of NCSA, to discuss John’s vision of a high-throughput computing (HTC) resource. An HTC resource seeks to provide a maximal amount of jobs per year, in contrast to the more traditional HPC resource which seeks to provide a maximal amount of floating point operations per second. HTC is a means through which a large number of resource-limited computing job requests can be accommodated. At the meeting John and Tim mentioned that their proposal to the campus was not funded but contained the following features:

● The HTC resources would be campus-wide, supported by campus-, college, and departmental-level units

● Usage would be “free” to all users

6

● Academic and research units would receive access to campus-wide HTC resources if they exposed their existing resources, i.e., idle desktops, to the HTC resource manager. Units who did not contribute idle cycles would not receive access to the HTC resource.

● The HTC should be compatible with the Campus Cluster. Compatible was not well defined.

● The HTC resource would use either (or both) of HTCondor and Boinc resource management systems, the former of which is compatible with the Open Science Grid (OSG, a nationwide HTC resource).

● Personnel requirements for the HTC were estimated to be 2 FTE during startup and 1 FTE a steady-state operating conditions.

Most importantly, John and Tim were seeking for an initial partner with whom the HTC resource could be established.

3.2 Getting ENGR-IT people in the right place The RWG did not discuss the placement of ENGR-IT people relative to the needs of the College.

3.3 Getting ahead of needs In early 2017 the retirement of the Taub instance of the campus cluster was announced as pending, with formal retirement occurring on June 1, 2017. Taub provided the CoE with an already-paid-for computing resource via CSE’s substantial investment in it , of approximately 6

300 Taub nodes, primarily through the CSE queue. Historically, CSE had been the de facto provider of “free” HPC resources to the CoE community. Without Taub, CoE members do not have access to “free” HPC resources, provided by the College or by the campus. The RWG asked ENGR-IT to provide usage information on how CoE members utilized the CSE-provided Taub resource. Mr. Jay Guelfi of ENGR-IT provided several pieces of information, including the distribution of job sizes run from January 1, 2015, to March 8, 2017, in the CSE queue, as shown in Table 1. The bulk of the time requested from the CSE queue used between 33 and 64 cores, with a secondary group utilizing between 9 and 16 cores or between 65 and 128 cores. Jobs with core requirements in excess of 128 cores consume a modest fraction of the CSE queue availability. The 2015-2017 usage numbers show that 427,000 hours are spent in 1-core jobs, and less than 1,000,000 hours spent in jobs that might fit on one Haswell (or older) socket suggest that Towns’ HTC resource would not alleviate pressure.

6 Taub was originally purchased ($1.5M) by many units within the CoE, the Beckman Institute, the Provost’s Office, and a handful of other campus units as a shared resource. The operations were underwritten by the Provost’s Office and user fees. The CoE indirectly supported Taub through underwriting a portion of the infrastructure fees assessed to new CoE investors in Taub.

7

1/1/2015 to 3/8/2017 usage number of days 797

Job size (cores)

Core hours total Core-hr/day

Cumulative core-hr/day Req. cores

Cumulative req. cores

1 427,728 537 537 22 22

2 76,341 96 632 4 26

3 - 4 105,187 132 764 5 32

5 - 8 142,618 179 943 7 39

9 - 16 6,218,521 7802 8746 325 364

17 - 32 3,489,518 4378 13124 182 547

33 - 64 17,101,626 21457 34582 894 1441

65 - 128 7,134,895 8952 43534 373 1814

129 - 256 3,856,106 4838 48372 202 2016

257 - 512 2,596,249 3258 51630 136 2151

513 - 1024 1,124,275 1411 53040 59 2210

1k - 2k 113,620 143 53183 6 2216

Total 42,386,684 53183

Table 1. Summary of CSE queue usage on Taub Replacing Taub initiated discussion as to what form should that replacement take, with several possibilities. The RWG discussed these options:

3.3.1 Should the CoE “just buy a Cray?” Indiana University purchased a Cray a couple of years ago with money from the President’s office. DJB spoke with an IU site rep at an NSF panel in March, 2017, and asked him how it was going. His response was instructive: the machine was very heavily used by a small number of users but had not impacted the core mission of the campus to as much degree as they had hoped. Their next purchase will more likely target an HTC-like machine.

3.3.2 Should the CoE examine Cloud-based services? There are two issues: cost of data ingress/egress and cost per cycle. Mr. Michael Chan of ENGR-IT indicated that UIUC’s current agreement with AWS is as follows: “We get access to AWS through the Net+ agreement through Internet2. Because much of the network traffic to/from here and the AWS datacenters travels over Internet2, AWS waives the fee for data

8

egress. There are some limitations to that fee waiver, but they aren’t likely to be a problem for research use. Specifically, bandwidth intensive applications (MOOCs, and media streaming) aren’t eligible for the data egress fee waiver, and the data egress fees can’t exceed 15% of the total monthly cost. It’s unlikely that we’d cross that line. Storage, and compute costs will far outweigh the data egress costs.” Follow-up questions regarding egress/ingress thresholds, storage and compute costs were not answered. The RWG acknowledged, in the absence of firm financial numbers, that Amazon needs to recover its costs and make a profit. While the campus budget should hopefully balance, it still seems to be true that the CoE’s near 100% utilization of CSE’s portion of Taub makes Cloud services less attractive while the campus has a funding model where (a) power and cooling costs are absorbed at the campus level, (b) data transfer costs are shared between ENGR-IT and Tech Services, (c) and the VCR is committed to keeping the ICCP running. In other words, for now, the CoE is not 100% responsible all of the costs associated with maintaining a computing infrastructure.

3.3.3 Costs to the CoE? The RWG made assessments of the costs of a Taub-like replacement. For reference, the Taub instance cost roughly $1.5M (paid by CSE, the Provost, Beckman, and several other units). The CSE portion of Taub was 296 nodes, with 12 Intel “Westmere” 2.66 GHz cores per node, for a total of 3,552 cores. A high-speed Voltaire QDR Infiniband cluster interconnect linked the nodes.

Costs of CoE-run Standalone Machine In March, 2017, the RWG asked Mr. Tim Boerner of NCSA what costs are involved in running the Campus Cluster and, in particular, what his comments were regarding the CoE maintaining Taub beyond its retirement. His comments are as follows:

“Thanks for the additional information. We currently have around 7 FTE working on the ICCP to support it for all of campus. Even if the resource were half the size (e.g. just Taub), we would still need these 7 FTE because of the complexity that comes with having so many different investors, adding to the resource pool, providing new features, etc.

“If we were planning a new cluster that would be static for 5 years and serve the needs of only one college, I think that would probably drop the requirements for staffing levels somewhat. However, given the number of departments supported by Engineering, perhaps not quite so much as one might hope. Looking at our breakdown for the ICCP

9

currently, an environment for all Engineering departments might need the following staff levels:

FTE ROLE 2.00 HPC Systems Engineering 0.50 HPC Network Engineering 0.50 HPC Storage Engineering 0.50 Cybersecurity 1.00 Research User Services 0.25 Project Lead 0.25 Project Manager

“Depending on feature development, number of issues experienced, etc, you might be able to drop the number of HPC Systems Engineers slightly. We currently have 2.5 FTE on the ICCP and it is simply not enough, but we have a fairly dynamic environment. The networking and storage support might drop to as low as 0.25 for a very static environment, as well. If you expect an adaptive feature set, with new features added regularly, or support for newer modalities (e.g. containerized HPC, cloud bursting, OpenStack, Hadoop, etc), then you should expect more admin time to implement and maintain these features. “As for actual numbers, that would be staffing costs somewhere in the ballpark of $300,000/yr to $375,000/yr assuming an average salary of $75,000/yr (unencumbered). For the Intel compiler licenses and Torque/Moab licenses, plan for about $30,000/yr. If you're looking to run GPFS, you would need to factor costs for software licenses there at maybe $10,000/yr to $15,000/yr depending on the size of the environment. There are schedulers and file systems that wouldn't incur licensing charges, as well, so those would be an option. “I am still waiting to hear back from Doug Miller at DCSS for input on power/cooling numbers. I pinged him again this morning to see if he has anything for me.”

Cost of CoE Utilizing the ICCP’s Research Computing as a Service In his e-mail to the CoE regarding the retirement of Taub, Mr. Jay Guelfi mentioned the Research Computing as a Service (RCaaS) option within the ICCP. His quoted price was $0.0217 per core-hour (2.17c / core-hour). To replace all 3,552 cores, assuming 100% utilization

10

, the costs would be approximately $675,000 / year. Presumably the CoE could negotiate a 7

lower price due to the purchase volume.

Cost of CoE Becoming an Investor in the Campus Cluster Cost estimates to replace the CSE investment in the Campus Cluster differ depending on whether the number of cores is held fixed, the number of nodes is held fixed, or some other metric is held fixed (such as flops / second). The cost also depends on the type of interconnect that links nodes, with high-speed Infiniband being the more costly option ($5,817.72 / node without Infiniband and $6,373.94 with Infiniband) and on the memory per node ($6,373.94 / node for 64 GB RAM to $9,311.82 / node for 256 GB RAM). Using the Infiniband interconnection option with 64 GB RAM / node, the RWG estimates that the CoE would need to invest $1.9M to recover the number of nodes and $810K to recover the number of cores.

CoE-Unit Cost Sharing Model Because members of the CoE will utilize any CoE-provided resource, it is arguable that each unit within the College should help support the resource. To estimate the amount of money the CoE could request from the units without causing undue burden on top of the campus-wide budget adjustment, the RWG asked one unit Head what amount of that unit’s budget could be used to support a computing resource. Using that number as a reference and assuming it scaled with the size of a Unit’s budget, an estimate of the amount the CoE could request from each unit is shown in Table 2 with a total around $280K per year. Unit expenditures were retrieved from DMI and are meant to be estimates only.

Unit Expenditures % Base contrib

Aero 9923 3.5 $9,887.42

ARI 3783 1.3 $3,769.44

Bio 7703 2.7 $7,675.38

Civil 35051 12.5 $34,925.32

CSE 1215 0.4 $1,210.64

CS 27338 9.7 $27,239.98

CSL 20994 7.5 $20,918.72

ECE 30538 10.9 $30,428.50

ENGR Admin 19451 6.9 $19,381.26

ENGR General 2827 1.0 $2,816.86

IESE 6913 2.5 $6,888.21

7 Professor Narayan Aluru, then Director of CSE, quoted CSE’s utilization as 99%. The RWG does not have independent confirmation of this number.

11

ITI 9091 3.2 $9,058.40

MRL 11938 4.3 $11,895.19

MSE 12663 4.5 $12,617.60

MechSE 32084 11.4 $31,968.96

MNTL 9663 3.4 $9,628.35

NPRE 7875 2.8 $7,846.76

Physics 30778 11.0 $30,667.64

TEC 1024 0.4 $1,020.33

TOTAL $279,844.97

Table 2. Estimate of unit contributions to a CoE computing resource.

12

4. Recommendations of the Committee The RWG recommends to the Associate Dean that supporting CoE-wide research requires additional leadership in making software training available to the members of the CoE as well as providing computing resources to the members. Specific recommendations are below. CSE should play the dominant role in software training but should be removed from leadership in providing the computing resource.

4.1 Software Training The RWG formulated the following training plan where CSE is responsible for bundling (when possible) and creating (where necessary) content for the CoE. The Education Working Group reviewed and approved of this plan.

● CSE coordinates all software training in the CoE ● CSE, the CoE, NCSA, and additional CoE units are encouraged to establish a sustainable

training coordinator position that will attract high-quality candidates by 8

○ Having a competitive salary with appropriate title (e.g., assistant teaching professor)

○ Allowing for research ● CSE, through its training coordinator, will

○ orchestrate, maintain and encourage all faculty to participate in a collaborative training knowledge base (e.g., https://github.com/UI-CCC)

○ Communicate XSEDE-related trainings to the CoE ○ Orchestrate, maintain, and schedule 3rd party training opportunities (e.g.,

Software Carpentry) ● CoE / ENGR-IT provides CSE with n (= 2?) 50% FTE graduate positions to

○ Assist the CSE training coordinator ○ Develop and maintain a series of 5-10 minute video introductions to CoE

computing resources (software, hardware, campus cluster) ○ Assist ENGR-IT research technology facilitator in application-specific questions ○ Assist CIO on CoE-specific requests

● CSE will be the CoE liaison for a computing resource clearinghouse, e.g., http://training.illinois.edu

● ENGR-IT will direct training inquiries back to CSE ● CoE commits to an investment in the campus cluster to provide compute cycles for

training and education

8 At the time of this report, this position was created and a search was on-going.

13

4.2 CoE-Provided Computing Resources Premise: the College shall provide a computing resource to all College members Oversight: the College shall provide oversight of its investment through the Associate Dean for Information Technology Suggested hardware infrastructure:

● At least 1,000 cores ● At least 500 TB of online storage ● At ½ nodes connected by a high-speed interconnect (e.g., Infiniband)

Default resource tiers:

● students (UG and G) ● staff (includes post-docs, academic professionals) ● faculty (includes tenured and tenure-track, as well as instructional and research faculty)

Allocation minimums for student usage: the College shall maintain an infrastructure supplying a minimum of

● 720 core-hours per month per user (= 1 core used 24 hours for 30 days) ● 5 GB disk storage per user

Allocation minimums for staff usage: the College shall maintain an infrastructure supplying a minimum of

● 2,880 core-hours per month per user (= 4 cores used 24 hours for 30 days) ● 20 GB disk storage per user

Resource minimums for faculty usage: the College shall maintain an infrastructure supplying a minimum of

● 11,520 core-hours per month per user (= 16 cores used 24 hours for 30 days) ● 50 GB disk storage per user in project space

Beyond-base usage: every user is guaranteed access to his / her base allocation. Resources needs exceeding the base allocation may be accommodated on a per-case basis using one of the following paths:

● Coursework-related: the instructor may request from CSE a course-specific allocation. The additional allocation will be distributed to each enrolled student who must submit jobs with the correct account identifier.

14

● Research-related: all research-related needs must have a sponsoring faculty member who will distribute his / her allocation(s). The faculty member may, in any combination,

○ Share his / her base allocation ○ Share his / her supplementary allocation(s) (e.g., Unit-provided research

allocation) ○ Share his / her investment in the ICCP, including

■ Purchased nodes ■ RCaaS core-hours

Evaluation: the ENGR-IT shall present to the College the utilization of its investment every month, including the following metrics:

● Number of jobs per month ● Distribution of job sizes ● Distribution of queue wait times ● Data ingress and egress ● Distribution of data movement per user

Cost recovery: the College may recover up-to 50% of its computing-related costs from the Units within the College.

15

Appendices

A. Charge Letter

16

B. Dr. Neal Davis’ Slides on CSE Software Training

17

18