AWS CloudFormation under the Hood (DMG303) | AWS re:Invent 2013
AWS re:Invent - Accelerating Research
-
Upload
chris-dagdigian -
Category
Technology
-
view
872 -
download
4
description
Transcript of AWS re:Invent - Accelerating Research
1
Accelerating Research 10-minute Vignette for 2012 Amazon re:Invent Conference
2
I’m Chris.
I’m an infrastructure/bio geek.
I work for the BioTeam.
Twitstuff: @chris_dag2
Who, what & whyBioTeam - Not your standard AWS user
‣ Independent consulting shop‣ Run by scientists forced to
learn IT, SW & HPC to get our own genetics research done
‣ 12+ years bridging the “gap” between science, IT & high performance computing
‣ www.bioteam.net
33
4
Meta-IssuesWhat you don’t know WILL hurt you.
4
A common theme across all our biotech, pharma, EDU and research clients ...
5
Lets Be Blunt
‣ It’s almost 2013‣ AWS is comprehendible in minute:hours; pilot projects
and proof-of-concepts can be done in hours:days. Really.‣ The TECHNICAL bits are easy or at least manageable‣ The POLICY & PROCEDURE bits will destroy your soul‣ If I could only give 1 piece of advice to researchers ...
5
For anyone doing research HPC or “science” on the cloud
6
One Piece of Advice ...
‣ The technical issues surrounding cloud-based scientific research are easy or at least surmountable
‣ The hard part is sorting out the internal policies and procedures that govern cloud use
‣ Ignore this at your peril
6
7
Five Minute Case StudyComputer Aided Engineering (CAE) on AWS
7
Next Generation Nuclear Magnetic Resonance
8
NMR Probehead Simulation on AWS
‣ CAE Simulation Project‣ via www.hpcexperiment.com‣ Software: CST Studio 2012‣ My role: Volunteer HPC Mentor
8
Simulating next-generation NMR probeheads
9
Why this was an interesting project
‣ Frontend interface is graphics heavy and requires Windows
‣ Studio ‘solvers’ run Linux or Windows; support GPUs and MPI task distribution
‣ Simultaneous use of local and cloud-based solvers actually works
‣ flexLM license server involved
‣ Non-trivial security and geo-location requirements
9
Initial Architecture1010
At experiment end it looked like this ...1111
12
Amazon FTWAWS makes the easy stuff fun and the hard stuff possible
12
Why this work was ‘easy’ on Amazon AWS ...
13
Nightmare on any other cloud
‣ Tick Tock ...‣ Don’t have the time to get into the CAE simulation details‣ Lets discuss why this simulation workload would be
much, much harder to do on some other cloud platform ...
13
Why this work was ‘easy’ on Amazon AWS ...
14
Nightmare on any other cloud
1. Virtual Servers2. Block Storage3. Object Storage4. ... and maybe some other
stuff if I’m lucky
‣ EC2, S3, EBS, RDS, SNS, SQS, SWS, GPUs, SSDs, CloudFormation, VPC, ENIs, SecurityGroups, 10GbE DirectConnect, Reserved Instances, ImportExport, Spot Market
‣ And ~25 other products and service features with more added monthly
‘Brand X’ Cloud AWS
14
Easy on AWS; much harder elsewhereOne very specific example
15
‣ The widely used FLEXlm license server uses NIC MAC addresses when generating license keys
‣ Different MAC? Science stops. Screwed.
‣ VPC ENIs allow separation of MAC address from Network Interface. Badass.
15
Why this work was ‘easy’ on Amazon AWS ...A few other examples ...
16
VPC
Spot Market
cc* & cg* ec2 instance
types
Incredibly powerful. Actually useful.
Approachable even if you are not an IPSEC or BGP routing god.
Compelling economics. Once you start you’ll likely never run anywhere else.
The competition can’t compete.
Fat nodes with bidirectional 10GbE bandwidth.
And don’t get me started on SSD or Provisioned-performance EBS volumes.
16
Wrapping up ...Science on AWS
17
‣ It’s not rocket science; yeah we can do this just about anywhere
‣ It’s just easier and faster on AWS because they have more building blocks; I can spend more time on actual science problems You mean I get paid for this?
17
Wrapping up ...Science on AWS, cont.
18
‣ There is no one-size-fits-all research design pattern ...
‣ Very few of us have a single pipeline or workflow that we can devote endless engineering effort to
‣ We are not going to toss out hundreds of legacy codes and rewrite everything for GPUs or MapReduce
‣ For research HPC it’s all about the building blocks { and how we can effectively use/deploy them }
I love my job
18
One last plug
‣ Want more info or details?‣ Watch http://bioteam.net for upcoming posts that
document in detail the CST Studio on AWS effort‣ And by the way ...‣ hpcexperiment.com round #2 is starting soon; They need
volunteers, resource providers and new use cass
1919
20
Thanks! Slides online at: http://slideshare.net/chrisdag/
Twitter: @chris_dag
20