Enabling Large Scale Scientific Computations for Expressed Sequence Tag Sequencing over Grid and...
-
Upload
emmeline-cameron -
Category
Documents
-
view
215 -
download
0
Transcript of Enabling Large Scale Scientific Computations for Expressed Sequence Tag Sequencing over Grid and...
Enabling Large Scale Scientific Computations for Expressed
Sequence Tag Sequencing over Grid and Cloud Computing Clusters
Sangmi Lee Pallickara, Marlon Pierce, Qunfeng Dong, Chin Hua Kong
Indiana University, Bloomington IN, USA
*Presented by Marlon Pierce
IU to lead New US NSF Track 2d $10M Award
The EST Pipeline• The goal is to cluster mRNA sequences
– Overlapping sequences are grouped together into different clusters and then
– A consensus sequence is derived from each cluster.– CAP3 is one program to assemble contiguous sequences.
• Data sources: NCBI GenBank, short read gene sequencers in the lab, etc.– Too large to do with serial codes like CAP3
• We use PaCE (S. Aluru) to do a pre-clustering step for large sequences (parallel problem).– 1 large data set --> many smaller clusters – Each individual cluster can be fed into CAP3. – We replaced the memory problem with the many-task problem.– This is data-file parallel.
• Next step: do the CAP3 consensus sequences match any known sequences?– BLAST also data-file parallel, good for Clouds
http://swarm.cgb.indiana.edu
• Our goal is to provide a Web service-based science portal that can handle the largest mRNA clustering problems.
• Computation is outsourced to Grids (TeraGrid) and Clouds (Amazon) – Not provided by in-
house clusters. • This is an open service,
open architecture approach.
Some TeraGrid Resources
Data obtained from NIH NCBI. 4.7 GB raw data processed using PACE on Big Red. Clusters shown to be processed with CAP3.
Swarm: Large scale job submission infrastructure over the distributed clusters• Web Service to submit and monitor 10,000’s (or more)
serial or parallel jobs.• Capabilities: – Scheduling large number of jobs over distributed HPC clusters
(Grid clusters, Cloud cluster and MS Windows HPC cluster)– Monitoring framework for the large scale jobs– Standard Web service interface for web application– Extensible design for the domain specific software logics– Brokers both Grid and Cloud submissions
• Other applications: – Calculate properties of all drug-like molecules in PubChem
(Gaussian)– Docking problems in drug discovery (Amber, Autodock)
(Revised) Architecture of Swarm Service
Windows Server Cluster
Windows Server Cluster
Swarm-Grid
Swarm-Dryad
Local RDMBSLocal RDMBS
Swarm-AnalysisStandard Web Service Interface
Large Task Load Optimizer
Swarm-Grid Connector
Swarm-Dryad Connector
Swarm-Hadoop Connector
Cloud Comp. Cluster
Cloud Comp. Cluster
Grid HPC/Condor Cluster
Grid HPC/Condor Cluster
Swarm-Hadoop
Swarm-Grid• Swarm considers
traditional Grid HPC cluster are suitable for the high-throughput jobs.– Parallel jobs (e.g. MPI
jobs)– Long running jobs
• Resource Ranking Manager– Prioritizes the resources
with QBETS, INCA• Fault Manager– Fatal faults– Recoverable faults
Resource Ranking Manager
Grid HPC/Condor pool Resource Connector
Condor(Grid/Vanilla) with Birdbath
Grid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC Clusters
Condor ClusterCondor Cluster
Standard Web Service Interface
Swarm-Grid
QBETS Web
Service
QBETS Web
Service
Local RDMBS
Local RDMBS
MyProxyServer
MyProxyServer
Hosted by TeraGrid Project
Hosted by UCSB
Request Manager
Job Distributor
Job Queue
Data Model Manager Fault Manager
User A’s Job Board
Swarm-Hadoop• Suitable for short running
serial job collections• Submit jobs to the cloud
computing clusters: Amazon’s EC2 or Eucalyptus
• Uses Hadoop map-reduce engine.
• Each job processed as a single Map function:
• Input/output location is determined by the Data Model Manager– Easy to modify for the
domain specific requirements.
Swarm-Hadoop
Hadoop Map Reduce Programming interface
Cloud Computing Cluster
Cloud Computing Cluster
Hadoop Resource Connector
Job Producer
DataModel Manager Fault Manager
Local RDMBSLocal RDMBS
Standard WebService Interface
Job Buffer
Request Manager
User A’s Job Board
Performance Evaluation• Java JDK 1.6 or higher, Apache Axis2• Server: 3.40 GHz Inter Pentium 4 CPU, 1GB RAM
• Swarm Grid:– Backend TeraGrid machines: Big Red (Indiana University), Ranger (Texas Advanced
Computing Center), and NSTG (Oak Ridge National Lab)
• Swarm-Hadoop:– Computing Nodes: Amazon Web Service EC2 cluster with m1.small instance (2.5 GHz
Dual-core AMD Opteron with 1.7GB RAM)
• Swarm-Windows HPC:– Microsoft Windows HPC cluster, 2.39GHz CPUs, 49.15GB RAM, 24 cores, 4 sockets
• Dataset: partial set of the human EST fragments (published by NCBI GenBank)
– 4.6 GB total– Groupings: Very small job(less than 1 minute), small job(1~3 minutes), Medium
job(3~10 minutes), large job(longer than 10 minutes)
Total Execution time of CAP3 execution for the various numbers of jobs (~1 minute) with Swarm-
Grid, Swarm-Hadoop, and Swarm-Dryad
Job Execution time in Swarm-Hadoop
Conclusions• Bioinformatics needs both computing Grids and
scientific Clouds– Problem sizes range over many orders of magnitude
• Swarm is designed to bridge the gap between the two, while supporting 10,000’s or more jobs per user per problem.
• Smart scheduling is an issue in data-parallel computing– Small Jobs(~1min) were processed more efficiently by
Swarm-Hadoop and Swarm-Dryad. – Grid style HPC clusters adds minutes (or even longer) of
overhead to each of jobs.– Grid style HPC clusters still provide stable environment
for large scale parallel jobs.
More Information
• Email: leesangm AT cs.indiana.edu mpierce AT cs.indiana.edu
• Swarm Web Site: http://www.collab-ogce.org/ogce/index.php/Swarm
• Swarm on SourceForge: http://ogce.svn.sourceforge.net/viewvc/ogce/ogce-services-incubator/swarm/
Computational Challenges in the EST Sequencing
• Challenge 1: Executing tens of thousands of jobs.– More than 100 plant species have at least 10,000 EST
sequences; tens of thousand assembly jobs are processed.– Standard queuing systems used by Grid based clusters do
NOT allow users to submit 1000s of jobs concurrently to batch queue systems.
• Challenge 2: Requirement of job processing is various– To complete EST assembly process, various types of
computation jobs must be processed. E.g. large scale parallel processing, serial processing, and embarrassingly parallel jobs.
– Suitable computing resource will optimize the performance of the computation.
Tools for EST Sequence Assembly
Cleaning Cleaning sequence readssequence reads
RepeatMaskerSEG, XNU,
RepeatRunner, PILER
Clustering Clustering sequence readssequence reads
PaCE
Cap3 Clustering, BAG, Clusterer,
CluSTr, UI Cluster and many more
Assemble readsAssemble reads Cap3FAKII, GAP4, PHRAP, TIGR Assembler
Similarity searchSimilarity search Blast
FASTA, Smith-Waterman, Needleman-
Wunsch
Swarm-Grid: Submitting High-throughput jobs-2
• User(personal account, community account) based job management: policies in the Gird clusters are based on the user.
• Job Distributor: matchmaking available resources and submitted jobs.
• Job Execution Manager: submit jobs through CondorG using birdbath WS interface
• Condor resource connector manages to job to be submitted to the Grid HPC clusters or traditional Condor cluster.
Resource Ranking Manager
Grid HPC/Condor pool Resource Connector
Condor(Grid/Vanilla) with Birdbath
Grid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC ClustersGrid HPC Clusters
Condor ClusterCondor Cluster
Standard WebService InterfaceSwarm-Grid
QBET Web
Service
QBET Web
Service
Local RDMBS
Local RDMBS
MyProxyServer
MyProxyServer
Hosted by TeraGrid Project
Hosted by UCSB
Request Manager
Job Distributor
Job Queue
DataModel Manager Fault Manager
User A’s Job Board
Job Execution Time in Swarm-DryAd with Windows HPC 16 nodes
Job Execution Time in Swarm-DryAd various number of nodes
EST Sequencing Pipeline• EST (Expressed Sequence Tag): A fragment of Messenger
RNAs (mRNAs) which is transcribed from the genes residing on chromosomes.
• EST Sequencing: Re-constructing full length of mRNA sequences for each expressed gene by means of assembling EST fragments.
• EST sequencing is a standard practice for gene discovery, especially for the genomes of many organisms which may be too complex for whole-genome sequencing. (e.g. wheat)
• EST contigs are important data for accurate gene annotation.
• A pipeline of computational steps is required:– E.g. repeat masking, PaCE, CAP3 or other assembler on
clustered data set
Computing resources for computing intensive Biological Research
• Biologically based researches require substantial amount of computing resources.
• Many of current computing is based on the limited local computing infrastructure.
• Available computing resources include:– US national cyberinfrastructure (e.g. TeraGrid) good fit for
closely coupled parallel application– Cloud computing clusters (e.g. Amazon EC2, Eucalyptus) :
good for on-demand jobs that individually last a few seconds or minutes
– Microsoft Windows based HPC cluster(DryAd) : Job submission environment without conventional overhead of batch queue systems