Building pipelines with Make
Click here to load reader
-
Upload
melvin-zhang -
Category
Technology
-
view
193 -
download
1
Transcript of Building pipelines with Make
Building pipelines with Make
Melvin ZhangNational University of Singapore
April 28, 2011http://www.slideshare.net/melvinzhang/
building-pipelines-with-make
1 / 15
Oh no! I found an error in raw data . . .
Do youI manually redo each step?I run ./gen trees.sh human-mouse-rat.mgr?I run make human-mouse-rat trees.tree?
2 / 15
Naı̈ve solution: write a shell script
3 / 15
Naı̈ve solution: write a shell script
4 / 15
maximilianh’s blog
It’s a huge mess and after a while I’ve completelyforgotten which part depends on which others.Then, at a visit to the Vingron group I’ve seenpeople typing in ”make” all the time. It’s sosimple, I don’t know why I’ve never thought ofthis.
– http://archive.nodalpoint.org/2007/03/18/a pipeline is a makefile
5 / 15
Makefile for compilation
all: helloworld
clean:-rm main.o hello
hello:g++ -g -o hello main.o
main.o: main.cppg++ -g -c main.cpp
6 / 15
Makefile for bioinformatics
all: HMR_trees.tree
clean:-rm *.tree
HMR_trees.tree: HMR_mgr.tree HMR_grappa.treecat HMR_mgr.tree HMR_grappa.tree > HMR_trees
HMR_mgr.tree: HMR.mgrbin/mgr HMR.mgr > HMR_mgr.tree
HMR_grappa.tree: HMR.mgrbin/grappa HMR.mgr > HMR_mgr.tree
7 / 15
Make is declarative
Light weight syntax to document dependency amongresults. Encourages small well defined steps.
# commentstarget1: dep_1 dep_2 dep_3<tab><commands><tab><commands><tab>...
dep_1: input...
8 / 15
Make is smartall: HMR_trees.treeclean:
-rm *.treeHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree
cat HMR_mgr.tree HMR_grappa.tree > HMR_treesHMR_mgr.tree: HMR.mgr
bin/mgr HMR.mgr > HMR_mgr.treeHMR_grappa.tree: HMR.mgr
bin/grappa HMR.mgr > HMR_mgr.tree
$ make HMR_trees.treebin/mgr HMR.mgr > HMR_mgr.treebin/grappa HMR.mgr > HMR_grappa.treecat HMR_mgr.tree HMR_grappa.tree > HMR_trees
Make only runs command if target is older thandependents
9 / 15
Make is smartall: HMR_trees.treeclean:
-rm *.treeHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree
cat HMR_mgr.tree HMR_grappa.tree > HMR_treesHMR_mgr.tree: HMR.mgr
bin/mgr HMR.mgr > HMR_mgr.treeHMR_grappa.tree: HMR.mgr
bin/grappa HMR.mgr > HMR_mgr.tree
$ make HMR_trees.treebin/mgr HMR.mgr > HMR_mgr.treebin/grappa HMR.mgr > HMR_grappa.treecat HMR_mgr.tree HMR_grappa.tree > HMR_trees
Make only runs command if target is older thandependents
9 / 15
Make is older than you
Figure: VP Engineering @ Google
Created by Stuart Feldman in 1977 at Bell Labs. In 2003Dr. Feldman received the ACM Software System Awardfor the authoring of this widespread tool.
10 / 15
Basics of a Make rule# commentstarget1: dep_1 dep_2 dep_3<tab><commands><tab><commands><tab>...
dep_1: input...
Example
# combine result of MGR and GRAPPA for# human-mouse-rat into a single fileHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree
cat HMR_mgr.tree HMR_grappa.tree > HMR_trees
11 / 15
Basics of a Make rule# commentstarget1: dep_1 dep_2 dep_3<tab><commands><tab><commands><tab>...
dep_1: input...
Example
# combine result of MGR and GRAPPA for# human-mouse-rat into a single fileHMR_trees.tree: HMR_mgr.tree HMR_grappa.tree
cat HMR_mgr.tree HMR_grappa.tree > HMR_trees
11 / 15
Don’t repeat yourself (DRY)
Automatic variables
output.png: output.pdfconvert -trim -density 300 $ˆ $@
# $ˆ is the list of dependents# $@ is the name of the target
User defined variables
CC = gcctest.out: test.cpp
$(CC) -o test.out test.cpp
12 / 15
Don’t repeat yourself (DRY)
Automatic variables
output.png: output.pdfconvert -trim -density 300 $ˆ $@
# $ˆ is the list of dependents# $@ is the name of the target
User defined variables
CC = gcctest.out: test.cpp
$(CC) -o test.out test.cpp
12 / 15
Generalize with Pattern rules
%.png: %.pdfconvert -trim -density 300 $ˆ $@
# % is like * in shell
13 / 15
Use descriptive filenames
data alg1 alg2 ... algn.format
HMR: HMR_mgr_prune.tree
%_mgr.tree: %.mgrbin/mgr $ˆ > $@
%_prune.tree: %.treebin/prune $ˆ > $@
$ make HMRbin/mgr HMR.mgr > HMR_mgr.treebin/prune HMR_mgr.tree > HMR_mgr_prune.tree
14 / 15
Use descriptive filenames
data alg1 alg2 ... algn.format
HMR: HMR_mgr_prune.tree
%_mgr.tree: %.mgrbin/mgr $ˆ > $@
%_prune.tree: %.treebin/prune $ˆ > $@
$ make HMRbin/mgr HMR.mgr > HMR_mgr.treebin/prune HMR_mgr.tree > HMR_mgr_prune.tree
14 / 15
Further readings
I Parker DS, Gorlick MM, Lee CJ. Evolving frombioinformatics in-the-small to bioinformaticsin-the-large. OMICS. 2003 Spring;7(1):37-48.http://www.ncbi.nlm.nih.gov/pubmed/12831555
I http://www.gnu.org/software/make/I http://biowiki.org/MakefileManifestoI http://www.slideshare.net/giovanni/makefiles-bioinfo
15 / 15