Grid Computing with BatchMake

National Alliance for Medical Image Computing http://na-mic.org

Grid Computing with BatchMake

Julien JomierKitware Inc.


Motivation

Algorithm Design & Implementation

Data Processing

Reports & Analysis


Outline

• Introduction to BatchMake• Grid Computing with BatchMake• Reporting with the BatchBoards• BatchMake with Slicer• Future plans


BatchMake Origins

• Started in 2004 at UNC (CADDLab)• A2D2 from NLM• Kitware funds: KWGrid• BW-NAC: Batchboards• Na-Mic• STTR Kitware-UNC (Styner)


Introduction to BatchMake

• Batch script generator• Process of large number of datasets • Common Language (CMake-like)• Generation for:

- Shell script - Condor - KWGrid

• Run locally or on grid• Cross platform command line and GUI tool• C++ with ITK and FTLK


BatchMake Workflow

CommandLine Executable

ApplicationWrapperBatchMake

Script

Local Execution

Condor Script

KWGrid Script

Shell Script


Application Wrapper

• Describe command line arguments- Manually (GUI)- Automatically using itkMetaCommand

• Currently using internal XML format• .bmm files (BatchMake Modules)• Switching to Slicer’s

ModuleDescriptionParser- importing Slicer Module automatically


MetaCommand

• C++ class for command line parsing• Output recognized by

- BatchMake (-vxml)- Slicer (--xml)

• command.SetOption("compileScript","c",false,"Compile a script");

command.AddOptionField("compileScript","filename", MetaCommand::STRING,true);

• http://www.itk.org/Wiki/MetaIO• Insight/Utilities/MetaIO


Example Script

1 - SetApp(imSeg @ImageSegmentation)2 - SetAppOption(imSeg.threshold 10)3 - 4 - Set(workdir '/BatchMake/Demo/Ellipse_Images')5 - ListFileInDir(files ${workdir} '*.mha')6 -7 - Foreach(file ${files})8 - Set(filename '${workdir}/${file}')9 - Set(outfile '${workdir}/out_${file}')10- SetAppOption(imSeg.inputFilename ${filename})11- SetAppOption(imSeg.outputFilename ${outfile})12- Run(output ${imSeg})13- EndForeach(file)


Condor

• Specialized workload management system for compute-intensive jobs

• Created by UW-Madison• 15 Years of development• Provides job queueing mechanism,

scheduling policy, priority scheme, resource monitoring, and resource management.

• Unix and Windows

http://www.cs.wisc.edu/condor


Condor Universes

• Standard Universe (checkpoint)• Vanilla• Parallel Virtual Machine • MPI• Globus• Java (JVM)• Scheduler (no matching)


Condor Example Script

Universe = vanillaExecutable = EMSegmenterArguments = ‘brain.mha segmentation.mha --brain’Output = emsegmentation.out Error = emsegmentation.error Log = emsegmentation.logRequirements = (OpSys =="LINUX" && Arch=="INTEL")Should_transfer_files = YES When_to_transfer_output = ON_EXIT_OR_EVICTTransfer_input_files = brain.mhaQueue


Condor Workflow (Vanilla)

Submit Machine

Central Manager

Execute Machines

Condor Pool

Submit job

Issue job

Idle Workstation

Busy Workstation

Available Execute Machine

Push out job to machine

Central Manager

Central ManagerUnavailable Execute Machine

Dispatch to Central manager

= Available

Dispatch to Central manager

= Unavailable


Condor Job Submission Workflow

Collector

Submit Machine

Scheduler

Starter

Negociator

Central Manager Execute Machine

MasterClassAd

Master MatchClassAd

Job

1

23

4

5


BatchMake Condor Generation

http://www.cs.wisc.edu/condor


BatchMake Condor Generation

• Application/Module description - Inputs/Outputs - Data to transfer to nodes

• Loop unrolling• Scope definition• Directed Acyclic Graph (DAG)• Ensure order of

execution of jobs


Monitoring Condor Jobs

• Flock Monitor (Condor)• Condor Watcher (FLTK)• Condor Web Watcher (Ajax/PHP)


Condor Summary

• Pros:- Well supported- (Fairly) easy installation- Complete resources management

• Cons:- Script can be complicated- Jobs don’t start right away- Hard to debug


Reporting with BatchMake

CVS/SVN

Results posted on web(i.e., the batchboard)

CTest/DART compilessource code

Developers review results

Developers check-in

code

MIDASXNat

BatchMake validateson datasets

+


Reporting with BatchMake


BatchMake Server

• PHP Scripts with MySQL database• Simple C++ HTTP post• Statistical Analysis (Factor analysis:

Taguchi).• Dashboard creation capability• User management


BatchBoard Script

DashboardHost(http://www.insight-journal.org/batchmake)DashboardUser(‘Julien Jomier')DashboardKey('FKH3BPM1y11VXsMf')CreateExperiment(exp ‘Liver Project‘ ‘Segmentation Experiment' '')CreateMethod(Meth exp 'Registration'

'Register a T1 with noise to a translated T1')AddMethodInput(noiseImageName Meth 'Fixed Image')Set(fixedSlice '${fixedImage}Slice.png')ExtractSlice('${fixedImage}' '${fixedSlice}' 2 1)AddMethodInput(fixedSlice Meth 'Fixed Slice with noise' png)DashboardSend(Meth)


BatchMake and Slicer

• BatchMake now distributed with Slicer• Some modules can be run on a

collection of data- EMSegment- GaussianBlur

• Generic way to run moduleson a collection


BatchMake and MIDAS

• STTR Kitware/UNC Martin Styner


Future

• Improve packaging• Improve documentation• Improve integration with Slicer• BatchMake and Xnat (discussion)


References

• www.batchmake.org• www.insight-journal.org/batchmake

Grid Computing with BatchMake

Documents

Transcript of Grid Computing with BatchMake