Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to...

18
Managing Scalable Data Workflows on HPC Clusters Erfan Nourbakhsh (UC Davis) Astroinformatics 2019 June 26 th , Caltech

Transcript of Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to...

Page 1: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

ManagingScalableDataWorkflowsonHPCClusters

Erfan Nourbakhsh (UCDavis)

Astroinformatics 2019

June26th,Caltech

Page 2: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

Outline

• Time-basedjobschedulers• Smartschedulingandscalability• Workflowmanagementsystemsandbigdatapipelines• IntroducingApacheAirflow• Airflowcomponents• UIwalk-through• AirflowandHPC

Page 3: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

Time-basedjobschedulers

Page 4: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

Smartschedulingandscalability

• SmartScheduling• SchedulingtaskexecutionminimalcriterionforaWMS• Wewanttaskexecutiontobemore“data-aware”

• Scalability• Notonlyaddressourcurrentdataprocessingneeds,butalsoscalewithourfuturegrowth• Thisshouldbewithoutrequiringsubstantialengineeringtimeandinfrastructurechanges

Page 5: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

Workflowmanagementsystemsandbigdatapipelines

Page 6: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

Workflowmanagementsystemsandbigdatapipelines

Pinball

Page 7: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

ApacheAirflow

• Airflowisaplatformtoprogrammaticallyauthor,scheduleandmonitorworkflows.

• DevelopedatAirbnb in2014• JoinedtheApacheSoftwareFoundation’sincubationprogramin2016

• Ithasbeenbuilttoscale• Pythonscript(configurationascode)• ActivedevelopmentonGitHub (835contributors,6,534commits)• RichwebUI• Officiallyusedbyabout300companies(Tesla,Twitter,Yahoo!,PayPal,UnitedAirlinesetc.)

Page 8: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

AirflowCom

pone

nts

Page 9: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

AirflowCom

pone

nts

Page 10: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin
Page 11: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

UIWalk-through

Page 12: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

UIWalk-through

Page 13: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

UIWalk-through

Page 14: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

UIWalk-through

Page 15: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

AirflowandHPC

• ApacheSparkIntegration• MPIwithSlurm Integration(notnativelysupported)

Page 16: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

Slurm Output

Yourjoboutput...

Page 17: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

TryAirflowandapplyittoyourwork

Thankyou!

Conclusion

Page 18: Managing Scalable Data Workflows on HPC Clusters · Apache Airflow • Airflow is a platform to programmatically author, schedule and monitor workflows. • Developed at Airbnbin

Sourceshttps://github.com/apache/airflow

https://www.alooma.com/answers/what-is-apache-airflow

https://sonra.io/2018/01/01/using-apache-airflow-to-build-a-data-pipeline-on-aws/

https://www.slideshare.net/mesodiar/intro-to-airflow-good-bye-cron-welcome-scheduled-workflow-management

https://qcon.ai/system/files/presentation-slides/sid_anand_qcon_ai_2018_v2.pdf

https://dabble-of-devops.com/learn-airflow-by-example-part-1-introduction/

https://towardsdatascience.com/why-quizlet-chose-apache-airflow-for-executing-data-workflows-3f97d40e9571


AAON Outdoor Airflow Measurement System - AAON... · The AAON Outdoor Airflow Measurement System includes an airflow station, airflow signal processor, and temperature transmitter

AAON Outdoor Airflow Measurement System - AAON... · The AAON Outdoor Airflow Measurement System includes an airflow station, airflow signal processor, and temperature transmitter

Cleaning Can Be Programmatically Improved · Cleaning Can Be Programmatically Improved 40 50 60 70 80 ... Testing immediately after cleaning and just before use is the standard. ...

Cleaning Can Be Programmatically Improved · Cleaning Can Be Programmatically Improved 40 50 60 70 80 ... Testing immediately after cleaning and just before use is the standard. ...

AIRFLOW MANAGEMENT PRODUCTS TURNING VANE & CHIMNEY AIRFLOW

AIRFLOW MANAGEMENT PRODUCTS TURNING VANE & CHIMNEY AIRFLOW

Apache Airflow Oozie Workflows to Migrating Apache · 2020. 7. 9. · Apache Oozie Apache Oozie is a workflow management system to manage Hadoop jobs. It is deeply integrated with

Apache Airflow Oozie Workflows to Migrating Apache · 2020. 7. 9. · Apache Oozie Apache Oozie is a workflow management system to manage Hadoop jobs. It is deeply integrated with

Programmatically Profiling Users, Personalising UI, And Targeting Content To Audiences

Programmatically Profiling Users, Personalising UI, And Targeting Content To Audiences

Programmatically Managing Python Workloads Across Multiple Clouds

Programmatically Managing Python Workloads Across Multiple Clouds

Airflow Tables

Airflow Tables

Scientific Workflows - University of Washingtonhomes.cs.washington.edu/~billhowe/cs410/lectures/sciwf.pdf · Business Workflows vs. Scientific Workflows • Business Workflows –

Scientific Workflows - University of Washingtonhomes.cs.washington.edu/~billhowe/cs410/lectures/sciwf.pdf · Business Workflows vs. Scientific Workflows • Business Workflows –

Apache Airflow

Apache Airflow

Airflow Documentation

Airflow Documentation

PowerPoint Presentation · 10.00 5.00 0.00 700 Airflow 1 Airflow 2 Airflow 3 Airflow 4 519-633-8801 Airflow (cfm at STP) Canada N5P 3R9 TEL: 33 Gaylord Road, St. Thomas, ON., Title:

PowerPoint Presentation · 10.00 5.00 0.00 700 Airflow 1 Airflow 2 Airflow 3 Airflow 4 519-633-8801 Airflow (cfm at STP) Canada N5P 3R9 TEL: 33 Gaylord Road, St. Thomas, ON., Title:

Airflow Management

Airflow Management

Airflow Modeling

Airflow Modeling

Airflow @ Agari

Airflow @ Agari

77-1131 - Building Airflow System Control Applications · BUILDING AIRFLOW SYSTEM CONTROL APPLICATIONS 2 Airflow Control Applications ... AIRFLOW CONTROL FUNDAMENTALS

77-1131 - Building Airflow System Control Applications · BUILDING AIRFLOW SYSTEM CONTROL APPLICATIONS 2 Airflow Control Applications ... AIRFLOW CONTROL FUNDAMENTALS

Get in control of your workflows with Apache Airflow

Get in control of your workflows with Apache Airflow

Petrel: A Programmatically Accessible Research Data Service

Petrel: A Programmatically Accessible Research Data Service

Outdoor Airflow Measurement System · Outdoor Airflow Measurement System DESCRIPTION ... accurately determine true airflow rates even when the primary airflow measurement stations

Outdoor Airflow Measurement System · Outdoor Airflow Measurement System DESCRIPTION ... accurately determine true airflow rates even when the primary airflow measurement stations

Airflow Documentation · Airflow is a platform to programmatically author, schedule and monitor workflows. Use airflow to author workflows as directed acyclic graphs (DAGs) of

Airflow Documentation · Airflow is a platform to programmatically author, schedule and monitor workflows. Use airflow to author workflows as directed acyclic graphs (DAGs) of

Programmatically mapping source variables to …...1 PharmaSUG 2019 - Paper BP-340 Programmatically mapping source variables to output SDTM (Study Data Tabulation Model) variables

Programmatically mapping source variables to …...1 PharmaSUG 2019 - Paper BP-340 Programmatically mapping source variables to output SDTM (Study Data Tabulation Model) variables