MapReduce Container ReUse

download MapReduce Container ReUse

of 11

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of MapReduce Container ReUse

  • 1. MR Container Re-Use(Not YARN specific) Siddarth Seth Member of Technical Staff Hortonworks Inc. 2012

2. Current AM Components JobRunningTA Listener Task TaskTaskAttempContainer tLauncherRMAllocatoRM r/ NMScheduler 3. TaskAttempt and Container operations aretightly coupled CLC construction, Container Launchinvocation is handled by the TaskAttempt Container Launch is tied to the TaskAttempt(instead of container size, LocalResources) Container shutdown. 4. AM post 3902 ContainerJobRunning JVMListenerRunningTA ListenerTaskTaskNM TaskAttempContainerCommunicattorNode RMAMSchedulerNM Rack ? 5. Container and Node have their own states. Containers interact with the NodeManager Tasks interact with the scheduler whichmatches containers to task attempts. Nodes take care of blacklisting simplifiesthe scheduler. Easier to write a custom scheduler. 6. Current State Most of the AM functional changes aredone. (Cleanup pending) Task side changes are required A re-use scheduler needs to beimplemented. 7. Facilitates Common MapOutputBuffer for mapsassigned to the same container. Merging per-node or per-rack map output Custom Task Types 8. Hortonworks Data Platform Simplify deployment to get started quickly and easily Monitor, manage any size cluster with familiar console and tools1 Only platform to include data integration services to interact with any data Metadata services opens the platform for integration with existing applications Dependable high availability architecture Reduce risks and cost of adoption Lower the total cost to administer and provision Tested at scale to future proof your cluster growth Integrate with your existing ecosystemPage 8 Hortonworks Inc. 2012 9. Hortonworks TrainingThe expert source forApache Hadoop training & certificationRole-based Developer and Administration training Coursework built and maintained by the core Apache Hadoop development team. The right course, with the most extensive and realistic hands-on materials Provide an immersive experience into real-world Hadoop scenarios Public and Private courses availableComprehensive Apache Hadoop Certification Become a trusted and valuableApache Hadoop expertPage 9 Hortonworks Inc. 2012 10. Next Steps?1 Download Hortonworks Data Use the getting started Learn more get support Hortonworks Support Expert role based training Full lifecycle technical support Course for admins, developers across four service levels and operators Delivered by Apache Hadoop Certification program Experts/Committers Custom onsite options Forward-compatible 10 Hortonworks Inc. 2012 11. Thank You!Questions & AnswersFollow: @hortonworksRead: Page 11 Hortonworks Inc. 2012