UpgradingfromHDP2.1toHDP2.4

2016/03/24@wyukawa

HadoopSCR#hadoopreading

About1yearago

hFp://www.slideshare.net/tagomoris/upgrading-from-hdp-21-to-hdp-22

2014/12-

MRv2/HDFS

Hive

Analysis3(HDP2.1)

Inputdatafluentd-plugin-webhdfs

shib

Executequeriesoverhiveserver2onMR

hivebatchonMR

HDP2.1.5.0

•  InstalloverAnsible,withoutAmbari•  Hadoop2.4.0•  YARNResourceManagerHA•  NameNodeHA•  Hive0.13

HDP2.3.4

•  InstalloverAmbaributuseAnsible•  Hadoop2.7.1– YARNResourceManagerRESTAPI(includekill)– shibuseyarncommandtokillonHDP2.1.5.0

•  Hive1.2.1– backportHive-12084HivequerieswithORDERBYandlargeLIMITfailswithOutOfMemoryErrorJavaheapspace.

•  Tez0.7.0

changeHadoop/Hivesebngbyambari

•  dfs.datanode.failed.volumes.tolerated=1•  hive.server2.authendcadon=NOSASL•  hive.server2.enable.doAs=false•  hive.auto.convert.join=false•  changememorysizebecauseofprestoandfluentd–  increasehadoop/datanode/resourcemanager/hiveserver2heap

– decreaseyarn.nodemanager.resource.memory-mbandyarn.scheduler.maximum-allocadon-mb

bluegreendeployment

MRv2/HDFS

Hive

Analysis3(HDP2.1)

Inputdatafluentd-plugin-webhdfs

shib

Executequeriesoverhiveserver2onMR

MRv2/Tez/HDFS

Hive

Analysis4(HDP2.3.4)

hivebatchonTezhivebatchonMR

OOMEoccuredatHiveonTez

•  reportasHIVE-13238–  increasehive.tez.container.size

•  reportasHIVE-13239

SwitchtohivebatchonMR!IhavethereasontouseMRbecauseofshibstatusfeature...

WhydoI useshibonMR?

•  canexecuteHiveonTez– sethive.execudon.engine=tez

•  can’tmonitorquerystatus•  can’tkillquery•  shibusemapreduce.job.nametoidendfysubmiFedquery

OMMEoccuredatHiveonMR

•  reportasHIVE-13273– hiveserver2memoryleak

needtoupgradeHDP2.4

hFps://community.hortonworks.com/quesdons/20116/logfdscacheflushdmer-thread-increase.html

HDP2.4.0

•  almostsameasHDP2.3.4exceptforKala,Spark

•  backportHIVE-12766!!!!!!!!!!!!!!!!!!!!!!!!!•  AMBARI-14389

upgradingfromHDP2.3.4toHDP2.4.0

•  reportasAMBARI-15418

Iwanthortonworkstofixthisissue.But,anyway,switchtohivebatchonTez!

hiveserver2isstable!!!

2016/3-

MRv2/HDFS

Hive

Analysis3(HDP2.1)

Inputdatafluentd-plugin-webhdfs

shib

Executequeriesoverhiveserver2onMR

MRv2/Tez/HDFS

Hive

Analysis4(HDP2.4.0)

hivebatchonTezhivebatchonMR

somedmescorrupt/missingblock...

monitoringwithgrafana,prometheus,hadoop_exporter

Conclusion

•  Don’tuseHDP2.3.4ifyouusehiveserver2•  useHDP2.4.0•  seeApachepatchinformadon•  I wanttoseeHDPsourcerepository•  somedmesneedtochangequery(HIVE-9390)•  IwantHivecommiFer