Table of Contents - Pivotal Software · What is HAWQ? HAWQ Architecture Table Distribution and...

19

10162228344251606465677381838788919499

100103104105106107108109110113120123127130133134136148152155159160166167170174176179180188194197

Table of ContentsTable of ContentsPivotal HDB 2.3.0 DocumentationPivotal HDB 2.3.0 Release NotesPivotal HDB 2.2.0 Release NotesPivotal HDB 2.1.2 Release NotesPivotal HDB 2.1.1 Release NotesPivotal HDB 2.1.0 Release NotesPivotal HDB 2.0.1 Release NotesPivotal HDB 2.0 Release NotesApache HAWQ System RequirementsSelect HAWQ Host MachinesSetting up HDB Software RepositoriesInstall Apache HAWQ using AmbariInstall HAWQ from the Command Line (Optional)PXF Post-Installation Procedures for Hive/HBaseAmazon EC2 ConfigurationInstalling Procedural Languages and Package Extensions for HAWQInstalling MADlibInstalling PL/RUpgrading to HDB 2.3.xWhat is HAWQ?HAWQ ArchitectureTable Distribution and StorageElastic Query Execution RuntimeResource ManagementHDFS Catalog CacheHAWQ Management ToolsHigh Availability, Redundancy and Fault ToleranceGetting Started with HAWQLesson 1 - Runtime EnvironmentLesson 2 - Cluster AdministrationLesson 3 - Database AdministrationLesson 4 - Sample Data Set and HAWQ SchemasLesson 5 - HAWQ TablesLesson 6 - HAWQ Extension Framework (PXF)Running a HAWQ ClusterIntroducing the HAWQ Operating EnvironmentManaging HAWQ Using AmbariUsing the Ambari REST APIStarting and Stopping HAWQExpanding a ClusterRemoving a NodeBacking Up and Restoring HAWQHigh Availability in HAWQUsing Master MirroringHAWQ Filespaces and High Availability Enabled HDFSUnderstanding the Fault Tolerance ServiceRecommended Monitoring and Maintenance TasksRoutine System Maintenance TasksMonitoring a HAWQ SystemHAWQ Administrative Log FilesHow HAWQ Manages ResourcesBest Practices for Configuring Resource Management

© Copyright Pivotal Software Inc, 2013-2017 1 2.3.0

198201206210213217219220224231233235237243244248256260263264267268271272275276278279280282285287291294296305306307309326327332344348349350352357358359360361362363364365

Configuring Resource ManagementIntegrating YARN with HAWQWorking with Hierarchical Resource QueuesAnalyzing Resource Manager StatusConfiguring Client AuthenticationUsing LDAP Authentication with TLS/SSLUsing Kerberos AuthenticationConfiguring HAWQ/PXF for Secure HDFSConfiguring Kerberos User Authentication for HAWQExample - Setting up an MIT KDC ServerDisabling Kerberos SecurityOverview of HAWQ AuthorizationUsing HAWQ Native AuthorizationOverview of Ranger Policy ManagementConfiguring HAWQ to use Ranger Policy ManagementCreating HAWQ Authorization Policies in RangerHAWQ Resources and PermissionsSQL Command Permissions SummaryUsing MADLib with Ranger AuthorizationAuditing Authorization EventsHigh Availability and HAWQ RangerHAWQ Ranger Kerberos IntegrationEstablishing a Database SessionHAWQ Client ApplicationsConnecting with psqlHAWQ Database Drivers and APIsTroubleshooting Connection ProblemsDefining Database ObjectsCreating and Managing DatabasesCreating and Managing TablespacesCreating and Managing SchemasCreating and Managing TablesIdentifying HAWQ Table HDFS FilesTable Storage Model and Distribution PolicyPartitioning Large TablesCreating and Managing ViewsUsing Languages and Extensions in HAWQUsing HAWQ Built-In LanguagesUsing PL/JavaUsing PL/PerlUsing PL/pgSQL in HAWQUsing PL/Python in HAWQUsing PL/R in HAWQEnabling Cryptographic Functions for PostgreSQL (pgcrypto)Managing Data with HAWQBasic Data OperationsAbout Database StatisticsConcurrency ControlWorking with TransactionsLoading and Unloading DataWorking with File-Based External TablesAccessing File-Based External Tablesgpfdist Protocolgpfdists ProtocolHandling Errors in External Table DataClient-Based HAWQ Load Tools


368369370371372373374378379380381382383384385386388389390391392393394395396397398399400401402403404405406407409410411412413415416417418419420421422423424425426427432

Using the HAWQ File Server (gpfdist)About gpfdist Setup and PerformanceControlling Segment ParallelismInstalling gpfdistStarting and Stopping gpfdistTroubleshooting gpfdistRegistering Files into HAWQ Internal TablesCreating and Using Web External TablesCommand-based Web External TablesURL-based Web External TablesLoading Data Using an External TableLoading and Writing Non-HDFS Custom DataUsing a Custom FormatImporting and Exporting Fixed Width DataExamples - Read Fixed-Width DataCreating External Tables - ExamplesHandling Load ErrorsDefine an External Table with Single Row Error IsolationCapture Row Formatting Errors and Declare a Reject LimitIdentifying Invalid CSV Files in Error Table DataMoving Data between TablesLoading Data with hawq loadLoading Data with COPYRunning COPY in Single Row Error Isolation ModeOptimizing Data Load and Query PerformanceUnloading Data from HAWQDefining a File-Based Writable External TableExample - HAWQ file server (gpfdist)Defining a Command-Based Writable External Web TableDisabling EXECUTE for Web or Writable External TablesUnloading Data Using a Writable External TableUnloading Data Using COPYTransforming XML DataDetermine the Transformation SchemaWrite a TransformWrite the gpfdist ConfigurationLoad the DataTransfer and Store the DataTransforming with GPLOADTransforming with INSERT INTO SELECT FROMConfiguration File FormatXML Transformation ExamplesCommand-based Web External TablesExample using IRS MeF XML Files (In demo Directory)Example using WITSML™ Files (In demo Directory)Formatting Data FilesFormatting RowsFormatting ColumnsRepresenting NULL ValuesEscapingEscaping in Text Formatted FilesEscaping in CSV Formatted FilesCharacter EncodingHAWQ InputFormat for MapReduceUsing PXF with Unmanaged Data


433435437444460462466470477479498502503505506507510512513515516517526533536541542543547548551553555558559561562564566567569570571574577579586587589590592594595596597603

Installing PXF Plug-insConfiguring PXFAccessing HDFS File DataAccessing Hive DataAccessing HBase DataAccessing JSON File DataAccessing External SQL Databases with JDBC (Beta)Writing Data to HDFSUsing Profiles to Read and Write DataPXF External Tables and APITroubleshooting PXFQuerying DataAbout HAWQ Query ProcessingAbout GPORCAOverview of GPORCAGPORCA Features and EnhancementsEnabling GPORCAConsiderations when Using GPORCADetermining The Query Optimizer In UseChanged Behavior with GPORCAGPORCA LimitationsDefining QueriesUsing Functions and OperatorsQuery PerformanceQuery ProfilingBest PracticesBest Practices for Configuring HAWQ ParametersBest Practices for Operating HAWQBest Practices for Securing HAWQBest Practices for Managing ResourcesBest Practices for Managing DataBest Practices for Querying DataTroubleshootingHAWQ ReferenceSQL CommandsABORTALTER AGGREGATEALTER DATABASEALTER CONVERSIONALTER FUNCTIONALTER OPERATORALTER OPERATOR CLASSALTER RESOURCE QUEUEALTER ROLEALTER SEQUENCEALTER TABLEALTER TABLESPACEALTER TYPEALTER USERANALYZEBEGINCHECKPOINTCLOSECOMMITCOPYCREATE AGGREGATE


607609611613619623624626630633637641643646654657659663664666667669670671672674675676678679680682684686688689690691692693694695696697698701704708710712714715716718719

CREATE CASTCREATE CONVERSIONCREATE DATABASECREATE EXTERNAL TABLECREATE FUNCTIONCREATE GROUPCREATE LANGUAGECREATE OPERATORCREATE OPERATOR CLASSCREATE RESOURCE QUEUECREATE ROLECREATE SCHEMACREATE SEQUENCECREATE TABLECREATE TABLE ASCREATE TABLESPACECREATE TYPECREATE USERCREATE VIEWDEALLOCATEDECLAREDROP AGGREGATEDROP CASTDROP CONVERSIONDROP DATABASEDROP EXTERNAL TABLEDROP FILESPACEDROP FUNCTIONDROP GROUPDROP LANGUAGEDROP OPERATORDROP OPERATOR CLASSDROP OWNEDDROP RESOURCE QUEUEDROP ROLEDROP SCHEMADROP SEQUENCEDROP TABLEDROP TABLESPACEDROP TYPEDROP USERDROP VIEWENDEXECUTEEXPLAINFETCHGRANTINSERTPREPAREREASSIGN OWNEDRELEASE SAVEPOINTRESETREVOKEROLLBACKROLLBACK TO SAVEPOINT


721723733734736738740741742744745747756806808812814818818822823825826828829830831832833834835836837838839840841842843844845846847848849850851853854855856857858859860861

SAVEPOINTSELECTSELECT INTOSETSET ROLESET SESSION AUTHORIZATIONSHOWTRUNCATEVACUUMServer Configuration Parameter ReferenceAbout Server Configuration ParametersConfiguration Parameter CategoriesConfiguration ParametersSample hawq-site.xml Configuration FileHDFS Configuration ReferenceEnvironment VariablesCharacter Set Support ReferenceData TypesTable 1. HAWQ Built-in Data TypesSystem Catalog ReferenceSystem TablesSystem ViewsSystem Catalogs Definitionsgp_configuration_historygp_distribution_policygp_global_sequencegp_master_mirroringgp_persistent_database_nodegp_persistent_filespace_nodegp_persistent_relation_nodegp_persistent_relfile_nodegp_persistent_tablespace_nodegp_relfile_nodegp_segment_configurationgp_version_at_initdbpg_aggregatepg_ampg_amoppg_amprocpg_appendonlypg_attrdefpg_attributepg_attribute_encodingpg_auth_memberspg_authidpg_castpg_classpg_compressionpg_constraintpg_conversionpg_databasepg_dependpg_descriptionpg_exttablepg_filespacepg_filespace_entry


862863864865866867868869870871872873874875876877878880881882883884885886887888889890891892893894895898899900905906909911914916918921924926929932934940943947955960962

pg_indexpg_inheritspg_languagepg_largeobjectpg_listenerpg_lockspg_namespacepg_opclasspg_operatorpg_partitionpg_partition_columnspg_partition_encodingpg_partition_rulepg_partition_templatespg_partitionspg_pltemplatepg_procpg_resqueuepg_resqueue_statuspg_rewritepg_rolespg_shdependpg_shdescriptionpg_stat_activitypg_stat_last_operationpg_stat_last_shoperationpg_stat_operationspg_stat_partition_operationspg_statisticpg_statspg_tablespacepg_triggerpg_typepg_type_encodingpg_windowThe hawq_toolkit Administrative SchemaHAWQ Management Tools Referenceanalyzedbcreatedbcreateuserdropdbdropusergpfdistgplogfilterhawq activatehawq checkhawq checkperfhawq confighawq extracthawq filespacehawq inithawq loadhawq registerhawq restarthawq scp


964966968971973975980984989

1004

hawq sshhawq ssh-exkeyshawq starthawq statehawq stoppg_dumppg_dumpallpg_restorepsqlvacuumdb


PivotalHDB2.3.0DocumentationPublished:July25,2018

Thisdocumentationdescribeshowtoinstall,configureandusethenewfeaturesofPivotalHDB2.3,whichincorporatesfeaturesfromApacheHAWQ®(incubating) .

KeytopicsinthePivotalHDB2.3.0documentationinclude:

ReleaseNotesandSystemRequirements

InstallingandupgradingPivotalHDB

SystemOverview

GettingStartedTutorial

ManagingDatawithHAWQ

UsingPXFwithUnmanagedData

UsingProceduralLanguages

BestPractices

Troubleshooting

Reference

Inaddition,specializedsubtopicsinclude:

ManagingHAWQUsingAmbari

AboutGPORCA

IntegratingYARNwithHAWQ

PXFJavaAPIReference


http://hawq.incubator.apache.org/http://hawq.incubator.apache.org/docs/pxf/javadoc/

PivotalHDB2.3.0ReleaseNotesPivotalHDB2.3.0isaminorreleaseoftheproductandisbasedonApacheHAWQ®(Incubating) .ThisreleaseincludesHAWQRangerHighAvailabilityandKerberossupport,BetasupportforthePXFHiveVectorizedORCandJDBCprofiles,andbugfixes.

SupportedPlatformsThesupportedplatformforrunningPivotalHDB2.3.0comprises:

RedHatEnterpriseLinux(RHEL)6.4+,7.2+(64-bit)(SeenoteinKnownIssuesandLimitationsforkernellimitations.)

HortonworksDataPlatform(HDP)2.5.3 .

Ambari2.4.2 (forAmbari-basedinstallationandHAWQclustermanagement).

EachPivotalHDBhostmachinemustalsomeettheApacheHAWQ(Incubating)systemrequirements.SeeApacheHAWQSystemRequirementsformoreinformation.

ProductSupportMatrixThefollowingtablesummarizesPivotalHDBproductsupportforcurrentandpreviousversionsofPivotalHDB,Hadoop,HAWQ,Ambari,andoperatingsystems.

PivotalHDBVersion

PXFVersion

HDPVersionRequirement

AmbariVersionRequirement

HAWQAmbariPlug-inRequirement

MADlibVersionRequirement

RHEL/CentOSVersionRequirement

SuSEVersionRequirement

2.3.0.0 3.3.0.0 2.5.3,2.6.1 2.4.2,2.5.1 2.3.0.0 1.10,1.11 6.4+,7.2+(64-bit) n/a

2.2.0.0 3.2.1.0 2.5.3,2.6.1 2.4.2,2.5.1 2.2.0.0 1.9,1.9.1,1.10 6.4+,7.2+(64-bit) n/a

2.1.2.0 3.2.0.0 2.5 2.4.1,2.4.2 2.1.2.0 1.9,1.9.1,1.10 6.4+(64-bit) n/a

2.1.1.0 3.1.1.0 2.5 2.4.1 2.1.1.0 1.9,1.9.1 6.4+(64-bit) n/a

2.1.0.0 3.1.0.0 2.5 2.4.1 2.1.0.0 1.9,1.9.1 6.4+(64-bit) n/a

2.0.1.0 3.0.1 2.4.0,2.4.2 2.2.2,2.4 2.0.1 1.9,1.9.1 6.4+(64-bit) n/a

2.0.0.0 3.0.0 2.3.4,2.4.0 2.2.2 2.0.0 1.9,1.9.1 6.4+(64-bit) n/a

1.3.1.1 2.5.1.1 2.2.6 2.0.x 1.3.11.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.1.0 2.5.1.1 2.2.6 2.0.x 1.3.1 1.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.3 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.2 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.1 1.3.3 2.2.4.2 1.7 1.11.7.1,1.8,1.9,1.9.1

6.4+ n/a

1.3.0.0 1.3.3 n/a n/a n/a1.7.1,1.8,1.9,1.9.1

n/a n/a

ProceduralLanguageSupportMatrixThefollowingtablesummarizescomponentversionsupportforProceduralLanguagesavailableinPivotalHDB2.x.TheversionslistedhavebeentestedwithPivotalHDB.Higherversionsmaybecompatible.Pleasetesthigherversionsthoroughlyinyournon-productionenvironmentsbeforedeployingtoproduction.

PivotalHDBVersion

PL/JavaJavaVersionRequirement

PL/RRVersionRequirement

PL/PerlPerlVersionRequirement

PL/PythonPythonVersionRequirement

2.3.0.0 1.7 3.3.1 5.10.1 2.6.2

2.2.0.0 1.7 3.3.1 5.10.1 2.6.2


http://hawq.incubator.apache.org/https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/ch_relnotes_v253.htmlhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-release-notes/content/ch_relnotes-ambari-2.4.2.0.html

2.1.x.0 1.7 3.3.1 5.10.1 2.6.2

2.0.1.0 1.7 3.3.1 5.10.1 2.6.2

2.0.0.0 1.6,1.7 3.1.0 5.10.1 2.6.2

PivotalHDBVersion





AWSSupportRequirementsPivotalHDBissupportedonAmazonWebServices(AWS)serversusingeitherAmazonblocklevelInstancestore(Amazonusesthevolumenamesephemeral[0-23])orAmazonElasticBlockStore(AmazonEBS)storage.Uselong-runningEC2instanceswiththeseforlong-runningHAWQinstances,asSpotinstancescanbeinterrupted.IfusingSpotinstances,minimizeriskofdatalossbyloadingfromandexportingtoexternalstorage.SeeAmazonEC2ConfigurationforadditionalHAWQAWS-relateddeploymentconsiderations.

PivotalHDB2.3.0FeaturesandChangesPivotalHDB2.3.0isbasedonApacheHAWQ(Incubating) ,andincludesthefollowingnewfeaturesascomparedtoPivotalHDB2.2.0:

HAWQRangerHighAvailabilityHDB2.3allowsyoutoconfigureastandbyHAWQRangerPluginService(RPS)forfailoverincaseswheretheprimaryRPSisunavailableornotrespondingonthelocalHAWQMasternode.ConfiguringastandbyRPSensuresthatHAWQcanauthorizeincomingrequestsatalltimeswithoutanydowntime.RefertotheHighAvailabilityandHAWQRangerdocumentationforadditionalinformation.

HAWQRangerKerberosSupportHDB2.3supportsHAWQRangerintegrationwhenKerberosisenabledforRangerand/orKerberosisenabledforHAWQuserauthentication.SeetheHAWQRangerKerberosIntegrationdocumentationfortheinformationnecessarytoconfigureKerberossupportforyourRanger-authorizedHAWQcluster.

PXFHiveVectorizedORCprofileHDB2.3includesaBetareleaseofOptimizedRowColumnar(ORC)fileformatwithVectorizedBatchReadersupport.RefertothePXFHiveplug-inORCFileFormatdocumentationforspecificinformationrelatedtothisnewfeature.

PXFJDBCplug-inHDB2.3includesaBetareleaseofthePXFJDBCplug-in.Withthisplug-in,youcancreateandqueryaHAWQexternaltablerepresentinganexternalSQLdatastoreusingaJDBCconnection.RefertotheAccessingExternalSQLDatabaseswithJDBC(Beta)documentationforspecificinformationrelatedtothePXFJDBCplug-in.

GettingStartedwithHAWQTutorialHDB2.3includestheGettingStartedwithHAWQTutorial.ThisguideprovidesaquickintroductionandvariousexercisestogetyouupandrunningwithyourHAWQinstallation.YouwilllearnabouttheHAWQruntimeenvironment,includingclusteranddatabaseadminstrationbasics.YouwillalsouseHAWQandPXFtoaccesssamplemanagedandunmanageddata.

PivotalHDB2.3.0UpgradeTheUpgradingfromPivotalHDB2.1.xor2.2guideprovidesspecificdetailsonapplyingthePivotalHDB2.3.0minorreleasetoyourHDB2.1.xor2.2.0installation.

Note:DirectupgradesfromHDB2.0.xarenotsupported.ThereisalsonodirectupgradepathfromPivotalHDB1.xtoHDB2.3.0.FormoreinformationonconsiderationsforupgradingfromPivotal1.xreleasestoPivotal2.xreleases,refertothePivotalHDB2.0documentation .ContactyourPivotalrepresentativeforassistanceinmigratingfromHDB1.xtoHDB2.x.

DifferencesComparedtoApacheHAWQ(Incubating)PivotalHDB2.3.0includesallofthefunctionalityinApacheHAWQ(Incubating) .

ResolvedIssuesThefollowingHAWQandPXFissueswereresolvedinPivotalHDB2.3.0.

ApacheJira Component Summary


http://hawq.incubator.apache.org/http://hdb.docs.pivotal.io/200/hdb/releasenotes/HAWQ20ReleaseNotes.html#upgradepathshttp://hawq.incubator.apache.org/

HAWQ-1404

PXF PXFnowleveragesthefile-levelstatisticsofanORCfile,andemitsrecordsfor COUNT(*)

HAWQ-1409

PXF HAWQsendsanadditionalheadertoPXFtoindicatetheaggregatefunctiontype

HAWQ-1417

Core Fixedacrashthatcouldoccurwhenexecuting ANALYZE after COPY

HAWQ-1422

Security HAWQnowprovidesuser-groupmembershipintheRangerPlug-inServicewhenrequestingaccess

HAWQ-1425

CommandLineTools

Fixedanincorrectinitclustererrormessagethatoccurredwhenansshconnectionfailed

HAWQ-1426

CommandLineTools

Fixedaproblemwhere hawq extract returnedanerrorafterthetablewasreorganized

HAWQ-1427

PXF ResolvedaPXFJSONProfile lang3 dependencyerror

HAWQ-1429

PXF PXFnolongerusesAggBridgewhena WHERE clauseisspecified

HAWQ-1430

Security UpdatedtheRanger-relatedlogleveltoavoidlogflooding

HAWQ-1431

PXF PXFnolongeruses StatsAccessor whenacolumnisspecifiedina SELECT clause

HAWQ-1433

ResourceManager

ALTER RESOURCE QUEUE DDLnowcheckstheformatoftheattributesMEMORY_CLUSTER_LIMITandCORE_CLUSTER_LIMIT

HAWQ-1434

PXF RemovedforceduppercasingofthetablenameinthePXFJDBCplug-in

HAWQ-1436

Security ImplementedRangerPlug-inServiceHighAvailabilityonHAWQ

HAWQ-1438

Core HAWQnowsupportsaresourceownerbeyondthetransactionboundary

HAWQ-1439

ResourceManager HAWQnowtoleratesthesystemtimebeingchangedtoanearlierpointwhencheckingtheresourcecontexttimeout

HAWQ-1440

PXF ANALYZE issupportedforHiveExternalTables

HAWQ-1443

Security ImplementedRangerlookupforHAWQwhenKerberosisenabled

HAWQ-1446

PXF IntroducedtheVectorizedprofileforORC

HAWQ-1449

CommandLineTools

HAWQstart/stopclusternowstarts/stopstheRangerPluginServiceonstandbynode

HAWQ-1451

CommandLineTools

The hawq state commandreportsthestatusofboththeRangerPluginServiceandtheStandbyRangerPluginService

HAWQ-1452

SecurityRemovedhawq_rps_address_suffixandhawq_rps_address_hostin hawq-site.xml tosimplifyconfigurationforRangerPlug-inServiceHighAvailability

HAWQ-1453

CoreFixedaproblemthatcausedtheerror:relation_close()reporterroratanalyzeStmt():isnotownedbyresourceownerTopTransaction(resowner.c:814)

HAWQ-1455

Core FixedincorrectresultsonCTASqueryovercatalog

HAWQ-1456

Security HAWQcopiesRangerPluginServiceconfigurationfilestothestandbyinspecificscenarios

HAWQ-1457

Core SharedmemoryforSegmentStatusandMetadataCacheisnolongerallocatedonsegments

HAWQ-1460

Core TheWALSendServerprocessnowexitsifpostmasteronmasteriskilled

HAWQ-1461

PXF ImprovedpartitionparametervalidationforthePXF-JDBCplugin

HAWQ-Security Improved enable-ranger-plugin.sh tosupportKerberos



https://issues.apache.org/jira/browse/HAWQ-1404https://issues.apache.org/jira/browse/HAWQ-1409https://issues.apache.org/jira/browse/HAWQ-1417https://issues.apache.org/jira/browse/HAWQ-1422https://issues.apache.org/jira/browse/HAWQ-1425https://issues.apache.org/jira/browse/HAWQ-1426https://issues.apache.org/jira/browse/HAWQ-1427https://issues.apache.org/jira/browse/HAWQ-1429https://issues.apache.org/jira/browse/HAWQ-1430https://issues.apache.org/jira/browse/HAWQ-1431https://issues.apache.org/jira/browse/HAWQ-1433https://issues.apache.org/jira/browse/HAWQ-1434https://issues.apache.org/jira/browse/HAWQ-1436https://issues.apache.org/jira/browse/HAWQ-1438https://issues.apache.org/jira/browse/HAWQ-1439https://issues.apache.org/jira/browse/HAWQ-1440https://issues.apache.org/jira/browse/HAWQ-1443https://issues.apache.org/jira/browse/HAWQ-1446https://issues.apache.org/jira/browse/HAWQ-1449https://issues.apache.org/jira/browse/HAWQ-1451https://issues.apache.org/jira/browse/HAWQ-1452https://issues.apache.org/jira/browse/HAWQ-1453https://issues.apache.org/jira/browse/HAWQ-1455https://issues.apache.org/jira/browse/HAWQ-1456https://issues.apache.org/jira/browse/HAWQ-1457https://issues.apache.org/jira/browse/HAWQ-1460https://issues.apache.org/jira/browse/HAWQ-1461https://issues.apache.org/jira/browse/HAWQ-1476

1476 HAWQ-1477

Security RangerPluginServiceconnectstoRangeradminunderkerberossecurity

HAWQ-1480

CommandLineTools

PackingacorefileinHAWQisnowpossible

HAWQ-1485

Security Theuser/passwordisusedinsteadofthecredentialscacheinRangerlookupforHAWQwhenKerberosisenabled

HAWQ-1486

PXF ResolvedaPANICthatoccurredwhileaccessingaPXFHDFStable

HAWQ-1487

Core Fixedadeadlockthatoccurredwhentryingtoprocessaninterruptinerrorhandling

HAWQ-1492

PXF Packagedthejdbc-pluginRPMwithPXFinstallation

HAWQ-1493

Security IntegratedtheRangerlookupJAASconfigurationintheranger-adminpluginJAR


KnownIssuesandLimitations

OperatingSystemHDBinstallationsrunningRHEL-7orCentOS7versionspriortoversion7.3mayexperienceanoperatingsystemissuethatcouldcauseHDBtohangwithlargeworkloads.RHEL7.3andCentOS7.3resolvethisissue.

RangerIntegrationwithHAWQRefertoLimitationsofRangerPolicyManagementforadiscussionoflimitationsrelatedtoHAWQintegrationwithRangerauthorization.

PXFGPSQL-3345-Totakeadvantageofthechangeinnumberofvirtualsegments,PXFexternaltablesmustbedroppedandrecreatedafterupdatingthedefault_hash_table_bucket_number serverconfigurationparameter.

GPSQL-3347-The LOCATION stringprovidedwhencreatingaPXFexternaltablemustuseonlyASCIIcharacterstoidentifyafilepath.Specifyingdouble-byteormulti-bytecharactersinafilepathreturnsthefollowingerror(formattedforclarity):

ERROR: remote component error (500) from 'IP_Address:51200': type Exception report message: File does not exist: /tmp/??????/ABC-??????-001.csv description: The server encountered an internal error that prevented it from fulfilling this request. exception: java.io.IOException: File does not exist: /tmp/??????/ABC-??????-001.csv (libchurl.c:897) (seg10 hdw2.hdp.local:40000 pid=389911) (dispatcher.c:1801)

PXFinaKerberos-securedclusterrequiresYARNtobeinstalledduetoadependencyonYARNlibraries.

InorderforPXFtointeroperatewithHBase,youmustmanuallyaddthePXFHBaseJARfiletotheHBaseclasspathafterinstallation.SeePost-InstallProcedureforHiveandHBaseonHDP.

HAWQ-974 -WhenusingcertainPXFprofilestoqueryagainstlargerfilesstoredinHDFS,usersmayoccasionallyexperiencehangingorquerytimeout.ThisisaknownissuethatwillbeimprovedinafutureHDBrelease.RefertoAddressingPXFMemoryIssuesforadiscussionoftheconfigurationoptionsavailabletoaddresstheseissuesinyourPXFdeployment.

The HiveORC profilesupportsaggregatequeries(count,min,max,etc.),buttheyhavenotyetbeenoptimizedtoleverageORCfile-andstripe-levelmetadata.

The HiveVectorizedORC profiledoesnotsupportthetimestampdatatypeorcomplextypes.

AmbariAmbari-managedclustersshouldonlyuseAmbariforsettingserverconfigurationparameters.Parametersmodifiedusingthe hawq configcommandwillbeoverwrittenonAmbaristartuporreconfiguration.

WheninstallingHAWQinaKerberos-securedcluster,theinstallationprocessmayreportawarning/failureinAmbariiftheHAWQconfigurationfor


https://issues.apache.org/jira/browse/HAWQ-1477https://issues.apache.org/jira/browse/HAWQ-1480https://issues.apache.org/jira/browse/HAWQ-1485https://issues.apache.org/jira/browse/HAWQ-1486https://issues.apache.org/jira/browse/HAWQ-1487https://issues.apache.org/jira/browse/HAWQ-1492https://issues.apache.org/jira/browse/HAWQ-1493https://issues.apache.org/jira/browse/HAWQ-974

resourcemanagementtypeisswitchedtoYARNmodeduringinstallation.ThewarningisrelatedtoHAWQnotbeingabletoregisterwithYARNuntiltheHDFS&YARNservicesarerestartedwithnewconfigurationsresultingfromtheHAWQinstallationprocess.

TheHAWQstandbymasterwillnotworkafteryouchangetheHAWQmasterportnumber.Toenablethestandbymasteryoumustfirstremoveandthenre-initializeit.SeeRemovingtheHAWQStandbyMasterandActivatingtheHAWQStandbyMaster.

TheAmbariRe-SynchronizeHAWQStandbyMasterserviceactionfailsifthereisanactiveconnectiontotheHAWQmasternode.TheHAWQtaskoutputshowstheerror, Active connections. Aborting shutdown... Ifthisoccurs,closeallactiveconnectionsandthentrythere-synchronizeactionagain.

TheAmbariRunServiceCheckactionforHAWQandPXFmaynotworkproperlyonasecureclusterifPXFisnotco-locatedwiththeYARNcomponent.

Inasecuredcluster,ifyoumovetheYARNResourceManagertoanotherhostyoumustmanuallyupdate hadoop.proxyuser.yarn.hosts intheHDFScore-site.xml filetomatchthenewResourceManagerhostname.Ifyoudonotperformthisstep,HAWQsegmentsfailtogetresourcesfromtheResourceManager.

TheAmbariStopHAWQServer(ImmediateMode)serviceactionor hawq stop -M immediate commandmaynotstopallHAWQmasterprocessesinsomecases.Several postgres processesownedbythe gpadmin usermayremainactive.

Ambaricheckswhetherthe hawq_rm_yarn_address and hawq_rm_yarn_scheduler_address valuesarevalidwhenYARNHAisnotenabled.InclustersthatuseYARNHA,thesepropertiesarenotusedandmaygetout-of-syncwiththeactiveResourceManager.ThiscanleadingtofalsewarningsfromAmbariifyoutrytochangethepropertyvalue.

AmbaridoesnotsupportCustomConfigurationGroupswithHAWQ.

CertainHAWQserverconfigurationparametersrelatedtoresourceenforcementarenotactive.ModifyingtheparametershasnoeffectinHAWQsincetheresourceenforcementfeatureisnotcurrentlysupported.Theseparametersinclude hawq_re_cgroup_hierarchy_name ,hawq_re_cgroup_mount_point ,and hawq_re_cpu_enable .TheseparametersappearintheAdvancedhawq-siteconfigurationsectionoftheAmbarimanagementinterface.

WorkaroundRequiredafterMovingNamenode

IfyouusetheAmbariMoveNamenodeWizardtomoveaHadoopnamenode,theWizarddoesnotautomaticallyupdatetheHAWQconfigurationtoreflectthechange.ThisleavesHAWQinannon-functionalstate,andwillcauseHAWQservicecheckstofailwithanerrorsimilarto:

2017-04-19 21:22:59,138 - SQL command executed failed: export PGPORT=5432 && source/usr/local/hawq/greenplum_path.sh && psql -d template1 -c \\\\\"CREATE TABLEambari_hawq_test (col1 int) DISTRIBUTED RANDOMLY;\\\\\"Returncode: 1Stdout:Stderr: Warning: Permanently added 'ip-10-32-36-168.ore1.vpc.pivotal.io,10.32.36.168'(RSA) to the list of known hosts.WARNING: could not remove relation directory 16385/1/18366: Input/output errorCONTEXT: Dropping file-system object -- Relation Directory: '16385/1/18366'ERROR: could not create relation directoryhdfs://ip-10-32-36-168.ore1.vpc.pivotal.io:8020/hawq_default/16385/1/18366: Input/output error

2016-04-19 21:22:59,139 - SERVICE CHECK FAILED: HAWQ was not able to write and queryfrom a table2016-04-19 21:23:02,608 - ** FAILURE **: Service check failed 1 of 3 checksstdout: /var/lib/ambari-agent/data/output-281.txt

Toworkaroundthisproblem,performoneofthefollowingproceduresafteryoucompletetheMoveNamenodeWizard.

WorkaroundforNon-HANameNodeClusters:1. PerformanHDFSservicechecktoensurethatHDFSisrunningproperlyafteryoumovedtheNameNode.

2. UsetheAmbari config.sh utilitytoupdate hawq_dfs_url tothenewNameNodeaddress.SeetheModifyconfigurations ontheAmbariWikiformoreinformation.Forexample:

$ cd /var/lib/ambari-server/resources/scripts/$ ./configs.sh set {ambari_server_host} {clustername} hawq-site$ hawq_dfs_url {new_namenode_address}:{port}/hawq_default

3. RestarttheHAWQconfigurationtoapplytheconfigurationchange.

4. Use ssh tologintoaHAWQnodeandrunthe checkpoint command:

$ psql -d template1 -c "checkpoint"


https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations

5. StoptheHAWQservice.

6. Themasterdatadirectoryisidentifiedinthe $GPHOME/etc/hawq-site.xml file hawq_master_directory propertyvalue.Copythemasterdatadirectorytoabackuplocation:

$ export MDATA_DIR=/value/from/hawqsite$ cp -r $MDATA_DIR /catalog/backup/location

7. ExecutethisquerytodisplayallavailableHAWQfilespaces:

SELECT fsname, fsedbid, fselocation FROM pg_filespace AS sp,pg_filespace_entry AS entry, pg_filesystem AS fs WHERE sp.fsfsys = fs.oidAND fs.fsysname = 'hdfs' AND sp.oid = entry.fsefsoid ORDER BYentry.fsedbid;

fsname | fsedbid | fselocation-------------+---------+------------------------------------------------cdbfast_fs_a | 0 | hdfs://hdfs-cluster/hawq//cdbfast_fs_adfs_system | 0 | hdfs://test5:9000/hawq/hawq-1459499690(2 rows)

8. Executethe hawq filespace commandoneachfilespacethatwasreturnedbythepreviousquery.Forexample:

$ hawq filespace --movefilespace dfs_system --location=hdfs://new_namenode:port/hawq/hawq-1459499690$ hawq filespace --movefilespace cdbfast_fs_a --location=hdfs://new_namenode:port/hawq//cdbfast_fs_a

9. IfyourclusterusesaHAWQstandbymaster,reinitializethestandbymasterinAmbariusingtheRemoveStandbyWizardfollowedbytheAddStandbyWizard.

10. StarttheHAWQService.

11. RunaHAWQservicechecktoensurethatalltestspass.

WorkaroundforHANameNodeClusters:1. PerformanHDFSservicechecktoensurethatHDFSisrunningproperlyafteryoumovedtheNameNode.

2. UseAmbaritoexpand Custom hdfs-client intheHAWQConfigstab,thenupdatethe dfs.namenode. propertiestomatchthecurrentNameNodeconfiguration.




PivotalHDB2.2.0ReleaseNotesPivotalHDB2.2.0isaminorreleaseoftheproductandisbasedonApacheHAWQ(Incubating) .ThisreleaseincludesRHEL/CentOS7support,BetasupportforApacheRangerintegration,andbugfixes.


RedHatEnterpriseLinux(RHEL)6.4+,7.2+(64-bit)(SeenoteinKnownIssuesandLimitationsforkernellimitations.)

CentOS7

HortonworksDataPlatform(HDP)2.5.3 .


EachPivotalHDBhostmachinemustalsomeettheApacheHAWQ(Incubating)systemrequirements.SeeApacheHAWQSystemRequirements formoreinformation.

ProductSupportMatrixThefollowingtablesummarizesPivotalHDBproductsupportforcurrentandpreviousversionsofPivotalHDB,Hadoop,HAWQ,Ambari,andoperatingsystems.

PivotalHDBVersion

PXFVersion







2.2.0.0 3.2.1.0 2.5.3,2.6.1 2.4.2,2.5.1 2.2.0.0 1.9,1.9.1,1.10 6.4+,7.2+(64-bit) n/a

2.1.2.0 3.2.0.0 2.5 2.4.1,2.4.2 2.1.2.0 1.9,1.9.1,1.10 6.4+(64-bit) n/a

2.1.1.0 3.1.1.0 2.5 2.4.1 2.1.1.0 1.9,1.9.1 6.4+(64-bit) n/a

2.1.0.0 3.1.0.0 2.5 2.4.1 2.1.0.0 1.9,1.9.1 6.4+(64-bit) n/a

2.0.1.0 3.0.1 2.4.0,2.4.2 2.2.2,2.4 2.0.1 1.9,1.9.1 6.4+(64-bit) n/a

2.0.0.0 3.0.0 2.3.4,2.4.0 2.2.2 2.0.0 1.9,1.9.1 6.4+(64-bit) n/a

1.3.1.1 2.5.1.1 2.2.6 2.0.x 1.3.11.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.1.0 2.5.1.1 2.2.6 2.0.x 1.3.1 1.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.3 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.2 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.1 1.3.3 2.2.4.2 1.7 1.11.7.1,1.8,1.9,1.9.1

6.4+ n/a

1.3.0.0 1.3.3 n/a n/a n/a1.7.1,1.8,1.9,1.9.1

n/a n/a

ProceduralLanguageSupportMatrixThefollowingtablesummarizescomponentversionsupportforProceduralLanguagesavailableinPivotalHDB2.x.TheversionslistedhavebeentestedwithPivotalHDB.Higherversionsmaybecompatible.Pleasetesthigherversionsthoroughlyinyournon-productionenvironmentsbeforedeployingtoproduction.

PivotalHDBVersion





2.2.0.0 1.7 3.3.1 5.10.1 2.6.2

2.1.x.0 1.7 3.3.1 5.10.1 2.6.2


http://hawq.incubator.apache.org/https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/ch_relnotes_v253.htmlhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-release-notes/content/ch_relnotes-ambari-2.4.2.0.htmlhttps://hdb.docs.pivotal.io/220/hawq/requirements/system-requirements.html

2.0.1.0 1.7 3.3.1 5.10.1 2.6.2

2.0.0.0 1.6,1.7 3.1.0 5.10.1 2.6.2

PivotalHDBVersion





AWSSupportRequirementsPivotalHDBissupportedonAmazonWebServices(AWS)serversusingeitherAmazonblocklevelInstancestore(Amazonusesthevolumenamesephemeral[0-23])orAmazonElasticBlockStore(AmazonEBS)storage.Uselong-runningEC2instanceswiththeseforlong-runningHAWQinstances,asSpotinstancescanbeinterrupted.IfusingSpotinstances,minimizeriskofdatalossbyloadingfromandexportingtoexternalstorage.

PivotalHDB2.2.0FeaturesandChangesPivotalHDB2.2.0isbasedonApacheHAWQ(Incubating) ,andincludesthefollowingnewfeaturesascomparedtoPivotalHDB2.1.2:

RangerIntegration(Beta)PivotalHDB2.2.0introducesBetasupportforApacheRanger.TheHAWQRangerPluginServiceisaRESTfulservicethatprovidesintegrationbetweenHDBandRangerpolicymanagement.RangerintegrationenablesyoutouseApacheRangertoauthorizeuseraccesstoHAWQresources.UsingRangerenablesyoutomanageallofyourHadoopcomponents’authorizationpoliciesusingthesameuserinterface,policystore,andauditingstores.RefertotheOverviewofRangerPolicyManagement forspecificinformationonHAWQintegrationwithRanger.

RHEL/CentOS7.2+supportPivotalHDB2.2.0nowprovidesproductdownloadsforRedHatEnterpriseLinux7andCentOS7operatingsystems.

PXFORCwithPivotalHDBPivotalHDB2.2.0nowfullysupportsPXFwithOptimizedRowColumnar(ORC)fileformat,whichwasformerlyBETA.

PivotalHDB2.2.0UpgradePivotalHDB2.2.0upgradepaths:

TheUpgradingfromPivotalHDB2.1.x guideprovidesspecificdetailsonapplyingthePivotalHDB2.2.0minorreleasetoyourHDB2.1.0,2.1.1,or2.1.2installation.

TheUpgradingfromPivotalHDB2.0.1 guidedetailsthestepsinvolvedtoupgradeyourPivotalHDB2.0.1installationtoHDB2.2.0.

Note:ThereisnodirectupgradepathfromPivotalHDB1.xtoHDB2.2.0.FormoreinformationonconsiderationsforupgradingfromPivotal1.xreleasestoPivotal2.xreleases,refertothePivotalHDB2.0documentation .ContactyourPivotalrepresentativeforassistanceinmigratingfromHDB1.xtoHDB2.x.

DifferencesComparedtoApacheHAWQ(Incubating)PivotalHDB2.2.0doesnotcurrentlysupportthePXFJDBCPlug-in.

Otherwise,PivotalHDB2.2.0includesallofthefunctionalityinApacheHAWQ(Incubating) .

ResolvedIssuesThefollowingHAWQandPXFissueswereresolvedinPivotalHDB2.2.0.


HAWQ-256 Security IntegratedSecuritywithApacheRanger(Partialbeta-releaseimplementation)

HAWQ-309 Build SupportforCentOS/RHEL7

HAWQ-944 Core Numutils.c: pg_ltoa and pg_itoa functionsallocatedunnecessaryamountsofbytes

HAWQ-1063 CommandLineTools FixedHAWQPythonlibrarymissingimport

HAWQ-1314 Catalog,Hcatalog,PXF Thepxf_get_item_fieldsfunctionwouldstopworkingafterupgrade

HAWQ-1347 Dispatcher QDshouldonlychecksegmenthealth


http://hawq.incubator.apache.org/https://hdb.docs.pivotal.io/220/hawq/ranger/ranger-overview.htmlhttps://hdb.docs.pivotal.io/220/hdb/install/HDB21xto22xUpgrade.htmlhttps://hdb.docs.pivotal.io/220/hdb/install/HDB20xto22xUpgrade.htmlhttp://hdb.docs.pivotal.io/200/hdb/releasenotes/HAWQ20ReleaseNotes.html#upgradepathshttp://hawq.incubator.apache.org/https://issues.apache.org/jira/browse/HAWQ-256https://issues.apache.org/jira/browse/HAWQ-309https://issues.apache.org/jira/browse/HAWQ-944https://issues.apache.org/jira/browse/HAWQ-1063https://issues.apache.org/jira/browse/HAWQ-1314https://issues.apache.org/jira/browse/HAWQ-1347

HAWQ-1365 Core Printoutdetailedschemainformationfortables,eveniftheuserdoesn’thaveprivilegesaccessprivilegesHAWQ-1366 Storage HAWQshouldthrowanerrorifitfindsadictionaryencodingtypeforParquet

HAWQ-1371 QueryExecution QEprocesshanginsharedinputscan

HAWQ-1378 Core Elaboratethe“invalidcommand-lineargumentsforserverprocess”error

HAWQ-1379 Core Donotsendoptionsmultipletimesinbuild_startup_packet

HAWQ-1385 CommandLineTools hawq_ctl stop failswhenthemasterisdown

HAWQ-1408 Core COPY…FROMSTDINcausesPANIC

HAWQ-1418 CommandLineTools Printexecutingcommandforhawqregister



MADlib1.9.xCompressionPivotalHDB2.2.0iscompatiblewithMADlib1.9,1.9.1,and1.10.IfyouhaveanexistingHDBinstallationwithMADlib1.9.xinstalled,orareinstallingMADlib1.9.x,youmustdownloadandexecuteascripttoremoveMADlib’suseofQuicklzcompression,whichisnotsupportedinHDB2.2.0.RunthisscriptifyouareupgradinganHDBinstallationwithMADlib1.9.xtoHDB2.2.0,orifyouareinstallingMADlib1.9.xonHDB2.2.0.

ThisprocedureisnotnecessaryifyouareusingorinstallingMADlib1.10,orifyouhavepreviouslydisabledQuicklzcompression.

IfyouareupgradinganHDB2.0systemthatcontainsMADlib:

1. CompletethePivotalHDB2.2.0upgradeprocedureasdescribedinUpgradingtoPivotalHDB2.2.0 .

2. DownloadandunpacktheMADlib1.9.xbinarydistributionfromthePivotalHDBDownloadPage onPivotalNetwork.

3. Executethe remove_compression.sh scriptintheMADlib1.9.xdistribution,providingthepathtoyourexistingMADlibinstallation:

$ remove_compression.sh --prefix

Note:Ifyoudonotincludethe --prefix option,thescriptusesthelocation ${GPHOME}/madlib .

FornewMADlibinstallations,completethesestepsafteryouinstallPivotalHDB2.2.0:

1. DownloadandunpacktheMADlib1.9.xbinarydistributionfromthePivotalHDBDownloadPage onPivotalNetwork.

2. InstalltheMADlib .gppkg file:

$ gppkg -i /madlib-ossv1.9.1_pv1.9.6_hawq2.1-rhel5-x86_64.gppkg

3. Executethe remove_compression.sh script,optionallyprovidingtheMADlibinstallationpath:



4. ContinueinstallingMADlibusingthe madpack install commandasdescribedintheMADlibInstallationGuide .Forexample:

$ madpack –p hawq install

OperatingSystemSomeLinuxkernelversionsbetween2.6.32to4.3.3(notincluding2.6.32and4.3.3)haveabugthatcouldintroducea getaddrinfo() functionhang.Toavoidthisissue,upgradeyourRHEL-6kerneltoversion4.3.3+.

IfyouarerunningRHEL-7,ensurethatyourkernelversionis3.10.0-327.27orabove,otherwiseyoumayexperiencehangswithlargeworkloads.

RangerIntegration(Beta)


https://issues.apache.org/jira/browse/HAWQ-1365https://issues.apache.org/jira/browse/HAWQ-1366https://issues.apache.org/jira/browse/HAWQ-1371https://issues.apache.org/jira/browse/HAWQ-1378https://issues.apache.org/jira/browse/HAWQ-1379https://issues.apache.org/jira/browse/HAWQ-1385https://issues.apache.org/jira/browse/HAWQ-1408https://issues.apache.org/jira/browse/HAWQ-1418https://hdb.docs.pivotal.io/220/hdb/install/HDB20xto22xUpgrade.htmlhttps://network.pivotal.io/products/pivotal-hdbhttps://network.pivotal.io/products/pivotal-hdbhttps://cwiki.apache.org/confluence/display/MADLIB/Installation+Guide

Beta-levelsupportofRangerPluginServicehasanumberofknownlimitations.Formoreinformation,refertoLimitationsofRangerPolicyManagement .





InorderforPXFtointeroperatewithHBase,youmustmanuallyaddthePXFHBaseJARfiletotheHBaseclasspathafterinstallation.SeePost-InstallProcedureforHiveandHBaseonHDP .

HAWQ-974 -WhenusingcertainPXFprofilestoqueryagainstlargerfilesstoredinHDFS,usersmayoccasionallyexperiencehangingorquerytimeout.ThisisaknownissuethatwillbeimprovedinafutureHDBrelease.RefertoAddressingPXFMemoryIssues foradiscussionoftheconfigurationoptionsavailabletoaddresstheseissuesinyourPXFdeployment.

The HiveORC profilesupportsaggregatequeries(count,min,max,etc.),buttheyhavenotyetbeenoptimizedtoleverageORCfile-andstripe-levelmetadata.

The HiveORC profiledoesnotyetusetheVectorizedBatchreader.

PL/RTheHAWQPL/RextensionisprovidedasaseparateRPMinthe hdb-add-ons-2.2.0.0 repository.ThefilesinstalledbythisRPMareownedby root .IfyouinstalledHAWQviaAmbari,HAWQfilesareownedby gpadmin .PerformthefollowingstepsoneachnodeinyourHAWQclusterafterPL/RRPMinstallationtoaligntheownershipofPL/Rfiles:

root@hawq-node$ cd /usr/local/hawqroot@hawq-node$ chown gpadmin:gpadmin share/postgresql/contrib/plr.sql docs/contrib/README.plr lib/postgresql/plr.so

AmbariAmbari-managedclustersshouldonlyuseAmbariforsettingserverconfigurationparameters.Parametersmodifiedusingthe hawq config commandwillbeoverwrittenonAmbaristartuporreconfiguration.

WheninstallingHAWQinaKerberos-securedcluster,theinstallationprocessmayreportawarning/failureinAmbariiftheHAWQconfigurationforresourcemanagementtypeisswitchedtoYARNmodeduringinstallation.ThewarningisrelatedtoHAWQnotbeingabletoregisterwithYARNuntiltheHDFS&YARNservicesarerestartedwithnewconfigurationsresultingfromtheHAWQinstallationprocess.

TheHAWQstandbymasterwillnotworkafteryouchangetheHAWQmasterportnumber.Toenablethestandbymasteryoumustfirstremoveandthenre-initializeit.SeeRemovingtheHAWQStandbyMaster andActivatingtheHAWQStandbyMaster .








https://hdb.docs.pivotal.io/220/hawq/ranger/ranger-overview.html#limitationshttps://hdb.docs.pivotal.io/220/hdb/install/install-ambari.html#post-install-pxfhttps://issues.apache.org/jira/browse/HAWQ-974https://hdb.docs.pivotal.io/220/hawq/pxf/TroubleshootingPXF.html#pxf-memcfghttps://hdb.docs.pivotal.io/220/hawq/admin/ambari-admin.html#amb-remove-standbyhttps://hdb.docs.pivotal.io/220/hawq/admin/ambari-admin.html#amb-activate-standby

















8. SELECT fsname, fsedbid, fselocation FROM pg_filespace AS sp,pg_filespace_entry AS entry, pg_filesystem AS fs WHERE sp.fsfsys = fs.oidAND fs.fsysname = 'hdfs' AND sp.oid = entry.fsefsoid ORDER BYentry.fsedbid;



PivotalHDB2.1.2ReleaseNotesHDB2.1.2isaminorreleaseoftheproductandisbasedonApacheHAWQ2.1.0.0(Incubating) .Thisreleaseincludesbugfixes,enhancementstotheBetareleaseofOptimizedRowColumnar(ORC)fileformatsupport,andenhancementstothePXFHiveplug-in.


RedHatEnterpriseLinux(RHEL)6.4+(64-bit)(SeenoteinKnownIssuesandLimitationsforkernellimitations.)

HortonworksDataPlatform(HDP)2.5 .

Ambari2.4.1 orAmbari2.4.2 (forAmbari-basedinstallationandHAWQclustermanagement).


ProductSupportMatrixThefollowingtablesummarizesPivotalHDBproductsupportforcurrentandpreviousversionsofHDB,Hadoop,HAWQ,Ambari,andoperatingsystems.

PivotalHDBVersion

PXFVersion







2.1.2.0 3.2.0.0 2.5 2.4.1,2.4.2 2.1.2.0 1.9,1.9.1,1.10 6.4+(64-bit) n/a

2.1.1.0 3.1.1.0 2.5 2.4.1 2.1.1.0 1.9,1.9.1 6.4+(64-bit) n/a

2.1.0.0 3.1.0.0 2.5 2.4.1 2.1.0.0 1.9,1.9.1 6.4+(64-bit) n/a

2.0.1.0 3.0.1 2.4.0,2.4.2 2.2.2,2.4 2.0.1 1.9,1.9.1 6.4+(64-bit) n/a

2.0.0.0 3.0.0 2.3.4,2.4.0 2.2.2 2.0.0 1.9,1.9.1 6.4+(64-bit) n/a

1.3.1.1 2.5.1.1 2.2.6 2.0.x 1.3.11.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.1.0 2.5.1.1 2.2.6 2.0.x 1.3.11.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.3 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.2 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.1 1.3.3 2.2.4.2 1.7 1.11.7.1,1.8,1.9,1.9.1

6.4+ n/a

1.3.0.0 1.3.3 n/a n/a n/a1.7.1,1.8,1.9,1.9.1

n/a n/a

Note:RHEL/CentOS7isnotsupported.

Note:IfyouareusingAmbari2.4.1andyouwanttoinstallbothHDPandHAWQatthesametime,seeInstallingHDPandHDBwithAmbari2.4.1beforeyoubegin.

ProceduralLanguageSupportMatrixThefollowingtablesummarizescomponentversionsupportforProceduralLanguagesavailableinPivotalHDB2.x.TheversionslistedhavebeentestedwithHDB.Higherversionsmaybecompatible.Pleasetesthigherversionsthoroughlyinyournon-productionenvironmentsbeforedeployingtoproduction.

PivotalHDBVersion





2.1.2.0 1.7 3.3.1 5.10.1 2.6.2

2.1.1.0 1.7 3.3.1 5.10.1 2.6.2


http://hawq.incubator.apache.org/https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_release-notes/content/ch_relnotes_v250.htmlhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-release-notes/content/ch_relnotes-ambari-2.4.1.0.htmlhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-release-notes/content/ch_relnotes-ambari-2.4.2.0.htmlhttps://hdb.docs.pivotal.io/212/hawq/requirements/system-requirements.html

2.1.1.0 1.7 3.3.1 5.10.1 2.6.2

2.1.0.0 1.7 3.3.1 5.10.1 2.6.2

2.0.1.0 1.7 3.3.1 5.10.1 2.6.2

2.0.0.0 1.6,1.7 3.1.0 5.10.1 2.6.2

PivotalHDBVersion






PivotalHDB2.1.2FeaturesandChangesPivotalHDB2.1.2isbasedonApacheHAWQ2.1.0.0(Incubating) ,andincludesthefollowingnewfeaturesascomparedtoPivotalHDB2.1.1.0:

ORCfileformatsupportenhancementsHDB2.1.2includesenhancementstotheOptimizedRowColumnar(ORC)fileformatsupportBetareleasedinHDB2.1.1.RefertotheORCBetadocumentation forspecificinformationrelatedtotheseenhancements.

PXFHivePlug-inenhancementsPXFnowselectstheoptimalHiveprofilefortheunderlyingfilestoragetypeofeachtable/partition/fragmentwhenrespondingtoaHAWQqueryofHivedata.TheoptimalprofileisselectedbothforexternaltableandHCatalogintegrationqueries.PXFimplementationchangesforthisimprovementrequireAmbari-managedclusterstoperformanadditionalpost-install/upgradeprocedure .

InstallingHDPandHDBwithAmbari2.4.1IfyouareusingAmbari2.4.1andyouwanttoinstallbothHDPandHAWQatthesametime,specialcaremustbetakenifyouwanttoinstalltheverylatestversionoftheHDPstackinsteadofthedefaultversion.Followthesesteps:

1. AfterinstallingAmbari,starttheClusterInstallWizardandproceeduntilyoureachtheSelectVersionscreen.

2. OntheSelectVersionscreen,selectHDB-2.5fromthelistofavailablestackversions.

3. WhilestillontheSelectVersionscreen,copytheBaseURLvaluesforthe HDP-2.5 and HDP-UTILS-1.1.0.21 repositoriesthatarelistedforyouroperatingsystem.Pastethesevaluesintoatemporaryfile;youwillneedtorestoretheseBaseURLvalueslater.

4. Usethedrop-downmenuforHDP-2.5toselectthestackoption,HDP-2.5(DefaultVersionDefinition).Verifythatthe hdb-2.1.2.0 andhdb-add-ons-2.1.2.0 repositoriesnowappearinthelistofRepositoriesforyouroperatingsystem.

5. ToinstalltheverylatestversionofHDP,replacetheBaseURLvaluesforthe HDP-2.5 and HDP-UTILS-1.1.0.21 repositorieswiththevaluesyoupastedintothetextfileinStep3.

6. ClickNexttocontinue,andfinishinstallingthenewHDPcluster.

7. InstallandconfigureHDPasdescribedinInstallingHAWQUsingAmbari .

Note:ThisworkaroundmaynotberequiredwithlaterversionsofAmbari2.4.

HDB2.1.2UpgradeHDB2.1.2upgradepaths:

TheUpgradingfromHDB2.1.x guideprovidesspecificdetailsonapplyingtheHDB2.1.2maintenancereleasetoyourHDB2.1.0orHDB2.1.1installation.

TheUpgradingfromHDB2.0.x guidedetailsthestepsinvolvedtoupgradeyourHDB2.0.xinstallationtoHDB2.1.2.

Note:IfyouareupgradinganHDBversionpriorto2.0,refertotheHDB2.0documentation .


http://hawq.incubator.apache.org/https://hdb.docs.pivotal.io/212/hdb/releasenotes/orc-support-beta.htmlhttps://hdb.docs.pivotal.io/212/hdb/install/install-ambari.html#post-install-212-reqhttps://hdb.docs.pivotal.io/212/hdb/install/install-ambari.htmlhttps://hdb.docs.pivotal.io/212/hdb/install/HDB21xto21xUpgrade.htmlhttps://hdb.docs.pivotal.io/212/hdb/install/HDB20xto21xUpgrade.htmlhttp://hdb.docs.pivotal.io/200/hdb/index.html

DifferencesComparedtoApacheHAWQ(Incubating)PivotalHDB2.1.2includesallofthefunctionalityinApacheHAWQ2.1.0.0(Incubating) andadditionalbugfixes,notedbyanasterisk(*)intheResolvedIssuestablebelow.

ResolvedIssuesThefollowingHAWQandPXFissueswereresolvedinHDB2.1.2.


HAWQ-762 PXF HiveaggregationqueriesthroughPXFwouldsometimeshang

HAWQ-870 QueryExecution Allocatetarget’stupletableslotinPortalHeapMemoryduringsplitpartition

HAWQ-1177 HCatalog,PXF UseprofilebasedonfileformatinHCatalogintegrationforHiveORCprofile

HAWQ-1208 Interconnect Fixedrandominterconnectfailure

HAWQ-1214 ResourceManager Removedresource_parameters

HAWQ-1215 PXF PXFHiveORCprofiledidnothandlecomplextypescorrectly

HAWQ-1227 CommandLineTools HAWQinitwouldfailifusernamecontainscapitalcharacter

HAWQ-1228 HCatalog,PXF UseprofilebasedonfileformatinHCatalogintegration(HiveRC,HiveTextprofiles)

HAWQ-1229 CommandLineTools Removedunusedoptionin‘hawqconfig’helpmessage

HAWQ-1240 QueryExecution Fixedbuginplanrefinementforcursoroperation

HAWQ-1241 Core Noneedtosetext/pythonin*PATHinfilegreenplum_path.sh

HAWQ-1242 Core hawq-site.xmldefaultcontenthadwronggucvariablenames

HAWQ-1258 ResourceManager Segmentresourcemanagerdidnotswitchbackwhenitcouldnotresolvestandbyhostname

HAWQ-1282 Core SharedInputScanwouldresultinendlessloop

HAWQ-1285 ResourceManager Resourcemanagerwouldoutputuninitializedstringashostname

HAWQ-1286 Security Reducedunnecessarycallstonamespacecheckwhenrun\d

HAWQ-1308 PXF FixedJavadoccompilewarnings

HAWQ-1309 PXF PXFservicemustdefaulttoport51200anduserpxf

HAWQ-1314 Catalog,HCatalog,PXF Fixedpost-upgradepxf_get_item_fields()functionbreak

HAWQ-1315 ResourceManager FunctionvalidateResourcePoolStatus()inresourcepool.cloggedthewronginformation

HAWQ-1317 Security Ported“Fixsomeregexissueswithout-of-rangecharactersandlargecharranges”frompg

HAWQ-1321 Core failNamesincorrectlyusedmemorycontexttobuildmessagewhenANALYZEfailed

HAWQ-1324 QueryExecution Querycancelcausedthesegmenttogointocrashrecovery

HAWQ-1326 QueryExecution Cancelthequeryearlierifoneofthesegmentsforthequerycrashes

HAWQ-1334 Dispatcher QDthreadnowsetserrorcodeiffailingsothatthemainprocessforthequerycouldexitsoon

HAWQ-1338 Core Insomecaseswriterprocesscrashedwhenrunning'hawqstopcluster’

HAWQ-1345* Catalog Couldnotcountblocksofrelation:Notadirectory

HAWQ-1347* Dispatcher QDwouldchecksegmenthealthonly

Note*:HDB2.1.2includesresolvedissuesHAWQ-1345andHAWQ-1347,additionalbugfixesappliedtotheApacheHAWQ2.1.0.0(Incubating)release.


MADlibCompressionPivotalHDB2.1.2iscompatiblewithMADlib1.9,1.9.1,and1.10.IfyouhaveanexistingHDBinstallationwithMADlib1.9.xinstalled,orareinstallingMADlib1.9.x,youmustdownloadandexecuteascripttoremoveMADlib’suseofQuicklzcompression,whichisnotsupportedinHDB2.1.2.RunthisscriptifyouareupgradinganHDBinstallationwithMADlib1.9.xtoHDB2.1.2,orifyouareinstallingMADlib1.9.xonHDB2.1.2.


http://hawq.incubator.apache.org/https://issues.apache.org/jira/browse/HAWQ-762https://issues.apache.org/jira/browse/HAWQ-870https://issues.apache.org/jira/browse/HAWQ-1177https://issues.apache.org/jira/browse/HAWQ-1208https://issues.apache.org/jira/browse/HAWQ-1214https://issues.apache.org/jira/browse/HAWQ-1215https://issues.apache.org/jira/browse/HAWQ-1227https://issues.apache.org/jira/browse/HAWQ-1228https://issues.apache.org/jira/browse/HAWQ-1229https://issues.apache.org/jira/browse/HAWQ-1240https://issues.apache.org/jira/browse/HAWQ-1241https://issues.apache.org/jira/browse/HAWQ-1242https://issues.apache.org/jira/browse/HAWQ-1258https://issues.apache.org/jira/browse/HAWQ-1282https://issues.apache.org/jira/browse/HAWQ-1285https://issues.apache.org/jira/browse/HAWQ-1286https://issues.apache.org/jira/browse/HAWQ-1308https://issues.apache.org/jira/browse/HAWQ-1309https://issues.apache.org/jira/browse/HAWQ-1314https://issues.apache.org/jira/browse/HAWQ-1315https://issues.apache.org/jira/browse/HAWQ-1317https://issues.apache.org/jira/browse/HAWQ-1321https://issues.apache.org/jira/browse/HAWQ-1324https://issues.apache.org/jira/browse/HAWQ-1326https://issues.apache.org/jira/browse/HAWQ-1334https://issues.apache.org/jira/browse/HAWQ-1338https://issues.apache.org/jira/browse/HAWQ-1345https://issues.apache.org/jira/browse/HAWQ-1347

ThisprocedureisnotnecessaryifyouareusingorinstallingMADlib1.10,orifyouhavepreviouslydisabledQuicklzcompression.



2. DownloadandunpacktheMADlib1.9.1binarydistributionfromthePivotalHDBDownloadPage onPivotalNetwork.

3. Executethe remove_compression.sh scriptintheMADlib1.9.1distribution,providingthepathtoyourexistingMADlibinstallation:












OperatingSystemSomeLinuxkernelversionsbetween2.6.32to4.3.3(notincluding2.6.32and4.3.3)haveabugthatcouldintroducea getaddrinfo() functionhang.Toavoidthisissue,upgradethekerneltoversion4.3.3+.




ORC-RefertoORCKnownIssuesandLimitations foralistofknownissuesrelatedtotheORCBeta.




PL/RTheHAWQPL/RextensionisprovidedasaseparateRPMinthe hdb-add-ons-2.1.2.0 repository.ThefilesinstalledbythisRPMareownedby root .Ifyou


https://hdb.docs.pivotal.io/212/hdb/install/HDB20xto21xUpgrade.htmlhttps://network.pivotal.io/products/pivotal-hdbhttps://network.pivotal.io/products/pivotal-hdbhttps://cwiki.apache.org/confluence/display/MADLIB/Installation+Guidehttps://hdb.docs.pivotal.io/212/hdb/releasenotes/orc-support-beta.html#hiveorc-known-issueshttps://hdb.docs.pivotal.io/212/hdb/install/install-ambari.html#post-install-pxfhttps://issues.apache.org/jira/browse/HAWQ-974https://hdb.docs.pivotal.io/212/hawq/pxf/TroubleshootingPXF.html#pxf-memcfg

installedHAWQviaAmbari,HAWQfilesareownedby gpadmin .PerformthefollowingstepsoneachnodeinyourHAWQclusterafterPL/RRPMinstallationtoaligntheownershipofPL/Rfiles:


AmbariAmbari-managedclustersshouldonlyuseAmbariforsettingserverconfigurationparameters.Parametersmodifiedusingthe hawq config commandwillbeoverwrittenonAmbaristartuporreconfiguration.

Incertainconfigurations,theHAWQMastermayfailtostartinAmbariversionspriorto2.4.2when webhdfs isdisabled.RefertoAMBARI-18837 .Toworkaroundthisissue,enable webhdfs bysetting dfs.webhdfs.enabled to True in hdfs-site.xml ,orcontactSupport.
















https://issues.apache.org/jira/browse/AMBARI-18837https://hdb.docs.pivotal.io/212/hawq/admin/ambari-admin.html#amb-remove-standbyhttps://hdb.docs.pivotal.io/212/hawq/admin/ambari-admin.html#amb-activate-standby











8. SELECT fsname, fsedbid, fselocation FROM pg_filespace AS sp,pg_filespace_entry AS entry, pg_filesystem AS fs WHERE sp.fsfsys = fs.oidAND fs.fsysname = 'hdfs' AND sp.oid = entry.fsefsoid ORDER BYentry.fsedbid;













PivotalHDB2.1.1ReleaseNotesHDB2.1.1includesaBetareleaseofOptimizedRowColumnar(ORC)fileformatsupportforHAWQ.


RedHatEnterpriseLinux(RHEL)6.4+(64-bit)(SeenoteinKnownIssuesandLimitationsforkernellimitations.)

HortonworksDataPlatform(HDP)2.5 .



ProductSupportMatrixThefollowingtablesummarizesPivotalHDBproductsupportforcurrentandpreviousversionsofHDB,Hadoop,HAWQ,Ambari,andoperatingsystems.

PivotalHDBVersion

PXFVersion







2.1.1.0 3.1.1 2.5 2.4.1 2.1.1 1.9,1.9.1 6.4+(64-bit) n/a

2.1.0.0 3.1.0 2.5 2.4.1 2.1.0 1.9,1.9.1 6.4+(64-bit) n/a

2.0.1.0 3.0.1 2.4.0,2.4.2 2.2.2,2.4 2.0.1 1.9,1.9.1 6.4+(64-bit) n/a

2.0.0.0 3.0.0 2.3.4,2.4.0 2.2.2 2.0.0 1.9,1.9.1 6.4+(64-bit) n/a

1.3.1.1 2.5.1.1 2.2.6 2.0.x 1.3.11.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.1.0 2.5.1.1 2.2.6 2.0.x 1.3.11.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.3 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.2 1.3.3 2.2.4.2 1.7 1.21.7.1,1.8,1.9,1.9.1

6.4+ SLES11SP3

1.3.0.1 1.3.3 2.2.4.2 1.7 1.11.7.1,1.8,1.9,1.9.1

6.4+ n/a

1.3.0.0 1.3.3 n/a n/a n/a1.7.1,1.8,1.9,1.9.1

n/a n/a

Note:RHEL/CentOS7isnotsupported.

Note:IfyouareusingAmbari2.4.1andyouwanttoinstallbothHDPandHAWQatthesametime,seeInstallingHDPandHDBwithAmbari2.4.1beforeyoubegin.

ProceduralLanguageSupportMatrixThefollowingtablesummarizescomponentversionsupportforProceduralLanguagesavailableinPivotalHDB2.x.TheversionslistedhavebeentestedwithHDB.Higherversionsmaybecompatible.Pleasetesthigherversionsthoroughlyinyournon-productionenvironmentsbeforedeployingtoproduction.

PivotalHDBVersion





2.1.1.0 1.7 3.3.1 5.10.1 2.6.2

2.1.0.0 1.7 3.3.1 5.10.1 2.6.2

2.0.1.0 1.7 3.3.1 5.10.1 2.6.2

2.0.0.0 1.6,1.7 3.1.0 5.10.1 2.6.2


https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_release-notes/content/ch_relnotes_v250.htmlhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-release-notes/content/ch_relnotes-ambari-2.4.1.0.htmlhttps://hdb.docs.pivotal.io/211/hawq/requirements/system-requirements.html

2.0.0.0 1.6,1.7 3.1.0 5.10.1 2.6.2PivotalHDBVersion






PivotalHDB2.1.1FeaturesandChangesPivotalHDB2.1.1isbasedonApacheHAWQ(Incubating),andincludesthefollowingnewfeaturesascomparedtoPivotalHDB2.1.0.0:

ORCfileformatsupportHDB2.1.1includesaBetareleaseofOptimizedRowColumnar(ORC)fileformatsupport.RefertotheORCBetadocumentation forspecificinformationrelatedtothisnewfeature.

InstallingHDPandHDBwithAmbari2.4.1IfyouareusingAmbari2.4.1andyouwanttoinstallbothHDPandHAWQatthesametime,specialcaremustbetakenifyouwanttoinstalltheverylatestversionoftheHDPstackinsteadofthedefaultversion.Followthesesteps:

1. AfterinstallingAmbari,starttheClusterInstallWizardandproceeduntilyoureachtheSelectVersionscreen.

2. OntheSelectVersionscreen,selectHDB-2.5fromthelistofavailablestackversions.

3. WhilestillontheSelectVersionscreen,copytheBaseURLvaluesforthe HDP-2.5 and HDP-UTILS-1.1.0.21 repositoriesthatarelistedforyouroperatingsystem.Pastethesevaluesintoatemporaryfile;youwillneedtorestoretheseBaseURLvalueslater.

4. Usethedrop-downmenuforHDP-2.5toselectthestackoption,HDP-2.5(DefaultVersionDefinition).Verifythatthe hdb-2.1.1.0 andhdb-add-ons-2.1.1.0 repositoriesnowappearinthelistofRepositoriesforyouroperatingsystem.

5. ToinstalltheverylatestversionofHDP,replacetheBaseURLvaluesforthe HDP-2.5 and HDP-UTILS-1.1.0.21 repositorieswiththevaluesyoupastedintothetextfileinStep3.

6. ClickNexttocontinue,andfinishinstallingthenewHDPcluster.

7. InstallandconfigureHDPasdescribedinInstallingHAWQUsingAmbari .

Note:ThisworkaroundmaynotberequiredwithlaterversionsofAmbari2.4.

HDB2.1.xUpgradeHDB2.1.1upgradepaths:

TheUpgradingfromHDB2.1.0 guideprovidesspecificdetailsonapplyingtheHDB2.1.1maintenancereleasetoyourHDB2.1.0installation.

TheUpgradingfromHDB2.0.x guidedetailsthestepsinvolvedtoupgradeyourHDB2.0.xinstallationtoHDB2.1.1.

Note:IfyouareupgradinganHDBversionpriorto2.0,refertotheHDB2.0documentation .

DifferencesComparedtoApacheHAWQ(Incubating)PivotalHDB2.1.1includesallofthefunctionalityinApacheHAWQ(Incubating) ,andaddsseveralbugfixesdescribedbelow.

ResolvedIssuesThefollowingHAWQandPXFissueswereresolvedinHDB2.1.1.


https://hdb.docs.pivotal.io/211/hdb/releasenotes/orc-support-beta.htmlhttps://hdb.docs.pivotal.io/211/hdb/install/install-ambari.htmlhttps://hdb.docs.pivotal.io/211/hdb/install/HDB210to21xUpgrade.htmlhttps://hdb.docs.pivotal.io/211/hdb/install/HDB20xto210Upgrade.htmlhttp://hdb.docs.pivotal.io/200/hdb/index.htmlhttp://hawq.incubator.apache.org/


HAWQ-583

PXFExtendedPXFtoenablepluginstosupportreturningpartialcontentfrom SELECT(column projection)statements

HAWQ-779

PXF SupportmorePXFfilterpushdown

HAWQ-931

PXF HiveORCAccessorwithsupportforPredicatepushdownandColumnProjection

HAWQ-964

PXF SupportforadditionallogicaloperatorsinPXF

HAWQ-1103

PXF FixtosendconstantdatatypeandlengthinfilterstringtoPXFservice

HAWQ-1111

PXF SupportforIN()operatorinPXF

HAWQ-1191

PXF GetridofDELIMITERpropertyforHiveORCprofile

HAWQ-1196

PXF PartitionedtablessupportforHiveORCprofile

HAWQ-1213

CommandLineTools

Incorrectcheckof hawq register incaseofrandomlydistributedtablewithnon-defaultdefault_hash_table_bucket_number value

HAWQ-1221

CommandLineTools

hawq register shoulderroroutwhenregisteringaYMLfilethatdoesn’texist


MADlibCompressionPivotalHDB2.1.1iscompatiblewithMADlib1.9and1.9.1.However,youmustdownloadandexecuteascriptinordertoremovetheMADlibQuicklzcompression,whichisnotsupportedinHDB2.1.1.RunthisscriptifyouareupgradingtoHDB2.1.1,orifyouareinstallingMADlibonHDB2.1.1.




3. Executethe remove_compression.sh scriptintheMADlib1.9.1distribution,providingthepathtoyourexistingMADlibinstallation:













https://issues.apache.org/jira/browse/HAWQ-583https://issues.apache.org/jira/browse/HAWQ-779https://issues.apache.org/jira/browse/HAWQ-931https://issues.apache.org/jira/browse/HAWQ-964https://issues.apache.org/jira/browse/HAWQ-1103https://issues.apache.org/jira/browse/HAWQ-1111https://issues.apache.org/jira/browse/HAWQ-1191https://issues.apache.org/jira/browse/HAWQ-1196https://issues.apache.org/jira/browse/HAWQ-1213https://issues.apache.org/jira/browse/HAWQ-1221https://hdb.docs.pivotal.io/211/hdb/install/HDB20xto210Upgrade.htmlhttps://network.pivotal.io/products/pivotal-hdbhttps://network.pivotal.io/products/pivotal-hdbhttps://cwiki.apache.org/confluence/display/MADLIB/Installation+Guide

OperatingSystemSomeLinuxkernelversionsbetween2.6.32to4.3.3(notincluding2.6.32and4.3.3)haveabugthatcouldintroducea getaddrinfo() functionhang.Toavoidthisissue,upgradethekerneltoversion4.3.3+.

PXFORC-RefertoORCKnownIssuesandLimitations foralistofknownissuesrelatedtotheORCBeta.




AfterupgradingfromHDBversion2.0.0,HCatalogaccessthroughPXFmayfailwiththefollowingerror:

postgres=# \d hcatalog.default.hive_tableERROR: function return row and query-specified return row do not matchDETAIL: Returned row contains 5 attributes, but query expects 4.

TorestoreHCatalogaccess,youmustupdatethePXF pxf_get_item_fields() functiondefinition.PerformthisprocedureonlyifyouupgradedfromHDB2.0.0.

1. LogintheHAWQmasternodeandstartthe psql subsystem:

$ ssh gpadmin@mastergpadmin@master$ psql -d postgres

2. Listallbutthe hcatalog and template0 databases:

postgres=# SELECT datname FROM pg_database WHERE NOT datname IN ('hcatalog', 'template0');

3. RunthefollowingcommandsoneachdatabaseidentifiedinStep2toupdatethe pxf_get_item_fields() functiondefinition:

postgres=# CONNECT ;postgres=# SET allow_system_table_mods = 'dml';postgres=# UPDATE pg_proc SET proallargtypes = '{25,25,25,25,25,25,25}', proargmodes = '{i,i,o,o,o,o,o}', proargnames = '{profile,pattern,path,itemname,fieldname,fieldtype,sourcefieldtype}' WHERE proname = 'pxf_get_item_fields';

4. Resetyour psql session:

postgres=# RESET allow_system_table_mods;

Note:Usethe allow_system_table_mods serverconfigurationparameterandidentifiedSQLcommandsonlyinthecontextofthisworkaround.Theyarenototherwisesupported.

PL/RTheHAWQPL/RextensionisprovidedasaseparateRPMinthe hdb-add-ons-2.1.1.0 repository.ThefilesinstalledbythisRPMareownedby root .IfyouinstalledHAWQviaAmbari,HAWQfilesareownedby gpadmin .PerformthefollowingstepsoneachnodeinyourHAWQclusterafterPL/RRPMinstallationtoaligntheownershipofPL/Rfiles:


Ambari


https://hdb.docs.pivotal.io/211/hdb/releasenotes/orc-support-beta.html#hiveorc-known-issueshttps://hdb.docs.pivotal.io/211/hdb/install/install-ambari.html#post-install-pxfhttps://issues.apache.org/jira/browse/HAWQ-974https://hdb.docs.pivotal.io/211/hawq/pxf/TroubleshootingPXF.html#pxf-memcfg

Ambari-managedclustersshouldonlyuseAmbariforsettingsystemparameters.Parametersmodifiedusingthe hawq config commandwillbeoverwrittenonAmbaristartuporreconfiguration.

Incertainconfigurations,theHAWQMastermayfailtostartinAmbariversionspriorto2.4.2when webhdfs isdisabled.RefertoAMBARI-18837 .Toworkaroundthisissue,enable webhdfs bysetting dfs.webhdfs.enabled to True in hdfs-site.xml ,orcontactSupport.



















https://issues.apache.org/jira/browse/AMBARI-18837https://hdb.docs.pivotal.io/211/hawq/admin/ambari-admin.html#amb-remove-standbyhttps://hdb.docs.pivotal.io/211/hawq/admin/ambari-admin.html#amb-activate-standbyhttps://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations








8. SELECT fsname, fsedbid, fselocation FROM pg_filespace as sp,pg_filespace_entry as entry, pg_filesystem as fs WHERE sp.fsfsys = fs.oidand fs.fsysname = 'hdfs' and sp.oid = entry.fsefsoid ORDER BYentry.fsedbid;

fsname | fsedbid | fselocation-------------+---------+------------------------------------------------cdbfast_fs_a | 0 | hdfs://hdfs-cluster/hawq//cdbfast_fs_adfs_system

Table of Contents - Pivotal Software · What is HAWQ? HAWQ Architecture Table Distribution and...

Documents

Transcript of Table of Contents - Pivotal Software · What is HAWQ? HAWQ Architecture Table Distribution and...