What’s New in Oracle Data Pump? OOW2017 Whats New Data Pump Author Roy Swonger Subject Oracle...
Transcript of What’s New in Oracle Data Pump? OOW2017 Whats New Data Pump Author Roy Swonger Subject Oracle...
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
What’sNewinOracleDataPump?OracleDatabase12cRelease2
RoyFSwongerVicePresidentOracleDatabaseUpgrade&Utilities
Updated:28-SEP-2017
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
What’sNewinDataPump?
ParallelExport/ImportofMetadataSubstitutionVariables&WildcardsREMAP_DIRECTORYLongIdentifiersupportTRUST_EXISTING_TABLE_PARTITIONSValidation&VerificationoptionsOther12.2FeaturesRecapof12.1Features
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ParallelMetadataExport:HowitUSEDTOWork
• StartwithESTIMATEphase– Gathertabledataobjects– Otherworkersremainidleuntildataobjectsaregathered
• Metadataexportedserially• Dataexportedinparallel
Pre-12.2
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ParallelMetadataExport:HowitWorksNow
• StartwithAnalysisstep– Metadataobjectspassedimmediatelytoworkersastheyarefound– E.g.Worker1findsasetofTABLEdefinitions,theyarehandedofftoworker2
• ESTIMATEphasestillhappens,butmetadatanolongerheldupbyestimate• Notes:
– Worksfordumpfilejobs,andfornetworkjobsifdestinationdatabaseis12.2– TransportableandFullTransportablejobsarenot(yet)parallelformetadata– ESTIMATEphasenowusesSTATISTICSonly– Restartworksasalways
NewFeaturein12.2
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ParallelMetadataExport:Logfile• 12.1.0.2 • 12.2.0.1
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
• Pre-12.2:– Oneworkerperpartition/subpartition– PQusedifpartitionsarelargeenough– Packagebodiesloadedinparallel
• Withpatchforbug22273229– Indexesbuiltinparallel– Constraintscreatedinparallel– Availableasbackportto12.1.0.2,11.2.0.4
• Startingwith12.2– Addedparallelimportofmostothermetadataobjects
– Someexceptions• Types(duetoinheritance)• Schemas• Proceduralactions
ParallelMetadataImport
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ParallelMetadataImport:Internals• MetadataisexportedinXMLdocuments
– EachXMLdocumentindumpfilecontainsnobjectsofagiventype
• XMLdocumentsareallocatedtoworkers1documentatatime
• Example:161userstoimport– Usersareexportedwithupto80usersperXMLdocument– WhathappenswithPARALLEL=8?
• Notes:– Worksforconventional(dumpfile)jobs– Not(yet)fortransportablejobsornetworkmode– Restartworkssameasalways– Statuscommandwillshowmultipleworkersonmetadata
XMLDoc1User1User2…User80
XMLDoc2User81User82…User160
XMLDoc1User161
Worker1
Worker2
Worker3
(Workers8-nwouldbeidle)
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ParallelMetadataImport:Logfile• ComparisonwithPARALLEL=8for27586objectgrantsandMETRICS=Y
– 12.1.0.2
– 12.2.0.1
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Performance:ParallelMetadataImport• ExamplesfromE-BusinessSuitetestdatabase
ObjectType
Count
11.2.0.4PARALLEL=32
12.1.0.2PARALLEL=8
12.1.0.2PARALLEL=32WithPatch
12.2.0.1PARALLEL=8
12.2.0.1PARALLEL=32
Comments
OBJECT_GRANT(owner)
27586 49 50 51 10 22 Hardconnectforeachgrant
SYNONYM 43254 105 109 111 25 44
TYPE 4364 108 114 119 111 110 Handledbysingleworker
PROCACT_SCHEMA 606 198 216 214 152 175 Handledbysingleworker
TABLE 33164 923 1160 1298 368 248
OBJECT_GRANT(table)
358649 541 543 578 142 157 Hardconnectforeachgrant
INDEX 53190 6721 5770 360 418 272
PACKAGE 53217 424 476 474 114 54
VIEW 34690 538 583 593 151 184
PACKAGE_BODY 52092 1363 1974 1186 1981 959 Alwaysparallelsince11.2
*importtimeinsecondsWhat'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
SubstitutionVariablesforDumpfileName• Substitutionvariablesfordumpfilename:
– Pre-12.2:%Ugeneratesafixed-width2-digitnumber• e.g.dumpfile=exp%U.dmp
– Newoptionfor12.2expdporimpdp:• %lor%L:Incrementingnumberfrom01upto2147483646
– Newoptionsin12.2expdponly:• %dor%D:DayofMonthinDDformat• %mor%M:NumberofMonthinMMformat• %yor%Y:YearinYYYYformat• %tor%T:FulldateinYYYYMMDDformat
$ expdp system/oracle directory=mydir \ filesize=50K dumpfile=exp%T_%L.dmp full=y ... ... ... . . exported "WMSYS"."WM$METADATA_MAP" 0 KB 0 rows Master table "SYSTEM"."SYS_EXPORT_FULL_01" successfully loaded/unloaded ****************************************************************************** Dump file set for SYSTEM.SYS_EXPORT_FULL_01 is: /home/oracle/exp20160917_01.dmp /home/oracle/exp20160917_02.dmp /home/oracle/exp20160917_03.dmp ... ... /home/oracle/exp20160917_67.dmp /home/oracle/exp20160917_68.dmp /home/oracle/exp20160917_69.dmp /home/oracle/exp20160917_70.dmp /home/oracle/exp20160917_71.dmp /home/oracle/exp20160917_72.dmp Job "SYSTEM"."SYS_EXPORT_FULL_01" successfully completed at Sat Sep 17 23:47:31 2016 elapsed 0 00:03:00
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
WildcardsforTRANSPORT_DATAFILES• Useawildcardin12.2insteadoflistingeveryfile• Pre-12.2:
• NewFeature:wildcards– *(asterisk)matchesmultiplecharacters– ?(questionmark)matchesasinglecharacter
TRANSPORT_DATAFILES=users01.dbf TRANSPORT_DATAFILES=users02.dbf … TRANSPORT_DATAFILES=data1.dbf TRANSPORT_DATAFILES=data2.dbf …
TRANSPORT_DATAFILES=users*.dbf TRANSPORT_DATAFILES=data?.dbf …
$ impdp system/oracle@pdb2 network_link=sourcedb \ version=12 full=y transportable=always metrics=y \ exclude=statistics \ directory=mydir \ logfile=pdb2.log \ transport_datafiles='/u02/oradata/CDB2/pdb2/user*.dbf’
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
REMAP_DIRECTORY• AppliestoDDLwheredirectoryspecsareused
– E.g.CREATETABLESPACE• Changedirectoryspecwithoutchangingfilenames• UsefulwhenmovingbetweenOSplatforms
– Example:importingdumpfilecreatedonOpenVMSintodatabaseonLinuxREMAP_DIRECTORY="'DB1$:[HRDATA.PAYROLL]':'/db1/hrdata/payroll/'"
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
LONGIdentifiers• Oracle12.1.0.2:1-30bytes
– CREATE TABLE abcdefghijklmnopqrstuvwxyzabcd;
• Oracle12.2.0.1:1-128bytes– IfCOMPATIBLE≥12.2.0– CREATE TABLE abcdefghijklmnopqrstuvwxyz_abcdefghijklmnopqrstuvwxyz_abcdefgh…
– Databasename:≤8byte– DiskGroups,PDBs,rollbacksegmentsandtablespacenames≤30bytes
• https://docs.oracle.com/database/122/SQLRF/Database-Object-Names-and-Qualifiers.htm#SQLRF51129
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
LongIdentifierSupportinDataPump• Long(128-byte)identifiersaresupportedin12.2by
– DataPumpexpdp/impdp– SQL*Loader– ORACLE_LOADERaccessdriver– ORACLE_DATAPUMPaccessdriver– ExternalTables
• Importingdatabasemustsupport128-byteidentifiers– BecarefulwhenexportingwithVERSION=12.1orearlier
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
TRUST_EXISTING_TABLE_PARTITIONS• Pre-12.2
– Importingintoexistingtablewasdoneserially– DataPumpcouldn’tbesurethatpartitioninginDBmatchedpartitioningindumpfile
• New12.2Parameter:DATA_OPTIONS=TRUST_EXISTING_TABLE_PARTITIONS – Bigperformanceboost– Ifpartitionsdon’tmatch…error:
ORA-31693: Table data object "SH"."SALES_BIG_PT":"SALES_2000" failed to load/unload and is being skipped due to error: ORA-29913: error in executing ODCIEXTTABLEFETCH callout ORA-14401: inserted partition key is outside specified partition
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
DataValidation&Verification
• DATA_OPTIONS=VALIDATE_TABLE_DATA – Importonly– Validatesdateandnumberformatsoftabledata– Defaultisnovalidation
• DATA_OPTIONS=VERIFY_STREAM_FORMAT – Exportonly– Defaultisnoverification
ExtraValidationforThingsThatShouldNeverHappen
Starting "SCOTT"."SYS_EXPORT_TABLE_01": scott/******** tables=t directory=dmpdir dumpfile=t.dmp reuse_dumpfiles=true Processing object type TABLE_EXPORT/TABLE/TABLE_DATA Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER Processing object type TABLE_EXPORT/TABLE/TABLE . . exported "SCOTT"."T" 5.570 KB 1 rows ORA-31694: master table "SCOTT"."SYS_EXPORT_TABLE_01" failed to load/unload ORA-02354: error in exporting/importing data ORA-26009: stream verification error: [1], [0], [0], [0]
ORA-02374: conversion error loading table "DPV"."TEST18" ORA-12899: value too large for column C1 (actual: 500, maximum: 498) ORA-02372: data for row: C8 : '
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Other12.2Features(1)• Usedirectpathloadinnetworkmode!
– SpecifyACCESS_METHOD=DIRECT_PATHwithNETWORK_LINK=<dblink> – AllowsnetworkimportofLONGandLONG RAW
• DataPumpavailableinInstantClient– ToolspackageforInstantClient– IncludesSQL*Loader,expdp,impdp,exp,imp
• Viewsthatdescribeavailabletransforms– DBMS_METADATA_TRANSFORMS – DBMS_METADATA_TRANSFORM_PARAMS – DBMS_METADATA_PARSE_ITEMS
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
SQL*PlusHistory• SQL> SET HISTORY ON • SQL> SET HISTORY 1000 • Example:
– SQL> hist • Listsallcommandsfromthehistory
– SQL> hist 7 run • Willrunthe7thcommandfromthelist
• https://docs.oracle.com/database/122/SQPUG/HISTORY.htm#SQPUG-GUID-CCF9B047-3122-4644-BA06-5FA4B5812E9F
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Other12.2Features(2)• Newinteractivecommands
– TRACEparametercanbesetforarunningjob• Noneedtostop/restartjobfortracingtotakeeffect
– STOP_WORKERcommand• Killanindividualworkeryoubelievetobehungorstuck
– BothwillbedocumentedinMOSnotes
• Enhancedlogfiles– WhenMETRICS=Y
• ShowworkerIDforeachitemprocessed• Showaccessmethodforeachtable
– Includecontentsofparfileinlogfile
18-SEP-16 15:24:30.950: ;;; ********************************************************** 18-SEP-16 15:24:30.951: ;;; Parfile values: 18-SEP-16 15:24:30.953: ;;; parfile: job_name=md_exp_16_12201 18-SEP-16 15:24:30.955: ;;; parfile: reuse_dumpfiles=Y 18-SEP-16 15:24:30.957: ;;; parfile: logtime=all 18-SEP-16 15:24:30.958: ;;; parfile: metrics=Y 18-SEP-16 15:24:30.960: ;;; parfile: parallel=16 18-SEP-16 15:24:30.962: ;;; parfile: full=Y 18-SEP-16 15:24:30.963: ;;; parfile: logfile=md_exp_16_12201.log 18-SEP-16 15:24:30.965: ;;; parfile: dumpfile=md16_12201_%U.dmp 18-SEP-16 15:24:30.966: ;;; parfile: directory=EBSIMP 18-SEP-16 15:24:30.968: ;;; *********************************************************
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
Recapof12.1Features• VIEWS_AS_TABLESparameter
– Letsyouexportthecontentsofaviewasatable
• TRANSFORMparameteroptions– TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y
• Willdisablearchiveloggingduringimportfortablesand/orindexes
– TRANSFORM=LOB_STORAGE:SECUREFILE – TRANSFORM=STORAGE:N – TRANSFORM=TABLE_COMPRESSION:<compression_clause>
• LOGTIME=[NONE|STATUS|LOGFILE|ALL]parameter– Willwritetimestampsonstatusand/orlogfilemessages
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
DataPumpBestPractices• Forfullexports:
– RoleEXP_FULL_DATABASEisrequired • Forexportconsistencyuse:
– FLASHBACK_SCN=<scn> – FLASHBACK_TIME=SYSTIMESTAMP alternative:
– CONSISTENT=Y [sinceOracle11.2–LegacyInterface]• ThiswillincreaseUNDOrequirementsforthedurationoftheexport
• Alwayssetparameters:– EXCLUDE=STATISTICS – METRICS=YES
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
DataPumpBestPractices
• UsePARALLEL=n – Typicallyn=2x<numberofCPUcores>
• Newfeaturein12.2:ParallelExport/ImportofMetadata– 11.2.0.4and12.1.0.2:applypatchforbug22273229toenableparallelimportofconstraintsandindexes
• EXCLUDE=STATISTICSonexport • Newfeaturein12c:TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y
– 12.1.0.2:Applypatchforbug20778442
PerformanceTips
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
DataPumpBestPracticesNetworkMode
• Directimportviadatabaselink– Parameter:NETWORK_LINK
• Runonlyimpdponthetargetsystem-noexpdpnecessary• Nodumpfilewritten,nodiskI/O,nofiletransferneeded
• Performance:Dependsonnetworkbandwidthandtarget'sCPUs• Restrictionsofdatabaselinksapply
– NewFeaturein12.2:SupportforDirectPathLoadoverdblink(includingLONGdata)• ACCESS_METHOD=DIRECT_PATH
$ expdp ...
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
DataPumpBestPracticesNetworkMode
• RealWorldCase:KaiserPermanente,Medicare(USA)– impdponNETWORK_LINKwith8vs16CPUcores• 10GBitconnectionleveragedupto8Gbit• 1TBtablecopiedin~15minð4TB/hour
– NetworkbandwidthandCPUbound
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
DataPumpNewsinOracle12c• TRANSFORMoptiontoenableAdvanced/HCCCompression
– Example:• TRANSFORM=TABLE_COMPRESSION:"compress for query high"
– But:Granularityonlyontheentireimport• Workarounds:
– Precreateobjects• Downside:Willslowdownimport!!!or:
– PrecreatethetablespacewithCOMPRESSoption• create tablespace ARCHIGH datafile 'archigh.ora' size 100G default compress for archive high;
– ThenrunDataPumpwithTRANSFORM=TABLE_COMPRESSION:N • Thiswilldropallembeddedcompressionattributesassociatedwiththetables• Nowtablespacecompressionoptionwillbeusedforallnewlycreatedtables
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ExportingViewsasTables
TABLE1
VIEW1 expdp system/mgr views_as_tables=scott.view1 ...
impdp system/mgr remap_table=view1:scott.table1 ...
TABLE1
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
ImprovedLogFiles:METRICS=YandLOGTIME=ALL
METRICS=Y(Added11.2)
BasicLogfile
LOGTIME=ALL(Added12.1)
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
EnhancedCompressionAlgorithm• COMPRESSION_ALGORITHM
– Definesthecompressionalgorithmwhencompressingdumpfiles• BASIC Thesamealgorithmusedinpreviousversions.Goodcompression,withoutseverelyimpactingonperformance• LOW: ForusewhenreducedCPUutilizationisapriorityovercompressionratio• MEDIUM: Recommendedoption.SimilarcharacteristicstoBASIC,butusesadifferentalgorithm• HIGH: Maximumavailablecompression,butmoreCPUintensive
– Performance:• Compressionratio• CPUusage
– RequiresAdvancedCompressionOptionlicense
$ expdp scott/tiger tables=emp directory=mydir dumpfile=emp.dmp logfile=expdp_emp.log compression=all compression_algorithm=medium
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
EnhancedCompressionAlgorithm• Customerevaluation
– BASICat3.5TB/hour
– MEDIUM at7.0TB/hour
2x
What'sNewinOracleDataPump?
Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|
DatabaseUpgradeBlog-Slides
What'sNewinOracleDataPump?
• https://MikeDietrichDE.com/