Oracle 10g Data Pump Say good bye to exp and imp (or not)!
-
Upload
benedict-davis -
Category
Documents
-
view
50 -
download
0
description
Transcript of Oracle 10g Data Pump Say good bye to exp and imp (or not)!
Oracle 10g Data Pump
Say good bye to exp and imp (or not)!2006/02/23
Simon Pane
2
Agenda
Data Pump Overview
Using Data Pump
Demonstration
Data Pump Test Cases
3
Data Pump Overview
4
What is Data Pump?
A replacement of the traditional export/import utilities?
The evolution of the traditional export/import utilities?
A completely new 10g utility serving a similar yet slightly different purpose?
5
Other Options for Moving Data
Traditional Export and ImportPros
Easy to use – most DBAs have years of experience using these utilities
Versatile – various options available; can specify what to include
Platform independent
Serial output
Cons Comparatively slow
Can be network intensive
Non-interruptible / resumable
Limited filtering options (for example, can exclude just VIEWS)
Limited remapping options (i.e. from one tablespace to another)
6
Other Options for Moving Data
Transportable TablespacesPros
Undoubtedly the fastest way to move data
Can use the traditional exp/imp or Data Pump to move meta-data
Cross-platform support if the platform byte-order is the same
Cons Tablespaces must be made read-only
Not selective (must move the entire tablespace)
Flashback is not possible (tablespace is read only when copied)
No physical reorganization is performed
Datafile sizes remain constant
Must use RMAN to convert the datafile if migrating to a platform with a different byte-order (check V$TRANSPORTABLE_PLATFORM)
7
Other Options Used Less Frequently
Extraction to a flat file and loading using SQL Loader
Direct copy using database links (SQL Plus COPY command)
Oracle Streams
3rd Party data ETL or reorg tools
8
Top 10 Reasons to Love DataPump
10. Similar look and feel to the old exp/imp
11. Can filter on the full range of object types
12. Can re-map datafiles and or tablespaces on import
13. Estimates the export file size (space needed)
14. Parallelizable
15. Significantly faster than the traditional exp/imp
16. PL/SQL Interface – programmable
17. A file is not actually required - can import through a network link
18. Track in v$session_longops
19. Resumable (interruptible and restartable)
9
Top 10 Reasons Not to Love Data Pump
10. Still generates redo (unlike direct path inserts)
11. Aggregation of exported data is not possible (sort only)
12. Performance on the server
13. Harder to tell what it’s doing at any given time
14. No equivalent to the STATISTICS option
15. Cannot be used with sequential media such as tapes and pipes (not read/written serially)
16. Only accesses files on the server, never the client
17. Oracle directories are required in the DB to access the files
18. Does not support COMMIT on imp or CONSISTENT on exp
19. If constraints are violated on import, the load is discontinued
10
Operation Fundamentals
Export/ImportThese utilities would basically connect to the Oracle database
via Oracle NET and run queries or DDL/DML
Processing of returned results and I/O operations were done on the client
Data PumpThe executables call PL/SQL APIs
Therefore processing is done on the database server
This can be an advantage or a disadvantage depending on the situation
“Self-Tuning”: no longer need to use BUFFER or RECORDSET
11
Export Operation
Network
Export File(s)
exp.exe
Oracle Database
12
Data Pump Export Operation
Network
Export File(s)
expdp.exe
Oracle Database
13
Key Differences
Dump and log files are on the server, not the client
Must have a DIRECTORY created in the Oracle database for I/OPermissions for the userid connecting to the instance, not the
schemas being exported or imported
Canceling the client process does not stop the job
Doesn’t automatically overwrite dump file if it already exists – returns an error instead
Parameters (command line) are reported in the log file
Exported objects order based on table size (descending) instead of alphabetically
14
Multiple Interfaces
1. Command line utilities expdb and impdb
1. Similar to the familiar exp and imp in usage
2. Use HELP=Y for a list of commands
3. Oracle documentation provides a comparison table to exp/imp
2. Enterprise Manager
3. PL/SQL Can be used independently but is difficult
All of these call the DBMS_DATAPUMP API Uses Oracle Advanced Queuing
Uses DBMS_METADATA
15
Unload Mechanisms
Data Pump automatically chooses to unload data either using:Direct path
External Tables (new driver called ORACLE_DATAPUMP)
Same “External Tables” mechanism that was introduced in Oracle9i
When will it use External tables:When parallelism can be used
When the table contains a complex data type or structure that prevents direct path unloads A lot of tables fall under this situation – see Oracle documentation for a
complete list
It doesn’t really matter to us which method is used
16
Multiple Processes
Master Control ProcessSpawns worker processes
Populates the master control table and log file
The master control table can be queried to track the job’s process
At the end of an export, the master control table is written to the dump file and dropped from the database
Worker ProcessesPerforms the loading/unloading
Number of processes depends on the degree of parallelism (the PARALLEL option)
17
Detaching and Re-Attaching
Issuing “Ctrl-C” from the data pump import will detachImport is running on the server so it will continue
Brings you into “interactive-command” mode
To re-attach, run impdp with the ATTACH= optionExample: impdp userid=system/oracle attach=JOB_01
Brings you back into “interactive-command” mode
18
New Views
DBA_DATAPUMP_JOBS and USER_DATABASE_JOBSIdentify all jobs regardless of their state
Identify any master tables not associated with an active job
DBA_DATAPUMP_SESSIONSIdentify user sessions that are attached to a job
Data pump sessions populate v$session_longopsDocumentation says that it is 100% accurate for imports but
testing proves otherwise!!!
19
Security Considerations
Still uses the EXP_FULL_DATABASE and IMP_FULL_DATABASE
A privileged user will have these two roles
A privileged user can:Export/import objects owned by other schemas
Export non-schema objects (metadata)
Attach to, monitor, and control jobs initiated by others
Perform schema, datafile, and tablespace remapping
Similar to the traditional export/import
Supports label securityIf exporting user has the EXEMPT ACCESS POLICY role
20
Object Statistics
From Oracle documentation regarding data pump exports:“A parameter comparable to STATISTICS is not needed.
Statistics are always saved for tables.”
From Oracle documentation regarding data pump imports:“A parameter comparable to STATISTICS is not needed. If the
source table has statistics, they are imported.”
21
Other Random Points
Can still use a parameter file and the PARFILE command line option
Fully supports Automatic Storage Management (ASM)
Can still flashback to a specified time or SCN
Can still extract (or backup) DDL (meta data)Using the SQLFILE option instead of the traditional INDEXFILE or SHOW options
Full support of LOBS
22
Using Data Pump
23
Oracle Directory Objects
Must first create an Oracle directory object and give the user who will be performing the Data Pump activities permission to use it (or rely on defaults):
SQL> create or replace directory dpump_demo as 'C:\temp';
Directory created.
SQL> grant read,write on directory dpump_demo to simon;
Grant succeeded.
SQL>
24
Key Data Pump Export Parameters
CONTENT={ALL | DATA_ONLY | METADATA_ONLY}
DIRECTORY=directory_object (default=DATA_PUMP_DIR)
DUMPFILE=[directory_object:]file_name [,…]
ESTIMATE={BLOCKS | STATISTICS}
ESTIMATE_ONLY={Y | N}
EXCLUDE=object_type[:name_clause] [,…]
FILESIZE=integer[B | K | M |G] (default=unlimited)
FLASHBACK_SCN=scn_value
FLASHBACK_TIME=“TO_TIMESTAMP(time_value)”
FULL={Y | N}
INCLUDE=object_type[:name_clause] [,…]
JOBNAME=jobname_string
25
Key Data Pump Export Parameters
LOGFILE=[directory_object:]file_name
NOLOGFILE={Y | N}
PARALLEL=integer (default=1)
QUERY=[schema.][table_name:]query_clause
SCHEMAS=schema_name [,…]
TABLES=[schema_name.]table_name[:partition_name] [,…]
TABLESPACES=tablespace_name [,…]
26
Data Pump Export Parameter Samples
Multiple dump files using a substitution variable (%U):DUMPFILE=DP_DIR1:SCOTT_20060223_%U.dmp
Excluding indexes that start with “EMP”:EXCLUDE=INDEX:“LIKE ‘EMP%’”
Excluding the SCOTT schema from a FULL export:EXCLUDE=SCHEMA:“=‘SCOTT’”
Mimicking the traditional CONSITENT parameter:FLASHBACK_TIME=“TO_TIMESTAMP”
Exporting only TABLES, FUNCTIONS and VIEWS:INCLUDE=TABLE,FUNCTION,VIEW
Using a query clauseQUERY=emp:‘“WHERE salary > 100000”’
27
Key Data Pump Import Parameters
CONTENT={ALL | DATA_ONLY | METADATA_ONLY}
DIRECTORY=directory_object (default=DATA_PUMP_DIR)
DUMPFILE=[directory_object:]file_name [,…]
EXCLUDE=object_type[:name_clause] [,…]
FULL={Y | N}
INCLUDE=object_type[:name_clause] [,…]
JOBNAME=jobname_string
LOGFILE=[directory_object:]file_name
NOLOGFILE={Y | N}
PARALLEL=integer (default=1)
QUERY=[schema.][table_name:]query_clause
28
Key Data Pump Import Parameters
REMAP_DATAFILE=source_datafile:target_datafile
REMAP_SCHEMA=source_schema:target_schema
REMAP_TABLESPACE=source_tablespace:target_tablespace
REUSE_DATAFILES={Y | N}
SCHEMAS=schema_name [,…]
SKIP_UNUSABLE_INDEXES={Y | N}
SQLFILE=[directory_object:]file_name
TABLE_EXISTS_ACTION={SKIP|APPEND|TRUNCATE|REPLACE}
TABLES=[schema_name.]table_name[:partition_name] [,…]
TABLESPACES=tablespace_name [,…]
29
Interactive Mode Commands
ADD_FILE=[directory_object:]file_name [,…]
CONTINUE_CLIENT
EXIT_CLIENT
FILESIZE=number
KILL_JOB
PARALLEL=integer
START_JOB
STATUS
STOP_JOB
30
Demonstration
31
Exporting and Importing Sample Schemas
expdp system/oracle@ORA1020 dumpfile=scott.dmp schemas=scott
impdp system/oracle@ORA1020 dumpfile=scott.dmp schemas=SCOTT remap_schema=SCOTT:LARRY
expdb system/oracle@ORA1020 dumpfile=larry.dmp schemas=larry
SELECT * FROM DBA_DATAPUMP_JOBS;
32
Using Interactive Mode
Ctrl-C to detach from the current export
Export> status
Export> stop_job
expdp system/oracle@ORA1020 attach=SYS_EXPORT_SCHEMA_01
Export> start_job
Export> exit_client
33
Data Pump Test Cases
34
Test Scenario #1
Generated sample SIEBEL dataBrushed off the dust on some old SIEBEL data population
scripts (circa 07/2000)
Designed for SIEBEL 6 on Oracle 8.1.6
Actual data is not important
Schema objectsTables: 218 (many empty tables)
Indexes: 1180 (SIEBEL is a heavily indexed application)
Schema size (from DBA_SEGMENTS)Tables: 1255MB
Indexes: 148MB
35
Export Performance Test Criteria
All work performed on laptop C681
SGA remains constant SGA_TARGET=0
SGA_MAX_SIZE=256MB
BUFFER CACHE=152MB
SHARED_POOL=60MB
Mostly default parameters
Not monitoring CPU utilization
Performed 4 runsDisregarded results from 1st run and averaged the other 3
36
Export Scripts
EXP sqlplus -s system/oracle @timestamp.sql > exp.log
exp.exe userid=system/oracle@ORA1020 file=SIEBEL.dmp log=SIEBEL.log owner='SIEBEL'
sqlplus -s system/oracle @timestamp.sql >> exp.log
EXPDP erase SIEBEL.dmp
sqlplus -s system/oracle @timestamp.sql > expdp.log
expdp.exe userid=system/oracle@ORA1020 dumpfile=SIEBEL.dmp logfile=SIEBEL.log schemas='SIEBEL' directory=test_dir
sqlplus -s system/oracle @timestamp.sql >> expdp.log
37
Export Performance Test Results
exp expdp
Average Export Time
6:02 3:18
Estimated File Size N/A 1.201 GB
Actual File Size 965 MB 621MB
38
Import Scripts
IMP sqlplus -s system/oracle @timestamp.sql > exp.log
imp.exe userid=system/oracle@ORA1020 file=SIEBEL.dmp log=SIEBEL.log fromuser='SIEBEL' touser='SCOTT' commit=y
sqlplus -s system/oracle @timestamp.sql >> exp.log
IMPDP sqlplus -s system/oracle @timestamp.sql > expdp.log
impdp.exe userid=system/oracle@ORA1020 dumpfile=SIEBEL.dmp logfile=SIEBEL.log schemas='SIEBEL' directory=test_dir remap_schema=SIEBEL:SCOTT remap_tablespace=TOOLS:SCOTT_DATA
sqlplus -s system/oracle @timestamp.sql >> expdp.log
39
Import Performance Test Results
imp impdb
Average Import Time 27:07
Average Import Time (no indexes)
25:19
Average Import Time (no rows) 27:27
Database was in ARCHIVELOG mode
Destination tablespace and archived log destination were both on ASM drives
Machine performance was degraded much more by impdb import
No import tuning performed (only COMMIT=Y)
40
Test Scenario #2
Data taken from an actual CGI internal application
Schema objectsTables: 22
Indexes: 26
Schema size (from DBA_SEGMENTS)Tables: 300MB
Indexes: 101MB
41
Export Scripts
EXP sqlplus -s system/oracle @timestamp.sql > exp.log
exp.exe userid=system/oracle@ORA1020 file=SCOTT.dmp log=SCOTT.log owner='SCOTT'
sqlplus -s system/oracle @timestamp.sql >> exp.log
EXPDP erase SIEBEL.dmp
sqlplus -s system/oracle @timestamp.sql > expdp.log
expdp.exe userid=system/oracle@ORA1020 dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT' directory=test_dir
sqlplus -s system/oracle @timestamp.sql >> expdp.log
42
Export Performance Test Results
exp expdp
Average Export Time
1:24 1:32
Estimated File Size N/A 290MB
Actual File Size 261MB 233MB
43
Import Scripts
IMP sqlplus -s system/oracle @timestamp.sql > exp.log
imp.exe userid=system/oracle@ORA1020 file=SCOTT.dmp log=SCOTT.log fromuser='SCOTT' touser='LARRY' commit=y
sqlplus -s system/oracle @timestamp.sql >> exp.log
IMPDP sqlplus -s system/oracle @timestamp.sql > expdp.log
impdp.exe userid=system/oracle@ORA1020 dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT' directory=test_dir remap_schema=SCOTT:LARRY remap_tablespace=SCOTT_DATA:LARRY_DATA
sqlplus -s system/oracle @timestamp.sql >> expdp.log
44
Import Performance Test Results
imp impdb
Average Import Time 5:48 2:26
Database was in NOARCHIVELOG mode
Destination tablespace and archived log destination were both on ASM drives
No import tuning performed (only COMMIT=Y)
45
Conclusions
46
Conclusions
Data Pump is an exciting new Oracle 10g tool that provides many benefits over the traditional export and import utilities
Whether to use Data Pump, Transportable Tablespaces, or even the traditional exp/imp will depend on the situation
Since the command line interface is easy to use and so similar to the traditional exp/imp, DBAs and developers should spend the time to learn how to use it
Final thought: since Data Pump dump files and traditional export dump files are not compatible/interchangeable, should a new file extension be used??? (.dmp vs .dpd)
47
The End
Comments, Questions ???