Post on 11-Feb-2018
7/22/2019 Getting the Best Out of Data Pump
1/49
1
7/22/2019 Getting the Best Out of Data Pump
2/49
Get the Best Out of Oracle Data Pump Functionality
Dean Gagne (Oracle)Viljo Hakala (Nokia)
7/22/2019 Getting the Best Out of Data Pump
3/49
3
Agenda
Moving large amounts of data with TransportableTablespaces
Filtering metadata using the INCLUDE and EXCLUDEparameters
Restarting stopped/failed jobs Hear about Nokia Corporation's database
environment How Nokia uses the REMAP_DATA parameter to
scramble data How Nokia regenerates primary keys without having
to use additional software or scripts
7/22/2019 Getting the Best Out of Data Pump
4/49
4
What is Oracle Data Pump?
New feature starting in Oracle Database 10 g Release 1 Enables very fast bulk data and metadata movement between
Oracle databases High-speed, parallel Export and Import utilities (expdp and
impdp) as well as a Web-based Oracle Enterprise Manager interface
Jobs can be restarted without loss of data, whether or not thestoppage was voluntary or involuntary
Jobs support fine-grained object selection. Virtually any type of object can be included or excluded
Supports the ability to load one instance directly from another (network import) and unload a remote instance (network export)
7/22/2019 Getting the Best Out of Data Pump
5/49
5
Moving Large Amounts of DataWith Transportable Tablespace
7/22/2019 Getting the Best Out of Data Pump
6/49
6
What is Transportable Tablespace?
An Oracle Database feature that allows data filetransfer from a database to another via a simple oscopy and a light specific export/import
Data Pump will move metadata only Data moves with data file copy Much faster than using direct path or external tables
7/22/2019 Getting the Best Out of Data Pump
7/49
7/22/2019 Getting the Best Out of Data Pump
8/498
Transportable TablespacesSelf Containment Check
Create table part_tab(id number) tablespace apartition by range (id)(partition low value less than (100) tablespace b,
partition hi values less than (1000) tablespace c); Create index part_ind on part_tab.id tablespace d; Transportable tablespace export requires all 4
tablespaces (a, b, c, d)
DBMS_TTS.TRANSPORT_SET_CHECK procedureto verify
accepts a comma separated list of tablespace names SELECT * FROM transport_set_violations;
7/22/2019 Getting the Best Out of Data Pump
9/49
7/22/2019 Getting the Best Out of Data Pump
10/49
7/22/2019 Getting the Best Out of Data Pump
11/49
7/22/2019 Getting the Best Out of Data Pump
12/49
7/22/2019 Getting the Best Out of Data Pump
13/49
7/22/2019 Getting the Best Out of Data Pump
14/49
7/22/2019 Getting the Best Out of Data Pump
15/49
7/22/2019 Getting the Best Out of Data Pump
16/49
7/22/2019 Getting the Best Out of Data Pump
17/49
17
Include Examples:
impdp system/manager tables=hr.employeesvs
impdp system/manager schemas=hr include=table:\"= \'EMPLOYEES\'\"
Same results Includes all objects that have table in the path
vsimpdp system/manager schemas=hr include=table/table:\"=
\'EMPLOYEES\'\ Only includes the table
Some of the TABLE object pathsSCHEMA_EXPORT/TABLE/TABLESCHEMA_EXPORT/TABLE/TABLE_DATASCHEMA_EXPORT/TABLE/GRANT
7/22/2019 Getting the Best Out of Data Pump
18/49
7/22/2019 Getting the Best Out of Data Pump
19/49
7/22/2019 Getting the Best Out of Data Pump
20/49
20
Restarting Stopped Export JobsInitial Export Command:
expdp system/manager job_name=datapump_exp
Restart command: expdp system/manager at tach=datapump_exp Export> start_job
Resets dumpfile pointers to last completed object type Restarts the job after the last completed object
7/22/2019 Getting the Best Out of Data Pump
21/49
7/22/2019 Getting the Best Out of Data Pump
22/49
7/22/2019 Getting the Best Out of Data Pump
23/49
7/22/2019 Getting the Best Out of Data Pump
24/49
2424
Nokia
Nokia is a world leader in mobility Head office in Finland; R&D, production, sales, marketing activities
around the world Worlds #1 manufacturer of mobile devices, with estimated 40%
share of global device market in 2009
Mobile device volumes 468 million units Net sales EUR 50.7 billion Operating profit EUR 5.0 billion 128 445 employees at year end (including Nokia Siemens
Networks) Strong R&D presence in 16 countries R&D investment EUR 6.0 billion Sales in more than 150 countries
7/22/2019 Getting the Best Out of Data Pump
25/49
2525
Oracle @ Nokia
20 DBAs, 2300 databases Grid Control: used for almost 5 years, #1 tool for all DBAs Applications:
OLTP: 50%, Hybrid databases: 40%, DW: 10% + Teradata About 1800+ Oracle; also MySQL 200+, MS SQL 300+
11g: 4%
10g: 80% 9i: 15% 8i: 1%
Host platforms: Solaris: 35% HP-UX: 35% Linux: 20% Windows 10%
7/22/2019 Getting the Best Out of Data Pump
26/49
7/22/2019 Getting the Best Out of Data Pump
27/49
7/22/2019 Getting the Best Out of Data Pump
28/49
7/22/2019 Getting the Best Out of Data Pump
29/49
7/22/2019 Getting the Best Out of Data Pump
30/49
7/22/2019 Getting the Best Out of Data Pump
31/49
3131
Data Masking with Oracle Data Pump
REMAP_DATA allows transformation of columns data while exporting(expdp) or importing (impdp) by using a remapping function in thedatabase.
REMAP_DATA with Data Pump is usually faster than a customUPDATE statement for masking data.
To mask multiple columns in the same process and command, theREMAP_DATA parameter can be used multiple times.
7/22/2019 Getting the Best Out of Data Pump
32/49
7/22/2019 Getting the Best Out of Data Pump
33/49
7/22/2019 Getting the Best Out of Data Pump
34/49
7/22/2019 Getting the Best Out of Data Pump
35/49
3535
Data Masking with Oracle Data Pump:Example
Create a table in the CUSTOMERS schemacalled phones
SQL>CREATE TABLE CUSTOMERS.PHONES(MODELNAME VARCHAR(20) NOT NULL,PHONENUMBER VARCHAR2(50)
);insert into CUSTOMERS.PHONES
values(N900,+3581234567); insert into CUSTOMERS.PHONES
values(N8,+3589817654);
insert into CUSTOMERS.PHONESvalues N7 +3584834819
D t M ki ith O l D t P
7/22/2019 Getting the Best Out of Data Pump
36/49
3636
Data Masking with Oracle Data Pump:Example
We then need to create a functionfor remapping
create or replace packagecustomers.maskpkg
asfunction masknumber(phonenumber
varchar2) return varchar2;end;/create or replace package body
customers.maskpkg asfunction masknumber (phonenumber
varchar2) return varchar2 isbegin return
substr(phonenumber,1,4)||round(dbms_random.value (100,999))||lpad(round(dbms_random.value(1,9999)),4,'0');
end;end;
The function masknumber willaccept a varchar2 type andreturns a random phone number in varchar2 type
7/22/2019 Getting the Best Out of Data Pump
37/49
7/22/2019 Getting the Best Out of Data Pump
38/49
D t M ki g ith O l D t P
7/22/2019 Getting the Best Out of Data Pump
39/49
3939
Data Masking with Oracle Data Pump:import/replace table
This example will import the masked dump fileand replace the existing table with masked column data
$ impdp customers/manager \tables=customers.phones \dumpfile=phones_masked.dmp \directory=dumpdir \table_exists_action=REPLACE
Data Masking with Oracle Data Pump:
7/22/2019 Getting the Best Out of Data Pump
40/49
4040
Data Masking with Oracle Data Pump:Example, after import
SQL> select * from customers.phones
MODELNAME PHONENUMBER---------- --------------------
N900 +35815499474N8 +3584800578N7 +3581247839
k h l
7/22/2019 Getting the Best Out of Data Pump
41/49
4141
Data Masking with Oracle Data Pump:Import
REMAP_DATA parameter can also be used in the import process
Use impdp if you have an existing dump file and you want to mask thedata when loading into the database
$ impdp customers/manager TABLE_EXISTS_ACTION=replacedumpfile=phones.dmp directory=dumpdir remap_data=customers.phones.phonenumber:customers.maskpkg.masknumber
Primary Key Regeneration with
7/22/2019 Getting the Best Out of Data Pump
42/49
4242
Primary Key Regeneration withREMAP_DATA
To change primary keys after development phase of an application, useREMAP_DATA parameter.
It is possible to use Data Pumps REMAP_DATA to change the primarykeys within a database by writing a remapping function which follows the
new primary key format.
Primary key conflicts can happen when loading data to an existing table ina database.
With REMAP_DATA, conflicts can be avoided by changing the columnvalues during import.
7/22/2019 Getting the Best Out of Data Pump
43/49
Other Methods to Secure Data in Oracle
7/22/2019 Getting the Best Out of Data Pump
44/49
44
Other Methods to Secure Data in OracleDatabase 11g Release 2 Oracle Virtual Private Database (VPD aka Fine Grained Access Control)
feature allows filtering of data at row-level for runtime SQL statementsaccording to a defined policy.
Data Masking Pack for Enterprise Manager. Centralized masking solutionwithin Grid Control.
Oracle Advanced Security Option for encrypting data at tablespace or column level.
Oracle Label Security for restricting access to data based on policies.
Oracle Database Vault for separating roles and restricting access to dataper roles.
7/22/2019 Getting the Best Out of Data Pump
45/49
7/22/2019 Getting the Best Out of Data Pump
46/49
7/22/2019 Getting the Best Out of Data Pump
47/49
7/22/2019 Getting the Best Out of Data Pump
48/49
48
7/22/2019 Getting the Best Out of Data Pump
49/49