SQL Loader

1

Basic Concepts

2

Sql *Loader

It is a high speed data loading utility supplied by Oracle that loads data from external files into table in an Oracle Database.

It can accepts data in variety of formats, can perform transformation, filtering and can load into multiple tables from multiple files in same loading session

3

Requirements- For Sql *Loader

Tables to be loaded must already exists in the Database. SQL *Loader never creates tables, it loads existing tables.

Table may be empty or may already contain data. Privileges:

INSERT privilege

DELETE privilege, when using REPLACE or TRUNCATE insert option

4

Files used in Sql *Loader

Control File - .ctl file Data File - .dat file Log File - .log file Bad File - .bad file Discard File - .dis file

5

Control File

It controls the Behavior of Sql *Loader

Source of Data to be loaded

Destination of Data to be loaded

Filtering of Data before Loading

Transformation of Data before Loading

Relating the Data file fields to table columns

6

Log File

It is a record of SQL *Loader’s activities during a load session

Control, Data, Bad, Discard file Name. Values of several command line parameters. Detailed breakdown of the fields & datatypes in data file

that was loaded. Error Message for records that cause error. Message indicating when records have been discarded. Summary of the Load (no: of records read from file, no: of

rows rejected because of errors, no: of rows discarded & elapsed time of load)

7

Data File – Contains the data to be loaded, it is optional

Discard File – Contains the data that are discarded by ‘when’ clause in control file, it is optional.

Bad File – Contains the data which are not loaded due to some errors, it is not optional, if any one error occurred, SQL *Loader will create the bad file and write the offending input records into it.

8

SQL *Loader Overview

SQL *Loader

Input Data File Control File

LogFile

BadFile

DiscardFile

DataBase

9

Loading Method

Basically there are 3 Loading Methods

Conventional Path Load

Direct Path Load

External Table Load( from Oracle 9i).

10

Conventional Path Load

Bind Array (Row Insert) is created by Sql *Loader based on field specification in control file

Data from the Bind Array are then insert into the table by the DB server, if data satisfies with the corresponding column datatype

11

Direct Path Load Parses the data according to the field specification

in Control File Converts the data to Column datatype (not to field

datatype in .ctl file) and builds Column array. Column array is passed to block formatter Newly formatted blocks are directly written

directly to the database file bypassing most SQL Processing

PARALLEL loading is possible.

12

PARALLEL Loading

Setup has to done, as the text file must be broken into several smaller files

We can run several SQLLDR session in parallel for each broken data files

It will increase the performance, reduce the time for loading

13

Advantage & Disadvantage of Direct Path Load

Advantage :

Very High Speed

Disadvantage :

Cannot Load Object Types, Collection Types (Nested Table, VARRAY),LOB(CLOB,NCLOB,BLOB, BFILE)

14

CHAR(5) Ctrl File Spec CHAR(5)

Field Conversion

aaa bbb

Column 1 Column 2

Table

a a a_ _ b b b

CHAR(5) Column Data Type VARCHAR(5)

Field 1 Field 2DATA FILE

BIND ARRAY

DATABASE

SQL LOADER

SERVER

a a a b b b

15

Record Filtering Record in Data file

SQL *LoaderField Processing

SQL *LoaderWhen-clauseEvaluation

DatabaseServer

Database

BadFile

DiscardFile

Read in

Accepted

SelectedDiscarded

InsertedRejected

RejectedControlfile

17

SQL> desc sampledata; Name Null Type

------------------------------- -------- ---- N1 VARCHAR2(25) N2 VARCHAR2(25)

LOAD DATA INFILE '/u01/xxvis/data1.dat'

replaceinto table sampledata

fields terminated by X'09'(

n2, n1

)

SQL> select * from sampledata;

N1 N2------------------------- ----

3456 12 0111 7891

3333 1234

Mapping between data fields in control file and columns of the table

12 34567891 01111234 3333

18

Data with in Control File

control1.ctlload datainfile *into table sql_ldr1 REPLACE(n position(1:2))begindata34567

Sql_ldr1N34567

$ sqlldr control=control1.ctl userid=apps/secretone

19

Data in DATA filedata2.dat111 222 666333 444 666555 666 666777 888 666999 000 666

control2.ctlload datainfile 'data2.dat'insert into table sql_ldr2

replace (n1 position(1:3),n2 position(5:7),n3 position(9:11))

Sql_ldr2:N1 N2 N3

111 222 666

333 444 666

555 666 666

777 888 666

999 000 666


20

Appending to a tabledata3.dat111 222 666333 444 666555 666 666777 888 666999 000 666

control3.ctlload datainfile 'data3.dat'insert into table sql_ldr3 append(n1 position(1:3),n2 position(5:7),n3 position(9:11))

Sql_ldr3 ( Before Loading):---------------------------------

N1 N2 N3

------ --------- ---------

111 222 666

333 444 666

555 666 666

777 888 666

999 0 666

Sql_ldr3 ( After Loading):---------------------------------

N1 N2 N3

------ --------- ---------

111 222 666

333 444 666

555 666 666

777 888 666

999 0 666

111 222 666

333 444 666

555 666 666

777 888 666

999 0 666

21

Loading the Selected datadata4.dat111 222333 444111 222333 444111 222

control4.ctlload datainfile 'data4.dat'insert into table sql_ldr4when n2 = '222'(n1 position(1:3),n2 position(5:7))

Sql_ldr4:

N1 N2

--------- ---------

111 222

111 222

111 222


$ sqlldr control=control4.ctl userid=apps/secretone dicard =

data4.dis

data4.dis

333 444333 444

22

Field Separatordata5.dat1111;2222;3333;4444;1111;2222;3333;4444;1111;2222;

control5.ctlload datainfile 'data5.dat'insert into table sql_ldr5 replacewhen n2='2222'fields terminated by ';'(n1 integer external,n2 integer external)

Sql_ldr5

N1 N2

-------- ---------

1111 2222

1111 2222

1111 2222


23

Loading Multiple Data Filesdata91.dat11 2233 4455 66

data92.dat77 88 99 0011 11

control9.ctl

load data

infile 'data91.dat'

infile 'data92.dat'

insert into

table sql_ldr9 replace

(

n1 position(1:2),

n2 position(4:5)

)


Sql_ldr9

N1 N2

11 22

33 44

55 66

77 88

99 0

11 11

24

Loading into Multiple Tablesdata10.dat11 2233 4455 66

control10.ctl

load data

infile 'data10.dat'

insert into


(n1 position(1:2),

n2 position(4:5))

into


(n1 position(1:2),

n2 position(4:5))


Sql_ldr101

N1 N211 2233 4455 66

Sql_ldr102

N1 N211 2233 4455 66

25

Loading from Multiple Data files to Multiple Tables

data111.dat11 2233 4455 66

data112.dat77 88 99 1199 99

control11.ctl

load data

infile 'data111.dat'

infile 'data112.dat'

insert into


(n1 position(1:2),

n2 position(4:5))

into


(n1 position(1:2),

n2 position(4:5))


Sql_ldr111

N1 N211 2233 4455 6677 8899 1199 99

Sql_ldr101

N1 N211 2233 4455 6677 8899 1199 99

26

Skipping recordsdata2.dat111 222 666333 444 666555 666 666777 888 666999 000 666

control2.ctlload datainfile 'data2.dat'insert into table sql_ldr2 replace (n1 position(1:3),n2 position(5:7),n3 position(9:11))

$ sqlldr control=control2.ctl userid=apps/secretone skip=2

Sql_ldr2

N1 N2 N3

555 666 666

777 888 666

999 000 666

27

Using LOAD optiondata14.dataabbbccdddeefff

control14.ctlload datainfile 'data14.dat'into table sql_ldr14

truncate/replace(a1 char(2),a2 char(3))

$ sqlldr control=control14.ctl userid=apps/secretone LOAD = 2

Sql_ldr14: A1 A2

aa bbbcc ddd

28

Fixed Record Formatdata12.dat12,3456,7891,0111,2

control12.dat load datainfile 'data12.dat' "fix 5"insert into table sql_ldr12 replace fields terminated by ','(n1 integer external,n2 integer external)


Sql_ldr12: N1 N2

12 3456 78 91 1

29

Variable Record Format

data1.dat

07aaa,bbb08ccc,dddd05ee,ff

control1.ctl

load data

infile ‘data1.dat’ “var 02”

into table sql_ldr1

fields terminated by ‘,’

(

a1 char,

a2 char

);

Desc sql_ldr1

a1 Varchar2(10)

a2 Varchar2(10)


Sql_ldr11: A1 A2

aaa bbbccc ddddee ff

30

Stream Record Format

data2.dat

aaa,bbb|ccc,dddd|eeee,ffff|

Control2.ctl

load data

infile ‘data2.dat’ “str ‘|’”

into table sql_ldr2

fields terminated by ‘,’

(

a1 char,

a2 char

)

Desc sql_ldr1

a1 Varchar2(10)

a2 Varchar2(10)


Sql_ldr12: A1 A2

aaa bbbccc ddddeeee ffff

31

Command Line Parameters BAD - Specifies the file in which all bad data is kept. The default

filename is the control filename with a .bad extension.

BINDSIZE - Size of the bind array in bytes. System dependent.

CONTROL - Specifies the name of control file.

DATA - Specifies the file that contains all data to be loaded.

DIRECT - If set to TRUE Direct Path Load is used. Otherwise Conventional Path will be used. Default value is FALSE.

DISCARD - Specifies the file where all discarded data is kept

32

DISCARDMAX – The maximum no: of invalid records, which is not satisfying the ‘when’ clause in control file, that may be encountered before SQL *Loader session is stop. Default value is ALL.

ERRORS – Specifies how many total errors can be encountered before SQL *Loader session is to stop. Default value is 50.

LOAD – Specifies the maximum no: of records to load before stopping. Default Value is ALL.

LOG – Specifies the log file’s name, where information on success or failure of the Loading session is reported. The Default filename is control_filename.log.

Command Line Parameters

33

PARALLEL – If set to TRUE, loads are performed in parallel where possible. Default value is FALSE

PARFILE – An additional file that contains more parameter specification.

ROWS – No: of rows to put in the path bind array, for Conventional path load. For Direct path load, ROWS specifies the no: of rows to read before a data save is performed. Default value is 64 in Conventional Path.

SKIP – No: of record to skip before starting the load. This parameter is important for restarting a load process after stopping an earlier session. Useful in Recovery from failure.


34

USERID – Specifies the username and password for the user conducting the SQL *Loader session

SILENT – Allows to suppress various header and feedback messages that SQL *Loader normally displays during a Load session.


Keyword for use with SILENT parameter• DISCARDS – Suppresses the discarded messages• ERRORS – Suppresses the error message• FEEDBACK – Suppresses the “commit point reached” messages.• HEADER – Suppresses the messages that SQL *Loader displays on the screen

when we first launch the executable.• PARTITION – Suppresses the per-partition statistics, when loading a direct

path load of a partitioned table.

SQL Loader

Documents

Transcript of SQL Loader