Lucas Jellema

OGH – SIG SQL & PL/SQL – 10 mei 2016

The Fourth Dimension of the Database

- On Flashback and Temporal Validity in

the Oracle 12c database

2

Time

3

The Pension Fund

4

The Pension Fund

• Administration for 2 milion participants

• Record their complete employment history

– employer, job role, salary and benefits

• Used to calculate invoices for monthly premium as well as the benefits that have been accrued

• MN has to be able to

– Report on the situation per end of each quarter

– Account for any invoice that has been sent (and the way it was calculated)

– Provide an audit trail of all mutations in the employment history including the nature

and timestamp of the mutation and the identity of the perpetrator

– Retain history for specified periods of time

5

The Pension FundCurrent approach

• Journal tables

• Audit columns to record identity of person responsible for change

• DML Triggers on all relevant tables to record history

• Long reporting batches at the end of the quarter

• Complex queries to reconstruct a certain moment in time that caused a certain invoice

• Logic to exclude data that is no longer relevant/allowed

• [lack of] scheduled batches to remove data once beyond retention

Glimpses of the past

Session 1 Session 2

Glimpses of the past

Session 1 Session 2

Glimpses of the past – same session

Glimpses from the pastHow a session can see its own past

Session 1

Formalized Glimpses of the pastFlashback Query

Compare current with previous

Powered by UNDO

UNDO

FLASHBACK facilities

• Flashback Database

• Flashback Standby Database

• Flashback Table

• Flashback Table to Before Undrop(Recycle Bin)

• Flashback Query

• Flashback Versions Query

• Flashback Transaction

DBA

Developer

11g Flashback Data Archive

1 Month10 Years

UNDO5 Days

Configuring Total Recall for Table EMP

• Set up Flashback Data Archive

• Enable Total Recall for EMP

• Some requirements:

– Tablespace must be under Automatic Segment Space Management to create the

Flashback Data Archive in it

– Automatic UNDO management must be enabled

– The FLASHBACK ARCHIVE ADMINISTER privilege

CREATE FLASHBACK ARCHIVE LONGTERM_ARCHIVE

TABLESPACE HISTORY

RETENTION 15 YEAR

ALTER TABLE EMP

FLASHBACK ARCHIVE LONGTERM_ARCHIVE;

Sessions Travel back in time

• Using Flashback Query and Flashback Versions Query, the past can be inspected

– Queries have to adapted

– Every table referenced in a query needs to be explicitly 'flashbacked'

• Non intrusive time travelling for an entire session is available too –through dbms_flashback

– Only travelling to the past

– The past is read-only

– Travel to Timestamp or SCN

– Note: SYSDATE is still the current system time, not the historic timestamp; Views or

Queries that use SYSDATE may need tending to

• For example to set from after-logon trigger or as option in a client application to set the “time scope”

– For example to create reports for the end of the year – no need to stop new data

coming in!

Move entire session to point in time

Flashback Row History aka flashback versions query

• See all flashback history data for each row, including:

– SCN range the row was “valid for”

– Time range (approx) the row was valid for

– The transaction id (XID) that modified the row

– The operation (I/U/D) that was performed on the row

– special constants: minvalue and maxvalue

select ename, sal,

versions_operation,

versions_starttime,

versions_endtime,

versions_startscn,

versions_endscn,

versions_xid

from emp versions between timestamp &A and &B

order by versions_startscn nulls first

from emp versions between timestamp minvalue and maxvalue

Trac-king's Salary History

Provide insight in trends and recent changes using flashback versions

10 Years

MOST_RECENT_SALARY_CHANGES

“Flashback Transaction”

• Select the UNDO_SQL for a certain transaction

select xid

, start_scn

, commit_scn

, operation

, undo_sql

, table_name

from

flashback_transaction_query

where xid = '0004000400000003D';

Transaction Back Out

• To undo the effects of one or more transactions, we can invoke DBMS_FLASHBACK.TRANSACTION_BACKOUT

– with a list of Transaction IDs

– and options for dealing with dependent transactions (later transactions partially on

the same records)• NonConflict_Only: only backout those parts of selected transaction that do not conflict with later transactions

• Nocascade: raise error if there is dependent transaction

• NoCascade_Force: only backout selected transaction

• Cascade: back out of dependent transaction too

– Backout does not commit: it performs compensating DML, constraints are enforced,

holds locks and reports

– Back out reports are available in views

*_FLASHBACK_TXN_STATE and *_FLASHBACK_TXN_REPORT

…_JOURNAL…

Date_modified

User_modified

Date_modified

User_modified

+ columns for

… columns

…_JOURNAL…

Date_modified

User_modified

Date_modified

User_modified

+ columns for

… columns

History is frequently kept in Audit columns and journaling tables

EMP_JOURNALEMP

Date_modified

User_modified

Date_modified

User_modified

+ columns for

… columns

…_JOURNAL…

Date_modified

User_modified

Date_modified

User_modified

+ columns for

… columns

Flashback data archive could mean The end of journaling

• Get rid of all journaling tables & triggers

• Productivity Gain

• Ease of use of archived data

– AS OF queries

– DBMS_FLASHBACK – transparent

• Performance Benefits

– Maintain flashback data archive is much more efficient than maintaining journaling

tables with triggers

10 YearsEMP… columns

Date_modified

… columns

Overhead of populating Flashback Data archive compared to triggers

Total Recall vs Triggers

6.0%8.9%

2.5%4.5%

54.0%53.2%56.9%

52.8%

0

10

20

30

40

50

60

70

100k/1k 100k/10k 1M/1k 1M/10k

# of Rows / Commit Interval

% Inc

reas

e in

Re

sp

on

se T

ime

Total Recall Triggers

Whodunnit?

• Flashback Data Archive does not record WHO made the change – only the changes that were made!

• Database Auditing does record the username and the client identifier for each completed transaction

– AUDIT INSERT,UPDATE,DELETE ON EMP

– Join Flashback Versions Query with user_audit_trail on audit.transactionid =

version.xid

10 YearsEMP… columns xid

Date_modified

… columns

USER_AUDIT

_TRAILClient Identifier

28

Whodunnit: Alternative Approaches

• USER_AUDIT_TRAIL is not meant to be part of business data – nor for long term storage and historical purposes

– If we need historical record of who performed transactions, we need to do something

in addition to or instead of USER_AUDIT_TRAIL

• One approach: maintain a custom transaction registry that records each(relevant) transaction with relevant values

– Record transaction identifier, SCN and relevant generic transaction context values

such as client identifier, db_user and os_user

– Options 1: Use FGA policy on each table with FLASHBACK ambition to record

transaction (requires Database Enterprise Edition to use dbms_fga)

– Option 2: Generate custom DML statement level trigger on each table with

FLASHBACK ambition to record transaction

– Join Flashback Versions Query with custom transaction registry on

audit.transactionid = version.xid

Flashback Weak Spots as of Oracle Database 11g R2

• Whodunnit?

• Also:

– when is the start of history?

– What went on before? What to do with existing archives?

– How to deal with Flashback Data during export/import?

– Some DDL is hard to absorb

• Oh, and by the way: Flashback Data Archive requires Enterprise Edition & Advanced Compression database option

30

License change in 12c

• Flashback Data Archive in all database editions

31

• Flashback Data Archive also available in 11.2.0.4 – all editions

Total Recall 12c

• Flashback Data Archive Improvements:

– Complete schema evolution support: all table definition, partitioning,

and space management DDLs are supported on FDA-enabled

tables.

• For example:

– add column – results from before the column was added will show

NULL for that column

– drop column – historic results will show the dropped column with a

system generated prefix (D_<number>_ColumnName)

– Note: drop table can only be done by user with extended privileges

Total Recall 12c

• Flashback Data Archive Improvements:

– The metadata information for tracking transactions including the

user context is now tracked. The addition of user-context tracking

makes it easier to determine which user made which changes to a

table.

• This could mean that journaling tables can now officially be deprecated

• Especially given the fact that the current contents of journaling tables can even be

migrated to Flashback Data Archive

34

Ensure transaction context is recorded (and set)

exec dbms_flashback_archive.set_context_level(level=> 'ALL');

exec dbms_session.set_identifier('The Creepy User from Finance ');

update oow.emp

set sal = sal * 1.4

where ename = 'ALLEN'

/

commit;

exec dbms_session.set_identifier('Scary Janitor from the Annex');

update oow.emp

set sal = sal * 0.7

where ename = 'MILLER'

/

commit;

35

Audit the generated history

SELECT versions_xid

, versions_starttime

, empno, ename, sal new_sal

, s.client_identifier

FROM oow.emp VERSIONS BETWEEN TIMESTAMP minvalue

AND maxvalue

join

sys.sys_fba_context_aud s

on (versions_xid = s.xid)

36

Alternative: retrieve context withdbms_flashback_archive.get_sys_context

SELECT versions_xid

, versions_starttime

, empno, ename, sal new_sal

, dbms_flashback_archive.get_sys_context

(versions_xid,'USERENV','CLIENT_IDENTIFIER') who

FROM emp VERSIONS BETWEEN TIMESTAMP minvalue

AND maxvalue

Total Recall (2)

• Import and export of Flashback Data Archive history using Data Pump

– Support for import and export using Data Pump for FDA-enabled

tables. Data Pump can now be used to export and import an FDA-

enabled base table along with its schema-evolution metadata and

historical row versions.

Total Recall (3)

• User generated history

– Support for importing user-generated history has been added.

Customers who have been maintaining history using other

mechanisms, such as triggers, can now import that history into Total

Recall.

• The solution for that question of 'what went on beforehistory started' with Flashback Database Archive prior to12c

39

Generate History –All actions by SYS

create table oow.emp as select * from scott.emp

grant execute on dbms_flashback_archive to oow;

grant execute on dbms_flashback to oow;

CREATE FLASHBACK ARCHIVE DEFAULT one_year TABLESPACE users

QUOTA 100M RETENTION 1 YEAR;

grant flashback archive on one_year to oow;

exec dbms_flashback_archive.create_temp_history_table('OOW', 'EMP');

-- This statement needs execution once in a database instance. This

will extend mappings to the past so that import of old history can

be done. Goes back to 01-JAN-88.

EXEC DBMS_FLASHBACK_ARCHIVE.extend_mappings();

---

-- after some history has been created:

EXEC DBMS_FLASHBACK_ARCHIVE.IMPORT_HISTORY('oow','EMP');

40

Generate History –Actions by Application

• Insert records describing each stage of history that has existed

– Including start and end time of historic state

– then import history [from temporary table into Flashback Data Archive]

insert into temp_history

(RID , STARTSCN , ENDSCN , XID, OPERATION

,EMPNO, ename, job, hiredate, sal, deptno )

values (NULL, timestamp_to_scn(to_date('01-04-2001', 'DD-MM-YYYY')),

timestamp_to_scn(to_date('01-07-2003', 'DD-MM-YYYY')), NULL, 'I'

,1567,'SELLERS','CLERK',to_date('01-04-2001','DD-MM-YYYY'),2200, 10);

insert into temp_history

(RID , STARTSCN , ENDSCN , XID, OPERATION

,EMPNO, ename, job, hiredate, sal, deptno)

values (NULL, timestamp_to_scn(to_date('01-07-2003', 'DD-MM-YYYY')),

timestamp_to_scn(to_date('01-10-2006', 'DD-MM-YYYY')), NULL, 'U'

,1567,'SELLERS','CLERK',to_date('01-04-2001','DD-MM-YYYY'),2200, 20);

…

41

Query the generated history

select ename

, job

from emp as of timestamp (sysdate - INTERVAL '10' YEAR)

minus

select ename

, job

from emp

select ename

, job

from emp as of timestamp (systimestamp - INTERVAL '3' YEAR)

minus

select ename

, job

from emp

DBMS_COMPARISON

• Supplied package DBMS_COMPARISON

• To compare tables or views

– Within the same schema or database or across databases (over a database link)

– Reports the number of differences – if any

– Reports the individual ROWIDs for all

rows that have differences

• To merge tables orviews in order tosynchronize one table with the other (master)

DBMS_COMPARISON

• First define a comparison:

• Execute the comparison:

DBMS_COMPARISON.CREATE_COMPARISON

( comparison_name => 'comp_emp_n_clone'

, schema_name => 'scott'

, object_name => 'emp'

, dblink_name => null –- same database

, column_list => null –- all columns participate

, remote_schema_name=>'scott'

, remote_object_name=>'EMP_COPY'

);

consistent := DBMS_COMPARISON.COMPARE

( comparison_name => 'comp_emp_n_clone'

, scan_info => scan_info

, perform_row_dif => TRUE );

dbms_output.put_line(' scan id = '||scan_info.id);

DBMS_COMPARISON

• Report differences:

SELECT r.INDEX_VALUE -- PK value

, case when r.LOCAL_ROWID is null

then 'REMOTE'

else case when r.REMOTE_ROWID is null

then 'LOCAL'

else 'BOTH'

end

end LOCAL_OR_REMOTE

FROM USER_COMPARISON_COLUMNS c

, USER_COMPARISON_ROW_DIF r

, USER_COMPARISON_SCAN s

WHERE c.COMPARISON_NAME = 'COMP_EMP_N_CLONE'

AND r.SCAN_ID = s.SCAN_ID

AND s.scan_id = &scan_id

AND r.STATUS = 'DIF'

AND c.INDEX_COLUMN = 'Y'

AND c.COMPARISON_NAME = r.COMPARISON_NAME

DBMS_COMPARISON

• Synchronize Local with Remote

– (Replicate Remote to Local):

DECLARE

scan_info DBMS_COMPARISON.COMPARISON_TYPE;

BEGIN

DBMS_COMPARISON.CONVERGE

( comparison_name => 'compare_emp_and_clone'

, scan_id => &scan_id

, scan_info => scan_info

, converge_options =>

DBMS_COMPARISON.CMP_CONVERGE_REMOTE_WINS

);

DBMS_OUTPUT.PUT_LINE('Remote Rows Merged: '

||scan_info.rmt_rows_merged);

DBMS_OUTPUT.PUT_LINE('Remote Rows Deleted: '

||scan_info.rmt_rows_deleted);

END;

Fine Grained Flashback Table

• Package DBaMiS_PARTIAL_FLASHBACK allows partial flashback revert [parts of] selected records to a previous state:

– only certain columns

– only specific rows

– Based on DBMS_COMPARISON

and Flashback Query

– Note: works flawlessly in 11g.

In 12c – dbms_comparison can

not run against a view that uses

AS OF in the from clause

Partial Table Flashback

T

a

b

l

e

A

r

c

h

i

v

e

Fine Grained Flashback Table

• Flashback DEPARTMENT 10

– and do not touch records from other departments

• Flashback Salary Data – but not any other columns

dbamis_partial_flashback.partial_flashback_table

( p_table_name => 'EMP'

, p_schema_name => 'SCOTT'

, p_to_timestamp => SYSTIMESTAMP – 30 –- go back one month

, p_column_list => 'EMPNO,SAL,COMM' -- only undo SAL & COMM

, p_undo_insert_flag => false — do not undo new inserts

);

dbamis_partial_flashback.partial_flashback_table

( p_table_name => 'EMP'

, p_schema_name => 'SCOTT'

, p_to_timestamp => SYSTIMESTAMP - 1/24*4 -- 4 hours back

, p_filter_condition => 'DEPTNO=10'

);

48

The Pension FundFuture approach

• Flashback Data Archives– Per retention period

• Associate relevant tables to FDA ofapplicable retention time

• Configure tracking of meta-data– To ensure transaction context is retained

• Report batches can be run at anymoment against any moment– Using dbms_flashback.enable_at_time

• Queries can easily be performed toreproduce a moment in the past– No impact of historic queries on SQL

• FDA management takes care ofdeprecated data

• Better transaction performance throughlower overhead

• Journal tables

• Audit columns

• DML Journaling Triggers

• Batches at the end of the quarter

• Complex queries to reconstruct

• Logic to exclude deprecated data

• batches to purge data

49

Commercial Interval

• Donderdag en vrijdag 2 en 3 juni – Vliegveld Valkenburg

• 120 sessies, 8 zalen, 1 keynote & 1 feestavond

• 80 sprekers uit 5 continenten

• Sprekers o.a.: Jonathan Lewis, Tim Hall, Graham Wood, Alex Nuijten, Patrick Barel, Heli

Helskyaho, Bryn Llewellyn, Pete Finnigan, Björn Rost, Alan Arentsen, Francisco Munoz

Alvarez, Toon Koppelaars, Frits Hoogland, Marco Gralike, Iloon Ellen-Wolff, Kellyn Potvin,…

50

Commercial Interval

• Registratie op http://www.amis-conference.com

• Kosten: 795 euro

– Maar bij mij kan je een kortingscode krijgen (voor de Early Bird fee)

51

Looking into the future…

OUR_PRODUCTS

NAME PRICEselect name, price

from our_products

52

Looking further into the future…

OUR_PRODUCTS

NAME PRICE

select name, price

from our_products

begin

DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (

level => 'ASOF'

, query_time => TO_TIMESTAMP('01-10-2018', 'DD-MM-YYYY')

);

end;

53

Current situation …

OUR_PRODUCTS

NAME PRICE

select name, price

from our_products

begin

DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (

level => 'CURRENT'

);

end;

54

All data in the table(the default setting)

OUR_PRODUCTS

NAME PRICE

select name, price

from our_products

begin

DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (

level => 'ALL'

);

end;

55

All data in the table(the default setting)

OUR_PRODUCTS

NAME PRICE

select name, price, start_date, end_date

from our_products

order

by start_date

START_DATE END_DATE

begin

DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME (

level => 'ALL'

);

end;

56

Part of SQL 2011 standard:Temporal Database

Make the database aware of the time based business validity of records

• Add timestamp columns indicating start and end of valid time for a record

• Specify a PERIOD for the table

• Note:

– A table can have multiple sets of columns, describing multiple types of temporal

business validity

create table our_products

( name varchar2(100)

, price number(7,2)

, start_date timestamp

, end_date timestamp

, PERIOD FOR offer_time (start_date, end_date)

);

Valid time aware flashback and flash foward queries

• Select all product prices on offer at a certain moment in time

• Perform all queries for records that are valid at a certain point in time –past or future

• Return all records currently (session time) valid

• Return all records (default)

SELECT *

FROM OUR_PRODUCTS AS OF PERIOD FOR offer_time

TO_TIMESTAMP('01-10-2014','DD-MM-YYYY')

EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time

( 'ASOF'

, TO_TIMESTAMP('01-05-2017','DD-MM-YYYY')

);

EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT');

EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');

59

Temporal Validity –Chairmanships with gaps and overlaps

• Chairperson of the board

– At any time there must be a chairperson – and at no time that can be more than one

• Table to hold chairperson-ships

60

Temporal Validity –Chairmanships with gaps and overlaps

• Select chairperson at a certain moment in time

• Perform all queries for records that are valid at a certain point in time –past or future

SELECT *

FROM chairmanships AS OF PERIOD FOR chairmanship

TO_TIMESTAMP('01-02-1980', 'DD-MM-YYYY')

EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time

( 'ASOF'

, TO_TIMESTAMP('01-10-2018','DD-MM-YYYY')

);

61

The future of temporal validity

Beyond 12.1.0.2 many Temporal Validity enhancements are expected:

• Gap and overlap checks ; support for PACKing consecutive records

• Valid time aware DML

• Temporal Primary and Unique Key constraints that allow multiple non-overlapping entries

• Temporal referential constraints that take into account the valid-time during which the rows exist.

– Child needs to have a valid Master at any time during its own validity

• Temporal Aggregation - group or order by valid-time

• Normalization - coalescing rows which are in adjacent or overlapping time periods

• Temporal joins – joins between tables with valid-time semantics based on 'simultaneous validity'

• For Information Lifecycle Management (ILM), the Valid Time information is used to assess records to move

62

Summary

• Time and History are supported in Oracle Database 12c along twodimensions:

– Transaction time – what been the committed state of the database at any point

– Business (valid) time – what was, is or will be the relevant data at any point

• Flashback is concerned with transaction time. As of 12c:

– Available in all editions (no license restrictions)

– Includes transaction (session) context at time of commit (whodunnit)

– Supports user defined history (as well as import and export)

– Flashback Query, Flashback Versions Query, Enable database session at timestamp

Flashback is ready for inclusion in database design and application development

• Valid Time Modeling (Temporal Database) is concerned with business validity of data – begin date and end date

– Based on SQL 2011

– Low impact use: alter table add (period for …)

– Easy querying across time – past, present and future

– Next steps are expected in enriching temporal database support (12.2?)