Post on 31-Oct-2019
SQL/MED and PostgreSQL
Peter Eisentraut
FOSDEM 2009
Peter Eisentraut SQL/MED and PostgreSQL
What is SQL/MED?
MED = Management of External DataMethods to access non-SQL data in SQLSQL/MED is ISO/IEC 9075-9
Peter Eisentraut SQL/MED and PostgreSQL
Applications and Use Cases
Connect to other DBMS (like DBI-Link)Connect to other PostgreSQL instances (like dblink)Read non-SQL data (CSV, XML)Manage data stored in file system
ImagesVideoEngineering data
Peter Eisentraut SQL/MED and PostgreSQL
Why do we care?
Powerful new functionalityUnifies existing ad-hoc solutions.Implementation has begun in PostgreSQL 8.4.Several people have plans for PostgreSQL 8.5.See status report later in this presentation.
Peter Eisentraut SQL/MED and PostgreSQL
Advantages
Schema integrationAccess controlStandard APICentralized control through DBMS
Peter Eisentraut SQL/MED and PostgreSQL
Two Parts
Wrapper interface Access other data sources, represent themas SQL tables
Datalinks Manage files stored in file system, represent filereferences as column values
Peter Eisentraut SQL/MED and PostgreSQL
Wrapper Interface Concepts
Define a foreign table . . .On a foreign server . . .Accessed through a foreign-data wrapper
Peter Eisentraut SQL/MED and PostgreSQL
Example: Foreign-Data Wrappers
Foreign-data wrapper (FDW): a library that can communicatewith external data sources
CREATE FOREIGN DATA WRAPPER foosqlLIBRARY ’foosql_fdw.so’LANGUAGE C;
PostgreSQL communicates with foosql_fdw.so usingSQL/MED FDW API.foosql_fdw.so communicates with FooSQL serverusing their own protocol.In theory, FooSQL, Inc. would ship foosql_fdw.so withtheir product.In practice, this is not so wide-spread.
Peter Eisentraut SQL/MED and PostgreSQL
Example: Foreign Servers
Foreign server: an instance of an external data sourceaccessed through a FDW
CREATE SERVER extradbFOREIGN DATA WRAPPER foosqlOPTIONS (host ’foo.example.com’, port ’2345’);
Options depend on FDW.
Peter Eisentraut SQL/MED and PostgreSQL
Example: User Mappings
User mapping: additional user-specific options for a foreignserver
CREATE USER MAPPING FOR peter SERVER extradbOPTIONS (user ’peter’, password ’seKret’);
Options depend on FDW.Putting connection options into server vs. user mapping isa matter of convention or convenience.
Peter Eisentraut SQL/MED and PostgreSQL
Example: Foreign Tables
Foreign table: a table stored on a foreign server
CREATE FOREIGN TABLE dataSERVER extradbOPTIONS (tablename ’DATA123’);
Now you can read and write the table as if it were local(depending on FDW features/implementation).Options specified for FDW, server, and user mapping areused as connection parameters (depending on FDW).
Peter Eisentraut SQL/MED and PostgreSQL
Another Wrapper Interface Example
Possible setup for accessing HTML tables stored in a web siteas SQL tables:
CREATE FOREIGN DATA WRAPPER htmlfileLIBRARY ’html_fdw.so’LANGUAGE C;
CREATE SERVER intranetwebFOREIGN DATA WRAPPER htmlfileOPTIONS (baseurl ’http://intranet/data’);
CREATE FOREIGN TABLE dataSERVER intranetwebOPTIONS (path ’foo.html#//table[@id="table1"]’);
Peter Eisentraut SQL/MED and PostgreSQL
More Wrapper Interface Features
GRANT and REVOKE
IMPORT FOREIGN SCHEMA
CREATE ROUTINE MAPPING
Peter Eisentraut SQL/MED and PostgreSQL
Status in PostgreSQL 8.4
PostgreSQL 8.4 has:CREATE FOREIGN DATA WRAPPER, but no librarysupportCREATE SERVER
CREATE USER MAPPING
ACL supportUsed by dblink and PL/Proxy to store connectioninformation
Peter Eisentraut SQL/MED and PostgreSQL
Status Worldwide
IBM DB2 provides a full implementationMySQL and Farrago use some syntax elementsNo other known implementations
Peter Eisentraut SQL/MED and PostgreSQL
Plan & Issues
Write wrapper library and foreign table support forPostgreSQL 8.5Supply a few foreign-data wrapper libraries withPostgreSQL 8.5Use standard wrapper interface API or design our ownAPI?
Peter Eisentraut SQL/MED and PostgreSQL
Datalink Concepts
Files are referenced through a new DATALINK typeDatabase system has control over external filesNo need to store file contents in database systemAccess control and integrity mechanisms of DBMS can beextended to file system
Peter Eisentraut SQL/MED and PostgreSQL
Example: Simple DATALINK Type
CREATE TABLE persons (id integer,name text,picture DATALINK [NO LINK CONTROL]
);
INSERT INTO persons VALUES (1,’Jon Doe’,DLVALUE(’file://some/where/1.jpg’)
);
This variant doesn’t do anything except store URLs.
Peter Eisentraut SQL/MED and PostgreSQL
DATALINK Attributes
Selection of additional possible attributes for DATALINK fields:FILE LINK CONTROL Datalink value must reference an
existing file.INTEGRITY ALL Referenced files can only be renamed or
deleted through SQL.INTEGRITY SELECTIVE Referenced files can be renamed or
deleted through SQL or directly.READ PERMISSION DB Database system controls file read
permission.RECOVERY YES PITR applies to referenced files.ON UNLINK DELETE File is deleted from file sytem when
deleted from database.
Peter Eisentraut SQL/MED and PostgreSQL
How to Implement Datalinks
How to implement this?
OS-dependentFile-system dependentApplication-dependentLots of hocus pocus neededPossibilities: kernel modules, LD_PRELOAD, extended FSattributesDon’t hold your breath.
Peter Eisentraut SQL/MED and PostgreSQL
Summary
SQL/MED:Wrapper interfaceDatalinksSubstantial support planned for PostgreSQL 8.5 andbeyond
References:http://wiki.postgresql.org/wiki/SqlMedConnectionManager (Martin Pihlak)http://www.sigmod.org/record/issues/0103/JM-Sta.pdf (Jim Melton et al.)http://www.sigmod.org/record/issues/0209/jimmelton.pdf (Jim Melton et al.)ISO/IEC 9075-9:2008 (“SQL/MED”)
Peter Eisentraut SQL/MED and PostgreSQL