SQLAlchemy Primer

SQLAlchemy Primer(extra content)

for Kobe Python Meetup #13 2017/09/15 Kobe Japan

Yasushi Masuda PhD ( @whosaysni )

Tech team, Core IT grp. IT Dept. MonotaRO Co., LTD.

Pythonista since 2001 (2.0～) • elaphe (barcode library) • oikami.py （老神.py） • PyCon JP founder

Japanese Translation works

Agenda

Myths

Core concepts in SQLAlchemy

Engine basics (+hands-on)

ORM primer (+hans-on)

References

Online Document:http://docs.sqlalchemy.org/

en/rel_1_1/

(Old) Japanese translation:http://omake.accense.com/static/doc-ja/sqlalchemy/

Preparationsakila DB on SQLite http://bit.ly/2fdeeft

https://github.com/jOOQ/jOOQ/jOOQ-examples/Sakila/sqlite-sakila-db/sqlite-sakila.sq

Sakila • Demonstration

DB for MySQL • Models a rental

video shop • BSD License

Schema described at: https://dev.mysql.com/doc/sakila/en/sakila-structure-tables.html

Myths[WRONG!] It's just an ORM libraryNot limited to. SQLAlchemy is a DB manipulation framework.

[WRONG!] SA is built on ORMNO. Connection management and SQL expression framework are its core.ORM is built on them.[WRONG!] SA cannot handle raw SQLTable definition is not required. Even textual SQL is available.

[WRONG!] SA automatically escapes value for youSQLAlchemy relies value escaping on DB-API, while it escapes schema, table, column names. SQLAlchemy generates parameterized query that helps DB-API level escaping.

[WRONG!] Only Python adept can handle SASQLAlchemy is EASY. You may need SQL and RDBMS experience. Let's see!

[AGREE] Documentation is somewhat difficult to understandAgreed.

Core concepts

http://docs.sqlalchemy.org/en/latest/index.html

http://docs.sqlalchemy.org/en/latest/index.html

SQLAlchemy Core

Engine (connection) Schema definitionSQL Expression

SQLAlchemy ORM

Mapper Declarative Mapper

Session

Dialect DB Backend-specific functionalities

Engine manages DB connection(s)

SQL Expressiondescribes SQL statement in Python

Mappingreflects DB record with Python object

Dialect DB Backend specific functionalities

DatabaseYour program

SQL construction

Query execution

Typeconversion

Parameter binding

Driver Setup

Connection management

Result data structure

Value escaping

Schema name

Type conversion

Dialect-specific

Query Construction

Query Execution

Result

Schema object Object-relational mapping

High-level Interface

DB Session

Engine

DB API DB ServerProgram

Issues around DB-API

Which DB-API to use How to initialize the DB-API

How to generate valid query for it How to handle cursor on the DB-API How to execute query on the DB-API How to retrieve result from DB-API How to reserve/reuse connection

DB API DB ServerProgram

Engine resolves issues

Select DB-API from DSN URL Initialize DB-API for you

Accept string and SQL expression Manage cursor for you

Unify execution/transaction API Handle result via ResultProxy Pool connection automatically

Engine

Engine DB API DB ServerProgram

>>>fromsqlalchemyimportcreate_engine>>>


>>>fromsqlalchemyimportcreate_engine>>>e=create_engine('sqlite://')#SQLitememoryengine>>>


#UseURLforspecifyingdatabase>>>e=create_engine('sqlite://')#SQLitememoryengine>>>e=create_engine('sqlite:///path_to_db.sqlite3')#sqlite3>>>e=create_engine('mysql://scott:tiger@dbserv/dbname')#mysql>>>e=create_engine('mssql://bill:gates@dbserv/dbname')#mssql>>>


#Becarefulwithnumberofslashes>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>e=create_engine('sqlite:////<absolute_path>/sqlite-sakila.sq')


>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>eEngine(sqlite:///sakila-data.sq)

#executereturnsResultProxy>>>q='selecttitlefromfilmlimit5')>>>res=e.execute(q)>>>res<sqlalchemy.engine.result.ResultProxyobjectat0x10da96990>>>>


>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>eEngine(sqlite:///sakila-data.sq)

#executereturnsResultProxy>>>q='selecttitlefromfilmlimit5')>>>res=e.execute(q)>>>res<sqlalchemy.engine.result.ResultProxyobjectat0x10da96990>>>forrowinres:#ResultProxycanbeiterated/namedtupleaccess...print(row.title)...ACADEMYDINOSAURACEGOLDFINGERADAPTATIONHOLESAFFAIRPREJUDICEAFRICANEGG>>>


>>>q='selectfilm_id,titlefromfilmlimit10'>>>rows=list(e.execute(q))>>>rows[2][1]#eachrowisaccessiblelikeastuple'ADAPTATIONHOLES'>>>rows[4]['title']#rowcanbeaccessiblelikeasdictionary'AFRICANEGG'

>>>res=e.execute(q)>>>forfid,titleinres:#canbeexpandedasnormaltuple...print((fid,title))...(1,'ACADEMYDINOSAUR')(2,'ACEGOLDFINGER')(3,'ADAPTATIONHOLES')(4,'AFFAIRPREJUDICE')(5,'AFRICANEGG')>>>rows=list(e.execute(q))>>>

transaction

>>>t=e.begin()>>>t.transaction<sqlalchemy...RootTransactionobjectat...>>>>t.transaction.commit()

#withstatementhandlestransactionsmart>>>withe.begin():e.execute(...)>>>#(transactioncommittedautomatically)

HANDS ON: Engine basics

Connecttosqlite-sakila.sqdatabase

Listactorsinfilm"DINOSAURSECRETARY"

HANDS ON: Engine basics

>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>q='''selecta.first_name,a.last_name...fromfilmasf...innerjoinfilm_actorasfa...onf.film_id=fa.film_id...innerjoinactorasa...onfa.actor_id=a.actor_id...wheref.title="DINOSAURSECRETARY"'''>>>forfirst_name,last_nameine.execute(q):...print('{}{}'.format(first_name,last_name))...LUCILLETRACYBURTDUKAKISJAYNENEESONRUSSELLBACALLPENELOPEMONROEMINNIEKILMER

SQL expression

Remember: SQL is a language

SELECT[ALL|DISTINCT[ON(expression[,...])]]

[*|expression[[AS]output_name][,...]]

[FROMfrom_item[,...]]

[WHEREcondition]

[GROUPBYgrouping_element[,...]]

[ORDERBYexpression[ASC|DESC|USINGoperator][NULLS{FIRST|LAST}][,...]]

[LIMIT{count|ALL}]





[WHEREcondition]



[LIMIT{count|ALL}]

STATEMENT





[WHEREcondition]



[LIMIT{count|ALL}]

CLAUSE

CLAUSE

CLAUSE

CLAUSE

CLAUSE

CLAUSE





[WHEREcondition]



[LIMIT{count|ALL}]

parameter

parameter

expression

expression

expression

expression

expression

expression

Remember: SQL is a Language

SELECT

FROM selectablesselectable

join

WHERE

selectable

selectable

GROUP BY ORDER BY

expression

expressioncondition

expression

expressionsexpression

expression

SELECT statement

SELECTA.name,B.title,C.title

FROMartistsasAINNERJOINalbumasBONA.id=B.artist_idINNERJOINmusicasCONB.id=C.album

WHEREA.nameLIKE'%Astor%'ANDA.yearBETWEEN1970AND2010ANDC.titleLIKE'%Tango%'

From SQL to Python

SELECT<column>,<column>,...

FROM<selectable>INNERJOIN<selectable>ON<condition>INNERJOIN<selectable>ON<condition>...

WHERE<condition>AND<condition>AND<condition>

GROUPBY...LIMIT...

Clauses

SELECT<[<column>,<column>,<column>]>

FROM<join(<selectable>,<selectable>,...)>

WHERE<and(<condition>,<condition>,...)>

Statement and subjects

SELECT<expressions>

FROM<selectable>

WHERE<conditions>

... simplifed

<selectstatement>

..., finally

engine.execute(<selectstatement>)

If query is "an object"...

Query object

>>>query=select(...)

>>>engine.execute(query)

... it can be "execute()-able"

Engine "compiles" query into string and execute it

(according to dialect)

query=select(

<expression>,

from_obj=<selectables>,

whereclause=<conditions>,

)

clauses as parameters

columns=[col1,col2,...]

fromobj=join(tbl1,tbl2,...)

where=and_(expr1,expr2,...)

query_expr=select(columns,from_obj=fromobj,whereclause=where)

query with SQL expression

>>>fromsqlalchemy.sqlimportselect,text>>>q=select([text('*')])

building sql statement with basic sql expression

>>>fromsqlalchemy.sqlimportselect,text>>>q=select([text('*')])>>>q<sqlalchemy.....Selectat...;Selectobject>>>>str(q)'SELECT*'>>>q=select([text('*')],...from_obj=text('foo'),...whereclause=text('id=3'))>>>str(q)'SELECT*\nFROMfoo\nWHEREid=3'

building sql statement with basic sql expression

elements: table and columnSchema

Table Table Table

...

Column

Column

Column

...

Column

Column

Column

...

Column

Column

Column

...

SchemaTable Table Table

Column

Column

Column

Column

Column

Column

...

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)>>>c<sqlalchemy....ColumnClauseat...;name>>>>str(c)'name'

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)>>>c<sqlalchemy....ColumnClauseat...;name>>>>str(c)'name'>>>c.table#None>>>t1=table('artist')>>>c.table=t1>>>str(c)'artist.name'

>>>t=table('tbl1',column('col1'),...)

defining table with columnsTable name List of columns


>>>t.c.col1

<sqlalchemy.....ColumnClauseat...;col1>



>>>t.c.col1

<sqlalchemy.....ColumnClauseat...;col1>

>>>t.schema='db1'

>>>str(t.c.col1)

'db1.tbl1.col1'


>>>t=table('tbl1',column('col1'),column('col2'))

select() with table element

>>>t=table('tbl1',column('col1'),column('col2'))>>>print(select([t]))SELECTtbl1.col1,tbl1.col2FROMtbl1



#columnlabeling>>>print(select([t.c.col1.label('col_alias1')])SELECTtbl1.col1as"col_alias1"FROMtbl1



#columnlabeling>>>print(select([t.c.col1.label('col_alias1')])SELECTtbl1.col1as"col_alias1"FROMtbl1

#tablealias>>>t_A,t_B=alias(t,'A'),alias(t,'B')>>>print(select([t_A.c.col1,t_B.c.col2]))SELECT"A".col1,"B".col2FROMtbl1AS"A",tbl1AS"B"


howtoworkwith

wheretbl1.col1=42

conditionals

howtoworkwith

wheretbl1.col1=42

conditionals

conditional expression (compare operation)

>>>cond=text('last_nameLIKE%KOV')>>>str(cond)'last_nameLIKE%KOV'

conditional by text()

>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>

conditional by like() method

>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>>>>str(cond)'last_nameLIKE:last_name_1'


placeholder for right value of LIKE

>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>>>>str(cond)'last_nameLIKE:last_name_1

>>>cond.rightBindParameter('%(4339977744last_name)s','%KOV',type_=String())


>>>column('first_name')='DAVID'File"<stdin>",line1SyntaxError:can'tassigntofunctioncall

conditional by operation


>>>column('first_name')=='DAVID'<sqlalchemy...BinaryExpressionobjectat...>




>>>cond=column('first_name')=='DAVID'>>>str(cond)>>>'first_name=:first_name_1'>>>cond.right>>>BindParameter('%(...)s','DAVID',type_=...)




>>>cond=column('first_name')=='DAVID'>>>str(cond)>>>'first_name=:first_name_1'>>>cond.right>>>BindParameter('%(...)s','DAVID',type_=...)

>>>str(column('last_name')==None)>>>'last_nameisNULL'


>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))

conditionals in select

>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))


>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))>>>query<sqlalchemy.....Selectat...;Selectobject>>>>print(query)SELECTactor.first_name,actor.last_nameFROMactorWHEREactor.last_name=:last_name_1>>>query.compile().params{'last_name_1':'TRACY'}


>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))>>>query<sqlalchemy.....Selectat...;Selectobject>>>>print(query)SELECTactor.actor_id,actor.first_name,actor.last_nameFROMactorWHEREactor.last_name=:last_name_1>>>query.compile().params{'last_name_1':'TRACY'}>>>list(e.execute(query))[(20,'LUCILLE','TRACY'),(117,'RENEE','TRACY')]


select(...)

SELECT...

select(...).select_from(...)

SELECT...FROM...

select(...).where(...)

SELECT...WHERE...

select(...).where(...).where(...)

SELECT...WHERE...AND...

select(...).where(...).order_by(...)

SELECT...WHERE...ORDERBY...

generative method

actor_tbl.select()SELECT...FROMactor

actor_tbl.select(actor_tbl.c.first_name='BEN')SELECT...FROMactorWHEREfirst_name=...

generative method

HANDS ON: SQL expression

• define actor table with table()/column()

• build query with sql expression to searchactor having initial A.H. (result should include actor_id)

Schema definition

howtoCREATEtable

schema definition

howtoCREATEtable

howtodefinecolumndetailhowtodefineconstraints

schema definition

>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>

defining table

>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>user_tableTable('user',MetaData(bind=None),Column...)

defining table

>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>user_tableTable('user',MetaData(bind=None),Column...)

#CREATETABLEbycrete()>>>user_table.create(bind=e)

#DROPTABLEwithdrop()>>>user_table.drop(bind=e)

defining table

Table/Column vs table/columnVisitable (base type)

ClauseElement

Selectable

FromClause [ table() ]

ColumnElement

ColumnClause [ column() ]

Column Table

SchemaItem

Table/Column vs table/columnVisitable (base type)

ClauseElement

Selectable

FromClause [ table() ]

ColumnElement

ColumnClause [ column() ]

Column Table

SchemaItem

Supprorts same operation

HANDS ON: Schema definition

• define "user" table with Table()/Column()

• create table with .create()

• insert records

• drop table with .drop()

name type options

id INTEGER primary_key

username VARCHAR (64) nullable=False

email VARCHAR(128) nullable=False

ORM basics

Object-Relational Mapping

• Map DB records to objects

• ActiveRecord pattern

• 1 table ~> 1 class, 1 record ~> 1 object

• FK reference -> reference to other object

• column -> property/attribute of object

ORM in SQLAlchemy

• mapper maps a table with Python classmapped class will have instrumented attribute

• Session manipulates DB for you to retrieve / save mapped objects

mappingTable

Column foo

Column bar

Column baz

...

entity class

"mapped" entity class

instrumented fooinstrumented barinstrumented baz

...

method quxmethod quux

method quxmethod quux

...

mapper

classic mapping

[new] declarative mapping

mapping patterns

>>>fromsqlalchemy.ormimportmapper>>>actor_tbl=Table(...)>>>classActor(object):pass>>>mapper(Actor,actor_tbl)>>>dir(Actor)['__class__',...'_sa_class_manager','actor_id','first_name','last_name']>>>Actor.actor_id<...InstrumentedAttributeobjectat...>

classic mapping

>>>fromsqlalchemy.ext.declarativeimportdeclarative_base>>>fromsqlalchmyimportDateTime,Integer,String>>>Base=declarative_base()>>>>>>classActor(Base):...__tablename__='actor'...actor_id=Column(Integer,primary_key=True)...first_name=Column(String)...last_name=Column(String)...last_update=Column(DateTime)

declarative mapping

>>>fromsqlalchemy.ext.declarativeimportdeclarative_base>>>fromsqlalchmyimportDateTime,Integer,String>>>Base=declarative_base()>>>>>>classActor(Base):...__tablename__='actor'...actor_id=Column(Integer,primary_key=True)...first_name=Column(String)...last_name=Column(String)...last_update=Column(DateTime)>>>dir(Actor)['__class__','_decl_class_registry','_sa_class_manager','actor_id','first_name','last_name','last_update','metadata']

declarative mapping

ORM in SQLAlchemy

• mapper maps a table with Python classmapped class will have instrumented attribute

• Session manipulates DB for you to retrieve / save mapped objects

sessionEngineProgram

query Aselect A

record Aobject Bstart

tracking A

...

(updates) flag A as "dirty"

reflect new/dirty changes

flush

starttracking B

...

add object B

update A insert B

commit

begin

Using Session

>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>>>>

Using Session

>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>query=session.query(Actor)>>>query<sqlalchemy.orm.query.Queryobjectat...>

Using Session

>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>query=session.query(Actor)>>>query<sqlalchemy.orm.query.Queryobjectat...>>>>actor=query.first()>>>actor<...ActorObjectat...>

Query object methods: query filtering

>>>q=session.query(Actor)>>>str(q)>>>'SELECT...\nFROMactor'>>>q1=q.filter(Actor.first_name=='BEN')>>>str(q1)'SELECT...\nFROMactor\nWHEREactor.first_name=?'>>>q2=q1.filter_by(last_name='SMITH')>>>str(q2)'SELECT...\nFROMactor\nWHEREactor.first_name=?ANDactor.last_name=?'

Query object methods: fetching record(s)

>>>q=session.query(Actor)>>>q.first()<...Actorobjectat...>

>>>q.all()#WARNING:SELECTsallrecords<...Actorobjectat...>,<...Actorobjectat...>,...

>>>q[:3]<...Actorobjectat...>,<...Actorobjectat...>,...

>>>q.count()200

Update/Insert in Session

>>>session=Session()>>>actor_a=Actor(first_name='KEN',...last_name='WATANABE')>>>session.add(actor_a)>>>session.commit()

>>>actor_b=session.query(Actor).get(1)>>>actor_b.last_name='GEORGE'>>>session.commit()

HANDS ON: ORM basics

• define Category model with declarative ORM

• select any record

• add "Nature" category

• delete "Nature" category

SQLAlchemy Primer

Engineering

Transcript of SQLAlchemy Primer