SQLAlchemy Primer

SQLAlchemy Primer(extra content)

for Kobe Python Meetup #13 2017/09/15 Kobe Japan

Yasushi Masuda PhD ( @whosaysni )

Tech team, Core IT grp. IT Dept. MonotaRO Co., LTD.

Pythonista since 2001 (2.0～) • elaphe (barcode library) • oikami.py （老神.py） • PyCon JP founder

Japanese Translation works

Agenda

Core concepts in SQLAlchemy

Engine basics (+hands-on)

ORM primer (+hans-on)

References

Online Document:http://docs.sqlalchemy.org/

en/rel_1_1/

(Old) Japanese translation:http://omake.accense.com/static/doc-ja/sqlalchemy/

Preparationsakila DB on SQLite http://bit.ly/2fdeeft

https://github.com/jOOQ/jOOQ/jOOQ-examples/Sakila/sqlite-sakila-db/sqlite-sakila.sq

Sakila • Demonstration

DB for MySQL • Models a rental

video shop • BSD License

Schema described at: https://dev.mysql.com/doc/sakila/en/sakila-structure-tables.html

Myths[WRONG!] It's just an ORM libraryNot limited to. SQLAlchemy is a DB manipulation framework.

[WRONG!] SA is built on ORMNO. Connection management and SQL expression framework are its core.ORM is built on them.[WRONG!] SA cannot handle raw SQLTable definition is not required. Even textual SQL is available.

[WRONG!] SA automatically escapes value for youSQLAlchemy relies value escaping on DB-API, while it escapes schema, table, column names. SQLAlchemy generates parameterized query that helps DB-API level escaping.

[WRONG!] Only Python adept can handle SASQLAlchemy is EASY. You may need SQL and RDBMS experience. Let's see!

[AGREE] Documentation is somewhat difficult to understandAgreed.

Core concepts

http://docs.sqlalchemy.org/en/latest/index.html

SQLAlchemy Core

Engine (connection) Schema definitionSQL Expression

SQLAlchemy ORM

Mapper Declarative Mapper

Session

Dialect DB Backend-specific functionalities

Engine manages DB connection(s)

SQL Expressiondescribes SQL statement in Python

Mappingreflects DB record with Python object

Dialect DB Backend specific functionalities

DatabaseYour program

SQL construction

Query execution

Typeconversion

Parameter binding

Driver Setup

Connection management

Result data structure

Value escaping

Schema name

Type conversion

Dialect-specific

Query Construction

Query Execution

Result

Schema object Object-relational mapping

High-level Interface

DB Session

Engine

DB API DB ServerProgram

Issues around DB-API

Which DB-API to use How to initialize the DB-API

How to generate valid query for it How to handle cursor on the DB-API How to execute query on the DB-API How to retrieve result from DB-API How to reserve/reuse connection

DB API DB ServerProgram

Engine resolves issues

Select DB-API from DSN URL Initialize DB-API for you

Accept string and SQL expression Manage cursor for you

Unify execution/transaction API Handle result via ResultProxy Pool connection automatically

Engine

Engine DB API DB ServerProgram

>>>fromsqlalchemyimportcreate_engine>>>

>>>fromsqlalchemyimportcreate_engine>>>e=create_engine('sqlite://')#SQLitememoryengine>>>

#UseURLforspecifyingdatabase>>>e=create_engine('sqlite://')#SQLitememoryengine>>>e=create_engine('sqlite:///path_to_db.sqlite3')#sqlite3>>>e=create_engine('mysql://scott:tiger@dbserv/dbname')#mysql>>>e=create_engine('mssql://bill:gates@dbserv/dbname')#mssql>>>

#Becarefulwithnumberofslashes>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>e=create_engine('sqlite:////<absolute_path>/sqlite-sakila.sq')

>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>eEngine(sqlite:///sakila-data.sq)

#executereturnsResultProxy>>>q='selecttitlefromfilmlimit5')>>>res=e.execute(q)>>>res<sqlalchemy.engine.result.ResultProxyobjectat0x10da96990>>>>

>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>eEngine(sqlite:///sakila-data.sq)

#executereturnsResultProxy>>>q='selecttitlefromfilmlimit5')>>>res=e.execute(q)>>>res<sqlalchemy.engine.result.ResultProxyobjectat0x10da96990>>>forrowinres:#ResultProxycanbeiterated/namedtupleaccess...print(row.title)...ACADEMYDINOSAURACEGOLDFINGERADAPTATIONHOLESAFFAIRPREJUDICEAFRICANEGG>>>

>>>q='selectfilm_id,titlefromfilmlimit10'>>>rows=list(e.execute(q))>>>rows[2][1]#eachrowisaccessiblelikeastuple'ADAPTATIONHOLES'>>>rows[4]['title']#rowcanbeaccessiblelikeasdictionary'AFRICANEGG'

>>>res=e.execute(q)>>>forfid,titleinres:#canbeexpandedasnormaltuple...print((fid,title))...(1,'ACADEMYDINOSAUR')(2,'ACEGOLDFINGER')(3,'ADAPTATIONHOLES')(4,'AFFAIRPREJUDICE')(5,'AFRICANEGG')>>>rows=list(e.execute(q))>>>

transaction

>>>t=e.begin()>>>t.transaction<sqlalchemy...RootTransactionobjectat...>>>>t.transaction.commit()

#withstatementhandlestransactionsmart>>>withe.begin():e.execute(...)>>>#(transactioncommittedautomatically)

HANDS ON: Engine basics

Connecttosqlite-sakila.sqdatabase

Listactorsinfilm"DINOSAURSECRETARY"

HANDS ON: Engine basics

>>>e=create_engine('sqlite:///sqlite-sakila.sq')>>>q='''selecta.first_name,a.last_name...fromfilmasf...innerjoinfilm_actorasfa...onf.film_id=fa.film_id...innerjoinactorasa...onfa.actor_id=a.actor_id...wheref.title="DINOSAURSECRETARY"'''>>>forfirst_name,last_nameine.execute(q):...print('{}{}'.format(first_name,last_name))...LUCILLETRACYBURTDUKAKISJAYNENEESONRUSSELLBACALLPENELOPEMONROEMINNIEKILMER

SQL expression

Remember: SQL is a language

SELECT[ALL|DISTINCT[ON(expression[,...])]]

[*|expression[[AS]output_name][,...]]

[FROMfrom_item[,...]]

[WHEREcondition]

[GROUPBYgrouping_element[,...]]

[ORDERBYexpression[ASC|DESC|USINGoperator][NULLS{FIRST|LAST}][,...]]

[LIMIT{count|ALL}]

[WHEREcondition]

[LIMIT{count|ALL}]

STATEMENT

[WHEREcondition]

[LIMIT{count|ALL}]

CLAUSE

[WHEREcondition]

[LIMIT{count|ALL}]

parameter

expression

Remember: SQL is a Language

SELECT

FROM selectablesselectable

selectable

GROUP BY ORDER BY

expression

expressioncondition

expression

expressionsexpression

expression

SELECT statement

SELECTA.name,B.title,C.title

FROMartistsasAINNERJOINalbumasBONA.id=B.artist_idINNERJOINmusicasCONB.id=C.album

WHEREA.nameLIKE'%Astor%'ANDA.yearBETWEEN1970AND2010ANDC.titleLIKE'%Tango%'

From SQL to Python

SELECT<column>,<column>,...

FROM<selectable>INNERJOIN<selectable>ON<condition>INNERJOIN<selectable>ON<condition>...

WHERE<condition>AND<condition>AND<condition>

GROUPBY...LIMIT...

Clauses

SELECT<[<column>,<column>,<column>]>

FROM<join(<selectable>,<selectable>,...)>

WHERE<and(<condition>,<condition>,...)>

Statement and subjects

SELECT<expressions>

FROM<selectable>

WHERE<conditions>

... simplifed

..., finally

engine.execute(<selectstatement>)

If query is "an object"...

Query object

>>>query=select(...)

>>>engine.execute(query)

... it can be "execute()-able"

Engine "compiles" query into string and execute it

(according to dialect)

query=select(

<expression>,

from_obj=<selectables>,

whereclause=<conditions>,

clauses as parameters

columns=[col1,col2,...]

fromobj=join(tbl1,tbl2,...)

where=and_(expr1,expr2,...)

query_expr=select(columns,from_obj=fromobj,whereclause=where)

query with SQL expression

>>>fromsqlalchemy.sqlimportselect,text>>>q=select([text('*')])

building sql statement with basic sql expression

>>>fromsqlalchemy.sqlimportselect,text>>>q=select([text('*')])>>>q<sqlalchemy.....Selectat...;Selectobject>>>>str(q)'SELECT*'>>>q=select([text('*')],...from_obj=text('foo'),...whereclause=text('id=3'))>>>str(q)'SELECT*\nFROMfoo\nWHEREid=3'

elements: table and columnSchema

Table Table Table

Column

SchemaTable Table Table

Column

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)>>>c<sqlalchemy....ColumnClauseat...;name>>>>str(c)'name'

elements: table and column>>>fromsqlalchemy.sqlimportcolumn,table>>>fromsqlalchemyimportINTEGER>>>c=column('name')#simplest>>>c=column('name',type_=INTEGER)>>>c<sqlalchemy....ColumnClauseat...;name>>>>str(c)'name'>>>c.table#None>>>t1=table('artist')>>>c.table=t1>>>str(c)'artist.name'

>>>t=table('tbl1',column('col1'),...)

defining table with columnsTable name List of columns

>>>t.c.col1

<sqlalchemy.....ColumnClauseat...;col1>

>>>t.c.col1

<sqlalchemy.....ColumnClauseat...;col1>

>>>t.schema='db1'

>>>str(t.c.col1)

'db1.tbl1.col1'

>>>t=table('tbl1',column('col1'),column('col2'))

select() with table element

>>>t=table('tbl1',column('col1'),column('col2'))>>>print(select([t]))SELECTtbl1.col1,tbl1.col2FROMtbl1

#columnlabeling>>>print(select([t.c.col1.label('col_alias1')])SELECTtbl1.col1as"col_alias1"FROMtbl1

#tablealias>>>t_A,t_B=alias(t,'A'),alias(t,'B')>>>print(select([t_A.c.col1,t_B.c.col2]))SELECT"A".col1,"B".col2FROMtbl1AS"A",tbl1AS"B"

howtoworkwith

wheretbl1.col1=42

conditionals

howtoworkwith

wheretbl1.col1=42

conditionals

conditional expression (compare operation)

>>>cond=text('last_nameLIKE%KOV')>>>str(cond)'last_nameLIKE%KOV'

conditional by text()

>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>

conditional by like() method

>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>>>>str(cond)'last_nameLIKE:last_name_1'

placeholder for right value of LIKE

>>>cond=column('last_name').like('%KOV')>>>cond<sqlalchemy....BinaryExpressionobjectat...>>>>str(cond)'last_nameLIKE:last_name_1

>>>cond.rightBindParameter('%(4339977744last_name)s','%KOV',type_=String())

>>>column('first_name')='DAVID'File"<stdin>",line1SyntaxError:can'tassigntofunctioncall

conditional by operation

>>>column('first_name')=='DAVID'<sqlalchemy...BinaryExpressionobjectat...>

>>>cond=column('first_name')=='DAVID'>>>str(cond)>>>'first_name=:first_name_1'>>>cond.right>>>BindParameter('%(...)s','DAVID',type_=...)

>>>str(column('last_name')==None)>>>'last_nameisNULL'

>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))

conditionals in select

>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))

>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))>>>query<sqlalchemy.....Selectat...;Selectobject>>>>print(query)SELECTactor.first_name,actor.last_nameFROMactorWHEREactor.last_name=:last_name_1>>>query.compile().params{'last_name_1':'TRACY'}

>>>fromsqlalchemy.sqlimportselect,table,column>>>actor_tbl=table('actor',column('actor_id'),...column('first_name'),column('last_name'))>>>query=select([actor_tbl],...whereclause=(actor_tbl.c.last_name=='TRACY'))>>>query<sqlalchemy.....Selectat...;Selectobject>>>>print(query)SELECTactor.actor_id,actor.first_name,actor.last_nameFROMactorWHEREactor.last_name=:last_name_1>>>query.compile().params{'last_name_1':'TRACY'}>>>list(e.execute(query))[(20,'LUCILLE','TRACY'),(117,'RENEE','TRACY')]

select(...)

SELECT...

select(...).select_from(...)

SELECT...FROM...

select(...).where(...)

SELECT...WHERE...

select(...).where(...).where(...)

SELECT...WHERE...AND...

select(...).where(...).order_by(...)

SELECT...WHERE...ORDERBY...

generative method

actor_tbl.select()SELECT...FROMactor

actor_tbl.select(actor_tbl.c.first_name='BEN')SELECT...FROMactorWHEREfirst_name=...

generative method

HANDS ON: SQL expression

• define actor table with table()/column()

• build query with sql expression to searchactor having initial A.H. (result should include actor_id)

Schema definition

howtoCREATEtable

schema definition

howtoCREATEtable

howtodefinecolumndetailhowtodefineconstraints

schema definition

>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>

defining table

>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>user_tableTable('user',MetaData(bind=None),Column...)

defining table

>>>fromsqlalchemyimportColumn,MetaData,Table>>>user_table=Table(...'user',MetaData(),...Column('id',INTEGER,primary_key=True),...Column('first_name',VARCHAR(45)),...Column('last_name',VARCHAR(45)))>>>user_tableTable('user',MetaData(bind=None),Column...)

#CREATETABLEbycrete()>>>user_table.create(bind=e)

#DROPTABLEwithdrop()>>>user_table.drop(bind=e)

defining table

Table/Column vs table/columnVisitable (base type)

ClauseElement

Selectable

FromClause [ table() ]

ColumnElement

ColumnClause [ column() ]

Column Table

SchemaItem

Table/Column vs table/columnVisitable (base type)

ClauseElement

Selectable

FromClause [ table() ]

ColumnElement

ColumnClause [ column() ]

Column Table

SchemaItem

Supprorts same operation

HANDS ON: Schema definition

• define "user" table with Table()/Column()

• create table with .create()

• insert records

• drop table with .drop()

name type options

id INTEGER primary_key

username VARCHAR (64) nullable=False

email VARCHAR(128) nullable=False

ORM basics

Object-Relational Mapping

• Map DB records to objects

• ActiveRecord pattern

• 1 table ~> 1 class, 1 record ~> 1 object

• FK reference -> reference to other object

• column -> property/attribute of object

ORM in SQLAlchemy

• mapper maps a table with Python classmapped class will have instrumented attribute

• Session manipulates DB for you to retrieve / save mapped objects

mappingTable

Column foo

Column bar

Column baz

entity class

"mapped" entity class

instrumented fooinstrumented barinstrumented baz

method quxmethod quux

mapper

classic mapping

[new] declarative mapping

mapping patterns

>>>fromsqlalchemy.ormimportmapper>>>actor_tbl=Table(...)>>>classActor(object):pass>>>mapper(Actor,actor_tbl)>>>dir(Actor)['__class__',...'_sa_class_manager','actor_id','first_name','last_name']>>>Actor.actor_id<...InstrumentedAttributeobjectat...>

classic mapping

>>>fromsqlalchemy.ext.declarativeimportdeclarative_base>>>fromsqlalchmyimportDateTime,Integer,String>>>Base=declarative_base()>>>>>>classActor(Base):...__tablename__='actor'...actor_id=Column(Integer,primary_key=True)...first_name=Column(String)...last_name=Column(String)...last_update=Column(DateTime)

declarative mapping

>>>fromsqlalchemy.ext.declarativeimportdeclarative_base>>>fromsqlalchmyimportDateTime,Integer,String>>>Base=declarative_base()>>>>>>classActor(Base):...__tablename__='actor'...actor_id=Column(Integer,primary_key=True)...first_name=Column(String)...last_name=Column(String)...last_update=Column(DateTime)>>>dir(Actor)['__class__','_decl_class_registry','_sa_class_manager','actor_id','first_name','last_name','last_update','metadata']

declarative mapping

ORM in SQLAlchemy

• mapper maps a table with Python classmapped class will have instrumented attribute

• Session manipulates DB for you to retrieve / save mapped objects

sessionEngineProgram

query Aselect A

record Aobject Bstart

tracking A

(updates) flag A as "dirty"

reflect new/dirty changes

starttracking B

add object B

update A insert B

commit

Using Session

>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>>>>

Using Session

>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>query=session.query(Actor)>>>query<sqlalchemy.orm.query.Queryobjectat...>

Using Session

>>>fromsqlalchemy.ormimportsessionmaker>>>Session=sessionmaker(bind=e)>>>session=Session()>>>query=session.query(Actor)>>>query<sqlalchemy.orm.query.Queryobjectat...>>>>actor=query.first()>>>actor<...ActorObjectat...>

Query object methods: query filtering

>>>q=session.query(Actor)>>>str(q)>>>'SELECT...\nFROMactor'>>>q1=q.filter(Actor.first_name=='BEN')>>>str(q1)'SELECT...\nFROMactor\nWHEREactor.first_name=?'>>>q2=q1.filter_by(last_name='SMITH')>>>str(q2)'SELECT...\nFROMactor\nWHEREactor.first_name=?ANDactor.last_name=?'

Query object methods: fetching record(s)

>>>q=session.query(Actor)>>>q.first()<...Actorobjectat...>

>>>q.all()#WARNING:SELECTsallrecords<...Actorobjectat...>,<...Actorobjectat...>,...

>>>q[:3]<...Actorobjectat...>,<...Actorobjectat...>,...

>>>q.count()200

Update/Insert in Session

>>>session=Session()>>>actor_a=Actor(first_name='KEN',...last_name='WATANABE')>>>session.add(actor_a)>>>session.commit()

>>>actor_b=session.query(Actor).get(1)>>>actor_b.last_name='GEORGE'>>>session.commit()

HANDS ON: ORM basics

• define Category model with declarative ORM

• select any record

• add "Nature" category

• delete "Nature" category

SQLAlchemy Primer

Engineering

Transcript of SQLAlchemy Primer

SQLAlchemy Session In Depth - zzzeek

for Pythonistas SqlAlchemy and pandas tips€¦ · Python 3 (includes sqlite3, Python DBI, etc.) SqlAlchemy IPython-SQL (%sql) pandas and Python. SqlAlchemy The de facto standard

SQLAlchemy quick reference

Object Relational Mapping - @let@token ORM & SQLAlchemy

sqlalchemy-migrate Documentation€¦ · sqlalchemy-migrate Documentation, Release Downgrading... done Success Our script runs on our database (sqlite:///project.db, as speciﬁed

An introduction to SQLAlchemy

Rest API using Flask & SqlAlchemy

sqlalchemy-redshift Documentation - Read the Docs › pdf › sqlalchemy-redshift › latest › ... · sqlalchemy-redshift Documentation, Release 0.7.2.dev0 Dialect class sqlalchemy_redshift.dialect.RedshiftDialect(*args,

sqlalchemy-redshift Documentation

Selena Deckelmann - Sane Schema Management with Alembic and SQLAlchemy @ Postgres Open

GeoAlchemy Documentation - GeoAlchemy: Using SQLAlchemy ...

SQLAlchemy Documentation

SQLAlchemy and Elixir (in a few minutes) Neil Blakey-Milner

SQLAlchemy-Searchable Documentation · 2019. 4. 2. · As of version 1.0 SQLAlchemy-Searchable comes with native PostgreSQL search query parser. The search query parser is capable

SQLAlchemy - discorporate · from sqlalchemy import MetaData, Table, Column, String, Integer metadata = MetaData() users = Table('users', metadata, Column('id', Integer, primary_key=True),

SQLAlchemy filters plus

SQLAlchemy-ImageAttach Documentation

Introduction to SQLAlchemy and Alembic Migrations

sqlalchemy-seeder Documentation

Flask-SQLAlchemy Documentation...CHAPTER 1 User’s Guide This part of the documentation will show you how to get started in using Flask-SQLAlchemy with Flask. 1.1Quickstart Flask-SQLAlchemy