PostgreSQL and JSON with Python - Przemek Lewandowski
-
Upload
sunscrapers -
Category
Software
-
view
154 -
download
6
Transcript of PostgreSQL and JSON with Python - Przemek Lewandowski
![Page 1: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/1.jpg)
sunscrapers.comYour favored tech partner.
PostgreSQL and JSONwith PythonPrzemek Lewandowski
16.06.2016
![Page 2: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/2.jpg)
sunscrapers.comYour favored tech partner.
1. Whys2. PostgreSQL types3. HStore vs JSON vs JSONB4. SQLAlchemy5. Django
Overview
![Page 3: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/3.jpg)
sunscrapers.comYour favored tech partner.
● Schema-less data● Schema-unknown data● Document storage● User defined schema
Why?
![Page 4: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/4.jpg)
sunscrapers.comYour favored tech partner.
● XML (since 8.2)
● HStore (since 8.2)
● JSON (since 9.2)
● JSONB (since 9.4)
PostgreSQL types
![Page 5: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/5.jpg)
sunscrapers.comYour favored tech partner.
● Only simple key/value pairs● No nesting● Strings only● Indexing● Many functions and operators
HStore
SELECT 'user => "fred", id => 1'::hstore -> 'user' AS user;
user------fred
![Page 6: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/6.jpg)
sunscrapers.comYour favored tech partner.
● Simple validation on input● Stored as text (like XML)● Preserves key order and duplicates● Indexes (only expression index)
JSON
SELECT '[{"a": {"b":{"c": "foo"}}}'::json#>'{a,b}';
?column?-------------{"c": "foo"}
![Page 7: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/7.jpg)
sunscrapers.comYour favored tech partner.
● Full JSON implementation● Binary storage● No key order or duplicate preservation● Fast access operations● Indexing● No date type!
JSONB
SELECT '{"a": {"b": 2, "c": 3}}'::jsonb @> '{"a": {"c": 3}}'::jsonb;
![Page 8: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/8.jpg)
sunscrapers.comYour favored tech partner.
SELECT feed_data.feed_id AS feed_data_feed_id, feed_data.feed AS feed_data_feed, feed_data.data AS feed_data_dataFROM Feed_dataWHERE CAST(feed_data.data ->> 'DISSEMINATION_ID' AS INTEGER) <= 90000 AND (feed_data.data ->> 'ACTION') = 'NEW'ORDER BY feed_data.data -> 'PRICE_NOTATION' DESCLIMIT 100 OFFSET 0
Query example
![Page 9: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/9.jpg)
sunscrapers.comYour favored tech partner.
SQLAlchemy
● PostgreSQL dialect● Close to database● HStore, JSON, JSONB data types and much more
![Page 10: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/10.jpg)
sunscrapers.comYour favored tech partner.
from sqlalchemy.dialects.postgresql import JSONB
# …
class FeedData(db.Model): feed_id = Column(String(), primary_key=True) feed = Column(Enum(*feed_urls.keys(), name='feeds_enum'), nullable=False) data = Column(MutableDict.as_mutable(JSONB), nullable=False)
def __init__(self, feed_id, feed, data): self.feed_id = feed_id self.feed = feed self.data = data
def as_dict(self): data = {field: getattr(self, field) for field in ['feed_id', 'feed']} data.update(self.data) return data
SQLAlchemy ORM example I
![Page 11: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/11.jpg)
sunscrapers.comYour favored tech partner.
def filter_by_date(db_field, dt_from, dt_to): return and_( db_field.cast(Date) >= dt_from, db_field.cast(Date) <= dt_to )
SQLAlchemy ORM example II
![Page 12: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/12.jpg)
sunscrapers.comYour favored tech partner.
Django ORM
● Django introduced PostgreSQL specific model fields● HStoreField since 1.8● JSONField since 1.9 (uses JSONB)● Support for migrations● Third party libs like: django-hstore
![Page 13: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/13.jpg)
sunscrapers.comYour favored tech partner.
More info
● https://www.pgcon.org/2014/schedule/events/684.en.html● https://www.postgresql.org/docs/9.4/static/datatype-json.html● https://docs.djangoproject.com/en/1.9/ref/contrib/postgres/fields/● http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html
![Page 14: PostgreSQL and JSON with Python - Przemek Lewandowski](https://reader036.fdocuments.in/reader036/viewer/2022082720/58ab4db51a28ab61758b6635/html5/thumbnails/14.jpg)
sunscrapers.comYour favored tech partner.
Thanks!Questions?