APEX and SQL/JSON in Action - nlOUG · Twitter API and the Oracle database: An approach •Download...

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

APEX and SQL/JSON in Action Twitter Analysis for everyone

Carsten Czarski Consulting Member of Technical Staff Oracle Application Express Oracle Deutschland B.V. & Co KG


About: Carsten Czarski

• With Oracle Germany since 2001 – based in Munich, Germany

• Member of the APEX development team since March 2016

• Blogs, Web, Twitter – German APEX community newsletter

blogs.oracle.com/apexcommunity_deutsch

– SQL and PL/SQL sql-plsql-de.blogspot.com

– Twitter @cczarski


About: Twitter Analysis in Oracle Database 12c Oracle12c SQL/JSON Showcase

• Understand data format

• Understand data structure

• Understand Oracle12c SQL/JSON

• Load Tweet data into the database

• Build queries and visualize results

3


Twitter data (from the Twitter API)

4


JSON: JavaScript Object Notation

• Very simple format

• Much simpler than XML

• Based on Javascript

• Increasing importance on the server node.js

• Broad support by web APIs – Google, Twitter, Facebook

{ "PONumber" : 1600, "Reference" : "ABULL-20140421“, "Requestor" : "Alexis Bull“, "User" : "ABULL“, "CostCenter" : "A50“, "ShippingInstructions" : { "name" : "Alexis Bull“, "Address" : { ... }, "Phone" : [ ... ] }, "Special Instructions" : null, "AllowPartialShipment" : true, "LineItems" : [{ "ItemNumber" : 1, "Part" : { "Description" : "One Magic Christmas“, "UnitPrice" : 19.95, "UPCCode" : 13131092899 }, "Quantity" : 9 }, {…}

] }


Understanding the twitter data structure

• Tweet represented by a JSON document

– 1 Tweet = 140 Characters 1 JSON = ~4900 Characters on average

• Many redundancies – Each JSON document contains full user details

– Retweet contains complete 'retweeted tweet'

– Twitter entities (Hashtags, Mentions) are being prepared as arrays

• Storage alternatives – Store JSON as is

– Parse JSON and store data in normalized, relational schema

6


Load Twitter data into the database

• Multiple approaches available: Twitter API, commercial offerings

• Simple approach with the Twitter API

7

http://sql-plsql-de.blogspot.de/2015/09/twitter-analyse-mit-nichts-als.html


Twitter API and the Oracle database: An approach

• Download "twitter4j" – Java Client library for the Twitter API

• Load Twitter4j into the Oracle Database (we use the Oracle JVM)

• Create Java Classes to access Twitter and map these to PL/SQL functions

• Prepare tables, views and PL/SQL packages

8

http://sql-plsql-de.blogspot.de/2015/09/twitter-analyse-mit-nichts-als.html


PL/SQL packages and functions for the Twitter API

9

SQL> desc tweet_pkg FUNCTION GET_GRANTS_SCRIPT RETURNS VARCHAR2 FUNCTION GET_TWEETS_JSON RETURNS TWEET_JSON_CT Argument Name Typ In/Out Defaultwert? ------------------------------ ----------------------- ------ -------- P_SEARCH VARCHAR2 IN P_ID NUMBER IN PROCEDURE STORE_TWEETS Argument Name Typ In/Out Defaultwert? ------------------------------ ----------------------- ------ -------- P_CATID NUMBER IN PROCEDURE STORE_TWEETS SQL> select * from table(tweet_pkg.get_tweets_json('@ogh_nl #apexworld', -1)) where rownum <= 10; COLUMN_VALUE -------------------------------------------------------------------------------- {"retweeted":false,"in_reply_to_screen_name":null,"possibly_sensitive":false,"tr {"retweeted":false,"in_reply_to_screen_name":null,"possibly_sensitive":false,"tr {"retweeted":false,"in_reply_to_screen_name":null,"possibly_sensitive":false,"tr :


The result: Our Twitter data is now in the database

10

SQL> select tweet from apextweets where rownum <= 3; TWEET -------------------------------------------------------------------------------- {"retweeted_status":{"contributors":null,"text":"#countdown naar #Apexworld @OGh _nl #iadvise_live @Yvke1983 geeft met #Robeco een client case over #twitterboots trap #formsmigratie #orclapex","geo":null,"retweeted":false,"in_reply_to_screen_ name":null,"truncated":false,"lang":"nl","entities":{"symbols":[],"urls":[],"has htags":[{"text":"countdown","indices":[0 {"retweeted_status":{"contributors":null,"text":"#countdown naar #Apexworld @OGh _nl #iadvise_live @Yvke1983 geeft met #Robeco een client case over #twitterboots trap #formsmigratie #orclapex","geo":null,"retweeted":false,"in_reply_to_screen_ name":null,"truncated":false,"lang":"nl","entities":{"symbols":[],"urls":[],"has htags":[{"text":"countdown","indices":[0 {"contributors":null,"text":"#countdown naar #Apexworld @OGh_nl #iadvise_live @Y vke1983 geeft met #Robeco een client case over #twitterbootstrap #formsmigratie #orclapex","geo":null,"retweeted":false,"in_reply_to_screen_name":null,"truncate d":false,"lang":"nl","entities":{"symbols":[],"urls":[],"hashtags":[{"text":"cou ntdown","indices":[0,10]},{"text":"Apexw


The Oracle Concept: JSON and SQL Duality

JSON

API / REST

access JSON storage in the

database

SQL Queries / Analytics

Oracle Database 12c

SQL


Native JSON Support in the Oracle Database

Store and manage JSON in tables • Store JSON as Text in tables (VARCHAR2, CLOB, BLOB) Schemaless storage – full flexibility

• No separate data type – All existing applications and tools can work with JSON data

API Access to JSON documents • REST Services (ORDS, SODA)

• Java API

Direct SQL queries on JSON documents Reporting, analysis and more.

'JSON and SQL duality'

12


SQL/JSON functions

• Query JSON data in tables with SQL/JSON functions – External tables do also work

– JSON data can even reside in a Hadoop Cluster or NoSQL DB Big Data SQL

• SQL/JSON functions available – JSON_VALUE extracts one scalar value

– JSON_QUERY extracts a JSON fragment

– JSON_EXISTS returns TRUE when given JSON path exists

– JSON_TABLE projects JSON data as a relational result

– JSON_TEXTCONTAINS performs full-text searches on JSON data

13


Example: Simple attribute extraction with JSON_VALUE

14

select distinct json_value( tweet, '$.user.screen_name' RETURNING VARCHAR2(35) ) from apextweets; JSON_VALUE ------------------------------------------------------------------ AdKosigan Enkitec berkleyp delrez_stephane :


More sophisticated analysis – on JSON data

15



16

SQL/JSON Part projects JSON attributes as a

relational result.



17

"Classic" SQL aggegation


Visualization with APEX

18


More analytics: Tweets by Day

• JSON Attribute: $.created_at

19


More analytics: Hashtag Combinations

20


More analytics: Hashtag Combinations

21


More Informationen

• Oracle Documentation – XML DB Developers Guide

– Chapter 39: JSON in Oracle Database

• Blogs, Web 2.0, Twitter – blogs.oracle.com/jsondb (English)

– json-rest-oracledb.blogspot.com (German)

– sql-plsql-de.blogspot.com (German and English)

22


APEX Connect 26th – 28th April 2016 in Berlin

Get your early-bird ticket now!

APEX.DOAG.ORG #APEXCONN16


[email protected] http://tinyurl.com/apexcommunity http://sql-plsql-de.blogspot.com http://plsqlexecoscomm.sourceforge.net http://plsqlmailclient.sourceforge.net Twitter: @cczarski

APEX and SQL/JSON in Action - nlOUG · Twitter API and the Oracle database: An approach •Download...

Documents

Transcript of APEX and SQL/JSON in Action - nlOUG · Twitter API and the Oracle database: An approach •Download...