Yahoo Query Language: Select * from Internet
-
Upload
drgath -
Category
Technology
-
view
4.812 -
download
1
description
Transcript of Yahoo Query Language: Select * from Internet
select * from internetYahoo Query Language:
Derek Gathright
var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
YQLYQL stmt
Why YQL?
Thousands of Web Services and sources that provide valuable data
Require Developers to read documentation and form URLs/queries.
Data is isolated
Needs filtering, combining, tweaking, shaping even after it gets to the developer.
What is YQL?
Cloud web service and SQL-Like Language • Familiar to developers• Synonymous with Data access• Expressive enough to get the right data.
Self describing - show, desc table
Allows you to query, filter, join and update data across any structured data on the web / web services
• And Yahoo’s sherpa cloud storage
Real time engine
YQL Statement Basics
show: lists the supported tables
desc: describes the structure of a table
select: fetches data
insert/update/delete: modify data
use: use an Open Data Table
set: define key-values across Open Data Tables
Insert / Update / Delete
With INSERT / UPDATE / DELETE SQL verbs you can modify data on the internet or cloud
• create tweets, add blog entries, update DBs etc• store tables or pieces of code in sherpa
Some tables with I/U/D• Twitter, Bit.ly shortener• Yahoo profile status, application update stream• Amazon SimpleDB
Demo: developer.yahoo.com/yql/
Get any data:
Open Data Tables
Open Data Tables
• http://github.com/yql/yql-tables/tree/master
• datatables.org mirrors the master and builds a single environment with all tables (faster serving)
• Console with community tables
• Brings the power of the table model to new APIs
• Schema defines mapping between YQL and Endpoint
• Keys can either be query, path or matrix parameters
• Keys can be marked as required
• YQL Compiler validates existence of required keys
YQL
Open Data Table example
Open Data Table example
Open Data Table example
ODT <execute> element
Extends Open Data Tables with server-side code
Javascript server-side scripting• No DOM, events etc, pure JS scripting• E4X support - XML literals, filtering ...
YQL no longer performs a GET request to the table binding URL
YQL provides additional useful global objects• request, response, y.rest, y.include, y.query...
Execute tag must return the data that will be the output of the YQL select
Move business logic of your application to the cloud
What you can do with <execute>
access APIs that require authentication• Netflix OAuth, FlickrAuth, Google AuthSub
join data across services• grab New York Times article tags and generate find associated flickr photos• combine multiple searches into a single result (twitter, web, news and images)
augment data:• city/state to APIs that just return zip code
create APIs from web page• celebrity birthdays scraped from imdb, caltrain
data transformation• convert the result from xml to Google's kml format
hello world
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table} where a=’hello' and b=’world';</sampleQuery> </meta> <bindings> <select itemPath="" produces="XML"> <urls><url>http://fake.url/{a}</url></urls> <inputs> <key id='a' type='xs:string' paramType='path' required="true" /> <key id='b' type='xs:string' paramType='variable' required="true" /> </inputs> <execute><![CDATA[ // Your javascript goes here. We will run it on our servers response.object = <item> <url>{request.url}</url> <a>{a}</a> <b>{b}</b> </item>; ]]></execute> </select> </bindings></table>
Scraping a page
<select produces="XML" itemPath="deals.deal"> <urls><url>http://deals.yahoo.com</url></urls> <execute><![CDATA[var dealhtml = y.query("select * from html where url='http://deals.yahoo.com' and xpath='//div[contains(@class, \"product-display\")]'").results;var deals = <deals/>;function compress(object) { if (!object) return ""; return object.toString().replace(/\s+/, " ")}for each (var div in dealhtml.div) { var deal = <deal/>; deal.name += <name>{compress(div.h3.a.text())}</name>; deal.link += <link>{div.div[0].a.@href}</link>;… deal.description += <description>{compress(div.p[0].*)}</description>; deal.logo += <logo>{compress(div.div[1].*)}</logo>; deals.deal += deal;}response.object = deals; ]]></execute> </select>
<select itemPath="" produces="XML"><urls><url>http://twitter.com/statuses/show/{id}.xml</url></urls><inputs>
<key id="username" type="xs:string" required="false" paramType="variable"/><key id="password" type="xs:string" required="false" paramType="variable"/><key id="id" type="xs:integer" required="true" paramType="path"/>
</inputs><execute><![CDATA[
var r = null; if (username && password) { y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); r = request.header("Authorization",authheader).get().response; } else { r = request.get().response; } response.object = r;
]]></execute></select>
select * from twitter.status where id="2108869549" and username="xxx" and password="yyy”
Twitter Select
<insert itemPath="" produces="XML"><urls>
<url>http://twitter.com/statuses/update.xml</url></urls><inputs>
<value id="username" type="xs:string" required="true" paramType="variable"/><value id="password" type="xs:string" required="true" paramType="variable"/><value id="status" type="xs:string" required="true" paramType="variable"/>
</inputs><execute><![CDATA[
y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); var content = "status="+status; response.object = request.header("Authorization",authheader).post(content).response;
]]></execute></insert>
insert into twitter.status (status,username,password) values ("new tweet from YQL",”xxx”,”yyy”);
Twitter Update
<delete itemPath="" produces="XML"><urls>
<url>http://twitter.com/statuses/destroy/{id}.xml</url></urls><inputs>
<key id="username" type="xs:string" required="true" paramType="variable"/><key id="password" type="xs:string" required="true" paramType="variable"/><key id="id" type="xs:string" required="true" paramType="path"/>
</inputs><execute><![CDATA[
y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); response.object = request.header("Authorization",authheader).del().response; ]]></execute></delete>
delete from twitter.status where id="2108869549" and username="xxx" and password="yyy”
Twitter Delete
Conclusion
More easily build applications• fewer lines of code• data processing done away from app• consistent and familiar syntax for all data access
Conclusion
Build applications that run faster• service in the cloud does the work
• store tables, environments, js, text files…
• conversion, filtering, parallel requests...
With smaller network footprint• fewer (client) calls
• smaller data amounts
• closer to the data, fatter pipes
Conclusion
Flexible• Supports XML, JSON, JSONP
• Transform documents between formats
• Easy to integrate into any development environment w/ HTTP
Questions?
http://developer.yahoo.com/yql
Derek Gathright
twitter.com/derek
meetup.com/socaljs