Web API, REST API and Web Scraping

download Web API, REST API and Web Scraping

of 40

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Web API, REST API and Web Scraping

  • Web and REST API + ScrapingMarco Brambilla, Riccardo Volonterio

    @marcobrambi@manudellavalleWeb Science #WebSciCo

  • The Data Analysis Pipeline

  • The 5 3 Ws of (Social) API & Scraping

    1) What2) Why3) Who4) When? Now.5) Where? Everywhere.6) How7) Use cases

    a. UrbanScopeb. La Scala

  • What

    An API (Application Program Interface) is a set of routines, protocols and tools for building software applications


    A WebAPI = HTTP based API Allows a programmatic access to data and platform

    And what if no API is available?

  • What

    An API (Application Program Interface) is a set of routines, protocols and tools for building software applications


    A WebAPI = HTTP based API Allows a programmatic access to data and platform

    And what if no API is available? Scraping!

  • Why

    Separation between model and presentation Regulate access to the data

    Traceable accounts Enable access to only a portion of the available data By Time By Location

    Avoid direct access to the platform website Request throttling

    Avoid server congestion Avoid DOS/DDOS

    Provide paid access to full (or higher volume) data

  • Who


  • How

    URLs API types

    RESTful APIs Authentication Scraping

  • How - URL

    URL format (RFC 2396):scheme:[//[user:password@]host[:port]][/]path[?query][#fragment]



    scheme credentials domain port path querystring fragment

  • How HTTP request

    When the scheme is HTTP(s), we can make HTTP requests as:VERB URL

    The VERB (or method) can be one of: GET POST DELETE OPTIONS and more: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

  • How - Endpoints

    Each API exposes a set of HTTP(s) endpoints (URLs) to call in order to get data or perform action on the platform.

    Example (retrieve data):GET https://graph.facebook.com/searchGET https://api.twitter.com/1.1/search/tweets.jsonGET https://api.instagram.com/v1/tags/nofilter/media/recent

    Example (platform actions):POST https://api.instagram.com/v1/media/{media-id}/commentsPOST https://api.twitter.com/1.1/statuses/update.jsonPOST https://graph.facebook.com/me/feed


  • How - Parameters

    Most of the endpoints can be tweaked via one or more parameters. Parameters can be added to an URL by appending to the endpoint some (URL encoded) strings.

    Example (Twitter):/search/tweets.json?q=%23expo2015milano&lang=it&result_type=recent&count=100&token=1235abcd

    The request in plain English is: Twitter, I am 1235abcd, can you please give me the 100 most recent tweets, in italian, by the user @expo2015milano?

  • How - REST

    RESTRepresentational state transfer (REST) is a style of software architecture.

    - Roy T. Fielding

    Client-Server: a uniform interface separates clients from servers.Stateless: the client-server communication is constrained by no client contextbeing stored on the server.Cacheable: clients and intermediaries can cache responses.Layered system: clients cannot tell whether are connected directly to theserver.Uniform interface: simplifies and decouples the architecture.

  • How - APIs

    WebAPI vs RESTful API

    WebAPI: An API that uses the HTTP protocol. Non standard/unpredictable URL format.

    POST http://example.com/getUserData.do?user=1234

    RESTful API: Resource based WebAPI. Standard/predictable URL format.GET http://example.com/user/1234

  • How RESTful API

    The RESTful API uses the available HTTP verbs to perform CRUD operations based on the context: Collection: A set of items (e.g.: /users) Item: A specific item in a collection (e.g.: /users/{id})

    VERB Collection Item

    POST Create anewitem. Notused

    GET Get listofelements. Gettheselected item.

    PUT Notused Updatetheselected item.

    DELETE Notused Delete theselected item.

  • How - Authentication

    Almost all the APIs require a kind of user authentication. The user mustregister to the developer of the provider to obtain the keys to access the API.Most of the main API providers use the OAuth protocol to authenticate auser.

    The user registers to the developer portal to obtain a key and a secret.Platform authenticates the user via key/secret pair and supply a token thatcan be used to perform HTTP(S) requests to the desired API endpoint.

  • Challenges

    Pagination Timelines

    Parallelization Multiple accounts API gaps

  • Challenges - Pagination

    Like SQL queries, most APIs support data pagination to split huge chunks of data into smaller set of data.

    Smaller chunks of data are easier to create, transfer (avoid long response time), cache and require less server computation time.

  • Challenges - Timeline

    Most of the Social Networks leverage on the concept of timelines instead ofstandard pagination to iterate through results.

    An ideal pagination will work like the following (Twitter example):

  • Challenges Timeline 2

    The problem is that dataare continuously added tothe top of the Twittertimeline.

    So the next requestrequest can lead toretrieving the alreadyprocessed tweets.

  • Challenges Timeline 3

    The solution is to use thecursoring technique.

    Instead of reading fromthe top of the timeline weread the data relative tothe already processedids.

  • Challenges - Parallelization

    When possible make parallel requests to gather more data in less time.

    Example:Get all the media from Instagram about @potus and #obama.

    We can make two parallel requests (one for each tag) and paginate the results instead of waiting for the first tag to finish.

    Some time this is not possible due to the requests interdependency like in pagination; each request depends on the result of previous one.

  • Challenges Multiple accounts

    Handling multiple accounts can increase the system throughput but increases the complexity of the system.

    Strategies to handle multiple accounts: Request based

    Round robin Account pool

    Account based Requests stack

  • Challenges Multi accounts Round robin


  • Challenges Multi accounts Account pool

    The account through which the request is made is chosen sequentially until its rate limitis reached, then we use the next available one.

  • Challenges Multi accounts Requests Stack

    This strategy overturn the roles. Now the accounts, inparallel, get the next request to do from the pool. Once anaccount completes a request the cycle restarts until the poolis empty.

  • Challenges Fill Gaps

    Sometimes the API does not give us all the data we want, so we need to fill those gaps.

    On quantity or time (Twitter) or on quality/content (Foursquare).

    Example: Twitter time limited data. We need to access data older then 10 days.

    Example:Foursquare check-ins timeline not available.Facebook like timestamps not available.


  • Use case UrbanScope (Crawler)

    An API Crawler is a software that methodically interacts with a WebAPI to download data or to take some actions at predefined time intervals.


  • Use case UrbanScope (Twitter)

    Goal: Get all the geolocated tweets in Milan and analyze the languages to detect anomalies.

    Problem: Twitter does not provide a simple endpoint give me all tweets in Milan.

    Solution: Create a grid of points (50m distance) over Milan and ask Twitter for each point.

    Problem: 70.000 points, with 1 account is 4 days.

    Solution: Use more keys!

    Problem: Handle multiple keys.

    Solution: Cry Then handle multiple keys.

  • Use case - UrbanScope - Explore Tweets

  • Use case UrbanScope (Foursquare)

    Goal: See how the check-ins progress in time to see where people go and visualize the pulse of Milan.

    Problem: Foursquare does not provide the timeline of the check-ins.

    Solution: Build our own timeline by collecting the number of check-ins every 4 hours for 90.000 venues.

    Problem: 90.000 api requests to do in 4 hours with an API time limit of 5000 requests per hour.

    Solution: See twitter with more tears and more keys (24)

    Problem: Foursquare ban accounts that seems to crawl data automatically -.-

  • Use case UrbanScope City magnets

  • Use case UrbanScope- Top Venues

  • 0








    Use case Teatro Alla Scala (Milan)

  • Scraping

    Web scraping means collecting information from websites by extracting them directly from the HTML source code.

  • Use case La Scala (Scraper)

    Goal: Steal Get data from twitter for 1 year ago.

    Problem: API allow access to 10 days in the past.

    Solution: The Twitter homepage allows to search without a time limit. Crete a web scraper that collect data from the HTML.

    Problem: The page needs to scroll to load more data.

    Solution: Use a (headless) fake browser to simulate the user interaction.

    Problem: You need to start a browser to get data, can be memory greedy.

    Solution: No. Maybe.

  • Use case La Scala (Scraper)

    Once we