CS 430P/530: Internet, Web, & Cloud Systems
Transcript of CS 430P/530: Internet, Web, & Cloud Systems
![Page 1: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/1.jpg)
APIs/abstractions
![Page 2: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/2.jpg)
Previously
Abstractions for infrastructure to ease operations (Ops) Manage complexity in deploying software systems
Examples covered… VMs, containers
Infrastructure as a Service
Platform as a Service
Functions as a Service
Infrastructure as Code
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 3: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/3.jpg)
But, what about the apps themselves?
Emerging software abstractions for building functionality
Example: Your homework (e.g. Flask) Common web application abstractions as a framework
No need to program sockets, HTTP, cookies, sessions, routing, HTML
generation, database communication, etc.
Imagine writing it in C (from scratch!)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 4: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/4.jpg)
Historically
From Unix to Windows in 90s/00s
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 5: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/5.jpg)
To this, 20 years later
What happened?
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 6: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/6.jpg)
Currently this…on the front-end
Mobile apps (Android/iOS SDKs)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 7: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/7.jpg)
Web apps (client-side JS frameworks)
Or both Progressive web apps
Seamless off-line/on-line experience
Native/hybrid applications
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 8: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/8.jpg)
Meanwhile…on the backend
Server-side web and API frameworks
Data and event management
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 9: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/9.jpg)
Now this…
Software as a service via APIs
Comprehensive lists at https://rapidapi.com/
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 10: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/10.jpg)
Even traditional businesses are building them…
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 11: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/11.jpg)
Others…
Payments (PayPal, Venmo)
Social media (Twitter, Facebook, Instagram, Pinterest, Spotify)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 12: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/12.jpg)
Multi-service instant
messaging and
communications (Twilio,
Nexmo) Communicate with
customers using whatever
they have!
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 13: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/13.jpg)
IoT (LiFX, GCP IOT Core, AWS IOT Core)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 14: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/14.jpg)
Food and nutrition APIs (Nutritionix)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 15: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/15.jpg)
Spoonacular
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 16: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/16.jpg)
On GCP…
Maps API, Search API, Knowledge Graph API, Analytics, Ads, Voice
transcription, Image analysis, Video analysis, Translation, etc.
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 17: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/17.jpg)
Accessing APIs
![Page 18: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/18.jpg)
#1: Python packages
(if they exist)
![Page 19: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/19.jpg)
PyPI
Python Package Index Packages installable via pip (as we have been doing all quarter)
Most popular web APIs come with supported Python packages
Many are auto-generated via Swagger
Note Code examples are written for simplicity
NEVER specify API keys within source code
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 20: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/20.jpg)
Example: Giphy
pip install giphy_client
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import giphy_client
# Create app and API_KEY at https://developers.giphy.com# Note insecure pattern: Should supply from environment variablesAPI_KEY = os.environ.get('YOUR_GIPHY_API_KEY')
api_instance = giphy_client.DefaultApi()api_response = api_instance.gifs_search_get(API_KEY,
'clapback', limit=1, rating='g', lang='en', fmt='json')if len(api_response.data) > 0:
print(api_response.data[0].embed_url)
![Page 21: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/21.jpg)
Example: Spotipy (Spotify)
pip install spotipy
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import spotipyfrom spotipy.oauth2 import SpotifyClientCredentials
# Obtain credentials via https://developer.spotify.com# Pass to client via environment variables that are not committed to gitCLIENT_ID=os.environ.get('SPOTIFY_CLIENT_ID')CLIENT_SECRET=os.environ.get('SPOTIFY_CLIENT_SECRET')
# Return a spotify playlist given querycredentials = SpotifyClientCredentials(client_id=CLIENT_ID,
client_secret=CLIENT_SECRET)spotify = spotipy.Spotify(client_credentials_manager=credentials)results = spotify.search('Foo Fighters', limit=1, type='playlist')if len(results['playlists']['items']) > 0:print(results['playlists']['items'][0]['external_urls']['spotify'])
![Page 22: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/22.jpg)
#2: API discovery
![Page 23: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/23.jpg)
OpenAPI specifications
OpenAPI initiative (Linux Foundation) Formerly known as Swagger (swagger.io)
"Standard, language-agnostic interface to REST APIs which allows
humans and computers to discover and understand the capabilities fo the
service without access to source code, documentation, or through
network traffic inspection"
Specify REST API and its semantics in standard way Code parses specification to automatically build calls that use it
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 24: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/24.jpg)
Google's API Discovery format
API discovery format Supports both REST APIs with OpenAPI specifications as well as gRPC's
generated API clients Recall gRPC (open-source RPC framework) supporting zero-copy, binary data
transfer with specifications written in Protocol Buffers language
More info at https://googleapis.github.io/
OpenAPI/Swagger tools here
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 25: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/25.jpg)
Example
In-platform access of storage API via apiclient.discovery Avoid use of key management for accesses done completely within GCP
pip install google-api-python-client google-auth
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import apiclient.discoveryimport google.auth
# Obtain credentials and instantiate APIcredentials, project_id = google.auth.default()gcs_api = apiclient.discovery.build('storage', 'v1', credentials=credentials)
# Obtain credentials and instantiate APIgcs_api.objects().insert(bucket='wuchang-bucket',
name='myfile.txt',media_body='./myfile.txt').execute()
![Page 26: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/26.jpg)
YouTube API via apiclient.discovery Access from anywhere, but requires API key
Portland State University CS 430P/530 Internet, Web & Cloud Systems
# https://developers.google.com/youtube/v3/docs/search/listimport apiclient.discoveryAPI_KEY = os.environ.get("API_KEY")
# Build the request for YouTube search contentyoutube = apiclient.discovery.build("youtube",
"v3",developerKey=API_KEY)
search_response = youtube.search().list( q = "Grumpy Cat",part = "id, snippet",maxResults = 1, ).execute()
for search_result in search_response.get("items", []):if search_result["id"]["kind"] == "youtube#video":print("%s" % search_result["snippet"]["title"])print("%s" % search_result["id"]["videoId"])
![Page 27: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/27.jpg)
Knowledge Graph API via apiclient.discovery
Translation API via apiclient.discovery
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import apiclient.discovery, os
API_KEY = os.environ.get('API_KEY')
kgsearch = apiclient.discovery.build('kgsearch', 'v1',developerKey=API_KEY)
request = kgsearch.entities().search(query='Portland State University',limit=1)
# Returns a dict of search result inforesult = request.execute()
import apiclient.discovery, os
translate = apiclient.discovery.build('translate', 'v2',developerKey=os.environ.get("API_KEY"))
request = translate.translations().list(source = 'en',target = 'fr',q=['flower','car'] )
result = request.execute()
![Page 28: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/28.jpg)
#3: HTTP requests to REST API
![Page 29: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/29.jpg)
Language support for HTTP
Most languages with native support for sending HTTP requests and
parsing HTTP responses Typically, GETs and POSTs with JSON being returned
Packages are wrappers for the underlying REST API call over HTTP
Python Requests (My) preferred package for HTTP
Most examples will use this
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 30: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/30.jpg)
Example: Google Places API (REST via Python)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import requests
def post(self):""" Sends search box information to Google Places API and returns response """api_key = os.environ.get('API_KEY')keyword = request.form['search']
# Base URL for Google Places APIurl = 'https://maps.googleapis.com/maps/api/place/nearbysearch/json?'
# Lat/Long of Portland, ORlocation = 'location=45.5122,-122.6587&radius=1500'
# Places type variableplace = 'restaurant'
# API responseresponse = requests.get(url + location + '&type=' + place +
'&keyword=' + keyword + '&key=' + api_key)r = response.json()return render_template('home.html', response=r['results'])
![Page 31: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/31.jpg)
Example: Knowledge Graph API (REST via Python)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import requests
# Grab Knowledge Graph API Key from APIs & Services:Credentials and# set environment variableapi_key = os.environ.get('API_KEY')
term = 'Portland State University'
# Use term and api_key string to formulate a URL for hitting the# Knowledge Graph API endpoint. Note f-strings are Python 3.6url_string = f'https://kgsearch.googleapis.com/v1/entities:search?query={term}&key={api_key}&limit=1&indent=True'
# Take given response, parse it as a json object, and return a dictresponse = requests.get(url_string)
result_dict = response.json()
![Page 32: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/32.jpg)
Example: Yelp API
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import requestsdef yelpSearch(self, title):
# Craft Yelp's API endpoint URL, set location to Portland and limit resultsurl = 'https://api.yelp.com/v3/businesses/search?term=' + title +
'&location=portland&limit=3'
# Declare the api header with api key (Fill in your own key)headers={'Authorization': "Bearer YOUR_API_KEY"}
# Query the API endpoint, parse JSON response and return dictresponse = requests.get(url, headers=headers)return response.json()
![Page 33: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/33.jpg)
Side-by-side (Google Maps)
pip install googlemaps
pip install requests
Portland State University CS 430P/530 Internet, Web & Cloud Systems
# Requires enabling geocoding API and generation of an API key in Google Cloudimport googlemapsimport os
API_KEY = os.environ.get('API_KEY')gmaps = googlemaps.Client(key=API_KEY)boba_query = 'Mo Cha Tea House, Portland, OR'boba_shops = gmaps.geocode(boba_query)for shop in boba_shops:print(shop['formatted_address'])
import requestsimport jsonimport os
API_KEY = os.environ.get('API_KEY')boba_query = 'Mo Cha Tea House, Portland, OR'resp = requests.get( f'''https://maps.googleapis.com/maps/api/geocode/json?address={boba_query}&key={API_KEY}''')boba_shops = json.loads(resp.text)for shop in boba_shops['results']:
print(shop['formatted_address'])
![Page 34: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/34.jpg)
Side-by-side (Nutritionix)
pip install nutritionix nutritionix-client
pip install requests
Portland State University CS 430P/530 Internet, Web & Cloud Systems
from nutritionix import NutritionixClientrecipe = 'Pasta Salad'
# Get the credentials from environment variables (more secure than in code)APP_ID=os.environ.get('NIX_APP_ID')API_KEY=os.environ.get('NIX_API_KEY')
nut_info = NutritionixClient(application_id=APP_ID, api_key=API_KEY)result = nut_info.search(q=recipe, limit=3, offset=0)
def nutritionix(self, ingredient):url = 'https://trackapi.nutritionix.com/v2/natural/nutrients'# Declare the api header with api key (Fill in your own App ID and Key)headers = {"Content-Type":"application/json", "x-app-id":"YOUR_APP_ID",
"x-app-key":"YOUR_APP_KEY"}
# Declare POST query as a dictionary, set query field to ingredientbody = {"query":ingredient,"timezone": "US/Eastern"}
# Make request to API endpoint, parse JSON response and return dictresponse = requests.post(url, headers = headers, json = body)return response.json()
![Page 35: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/35.jpg)
AWS API Gateway
Building and deploying APIs
![Page 36: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/36.jpg)
Types of Web APIs
RESTful APIs (request-response) Online transactions
Weather application
Blogs, social media
Via HTTP methods PUT, GET, POST, DELETE, etc.
Take in and output HTML, XML, or JSON
WebSocket APIs (bi-directional) Chat applications
Real-time alerts and notifications
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 37: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/37.jpg)
API Gateway
Make it easy to scale and manage versions of an API Fully managed, serverless API platform
REST and WebSocket support
Lifecycle management support for APIs
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 38: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/38.jpg)
API Gateway overview
https://www.alexdebrie.com/posts/api-gateway-elements/
Portland State University CS 430P/530 Internet, Web & Cloud Systems
1. Authorization
and access
control
Routing to
endpoint,
access control
Packaging
request to API
logic in
integration
Transform response, format
HTTP response headers
2. Request flow
3. Response flow
![Page 39: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/39.jpg)
1. Authorization and access control
User access management to API IAM-based, resource-based policies
AWS Cognito
Custom (e.g. Authorizer built in a Lambda)
OAuth/SAML
User API key generation and management
Usage-plans for billing and throttling Per-API key metering
Usage per second or daily quota
Configuration of external site access via Cross-Origin Resource
Sharing (CORS) Restrict sites that can access API via cross-domain requests
TLS certificate management for endpoint
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 40: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/40.jpg)
2. Request Flow
Endpoint configuration, request routing based on HTTP method
Method request Validation of request parameters
Validation of request bodies Done via pre-defined models (data structure for specifying request and response
payloads in a JSON Schema format)
Integration request Transformation to format method request into backend integration
request (via request mapping templates)
Integration invocation to perform API logic
Integration Implementation of API logic (EC2, ECS, Lambda, etc)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 41: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/41.jpg)
Example model and mapping templates
Cost calculator request model that contains a price, size, and unit API Gateway ensures requests supply all parts of the model
Mapping templates Transform from one model to another
For requests, translate method request model into integration request
model for use by backend integration
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 42: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/42.jpg)
3. Response Flow
Transform integration responses and return values back into HTTP
response Done similarly to request flow via response mapping templates and
method response models
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 43: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/43.jpg)
API Gateway miscellany
Caching support (via CloudFront) Endpoint types
Edge (Latency optimized via deployment in CloudFront) Regional (For requests and responses residing in the same AWS region) Private (Exposed only to services and resources inside your VPC)
Stages Snapshots or versions of the API available for clients to call Create an API deployment and associate it with a stage (stage settings include
caching enabled, request throttling, logging config) SDK generation
Automatically generate language SDKs (Java, JavaScript, Objective-C, Swift, and Ruby)
CloudWatch monitoring built-in (metrics collected per minute) Count Integration latency CacheHitCount CacheMissCount HTTP 400 and 500 errors
Portland State University CS 430P/530 Internet, Web & Cloud Systems
![Page 44: CS 430P/530: Internet, Web, & Cloud Systems](https://reader033.fdocuments.in/reader033/viewer/2022052722/628ee0c4ec776203633b2eee/html5/thumbnails/44.jpg)
Labs
Leverage APIs to build applications (Slack/Knowledge Graph)
Manually deploy APIs (Guestbook) A lot of work!
Next week IAC + API deployment
ML APIs
Backend as a Service APIs
Portland State University CS 430P/530 Internet, Web & Cloud Systems