Central Person Registry ITS ITANA Architecture Review 10 November 2010.
ITANA 2016: API Architecture and Implementation
-
Upload
colin-bell -
Category
Technology
-
view
120 -
download
0
Transcript of ITANA 2016: API Architecture and Implementation
![Page 1: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/1.jpg)
ITANA API / Governance Working Group
U N I V E R S I T Y O F W AT E R L O O
API ARCHITECTURE AND IMPLEMENTATIONP O W E R I N G S T U D E N T A N D I N S T I T U T I O N A L I N N O VAT I O N
Colin BellDirector Enterprise ArchitectureEA, ISTBMath2008uwat.ca/ist-ea
Zachary SeguinOpen Data API Incoming Dev Lead Client Services, ISTBCS Computer Science2017uwaterloo.ca/open-data
Kartik TalwarOpen Data API Outgoing Dev Lead Client Services, ISTBSc Physics2016uwaterloo.ca/open-data
Nate VexlerOpen Data API Service LeadClient Services, ISTBASc Systems Design Eng.2012uwaterloo.ca/open-data
![Page 2: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/2.jpg)
ITANA API / Governance Working Group
AGENDA
Background1. Prime Motivators2. Routine Disclosure3. Why Open Data First?4. 5 Star Data5. Open Data License6. History of Open Data
Technical Specifics7. Data Sets8. Data Acquisition9. Technology
![Page 3: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/3.jpg)
ITANA API / Governance Working Group
AGENDA
Growing1. Governance and Policy2. Private Data (PbD)3. Business Process4. Data Warehousing/ESB/iPaaS5. Future Directions
Questions6. Buy vs. Build?7. How to build today?8. Documentation?9. How do you deal with the ERP
culture clash?
![Page 4: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/4.jpg)
ITANA API / Governance Working Group
BACKGROUND
![Page 5: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/5.jpg)
ITANA API / Governance Working Group
ORIGINAL PRIME MOTIVATORS
• Enable Student Development• Enable Homebrew Student Portal
![Page 6: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/6.jpg)
ITANA API / Governance Working Group
PROTECTING PRIVACY
http://eaves.ca/2013/01/07/the-journal-news-gun-map-open-vs-personal-data/
![Page 7: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/7.jpg)
ITANA API / Governance Working Group
MAINTAINING TRANSPARENCYPROTECTING PRIVACY
bit.ly/ipc_abd bit.ly/ipc_pbd
![Page 8: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/8.jpg)
ITANA API / Governance Working Group
ROUTINE DISCLOSURE• Pol icy of Ontar io’s
Information Privacy Commissioner
• 7 pr inciples• bit . ly / ipc_abd
![Page 9: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/9.jpg)
ITANA API / Governance Working Group
ROUTINE DISCLOSURE1. P r o a c t i v e , n o t R e a c t i v e2. A c c e s s E m b e d d e d i n t o D e s i g n3. O p e n n e s s a n d Tr a n s p a r e n c y =
A c c o u n t a b i l i t y4. F o s t e r s C o l l a b o r a t i o n5. E n h a n c e s E f f i c i e n t G o v e r n m e n t6. M a k e s A c c e s s Tr u l y A c c e s s i b l e7. I n c r e a s e s Q u a l i t y o f I n f o r m a t i o n
![Page 10: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/10.jpg)
ITANA API / Governance Working Group
WHY OPEN DATA FIRST?
Personally Identifiable Data
![Page 11: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/11.jpg)
ITANA API / Governance Working Groupbit.ly/ipc_abd
WHY OPEN DATA FIRST?
![Page 12: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/12.jpg)
ITANA API / Governance Working Group
Emerging trend in Government (Toronto, Vancouver)
New Vertical (Higher Ed)
WHY OPEN DATA FIRST?
![Page 13: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/13.jpg)
Open Data License
![Page 14: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/14.jpg)
ITANA API / Governance Working Group
OPEN DATA
5stardata.info
![Page 15: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/15.jpg)
ITANA API / Governance Working Group
IMPROVING OPEN DATA
inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/
![Page 16: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/16.jpg)
ITANA API / Governance Working Group
WHY IS OPEN DATA IMPORTANT
• Economies of Scale • Value of Data • Cost of Using Data
![Page 17: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/17.jpg)
ITANA API / Governance Working Group
WHY IS OPEN DATA IMPORTANT
DO MORE W/ LESS
![Page 18: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/18.jpg)
Fall 2009• Nathan lobbies with
others at Student Technology Advisory Committee re Portal/Open Data
March 2010• Presentation at
High Level Computing Committees UCIST/CTSC
Fall 2010• Jeff Verkoeyen’s
uwdata.ca becomes official
Aug 2011• Kartik Talwar
creates api.youwaterloo.ca
Feb 2012• api.uwaterloo.ca
becomes official
Open Data @ uwaterloo Timeline
![Page 19: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/19.jpg)
2012• Nathan Joins IST full time• Open-Data powered
apps:• - Student Portal (internal)
• - uwflow.com (external)
2013• V2 launches• Drupal-Powered Content
Management System is leveraged to deliver Open Data for use in Student Portal
2014• Open Data-powered
Student Portal launches;• mandate to bring
api.uwaterloo.ca deeper into the enterprise
2015• Campus Map Project
(powered by Open Data) is in development
• Student Portal project continues to further the momentum of Open Data
Open Data @ uwaterloo Timeline
![Page 20: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/20.jpg)
ITANA API / Governance Working Group
IPC CORE DESIGN PRINCIPLES
bit.ly/ipc_abd bit.ly/ipc_pbd
![Page 21: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/21.jpg)
Student PortalAbD/PbD Strategy(Colin Bell)
![Page 22: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/22.jpg)
ITANA API / Governance Working Group
IMPROVING OPEN DATA
inkdroid.org/journal/2010/06/04/the-5-stars-of-open-linked-data/
![Page 23: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/23.jpg)
ITANA API / Governance Working Group
API: APPLICATION PROGRAMMING INTERFACE
![Page 24: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/24.jpg)
ITANA API / Governance Working Group
API: APPLICATION PROGRAMMING INTERFACE
![Page 25: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/25.jpg)
![Page 26: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/26.jpg)
CORE AREAS OF SERVICE OF CURRENT API
Open Data API
Student Engagement
(Portal)
Core IT Service (WCMS)
For Students by Students
Emerging Academic
Use
![Page 27: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/27.jpg)
uWaterlooGitHub
github.com/uwaterloo
![Page 28: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/28.jpg)
IRL Community with
DemoCamps
![Page 30: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/30.jpg)
Community Created Toolsuwaterloo.xyz
![Page 31: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/31.jpg)
ITANA API / Governance Working Group
APPLIED EXAMPLE: RESPONDING TO COMMUNITY NEED
GOOSE WATCH
BACK to April 9, 2013
![Page 32: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/32.jpg)
ITANA API / Governance Working Group
Three Days PRIORAppeal For Data
THE STUDENT SUCCESS OFFICE SWINGS IN ACTION TO PROTECT STUDENTS
Three Days PRIOR Sketch/Prototype
![Page 33: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/33.jpg)
ITANA API / Governance Working Group
Mid Afternoon the Day BeforeThe endpoint launches
THE UWATERLOO COMMUNITY S W I N G S I N TO A C T I O N TO R E Q U E S T / D E L I V E R D ATA
Two Days PRIORThe reddit Data Request
![Page 34: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/34.jpg)
ITANA API / Governance Working Group
THE RESPONSE
![Page 35: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/35.jpg)
ITANA API / Governance Working Group
PROGRESS!
5stardata.info
API
PNG
![Page 36: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/36.jpg)
ITANA API / Governance Working Group
TECHNICAL SPECIFICS
![Page 37: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/37.jpg)
ITANA API / Governance Working Group
DATASETS
Food Services
News and Events CO-OP WATpark
New Campus Map Tutors Services Awards
![Page 38: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/38.jpg)
ITANA API / Governance Working Group
Food ServicesLocations and Hours
Daily menuNutritional Information
![Page 39: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/39.jpg)
ITANA API / Governance Working Group
CO-OP DataEmployer Info-Sessions
Career Centre Workshops
![Page 40: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/40.jpg)
ITANA API / Governance Working Group
TutorsFind a Tutor for a courseSee their availability and
get help
![Page 41: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/41.jpg)
ITANA API / Governance Working Group
WATParkSee live parking lot capacity
Get updates on opening and closing hours
![Page 42: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/42.jpg)
ITANA API / Governance Working Group
GithubStudents Contributing Data
Students submitting datasets
![Page 43: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/43.jpg)
ITANA API / Governance Working Group
New Campus Map
![Page 44: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/44.jpg)
ITANA API / Governance Working Group
CURRENT STATE: DATA ACQUISITION
ScrapersGithub• Crowdsourced• Maintained
CMS (Drupal)• Proxy requests• Pub/sub
Other Proxy Requests
Direct Database
Integrations
![Page 45: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/45.jpg)
ITANA API / Governance Working Group
FUTURE STATE: DATA ACQUISITION
ScrapersGithub• Crowdsourced• Maintained
CMS (Drupal)• Proxy requests• Pub/sub
Other Proxy Requests
Direct Database
Integrations
WarehousingESB / iPaaS
![Page 46: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/46.jpg)
ITANA API / Governance Working Group
CURRENT STATE: DATASETS
Food ServicesDB
News/Events/etcPub Sub
CO-OP WATparkProxy
New Campus Map
Curated GH
TutorsProxy
AwardsProxy
![Page 47: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/47.jpg)
ITANA API / Governance Working Group
TECHNICAL ARCHITECTURE
• Student initiated, no fancy ($) tech involved.
• After a number of iterative improvements, this is our stack.
![Page 48: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/48.jpg)
api.uwaterloo.ca
App Server 1 App Server 2
Incoming Webhooks
Workers
Other Databases
Proxy ServicesAPI DB
![Page 49: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/49.jpg)
App Server
Incoming Webhooks Proxy Services Workers
Github sync, scrapers, updaters
Live requests from other internal APIs
New posts from university websites
API DB Other DBs
![Page 50: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/50.jpg)
App Server
API Request Router (PHP Klein)
Tutoring Services Food Services
FS DB API DB
GET METHOD 1GET METHOD 2GET METHOD 3
Serve from FS DB
Campus Events
GET METHOD 1GET METHOD 2GET METHOD 3
Proxied via internal tutoring API
Tutoring API (other department)
Crons
- sync datasets- run scrapers- cache things
Keep data on file system
GET Serve from DB
POST (webhook) Process and update DB
![Page 51: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/51.jpg)
ITANA API / Governance Working Group
DOCUMENTATION / ISSUE TRACKING
• Docs: Github w/ Markdown generator• Issue Tracking
Community Issues: GithubService Desk: RTInternal: Gitlab/Jira
![Page 52: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/52.jpg)
ITANA API / Governance Working Group
GROWING API CAPABILITY
![Page 53: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/53.jpg)
ITANA API / Governance Working Group
GOVERNANCE AND POLICY
• Administrative Information Governance Committee (AIGC)
Vice Presidents, Senior Management, and Associate Provosts2015 decision to consolidate IM related Policies.Procedures and Guidelines to follow Policy. Draft of Policy being moved through review now.
![Page 54: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/54.jpg)
ITANA API / Governance Working Group
PRIVATE DATA
• We have few private data APIs on campus. Faculty developed advising system provides student information backbone.
• They are starting to grow. Principle established prioritizing API integration pathways.
• Real uptake depends on Identity and Access Management implementations.
Shibboleth, SAML, OAUTH2 to come 2016/2017.
![Page 55: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/55.jpg)
ITANA API / Governance Working Group
BUSINESS PROCESS
• Working to change the relationships.Initially we leaned on AbD, many units felt ’bad things’ were happening.Proven that the sky has not fallen.Developing a Steering Committee model to help direct future development merging community needs with institutional goals.Develop SLA with Information Stewards.
![Page 56: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/56.jpg)
ITANA API / Governance Working Group
DATA WAREHOUSING / ENTERPRISE SERVICE BUS / IPAAS• Integrations on campus have been file or DB based.
Times have changed, we are starting to change.• Open Data has provided us with knowledge and
expertise in APIs.• As warehousing improves and we bring either an
ESB or iPaaS solution on campus, our data architecture will start to take shape.
![Page 57: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/57.jpg)
ITANA API / Governance Working Group
DATA WAREHOUSING / ENTERPRISE SERVICE BUS / IPAAS
Data Warehousing
Integration Engine (ESB/iPaaS)
HR
Finance
SIS
Co-Op
…
OR
AbD(Open)
PbD(Priv)
API
![Page 58: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/58.jpg)
ITANA API / Governance Working Group
FUTURE DIRECTIONS
• Composable Microservices• Create Once, Publish Everywhere (COPE) -
NPR Model
![Page 59: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/59.jpg)
University of Waterloo WCMS Create Once Publish Everyone COPE Strategy(Andrew McAlorum)
News and Events
![Page 60: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/60.jpg)
GROWING AREAS OF SERVICE
Open Data API
Student Engagement
(Portal)
Core IT Service (CMS + more)
For Students by Students
Emerging Academic
Use
Campus Map
Research
![Page 61: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/61.jpg)
ITANA API / Governance Working Group
THE OPEN DATA ENGINE
(Demand-First)• Exam Schedule• CO-OP workshops• Tutors• Food Services
(Supply-First)• Services• News Events• Awards• Publication
![Page 62: ITANA 2016: API Architecture and Implementation](https://reader035.fdocuments.in/reader035/viewer/2022062523/5884855c1a28ab6d1a8b5001/html5/thumbnails/62.jpg)
ITANA API / Governance Working Group
[email protected]@[email protected]@uwaterloo.ca
Buy vs. Build?How to build today?
How do you Document?How to mitigate ERP culture clash?
QUESTIONS?