Copyright © 2019 Oracle and/or its affiliates.

Delivering Developer Tools at Scale:

Microsoft Azure & Oracle Cloud

Perspectives

Senior Software Dev Manager

Oracle Cloud Infrastructure

Twitter: @jodoglevy

Joe Levy

Program Manager

Microsoft Azure

Mohammed Zehgir

Setting the Context: Microsoft Azure / Oracle Cloud

Collection of IaaS, PaaS, SaaS services Compute, Virtual Network, Object Storage services, etc

Collection of clients for those services GUIs (in-browser, phone / computer apps, etc)

SDKs (Java, Python, Go, JavaScript, etc)

CLIs (Bash, PowerShell, etc)

Custom clients written by users

Setting the Context: Our Responsibility

Collection of IaaS, PaaS, SaaS services Compute, Virtual Network, Object Storage services, etc

Collection of clients for those services GUIs (in-browser, phone / computer apps, etc)

SDKs (Java, Python, Go, JavaScript, etc)

CLIs (Bash, PowerShell, etc)

Custom clients written by users

The Problem

March 2015

2018

The Problem

Compute Block Storage

Virtual Networking

Load Balancing Object Storage Audit Database File Storage …

Java SDK

Python SDK

Ruby SDK

Go SDK

C# SDK

JavaScript SDK

…

The Problem

Boxes: N dev tools X M services

Team: 8 devs

The Problem

Boxes: N dev tools X M services X Z features X Y pace of innovation X W release scopes

Team: 8 devs

The Problem

One developer experience team

Responsible for SDKs, CLIs, dev docs

Many services

Many dev tools & programming languages customers want to use

How to handle delivery of many SDKs, CLIs, other dev tools, and dev docs for all these services, quickly and with high quality?

The Solution: OpenAPI

Declaratively describe REST APIs in a machine-readable way

Manage your services’ interface specifications as you would your code

Open Source

Useful for: Generating REST API documentation

Code reviewing REST APIs (to ensure good REST use, and consistency across APIs)

Creating automated pipelines around service specifications

Service (scaffolding) generation

Client generation

OpenAPI spec: Launch instance

/instances/: post: description: | Creates a new instance in the specified compartment and the specified Availability Domain. <shortened for brevity> operationId: LaunchInstance parameters: - description: Instance details in: body name: LaunchInstanceDetails required: true schema: $ref: '#/definitions/LaunchInstanceDetails' responses: 200: description: The instance is being launched. headers: etag: $ref: '#/x-descriptions/etag' schema: $ref: '#/definitions/Instance'

LaunchInstanceDetails: properties: availabilityDomain: type: string compartmentId: type: string displayName: type: string hostnameLabel: type: string imageId: type: string ipxeScript: type: string metadata: additionalProperties: type: string type: object shape: type: string required: - compartmentId - shape - availabilityDomain

Results when generated

• REST API docs

• Java SDK

• Python SDK

• Ruby SDK

• Go SDK

• CLI: oci compute instance launch

Standardize to Survive: OpenAPI

Q: How to scale development of 10+ clients X 50+ services?

A: Not manually

Q: How to ensure service APIs follow consistency guidelines easily?

A: OpenAPI makes API design reviews easy

Q: How to write bug-free clients w/ consistent interfaces quickly?

A: OpenAPI to generate SDK / CLI clients

Q: How to write consistent HTTP API docs effortlessly?

A: OpenAPI to generate API docs

Beyond OpenAPI

Domain Expressiveness

What if standard OpenAPI fields aren’t enough?

Scaling Processes

How do orgs with 1K+ developers use OpenAPI without tripping?

Validation / Testing

How do we scale manual reviews of OpenAPI specs?

How do we know services match their specs?

How do we know the generated code actually works?

Domain Expressiveness: How Azure solves it

OpenAPI Extensions

x-ms-pageable

x-ms-client-flatten

x-ms-enum

AutoRest Code Generator

Idiomatic, Fluent, OSS

Azure and Generic generators

Generators can be written in any language

Domain Expressiveness: x-ms-pageable extension

Paging in Azure services has more

than one pattern

Enables description of service

paging behavior

Domain Expressiveness: x-ms-pageable extension

Before:

After:

Domain Expressiveness: x-ms-client-flatten

OpenAPI Spec: Wire Format (json):

Domain Expressiveness: x-ms-client-flatten

After: Before:

Domain Expressiveness: How Oracle Cloud solves it

Custom fields in OpenAPI (x-…) x-obmcs-signing-strategy

x-obmcs-feature-id

x-obmcs-owner-info

Forked OpenAPI parser / generator

Graceful fallback if using standard OpenAPI tools

Domain Expressiveness: x-obmcs-ownership-info

OpenAPI spec snippet: Test snippet:

Domain Expressiveness: x-obmcs-feature-id

OpenAPI spec snippet: Process snippet:

Client

docs

Client

tests vs

spec

Client

samples

HTTP

API

docs

Client

docs

gen

Client

tools

Scaling Process: How Oracle Cloud solves it Service teams responsible for:

Creating / updating OpenAPI spec for their service

Ensuring service matches its spec

Writing samples of service operations in client tools

Maintaining service operations in client tools

Triaging bugs, fixing issues in service or spec, replying to customer issues, requests

API Review Board responsible for:

Reviewing OpenAPI specs for ease of use / consistency

Maintaining automated review tool & review pipeline

SDK / CLI team responsible for:

Maintaining client codegen / testgen platforms and associated pipeline

Ensuring client tools work against services which match each spec

Generating client docs

Implementing higher-level & non-generatable functionality in clients

Releasing the client tools publicly

Docs team responsible for:

Generating HTTP API docs and maintaining docs pipeline

Client

testgen

Client

codgen

HTTP

API

docs

gen

Manual

review

Auto

review

Open

API spec

Service

tests vs

spec

Scaling Process: How Azure solves it

API Review Board Responsible for enforcing Microsoft API design guidelines

https://github.com/Microsoft/api-guidelines

Service Teams Responsible for creation and maintenance of their specifications

https://github.com/Azure/azure-rest-api-specs

Developer Experience Team Code generation / validation / publication

https://github.com/Azure/autorest

Documentation Team Documentation generation pipeline

Verification / Testing: How Azure solves it

Ensure OpenAPI high quality specifications Syntax Validation

Linting

Breaking Changes

Ensure generated code works Matrix of Language x Features

AutoRest test server and verification by language

Ensure specifications match the service Live Traffic Validation

Verification / Testing: How Azure solves it w/ GitHub Checks

Verification / Testing: How Oracle Cloud solves it

Ensuring OpenAPI specs follow rules

Manual API Review

Spec Validator tool

Ensuring service implementation matches OpenAPI spec

Service teams’ responsibility, using either: Dredd

Generating service interfaces from spec

Writing service tests using any of the generated SDKs

Ensuring generated client code works against service

Client test generation

Generated tests run against “mock service”

Mock service takes in and responds to HTTP requests such

that it matches the OpenAPI spec

OpenAPI

spec

Service

Tests service

matches spec

Tests clients work

against a service

matching spec

Client tools

Verification / Testing: How Oracle Cloud solves it

Ensuring OpenAPI specs follow rules

Manual API Review

Spec Validator tool

Ensuring service implementation matches OpenAPI spec

Service teams’ responsibility, using either: Dredd

Generating service interfaces from spec

Writing service tests using any of the generated SDKs

Ensuring generated client code works against service

Client test generation

Generated tests run against “mock service”

Mock service takes in and responds to HTTP requests such

that it matches the OpenAPI spec

OpenAPI

spec

Service

Tests service

matches spec

Tests clients work

against a service

matching spec

Client tools

Verification / Testing: How Oracle Cloud solves it

Why generate tests? Avoid rewriting the same test logic many times in different languages

Requires lots of time and polyglot expertise

Avoid human error Bugs in the test itself

Inconsistent test bar

Why test against mock service instead of real service? Limitations of real service

Has dependencies before operations can be run

Has restrictions on which params can be used together & param values

Has resource limits, rate limits, limits on parallelization

Takes time to spin up / tear down resources

Could have bugs, reliability issues (in development)

Azure All Up Process Flow

OpenAPI Spec

Pull Request

• Linting

• Breaking Changes

API Design Review

• Behavior

Generation

• Documentation

• Code

Release Automation

Live HTTP Traffic Validation

Oracle Cloud All Up Process Flow

Update OpenAPI

Spec

Service teams API Review Board Docs team SDK / CLI team

Review OpenAPI

Spec changes

(manual)

Review OpenAPI

Spec changes

(manual)

Review OpenAPI

Spec changes

(auto tooling)

Generate client

code

Test service

implementation

(vs spec)

Create client

samples

Review samples

& client test

failures

Generate HTTP

API docs

Review HTTP API

docs

Release updated

clients

Release updated

HTTP API docs

Spec Validator Docs Generator

Code Generator

Generate client

tests (vs spec)

Test Generator

Mock Service

Thank You

Copyright © 2019 Oracle and/or its affiliates.

Senior Software Dev

Manager

Oracle Cloud Infrastructure

Twitter: @jodoglevy

Joe Levy

Program Manager

Microsoft Azure

Mohammed Zehgir