RESTFul API DesignPrinciples, Guidelines and Best Practices

Who am IChristian Günther

Solution Architect

SAP / Non-SAP

Cloud Computing

NoSQL and Open Source evangelist

Twitter: @siliconchris

The storyline behindThe company Microdolls Inc. produces miniature action figures.

The company runs a website with a web-shop, a doll-designer and discussion boards.

The website is quite old and they would like to redesign it.

They would also like to create a mobile version of the shop and the doll designer.

microdolls.com

Where imagination meets action

Current situationThe website and all offerings (such as the shopping cart, the doll designer and the discussion board) is written in PHP

The backend is a MySQL database

Website and DB are directly coupled using embedded sql-statements in the php-code

The designer consists of php-code that executes scripted ImageMagick functions on the OS level to manipulate images and return the new image to the browser

All in all the technical implementation is quite old, but it is not unlike many other sites with a shop one can find on the internet

Current interaction modelCustomers of microdolls.com open their browser and enter the shop’s url The web server takes the browsers request (this of course being GET www.microdolls.com/shop/index.php), possibly executes some php code and returns plain html The user’s browser then interpreted and renders the html code and presents the visual representation of the site Any element, which is not directly included in the html payload (such as images), are retrieved through subsequent GET-requests with a URL pointing to the image location on the server.

Issues with the current setupThere is no real separation (be it layered or otherwise) between the different technical elements such as presentation, application and persistence - everything is very tightly coupled.

This programming paradigm and the underlying architecture makes it extremely difficult and time consuming to introduce new functionality.

Reuse of code is mainly done by duplicating needed functionality from one page in another - that is to say, it is not really reused, but rather used again.

Changes to any element (be it shop, board, designer or database) need to be thoroughly tested throughout the whole website and all areas.

To enter the mobile market, the company would need to re-create the whole website with all functionality and would also need to implement new abstraction layers.

The conclusionInstead of simply trying to recreate the website on mobile devices, the company decides to redesign their infrastructure and programming model.

In essence, tear down the monolithic php site and establish a micro services architecture

SOAP? REST? THRIFT?Like our fictional company microdolls.com, a lot of companies have designed their applications in a very monolithic way (in the past, one has to say).

A monolith is an application which provides all of its services through a giant, singular application. Due to the drawbacks of such an approach, the modern API development architecture has been steadily moving towards micro services.

Unlike monolithic systems, Microservices deliver their functionality split between multiple, separate applications and processes, and provide their functionality across multiple servers when needed.

As Microservice development has grown, so has the need for more diverse and extensible architecture designs.

There are basically two approaches (technologies) on how to create an API for a micro services based architecture and one that is rather new: SOAP, REST and Apache THRIFT

SOAP - Simple Object Access ProtocolSOAP is an architectural concept that was created in 1998 by Dave Winer, Don Box, and Bob Atkinson, in collaboration with Microsoft.

Designed explicitly for the purpose of making communication between web services easier, SOAP has four main characteristics that made it so popular in its infancy:

Extensibility through user generated extensions tying into the base structure;

Neutrality by operating over any transport protocol;

Independence by allowing variable programming paradigms and structures;

Large Data Handling through asynchron calls to conversions, calculations, etc.

REST - Representational State TransferREST was created in 2000 by Roy Fielding in UC, Irvine; later versions of this architecture were created in collaboration with the W3C Technical Architecture Group.

While SOAP aimed to be a complete system, REST was designed to be more lightweight for building the following right into the design;

Scalability by using cached data from the client and intermediate nodes built into HTTP to self-define;

Portability by tying the transfer of data to the program code during transfer;

Extensibility by allowing individual elements of the greater network to develop independent of one another, using uniform interfaces.

Apache THRIFTThrift, in 2007 designed by Facebook, was soon released as an open source project under the Apache Software Foundation label.

The architecture, designed specifically to compete with SOAP and REST while delivering data quickly in a number of formats, includes a complete stack for the manufacture, maintenance, and expansion of clients and servers.

The system was designed specifically to include:

Scalability by supporting cross-language services seamlessly between C#, C++, Cappuccino, Cocoa, Haskell, and more

Simplicity by eschewing the frameworks of REST and the XML of SOAP in favor of a simple library

Speed by utilizing binary serialization to handle data

Evolution by allowing for soft versioning, supporting third party teams to develop RPC calls as needed

Text

REST in Peace SOAPREST vs SOAP: Simplicity wins, again

REST vs SOAPREST is not always better than SOAP

There are circumstances, where a SOAP-based approach could be preferable over REST

Those situations, however, are scarce to find

Eg, if you need highly coupled applications, build by your own or your co-team, you could use SOAP, but

in general, if you build an API that is to be consumed by clients outside your company or application domain, NEVER use SOAP

What is REST?

REST = Representational State Transfer is an architectural style and programming paradigm for distributed systems, especially (but not necessarily only) those found in applications running on the world wide web

REST gives a coordinated set of constraints to the design of components in a distributed hypermedia system that can lead to a higher-performing and more maintainable architecture

To the extent that systems conform to the constraints of REST they can be called RESTful.

What is REST?

RESTful systems typically, but not always, communicate over HTTP with the same HTTP verbs (this being GET, POST, PUT and DELETE) used by a web browser that access a web page

REST interfaces with external systems using resources identified by a URI which can be operated upon using the named standard verbs

REST demands that the response to a resource-request must always return the same content regardless how many times the URI was accessed

The idea behind RESTWith REST the idea is that, rather than using complex mechanisms such as CORBA, RPC or SOAP to connect between clients and servers, simple HTTP queries are used. 

RESTful applications use HTTP requests to POST data (create), PUT data (update), GET data (make queries), and DELETE, to delete data on the server.

Thus, REST uses HTTP for all four CRUD (Create/Read/Update/Delete) operations

The idea behind RESTFor example, if a client application wants to get information on the action figure Neo, it sends our API server an HTTP GET with the URL:

Using jQuery we can implement this AJAX request in our client application simply like this:

The HTTP reply our server sends back to the client is the raw result data — not embedded inside an HTML page, not XML-encoded, just the data you need in a way you can immediately use – a String or a JSON object you can use directly.

$.getJSON("http://microdolls.com/dolls/neo", function(data) { console.log("The Action figure neo fancies "+data );

});

http://microdolls.com/dolls/neo

The idea behind RESTREST can easily handle more complex requests, including multiple parameters. In most cases, you’ll just use HTTP GET parameters in the URL.

For example, if a specific doll can only uniquely specified by both template and sex, our API might accept a request like this:

As a convention (although I’d rather see this as a strict guideline), HTTP GET requests should be for read-only queries; GET should not change the state of the server and its data. For creation, updating, and deleting data, use POST, PUT and DELETE requests.

http://microdolls/dolls/?template=matrix&sex=male

WHAT is a RESTFul API?Short Answer: „A RESTFul API is one that exposes it’s services in a way that complies to the principles of REST“

WHAT is a RESTFul API?The long answer is this: A RESTFul API is one, that can be consumed by a system, in the same intuitive way a user consumes and uses a website.

A user does not need to know the structure of a website prior using it. He/She will learn about the site’s structure along the way, simply by following presented navigation links (URLs)

A RESTFul API provides the same smooth consumption pattern. A system connecting to your API the first time, will learn about the internal structure of the API by navigating the API’s response

RESTFul APIGuidelines on RESTFul API Design

RESTFul API DesignHTTP is your driver GET never changes an objects state Use nouns instead of verbs

Always use plural Use sub-locations for relations Use grouping, if applicable HTTP headers contain the serialization format Use HATEOAS

Provide operations for collections Version your API Transport Errors as HTTP Return Codes Allow overriding of HTTP methods

HTTP is your driverAlthough this should be rather obvious:

A RESTFul API relies entirely on HTTP

All operations on exposed objects are done through well defined methods

GET - retrieves an object, a collection or displays specific data

POST - creates an object

PUT - updates an object (or a predefined collection of objects)

DELETE - hm? any guess?

GET never changes stateMore precisely GET method and query parameters should not alter the state of an object or data

Use PUT, POST and DELETE methods instead of the GET method to alter the state.

Do not use GET for state changes:

GET /users/711?activate

or

GET /users/711/activate Nooooooooh!!!

Use nouns instead of verbsFor easy readability nouns are always preferable over verbs.

After all, your API will expose an object as first citizen.

The methods to alter or work with your object are of second concern - the are implemented using standard HTTP methods.

Therefore never use anything like these:

/getAllUser

but always use the simple

/users

along with a GET/POST/PUT/DELETE HTTP Method

Use plural nounsAs well for easy readability always use plurals

Do not mix up singular and plural nouns. Keep it simple and use only plural nouns for all resources.

/cars instead of /car

/users instead of /user

/products instead of /product

/settings instead of /setting

Use sub-locations for relationsIf a resource is related to another resource use sub-resources.

GET /cars/711/drivers/

Returns a list of drivers for car 711

GET /cars/711/drivers/4

Returns driver #4 for car 711

Use grouping, if applicable

Identify common or shared properties:

Is there something you can group in a domain specific context? Something that is shared by most or at least a majority of objects/services?

Create an abstract class (if we talk of objects) or a group (if we talk of a service) and apply the shared property to it - then derive your class from the abstract class or put your services in that group, respectively.

HTTP headers contain the serialization format

Both, client and server, need to know which format is used for the communication.

The format has to be specified in the HTTP-Header.

Content-Type defines the request format.

Accept defines a list of acceptable response formats.

Use HATEOAS

HATEOAS = Hypermedia As The Engine Of Application State is a principle that hypertext links should be used to create a better navigation through the API.

In essence that means, in an API designed after the HATEOAS principle, a client will be able to navigate through the API by following URLs and links provided by the server - this is much the same, as a user navigating through a website by following links on that site.

As this is a bit of a complicated topic, a more in deep description is provided within this presentation

Provide operations for collections

Filtering - filter a return list

Sorting - sort a collection

Field Selection - select fields

Paging - display a limited set of resources per query

FilteringUse a unique query parameter for all fields or a query language for filtering.

To return a list of alien action figures

GET /dolls?family=alien

To return a list of action figures with a maximum of 2 heads

GET /dolls?heads<=2

Sorting

Allow ascending and descending sorting over multiple fields.

To return a list of action figures sorted by descending manufacturers and ascending models.

GET /dolls?sort=-manufactorer,+model

Field Selection

Mobile clients display just a few attributes in a list.

They don’t need all attributes of a resource.

Give the API consumer the ability to choose returned fields.

This will also reduce the network traffic and speed up the usage of the API.

GET /dolls?fields=manufacturer,model,id,family

Paging

Use limit and offset. It is flexible for the user and common in leading databases. The default should be limit=20 and offset=0

GET /dolls?offset=10&limit=5

To send the total entries back to the user use the custom HTTP header: X-Total-Count.

Links to the next or previous page should be provided in the HTTP header link as well.

It is important to follow this link header values instead of constructing your own URLs. <https://blog.microdolls.com/catalog/api/v1/dolls?offset=15&limit=5>; rel=„next",

<https://blog.microdolls.com/catalog/api/v1/dolls?offset=50&limit=3>; rel=„last",

<https://blog.microdolls.com/catalog/api/v1/dolls?offset=0&limit=5>; rel=„first",

<https://blog.microdolls.com/catalog/api/v1/dolls?offset=5&limit=5>; rel="prev",

Version your API

Make the API Version mandatory and do not release an unversioned API.

Decide on your API versioning scheme - simple ordinals or year plus month or the like. It doesn’t really matter, but stick to it for all your APIs within a domain and/or application system

If using numbers, use simple ordinal number and avoid dot notation such as 2.5.

Finally it is common sense to prefix the url for the API versioning with the letter „v“

/catalog/api/v1

Transport errors as HTTP status codes

It is hard to work with an API that ignores error handling.

Pure returning of a HTTP 500 with a stacktrace is not very helpful.

Also, inventing new meanings to established http status codes makes using your API a nightmare on elm street

Instead use standard HTTP status codes

Use HTTP status codesThe HTTP standard provides over 70 status codes to describe the return values.

Use these 4 codes for successful requests

200 – OK – Here is your result

201 – OK – New resource was created / resource was updated

204 – OK – The resource was successfully deleted

304 – Not Modified – The client can use cached data

HTTP error status codesUse these 5 codes for failed requests

400 – Bad Request – The request was invalid or cannot be served. The exact error should be explained in the error payload. E.g. „The JSON is not valid“

401 – Unauthorized – The request requires an user authentication

403 – Forbidden – The server understood the request, but is refusing it or the access is not allowed.

404 – Not found – There is no resource behind the URI.

422 – Unprocessable Entity – Should be used if the server cannot process the enitity, e.g. if an image cannot be formatted or mandatory fields are missing in the payload.

HTTP 500

API developers should avoid this error in normal situation and use specific error codes (as shown in the previous slide).

If an error occurs in the global catch blog (bad luck), the stracktrace should be logged and not returned as response.

500 – Internal Server Error

Use error payloadsAll exceptions should be mapped in an error payload.

Here is an example how a JSON payload should look like.

04 "userMessage": "Sorry, the requested resource does not exist“,

05 "internalMessage": "No doll found in the database",

Allow overriding HTTP methods

Some proxies support only POST and GET methods.

To support a RESTful API with these limitations, the API needs a way to override the HTTP method.

Use the custom HTTP Header X-HTTP-Method-Override to override the POST Method.