ServersServers

Servers allow web sites to do things, not just displaythings

choices test server URLs routes

1

ChoicesChoicesFor a dynamic web site, you have to choose a webserver, with a framework and a scripting ordevelopment language for writing the pages:

CGI Perl or C (obsolete)

ASPVBScript orC#

(proprietary/1-platform/obsolete)

JSP Java (clunky)LAMP PHPDjango PythonRails RubyNode JavaScript

2

Two choicesTwo choicesFor this course, two choices are recommended, and willbe covered in this chapter:

RawRaw Node, with standard libraries only

ExpressExpress Node, with express and/or other libraries

Take the raw approach to learn more, take the expressapproach to get further

3

Development serverDevelopment serverIf you are using Node (raw or express), the developmentserver server.js provided at the start of the coursecontains some potentially useful details

But remember that it was designed for development andnot production:

It only allows connections from the same computer, andits security is otherwise weak and, in particular it has aserious security vulnerability

4

Preliminary securityPreliminary securitySecurity is essential for a potentially public-accessservice - we will come back to it several times

Step one is to have a dedicated computer for a web site,and put it somewhere physically secure, e.g. a lockedmachine room

Step two is to restrict login to that computer to alimited number of other computers or people or both

Step three is to establish a firewall, especially for anintranet

5

ExperimentsExperimentsIn this chapter, we will concentrate on the rawapproach, in order to work out what the various issuesare, and summarise the other approaches

The first thing to do is to find out what kinds of requestthere are, and what information they contain

So, let's set up a minimal Node server which prints outrequest details - it may be helpful to keep it around forfuture use as well

6

DocumentationDocumentationThe documentation is in the Node docs for the httpand later the https modules

The documentation is very difficult to make sense ofbecause (a) it includes very low level things we don'tneed (b) it mixes client and server info (c) it isn't explicitabout types (d) it is poorly explained (e) it assumes thecallback style

Two important pieces of information are that a requesthas type IncomingMessage and a response has typeServerResponse

7

server1.jsserver1.js

Minimal serverMinimal server// Minimal server// Minimal server letlet HTTP = HTTP = requirerequire(('http''http');); start(start(80808080);); ......

Download server1.js to experiment

The 'http' module comes with Node, and so does 'https'

Use 8080, to add firewall security, to avoidroot/administrator privileges, or if port 80 is being usedby another program

8

server1.jsserver1.js

Starting the serverStarting the server...... // Provide a service to localhost only.// Provide a service to localhost only. functionfunction startstart((portport) ) {{ letlet service = HTTP.createServer(handle); service = HTTP.createServer(handle); service.listen(port, service.listen(port, 'localhost''localhost');); }} ......

The server is now listening for browser requests, butonly from the computer it is running on (do this whiledeveloping, until you are happy with your security)

Every request will cause a call to the handle function

9

server1.jsserver1.js

Handling requestsHandling requests...... // Deal with a request.// Deal with a request. functionfunction handlehandle((request, responserequest, response) ) {{ consoleconsole.log(.log("Method:""Method:", request.method);, request.method); consoleconsole.log(.log("URL:""URL:", request.url);, request.url); consoleconsole.log(.log("Headers:""Headers:", request.headers);, request.headers); reply(response); reply(response); }}

Print details - the most important fields of the request

Look up http.IncomingMessage in the Node APIdocs for the 'http' module to get request details

10

server1.jsserver1.js

RespondingResponding...... // Send a reply.// Send a reply. functionfunction replyreply((responseresponse) ) {{ letlet hdrs = { hdrs = { 'Content-Type''Content-Type': : 'text/plain''text/plain' }; }; response.writeHead( response.writeHead(200200, hdrs); , hdrs); // 200 = OK// 200 = OK response.write( response.write("OK""OK");); response.end(); response.end(); }}

Send a minimal response (as if delivering a plain textfile with one line containing OK)

Look up http.ServerResponse in the Node APIdocs for the 'http' module to get response details

11

TestingTestingWhen you visit http://localhost:8080/ withyour browser, the server will describe the request

You are quite likely to see two requests, one for URL /which you asked for, and one for URL /favicon.icowhich is the browser asking for the site icon

The test server responds to every request the same way

12

server2.jsserver2.js

Delivering pagesDelivering pagesletlet HTTP = HTTP = requirerequire(('http''http');); letlet FS = FS = requirerequire(('fs''fs').promises;).promises; letlet OK = OK = 200200, NotFound = , NotFound = 404404, BadType = , BadType = 415415;; start(start(80808080););

The next server server2.js delivers HTML pages

Names are defined for success and failure codes

start can't be called until global data has been defined(some people call it at the bottom of the script)

13

Handling page requestsHandling page requests

Add index.html if a URL ends with /, and give anerror if it is not .html, return after calling fail tostop further processing, convert the url into a file path

Read the file and use the content to reply

server2.jsserver2.jsasyncasync functionfunction handlehandle((request, responserequest, response) ) {{ letlet url = request.url; url = request.url; ifif (url.endsWith( (url.endsWith("/""/")) url = url + )) url = url + "index.html""index.html";; ifif (! url.endsWith( (! url.endsWith(".html"".html")))) returnreturn fail(response, BadType, fail(response, BadType, "Not .html""Not .html");); letlet file = file = "./public""./public" + url; + url; letlet content; content; trytry { content = { content = awaitawait FS.readFile(file); } FS.readFile(file); } catchcatch (err) { (err) { returnreturn fail(response, NotFound, fail(response, NotFound, "Not found""Not found")) reply(response, content); reply(response, content); }}

14

ReplyingReplying

the content comes from readFile

Deliver file content as application/xhtml+xml(can only handle web pages)

server2.jsserver2.jsfunctionfunction replyreply((response, contentresponse, content) ) {{ letlet hdrs = { hdrs = { 'Content-Type''Content-Type': : 'application/xhtml+xml'application/xhtml+xml response.writeHead(OK, hdrs); response.writeHead(OK, hdrs); response.write(content); response.write(content); response.end(); response.end(); }}

15

ErrorsErrors

In the case of an error, deliver a one-line plain textresponse, with a given error code

server2.jsserver2.jsfunctionfunction failfail((response, code, messageresponse, code, message) ) {{ letlet hdrs = { hdrs = { 'Content-Type''Content-Type': : 'text/plain''text/plain' };      };      response.writeHead(code, hdrs); response.writeHead(code, hdrs); response.write(message); response.write(message); response.end(); response.end(); }}

16

URL ValidationURL ValidationSecurity inside the server is a difficult topic; the firstline of defence is to validate the URL in each request

Warning:Warning: the original server.js provided doesn't dofull URL validation

Warning:Warning: expressexpress doesn't do full URL validation

Warning:Warning: other servers/frameworks don't do full URLvalidation

Security is youryour responsibility

17

File systemsFile systemsThe main problem is that servers convert the URL pathinto a file path by prefixing the root folder

Then they get the file system to find the resource

That means there are security loopholes, depending onodd features of the file system

18

Dot dotDot dotAn obvious attack is to use .. in URLs to try to reachfiles outside the root folder

Suppose you have a root folder /site/public and asensitive file /site/secret.txt

An attack might be to visit/products/../../secret.txt

This must be prevented and it is stronglystrongly recommendedthat you test it (express and other servers do preventit, if you use them properly)

19

TestingTestingTesting can be difficult, e.g. if you type a URL directlyinto your browser, and then get the server to print outthe URL, you may find that the browser has changed theURL before sending the request (especially /../)

The same happens with tools like wget or curl

For total control, write a program such as get.js

19a

SynonymsSynonymsSuppose you have an admin folder insideinside your site

And you have password protection of the form "if a URL starts with "if a URL starts with /admin//admin/ then require a password" then require a password"

Attacks may use synonyms to bypass the protection /products/../admin/adduser /./admin/adduser //admin/adduser

Express and other servers (and browsers) handle /../and /./ but notnot // so test this!!!

20

Restricting URLsRestricting URLsThe simplest approach to solve these problems is tovalidate the URL

Simplest is to reject a URL if it contains /. or //, orcontains non-ascii characters, or doesn't start with /, oris too long, see dotdot, hidden, spaces

Safer is to check that the URL matches one of the stringsor patterns that you regard as OK, and reverse yoursecurity checks

21

DotdotDotdotIf you reject URLs containing /., then you are rejecting/./, which is good, but also /../ which is used quitecommonly

But that's probably OK, because /../ is usuallyresolved by the browser before the request gets to theserver

If you prefer, you can canonicalise the URL, and thenreject it if it starts with .. which indicates that it istrying to climb out of the public folder, and thenapply security restrictions to the canonical URL

21a

Hidden filesHidden filesIf you reject URLs containing /., then you are rejectinghidden files /.hidden, which may be a good thing

On Linux and MacOs, any file or directory whose namestarts with a dot is automatically hidden

On Windows, a flag is used for hidden files, and namesstarting with a dot have no special meaning, but manytools such as git create files or folders that start withdot andand are marked as hidden

So probablyprobably a dot file is hidden, even on Windows

21b

SpacesSpacesSpaces are officially only allowed in URLs if they areescaped as %20

If you are going to allow them, you need to work out (a)how different browsers handle them (b) what gets sentto your server (c) how to handle them safely in yourserver on your file system (d) what effect, probablyconfusion, they will have on your users

It is almost always simpler and better to ban them

21c

Need to knowNeed to knowSecurity can be improved by reversing it (on the 'needto know' principle)

Suppose your site has just two folders, products andadmin

Instead of "if a URL starts with "if a URL starts with /admin//admin/ then require a then require apassword"password" do "if a URL doesn't start with "if a URL doesn't start with /products//products/then require a password"then require a password"

That would catch more of the problems without havingto think too hard

21d

CaseCaseAttackers might try /ADMIN/adduser to get pastyour password protection on /admin/...

Usually Linux servers are case sensitive and Windowsservers are case insensitive (the filestore decides!)

I recommend making all requests case sensitive, as inthe provided server, and using a whitelist or blacklistfor exceptions

22

AccessAccessAnother thing that needs to be done is to preventvisitors from accessing files which are not intended tobe accessible

That means e.g. being careful not to put the serverprogram, encryption keys, database files, config files,source code, editor backups, hidden files, and so oninside the public root folder

It is also a bad idea to have hard links, symbolic links,shortcuts, or anything equivalent, inside a root folder

23

jsjs

Content typesContent typesEach successful response needs a content type: the normis to maintain a table in the server code which maps fileextensions to content types (or use the mime module)

letlet types = { types = { csscss : : 'text/css''text/css',, jsjs : : 'application/javascript''application/javascript',, ... ... }} ...... let type = types[extension];let type = types[extension]; letlet hdrs = { hdrs = { 'Content-Type''Content-Type': type };: type }; response.writeHead(response.writeHead(200200, hdrs);, hdrs); ......

24

ExtensionsExtensionsWhat happens with a URL like /project/Makefilewhich doesn't have an extension?

The server doesn't know, just from the URL, whether itis a file with no extension or a folder with no /

If it is a file, the server doesn't know what type to use,so the default is often to treat it as a folder

If it is folder, you mustmust redirect the browser, so it issimpler to reject such URLs (maybe with exceptions)

25

RedirectionRedirectionWhat happens if you treat /x/y as /x/y/ and deliverthe file /x/y/index.html?

If there is a relative link on that index page, the browserthinks it is relative to /x/ and not /x/y/ so the linkfails

So you must redirect the browser, by sending aresponse with the corrected URL (using code 301, 302,303, 307, 308 - good luck working out which is correct)

I think it is better to avoid the inefficiency of multiplerequests by banning the original URL

25a

jsjs

Content negotiationContent negotiationFor XHTML, content negotiation is needed to serve an.html page to old and new browsers

// For the .html extension:// For the .html extension: letlet otype = otype = "text/html""text/html";; letlet ntype = ntype = "application/xhtml+xml""application/xhtml+xml";; letlet header = request.headers.accept; header = request.headers.accept; letlet accepts = header.split( accepts = header.split(","",");); ifif (accepts.indexOf(ntype) >= (accepts.indexOf(ntype) >= 00) type = ntype;) type = ntype; elseelse type = otype; type = otype;

Warning:Warning: express and other servers deliver all HTMLpages as "text/html" by default

26

Dynamic URLsDynamic URLsFor dynamic resources, e.g. pages generated fromdatabase data, it is up to you to invent URL conventions

As well as dynamically generated HTML pages, dynamicURLs may present requests to the server to update thedatabase, or deal with an upload - you could call theseaction URLs

There are lots of possibilities, including using express

27

ExpressExpressIn an express server, I personally find it extremelydifficult to understandunderstand what happens to a particularrequest, or to debugdebug what is happening to a particularrequest, or work out how to do 'unusual' things such ascontent negotiation

On the other hand, express has a very good 'mix-and-match' style, with a wide range of off-the-shelf tools

Maybe the best approach is to use those tools directly,without using express's obscure routing

27a

URL RewritingURL RewritingURL rewriting allows you to make logical and intuitiveURLs available to users

Suppose you publish a URL /pets/dogs/odie.html

When it arrives in a server request, you can change it to/pets/dogs.html?name=odie so you can use atemplate for all dogs, filled in with data about aparticular dog

Or rewrite to /pets.html?type=dog&name=odieso you can use a single template for all pets

28

RoutesRoutesA route is a rule for sending a dynamic URL to the righthandler function for dealing with the request

The handler function may use a template and a databaseto construct a web page to be sent back to the browser

OROR the handler may carry out some update to thedatabase, or deal with an upload, and send aconfirmation message back to the browser

Routes are most often based on URL prefixes

29

PrefixesPrefixesIn the prefix case, you recognise any URL which startswith a fixed set of handler prefixes, e.g /set/x/42 or/set?x=42 need to be handled by a function set

The server contains code to recognize the action URLsand call the right handler function, passing it therequest, response, etc.

The advantage is simplicity when there are fewhandlers, and the danger is a poor, non-modular,monolithic design, i.e. all in a one-piece program

30

ModularityModularityA good way to make routing more modular in a DIYserver is for dynamic or action URLs to be of the form/do/set?x=42

The /do/ prefix is used to recognize the URL as anaction URL

The next part set is used as the name of a moduleset.js in your server module folder (notnot insidepublic!)

You pass it the parameters and it handles the request,see table of handlers

31

jsjs

jsjs

Table of handlersTable of handlersYou can maintain a table of handlers

letlet handlers = ...; handlers = ...; letlet f = handlers[name]; f = handlers[name]; f(request, response);f(request, response);

If you are using modules, you can load a module if thehandler hasn't been used before (using node's old non-standard module system)

letlet m = m = requirerequire(("./do/""./do/" + name + + name + ".js"".js");); letlet f = m.handle; f = m.handle; handlers[name] = f;handlers[name] = f;

31a

MethodsMethodsThe method GET represents a normal request for a pageor file - it has no body

The method POST sends data from a form - it has abody with the parameters and their values

The method PUT is very useful for simple uploads - youcan arrange for the filename to be in the URL, and thebody to be the file contents (the alternative, POST withtype multipart/form-data, is complex andinefficient)

32

Going furtherGoing furtherTo avoid nasty callback issues and allow theasync/await keywords to be used, a promise versionof the http library is needed

At present, there isn't one built into node, so try this:

npm install q-io -savenpm install q-io -save

It provides require("q-io/fs") equivalent torequire("fs").promises, and require("q-io/http") which is like http withpromises

33

Reading the bodyReading the body

Like server 1, this responds OK to every request

The body is not instantly available - it has to be read in

server1b.jsserver1b.jsasyncasync functionfunction handlehandle((request, responserequest, response) ) {{ consoleconsole.log(.log("Method:""Method:", request.method);, request.method); consoleconsole.log(.log("URL:""URL:", request.url);, request.url); consoleconsole.log(.log("Headers:""Headers:", request.headers);, request.headers); letlet body = body = awaitawait request.body.read(); request.body.read(); consoleconsole.log(.log("Body:""Body:", body.toString());, body.toString()); response.status = response.status = 200200;; response.headers = { response.headers = {'Content-Type''Content-Type': : 'text/plain''text/plain' response.body = [ response.body = ["OK""OK"];]; returnreturn response; response; }}

34

Test pageTest pageTo test, create a minimal form page

Double click to view with browser (with a file: URL,not via the server) then press Send

htmlhtml<!DOCTYPE html><!DOCTYPE html> <<htmlhtml xmlnsxmlns=="http://www.w3.org/1999/xhtml""http://www.w3.org/1999/xhtml">> <<headhead>><<titletitle>>TestTest</</titletitle>></</headhead>> <<bodybody>> <<formform methodmethod=="post""post" actionaction=="http://localhost:8080/"http://localhost:8080/ <<inputinput namename=="pet""pet" valuevalue=="dog""dog"/>/> <<inputinput namename=="car""car" valuevalue=="bmw""bmw"/>/> <<inputinput typetype=="submit""submit" valuevalue=="Send""Send"/>/> </</formform>> </</bodybody>> </</htmlhtml>>

35

QuerystringQuerystringThe body of the request posted from a form contains: pet=dog&car=bmw

This can be unpacked with the querystring module

letlet QS = QS = requirerequire(("querystring""querystring");); ...... let params = QS.parse(body);let params = QS.parse(body); consoleconsole.log(params.pet, params.car);.log(params.pet, params.car);

36

MiddlewareMiddlewareIn the express world, middleware refers to a piece ofsoftware which sits between the main handle functionand the individual handlers which do the real work -they adapt the request in some way, e.g. checkpasswords and permissions

A middleware function is given the request andresponse by handle, chooses a handler, and passes itthe possibly modified request and response

Middleware functions are also called filters

37

ChoicesChoicesRoughly speaking, your choices are:

write your own server, using callbackswrite your own server, using async/await(maybe using flows)use express, with callbacks (as tutorials tell you todo)use express, with async/await (much better,but harder to find out how)

38

TemplatesTemplatesA template is a web page, reused with different data

Some convention is used to indicate points in thetemplate where data is inserted

If you are taking the raw approach to building a server,you could invent your own conventions, but you maywant to use some ready-made template system

Express allows you to choose between template engines(ejs, pug, haml, handlebars, combyne, swig, blade, ...)

39

Password protectionPassword protectionPasswords must not be sent over the network withoutencryption

Using http, passwords are sent using base64 encoding

This is mostly to turn special characters into ascii, andto prevent accidental prying, but it is not encryptionit is not encryption

To encrypt properly, you mustmust switch to https

encryption protocols, saving passwords, HTTPS sites, HTTPS forwarding

40

Encryption protocolsEncryption protocolsFor serious https work, you need to configure yourserver to avoid encryption algorithms or protocolswhich are now thought to be weak

Look for a web site (e.g. ssllabs) which will carry out an"SSL server test" on the encryption facilities offered byyour site

40a

Saving passwordsSaving passwordsHow are passwords saved in databases?

Usually, they are 'encrypted' first with MD5

Note (a) MD5 is really a digest (hash) mechanism and not anencryption (b) it is now regarded as weak and something likeSHA256 or bcrypt is recommended instead (c) it is only secondary security - the main security isnot letting anyone access the database in the first place

40b

Https sitesHttps sitesIt used to be common for a site to provide both HTTPand HTTPS (technically two servers, but running in thesame program)

It is becoming common to provide only HTTPS

HTTP/2 based on SPDY, and Let's Encrypt, are likely toincrease this trend

40c

Https forwardingHttps forwardingSuppose your server provides both an http service andan https service for the same site

And suppose that some areas of the site are protectedand some are not

If an http request is made for a protected area, it isimportant that the request should be redirected tohttps

That means sending a response to the browser, asking itto try again on the https URL

40d

CertificatesCertificatesTo use https, you need to provide a private key (from apublic/private pair) and a certificate

For this unit, a self-signed certificate is fine (though abrowser will ask the user to confirm trust), or if youhave a domain you can use Let's Encrypt

These can be loaded from files before the server starts(or even embedded in the code) but must be kept secret

41

Form submissionForm submissionSuppose your site has a form page for a privileged userto fill in

The form data will be sent to an action (specified in theaction attribute), normally a dynamic URL

The form page should be protected, so the user can seea comforting protection icon

But the actionaction URL really mustmust be protected, because itis the one that carries sensitive data and/or asks theserver to do updates

42

Input ValidationInput ValidationData validation can optionally be done on a form page(by HTML5 or script) for quick response

But validation mustmust also be done on the server side, onthe action URL that receives the data, for security anddata consistency

For good security, your attitude should not just be"what damaging data could a hacker send" but "whatrange of data is safe"

It is better to start very restrictive and expand than theother way round

43

Output ValidationOutput ValidationSuppose a user can enter text which ends up displayedon a web page (e.g. in a blog post or even in an errormessage)

Care needs to be taken that it can't do any harm byhaving any special HTML/CSS/JS meaning

The main danger is XSS (cross-site scripting) attacks

The issues are subtle, so it is best to use a security-aware library feature

44

SessionsSessionsA session is needed to track multiple requests comingfrom the same user, and keep some state data

This is particularly necessary for protected pages, just toremember that the user has already logged in andneedn't login again

It is possible to use OAuth to delegate logins

But otherwise, how do sessions work?

45

Shared secretsShared secretsIt is not enough to use ip addresses and port numbers,partly because they can be forged, and partly becausethey are not always specific to users and their browsers

It is normally done by having a shared secret, i.e. anunguessable random number held by both the browserand server

Since nobody must see it, https must be used, even ifpassword login isn't used

46

Generating a shared secretGenerating a shared secretIt is the server which must generate the shared secreton a user's first visit

A username or fixed id number or pseudo-randomnumber is notnot good enough

A time-of-day random number is very often used, whichis OK but not excellent

An id signed with an encryption key is good, and so is acrypto-random number (usually entropy-based)

47

CookiesCookiesA cookie is a name/value pair which the server sets in aresponse header

After that, the browser will send the same cookie in arequest header every time it visits the same server

This is ideal for a shared secret and is then called asession cookie and it lasts as long as you choose

There is no need to store any data in the browser otherthan a session id - everything else can be in the server

(The main exception is using browser localStorage whenthere is no configurable server)

48