Student Research ProjectManagement System

Dajie Yang

A report submitted for the courseCOMP8755 Individual Computing Project

Supervised by: Peter StrazdinsThe Australian National University

October 2019

c© Dajie Yang 2019

Except where otherwise indicated, this report is my own original work.

Dajie Yang25 October 2019

Acknowledgments

I would like to firstly thank my parents. No words can express my gratitude to myparents for supporting my study here in Australia. And I would also like to thankmy friends that helped me during this project, either emotionally or technically. Also,thanks my co-worker Euikyum for his contribution in this project, and my supervisorPeter for his valuable feedback.

iii

Abstract

The Research School of Computer Science currently uses a paper-based system tomanage study contracts. The system is currently experiencing efficiency and scal-ability issues, since the system relies on manual operation for the entire contractadministration process.

Meanwhile, web technologies has evolved dramatically over the past decade.With the stress on user interaction, new frameworks and architectures emerged, andchange the experience of web application significantly. Moreover, new technologies,especially docker, is bringing the software development & deployment process to anew era.

The project of building a Student Research Project Management System (SRPMS)is planned with the above backgrounds. In this project, we build an informationsystem that digitize the entire contract administration process, with careful designto ensure data integrity and security. The system is implemented as a moderndecoupled web application, employing a RESTful API in the back-end, and a single-page application in front-end. Also, the system uses docker to promise portabilityand avoid dependency issues, and enables automatic testing and deployment withGitLab continuous integration support.

v

vi

Contents

Acknowledgments iii

Abstract v

1 Introduction 11.1 Problem Statement and Motivations . . . . . . . . . . . . . . . . . . . . . 11.2 Project Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Report Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Background and Related Work 32.1 Decoupled Web Application . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Web Service with RESTful API . . . . . . . . . . . . . . . . . . . . . . . . 42.3 Single Page Application (SPA) as Front-end . . . . . . . . . . . . . . . . . 42.4 Light Weight Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.5 Web frameworks and Object Relational Mapping (ORM) . . . . . . . . . 52.6 DevOps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Design and Implementation 73.1 Architecture Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Normal Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.3 Django Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.4 Database Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.4.1 Django Object Relational Mapping (ORM) . . . . . . . . . . . . . 133.4.2 Model Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.5 API Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.6 Authentication in Back-End . . . . . . . . . . . . . . . . . . . . . . . . . . 163.7 Business Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.7.1 Placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.7.2 Permission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.7.3 Create and Update . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.7.4 Special Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.7.5 Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.7.6 Data Export . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.8 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.9 Angular Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.9.1 Authentication in Front-End . . . . . . . . . . . . . . . . . . . . . 27

vii

viii Contents

3.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4 Deployment 294.1 Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.2 SSL certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.3 Continuous Integration with GitLab . . . . . . . . . . . . . . . . . . . . . 314.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5 Conclusion and Future Works 35

Bibliography 37

Appendices 39

Appendix Study Contract of SRPMS Project 41

Appendix Sample PDF Export for Finalized Contract in SRPMS 45

Appendix Contribution Statement 49

Appendix Deployment Guide 51

Appendix Back-End Design in File Structure Level 81

List of Figures

3.1 General Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 Normal Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.3 Django REST Application Request Cycle . . . . . . . . . . . . . . . . . . 103.4 EER Model of Objects in Normal Workflow . . . . . . . . . . . . . . . . . 123.5 Diamond Shape Relation between SrpmsUser and Assessment . . . . . 133.6 Example Database Model Implementation . . . . . . . . . . . . . . . . . 153.7 Example Permission Logic Implementation . . . . . . . . . . . . . . . . . 183.8 Example Custom Create/Update Logic . . . . . . . . . . . . . . . . . . . 193.9 Example Special Action Definition . . . . . . . . . . . . . . . . . . . . . . 213.10 Notification Workflow (in Red Square) . . . . . . . . . . . . . . . . . . . . 223.11 Example Signal Implementation . . . . . . . . . . . . . . . . . . . . . . . 233.12 Example CSV Export . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.13 Example Test Class Implementation . . . . . . . . . . . . . . . . . . . . . 253.14 Test Coverage in Back-end . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.15 HTTP Interceptor in SRPMS Client . . . . . . . . . . . . . . . . . . . . . . 27

4.1 Docker Containers for Development Environment . . . . . . . . . . . . . 304.2 Docker Containers for Deployment Environment . . . . . . . . . . . . . 304.3 Example GitLab CI Configuration . . . . . . . . . . . . . . . . . . . . . . 324.4 SRPMS CI Pipeline Example . . . . . . . . . . . . . . . . . . . . . . . . . . 32

ix

x LIST OF FIGURES

List of Tables

xi

xii LIST OF TABLES

Chapter 1

Introduction

The Reseach School of Computer Science (RSCS) currently providing a number ofstudy-contract-based courses. All of these courses require a study contract beforeformal enrollment. However, the paper-based system RSCS currently employs isinconvenient to use and hard to maintain, especially considering the growing numberof contracts each semester.

This project represents the effort of addressing the above issue by building anoperational information system, i.e. the Student Research Project Management System(SRPMS), in order to provide better contract management experience for students andRSCS staffs.

In this chapter, Section 1.1 will gives details about current issues, while Section 1.2discuss the scope of this project. Lastly, Section 1.3 gives an overview of the structureof this report.

1.1 Problem Statement and Motivations

Currently, RSCS use a paper-based system to manage study contracts. In order tofinish a contract, the student firstly need to fill the contract word document, andthen print and hand to supervisor for supervisor’s approval. The supervisor willalso be responsible for finding examiners for assessments of the contract, and askfor examiners’ approval. At the final stage, the contract will be presented to thecourse convener for final approval. The student can then use the finalized contract toobtain a permission code for course enrollment. A finalized contract of the currentpaper-based system can be found in Appendix 5.

This paper-based system has several major disadvantages. Firstly, its labor-intensive. Supervisor needs to physically reach out examiners and course convenerin order to process the contract. Secondly, the process is not informative after the con-tract is submitted. As a result, a lot of email communication occurred after contractsubmitted, solely for the simple purpose of asking progress. Lastly, the paper-basedsystem is currently having a hard time maintaining records of past contracts. Thereare more than 100 contracts every semester at the moment, and the size of contractarchive will only grows larger in the future.

As one of the solution, it would be nice if the paper-based system can be con-

1

2 Introduction

verted to a digital system that can support the entire contract administration process.Considering various kind of user, it would be best if this digital system can be servedas a web application.

1.2 Project Scope

The main goal of the Student Research Project Management System (SRPMS) isto build an operational information system for handling the contract administrationprocess. The scope of this system includes the original workflow, and some additionalconvenience a digital system would typically provide. For example, student can usetheir ANU credentials to login, and such information would not be required againwhen creating a new contract.

The system should also be able to display relevant information to each partyduring the process, and can talk to other systems for re-using current information.Moreover, the system should be able to export contract data at the end of the process.

1.3 Report Outline

The implementation of SRPMS is executed by a team of two people, as such tworeports are produced corresponding to each person’s main contribution. This reportwill mainly focus on back-end implementation and system deployment, while theother one focuses on front-end implementation.

This report consists of multiple chapters. Chapter 2 will introduce the technicalbackground of this project, including the current technology and practices commonlyused in this kind of system. Chapter 3 will discuss the design and implementation ofthis system in details. Chapter 4 will talk about how this system is being deployed,with Chapter 5 concludes the project.

Chapter 2

Background and Related Work

With the arrival of Web 2.0, the idea of web application has changed dramatically.In this chapter, We’ll discuss the modern decoupled web application in Section 2.1,with Section 2.2 giving a common approach of providing web service in decoupledweb application, Section 2.3 discussing how web service is presented to client, andSection 2.4 showing the current trend of web server.

Starting from Section 2.5, we’ll discuss the choice of back-end implementation,with Section 2.5 focusing on the choice of web framework, and Section 2.6 focusingon deployment and testing strategies.

2.1 Decoupled Web Application

In the era of Web 2.0, most today’s websites feature a decoupled front-end and back-end, in order to enhance the experience of user interaction, provides cross platformsupport, and handle scalability issue better.

To better illustrate, we compare Web 2.0 and Web 1.0 in details based on researchof Viswanathan et al. [2009]. Firstly, Web 2.0 uses dynamic HTML, which means apage can change in browser in real-time according to the data it loads without anyrefreshing. Secondly, Web 2.0 moves towards a universal standard, instead of usingbrowser-specific HTML tags. Thirdly, Web 2.0 focus more on user interaction ratherthan simply serving contents.

With the background of Web 2.0, decoupled web application appeared. In atypical decoupled application, the back-end only exists for the purpose of exchangingraw data, instead of HTML pages, between the front-end. The front-end is servedfrom a static file server, and runs in users’ browser as a client to interact with the back-end. This provides several benefits [AV, 2019]. Firstly, the back-end and front-end’sdevelopment can have separate workflow, thus can progress in parallel. Secondly,it isolates bugs between front-end and back-end, and make it possible to test themseparately. Thirdly, the back-end API can be re-used for multiple platforms, and itseasily scalable.

3

4 Background and Related Work

2.2 Web Service with RESTful API

Representational state transfer (REST) [Fielding, 2016] architectural style is one of themost common approaches for implementing web API. It defines a set of constraintsdescribing how the requesting system can access and manipulate textual represen-tation of resources hosted in a remote system. Operations between the requestingsystem and the remote system is designed to be uniform and stateless.

In the perspective of web service, the set of operations is HTTP methods, includingGET, POST, PUT, PATCH, and DELETE. These operations correspond to the retrieve,create, update, and delete actions on resources [W3C, 2004]. The textual representa-tion of resources is often in JSON format, although other representation can also beused (e.g. xml). Most of today’s web application provides JSON RESTful API in theback-end, for example, both WordPress and Drupal 8 have such API built-in, and willdeepen support in future releases.

2.3 Single Page Application (SPA) as Front-end

While back-end focus on serving data, front-end evolves towards the enhancement ofuser experience. The most significant change during this evolution is the appearanceof single page application.

Single page application provides significant advantage on loading time compareto multi-page application. Previous to decoupled applications, pages are rendered inserver side, and a user needs to wait for the entire page to loads almost every time thedata or URL change, this is known as multi-page application. Single page application,on the other hand, cache all view templates on initial load. The application thenretrieves data asynchronously from the back-end, and render the page based on theretrieved data. The front-end would also employ a router intercepting the navigationaction. Change of url then simply become loading of an already cached template,with little to no latency. For SPAs, the client would only communicate to back-endwhen it needs data to fill the template.

2.4 Light Weight Web Server

With the thrift of microservice and container technology, the market share of webservers has changed dramatically. Nginx is currently the most popular web server,with 35% market share. Apache, on the other hand, has decreased to 29% over thepast several years [Netcraft, 2019]. Unlike Apache’s process-or-thread-per-connectionapproach, each worker in Nginx can handle thousands of HTTP connections simulta-neously, thus make Nginx light in resources, and high in performance [Nginx, 2015].For this reason, Nginx is widely used as load balancer, gateway server, or static fileserver in modern web applications. While the event-driven model does make Nginxmuch harder to customize, the configuration Nginx provides is enough for the scopeof most web applications.

§2.5 Web frameworks and Object Relational Mapping (ORM) 5

2.5 Web frameworks and Object Relational Mapping (ORM)

As back-end only needs to provide RESTful API, the difference between web frame-works reduced. The key component of RESTful API only includes data serializationand HTTP request handling. Common frameworks, including Spring (Java), Laravel(PHP), .NET(C#), Django (Python), all supports RESTful API very well.

In addition, almost every programming language today has its own Object Rela-tional Mapping (ORM) library, which provides a layer to operate database, insteadof writing raw SQL commands. Developers simply need to define the model in theirpreferred language, and ORM will talk to the database for creating correspondingschema. ORM also wraps query results into language callable for easy manipulation,and type inference. Moreover, some ORM provides migration support for change ofdatabase schema, dramatically simplify the maintenance of database.

For the above reasons, the selection of framework and language depends moreon personal preference and team background. While the performance of differentframeworks still exists, unless targeting for millions of users from beginning, havingsuch worry is not necessary. Moreover, even if based on a relatively inefficientlanguage, Python, the web framework Django can still handle billion-level pageviews [Robenolt, 2013].

2.6 DevOps

With the growing of complication of modern web application, DevOps plays a moreand more important part in software development and product delivery.

DevOps refer to a set of practices that combines software development(Dev) and information-technology operations (Ops), and aims to shortenthe systems development life cycle and provide continuous delivery withhigh software quality. [Loukides, 2014]

Continuous Integration (CI) is one of the most common DevOps practice. Con-tinuous integration aims to integrate newly developed code into existing system asoften as possible. In common practice, a set of predefined operations will be executedon every git push, including build and test of the system. This ensures the systemworks correctly after the new code has been merged. The main benefits of CI is that itensures the stability of the system even under fast development pace, and developerscan fix bugs as soon as possible based on feedback from the CI.

To facilitate CI, modern collaboration platforms provides a range of tools. Forexample, GitLab (gitlab.com) provides CI/CD support by specifying CI operationsin .gitlab-ci.yml file in the root directory of the git repository. Users can eitherconfigure to use the public machine provided by GitLab to run the CI operations, orregister their own machine as CI runner.

Alone side with CI, container technology is also widely adopted to ensure theconsistent and portability of environments. Currently, docker is the most popular

6 Background and Related Work

choice for containerization. Docker is an OS-level virtualization technology thatshares the host system’s kernel. However, docker may run different Linux distributionon top of the kernel, with isolated file system and package management. Thesefeatures make docker light in resources, and high in performance. Docker buildsdocker image, where Linux distribution, system configuration, and start up behaviorwould be specified. An instance of a docker image is called docker container. Dockercan share data between containers using a docker volume, or bind mount a directoryfrom the host system to a running container.

2.7 Summary

In this chapter, we discussed commonly used approaches in developing modern webapplication, including decoupled front-end and back-end, JSON RESTful API, singlepage application, selection of web framework, and continuous integration.

In the next chapter, we would discuss detail design and implementation of theSRPMS based on approaches discussed in this chapter.

Chapter 3

Design and Implementation

In this chapter, we’ll discuss design and implementation of SRPMS. Section 3.1 woulddiscuss the design choice and general architecture of the back-end, with Section 3.2demonstrate the normal workflow of the contract approval process, while Section 3.3provides an overview of our Django application, and Section 3.4 presents the designof relational database schema for the purpose of saving data in normal workflow.

Dive deeper, Section 3.5 describes how API of the back-end organized, and theirfunctionalities, Section 3.6 discuss how the back-end authenticates users, and Sec-tion 3.7 elaborates how business logic (e.g. permission, create & update, notifications)is designed and implemented.

At last, Section 3.8 shows how the back-end is tested, with the last section (Sec-tion 3.9) giving an overview of the front-end application, and how a decouple front-end authorizes its request to the back-end.

3.1 Architecture Overview

Based on Section 2.5, the selection of tools for this project is based on the followingprinciples: light-weight, mainstream, small learning curve, and adhere to the team’scurrent background. As such, Nginx, Django (with Gunicorn), and PostgreSQL ischosen, with the front-end client implemented by Angular. Figure 3.1 shows howthese components are connected.

A user of this system would use an Angular client (which is retrieved as staticfile when the user first access the website) to communicate with the RESTful API inback-end. All API related communication would be redirected by Nginx gateway toGunicorn HTTP server, where HTTP request would be translated to Python callable,and being sent to Django. Django then process the request, and communicate withPostgreSQL database as necessary.

We selected Nginx for its light-weight and popularity. In SRPMS, it acts as thegateway server for forwarding API requests to the API server, and also acts as a staticfile server for serving static resources (e.g. the Angular client).

As for web framework where the API logic resides, we choose Django. Djangois a web framework written in Python, and is selected since the team member ofthis project all have Python programming experience. Compare to other Python

7

8 Design and Implementation

Nginx Gunicorn Django PostgreSQL

Response R

equest

Angular

SRPMS

Clie

nt (B

row

ser)

Serv

er

Figure 3.1: General Architecture

web frameworks, the main advantage of Django is its detail documentation, and thepowerful built-in Django Object Relational Mapping (ORM) library. Details aboutDjango ORM would be discussed in Section 3.4.1.

However, Django is only a web framework, and is not designed to act as a webserver, as such Gunicorn is chosen as the Python Web Server Gateway Interface(WSGI) HTTP Server. WSGI is Python’s standard for translating HTTP request toPython callable, and can be seen as the Python equivalence of Java Servlets.

Alone side with Django, PostgreSQL is selected, for Django ORM’s well supportof this database.

For the front-end, Angular is selected for the completeness of its documenta-tion. Contrary to other popular choices (e.g. React) that focus only on one thing(e.g. UI, HTTP, or Async), Angular provides a complete framework that includesfunctionalities ranging from the basic HTTP client to page layout and dynamic form.

3.2 Normal Workflow

The normal workflow describe the process from the initialization of a study contractto its finalization. Figure 3.2 illustrates the entire normal workflow.

At the beginning of the workflow, student initialize the contract, and edit the con-tract to include information, e.g. project title, description, objectives, and assessments.The student then nominates a formal supervisor for this contract, and submit thecontract to SRPMS.

After the first contract submission, the contract would be visible to the nominatedformal supervisor. The supervisor can then nominates examiner for each assessments,and optionally non-formal supervisors for the contract. Alternatively, if the supervisordisagree with the contract’s content, he/she can disapprove the contract. In this case,the contract would be returned to the student for further revision.

When the formal supervisor finally approve the contract, examiners would be

§3.2 Normal Workflow 9

Normal FlowC

onve

ner

Yes ApproveFinalized

Exam

iner

Approve

Supe

rvis

or

Approve

Nominate Examiner

Formal SupervisorNominate Non-formal Supervisor

Stud

ent

Initialize Contract Edit Contract Submit Contract

Nominate Formal Supervisor

Edit Assessments

No

Yes

Yes

No

No

Figure 3.2: Normal Workflow

informed to approve assessments. Any examiner’s disapproval would result thecontract being sent back to formal supervisor. The formal supervisor can then editother supervisor information and approve again to send to examiners, or disapproveto send to the student for content revision.

At the time all examiners approved, the contract would be presented to courseconvener. The contract would be finalized (i.e. read-only, even for course convener)if the course convener approve, or return to formal supervisor on disapproval.

The details of workflow can vary depending on the contract type. For example,the assessment items of individual project contract is fixed, but the special topiccontract can have flexible assessment items.

During the entire normal workflow, only student has the permission to edit con-tract content, however only in the circumstance that the contract haven’t been submit-ted. Formal supervisor, on the other hand, is a group of users that are recognized bythe Research School of Computer Science. Students can only nominate formal super-visors, other supervisor, if needed, require formal supervisor’s nomination (after thecontract submitted).

10 Design and Implementation

Response

Web Server(Nginx)

WSGI

ResponseMiddlewares

URL Router

Views

Generate Response PermissionChecking

Model(Django ORM)

Database(PostgreSQL)

RequestMiddlewares

Serializers

Create/UpdateSerializeResult De-serializeData

Validation

Django

Request

Client

Gunicorn

Figure 3.3: Django REST Application Request Cycle

3.3 Django Applications

A Django project employs one or more Django application, with each applicationprovides a distinct set of functionalities. This separation means each applicationdefines its own router, database schema, and more. Routers of different apps existin their app’s namespace, and are collected by the router defined in project root.Database schema, on the other hand, can be referred from other applications.

For the purpose of implementing RESTful API, SRPMS use the third party pack-age django-rest-framework. As a 14 years old web framework, Django is notoriginally designed for the purpose of serving RESTful API. In Django, each routerentry is connected to a view, which defines what response should be generated basedon the request. While generic Django applications render and return an HTML page,the REST framework returns the text representation of the requested resource. Atypical request cycle of Django application with REST framework is shown in Fig-ure 3.3. In Figure 3.3, serializer is the component that handles the conversion betweenresources and their text representation. Details about these components would bediscussed in Section 3.7.1.

Conveniently, Django REST framework includes a ViewSet class for creating

§3.4 Database Schema 11

views for a database schema. A ViewSet includes a list view and a detail view1.List view supports GET and POST method, for listing all objects of this schema orcreating a new object. Detail view supports GET, PUT, POST, and DELETE method,for showing or manipulating a single object. For example, the ContractViewSet,which corresponds to the contract schema, includes /api/research_mgt/contracts/ as listview, and /api/research_mgt/contracts/<id>/ as detail view, where <id> is the primarykey of the contract.

A ViewSet class can also include custom actions for list view or detail view. Forexample, api/research_mgt/contracts/<id>/submit/, or api/research_mgt/contracts/export_csv.

The implementation of SRPMS includes two Django applications, with one pro-viding account service, and one for contract management service. This is for theconsideration that user management should be reused across multiple applications,instead of being application specific. The account service provides authenticationAPI, including session based login/logout, and token authentication. The contractmanagement service provides API related to the normal workflow.

3.4 Database Schema

The design of database schema plays a crucial part of the implementation of thenormal workflow. The schema needs to appropriately describe the multiplicity be-tween objects, and correctly include attributes for relations. The Enhanced-EntityRelationship (EER) model designed for SRPMS is illustrated in Figure 3.4.

To be specific, this EER model describe the following relations:

• A SrpmsUser may or may not supervise/convene/own one or more Contract

• A SrpmsUser may or may not examine one or more Assessments

• A SrpmsUser may or may not be nominated by one and only one SrpmsUser

• A SrpmsUser may or may not nominate one or more SrpmsUser

• A Contract must be supervised by one or more SrpmsUser

• A Contract must be convene by one and only one SrpmsUser

• A Contract must be owned by one and only one SrpmsUser

• A Contract have the following sub-classes: Individual Project, and Special Topic

• A Contract must related to one and only one Course

• A Contract must be assessed by one or more assessments

1This report use "detail view", instead of "details view" for adhering to Django REST Framework’sofficial documentation

12 Design and Implementation

Figure 3.4: EER Model of Objects in Normal Workflow

• A Course may or may not related to one or more Contract

• An Assessment must assess one and only one Contract

• An Assessment must be examine by one or more SrpmsUser

• An Assessment must be defined from one and only one Assessment Template.

• An Assessment Template may or may not define one or more Assessments

Moreover, this EER model is designed with the consideration of future projectscope, in order to prevent dramatic change of database schema in the future. Forexample, while both individual project and special topic contract only support oneexaminer per assessment, the EER model supports multiple examiner for the sameassessment, as honors degree’s study contract requires two examiners for a singleassessment item. However, support of honor’s contract is not currently part of theproject scope, and would not have corresponding API entry.

During implementation, the EER model is converted to database schema usingER-to-Relations algorithm. The following schema are created for this EER model:SrpmsUser entity, Contract entity, Supervise relation, Assessment entity, Assessment-Template entity, Course entity, IndividualProject subclass, SpecialTopic subclass, As-sessmentExamine entity, Examine relation (between Contarct, SrpmsUser, and Assess-mentExamine), and Supervise relation.

§3.4 Database Schema 13

Figure 3.5: Diamond Shape Relation between SrpmsUser and Assessment

3.4.1 Django Object Relational Mapping (ORM)

Django ORM, similar to other ORM, converts data between incompatible type systems.In the case of SRPMS, its PostgreSQL data record and Python callable. To create adatabase schema with the ORM, developers simply need to define a Python classinheriting the Model class provided by the ORM, and the member variables of thisclass using special objects provided by the ORM (e.g. CharField, IntegerField).Django ORM would then automatically generate SQL commands based on theseclasses, and execute on the next database migration, i.e. the manual operation ofinforming Django to apply changes to the database.

However, Django ORM does not support all types of entities and relations in theEER model standard. To be specific, weak entities and determine relations are notsupported. However in SRPMS, the Assessment is a very important weak entity, andhave a determine many-to-many relation with the SrpmsUser entity.

In order to correctly represent this relation, we make an equivalent diamond shaperelation between SrpmsUser and Assessments. Figure 3.5 illustrates this diamondshape relation. It defines the following relations:

• SrpmsUser may or may not examine one Contract, with one or more AssessmentExamine

• Contract must be examine by one and only one SrpmsUser and one or moreAssessment Examine

• Contract must be assessed by one or more Assessment

• Assessment must be examine by one or more Assessment Examine

• Assessment must assess one and only one Contract

14 Design and Implementation

• Assessment must be defined from one and only one Assessment Template

• AssessmentExamine must examine one and only one Assessment

• AssessmentExamine must be examine by one and only one SrpmsUser, withone and only one Contract

3.4.2 Model Constraints

Django ORM also provides a convenient way to model constraints, with each modelobject have its own clean() method for customizing model level constraint. Modellevel constraint specifies the relation between fields or even related objects, and ischecked every time before an object of this model is being stored to the database.

The following constraints are implemented for the EER model given in Figure 3.4and Figure 3.5:

• For contract entity, if submitted, the weight of all its assessments must sum to100, and must have at least one supervisor

• For contract entity, if finalized, the contract must be submitted, with all assess-ment approved, and all supervisor approved

• For supervise relation, un-submitted contract is not allowed to be approved

• For supervise relation, its supervisor and contract must be unique

• For examine relation, its examiner and contract must be unique

• For assessment examine relation, its related examine relation and assessmententity must be unique

• For contract entity, supervise relation, assessment entity, examine relation, andassessment examine entity, modification is not allowed if the related contract isfinalized.

Ensuring model constraints is important, as they specify the minimal level of dataintegrity, and exist beyond user permissions. Modification of model constraints canonly be done in source code, no user permission can override model level constraints.Figure 3.6 shows an example implementation of a model using Django ORM.

3.5 API Organization

The organization of RESTful API represents the resources exposed to clients. However,representing resources according to database model would be a waste to the extraabstraction capability the API layer provides. When converting an object of a databaseschema to its text representation, it is common to only expose a subset of fields fordifferent API entries, so as to apply different behavior and permissions for different

§3.5 API Organization 15

1 from django.db import models2

3 class Contract(models.Model):4

5 year = models.IntegerField()6 course = models.ForeignKey(Course, on_delete=models.PROTECT)7

8 ...9

10 def clean(self):11

12 ... # model constraints

Figure 3.6: Example Database Model Implementation

fields of the same model. It is also possible to combine multiple objects into one textrepresentation, which would be seen as a single resource by the client.

For example, the Contract model has field convener_approval_date represent-ing the date time convener finalize the contract. Empty of this field would meanconvener hasn’t approve this contract, or a date time should be presented. Such fieldshould be read-only when a client requests the resource from the contract API, andbe provided in a dedicated API entry where only users with convener permissioncan access. In SRPMS’s implementation, client can interact with contract resourcesthrough API entry /api/research_mgt/contracts/ or /api/research_mgt/contracts/<id>, withthe approval function available at API entry /api/research_mgt/contracts/<id>/approve/.

Furthermore, the SRPMS API presents related resources through nested router.This ensures that only relevant resources would be accessed by the client. For example,assessments of a contract is available at api/research_mgt/contracts/<id>/asssessments/,and examine relation of an assessment is available at api/research_mgt/contracts/<contract_id>/asssessments/<assessment_id>/examine. In the case that an assessment exists butdoes not belongs to the contract specified in URL, HTTP 404 would be returned.

The list of all current SRPMS API entries are shown below:

/api/+-- accounts/| +-- token/| +-- token/refresh/| +-- users/| | +-- <id>/| +-- login/| +-- logout/+-- research_mgt/| +-- users/| | +-- <id>/| | | +-- set_formal_supervisor/| | | +-- set_course_convener/

16 Design and Implementation

| +-- courses/| | +-- <id>/| +-- assessment-templates/| | +-- <id>/| +-- contracts/| | +-- export_csv/| | +-- <id>/| | | +-- submit/| | | +-- approve/| | | +-- print/| | | +-- supervise/| | | | +-- <id>/| | | | | +-- approve/| | | +-- assessments/| | | | +-- <id>/| | | | | +-- examine/| | | | | | +-- <id>/| | | | | | | +-- approve/

3.6 Authentication in Back-End

The back-end support both ANU and SRPMS credentials, through session or tokenbased authentication. The ANU credential is supported through ANU LDAP service,which is only accessible within ANU network. SRPMS credential login, on the otherhand, use credentials stored in SRPMS user database.

The ANU credential login allows the majority user of this system, i.e. ANUstudents and staff, access SRPMS without both the need of creating extra account,and the need to enter user information again in a different system. SRPMS configurethis by using library django-auth-ldap, which authenticate the user with theirinput credential against ANU LDAP service through standard LDAP protocol. Itwould also automatically creates a dummy entry for users on their first login, thedummy account stores user information returned from ANU LDAP service, and isconfigured to have unusable password to prevent direct login.

Being a decoupled web application, the client use token authentication to au-thorize its call to the back-end. Traditionally in session authentication, a user’s statewould be maintained in the server side, providing context for HTML rendering. How-ever, since the principle of RESTful is to be stateless, session authentication wouldviolate such constraint directly. Furthermore, as the page is rendered in front-endclient with data from the back-end, the context in server side become pointless, as thecontext need to be fetched to front-end client eventually.

In the front-end, token authentication is done by appending an access token to theheader of every request to the back-end. On user login, rather than creating a state inserver side, a token would be generated based on username and expire date of this

§3.7 Business Logic 17

token, and return to the client. Client then append the token to the ”Authorization”header on every request. Server would then be able to identifies the user based onthe token in the header.

Despite the use of token authentication, session-based authentication is still im-plemented, consider that administrators and developers would sometimes need touse API directly through the browsable API interface. This interface is provided bythe Django REST framework, and is rendered in server side.

3.7 Business Logic

Business logic refers to the part of program that encodes the business rules thatdetermines how the end user interface communicates to the database. In SRPMS,business logic mainly includes user permission checking, type of contract checking,and some special actions.

3.7.1 Placement

Placing the business logic correctly is crucial for good architecture and maintainability.Logics that are request related should always be processed first. For example, checkuser permission, and assign automatic attribute according to the request sender.Later, logic that is request unrelated would be processed, including create and updatedata, and send notifications. The separation of logic layer ensures the number ofparameters would decrease as layer deepen, and help improve the readability of thecode by avoid placing logic sporadically.

For Django application with REST framework, most business logics are beingplaced in views and serializers, where views handle request related logic, and serial-izers handle request unrelated logic, as shown in Figure 3.3.

View is the first stop after the request getting out of the router, and a request mustcorrespond to one and only one view. A view would first check if the requester senderhas the permission regarding the resource it requests, method it use, and permissionof the request sender. Then the request content, which is in text representation,would be sent to a serializer for parsing. The return of the serializer can be referredas validated data, including objects the text representation refers to, and other relatedresources in the form of Python callable. The view can then further process thevalidated data. For example, in the AssessmentExamineViewSet, an nominatorattribute would be generated according to the request sender. After finished, resultswould be sent to serializer for persistence.

Serializer is the final stop of a request in most case, as create/update/deletewould be performed here. Serializer would receive processed information from view,i.e. converting information that directly related to the request to request neutral, andthen perform create, update, or delete on the object.

18 Design and Implementation

1 from rest_framework.settings import api_settings2 from .permissions import (AllowSafeMethods,3 IsConvener, IsSuperuser, IsContractFormalSupervisor,4 ContractSubmitted,5 IsThisSupervisorNotApprove, ContractNotFinalApproved,6 IsExaminerNominator)7

8 default_perms: list = api_settings.DEFAULT_PERMISSION_CLASSES9

10 class AssessmentExamineViewSet(...):11

12 ...13

14 permission_classes = default_perms + [AllowSafeMethods | IsSuperuser |15 (IsConvener & ContractSubmitted &16 ContractNotFinalApproved) |17 (IsContractFormalSupervisor &18 ContractSubmitted &19 IsThisSupervisorNotApprove &20 ContractNotFinalApproved &21 IsExaminerNominator), ]22 ...

Figure 3.7: Example Permission Logic Implementation

3.7.2 Permission

SRPMS implemented three levels of permission, view level, object level, and fieldlevel. View level permission checks whether a user is allowed to access a view, objectlevel checks whether a user is allowed to access an object in the view, while field levelchecks if a user is allowed to modify a field on that object.

SRPMS utilize the permission class Django REST framework provided, and imple-ments permission classes in a very intuitive way. For example, Figure 3.7 shows thepermission class of AssessmentExamineViewSet. The permission_class speci-fies a list of permission classes, with all of them inherit from the BasePermissionclass provided by REST framework. On request arrive, this list of permissions wouldbe evaluated against the request, and allows the request only if all the direct elementsof the list are True. In this example, excluding the default permission, the permissionlist only have one direct element, where AND and OR logical operators are used toapply different permission to different user.

Consider the number of permissions and the complication of the logic, this reportwould not include a list of permission logics.

For field level permission checking, while SRPMS does support, it should gen-erally be avoided during implementation, for the reason that it would easily in-troduce complicate logic, and would be executed every time regardless whetherthis field would be updated. Instead, a dedicated view for field should be used,with custom permission in view level. For example, the submit date and con-vener approval date are all direct attributes of the contract schema, but they are

§3.7 Business Logic 19

1 from rest_framework import serializers2 from research_mgt.models import Contract3

4 class ContractSerializer(serializers.ModelSerializer):5

6 ...7

8 class Meta:9 model = Contract

10

11 def create(self, validated_data: dict) -> Contract:12

13 ...14

15 def update(self, instance: Contract, validated_data: dict) -> Contract:16

17 ...

Figure 3.8: Example Custom Create/Update Logic

excluded from the contract API at /api/research_mgt/contracts/<id>/, and are separatedin /api/research_mgt/contracts/<id>/submit/ and /api/research_mgt/contracts/<id>/approve/respectively. SRPMS currently only have field level permission checking for superviserelated API.

Lastly, SRPMS also limits the visibility of objects to each user. After the view levelpermission check, SRPMS would get a query of objects this user can view (i.e. query).For objects that a user cannot view, object and field level permission would alwaysfail. SRPMS currently limits the object visibility in the following way:

• Superuser can see all contracts

• Course convener can see all contracts

• Login users can see all finalized contracts 2

• Login users can see contracts they own

• Login user can see contracts they supervise or examine, if the contract has beensubmitted at least one time.

3.7.3 Create and Update

In Django REST framework, the actual create and update happen in serializer mostof the time. Generally speaking, serializer is tight to a model, i.e. database schema,so that it can automatically infer fields of a model and validation method it shoulduse for the field. This kind of serializer is also known as model serializer.

2This is the current CSPROJECT website’s behavior

20 Design and Implementation

The majority of business logics are implemented in serialize create/update stepby override the default method, as shown in Figure 3.8. Part of these business logicsare:

• Supervise relation only allows one formal supervisor for each individual projectcontract

• Supervise relation would automatically clear supervisor approval if the super-visor is being changed

• AssessmentExamine relation would automatically clear examiner approval ifthe examiner is being changed

• Contract can have one and only one subclass (currently one of individual projectand special topic)

• When creating new individual project type of contract, relevant assessments (i.e.report, artifact, and presentation) would be created.

For business logics that involve manipulation of more than one object, transactionis used to ensure atomicity. In details, the commit of these objects are wrapped in atransaction, the failure of any commit would result the failure of the entire transaction,and the database would be reverted to the state before the transaction begins.

For example, the creation of an individual project contract implies the creationof its three associates assessments, namely report, artifact, and presentation. Thecreation of the individual project contract should fail if any of these assessmentscannot be created for some reason. However, as an assessment object must has aforeign key referring to a contract, the contract object need to be created before anyassessment can be created. In this case, transaction make sure the created contractwould be destroyed if any of the assessment fail to create.

3.7.4 Special Actions

Normally, a ViewSet class from REST framework only includes one list view and onedetail view. However, the REST framework provides a convenient to add extra viewsto a ViewSet, as shown in Figure 3.9. These views are referred as special actions inthis report.

Special action are just another view in a ViewSet class, but it often denotessignificant transition in the normal workflow. For example, the submit action wouldforbid the student from further editing the contract, and the contract would be sent tosupervisors for approval. Durnig this process, SRPMS would check if all assessments’weight sum to 100, and at least one supervisor is nominated for the contract. Suchaction also have side effects, especially the case of its revert. For example, the revertof submit action would reset all approvals from supervisors and examiners. This isfor the consideration that the owner would be able to modify the contract content,but these modifications should not be approved automatically.

SRPMS currently employs the following special actions:

§3.7 Business Logic 21

1 from django.http import HttpResponse2 from rest_framework.decorators import action3

4 class ContractViewSet(...):5

6 ...7

8 @action(methods=[’PUT’, ’PATCH’], detail=True, serializer_class=...,9 permission_classes=default_perms + [...])

10 def submit(self, request, pk=None) -> HttpResponse:11

12 ...13

14

15 @action(methods=[’PUT’, ’PATCH’], detail=True, serializer_class=...,16 permission_classes=default_perms + [...])17 def approve(self, request, pk=None) -> HttpResponse:18

19 ...

Figure 3.9: Example Special Action Definition

• Detail views of ContractViewSet:

– Submit: for contract owner to submit a contract. Revert of this actionwould reset all approvals from supervisors and examiners.

– Approve: for course convener to finalize a contract. Revert of this actionwould reset the formal supervisor’s approval.

• Detail views of SuperviseViewSet:

– Approve: for supervisors to approve a contract. Revert of this action wouldreset approvals from examiners that are nominated by this supervisor. Inthe case that the supervisor is a formal supervisor, the contract’s submitstatus would be reverted.

• Detail views of AssessmentExamineViewSet:

– Approve: for examiners to approve an assessment. Revert of this actionwould also reset the approval from supervisor that nominated this exam-iner.

Special actions also help managing field level permission. For example, the sub-mit_date and convener_approval_date fields all belongs to the model of thecontract entity. However, these fields should not be edited alone side with otherattributes like learning objectives and project description, and checks like the sum ofassessment weight should not be executed every time either.

Moreover, serializers for special actions are not connected to models. In the caseof contract, while the submit_date is a date-time field of the contract model, it is

22 Design and Implementation

Res

pons

e

Web Server(Nginx) WSGI

ResponseMiddlewares

URL Router

Views

Generate Response PermissionChecking

RequestMiddlewares

Serializers

Create/UpdateSerializeResult De-serializeData

Validation

Django

SignalSender

SignalReceiver

NotificationRequired?

Create and StoreActionLog

Send Notifications

Req

uest

Client

Approve / Submit

Figure 3.10: Notification Workflow (in Red Square)

being represented as a boolean field in the API. The serializer of the submit actionwould translate the boolean value to date time according to the current server time.To be specific, "True" would set the field with the current server time, and "False"would set the field to null.

3.7.5 Notifications

SRPMS uses Django signal mechanism to implement notifications. Django signalallows custom signal sender and receiver. SRPMS currently implements four signalscorresponding to each special action mentioned in Section 3.7.4. These signal sendersare placed after all business logics finished, and before the response is being returnedby the view. This ensures that no signal would be triggered on the failure of anybusiness logic. Figure 3.10 shows how signals are triggered during the request cycle,with an example implementation given by Figure 3.11. Currently, SRPMS sendsnotifications in the following scenario:

• Contract owner would be notified if the contract’s submit_date field changed

• Contract owner would be notified if a supervisor approve his/her superviserelation

§3.7 Business Logic 23

1 from django.dispatch import receiver, Signal2 from research_mgt.models import Contract, ActivityLog3

4 CONTRACT_SUBMIT = Signal(providing_args=[’contract’, ’activity_log’])5

6 @receiver(CONTRACT_SUBMIT, dispatch_uid=’contract_submit’)7 def contract_submit_notifications(contract: Contract,8 activity_log: ActivityLog,9 **kwargs):

10

11 ... # Send email

(a) signals.py

1 from django.http import HttpResponse2 from rest_framework.decorators import action3

4 from .signals import CONTRACT_SUBMIT5

6 class ContractViewSet(...):7

8 ...9

10 @action(methods=[’PUT’, ’PATCH’], detail=True, serializer_class=...,11 permission_classes=default_perms + [...])12 def submit(self, request, pk=None) -> HttpResponse:13

14 ...15

16 CONTRACT_SUBMIT.send(sender=self.__class__, ...)17

18 ...

(b) views.py

Figure 3.11: Example Signal Implementation

24 Design and Implementation

1 COMP8755; 19S2, 20S1; u0000001; Ed, Admin; aiosdf; 60; 30; Admin Ed <admin@srpms.cecs.anu.edu.au>; Test Ted <test@srpms.cecs.anu.edu.au>; Dajie Yang <Dajie.Yang@anu.edu.au>;

Figure 3.12: Example CSV Export

• Contract owner would be notified if an examiner approve an assessment ofhis/her contract

• Contract owner would be notified if course convener approve/disapprove his/hercontract

• Supervisor would be notified if a contract nominated him/her as supervisor isbeing submitted

• Supervisor would be notified if an examiner he/she nominated disapprove anassessment

• Supervisor would be notified if course convener approve/disapprove a contracthe/she supervise

• Examiner would be notified if a contract him/her examine is being approvedby the supervisor that nominated him/her

• Course convener would be notified if all supervisors and assessments of acontract are approved

SRPMS also implements a special data model to record actions that would triggerthe sender. Normally in a data model, a foreign key must refer to another model.However, Django’s content type module supports generic foreign key that can beused for logging any kind of data model. For example, in one action log the foreignkey refers to the contract model, but other action log may refers to the model ofsupervise relation.

3.7.6 Data Export

While not presented in the normal workflow, the course convener need to exportthe current list of contracts to csv in order to integrate with the current system,and students also need to export the PDF version of contracts. For these purposes,SRPMS implemented two additional special actions. The first one is print, whichis only available for contracts that have been finalized. The contract would firstbe rendered to HTML, and then be converted to PDF using a third party libraryWeasyPrint. A sample PDF export is shown in Appendix 5. The other one is csvexport, which would export all current finalized contracts to csv. A sample csv exportis shown in Figure 3.12.

§3.8 Testing 25

1 from django.test import TestCase2

3 class TestContract(TestCase):4 def setUp(self) -> None:5

6 ... # set up data that would be reused7

8 def test_contract_valid(self):9

10 ... # test and assert

Figure 3.13: Example Test Class Implementation

3.8 Testing

Django provides a very useful testing library, as well as convenient test managementtools. Firstly, Django has the ‘python manage.py test‘ management command thatwould automatically detect and run all tests. This includes functions with prefixtest_* that belong to any class that inherits django.test.TestCase. Secondly,all test functions would be wrapped in a transaction, and would be reverted afterfinish. In this way, the test data would remain unchanged after a test, and can bere-used across multiple tests. Django would also automatically create and destory adedicated test database when test begins and finishes. An example test class is shownin Figure 3.13.

The other advantage of testing in Python is the coverage tool. coverage.py isa Python library that can act as a wrapper to run tests, and automatically calculatestest coverage. It would automatically count the total number of statements in eachfile, and then record the number of statements the test missed. The coverage reportfor SRPMS back-end is shown in Figure 3.14.

Tests of SRPMS back-end is mostly conducted in API level, with some othersupplementary tests for non-API functions. API tests would check if the responsecode and content is correct after the test data is being sent to API. Since API alsoincludes communication with data models, API tests covers the majority of functionsin SRPMS. Supplementary tests include email notification verification, and customcomponents behavior check (e.g. the serializer). In total, the current tests of SRPMSback-end reach 92% code coverage, as shown in Figure 3.14.

3.9 Angular Application

SRPMS use Angular to implement the client in front-end. The client is a single pageapplication that provides friendly user interface that bridges normal users to theback-end. Due to task separation, details of Angular client implementation wouldbe skipped in this report, but focus on how the client use token authentication toauthorize every request to the back-end.

26 Design and Implementation

1 Name Stmts Miss Cover2 --------------------------------------------------------------------------------3 accounts/__init__.py 0 0 100%4 accounts/admin.py 4 0 100%5 accounts/apps.py 3 0 100%6 accounts/authentication.py 8 0 100%7 accounts/migrations/0001_initial.py 10 0 100%8 accounts/migrations/0002_srpmsuser_display_name.py 4 0 100%9 accounts/migrations/__init__.py 0 0 100%

10 accounts/models.py 24 0 100%11 accounts/serializers.py 19 0 100%12 accounts/urls.py 10 0 100%13 accounts/views.py 51 11 78%14 manage.py 12 2 83%15 research_mgt/__init__.py 0 0 100%16 research_mgt/admin.py 14 0 100%17 research_mgt/apps.py 11 0 100%18 research_mgt/csv_export.py 48 4 92%19 research_mgt/filters.py 29 0 100%20 research_mgt/migrations/0001_initial.py 8 0 100%21 research_mgt/migrations/0002_create_group_permission.py 32 4 88%22 research_mgt/migrations/0003_create_assessment_template.py 20 2 90%23 research_mgt/migrations/0004_create_courses.py 23 2 91%24 research_mgt/migrations/0005_activity_actions.py 24 2 92%25 research_mgt/migrations/__init__.py 0 0 100%26 research_mgt/mixins.py 51 7 86%27 research_mgt/models.py 226 18 92%28 research_mgt/permissions.py 238 32 87%29 research_mgt/print.py 61 3 95%30 research_mgt/serializer_utils.py 31 5 84%31 research_mgt/serializers.py 171 6 96%32 research_mgt/signals.py 88 2 98%33 research_mgt/urls.py 19 0 100%34 research_mgt/views.py 254 16 94%35 --------------------------------------------------------------------------------36 TOTAL 1493 116 92%

Figure 3.14: Test Coverage in Back-end

§3.9 Angular Application 27

AngularO

ther

Com

pone

nts

GenerateRequest Handle RespondPrompt Login

HTT

P C

lient

SendRequest Process Respond

HTT

P In

tece

ptor

Yes

No

Have Acces Token Attach Token

Send Request

Yes HTTP 401?Have Refresh TokenAttempt TokenRefresh

No

Back-end

Figure 3.15: HTTP Interceptor in SRPMS Client

3.9.1 Authentication in Front-End

Requests from Angular client is being authorized by implementing an HTTP inter-ceptor service. Service in Angular is a special class that would be injected to a rangeof components by using Angular’s dependency injection mechanism. The HTTPinterceptor service is configured to be injected in every components, as such all HTTPrequests would be intercepted by this service. The service would then attach theaccess token to the HTTP header of every request.

However, the access token only lasts for a short period of time. On every loginsuccess, the back-end would return an access token and a refresh token. The accesstoken is currently configured to expire after 5 minutes, while the refresh token canlast for 24 hours. On token expired, the back-end would return HTTP 401, denotingthat the request is unauthorized. In this case, the HTTP interceptor service wouldattempt to refresh token, and attach the new access token to the original request. If

28 Design and Implementation

the token refresh attempt still returns HTTP 401, the HTTP interceptor service wouldprompt a login dialog on the current page. Figure 3.15 denotes this process.

3.10 Summary

In this chapter, we discuss the design and implementation of SRPMS, includingthe normal workflow this system is trying to achieve, the database schema for thedatabase, the authentication methods, and details business logic implementation.

In the next chapter, we will discuss how the system is being deployed, and howthe project manages environments.

Chapter 4

Deployment

In this chapter, we would discuss how the SRPMS project manages environmentsduring development in Section 4.1, and how the system obtains a valid SSL certificatein Section 4.2, with the last section (Section 4.3) presenting how SRPMS utilize GitLabCI.

4.1 Environments

The SRPMS project built a total of three environments in order to support threedifferent stages, i.e. develop, test, and deploy. All stages are supported by Dockerto ensure environment consistency and prevent dependency issue, with some stagesalso support local machine setup.

Within the three environments, the development one is built to provide a conve-nient debugging environment for developers. This environment can either be runusing docker, or on local machine directly. Figure 4.1 shows how containers areorganized for development environment. In this environment, the back-end sourcecode would be mapped directly into the Django container, where source code wouldbe monitored, and trigger a Django reload on every file modification. The front-endcode also does the similar mapping and monitoring, however, the compiled resultwould be sent to the data volume share between the Angular container and Nginxcontainer, and being served as static file.

The development environment can also be run on local machine, so that develop-ers can use IDE debugging tools to quickly identify issues.

On the other hand, the deployment environment is built for production deploy-ment. Figure 4.2 shows how containers are organized for this environment. Unlikethe development one, this environment itself does not map any host file into the con-tainer, for the purpose of making containers self-contained and increases portability.Also, different from the development environment, the network would not exposethe database port to the outer world, in order to present possible attacks. Moreover,static files and media files from Django would be served through Nginx to maximizethe serving speed.

SRPMS also has a test environment that mimics the deployment environment.To be specific, apart from the certbot container that is being used to obtain valid

29

30 Deployment

Docker

Networks

srpms_network(bridge)

5432, 8000, 8001

Services

django-gunicorn angular-client nginxdb-postgres

5432 80, 4438000

Volumes

db_postgres_volume angular_volume

./srpms-client./srpms

Figure 4.1: Docker Containers for Development Environment

Docker

Networks

srpms_network(bridge)

80, 443

Services

django-gunicorn angular-client nginxdb-postgres certbot

80, 4435432 8000

Volumes

db_postgres_volume angular_volume

static_volume

media_volume letsencrypt_volume

certbot_volume

/var/run/docker.sock

Figure 4.2: Docker Containers for Deployment Environment

§4.2 SSL certificate 31

SSL certificate, all other configurations are identical. The similarity between test-ing and deployment environment ensures that tests can reflect errors in productionenvironment as close as possible.

Among all three SRPMS environments, only the deployment development usesvalid an SSL certificate, with the other two using an invalid certificate generated forlocalhost.

4.2 SSL certificate

The SSL certificate plays an important role in decoupled web application. As men-tioned in Section 3.6 and Section 3.9.1, the front-end authorizes its request throughtoken authentication. This would effectively means un-encrypted transmission canresult the expose of access token, and allow a third person to use the interceptedtoken to authorize malicious request. For this reason, SRPMS is HTTPS only forproduction deployment, and would redirect all HTTP requests to HTTPS.

To support HTTPS, SRPMS use the Let’s Encrypt facility to obtain free SSL certifi-cates. Generally speaking, Let’s Encrypt requires the prove of control over a domain.In the case of SRPMS, a special file would be served in a special location of thedomain.

Let’s Encrypt is a non-profit certificate authority run by Internet SecurityResearch Group that provides X.509 certificates for Transport Layer Secu-rity encryption at no charge. The certificate is valid for 90 days, duringwhich renewal can take place at any time. [Let’s Encrypt, 2019]

This certificate obtaining process is automated in SRPMS by using certbot, a smallprogram that communicates with the Let’s Encrypt facility. As shown in Figure 4.2,certbot have its own container, with isolated network to ensure max security. Duringthe first time set up, certbot would communicate with Let’s Encrypt to negotiatea special file, which would then be shared to Nginx and serve to public. Let’sEncrypt would then check if the file content is valid, and issue the correspondingcertificate for the domain. The certificate would be stored to the volume beingshared between Nginx and certbot container, and trigger certbot to inform Nginx toreload configurations. Specially, for the first time set up, Nginx would first generatean invalid certificate for localhost if no certificate is detected, and reload the validcertificate once certbot container informs.

4.3 Continuous Integration with GitLab

The SRPMS project is hosted on CECS GitLab, and use GitLab CI for DevOps. GitLabCI support easy pipeline configuration through including a .gitlab-ci.yml fileas pipeline configuration in project root directory. The pipeline configuration firstlyspecifies stages a pipeline includes, then specifies jobs a stage would execute. Jobsare essentially shell scripts, with each statement in a bullet point. Jobs belong to

32 Deployment

1 stages:2 - build3 - test4 - clean5 - ...6

7 job_01:8 stage: build9 script:

10 - echo start11 - ...

Figure 4.3: Example GitLab CI Configuration

Figure 4.4: SRPMS CI Pipeline Example

the same stage can be executed in parallel. Figure 4.3 shows an example GitLab CIconfiguration.

The SRPMS currently employs 5 stages for a pipeline, as shown in 4.4. Withinthese 5 stages, only build, test, clean would be executed on non-master branches. Thisensures every branch would be thoroughly tested, while only the master branch canaffect the production environment. In detail, the build stage would attmpt to buildall docker images, with the test stage conducting tests in containers instantiated fromdocker images built in the previous stage. At last, regardless the success or failureof the test stage, all containers, data volumes, and networks created during the teststage would be purged.

During development, CI brings the benefit of minimizing bugs. During the execu-tion of a pipeline, failure of a job would cause the failure of the entire pipeline, andthe developer who pushed the commit would be informed by GitLab automatically.With the help of CI, we can now give the promise that the code would functioncorrectly outside of development environment.

At deployment stage, CI allows the code being deployed automatically afterpipeline passed, eliminating risks introduced by manual operation, and enable prod-uct to be delivered in a much faster pace.

4.4 Summary

In this chapter, we discussed how SRPMS is being deployed, ranging from how theenvironment is managed, how the SSL certificate is initialized, to the use of GitLabCI. For detail deployment manual, please refer to Appendix 5

In the next chapter, we would make a conclusion for this project, and discuss

§4.4 Summary 33

future works.

34 Deployment

Chapter 5

Conclusion and Future Works

At this stage, the SRPMS fully implemented all necessary APIs for accomplishingthe normal workflow. The system put thorough consideration on business logic toenforce strict permission and data integrity. Additional, the system uses modernpractices, including the use docker and CI, to efficiently manage environments andcontrol software quality.

The system is built from scratch in three months by two developers with little mod-ern web development experience, but enthusiastic for learning the current practices.Throughout this process, I developed more recognition of the importance of softwaretesting and continuous integration. The time I spent in building CI pipeline andwriting test turns out to worth much more that I thought, and allows the back-end toiterate in a much faster pace in this tight schedule.

However, the current implementation of SRPMS still has ample amount of spaceto improve. For example, most text representation still use primary key to representrelated resources, but hyper-link is more preferred. Also, API currently use nestedrouter to represent dependent resources, but this is not recommended in RESTfulAPI design.

In the future, the system has a plan to support examiner review at the end of everysemester. In this way, the RSCS staff can process the entire course administrationprocess digitally, i.e. from contract initialization to final review within one system,and save time from tedious manual operations.

35

36 Conclusion and Future Works

Bibliography

AV, S., 2019. Perks of a decoupled architecture. https://medium.com/@aliasav/perks-of-a-decoupled-architecture-fb6983890ed0. (cited on page 3)

Fielding, R. T., 2016. Architectural styles and the design of network-based softwarearchitectures. university of california, irvine, 2000. (cited on page 4)

Let’s Encrypt, 2019. About let’s encrypt. https://letsencrypt.org/about/. (cited on page31)

Loukides, M., 2014. Revisiting âAŸwhat is devopsâAZ. OReilly Radar, (2014). (citedon page 5)

Netcraft, 2019. September 2019 web server survey. https://news.netcraft.com/archives/category/web-server-survey/. (cited on page 4)

Nginx, 2015. Nginx vs. apache: Our view of a decade-old question. https://www.nginx.com/blog/nginx-vs-apache-our-view/. (cited on page 4)

Robenolt, M., 2013. Scaling django to 8 billion page views. https://blog.disqus.com/scaling-django-to-8-billion-page-views. (cited on page 5)

Viswanathan, G.; Mathur, P. D.; and Yammiyavar, P., 2009. From web 1.0 to web2.0 and beyond: Reviewing usability heuristic criteria taking music sites as casestudies. Hämtad februari, 27 (2009), 2015. (cited on page 3)

W3C, 2004. Web services architecture. https://www.w3.org/TR/2004/NOTE-ws-arch-20040211/#relwwwrest. (cited on page 4)

37

38 Bibliography

Appendices

39

Appendix

Study Contract of SRPMS Project

41

44 Study Contract of SRPMS Project

Appendix

Sample PDF Export for FinalizedContract in SRPMS

45

Independent Study Contract

Section A (Student and Supervisors)

UniID: u0000001

SURNAME: Ed FIRST NAME: Admin

PROJECT SUPERVISOR: Admin Ed

FORMAL SUPERVISOR: Admin Ed

COURSE: COMP8755 - Individual Computing Project (12 units)

COMMERCING SEMESTER YEAR: 2019 SEMESTER: 2 Two-semester project (12u courses only): Yes

PROJECT TITLE:

aiosdf

LEARNING OBJECTIVES:

asdoiusiodfoqieuroiadfioauodsifu

PROJECT DESCRIPTION:

ASSESSMENT:

Assessed project components % of mark Due date Examiner

presentation 10 Admin Ed

artifact 30 Dajie Yang

report 60 Test Ted

Research School of Computer Science Page 1 of 2

STUDENT DECLARATION:

I agree to fulfill the above defined contract.

Student: Admin Ed Date: Oct. 13, 2019

Section B (Supervisor and Examiner)

I am willing to supervise and support this project. I have checked the student's academicrecord and believe this student can complete the project. I nominate the followingexaminer, and have obtained their consent to review the report.

Supervisor: Admin Ed Approval Date: Oct. 13, 2019

Examine Item: presentation

Examiner: Admin Ed Approval Date: Oct. 13, 2019

Examine Item: artifact

Examiner: Dajie Yang Approval Date: Oct. 13, 2019

Examine Item: report

Examiner: Test Ted Approval Date: Oct. 13, 2019

(Nominated examiners may be subject to change on request by the supervisor or courseconvenor)

REQUIRED DEPARTMENT RESOURCES:

Section C (Course convenor approval)

Convener: Admin Ed Approval Date: Oct. 13, 2019

Research School of Computer Science Page 2 of 2

48 Sample PDF Export for Finalized Contract in SRPMS

Appendix

Contribution Statement

I declared that the content of this report, unless otherwise specified, correspondsand based on my contribution to the source code of this project. Details of files anddirectories I contributed are listed below.

./+-- certbot/ 100%+-- config.dev/ 100%+-- config.prod/ 100%+-- docs/| +-- AgileOverPlanDriven.md 100%| +-- AgilePhasePlan.md 100%| +-- Back-end.md 100%| +-- Deployment.md 100%| +-- example_coding_style.py 100%| +-- meetings.md 75%| +-- relational-model.md 70%| +-- ...+-- gitlab-ci/ 100%+-- nginx/ 100%+-- srpms/ 90%+-- srpms-client/| +-- src| | +-- app| | +-- contract-form/ 8%| | +-- contract-form-element/ 8%| | +-- header/ 40%| | +-- login-dialog/ 100%| | +-- accounts.service.spec.ts 100%| | +-- accounts.service.ts 100%| | +-- api-url.ts 100%| | +-- auth-interceptor.service.spec.ts 100%| | +-- auth-interceptor.service.ts 100%| | +-- element.service.ts 15%

49

50 Contribution Statement

| | +-- element-user.ts 100%| | +-- reseach_mgt-objects.ts 100%| | +-- ...| +-- ...+-- tests/ 100%+-- dev_setup.sh 100%+-- docker-compose.dev.yml 100%+-- docker-compose.prod.base.yml 100%+-- docker-compose.prod.deploy.yml 100%+-- docker-compose.prod.test.yml 100%+-- README.md 50%

Files and directories that are not included in the above diagram, as well as thecomplement part to 100%, are contributed by my co-worker Euikyum Yang.

Appendix

Deployment Guide 1

1Export from markdown documentation of the project repository

51

SSL CertificateSpecial note for Let's Encrypt certificate

Setup local development environmentRunning tests manually

Deploy - ProductionArchitecture overviewCECS VM Clean upInstall dockerConfigure iptableConfigure repository for group accessCI/CD pipeline setup

Secrets for testingSecrets for depolymentSetup pipeline runner

Manually start the serviceCleaning

Deploy - DevelopmentThrough dockerOn local machine

Front-end Back-end communicationAccess ANU LDAP outside campusAttach to a running docker containerAccessing API directlyDatabase migration

Import existing database dumpInitialize database schemaReverse migrationModify existing schemaManually clear database

ReferenceDockerSSL certificatesCI/CDServer iptables ruleMisc.

52 Deployment Guide

SSL Certificate NOTE: This section should be read before attempting any deployment

The project have below configurations for different deployment methods

Production deployment: a free and valid SSL certificate forsrpms.cecs.anu.edu.au would be issue by Let's Encrypt, the certificaterequest/update and configuration process is autonomous.Development deployment: a invalid SSL certificate for domain localhost wouldbe generated by OpenSSL command line tool, the configuration process isautonomousTesting: a invalid SSL certificate for domain localhost would be generated byOpenSSL command line tool, the configuration process is autonomous

Special note for Let's Encrypt certificate

The certificate Let's Encrypt can issue per Registered Domain is 50 per week

For example, api.srpms.cecs.anu.edu.au and srpms.cecs.anu.edu.auwould be in the same registered domain.Though we only have one domain currently, the rate limit is still worthmentioning.

The renewal of an existing certificate is 5 times per week

For example, once srpms.cecs.anu.edu.au obtain a certificate, it can onlybe renew 5 times each weekEach renewal is valid for 90 days

For more information, refer to Let's Encrypt's rate limit

Setup local developmentenvironment

We recommend using miniconda to manage your local environment.

53

In project root directory, the dev_setup.sh file is provided for setting up the localenvironment with miniconda, but please be aware that it only supports Debian-based Linux distribution, as it relies on apt-get for some dependencies.

To setup environment, simply run ./dev_setup.sh , it'll prompts you for necessaryinputs.

Running tests manually

To run test for back-end

To run test for front-end

Deploy - Production NOTE: Make sure the production machine have a public IP

NOTE: Production deployment does not support server that is outside ANUnetwork

Architecture overview

NOTE: For the below diagram, the swimlane Services can be regarded asContainers . A service defined in docker compose can have multiple container ofthe same docker image, however in our case each service is only configured to haveone container.

cd srpms

python manage.py test

cd srpms-client

ng test

54 Deployment Guide

In production deployment, file system mapping (i.e. bind mount) between thephysical machine and docker container should be avoided, as the deploymentmight be executed within other docker container. For more details, refer tomanage data in docker.

The /var/run/docker.sock is a unix socket that can be used to access dockerin the host machine, the certbot service has this mapping so that it can reloadnginx service when SSL certificate update.

The srpms_network expose port 80 and 443 to the outside network, bothports are mapped to the same ports on nginx service

The db-postgres service expose port 5432 internally, so that other containerinside srpms_network can access the database

The django-gunicorn service expose port 8000 internally, and nginx wouldforward API request there

The certbot container is not part of the srpms_network by design, instead, itconnect directly using the host machine's network.

NOTE: The certbot service exist solely because the website is currentlyusing free SSL certificate provided by Let's Encrypt. Though ideally it should

55

use SSL certificate issue by ANU.The angular-client service does not have any network connectivity by design,the service would exist automatically after it finished compiling the angularsource code.

The compile result would be copied to the data volume shared betweennginx and angular-client , i.e. the compile result is hosted by nginx .

CECS VM Clean up

Note that the provided VM currently has many unrelated packages installed, we'reNOT going to clean them all, but just free up ports as much as possible.

Install docker

# For srpms.cecs.anu.edu.au only

# Since we are using docker, we'll disable un-necessary services

sudo systemctl disable nginx.service

sudo systemctl stop nginx.service

sudo systemctl disable postgresql.service

sudo systemctl stop postgresql.service

sudo pkill -U tomcat8

sudo apt-get update

# Install packages to allow apt to use a repository over HTTPS

sudo apt-get install \

apt-transport-https \

ca-certificates \

curl \

gnupg2 \

software-properties-common

# Add Docker’s official GPG key

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-

key add -

# Setup the stable source for docker binary

56 Deployment Guide

Configure iptable

Please refer to Server iptables rule section at the bottom of this file for iptablerules

Configure repository for group access

Sometimes CI runner cannot take care of all the situations, and you need to dothings manually. This section is for this purpose.

sudo add-apt-repository \

"deb [arch=amd64] https://download.docker.com/linux/debian \

$(lsb_release -cs) \

stable"

# Update apt and install docker

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

# At this stage, you might want to put your linux user inside the

'docker' group, otherwise sudo privilege is required for using docker

command

# Enter swarm mode since we need to use docker secrets

docker swarm init

sudo apt-get install iptables-persistent

# Put the iptables rule given in following section under

corresponding folder

#mv iptable-rules /etc/iptables/rules.v4

# Reload docker's its iptables rules

sudo systemctl restart docker.service

# Reload the configuration

sudo systemctl restart netfilter-persistent

57

CI/CD pipeline setup

NOTE: secrets still work even if you configure this on the local machine, and usedocker inside a container through socket passing, and does not require any specialconfiguration

Secrets for testing

NOTE: You can do things in this section on a different machine (in the casethat the production server is not suitable for running test for some reason)

The project use GitLab's shell executor with docker runner for this purpose. Below isthe steps to configure necessary files and parameters.

# Create folder at root for group access

sudo mkdir /srpms

sudo addgroup srpms

sudo chown root:srpms /srpms

sudo chmod g+w /srpms

# Download source code

git clone https://gitlab.cecs.anu.edu.au/u6513788/comp8755-srpms.git

.

# Configure git for group access

git config core.sharedRepository group

# Change file ownership and permission to allow group access

chgrp -R srpms .

chmod -R g+w .

# Git pack files should be immutable

chmod g-w objects/pack/*

# Configure new file to inherit directory's group id

find -type d -exec chmod g+s {} +

# cd to the runner image dir

cd gitlab-ci

# Register first

docker-compose run runner register

58 Deployment Guide

About django_test_ldap_username and django_test_ldap_password :

Must be a valid ANU username and passwordEvery time user login using ANU credential, Django would pop user informationfrom ANU LDAP, this test is to make sure that user info popping is workingcorrectly.

NOTE: If your test machine is outside ANU network, you'll need to set up a sshtunnel, refer to Access ANU LDAP outside campus for instructions.

Secrets for depolyment

# Start the runner

docker-compose up -d

# Create directory for storing secrets, and give group access

sudo mkdir /srpms-secrets-test

sudo chown root:docker /srpms-secrets-test

sudo chmod 750 /srpms-secrets-test

sudo chmod g+s /srpms-secrets-test

# Set up secrets for testing (on your host machines)

sudo printf '<secret content>' > /srpms-secrets-test/postgres-db.txt

sudo printf '<secret content>' > /srpms-secrets-test/postgres-

user.txt

sudo printf '<secret content>' > /srpms-secrets-test/postgres-

passwd.txt

sudo printf -- '<secret content>' > /srpms-secrets-test/postgres-

init-args.txt

sudo printf '<secret content>' > /srpms-secrets-

test/django_secret_key.txt

# For testing LDAP connection and user property mapping, you'll need

to set up username and password on you test machine, not setting

these two variables would cause test error.

sudo printf '<secret content>' > /srpms-secrets-

test/django_test_ldap_username.txt

sudo printf '<secret content>' > /srpms-secrets-

test/django_test_ldap_password.txt

59

Docker secrets provide a convenient way for providing sensitive information (e.g.username, password of the database). You don't want to include these informationinside the git repository.

Setup pipeline runner

NOTE: For running testing pipeline, please follow Access ANU LDAP outsidecampus to setup ssh tunnel, even if you're inside ANU network.

# Create directory for storing secrets, and give group access

sudo mkdir /srpms-secrets

sudo chown root:srpms /srpms-secrets

sudo chmod 750 /srpms-secrets

sudo chmod g+s /srpms-secrets

# Create secrets

# NOTE: DO NOT USE "echo" AS IT WOULD SUFFIX NEW LINE CHARACTER

# The below information would be used to initializa postgres database

(if the database does not exist), and would be used by Django to

access the database

# Database name

sudo printf '<secret content>' > /srpms-secrets/postgres-db.txt

# Database username

sudo printf '<secret content>' > /srpms-secrets/postgres-user.txt

# Database password

sudo printf '<secret content>' > /srpms-secrets/postgres-passwd.txt

# "--" would prevent "--" raise error when its at the start of the

content string

sudo printf -- '<secret content>' > /srpms-secrets/postgres-init-

args.txt

# Django secret key is used for various purpose, for example, hashing

user's password so that they won't store in plain text, exposure of

this secret key would result all user's password being compromised.

sudo printf '<secret content>' > /srpms-secrets/django_secret_key.txt

60 Deployment Guide

On the machine that you wants to run the pipeline:

1. Change working directory to CI

cd gitlab-ci

2. Run the register command:

docker-compose run runner register

3. Enter GitLab instance URL:

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com

)

https://gitlab.cecs.anu.edu.au

4. Enter the token you obtained to register the Runner (go to Settings -> CI/CD -> Runner -> Specific Runners to obtain the token for your repository)

Please enter the gitlab-ci token for this runner:

xxx

5. Enter a description for the Runner, you can change this later in GitLab’s UI:

Please enter the gitlab-ci description for this runner:

[hostname] my-runner

6. Enter the tags associated with the Runner, you can change this later in GitLab’sUI:

Please enter the gitlab-ci tags for this runner (comma separated):

my-tag,another-tag

For production deployment, please enter comp8755, srpmsFor running test, please enter comp8755, cooper

7. Enter the Runner executor:

61

Please enter the executor: ssh, docker+machine, docker-ssh+machine,

kubernetes, docker, parallels, virtualbox, docker-ssh, shell:

shell

8. The previous step would exit after finished, then you need to start the runnermanually

docker-compose up -d

Manually start the service

This section covers how to start the back-end from command line through docker.For production deployment, it currently only support docker, in order to guaranteethe consistency of its behavior. Refer to Deploy - development section if you want torun without docker.

# For simplicity, we wrap the docker command to a shell variable

COMPOSE_DEPLOY='docker-compose -f docker-compose.prod.base.yml -f

docker-compose.prod.deploy.yml'

# Start up

$COMPOSE_DEPLOY up -d

# Stop

$COMPOSE_DEPLOY stop

# Check logs, append '-f' for following

$COMPOSE_DEPLOY logs

# Shutdown and remove containers

$COMPOSE_DEPLOY down

Cleaning

CAUTION: removing the letsencrypt_volume would effectively remove thecertificate issued by Let's Encrypt, and Let's Encrypt has a rate limit forobtaining new certificates

62 Deployment Guide

$COMPOSE_DEPLOY down given by the previous section would not remove dockerimages, data volumes, and networks

That is, the data of the database, issued certificates, static files, etc. would stillexist, only the docker container (i.e. services) would be removedThe srpms_network also continue to exist.The docker image of corresponding service would exist as well. Recall thatcontainers are instantiated docker image.

To see the list of docker data volumes: docker volume ls

To see the list of docker networks: docker-network ls

To clean all data volume, network, images, and containers: $COMPOSE_DEPLOY down--rmi 'local' -v --remove-orphans

Deploy - Development NOTE: development deployment is only meant to be accessed throughlocalhost

Through docker

63

The srpms_network expose port 5432 , 8000 and 8001 to the outside network,of which 5432 map to 5432 port of db-postgres service, 8000 map to 80 portof nginx service, 8001 map to 443 port of nginx service

The django-gunicorn service expose port 8000 internally, and nginx wouldforward API request there

./srpms refers to the folder (that contains Django source code) locates on yourphysical machine, in development mode, Django would automatically reloadwhen it detect changes on any file within this directory, so that your latestmodification can be applied instantly.

./srpms-client refers to the folder (that contains Angular source code) locateson your physical machine, in development mode, Angular would automaticallycompile the new source code when it detect changes on any file within thisdirectory, so that your latest modification can be applied instantly.

However, you still need to refresh the page, or possibly reload all resources( Ctrl + F5 for chrome based browsers) in order to see new changes

64 Deployment Guide

angular-client runs ng build --watch to produce front-end code, the compileresult is hosted by nginx through a shared data volume

django-gunicorn runs gunicorn --reload --bind :8000 to serve back-endcontent (including back-end static and media files)

Please make sure you are under the project root directory when usingfollowing commands

# If your machine is inside campus, use ldap://ldap.anu.edu.au,

otherwise refer to the 'Access ANU LDAP outside campus' section

export LDAP_ADDR='<ANU_LDAP_ADDR>'

# Build images if this is your first run

docker-compose -f docker-compose.dev.yml build

# Start, will run in background

docker-compose -f docker-compose.dev.yml up -d

# To apply changes of Dockerfile and compose file to running

containers, use the following command (remove --build if you did not

change any Dockerfile)

docker-compose -f docker-compose.dev.yml up -d --build

# Stop containers

docker-compose -f docker-compose.dev.yml stop

# This command will remove container, network, and volumes, which means

the database would be removed as well

docker-compose -f docker-compose.dev.yml down --rmi 'local' -v --

remove-orphans

On local machine

Having everything running inside docker can sometimes make debug very painful,as you won't have IDE debugging tools support, e.g. breakpoint. As such wesometimes need to run code outside container.

To run Django on local machine:

65

export DEBUG=True

# NOTE: Activate your conda environment first

# Run the database

docker-compose -f docker-compose.dev.yml up -d db-postgres

cd srpms

python manage.py runserver

Please note that HTTPS settings are defined in a way that would not beapplied if you run Django in local machine.Please note that angular-client and nginx is depend on django-

gunicorn , and they would not be able to start if you run Django locally.To run Angular on local machine, with everything else running inside docker

export DEBUG=True

# NOTE: Activate your conda environment first

# Run the database

docker-compose -f docker-compose.dev.yml up -d

docker-compose -f docker-compose.dev.yml stop angular-client

cd srpms-client

ng serve

To run both Angular and Django on local machine

export DEBUG=True

# NOTE: Activate your conda environment first

# Run the database

docker-compose -f docker-compose.dev.yml up -d db-postgres

# Run Django

cd srpms

python manage.py runserver

66 Deployment Guide

# Please do this in a separate terminal, remember to `export

DEBUG=True`

cd srpms-client

ng serve

Front-end Back-end communication

The API URL by default is set (i.e. hard-coded) to /api/ in Angular client code.However:

When you run angular outside of docker container, ng serve would by defaultruns on localhost:4200 , while the docker-compose service is configured torun at localhost:8000 (HTTP) and localhost:8001 (HTTPS).

You'll need to change API_URL temporarily in order to point to correct API url,localhost:8001/api/ in this case.

API_URL variable is defined in srpms-client/src/app/api-url.ts

NOTE: Please change the variable back before you commit!If you are running Django locally, the address is normallylocalhost:8000/api/ , unless you configure otherwise

Access ANU LDAP outside campus If you're not using the ANU-Secure WIFI (or any other network provided by ANU), theANU LDAP server would be invisible for you. In this case, you'll need to configurethe connection to ANU LDAP server yourself. Here we give an example ofconnection ANU LDAP server from the outside through an SSH Tunnel that connectto a machine located inside ANU network.

67

DOCKER_GATEWAY="$(docker network inspect bridge --format='{{(index

.IPAM.Config 0).Gateway}}')"

export LDAP_ADDR="ldap://$DOCKER_GATEWAY"

# Make sure your ssh connection is alive when the container is

running

# Port 389 is in the range of 1~1024, as such we need sudo

privilege.

sudo ssh -L "$DOCKER_GATEWAY":389:ldap.anu.edu.au:389

<UniID>@srpms.cecs.anu.edu.au

You also need to make sure your iptables allow incoming traffic from the srpmsnetwork subnet, otherwise connections from the container would be blockedand won't reach the ssh tunnel.

For Linux:

sudo iptables -A INPUT -d $(docker network inspect bridge --

format='{{(index .IPAM.Config 0).Subnet}}') -p tcp -m tcp --

dport 389 -j ACCEPT

ANU LDAP service is available at

url: ldap://ldap.anu.edu.au

port: 389

Attach to a running dockercontainer

Attach to a running container can be useful when you need to do manualoperations

For example, re-applying database migration, or

To attach to a running container, use docker-compose -f docker-

compose.dev.yml exec <service_name> <command>

68 Deployment Guide

For example, to attach to the Django container for trouble shooting, usecommand docker-compose -f docker-compose.dev.yml exec django-

gunicorn /bin/bash

To run a single container, use docker-compose -f <compose file> run

<service_name> <command>

NOTE: docker-compose file contains variables required to run the dockercontainer, regular docker run does not. Please don't start these dockercontainers with plain docker command.

Accessing API directly Currently, not all functionalities provided by the API have been supported by thefront-end.

For example, contract printing and contract csv export, and the modification ofcourse and assessment template is not yet supported.

Using the API directly:

To access API directly, enter the url <host_addr>[:<port>]/api/ , port isoptional depending how you deploy the project

You can authenticate yourself by assessing api/accounts/login/ , and loginwith your credential, all access to the API would be authenticated afterwards.

api/accounts/ contains authentication related API

api/research_mgt/ contains API for contract management

The top of the page shows your current location, in this example is ApiRoot/Api Root/Contract List/Contract Instance/Assessment

List/Assessment Instance/Assessment Examine List/Assessment Examine

Instance

69

Locations like Contract List , or Assessment List is called list view, asthey contain a list of all instances (note that the list of instances might varydepending on user's privilege)

Locations like Contract Instance , or Assessment Instance is call detailview, and only shows a particular instance

For example, api/contracts is a list view, api/contracts/1/ is a detailview.

Functionality like print , submit , and approve are only available atdetail view levelFunctionality like export_csv is only available at list view level

The view provides documentation about the functionalities available for thisview and its detail view, for example

NOTE: Currently, the browsable API provided by REST framework is enabled in alldeployment settings, for the convenient of demonstration. But it should be disabledin production.

Database migration

70 Deployment Guide

CAUTION: ONLY certified person should be allowed to interact with databasedirectly, any user, including course convener, should not be allowed toperform operations in this section.

CAUTION: Direct interaction with database would effectively bypassing allsecurity check, ONLY certified person should be allowed to do this.

NOTE: Make sure you have already deployed before reading this section.

The project uses Django ORM to manage data in the relational database. Unlessunder extreme circumstances, database should NOT be accessed directly.

What is database migration (from Django's official documentation about migration)

Migrations are Django’s way of propagating changes you make to your models(adding a field, deleting a model, etc.) into your database schema. They’redesigned to be mostly automatic, but you’ll need to know when to makemigrations, when to run them, and the common problems you might run into

Import existing database dump

If you already have a database dump named <db_dump> , you can import it to thedatabase container by

1. Copy to the database container docker cp <db_dump> srpms_db-postgres:/

2. Import to the database (make sure the db-postgres service is running first)docker-compose -f <compose file> exec db-postgres psql -U <db_name> <

/<db_dump>

Initialize database schema

Although this would be done automatically in production deployment, you mightneed to do this from time to time in development environment

docker-compose -f <compose file> run django-gunicorn python manage.py

migrate

Reverse migration

71

If you already have data inside the database

You might run into trouble if you already have data (e.g. Contract ,ActionLog , ...) in the database, and this is the expected behavior to preventaccidental mis-operation. In this case, you need to first clear all existingdata.

from research_mgt.models import Contract, ActivityLog

Contract.objects.all().delete()

ActivityLog.objects.all().delete()

NOTE: while it is specified in Django ORM that IndividualProject wouldbe CASCADE if the Contract it refers to is being deleted, this constraintexist solely on Django ORM. Attempt to delete Contract through databaseshell would not result its related IndividualProject being deleted.

Follow the normal reverse steps

Normal reverse steps:

# To reset all migration given an app (e.g. accounts,

research_mgt)

python manage.py migrate <app_name> zero

# To reverse migrations to certain step, migrate_num can be

0001, 0002,

# according to the migrates created for the app.

# For example, research_mgt has a total of 5 migrations, then

# 'migrate research_mgt 0004' would migrate it to a stage where

0001-0004

# have been migrated, but 0005 hasn't

python manage.py migrate <app_name> <migrate_num>

Modify existing schema

At the time you are reading this documentation, the database schema should bematured, and it is very likely that there are already some data in the database

CAUTION: Please note that modifying database schema would affect all existingdata

72 Deployment Guide

CAUTION: Any modification to the database schema should be done throughDjango ORM, i.e. change the models.py file, modify the database schemadirectly using database shell should be avoided in most circumstance.

Steps:

After you modify the models.py file, run python manage.pymakemigrations , which would automatically detects any change in modelsfile, an generate new migration in migrations directory.

Issue python manage.py migrate to apply new migrations

You might be asked to provide input for default values based on yourchange

Manually clear database

When switching between different branches, you might run in to trouble thatthe migrations in the current branch does not match the migrations you appliedfor the database. In this case, you'll likely to run into situation that all migrationattempt would failed.

Below is an example of how to manually clear migrations related toresearch_mgt

1. Run python manage.py shell , and then

# Clear data through django ORM

from research_mgt.models import ActivityLog, Contract

ActivityLog.objects.all().delete()

Contract.objects.all().delete()

2. Run python manage.py migrate --fake research_mgt zero to reset themigration record

3. Run python manage.py dbshell to manually clear schema:

73

-- Order is important

DROP TABLE research_mgt_activitylog;

DROP TABLE research_mgt_activityaction;

DROP TABLE research_mgt_individualproject;

DROP TABLE research_mgt_specialtopic;

DROP TABLE research_mgt_assessmentexamine;

DROP TABLE research_mgt_examine;

DROP TABLE research_mgt_assessment;

DROP TABLE research_mgt_assessmenttemplate;

DROP TABLE research_mgt_supervise;

DROP TABLE research_mgt_contract;

DROP TABLE research_mgt_course;

4. Because research_mgt 's migration include Permissions and Groupobjects, we need to clear these two schema as well:

# This would also have the side effect of removing all existing

users

python manage.py migrate auth zero

5. Reapply all migration

python manage.py migrate

Reference How To Get Angular and Nginx Working Together Properly for Development

Docker

Environment Variable priorities when using compose

1. Compose file2. Shell environment variables3. Environment file4. Dockerfile5. Variable is not defined

74 Deployment Guide

Building Django Docker Image with Alpine

Docker Compose with NginX, Django, Gunicorn and multiple Postgres databases

Understanding Docker Networking Drivers and their use cases

Deploying Gunicorn

Deploying nginx + django + python 3

Why Your Dockerized Application Isn’t Receiving Signals

Trapping Signals in Docker Containers

Using docker-compose with CI - how to deal with exit codes and daemonized linkedcontainers?

SSL certificates

To generate a self-signed SSL certificate for localhost (for development purpose)

openssl req -x509 -out localhost.crt -keyout localhost.key \

-newkey rsa:2048 -nodes -sha256 \

-subj '/CN=localhost' -extensions EXT -config <( \

printf "[dn]\nCN=localhost\n[req]\ndistinguished_name =

dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\next

endedKeyUsage=serverAuth")

This self-sign certificate would not accept by chrome, as such, you need to go tochrome://flags/#allow-insecure-localhost , and set it to enable

How to Setup a SSL Certificate on Nginx for a Django Application

Nginx and Let’s Encrypt with Docker in Less Than 5 Minutes

Certificates for localhost

CI/CD

Getting started with GitLab CI/CD

Run GitLab Runner in a container

75

Register Runners

Using components of Auto-DevOps

Server iptables rule

# Title: iptables rules for SRPMS Server

# Author: Dajie Yang

# Last Modify: Aug 23 2019

################

# Mangle Table #

################

# *mangle

################

# Filter Table #

################

*filter

:INPUT DROP [0:0]

:FORWARD DROP [0:0]

:OUTPUT ACCEPT [0:0]

:addrtype-general - [0:0]

:anti-scan - [0:0]

:app-limit - [0:0]

:app-reject - [0:0]

# Allow all loopback traffic

-A INPUT -i lo -j ACCEPT

-A OUTPUT -o lo -j ACCEPT

# Quickly process packets for which we already have a connection

-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Drop INVALID packets

-A INPUT -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-

burst 10 -j LOG --log-prefix "[INVALID] "

-A INPUT -m conntrack --ctstate INVALID -j DROP

# Ok icmp codes for INPUT

-A INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT

76 Deployment Guide

-A INPUT -p icmp --icmp-type source-quench -j ACCEPT

-A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT

-A INPUT -p icmp --icmp-type parameter-problem -j ACCEPT

-A INPUT -p icmp --icmp-type echo-request -j ACCEPT

# Ok icmp code for FORWARD

-A FORWARD -p icmp --icmp-type destination-unreachable -j ACCEPT

-A FORWARD -p icmp --icmp-type source-quench -j ACCEPT

-A FORWARD -p icmp --icmp-type time-exceeded -j ACCEPT

-A FORWARD -p icmp --icmp-type parameter-problem -j ACCEPT

-A FORWARD -p icmp --icmp-type echo-request -j ACCEPT

-A FORWARD -p icmp --icmp-type echo-reply -j ACCEPT

# Allow DHCP client to work

-A INPUT -p udp --sport 67 --dport 68 -j ACCEPT

# Procss special address, generally

-A INPUT -j addrtype-general

-A addrtype-general -m addrtype --dst-type LOCAL -j RETURN

-A addrtype-general -m addrtype --dst-type MULTICAST -j RETURN

-A addrtype-general -m addrtype --dst-type BROADCAST -j RETURN

-A addrtype-general -m addrtype --dst-type UNICAST -j DROP

-A addrtype-general -m limit --limit 3/min --limit-burst 10 -j LOG --

log-prefix "[SPECIAL ADDR] "

-A addrtype-general -j DROP

# allow MULTICAST mDNS for service discovery

-A INPUT -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT

# allow MULTICAST UPnP for service discovery

-A INPUT -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT

# Blocking port scan

-A INPUT -p tcp --tcp-flags ACK,FIN FIN -j anti-scan

-A INPUT -p tcp --tcp-flags ACK,PSH PSH -j anti-scan

-A INPUT -p tcp --tcp-flags ACK,URG URG -j anti-scan

-A INPUT -p tcp --tcp-flags ALL ALL -j anti-scan

-A INPUT -p tcp --tcp-flags ALL NONE -j anti-scan

-A INPUT -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j anti-scan

-A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j anti-scan

-A INPUT -p tcp --tcp-flags FIN,RST FIN,RST -j anti-scan

-A INPUT -p tcp --tcp-flags ALL SYN,FIN -j anti-scan

-A INPUT -p tcp --tcp-flags ALL URG,PSH,FIN -j anti-scan

-A INPUT -p tcp --tcp-flags ALL FIN -j anti-scan

-A INPUT -p tcp --tcp-flags ALL URG,PSH,SYN,FIN -j anti-scan

-A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j anti-scan

77

-A anti-scan -m limit --limit 3/min --limit-burst 3 -j LOG --log-prefix

"[PORT SCAN] "

-A anti-scan -m recent --set --name PORT-SCAN --mask 255.255.255.255 --

rsource

-A anti-scan -m recent --update --seconds 30 --hitcount 10 --name PORT-

SCAN --mask 255.255.255.255 --rsource -j DROP

# Exception for applications

# SSH

-A INPUT -p tcp -m tcp --dport 22 -j LOG --log-prefix "[SSH] "

-A INPUT -p tcp -m tcp --dport 22 -j app-limit

# HTTP/HTTPS, does not currently apply any connection limit

-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

-A INPUT -p udp -m udp --dport 80 -j ACCEPT

-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

-A INPUT -p udp -m udp --dport 443 -j ACCEPT

# Limit connection frequency to 10 hits per 30s, after that block the

client for 5 minutes

-A app-limit -m conntrack --ctstate NEW -m recent --set --name APP-

LIMIT --mask 255.255.255.255 --rsource

-A app-limit -m conntrack --ctstate NEW -m recent --update --seconds 30

--hitcount 10 --name APP-LIMIT --mask 255.255.255.255 --rsource -j app-

reject

-A app-limit -j ACCEPT

-A app-reject -m limit --limit 5/min -j LOG --log-prefix "[APP REJECT]

"

-A app-reject -j REJECT --reject-with icmp-port-unreachable

COMMIT

# Done

Misc.

Keep exit codes when trapping SIGINT and similar?

Trapping Signals in Docker Containers

How to configure an existing git repo to be shared by a UNIX group

How do I parse command line arguments in Bash?

When do we need curly braces around shell variables?

78 Deployment Guide

 

79

80 Deployment Guide

Appendix

Back-End Design in File StructureLevel 2

2Export from markdown documentation of the project repository

81

File Structure srpms/

├── accounts # Account app│ ├── migrations/ # Database migration for this app│ ├── __init__.py│ ├── admin.py # Django admin page configuration│ ├── apps.py # App config and initialization│ ├── authentication.py # ANU LDAP Authentication backend│ ├── models.py # Database schema for this app│ ├── serializers.py # Data serializers for encode/decode JSON for API

│ ├── tests.py # Tests for this app│ ├── urls.py # How this app locate itself in url│ └── views.py # Views handle requests and generate response

├── research_mgt # Research management system app│ ├── migrations/ # Database migration for this app│ ├── static/ # Static resources for generate contract PDF

│ │ └── research_mgt/│ ├── templates/ # HTML templates for generate contract PDF

│ │ └── research_mgt/│ ├── tests/ # Tests for this app│ ├── __init__.py│ ├── admin.py # Django admin page configuration│ ├── apps.py # App config and initialization│ ├── filters.py # Filters that enable url query, e.g. '?key=value'

│ ├── mixins.py # Custom mixins to support nested url

│ ├── models.py # Database schema for this app│ ├── permissions.py # Permission control for request sender

│ ├── print.py # Print contract to PDF

82 Back-End Design in File Structure Level

Django project overview This section would only give simple overview of a typical Django project, for moredetails please read Django's nice documentation.

A overview of a typical Django project directory structure

wsgi.py and manage.py are generated by Django when create the project withdjango-admin startproject <project_name>

│ ├── serializers.py # Data serializers for encode/decode JSON for API

│ ├── serializer_utils.py # Custom serializer field│ ├── signals.py # Define signals, and send notifications on signal

│ ├── urls.py # How this app locate itself in url│ └── views.py # Views handle requests and generate response

├── srpms # System configuration│ ├── __init__.py│ ├── settings.py # System settings│ ├── urls.py # Root url│ ├── utils.py # Custom exception handler for server error

│ └── wsgi.py├── Dockerfile # Dockerfile for building the django image

├── manage.py└── start.sh # Start up script (only for docker use)

project_name/ # Project root

├── app_01/├── app_02/├── project_name/ # Project configurations│ ├── settings.py # Overall project settings│ ├── urls.py # URLs configuration of the project│ └── wsgi.py└── manage.py

83

django-admin is a command available from Django's python packagemanage.py provides the Django's management interface, issue pythonmanage.py help for detail information

It includes support for database migration, superuser creation, django appskeleton generation, and more

Apps in Django Project

A Django project consists of one or more app, each app represents a distinct set offunctionalities you want to deliver to user. The skeleton of an app can be generatedthrough python manage.py startapp <app_name>

In this project, our main goal is to provide the research contract managementsystem, and all functionalities regarding this goal would provided through theresearch_mgt app.

We also have a separate app accounts providing user management andauthentication functionalities. This set of functionalities is separated fromresearch_mgt for the consideration that user management and authentication isnot app specific, and might be used for other apps (in the case that we have otherapps in the future).

General Structure of a Django App with RESTframework

The typical structure of a Django app is as follow:

app-name/

├── migrations/├── __init__.py├── admin.py├── apps.py├── models.py├── serializers.py├── tests.py├── urls.py└── views.py

84 Back-End Design in File Structure Level

migrations/ correspond to database operations through Django ORM

These operations are normally auto generated through Django'smakemigrations management commandCreation of initial database data should also be placed here, e.g. createcourses and assessment templates when the database first initialize

admin.py includes support for Django's admin site for this app, note that thisproject currently does not utilize this. Read more about Django admin site

apps.py is the entry when Django try to load this app, normally don't modify

Signals (if any) are recommended to be initialized heremodels.py defines the database model that would be used inside this app

serializers.py defines how data from database models is being converted toJSON, and how JSON data would write to database models

The JSON data structure and database model need not to be the same, oneJSON data serializer can write to multiple database model, and onedatabase model can correspond to multiple serializer.Request sender unrelated business logic are defined here.

tests.py includes test for this app

Any function define in this file that follows test_* pattern would beautomatically included when issue python manage.py test

urls.py defines the URL layout of this app

views.py defines how to response request for each URL.

You would also see the following addition file structure more common on Djangoproject without REST framework. They are less common for projects using RESTframework as Django is only being used for sending/receiving JSON data, and don'tdo any server side page rendering.

app-name/

├── templates/│ └── app-name/└── static/ └── app-name/

85

templates/app-name/ includes HTML templates and CSS that would be used togenerate pages for this appstatic/app-name/ includes resources (e.g. images) that would be used torender the HTML page

Request Life Cycle in Django RESTFramework

Authentication The project is using both Django model back-end( django.contrib.auth.backends.ModelBackend ) and django-auth-ldap package tooffer authentication service. Of which:

86 Back-End Design in File Structure Level

Django model back-end is our own user database, for the purpose ofauthenticating users don't exist in ANU (but exist in our system).

django-auth-ldap is configured to connected to ANU LDAP server, so that userscan use their ANU username and password to login to this system

User information stored in ANU LDAP server would be poped/updated toour own user database on every login attempt.

For every login request, it'll first try to login using ANU LDAP service, and try ourown user database if the previous one failed.

Please note that information poped from ANU LDAP service only include userinformation (e.g. name, id, affiliations), and does not include password.User login using their ANU account would not have their password stored in ouruser database, but a user instance that cannot be used for login, with unusablepassword generated by Django.

Models, serializers and theirvalidation

This project implemented a total of three 3 levels for data validation

1. Model level. Regardless of the requester's privilege and all data instancefrom this model must comply

2. Instance level. Might vary depending on user's privilege, only apply toinstance that have certain property

3. Field level. Regardless of the requester's privilege, e.g. integer field shouldnot have string value

Validation for business logic and database model should be separated clearly,business logic related validation should be done by serializers, and databasemodel should not touch any business logic that are instance related.

Differences between ModelSerializer validation and ModelForm.Django models, encapsulation and data integrity

Database model's constraint only apply at model level, e.g. contract'sassessment must sum to 100

87

Instance level constraint (e.g. this type of contract should belong to a courseinstance) should be put in higher level.

Permissions This project implemented a total of three 3 levels of permissions

1. Field level. Decide whether the user have the privilege to edit a field of ainstance

2. Instance (Object) level. Decide whether the user can modify/delete a instance,i.e. PUT, PATCH, or DELETE.

3. View level. Decide whether a user can GET, POST, to a certain URL

List of Groups & Permissions

Permissions

can_convene , users with this permission can become course convenercan_supervise , users with this permission can become formal supervisoris_mgt_superuser , users with this permission can almost do anything, evenedit a finalized contract

Groups:

approved_supervisors , users in this group has permission can_supervisecourse_convener , users in this group has permission can_convenemgt_superusers , users in this group has permission is_mgt_superuser

88 Back-End Design in File Structure Level