MODERN WEB APPLICATIONS INFRASTRUCTURE

PRACTICES, FLOWS AND TOOLS

!

EPAM Systems 2

!

!

"

# IGOR_ALPERT@EPAM.COM

HTTP://UA.LINKEDIN.COM/IN/IALPERT

IGOR ALPERT

EPAM Systems 3

Application !

!

EPAM Systems 4

Application

CSS

JS

HTML

!

!

5

Application

EPAM Systems

$

Feature #1

Feature #2

Bug fixing

!

Logs

6

Application

EPAM Systems

$ %

!

7

Application

EPAM Systems

Logs

Doc

!

$ %

8

Application

$

Logs

Config

Docs

!

%

EPAM Systems

Packages

9

Application

$

Config

!

%

EPAM Systems

Logs

Docs

10

Application

$

Logs

Docs

Config

Tests

!

Packages

%

EPAM Systems

11

Application

$ &

!

%

EPAM Systems

12

Application

0.5

$ 0.8

1.0

!

&%

EPAM Systems

13

Application

0.5

$ 0.8

1.0

!

Dev

Test

Prod

&%

EPAM Systems

14

Application

0.5

$ 0.8

1.0

!

Dev

Test

Prod

APP

DB

SV

&% '

EPAM Systems

DB

DB

DB

DB

' '

' '

15

Application

$

!

&% "

EPAM Systems

16

Application

$

!

&% " 7Errors

Logs

Metrics

EPAM Systems

17

Agenda !

$DEVELOPMENT

WORKFLOW

%SOURCE

MANAGEMENT

&BUILD

PIPELINE

"DEPLOY

PLATFORMS

7MONITORING

METRICS

CONTINUOUS

INTEGRATION

#

EPAM Systems

$

DEVELOPMENT

WORKFLOW

EPAM Systems 19

DEVELOPMENT WORKFLOW $

DEV$(

Managers)QA Support

%

EPAM Systems 20

DEVELOPMENT WORKFLOW $

DEV$(

Managers)QA Support

%

EPAM Systems 21

DEVELOPMENT WORKFLOW $

(Managers

)QADEV

$

Support%

EPAM Systems 22

DEVELOPMENT WORKFLOW $

(DESIGN

)TESTING

&BUILD, DEPLOY

aBUG FIX

'LAUNCHDEVELOP

$

2-4 WEEKS ITERATION

EPAM Systems 23

DEVELOPMENT WORKFLOW

&

(

*

$

%

UP TO 1 DAY ITERATION

$

24

DEVELOPMENT WORKFLOW

EPAM Systems

SERVERS

1. NGINX

2. APACHE HTTPd

3. TOMCAT

4. IIS

5. NODEJS

6. EXPRESS

7. …

STYLES

1. TWITTER BOOTSTRAP

2. ZURB FOUNDATION

3. COMPASS

4. SASS

5. LESS

6. STYLUS

7. CUSTOM SOLUTION

FRAMEWORKS

1. ANGULARJS

2. BACKBONE

3. EXTJS

4. SENCHA TOUCH

5. EMBERJS

6. METEOR

7. DERBY

8. CUSTOM SOLUTION

TESTING

1. JASMINE

2. MOCHA

3. BUSTER

4. QUNIT

5. KARMA

6. PROTRACTOR

7. NIGHTWATCH

8. INTERN

9. WEBDRIVER.IO

10.CUSTOM SOLUTION

OS

1. WINDOWS

2. OSX

3. UBUNTU

4. CENTOS

5. FREEBSD

6. COREOS

TOOLS

1. GRUNT

2. GULP

3. NPM

4. BOWER

5. YEOMAN

6. CUSTOM SOLUTION

DATABASES

1. MONGODB

2. COUCHDB

3. RIAK

4. REDIS

5. CASSANDRA

SERVICES

1. AMAZON

2. NODEJITSU

3. HEROKU

4. MONGOLAB

5. FIREBASE

6. JOYENT

7. PUSHER

8. DOCKER

$

SERVER SIDE

1. PHP

2. RUBY

3. PYTHON

4. …

!

EPAM Systems 25

DEVELOPMENT WORKFLOW $

VIRTUAL MACHINE

Your toolset will ask for them as a dependency

Works better and faster on Unix based systems

NODEJS, RUBY, PYTHON

Ability to compile native modules — easier to “make from source” then in Windows

Building a Module in Windows

WINDOWS

node-gyp, Python, shell, VS, …

EPAM Systems 26

DEPENDENCY MANAGEMENT $

Packager for OSX, uses formulas to install software

Homebrew

Mac OSXbrew install [package]

Package manager for Debian GNU/Linux distribution and its variants

Advanced Packaging Tool

Ubuntuapt-get install [package]

The Yellowdog Updater, Modified (yum) is an open-source command-line package management utility for Linux operating

Yum

CentOS

yum install [package]

Chocolatey NuGet is a Machine Package Manager, somewhat like apt-get, but built with Windows in mind

Chocolatey

Windows

choco install [package]

EPAM Systems 27

DEPENDENCY MANAGEMENT $

PRCE

NGINX

OPENSSL

MAKEDEPEND

PKG-CONFIG

GIT

GIT-FLOW

brew install git-flow brew install nginx

EPAM Systems 28

DEPENDENCY MANAGEMENT $

NVM

APP1

Node 0.11x

NVM

APP2

Node 0.10.31

EPAM Systems 29

DEPENDENCY MANAGEMENT $

Node Version Manager - Simple bash script to manage multiple active node.js versions

NodeJS

NVMhttps://github.com/creationix/nvm

RVM is the Ruby enVironment Manager. It manages Ruby application environments and enables switching between them

Ruby

RVMhttps://github.com/wayneeseguin/rvm

It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments

Python

virtualenv

https://github.com/pypa/virtualenv

A Node version manager for the windows folks out there. Inspired by n and nodenv

NodeJS

Nodisthttps://github.com/marcelklehr/nodist

EPAM Systems 30

DEPENDENCY MANAGEMENT $

NPM BOWER

manager install

"dependencies": { "<name>": "<version>", "<name>": "<folder>", "<name>": "<package>" }, "devDependencies": { "<test-framework-name>": "<version>" }

"dependencies": { "<name>": "<version>", "<name>": "<folder>", "<name>": "<package>" }, "devDependencies": { "<test-framework-name>": "<version>" }

manager install

manager install

EPAM Systems 31

DEPENDENCY MANAGEMENT $

NPM

GRUNT

PACKAGE.JSON

LODASH

ASYNC

KARMA

Packages:

npm install

npm install —save

Downloads:

Corporate mirrors

npm init

DEV-DEPS

https://docs.npmjs.com/

EPAM Systems 32

DEPENDENCY MANAGEMENT $

BOWER

LODASH

BOWER

BACKBONE

JQUERY

UNDERSCORE

BOOTSTRAP

Packages:

bower init

bower install

npm install -g

http://bower.io/

&BUILD

PIPELINE

EPAM Systems 34

BUILD PIPELINE &

Source

Concatenate

Uglify

SourceMaps

Test

Karma

Protractor

Mocha

Coverage

Preprocess

LESS

SASS

Compass

Assets

Templates

CSS

HTML processing

Images optimization

Watch

LiveReload

Rebuild

Serve

Custom

ChangeLog

Notifications

console.debug

EPAM Systems 35

GRUNT

BUILD PIPELINE !

Packages:

grunt

grunt coffee:app

jshint: { // define the files to lint files: ['gruntfile.js', 'src/**/*.js', 'test/**/*.js'], // configure JSHint options: { // more options here globals: { jQuery: true, console: true, module: true } } }

Gruntfile.js

Gruntfile.coffee

http://gruntjs.com/http://gruntjs.com/plugins

Good for file operations like copy/move/save. Configuration over code. Better for small projects with small amount of files.

FILE BASED

Most popular JS task runner. Almost any possible task is available as plugin. You can perform build-in or custom operations. Sync/Async

TONS OF PLUGINS

1

2

EPAM Systems 36

GULP

BUILD PIPELINE !

var gulp = require('gulp'); !gulp.task('default', function() { // place code for your default task here });

gulpfile.js

http://gulpjs.comhttp://gulpjs.com/plugins/

Packages:

gulp

grunt coffee

npm install --save-dev gulp

No temporary files. Code over Configuration. Much more faster then Grunt for file-content processing operations.

STREAM BASED

Only 4 API methods: task, watch, src, dest. Easy to write complex flows.

EASY API

1

2

EPAM Systems 37

YEOMAN

BUILD PIPELINE !

http://gulpjs.comhttp://gulpjs.com/plugins/

Generators: ~1100

npm install -g generator-angular

yo angular

npm install -g yo$ yo angular:controller myController $ yo angular:directive myDirective $ yo angular:filter myFilter $ yo angular:service myService

Pre-packed with test suites

KARMA TESTS

Easy scaffolding system to generate application components

SCAFFOLDING

Comes with build-in live-reload and grunt task

SERVER

1

2

3

EPAM Systems 38

BUILD PIPELINE

SLUSHhttps://www.npmjs.org/package/slush

BROCCOLIhttps://www.npmjs.org/package/broccoli

http://indigounited.com/automaton/

AUTOMATIONJAMhttps://github.com/caolan/jam

BRUNCHhttp://brunch.io/ https://github.com/nickjanssen/angus

ANGUS

!

!"

SOURCE MANAGEMENT

EPAM Systems 40

SOURCE "

GIT

Cheap branching — just a reference to commit. Lots of possible merging strategies and conflict resolving. History rewrite

EASY BRANCHING AND MERGING

You can work alone or with a team. You can work with or without central server. Even without network connection.

DISTRIBUTED

You can clone whole repo/branch or just a few commits. All meta-data is stored in one place ~/.git

FAST

1

2

3"

git init !git add git commit !git push —force !git rebase -i git merge !git tag -a

EPAM Systems 41

SOURCE "

GITLAB

You can integrate it with your own infrastructure, users base, private services etc

PRIVATELY HOSTED

Fully manageable and configurable Same capabilities — collaboration, forking, wiki and JIRA

TRUSTED BY ENTERPRISE

Hosted on GitHub. You can contribute and write your own features

OPEN-SOURCE

1

2

3

EPAM Systems 42

SOURCE "

GITHUB

You are able to create branches, tags, pull-requests, edit and commit files directly from browser

WEB INTERFACE

Do code-review with your team, leave comments, track issues and maintain WIKI pages

CODE REVIEW AND COLLABORATION

You can fork and contribute to any public repository. Home for open-source projects

FORK, PULL REQUEST

1

2

3

Mirror mirror on the wall, who forks best of them all?

EPAM Systems 43

SOURCE "

BITBUCKET

Unlike GitHub this service will allow you to create *unlimited* amount of private repositories.

FREE PRIVATE REPOS

Natively integrates with Atlasssian software stack. Trusted by Enterprise.

ENTERPRISE ORIENTED

Spooning - with Bitbucket is a pair programming process as an opposite to forking

SPOONING vs. FORKING

1

2

3

EPAM Systems 44

SOURCE "

BRANCHING MODELS

Centralized

Gitflow

Github

Feature branch

Commit

Commit

Commit

Commit

REBASE SQUASH

FEATURE BRANCH

…

REBASE

FEATURE

FEATURE

FEATURE

FEATURE

UPSTREAM BRANCH

EPAM Systems 45

SOURCE "

HOOKS

All Git hooks are ordinary scripts that Git executes when certain events occur in the repository. This makes them very easy to install and configure

PRE-COMMIT

PREPARE-COMMIT-MSG

COMMIT-MSG

POST-COMMIT

POST-CHECKOUT

PRE-REBASE

PRE-RECEIVE

UPDATE

POST-RECEIVE

Client Server

http://githooks.com/

SPELLING ERRORS

ENFORCE RULES

EMAIL / SMS NOTIFICATIONS

DEPLOY TRIGGER

Use Cases

EPAM Systems 46

SOURCE "

COMMIT MESSAGES

[FIX] Closes: #1234 [BREAKING] Interface of method has been changed… [COMPLETE] Feature (F16:Raptor) is ready for integration [ENV] Some minor refactoring

MAKE YOUR CONVENTION

ISSUE MANAGEMENT

1

2

3 RELEASE NOTES AND CHANGE LOGS

4 ANNOTATIONS: squash! fixup!

CONTINUOUS

INTEGRATION

#

EPAM Systems 48

TECHNICAL DEPT

TESTS WILL BE IN THE NEXT RELEASE

LET’S JUST COPY/PASTE FOR NOW

1

2

3

CODE ENTROPY: “IF I TOUCH THAT CODE EVERYTHING WILL BREAK”

4

TODO/FIXME STATEMENTS

CONTINUOUS INTEGRATION #

Fix this code, the only one who can. Is in another company already.

http://en.wikipedia.org/wiki/Technical_debt

5

DOCS? MY CODE IS STATE OF ART

EPAM Systems 49

CODE QUALITY

CODE-STYLE: TABS/SPACES, INDENTATION

STATIC ANALYZE AND COMPLEXITY: BUGS, DUPLICATION, STANDARDS

1

2

3

UNIT-TESTS AND COVERAGE4

DOCS AND COMMENTS

AUTOMATE

CONTINUOUS INTEGRATION #

5

TECH BACKLOG

EPAM Systems 50

CODE STYLE

CONTINUOUS INTEGRATION #

- JSHINT + STYLISH - PLATO - CODE PAINTER - EDITORCONFIG - JSCS - ESLINT

- AIRBNB - CROCKFORD - GOOGLE - JQUERY - MDCS (THREE.JS) - WIKIMEDIA - YANDEX

- COFFEELINT - CSS - HTML

Other validations JSCS Presets

1

2

3

4

5

Tools

READABILITY

GOOD NAMES

CLEAR LOGIC

POTENTIAL ISSUES

GUIDELINES

EPAM Systems 51

CONTINUOUS INTEGRATION

TESTS

#

LONG (hours)

MEDIUM (minutes)

FAST (seconds)

FEEDBACK TIME

UNIT-TESTS

UI

API

TDD (BDD)

BETTER CODE UNDERSTANDING

1

2

3

RELIABILITY

4 RELEASE FASTER

5

MOTIVATIONFILE SYSTEM

DATABASE

HEADLESS

END-TO-END

SERVICES

SMOKE TESTS

EPAM Systems 52

CONTINUOUS INTEGRATION

FIXTURES

#

module.exports.Permission = login: name: 'login' title: 'Login Page' ! home: name: 'home' title: 'Home Page' ! student: name: 'student' title: 'Student Section' ! moderator: name: 'moderator' title: 'Admin Section'

module.exports.Role = guest: name: 'Guest' permissions: [ Permissions.login Permissions.home ] ! user: name: 'User' permissions: [ Permissions.home Permissions.student ] ! admin: name: 'Admin' permissions: [ Permissions.home Permissions.student Permissions.moderator ]

Permission.coffee Role.coffee

db.Role

Clean

Role.coffee

EPAM Systems 53

CONTINUOUS INTEGRATION

CODE COVERAGE

#

97%

NOT TESTED AREA OF APP

DEAD CODE DETECTION

1

2

3 TESTING QUALITY

4ACCEPTANCE THRESHOLD 70-90%

5 REPORTS

JSCoverage

Istanbul

COVERALS.IO

Blanket

CODECLIMATE

EPAM Systems 54

CONTINUOUS INTEGRATION

DOCUMENTATION GENERATION

#

KEEP THEM UP-TO-DATE

METHODS AND CLASSES

1

2

3 PUBLIC API

4USAGE EXAMPLES

5 (RE) USE YOUR TESTS

HTML MARKDOWN

JSDoc

Output

JSDOC* CROJSDOC APIDOCJS DOXMOXDOCCO CODO

Tools

@param @name @class @method @option

@example @depricated @package @private @api

EPAM Systems 55

CONTINUOUS INTEGRATION

.dot files

#

.gitignore

.gitattributes

.jshintrc

.jshintignore

.editorconfig

.jscs.json

.travis.yml

SHAREABLE CONFIGURATION

EASY TO MANAGE

1

2

3

IDE INTEGRATION4

ESSENTIAL PART OF YOUR ENVIRONMENT

5 NOT A TRASH

Invest time learning to configure your machine and automate processes

Project System

.bashrc

.zshrc

.gitconfig

.aliases

.functions

.osx

http://dotfiles.github.io/

EPAM Systems 56

CONTINUOUS INTEGRATION

IDE

#

Code Completion and Inspection

Refactoring, Metrics

1

2

3 Test framework integration

4 Debugger, Frameworks support

5 Source Version control Issue trackers integration

CODIO CLOUD9 CODEBOX CODENVY

Online IDE

EPAM Systems 57

CI SERVERS

CONTINUOUS INTEGRATION #

BUILDS HISTORY AND VISIBILITY

LAST GREEN/RED STATUS

1

2

3 PARALLEL BUILDS

4 MULTIPLE ENVIRONMENTS

5 ROLLBACK

JENKINS TEAMCITY GRUNT CONCRETE STRIDER CI

TOOLS

And badges: http://shields.io/

EPAM Systems 58

HOSTED SERVICES

CONTINUOUS INTEGRATION #

CI AS A SERVICE

INTEGRATED WITH GITHUB AND BITBUCKET

1

2

3 BUILD ON SCHEDULE

4 DEPLOY GREEN

5 REPORTS AND STATUS DASHBOARD

circle.yml .travis.yml

Options

TRAVIS CI CODESHIP CIRCLE CI BAMBOO MAGNUM CI SEMAPHORE GONDOR RULTOR

Services

"DEPLOY

PLATFORMS

EPAM Systems 60

DEPLOY PLATFORMS

SERVER

#

CUSTOMIZABLE STACK AND ENVIRONMENT

NEED FOR (DEV) OPS

DATA: COMPANY POLICY

OWN SERVICES TO USE

d ef g

g / NGINX APACHE TOMCAT

Ñ SSH BASH 7 SYSLOG

# FILES TMP CACHE

$ SQL NOSQL KEY:VAL

( SERVICES DAEMONS CRON

W IPTABLESm SENDMAIL

w

1

2

3

4

5 INFRASTRUCTURE

K

EPAM Systems

%61

DEPLOY PLATFORMS

PaaS

#

CLI DEPLOYS

CONFIGURATION DASHBOARD

1

2

3BUILDING BLOCKS (DISK SPACE, DB, MONITORING, ADDONS)

4 MONITORING

5 SLA AND NODES MANAGEMENT

NODEJITSU HEROKU CLOUD FOUNDRY JOYENT ENGINE YARD

Services

EPAM Systems 62

DEPLOY PLATFORMS

Docker

#

YOUR OWN PAAS

OPEN SOURCE

1

2

3EASY SCALE

4 MANY READY TO USE STACKS

5 EASY TO MIGRATE

DEIS FLYNN TSURU OCTOHOST !

TOOLS

7MONITORING EVENTS

METRICS

EPAM Systems 64

MONITORING EVENTS

Logs management

#

FORMAT IS IMPORTANT

AGGREGATION AND ROTATION

1

2

3 SEARCH

4 ALERTS AND NOTIFICATIONS

5 WATCHERS

EPAM Systems 65

MONITORING EVENTS

Example

#

2014/10/02 13:58:32 [error] 925#0: *14 connect() failed (111: Connection refused) while…

SIZE

DATE

10Mb

LOGSSept

10Mb

Oct

07

EPAM Systems 66

MONITORING EVENTS

Logging services

#

PAPERTAIL LOGENTRIES SUMOLOGICBOUNDARY SENTRY GRAYLOG2

Services

APPDB SRV

SYS

%ELASTIC SEARCH

n

8

!

EPAM Systems 67

MONITORING

Errors

#

CONTEXT IS IMPORTANT

GROUPING

REPRODUCE

FAIL GRACEFULLY

HISTORY

… RAYGUN TRACKJS AIRBRAKE QBAKA MUSCULA JSERRLOGTRACEKIT

Services

1

2

3

4

5

EPAM Systems 68

MONITORING METRICS

HealthPage

#

SERVICE #1 (UP) Ã

SERVICE #3 (10k users / s) Ã

SERVICE #1 (10k ops / s) Ã

ÂFREE SPACE (~2Gb)

HARDWARE

SERVICES AND API

1

2

3 TRANSPORTS

4 PERFORMANCE

5 UPTIME

&SUMMARY

70

SUMMARY

EPAM Systems

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

71

SUMMARY

EPAM Systems

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

72

SUMMARY

EPAM Systems

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

73

SUMMARY

EPAM Systems

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

' " ! " 7#

74

SUMMARY

EPAM Systems

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

DevOps

Production alike

…as a Service

' " ! " 7#

75

SUMMARY

EPAM Systems

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

DevOps

Production alike

…as a Service

Metrics

Checklist

' " ! " 7#

Logs rotation

76

SUMMARY

EPAM Systems

&

DEVELOPMENT

WORKFLOW

SOURCE

MANAGEMENT

DEPLOY

PLATFORMS

MONITORING

METRICS

Automate

Tests

Small features

Code style

Commits convention

12 factor app

Build Blocks

Keep it fast

Run cycles

Fail early

Fast feedback

Step by Step

BUILD

PIPELINE

CONTINUOUS

INTEGRATION

DevOps

Production alike

…as a Service

Metrics

Checklist

' " ! " 7#

Logs rotation

77EPAM Systems

QUESTIONS?

?

THANKS!