From Pilot To ProductBy Ivan Verhun

Notice• Ask questions during presentation• Time is limited• Business domain is not important

Agenda• Core product

overview• Pilot architecture• • Dev Process

Core Product Overview• B2B• Import data from clients• Process data

– Validate– Filter– Generate report

Data Flow

SaaS Maturity Models

Main Disadvantages• Single tenant• Tightly coupled components –

impossible to reuse• Business logic in stored procedures

This slide intentionally left blank

Pilot Overview

Pilot Overview• Target audience - SMB• Business pilot• Limited TTL• Preferably short TTM• Isolated from the core platform

Individuals

Businesses

Pilot Overview• Target audience• Business pilot• Limited TTL• Preferably short TTM• Isolated from the core platform

This slide intentionally left blank

Pilot Architecture• 1000 meters view• Technologies and Tools• Mistakes

Initial Key Decisions• Multitenant• Extract data from single tenant DBs• Functionality spread between modules• All business logic in Java

– No stored procedures

Multitenant

Initial Key Decisions• Multitenant• All business logic in Java

– No stored procedures• Extract data from single tenant DBs• Functionality spread between modules

Extract data from single tenant DBs

Modules (Services)

Endpoints• Asynchronous – AMQP endpoints for

prod, qa, dev • Synchronous – HTTP endpoints for

IT • Endpoints switched on/off depending

on spring profile

This slide intentionally left blank

No Downtime Deployment

• Maintain an application mode – LIVE– MAINTAINCE

• Application mode is used by LB• Switch mode to MAINTAINCE before

updating the application• Switch mode to LIVE after re-

deployment

No Downtime Deployment

This slide intentionally left blank

Tech Stack – BeforeTools• Java 6• Jenkins• Maven 3• MySQL 5.1• Jetty 6• RabbitMQ• Git/Github

Libs and Frameworks• Spring

– IOC– AMQP– MVC– JDBC Template– Aspects

• JUnit• slf4j• Mockito, Powermock• Joda Time

Database Schema Migrations

• Startup• Patch Level (patch0001_test_patch.sql)• Patch Loading• Patch Ordering• Patch Execution• Post-Patch Tasks

Newest DB Schema• Shell script generating DB dump• Runs on Jenkins• Commits newest dump to git repo• Used to create empty test DBs

ATDD:

http://wakaleo.com/thucydides-sample-reports/

http://thucydides.info/

BDD and JBehaveBDD:• is an evolution of test-driven development (TDD) and

acceptance-test driven design…• shifts the vocabulary from being test-based to

behaviour-based• Was developed as a response to the issues

encountered teaching test-driven development

Author: Dan North

http://en.wikipedia.org/wiki/Behavior-driven_development

JBehave: example of story

This slide intentionally left blank

Centralized LoggingA tool for managing events and logs. You can use it to • collect logs • parse them • and store them for later

use (like, for searching)

http://logstash.net/

Kibana – ES data visualization engine

Performance Monitoring• Reporting Via JMX, HTTP,

slf4j, CSV, Graphite etc

• The Registry• Counters• Meters• Histograms• Timer (@Timed) http://metrics.codahale.com/

Java 7• Java 6 is dead• Java 8 is baby

This slide intentionally left blank

Persistence• JDBC was not OK• JPA/Hibernate was not enough• MyBatis – good for complex

queries

Web UI

http://www.thymeleaf.org/

TSV Generation

Report Printer

PDF Validation

Impossible to:• reliably verify absence of an element• associate present text to particular

report attribute– there may be false positives

• verify correct order of the text elements• parse/verify graphs

http://pdfbox.apache.org/

LimitationsVerify:• number of missing files• number of unexpected files• file content

Scope

Web UI (Admin)

https://angularjs.org/

Tech Stack – AfterInitial tool kit• Java 6• Jenkins• Maven 3• MySQL 5.1• Jetty 6• RabbitMQ

Added/updated tools• Autopatch• Configuration Service • JBehave + Thucydides• Java 7• Logstash• Metrics library• Amazon S3

Tech Stack – AfterOriginal tech stack• Spring

– IOC– AMQP– MVC– JDBC Template– Aspects

• JUnit• slf4j• Mockito, Powermock• Joda Time

Added later• JPA/Hibernate• MyBatis• Freemarker• Thymeleaf• Js:

– AngularJS – RequireJS– Bower

• Apache PDFBox

Mistakes• JDBC code is poorly maintainable• JUnit – @Parameterized tests

Development Process• Methodology• Techniques and Tools• Mistakes and Lessons

Scrum RulesPrinciple Is Adopted

Product Owner +

Development Team +

Scrum Master +/-

Sprint planning meeting +

Daily Scrum meeting +

Sprint Review +

Sprint Retrospective +

Product backlog +

Sprint backlog +

Burndown chart -

Sprint Structure

Planning process• Preplanning (30-60 min)

– high level plans and priorities• Grooming (up to 3 hours)

– discussion– estimation– Poker– Story points (0-8)

• Planning (30-60 min)– final estimation– defining sprint scope (commitment)

Scrum Board

Task Workflow• Analysis and writing AC• AC Review• Coding

– Unit tests– Functionality – Integration tests– Writing resolution comments

• Code review• Verification on QA env

Daily Scrum

Daily Scrum

Daily Scrum

Daily scrum

Advertising Ukraine

Demo

Retrospective• What is working• What’s not working• To improve next

time

Code Review TypesAd hoc

Passaround

Pair programming

Walkthrough

Team review

Inspection

Code Review Tools– Github

• Pros– inline comments– no extra software required

• Cons– emails per comment (it's even not configurable)– inline diffs, no side-by-side comparison– small diff context– directory structure is not visible

– Side-by-side comparison (Beyond compare)• lines comparison in the bottom (above status bar)• compare different files• compare with clipboard content

– Reviewer checklist

Beyond Compare

Beyond Compare

Review Checklist• The Obvious

– Code compiles, it’s clear, test cases etc.

• Design• Coding Standards• Error Handling• Security• Thread Safety

• Control Structures• Functions• Performance• Bug Fixes• Math• Maintainability• Documentation• Requirements Tracing

This slide intentionally left blank

People Management• Let people choose the task they like (if

possible)• Avoid knowledge islands• Let people be initiative• Minimize micromanagement• Semi-periodic face to face meetings

People management• Dealing with highly

motivated/enthusiastic people• Trainings

– Agile (scrum)– Code review– Thucydides– Virtualization– TDD and Refactoring

• Dinners

Team Buildings

Thursdays Dinner

Cross functional team

Team CompositionIchak Adizes:• Producers• Administrators• Entrepreneurs• Integrators

This slide intentionally left blank

Mistakes

We Don’t Need Testers

We Managed to Avoid This

Code Review(ers)• Start with seniors only• Establish code review process before

extending reviewers list• Extend reviewers list gradually• “Code Reviewer” title should be

honorable

Learnings• Testers are essential• Sometimes micromanagement is a

must• Establish code reviews right

This slide intentionally left blank

Customer Management

Why Customer Need Reports

Trus

tLo

w

Hig

h

Low HighTransparency

This slide intentionally left blank

From Pilot To Product• Successful production launch• Unexpectedly big demand• Extended contract• Rewriting on core platform

– New set of exciting technologies

ResourcesSaaS Maturity Model:• http://www.avuetech.com/wp-content/uploads/sites/7/2012/03/SaaS-Maturity-Levels.pdf• http

://blogs.msdn.com/b/architectsrule/archive/2008/08/18/saas-maturity-model-according-to-forrester.aspx

• http://www.infoq.com/news/2008/02/saas-architecture-maturity-model Code review:• http://en.wikipedia.org/wiki/Code_review#Types• http://www.softwaretestinggenius.com/istqb-foundation-level-exam-crash-course-part-6• https://www.owasp.org/images/2/2e/OWASP_Code_Review_Guide-V1_1.pdf• http://smartbear.com/SmartBear/media/pdfs/WP-CC-11-Best-Practices-of-Peer-Code-

Review.pdfCross functional team• http://en.wikipedia.org/wiki/Cross-functional_team Customer management• http://habrahabr.ru/company/stratoplan/blog/202028/

Picture Sources• http://en.wikipedia.org/wiki/File:American_truck.JPG • http://www.expresstaxiinc.us/ • http://www.dreamstime.com/ • http://www.durhamcharterservices.com/ • http://www.galingas.com.ua/ • http://provse.te.ua/ • http://www.huffingtonpost.com/ • https://en.wikipedia.org/wiki/Scrum_(software_development)#mediaviewer/File:Scrum_process.svg

??