Extreme Programming for ETL and Data Analytics
74
eXtreme Programming for ETL Agile Pune 2014 _________________________ Prafulla Girgaonkar [@pgirgaonkar] Naresh Jain [@nashjain] Illustration: Umesh Kale
-
Upload
naresh-jain -
Category
Software
-
view
389 -
download
0
description
Over the last decade, eXtreme Programming practices like User Stories, Evolutionary Design, Test-Driven Development (TDD), Behavior Driven Developer (BDD), Refactoring, Continuous Integration and Automation have fundamentally changed software development processes and inherently how engineers work. Having experienced various benefits from XP practices on our J2EE stack, our team started to apply these practices to extract, transform, and load (ETL) and Data Analytics side of our product. Unfortunately, there is very little guidance available in this context, esp. for the SAS Platform. Right from finding the unit testing framework to structuring the code to designing our modules and setting up a Continuous Integration builds, our team had to figure out everything, the hard way. Join us to understand the challenges we faced during this process and how we resolved these challenges. Details: http://confengine.com/agile-pune-2014/proposal/535/extreme-programming-for-etl-and-data-analytics Conference: http://pune.agileindia.org/
Transcript of Extreme Programming for ETL and Data Analytics
eXtremeProgramming for ETL
Agile Pune 2014_________________________
Prafulla Girgaonkar [@pgirgaonkar]Naresh Jain [@nashjain]Illustration: Umesh Kale
25YEARS 6000+CLIENTS Nr 1M ROOMS 94COUNTRIES 95%+
ago we started
helping clients
manage their
revenue
later we’re still finding
the revenue opportunity
in every situation
around the world are
priced every single day
using IDeaS
on 6 continents have
hotels that now run
on IDeaS Revenue
Management
Systems
More than 95
percent of all
our clients are
still working
with us today.
About IDeaS
ETL
DECISION
IP
PMS/CRS
Competitor
WebRate
Social Media
Periodically or
Real Time
Full or
Delta
Messages or
File
****
ETL & IDEAS
Concept Exploration
Structuring Requirements
Discovery
Earlier ETL Practices
Concept Data Exploration “Structurization”
ETLDevelopment
Certification GA
Typical Development Cycle on an ETL Project
Vendor
IDEAS
Development
Earlier ETL Practices
Challenges and Flaws
Source : www.dogri.org/
Challenges and Flaws
Challenges and Flaws
Challenges and Flaws
Challenges and Flaws
Challenges and Flaws
Time
Discovery Development
Challenges and Flaws
Source : http://roselawgroupreporter.com/
Challenges and Flaws
Requirement Conformity
Requirement Misunderstanding
Late Data Change
Delays
API Performance
Priorities
Challenges and Flaws - Vendor
Naresh Jain
• nareshjain.com
• @nashjain
Prafulla S Girgaonkar
Product Development for
Hospitality | Revenue Optimization | Finance
@pgirgaonkar
Journey
Hilstar Opera Pegasus
Old Approach
Existing Approach - Iterative & Incremental Approach
Initial XP Practices - Hilstar
DBA
QA
DEV PM
UX
Initial XP Practices – Whole Team
Source: http://www.selfishprogramming.com/
BO
KM
Iteration #1 Iteration #2 Iteration #3
Initial XP Practices – Small Releases
Initial XP Practices - Continuous Integration
Initial XP Practices – User Story
Evolution of the Practices… OPERA
Source: http://www.freedomchurchunited.org/
Evolution of the Practices - Pairing
Evolution of the Practices – Planning/Grooming
Source: http://www.h4-entertainment.com/
Evolution of the Practices – Test Driven Development
Evolution of the Practices – Refactoring
Evolution of the Practices – Coding Standards
Evolution of the Practices – Collective Ownership
Evolution of the Practices – Simple Design
• The customer is always available
• Code the unit test first
• Leave optimization for later
Evolution of the Practices – Optimization at the End
Write a TEST before FIXING the BUG
Evolution of the Practices – Test Driven Bug Fixing
Further…Evolution of the Practices… Pegasus
Data Management
4GLMACROS
PROCS FUNCTIONS
Learn SAS 4gl @
http://support.sas.com/software/products/ondemand-
academics/
Further…Evolution of the Practices – SAS 4GL
Further…Evolution of the Practices – Earlier Code
No Unit testNo Build
Integration
Language Unawareness
Discover the Tool
SASUNIT
Further…Evolution of the Practices – Testability
Further…Evolution of the Practices – Code Later
Part of build Process
Fail Fast
Further…Evolution of the Practices – Build Process Correction
Integrated With Jenkins
Fail Fast
Fail Reports
Further…Evolution of the Practices – Jenkins for SASUNIT
Engineering Practices are always at the core.
Tools May not be there.
There is WILL, there is WAY
CRUX
Benefits
Source: http://flickr.com
Source: http://docustar.com/blog/quality-check-is-99-9-good-enough/
Benefits –
Source: http://www.ftjco.com/wp-content/uploads/2012/10/diamonds.jpg
Benefits
Benefits
Benefits
Benefits
Relative Time
Hilstar Opera Pegasus
UNIT Test …UNIT Test …UNIT Test …
Source: www.topnews.in
Benefits
Source: http://en.wikipedia.org/wiki/Cello
Benefits
Change Management
Source: http://www.animalnational.com/monarch-butterfly-migration/
Change Management – Usual Reactions
Source: http://ericgerlach.com/introphilosophy8/
DEVs | QAs | POs | BOs
Change Management – Looking for Rituals
Source: http://www.theguardian.com/books/2014/apr/27/reading-print-books-ereaders-technology
DEVs | QAs | POs | BOs
Courtesy: Jeff Patton
Change Management – Smaller Release/Stories were Shaker
DEVs | QAs | POs | BOs
Change Management – Incremental Development
DEVs | QAs | POs | BOs
Change Management – Story Map
DEVs | QAs | POs | BOs
Change Management – Spikes were new
Source: http://www.kitchology.com/
DEVs | QAs
Change Management – CI & Failing Builds
DEVs | QAs
Change Management – Pairing
Source: https://www.airpair.com/mean-stack/posts/2014-10-job-post-mean-stack-developer
DEVs | QAs
Change Management – Team Ownership
DEVs | QAs
Source: https://dilbert.com
Change Management – Testers part of Design Discussion
Source: http://www.mercatornet.com/articles/view/health_economics_101_contraceptives_pay_for_themselves_usually_babies_dont
DEVs | QAs
Change Management – Defect are discussed; not entered
QAsSource: http://letstalkaboutdiscussion.ca/
Change Management – Test Automation
QAsSource: http://www.mobileqazone.com/profiles/blogs/software-testing-by-human-or-machine-what-should-shouldn-t-be
Road Ahead
Source: http://www.gettyimages.in/
Engineering Practice - Refactoring / Managing Code Smells
Source: www.123rf.com
• Engineering Practice - Simple Design & Solving critical
challenges by taking them out of context
Road Ahead - Engineering Practice – Simple Design
Source: http://flickr.com/
• Engineering Practice - Simple Design & Solving critical
challenges by taking them out of context
Road Ahead - Engineering Practice –Sandbox
Source: http://ifum.com/
Road Ahead - Engineering Practice - Evolutionary Design Extended
Source: http://nordicedge.com/
Road Ahead - Engineering Practice – Programming skills and polyglot
Road Ahead - Real Time Integration
WE I JAD
Road Ahead - Joint Application development with Vendor
Source: http://www.p-r-i.org/nadcap/nadcap-meetings/
Take Away
• Engineering Practices are universal
– determination and perseverance
• XP Practices lead to T-Shaped individual or
Generalizing Specialist
• Product Discovery & Story Mapping vital for
Alignment.
Prafulla Girgaonkar [@pgirgaonkar]Naresh Jain [@nashjain]Illustration: Umesh Kale
Session Ends.. Efforts Continue…
