Agile and BI – a perfect match Agile BI congres | Marco Coopmans | 13 december 2012

?

What is Agile?

Why start working differently?

A software development project …

• Takes too long

• Is too expensive

• Does not deliver what is asked

• Does not deliver what is needed

The Agile Manifesto

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

[Agile Manifesto, Utah, 2001]

Values

The main goal of agile is:

Delivering as much value as possible for the given time and money

How?

• Only make realy needed functionality, the highest value first

• Eliminate waste (lean) – only added value

• Mitigate risks – show working, production feasible software

The goal of agile

The characteristics of agile vs. waterfall

The user knows what he needs

The developer knows how to build it

During the project nothing changes

Waterfall

The user discovers during the project what he needs

The developer discovers during the project how to build it

There is a lot of dynamics and changes during the project

Agile

The devils triangle - Waterfall vs. Agile

Waterfall Agile

Time

Money

Quality

Functionality

Functionality

Time

Money

Quality

Fixe

d

Var

iab

le

Fixe

d

Var

iab

le

Determ

ines

Determ

ines

Agile and architecture

See the curve coming, but don’t overdrive

your headlights

• Keep it simple – As simple as possible

• Eliminate waste – Don’t carry any waste as long as it is

not necessary

– Don’t invest (too much) in the future

• Postpone choices as long as possible – Do things only when they are really

needed

Some architectural agile principles

Agile is delivering working software as quickly as

possible (business value)

Be a minimalist

• Architecture is not determined at the beginning,

but “emerges” (form follows function)

• Determine a start architecture: a (minimal) set

of guide lines

• That gives a lean-and-mean

framework that

grows during development

Emerging architecture

Refactoring is a fact!

Agile is not short term solutions

To avoid a dead end and minimalize the impact of

refactoring you have to look ahead,…

but not too far!

An experienced

SCRUM team will not

let this happen!

SCRUM is the most applied form of agile

The SCRUM process

Grooming/Poker sessions

Sprint Backlog

Demo

Sprint (2 to 4 weeks)

Production feasible Product/software

Feedback, new input

Product Backlog

Priority Sprint planning

•Product Vision •Solution outline

(architecture, tooling) •Business Case / ROI

•Release planning (1st release = MVP)

Development

Daily

stand up

Re

leas

e 1

R

ele

ase

2

Re

leas

e x

R

ele

ase

..

SCRUM Roles and Organisation

Product Owner

Scrum Master

Stake Holders Scrum Team

Tune Tune

Supports

Project Team

Roles and Organization – Product Owner

Is a delegate of the business who has the mandate to determine what is on the product backlog and the prioritization of it

Holder of the product vision

Owner of the product backlog and user stories (requirements)

Represents all stake holders (business, IT, external, …)

Accepts or rejects the product of a sprint

Decides release date and content

Product Owner

Product Owner

Product Owner sits

in the driver seat

Roles and Organization – Scrum Team

A multidisciplinary team (designers, developers, testers, architects, …) – all expertise necessary to deliver a production feasible product

Design, develop and test the user stories

Is self-organizing, without hierarchy

Determines, together with the Product Owner, the content off a sprint and commits itself to it!

Scrum Team

Scrum Team

Scrum Team is self-

organizing

Roles and Organization – Scrum Master

Facilitates the SCRUM process of the team

Ensures optimal conditions for the productivity of the team

Solves impediments (blocking issues)

Is part of the team, … but is supporting (has no authority)

Scrum Master

Scrum Master

Scrum Master is a

facilitator

Roles and Organization – Stake Holders

Everybody who has an interest in the product (endusers, managers, security officers, architects, system administrators, controllers, supervisors, …)

Are being represented by the Product Owner

Stake Holders don’t have direct access to the Scrum Team! – takes place via the Product Owner

Stake Holders

Stake Holders

Agile and BI

Why start working differently?

A software development project …

• Takes too long

• Is too expensive

• Does not deliver what is asked

• Does not deliver what is needed

Does this also apply for BI projects?

According to Gartner research estimation is that

70 to 80% of BI projects fail. And the number

one reason cited for this disturbing rate of

debacle? Poor communication. (Patrick Meehan, President and Research Director at Gartner, 2011)

Agile BI is about tools?

Agile BI = Delivering functionality as soon as

possible

Agile BI = Self-service

Agile BI = Tools

It’s not only

about

delivering

data fast, it’s

also about the

right data and

quality

Agile BI is only suited for the front-end?

Back-end

Front-end

Waterfall

Agile

Only 20/30% would be agile!

70/80%

Agile BI is layer driven?

Sprint 1

Sprint 2

Sprint 3

Sprint 4

Sprint 5 First visible result in sprint 5

Fro

m s

ou

rce

to

info

rmat

ion

business value

Agile is delivering business value (each sprint)!

But how are you

gonna do this every

sprint, from the

beginning?

The minimal architecture

Choose a minimal set of components to achieve the requested functionality. This depends on tools, source systems, transformation, integration, …

2 ways - #1 Report driven (slicing)

A report is growing with

each sprint

This will need a few sprints to get a sustainable report to get efficient feedback

2 ways - #2 Data / information driven (refining)

Reports are refining /

maturing with each

sprint

The ‘Mona Lisa model’ – from raw data to information

Production Base

Dimensional model (Facts and Dimensions)

Initiate & Model Storming

Making accessible

Refine Raw

ETL

Universe(s)

Standard reports

Experienced Domainexperts

User documentation

System- & Operatingdocumentation

Trained key-users

Sprint … Sprint …

Sprint … Sprint …

Datastorage (DWH/DM/…)

Unlock as much as possible (raw)

data

Refine content: - Refine Dimensional

model - Refine Universe - Make basic

report(s) - Add business rules,

transformations, integrations, …

- Add history - ….

Refine to end users:

- Make final rapports - Add security/

authorization - Finalize metadata - Finalize

documentation - ….

Bring to Production

Set op a base version

(dimensional) information-

model

Confronting users with data as soon as

possible

Organisation – BICC and SCRUM team

IV IT-supplier Business

Business Intelligence Competence Center

Legacy Designer/Developer

Technical Consultant (dba, network, infra, etc)

ETL Designer/Developer

BI/Warehouse Consultant

BI Tool Specialist

Power User

Key-User

Proces/Data owner BICC manager

BI Business Modeler (BI Tool Specialist)

Enterprise WH Architect

BI Domain Expert

Management Reporting Authority

End-User

BI/ETL Designer/Developer

Production Guard

Service Coördinator

Project Team

Product owner

Scrum master

SCRUM Team

BI/ETL Designer/Developer

Enterprise WH Architect BI Domain Expert BI Business Modeler Developer/ Specialist Power User

Business IT

BI issues Agile

Why agile is so well suited for BI

Unclear information need Agile assumes that requirements are not clear and/or change, that is part of the approach. By a constant feedback requirements become clear.

Management / business involvement

Business is in the lead (Product Owner). Business is part of the team (domain experts and power users). Stake holders are being involved through demo’s, grooming sessions, sprint plannings.

Quality of data Quality of data is being tackled at the moment the issue arrises. The issue is made clear from a concrete information need and is better negotiable than.

Definitions Definitions are being established during development (by the business), when it is necessary, based upon real data/visible results.

Reliability of information Reliability is being guaranteed by the participation of domain experts and power users in the SCRUM team (not by IT). They also give the demo!

Organisation A SCRUM team fits realy good with a BICC. The business (power users, etc.) will grow into their role during the project.

Agile and BI are a perfect match!

Preconditions and Pitfalls

Preconditions - Culture

• Give confidence

• Learning organisation - making mistakes is allowed

• Transparancy - no judgement

• Entrepenearship and

creativitity

• Belief

Agile means a change of culture

Preconditions – Process en organisation

• Committed Product Owner, with

mandate

• Sufficient business (customer)

participation – Preferably Power Users

• IT Professionals (craftsmanship)

• (SCRUM) process is being

followed – meetings, stand-ups etc. are not for

discussion

The business sits in the driver seat

Preconditions – Technique

• Enough complexity

• Emergent architecture

• Continuous build and

deploy

• Automated testing

• Production(like) data available

• The SCRUM team determines the solution – no

one else

Pitfalls - Sprint

• No ‘closed’ sprint – New features, requirements added during

the sprint

• No real commitment for a sprint

• Laying the bar too high – Don’t do too much at the same time

• Laying the bar too low – Don’t be satisfied too easily

• Too much production-oriented – Communicating is production! It’s not

about the amount of code

– 10 to 30% preserved for grooming etc.

• No Definition of Done (DoD)

Pitfalls - Solution

• Holding on to standard

solutions, architectures – New techniques provide many

opportunities, try them

• Change to generic / “flexible”

solutions too quickly – Generic datamodels, Datavault, …

• Too little focus on quality – short

term solutions

Pitfalls - Leadership

• Agile is not about working late

(on the contrary)

• Fall back on classical

management – The Team is self-organizing, even at “bad

weather”

It’s not about working harder

but working smarter

Pitfalls – Client/Business

• No clear business objective, lack

of product vision – Why are we doing this? What are we trying

to achieve?

• Insufficient coordination with

stakeholders (by the Product

Owner) – Lack of support

Conclusion

BI and Agile are a perfect match,

but ….

It’s a difficult game to play

Thank you

Marco Coopmans Marco.coopmans@gnie.nl +31 6 11916413