Endasol Streamlining Bi Implementation, Euci March 2008

Considerations for Streamlining YourBusiness Intelligence Implementation

March 2008

© 2008 Endasol, Inc. All Rights Reserved.


Brief Introduction - Endasol (www.endasol.com)

Focus AreasData Warehouses/ BI Efforts

Lead Roles in PJM Enterprise-Wide Data Warehouse prior to forming EndasolCalled back to PJM to help develop a separate Market Monitoring Data MartJust underway with helping to identify business requirements and early design for a major data mart at an ISO.Business Analysts for a couple different energy companies’ major data consolidation & reporting projectsDeveloped Comprehensive Solution for Wholesale Power P&L + Other KPIs

Systems Integration – ETRM, Scheduling, Settlements, Generator Management, Fuel InventoryISO Operations & Interfaces - PJM, MISO, NYISO, ISO-NE

Developed Streamlined Interfaces to many of the PJM eSuite ApplicationsDirect Experience in estimating Major Settlement Line Items

Business Process Development/ Refinement


Streamlining BI Implementation - Topics

Components of Business Intelligence

Methodologies for Streamlining a BI Implementation

Relationship to Business Process Management


Components of BI Systems


BI System Integration & Information Layers

Data WarehouseData movement from Sources to Target Tables

Usually scheduled in batched ETL processesStaging Tables or Operational Data Store often an intermediate step

Data Model promotes business user reporting & querying – DimensionalData Propagation

Data movement among SystemsUsually in Real Time or near Real TimeUse of Enterprise Application Integration (EAI) tools to publish & subscribe to data

Data Warehouse can be a SubscriberData Federation

No data movementPre-established Views are often constructed to relate data across systems

More suitable for highly decentralized organizationsA sensible approach may involve elements of each method


BI System Integration Layer

Pre-established ReportsRoutine reports designed & developed in advance – often scheduled Typically ‘locked down’ and controlled by a couple of individuals - often ITMay build intermediate tables, views and/or calculated fields to feed reports

Self Service Reports and QueriesBusiness users should be able to do reporting & querying without the need for IT

Business users do not always know exactly what they want until they start using the data in ways not previously possibleRequires a sound foundation (typically dimensional data model), data dictionary or other metadata, and training

OLAP CubesOLAP provides interactive analysis and reporting using multidimensional views of data

Users can slice, dice, drill-down and roll-up data – more than Excel pivotsCubes are often pre-built and refreshed right after batch ETL processes so that they are immediately available to users


BI System Intelligence Layer

The value of the BI System is often not realized until deploying this Layer

Advanced Visualization, including Dashboards & ScorecardsEnable rapid assessment of Business Metrics or Key Performance IndicatorsDashboards display information in tables, charts, gauges, and other indicatorsScorecards display metrics against pre-established thresholds grouped into red, yellow, green or numerical weightingsUltimately, should have the capability to drill down into information feeding the dashboard and scorecard items


BI System Intelligence Layer

Advanced AnalyticsExploratory Data Mining – seeking new insights through examining the information with statistical analysis or other techniquesPredictive Analytics – formulating rule-based models to predict what will happen under various scenarios with high degree of confidence

Many BI efforts are abandoned before reaching this LayerWhy? Because implementation of the other layers have been overly costly with mediocre results

Predictive Analytics; Extending the Value of Your Data Warehousing Investment, 2007 TDWI Best Practices Report, Wayne W. Eckerson


Employ methods that get to the higher value BI layers sooner:

Establish Fundamentals Early

Consider using a Packaged Solution or Starter Model

Consider Agile Development Approach

Consider Hybrid Approach

Considerations for the Interactive Layer

Considerations for the Intelligence Layer

Streamlining BI Implementation


Implement a Data Governance Program UpfrontAssign Data Stewards from the Business responsible for quality and use of data

Establish Monitoring and Methods for Correcting Issues

Consider a Master Data Management ProgramManaging your Reference Data, such as customer, generatorsEnsuring consistency and dynamic mapping among systemsSome examples of the need:

How many systems or databases house customers (individuals or organizations) and how many different identifications, abbreviations, or alias’s are used?How are you tracking who owns whom? Frequently changes - if not careful, you may have excess credit exposure to entities or entities may be exercising unfair market power.

Various technologies are availableEnsuring consistency among systems regarding Master Data will streamline BI implementation, but not absolutely needed since the data warehouse will ultimately solve this issue for users of the data warehouse

Establish Fundamentals Early for Any Phase


Set realistic expectations from both business & IT perspectivesDefine precise & measurable objectivesEnsure that the BI system will be scalable to accommodate other corporate initiatives in the futureMore importantly, align the project with overall corporate BI objectives

Minimize scope creepWell-defined business requirementsEstablish Change Control policies upfront – know how changes will be processed and who needs to approve these

If you don’t have the expertise in-house, then hire professionalsEnsure they have experience in the phase you are embarking onEnsure those on the team have experience with your businessBe involved – you still need to ‘own’ the program & business must be involved

BI systems are not static – you will need skills to advance & maintain your system

If not prepared to adopt these & other fundamentals, then don’t start!

Establish Fundamentals Early for Any Phase


Packaged BI Solutions: 3rd party products fulfilling a slice of your BI needs

A Packaged BI Solution should:Provide the business with a clear definition of the end results – what you will get is visible from the start!Integrate and merge data across all appropriate systemsEnable easy access to underlying data and the information it providesServe as a starting point for other analysis and reportsBuild on an existing data warehouse as appropriate

Use of Packaged BI Solutions & Starter Models


A Packaged BI Solution is typically not:A front line application, such as an Energy Trade & Risk Management System (ETRM), or a Customer Information System (CIS) - too many try to elevate these transactional systems to be an enterprise data mart for reports & analysis

Are there Packaged BI Solutions available in our industry?For wholesale power, including financial transactions, we have our PREMIRTM SolutionThere may be solutions available for the Retail, and Transmission/Distribution SectorsConsulting Companies focused in your business may offer at least Starter Models



Endasol’s Flow Diagram

Our present focus is the Wholesale Power & Financial Transactions Data Models and Reporting

premir SolutionsTMpremir SolutionsTM


Advantages:Requirements/Vision already defined – you see what the end result looks likeData Model provided and can be built uponKey Reports may be provided and others can be addedKPI’s & Dashboard may be provided and can be expandedIn any case, they can provide a head start enabling quicker deployment

Potential Pitfalls:Don’t expect a packaged solution or even starter model to meet all of your objectives – expect that some customization will be neededCompatibility with existing data warehouse, data marts must be consideredCompatibility of technology used – database, reporting tools – must be consideredCapability to expand the solution to other areas of the enterprise must exist

Conclusion: A packaged solution or starter model can significantly expedite your BI implementation, but do your homework first



Waterfall Methodology – Traditional ApproachFrom Wikipedia: “The waterfall model is a sequential software development model (a process for the creation of software) in which development is seen as flowing steadily downwards (like a waterfall) through the phases of requirements analysis, design, implementation, testing (validation), integration, and maintenance.”Often characterized by lengthy requirements, design and signoff before any development is allowed to begin.

Agile Methodology – Currently gaining acceptanceFrom Ralph Kimball: “…In general, it (Agile) refers to a group of methodologies, including Extreme Programming, SCRUM, Adaptive Software Development and others, which share a common focus on iterative development and minimizing risk by delivering new functionality in short timeframes, often measured in weeks.”Less structured with emphasis on deliverables, not documents & signoffs

Consider Agile Development for BI - Background


Consider Agile Development for BI

Agile Development is gaining interest - Agile is now included in training classes and conferencesScrum is one structured methodology for Agile Development

Product/Project broken into deliverables, called ‘Sprints’ of 2 to 4 weeks.Sprints established to bring priority, but achievable results within time frame.Sprints typically locked down with high resistance to changing scope once started.Product Owner - The Product Owner represents the voice of the client. He/She ensures that the Scrum Team works with the right things from a business perspective.ScrumMaster (or Facilitator) - Scrum is facilitated by a ScrumMaster, whose primary job is to remove impediments to the ability of the team to deliver the sprint goal. The ScrumMaster is not the leader of the team (as they are self-organizing) but acts as a buffer between the team and any distracting influences. Empowered teams without traditional Project Management.Scrum Team - A small team of 2+ people with cross-functional skills to do the actual work (designer, developer etc.).Daily meetings among players – IT & Business.Development proceeds immediately and documentation often lags.


Consider Agile Development for BI Implementation

Biggest Issue with many DW/BI projectsChanges during the project!

Source system changesRequirementsData quality issues – some sources are just not very good.Change in business sponsor / business ownerMarket changes

ISO rule changesSubscription services

Business Priorities



Advantages of Agile Development Design at the beginning of each sprint - Learn from each sprint Phased / iterative approach – Useable tools delivered quickerDetermine next sprint based on priorities.Working closely with the business users during data cleansing and development of reports, cubes and dashboards

Increase user satisfaction and ‘ownership’Decrease ‘do-overs’

Building what the business needs overrides building just to meet documentation



Disadvantages of Agile DevelopmentBudgeting difficulties due to minimal upfront analysis & design

Hardware needsStaffing – how long?When is the project considered finished?

How does the present sprint fit into the solution?Some rework may be neededHave a risk of delivering isolated silos of data which meet individual releases, but don’t integrate well with each other

New to the organization and not well understood by IT and Business. Users are confused about their role.Users may not have sufficient time.Minimize risk by using company standards & following corporate SDLC process

Documentation – Agile methodology minimizes documentation. What about support and future enhancements?


Hybrid Approach for BI Implementation

Optimum Approach may be a Hybrid between Waterfall & AgileTake time upfront to develop overall BI plan/ roadmap

Spend some time on how the overall BI solution fits into the enterprise pictureIdentify subject areas for potential phasesDevelop framework of data modelSet standards Select hardware / software

Break into focused phasesPhase 1 may be 4 – 6 months due to infrastructure, overall requirements, design & documentation Focus mainly on current phase subject area but know how it fits with the other areas.Other phases may decrease in length as more and more subject areas are added.

Take each phase all the way through user reports, OLAP cubes, dashboard & scorecardInclude support with the delivery of each phase


Hybrid Approach for BI Implementation

DetailedRequirements

Establish Project•BI System Objectives•Business Requirements•Prioritize Phases •Project Plan

High Level Design•Subject Area Data•Subject Area Relationships•High Level Data Model•Set Standards

Infrastructure•Hardware•Software/ Licenses•Set Up Environments

DetailedDesign

ETL &Test

UserReports

OLAPCubes

Dashboard/ Scorecard

Phase1

Phase 2

Phase3

Phase4

Repeatfor

eachPhase

EachPhase

may getshorter


Choosing Reporting and Querying ToolsBusiness Ease of Use

Deploying a mixture of tools

What about Excel?MS Excel will continue to be the tool of choice for many business usersEven with well managed data warehouses, end users still resort to ExcelBut this may not be bad as long as the source is the data warehouse and not direct pulls from source systems

How important is OLAP?

Streamlining Interactive Layer


Give Business Early Access to DataStandard reports should be pre-established using the corporate reporting tool & ‘Locked Down’Underlying structure/ data model should be such that business users can easily understandAllow them to develop their own reports and analysis - Stress using corporate reporting tools, but be flexible as long as they are sourcing from the data mart

Streamlining Interactive Layer


Restrict Key Performance Indicators (KPIs) to a small number (6 –10) specific to the organization’s goalsDevelop different set of KPIs for each business unit, sub-unit

The business unit’s KPIs should relate to corporate KPIsThe business unit’s KPIs should also include what’s important to bonuses

Most effective use is often a mixture of financial & non-financial KPIs as in the Kaplan and Norton Balanced Scorecard approachKPIs should follow the SMART test:

Specific, Measurable, Achievable (Actionable), Realistic, Timely

Once implemented, continue to refine KPIsRevise as corporate and business unit objectives changeMonitor who is accessing and how often –retire KPIs not being usedEnsure individuals can easily access KPIs and understand what they are viewing

Streamlining Intelligence Layer - KPIs


Choosing TechnologyNumerous products available with nice dashboard & scorecard features

Some are limited to specific technologies, such as using MS Analysis Services (cubes) to drive dashboard components

Most effective is ability to drill down – start with the picture & drill to underlying information

Don’t need to go overboard on dashboard/ scorecard technologyIf can’t justify expense upfront, then consider publishing output of certain reports & OLAP such as graphs or charts in a web-portal – this can be automatedOnce base functionality is working, can always buy advanced technology later

Streamlining Intelligence Layer – Visualizing


Implementing Dashboards & Scorecards

Keep it simple – don’t clutter the viewing screen

Consider allowing viewers to construct their own ‘myBI’ pageSome items may be required viewing such as important metricsOthers may be selected from a list of available dashboard reports, graphs, charts

Continue to monitor usage and effectiveness & continue to refine

Streamlining Intelligence Layer – Visualizing


Determine Data Mining or Advanced Analytical Methodology Early

Exploratory – looking for new insights and/ orPredictive – developing models to predict behavior/ market opportunitiesDifferent methodologies are available for each focusMethodology may require very specific data model or structure

Data model used for advanced analytics may conflict with that optimal for reporting and user-queriesIf you know this upfront, you may be able to build a hybrid model that can serve both purposes, orYou may be able to implement a model structure that can be more easily transformed into that needed for the advanced analytic application

Streamlining Intelligence Layer – Data Mining


Ultimately, your BI System should be aligned to corporate objectives and provide monitoring & feedback for continued alignment

Corporate KPIs should trickle down to corresponding KPIs pertinent to individual Business Units

Business Unit KPIs should link to corporate & adjusted as corporate objectives change

The above is part of the broader initiative referred to as Business Performance Management (BPM), Corporate Performance Management (CPM), or Enterprise Performance Management (EPM)

There are advancement in software tools which support implementation & monitoring

Align Your BI System with Corporate Objectives


Streamlining BI Implementation - Summary

Align your BI System with Corporate Objectives

Value of your BI System is often not realized until deployment of Visualization and Advanced Analytics

Dashboards & Scorecards to track KPIs & other important measures

Drill-down into information driving displays is an important capability

Data Mining for discovering new insights

Predictive Analytics to gain edge in market


Streamlining BI Implementation - Summary

Objective is to get to these value BI Layers as soon as possible:Use Packaged BI Solutions or Starter Models if available and if applicableUse a Hybrid Approach of Agile & traditional Waterfall Development

Do some detailed requirements & design upfrontBreak Project into short phases of 1 – 6 months covering selected subject areasEach Phase should include all BI Layers - Data Model, User Reports, Cubes, Dashboard & Scorecard items

Technology – decide what’s important to you from a budget and capability perspective, be flexible and don’t get stuck in analysis paralysis.

Endasol Streamlining Bi Implementation, Euci March 2008

Technology

Transcript of Endasol Streamlining Bi Implementation, Euci March 2008