Bringing Analytical Models in to The Business Field

MODEL DEPLOYMENT: THE MOMENT OF TRUTH

Presenter: Robin Way IIA Lead Faculty and President, Corios

MODEL DEPLOYMENT: THE MOMENT OF TRUTH

Presenter: Robin Way

IIA Lead FacultyPresident,

Corios

Inquiries about IIA services should be directed to [email protected] 2 | 2013 © IIA All Rights Reserved

About IIA

IIA is an independent research firm for organizations committed to accelerating their business through the power of analytics. We believe that in the new data economy only those who compete on analytics win. We know analytics inside and out - it’s what we do. IIA works across a breadth of industries to uncover actionable insights gleaned directly from our network of analytics practitioners, industry experts and faculty. The result? Our clients learn how best to leverage the power of analytics for greater success in the new data economy.

3

BIG IDEAS

•Many companies are not realizing the full economic potential of their analytic model assets due to lack of adoption.

• A common language for understanding analytic models and using them in the business needs to be shared with IT and the field.

• The financial benefits of proper execution and deployment of analytic models is exceptionally compelling.


Copyright© 2013 IIA All Rights Reserved

BACKGROUND

• Corios has conducted over 50 model development and deployment projects with North American businesses.

• Unlike in model development, few common processes exist in the world of model deployment.

• This leads to long project cycles, increased technical and business risk, reduced quality, and reduced adoption of model-based insights by the field.


WHAT NEEDS TO CHANGE?

1. Establish explicit connections between model scores and business decisions.

2. Data structures and systems used in development and deployment are different, and are changing even faster.

3. Changes in models and their performance over time need to be easily interpretable, assessed and catalogued.

4. To deploy and execute models in production requires practices that are not commonly adopted by model developers.


THE FIVE D’S OF MODEL DEPLOYMENT

Develop: Model development and packaging

Decisions: Tie operational business decisions to model scores

Data: Operationalizing analytic model deployment in a specific data architecture

Delta: Monitor the workflow and performance of models in the field

Deploy: Implement analytic models via a software development life cycle


#1: MODEL DEVELOPMENT


DEVELOP: LESSONS LEARNED

• Understand the capabilities and limitations of the IT production facilities and standards up front.

• Educate IT about the analytics process.

• The tools you use and the models you build need to be robust to changing field conditions.


#2: DECISIONS

• Scores are not decisions. A decision is the proactive response of the business to the prospective customer behavior, involving the expenditure of resources.

• An evidence-based, virtuous cycle needs to be constructed for making the best decisions.

• Test and learn practices are the best option for businesses willing to commit to longer-term testing strategies that maximize the value of information.


CASE STUDY: NEXT BEST OFFER STRATEGYThe challenge• Client: The retail marketing group for one of the top 50 banks in the world. • The bank issues hundreds of offers every month via 30+ campaigns.• Marketing leadership knew that ranks and scores weren’t enough to make the

final decision about which offer to assign to each customer. They also had to balance product sales goals, cross-product halo effects, over-time contact strategies and offer cost and margin contribution alongside customer likelihood to respond.

• In short, which offer should they give each customer in order to grow profitably?

The solution • The bank implemented a mathematical campaign optimization routine to allocate

the margin-maximizing offer per client for each monthly campaign wave. • After only four months of design and implementation, the bank released their first

optimized campaigns out the door.

The results• Comparing year-over-year campaign results, the optimized campaigns executed

over a two-month period in summer 2012 produced a net increase of $22 in gross sales per customer. In the aggregate, profitable sales growth averaged $3.5 million in monthly incremental campaign-driven income.


DECISIONS: LESSONS LEARNED

•Don’t only implement business rules based on judgment and experience; also build prescriptive rules optimizing for trade-offs between alternatives in pure dollars and cents terms.

• The best performing offers and treatments are not the ones you will issue tomorrow; they will be the offers that your organization has refined over several waves of disciplined, rigorous trials, paired with conscientious measurement.


#3: DATA ARCHITECTURES

• Batch scoring in the warehouse worked fine 10 years ago.

• Larger pools of models, more rapid refresh, and innovative model scoring methods now dominate attention.

In-database analytics In-memory analytics

Over-time analytics On-demand analytics


CASE STUDY:IN-DATABASE ANALYTICSThe challenge• Client: The data sciences team for a large investment brokerage. • The brokerage has built response scoring models for their customers,

used to advise financial advisors on a periodic basis. • The brokerage needed to dramatically reduce the cycle time needed to

develop new models and to refresh their scores, so that financial advisors could have daily updates on the best treatment for their customers, and hence grow their most profitable relationships.

• The solution • Corios designed and implemented massively-parallel model training and

model scoring routines, and trained the client team to modernize the remainder.

• The results• Compute tasks that previously required days to run, now run in a few

minutes or less, on billions of transactions. • This also returned lifts on models of 10-20% by taking more

transactional predictors into account, not previously possible, due to computational scale limits.


DATA: LESSONS LEARNED

• Batch warehouse updates still work well if your refresh timelines are monthly and customers don’t migrate rapidly.•Otherwise:•Consider in-database analytics when transaction-level trends are important for predictions, and you’ve already made a commit to data appliances or distributed file systems.•Consider in-memory analytics when stakeholders need to make detail-sensitive decisions with intense, interactive visual input.•Consider over-time analytics for advanced trending, such as stress testing, customer survival, and new product adoption. •Consider on-demand scoring for deployment on individual customers, given a high degree of confidence in your models’ reliability, strong believability in the field, and patience for testing.


#4: DELTA: MONITORING MODEL PERFORMANCE

•Document the scope, ownership, quality assurance, approval and signoff, and lifecycle of a model asset.

• Understand the model’s insights, including the interpretation, defensibility, and identification of the best business actions to take based on a model score.


EXAMPLE: CREDIT RISK MODEL CALIBRATION


DELTA LESSONS LEARNED

• Lack of visibility into model inventories is a source of regulatory pressure that needs to be met now, rather than “a few years from now”.

•Workflows for tracking the model asset through its stages of development, review, approval and retirement, and commit the appropriate staffing to support.

•Models are worth hundreds of times their development cost, deserving prudence comparable to managing assets such as human resources or physical plant.


#5: DEPLOY

•Deploying models in production needs to be a structured, disciplined activity.

• The software development life cycle (“SDLC”) is an apt model, with some minor adaptations.


THE SDLC APPLIED TO MODEL DEPLOYMENT


DEPLOYMENT: LESSONS LEARNED

• Select a single platform for model development and deployment in order to reduce the quality assurance challenges that arise when translating from one platform’s code base to another.

•Quality assurance can be improved by using peer review, version and change management practices, and standards for model documentation and packaging.

•Model performance should be tracked both on accounts that get scored on-demand during the day, as well as on all accounts on a periodic basis, for tracking model stability.


THE FIVE D’S OF MODEL DEPLOYMENT

Develop: Model development and packaging

Decisions: Tie operational business decisions to model scores

Data: Operationalizing analytic model deployment in a specific data architecture

Delta: Monitor the workflow and performance of models in the field

Deploy: Implement analytic models via a software development life cycle

JOIN US FOR THE NEXT IIA WEBCASTS

July 24, 2013 Analytics 3.0: Opportunities for Healthcare

Thomas H. Davenport and Jack Phillips

Register at iianalytics.com


[email protected]

Bringing Analytical Models in to The Business Field

Business

Transcript of Bringing Analytical Models in to The Business Field