7/18/16

1

July20,2016ö ACCChicago

Zev J. EigenGlobal Director of Data Analyticszeigen@littler.com

@MMAFA_z

David Christliebdchristlieb@littler.com

1. A brief history of prediction modeling (from 2000 BCE to 2016)

2. A briefer explanation of data science3. An overview of data scientific applications in HR4. A few examples of those applications5. Values and risks of data science approach6. Recommendations

Overview

2

7/18/16

2

The Basic Progression

3

Expertise, knowledge, & intuition

Statistics and econometrics

Data science, machinelearning and cognitive computing

Predictive Analytics are profitable

4

7/18/16

3

“Big Data” are sources of information (transactions, observations, interactions) too big or fast for traditional

analysis

EE demographics; HRIS data; payroll data; compliance tracking

Performance data; Engagement data; talent and performance assessments

A/B testing; web logs; network analyses; behavioral data

Real time behavioral data; sentiment analysis; click-stream data; video, audio, images, SM/MMS, email meta data, publically available high velocity information

“Big Data”

Increasing variety, complexity, and velocity

1. Data used to benchmark (Why is A performing better than B?)2. Data used for recommendation and filter systems

– Content based recommendation engines– Collaborative filtering– Hybrid

3. Data used for predictions4. Data used to describe and understand statistical relationships

4 distinct data driven services

6

7/18/16

4

7

Sales, Marketing, Advertising

Client facing applications

Internal applications (Finance, Strategy...)

HR

LAW

HR analytics are profitable

• HR organizations that “regularly use data to make talent and HR strategy decisions” generate 30% greater stock returns than the S&P 500 over the last 3 years.

8

7/18/16

5

78% of HR directors rate people analytics in their organization as “developing”. The others (22%) are in the early stages.

INITIATINGDEVELOPING

ADVANCING

9

Administrative & Compliance Data Talent Management Data Social & Behavioral Data

Real time behavioral dataPassive candidate employment and personal preferences

Performance evaluationsEE engagement resultsTalent assessment results

Past experiences, skills and languages

Learning effectiveness Professional & social networks

EE DemographicsHRIS / Payroll dataCompliance tracking

Email meta data; publically available EE data

High quality data sources with predictive power are proliferating

1990 2016 10

7/18/16

6

Hiring Performance Evaluation

Promotion & Talent-position

matching

Policy Optimization Attrition

5 areas in which data science have been applied in HR decision-making

11

• Which new locations are most likely to result in the ability to hire and retain the most talented employees?

• Which of 3 contemplated policies is optimal to reduce turnover, reduce workplace conflict, and improve morale?

• Which applicants pose the lowest risk of theft or abuse of paid time off?

• Which employees are likely to voluntarily quit in the next 6 months?

Examples of predictive model questions

12

7/18/16

7

Data Source Best at predicting: Risk(s):

Video data from interviews Ruling out bad candidates (reliability) Replication of bias in favor of homogeneity

Unstructured text data (resumes and writing samples, etc.) “Fit” matching; field competence; Failure to identify value from

exogenous sources/dissimilarity

Behavioral indicators (public data like LinkedIn, Twitter, Reddit)

“Fit” matching, reliability, ruling out bad candidates Cohort effects, type I errors

Behavioral data (video game play) Risk-taking, impulsivity, conscientiousness, patience Unstable, expertise “bias”

User entered data (surveys,psychometrics, demographic info, etc.) Conformity to existing benchmarks Self-report bias

Hiring: Whom should the employer hire?

13

Hiring: Using Data Science to Improve Hiring Processes & Outcomes

7/18/16

8

An Example: Cherry Tree Data Science

15

% Likelihood of Criminal Behavior

Applicants WITH criminal records

Applicants WITHOUT criminal records

CTDS shows employers which applicants with

criminal records are no more “risky” than applicants already

deemed acceptable to hire.

7/18/16

9

Micro-level Example: Policy Optimization

17

Micro-Behavioral Example: Predicting Lying

18

7/18/16

10

Attrition: Who is likely to voluntarily quit?

19

Data Source Analytic Concern(s): Legal Risk(s):Behavioral data (email, RFID, etc.) Undetectable systematic variation Discrimination, privacy

Behavioral indicators (public data like LinkedIn, Twitter, Reddit)

Cohort effects; interpretation of missing data; system-gamable(?)

Discrimination, NLRAissues

User entered data (surveys,psychometrics, demographic info, etc.)

System-gamable (type I and type II errors); validity, failure to detect mediating effects, endogeneity

Discrimination

Social Network Analysis Data Unreliability of self-reported data; methodological problems Discrimination

19

An Example: HiQ Labs

20

7/18/16

11

1. Increases the availability of information on which to rely for decisions

2. Increases the reliability of information drawn from diffuse sources3. Increases the accuracy of decision-making criteria (by applying

algorithms to tether hiring criteria to objective performance measures)

4. Reduces bias (both illegal bias and bias that is not illegal but is inefficient)

What is the value of a data-analytic approach to organizational decision-making?

21

1. Analytics done incorrectly yield sub-optimal, erroneous, and costly decisions:

– Type-I errors (erroneous conclusion that risk exists when it doesn’t)

– Type-II errors (failure to properly detect risk)2. Reports not protected by attorney-client privilege may result in

discoverable “smoking gun” evidence in litigation.3. ”Insider” analytics may yield erroneous or self-interested results.

What are the risks of a data-analytic approach to HR decision-making?

22

7/18/16

12

• “4 out of 5” dentists problem.

• Agency costs.• Difficulty seeing

problems in one’s own organization.–Example: hidden bias

The drawback of “insider” analytics

23

Big Data offers great value, but may pose huge risks

• FCRA• ECOA• Title VII• ADA• ADEA• FHA• GINA

FTC advises that companies “should

review these laws and take steps to ensure their use of big data analytics

complies with the discrimination

prohibitions that may apply.”

24

7/18/16

13

• “Big data will face huge challenges around privacy, especially with the new privacy regulation by the European Union. Companies will be forced to address the “elephant in the room” around their privacy controls and procedures.” (Forbes, 3/15/2016)

• Gartner Inc. predicts that by 2018, 50% of business ethics violations will be related to data.

• We predict that by 2020, most privacy causes of action will be related to data.

Privacy Issues

25

• Ensure that multiple data sources are examined and cross-validated. • Don’t replicate one uniformly dictated approach with another. Test and re-test.• Integrate data scientific approaches into decision making processes, but avoid

replacing discretion with algorithms wholesale.• Legal risk is VERY costly, and often personally costly. Avoid/reduce it by ensuring

legal risk evaluation at 2 process points:– Feature identification / Model building– Report generation: don’t create “smoking gun” documents– Taking action on recommendations/ output

• Remember: Data are NEVER protected by attorney-client privilege. Reports generated internally in the ordinary course of business are often NOT protected by attorney-client privilege.

Recommendations

26

7/18/16

14

Thank you

Zev J. EigenGlobal Director of Data Analyticszeigen@littler.comTwitter: @MMAFA_z

David Christlieb

dchristlieb@littler.com

www.littler.com Page 1 C O N F I D E N T I AL

Speaker Biographies

ZEV J. EIGEN GLOBAL DIRECTOR OF DATA ANALYTICS

DAVID L. CHRISTLIEB SHAREHOLDER

www.littler.com Page 2 C O N F I D E N T I AL

Zev J. Eigen Global Director of Data Analytics

L.A. - Century City 2049 Century Park East 5th Floor Los Angeles, CA 90067

Direct: (310) 772-7242 zeigen@littler.com

Practice Areas Littler Big Data Initiative Class Actions Wage and Hour Employment Practices Audits Hiring, Performance Management and Termination Robotics, Artificial Intelligence (AI) and Automation Workplace Policy Institute

Overview

Dr. Zev J. Eigen combines his expertise in labor and employment law with his deep experience in complex data analytics and social scientific research to handle three categories of work: (1) predictive analytics using artificial intelligence or “machine learning” algorithms as applied to HR and related business decisions; (2) statistical analysis and econometric modeling of issues arising out of class actions and pattern and practice matters; and (3) statistical analysis of labor, employment and HR related data.

He is a nationally recognized expert in these fields appearing in the media frequently (Wall Street Journal, New York Times, Forbes, NPR, Bloomberg News, Reuters, Chicago Tribune, NBC, CBS, FOX, etc.). Zev is a frequent speaker on matters pertaining to workplace data analytics.

Zev is an accomplished public speaker and is invited regularly to lecture and present papers at academic institutions and professional organizations across the country.

Recognition

• Named, Fastcase 50, 2016 • Named, 40 Under 40 Rising Legal Stars, National Law Journal, 2013

• Refereed Paper Competition Winner, Labor and Employment Relations Association (LERA), 2011

• Recipient, Joel Seideman Memorial Prize, Cornell University, 1996

www.littler.com Page 3 C O N F I D E N T I AL

Education Ph.D., Massachusetts Institute of Technology, 2009 J.D., Cornell University Law School, 1999 B.S., Cornell University, 1996, With Honors

Bar Admissions California

Courts U.S. District Court, Central District of California U.S. District Court, Northern District of California U.S. District Court, Eastern District of California U.S. District Court, Southern District of California U.S. Court of Appeals, 9th Circuit U.S. Court of Appeals, 11th Circuit U.S. Court of Appeals, D.C. Circuit

Publications and Press • Data Science Opportunities and Challenges in the Legal and Human Resources Space , Littler

Podcast, May 27, 2016

• Law Firms Increasingly Use 'Moneyball' Analytics In Lateral Partner, TaxProf Blog, April 7, 2016 • Big Data, Artificial Intelligence and Analytics in the Workplace, Part 1, Littler Podcast, April 6,

2016 • Big Data in Employment, Today's General Counsel, February 26, 2016 • The Rise of the Chief Data Scientist, Legaltech News, February 1, 2016 • FTC Report: With Big Data Can Come Big Responsibility, Littler ASAP, January 22, 2016 • Machine Learning: Cybersecurity Dream-Come-True or Pipe Dream?, CSO, December 18, 2015 • Legal Tech Digest: 4 Trends to Watch, Legaltech News, October 30, 2015 • Employers Underuse 'Big Data', SHRM, October 19, 2015 • Post-Racial Hydraulics: The Hidden Dangers of the Universal Turn, New York University Law

Review • Littler Hires Its First National Director of Data Analytics and Launches Data Center, Littler Press

Release, August 31, 2015 • When Rules are Made to be Broken, 129 Northwestern University Law Review 107, 2014 • Less is More: A Case for Structural Reform of the National Labor Relations Board, 98 Minnesota

Law Review 1880 , 2014 • Justice or Just between Us? Empirical Evidence of the Trade-off between Procedural and

Interactional Justice in Workplace Dispute Resolution, 67 Industrial and Labor Relations Review 171, 2014

• Do Lawyers Really Believe Their Own Hype and Should They? A Natural Experiment, 41 Journal of Legal Studies 239 , 2012

• Empirical Studies of Contract, 8 Annual Review of Law and Social Science 291 , 2012 • When and Why Individuals Obey Contracts: Experimental Evidence of Consent, Compliance,

Promise, and Performance, 41 Journal of Legal Studies 67 , 2012

www.littler.com Page 4 C O N F I D E N T I AL

• Experimental Evidence of the Relationship Between Reading the Fine Print and Performance of Form-Contract Terms, 168 Journal of Institutional and Theoretical Economics 124, 2012

• A Moral / Contractual Approach to Labor Law Reform, 63 Hastings Law Journal 101, 2012 • Shifting the Paradigm of the Debate: A Proposal to Eliminate At-Will Employment and Implement

a ‘Mandatory Arbitration Act’, 87 Indiana Law Journal. 271, 2011 • The Devil in the Details: The Interrelationship among Citizenship, Rule of Law and Form-

Adhesive Contracts, 41 Connecticut Law Review 381 , 2008 • Don’t Train Your Employees and Cancel Your 1-800 Harassment Hotline: An Empirical

Examination and Correction of the Flaws in the Affirmative Defense to Sexual Harassment Charges, 69 Fordham Law Review 1265, 2001

• In Defense of Mandatory Arbitration of Employment Disputes: Saving the Baby, Tossing Out the Bath Water and Constructing a New Sink in the Process, 2 University of Pennsylvania Journal of Labor and Employment Law 73 , 1999

Events & Speaking Engagements • Closing the Gender Pay Gap: The Role of State Fair Pay Laws, The 2016 Executive Employer®

Conference, May 5, 2016

• Using Data Analytics in Assessing Litigation Risks, May 4, 2016 • Using Data Analytics in Onboarding and Employee Performance Measurement, May 4, 2016 • California Passes Sweeping New Law Aimed to Bridge the Gender Wage Inequality Gap ,

October 19, 2015 • HR in Hospitality Law Roundtable, Las Vegas, NV, March 16-18, 2015 • Lecturer, MIT/Harvard Economic Sociology Seminar , 2014 • American Law & Economics Association Annual Conference , 2011-2014 • Conference on Empirical Legal Studies , 2010-2014 • When and Why Individuals Obey Form-Adhesive Contracts: Experimental Evidence of Consent,

Compliance, Promise, and Performance, University of Chicago Law & Economics Workshop , April 12, 2011

• HR in Hospitality Annual Conference, Washington, D.C. , April 2011 • Labor & Employment Research Relations Association Annual Meeting, Denver, CO, January 7-

9. 2011

Books & Book Chapters

• Foundations of Labor and Employment Law, Matthew Bender Publishing, with Samuel Estreicher and Stewart Schwab, 2012

• The Forum for Adjudication of Employment Disputes, Research Handbook on the Economics of Labor and Employment Law, Michael L. Wachter, Cynthia L. Estlund, eds. Edward Elgar Publishing Ltd., with Samuel Estreicher, 2012

• Labor and Employment Law Initiatives and Proposals Under the Obama Administration: Proceedings of the New York University 62nd Annual Conference on Labor, volume editor, Kluwer Law International, 2011

www.littler.com Page 5 C O N F I D E N T I AL

David L. Christlieb Shareholder

Chicago 321 North Clark Street Suite 1000 Chicago, IL 60654

Direct: (312) 795-3264 dchristlieb@littler.com

Practice Areas Class Actions Affirmative Action/OFCCP Compliance Labor Management Relations Discrimination and Harassment Litigation and Trials

Overview

David L. Christlieb litigates in a broad range of employment law areas, including:

• Discrimination

• Wrongful termination • Retaliation

• Wage and hour issues

• Restrictive covenants • Trade secrets

• Expert witness issues

• Adverse impact analyses • Government investigations and audits

• National Labor Relations Board elections

• Collective bargaining • Diversity initiatives

• Workplace violence issues

• Employee/applicant assessment issues • Traditional labor matters

www.littler.com Page 6 C O N F I D E N T I AL

He has particular expertise in the area of affirmative action and has created more than 500 affirmative action plans, and he also advises clients regarding compliance, labor and management relations, and EEO policy.

A member of the firm's Class Action Practice Group, David's litigation experience includes complex nationwide collective and class actions under:

• The Fair Labor Standards Act

• Title VII

• Section 1981 • Various state statutes

He has represented clients, including manufacturers, retailers, restaurants and consulting firms, in federal and state courts and in arbitrations and mediations before the Illinois Department of Human Rights, the National Labor Relations Board, the Equal Employment Opportunity Commission, the Office of Federal Contract Compliance Programs, and the Department of Labor. David is frequently called upon to cross-examine or depose expert witnesses including economists, statisticians, psychologists, and sociologists.

Among David's notable successes, he has successfully argued an appeal in a discrimination lawsuit for a national manufacturer before the 2nd Circuit Court of Appeals, has won a high-stakes unfair labor practice trial over allegations that his client’s lockout of union employees was unlawful, and tried one of the first cases applying the “strike then lockout” exception to New Jersey’s unemployment compensation law. He has provided pro bono representation of indigent students in school expulsion cases.

David serves as the hiring shareholder in Littler Mendelson's Chicago office, and developed a litigation training program for associates.

In addition to his law degree, David holds a masters degree in human resources and industrial relations from the University of Illinois. In law school, he served as the notes editor for the Elder Law Journal.

Professional and Community Affiliations

• Member, Illinois State Bar Association • Member, Chicago Bar Association

Recognition

• Recipient, Rickert Award for Excellence in Legal Publications

Education J.D., University of Illinois College of Law, 2003, magna cum laude M.H.R.I.R., University of Illinois, 2003 B.A., Knox College, 2000, cum laude

Bar Admissions Illinois

www.littler.com Page 7 C O N F I D E N T I AL

Courts U.S. Court of Appeals, 2nd Circuit U.S. Court of Appeals, 6th Circuit U.S. Court of Appeals, 7th Circuit U.S. District Court, Northern District of Illinois U.S. District Court, Central District of Illinois U.S. District Court, Western District of Michigan U.S. District Court, Northern District of Florida

Publications and Press • Littler Mendelson Announces 13 Newly-Elevated Shareholders, Littler Press Release, January

19, 2010

• Illinois Supreme Court Applies Strict Liability to All Workplace Sexual Harassment By Supervisors, Littler Insight, April 23, 2009

• The Employee Free Choice Act: A Critical Analysis, Littler Report, July 24, 2008 • The Perils of Union Activism Have Been Greatly Exaggerated, Littler Article, May 21, 2007

Events & Speaking Engagements • OFCCP's Final Regulations for Veterans and Individuals with Disabilities, March 18, 2014

• Best Hiring Practices for Screening Talent, Chicago, IL, October 2, 2013 • Affirmative Action Boot Camp, Chicago, IL, October 3, 2012 • Grand Slam: Covering the Four Agencies That Effect Employers in a Major League Way, Littler

Mendelson, St. Louis, MO, June 13, 2012 • The Retail Industry Summit, Littler, Scottsdale, AZ, May 9, 2012 • Littler Presents Our Affirmative Action Boot Camp, Chicago, IL, June 16, 2011 • The Employment Compliance Costs of Doing Business with the Federal Government , Chicago,

IL, November 6, 2010 • Class Action Summit (Invitation Only), Rancho Palos Verdes, CA , September 23, 2010