Two Numeric Concerns for IVF Laboratorians:

David H. McCulloh, Ph.D., H.C.L.D., C.C.

New York University Langone Medical Center

NYU Fertility Center

1) Staffing the lab, and 2) Statistical Significance versus Clinical Relevance

Disclosures:

Assistant Laboratory Director -

Andrology Laboratory Director -

Director of Clinical Science -

President & Principal Scientist -

Learning Objectives:

• The types of procedures that we perform in the ART laboratory are changing – The complexity of procedures impacts personnel

requirements

• There are two components to staffing the ART lab – Fixed component – Variable component

• Adjustments to the calculator for part-time programs • There are ways to estimate the clinical relevance of

statistically significant observations

Volume of Procedures each Year

0

500

1000

1500

2000

2500

2000 2005 2013 2014

Year

Tota

l Nu

mb

er

of

Pat

ien

t-P

roce

du

res

Distribution of Procedures

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2000 2005 2013 2014

Year

Frac

tio

n o

f P

atie

nt-

Pro

ced

ure

s (%

)

Thaws

Banking

IVF/Bx

IVF/ET

It isn’t the same anymore…

• NYUFC is getting busier

• NYUFC has changed the types of procedures that are performed

– More diverse

– More complex

• Now we treat individual oocytes not just patients

• Must track each oocyte/embryo for PGS/PGD

– More witnessing

Old Rule of Thumb:

The IVF lab needs one embryologist for every 100 IVFs

IVF Traditional

IVF Contemporary

IVF /PGS

Procedure Time

Witness Complexity Procedure

Time Witness Complexity

Procedure Time

Witness Complexity

Preparation All 30 0 6 60 0 12 80 0 16

Oocyte Retrieval 60 10 12 60 10 12 60 10 12

Sperm Preparation 60 10 6 60 10 6 60 10 6

Insemination/ICSI 40 20 12 40 20 12 40 20 12

Fertilization Check 40 10 6 40 10 6 40 20 6

Day 2 Check 20 0 2 20 0 2 20 0 2

Day 3 Check 20 0 2 20 0 2 20 0 2

Day 3 Transfer 40 10 16 40 10 16 0 0 0

Day 3 Cryo 40 10 8 0 0 0 0 0 0

Assisted Hatching 20 0 4 20 0 4 60 0 12

Extended Culture 0 0 0 40 10 4 40 10 4

Day5 Check 0 0 0 20 0 2 20 0 2

Day 5 Transfer 0 0 0 0 0 0 0 0 0

Day 5 Biopsy 0 0 0 0 0 0 80 40 32

Day 5 Cryo 0 0 0 40 20 8 80 40 16

Day 6 Check 0 0 0 20 0 2 20 0 2

Day 6 Biopsy 0 0 0 0 0 0 80 40 32

Day 6 Cryo 0 0 0 40 20 8 80 40 16

Document Management 30 0 0 30 0 0 30 0 0

Number of hours 6.67 1.17 1.23 9.17 1.83 1.60 13.50 3.83 2.87

Total Time (hrs) 9.07

12.60

20.20

Complexity has increased!

• More person-hours per case

– Witnessing

– Tracking individual oocytes/embryos

– Prolonged culture duration

– Multiple cryopreservation events per patient

– Biopsy

Table 3. Activities in the IVF laboratory including components and estimated complexity level

Activity Component 1 Component 2 Component 3 Component 4 Component 5 Component 6 Complexity

Case Set-up Day -1 Record Review Need Assessment

Dish Labeling Media Preparation

Dish Preparation

2

Oocyte Retrieval; Day 0 12 Oocytes

Lab Preparations

Follicular Fluid Search

Cumulus Dissection/Wash

Oocyte Culture Witnessing

2

Oocyte Cryo; 10 Oocytes

Record Review/Pt ID

Media/Dish Preparation

Cryo Container Preparation

Denuding/ Evaluating Eggs

Vitrification Witnessing 4

Oocyte Thaw; 10 Oocytes

Record Review/Pt ID

Media Preparation

Dish Preparation

Oocyte Warming

Oocyte Culture Witnessing 4

Surgical Sperm Retrieval

Lab Preparations

OR Procedures

Tissue/Sample Processing

Tissue/Sample Cryo

Witnessing

2

Sperm Preparation; Simple

Semen Analysis

Gradient Preparation

Sample Preparation

Analysis Witnessing 1

Sperm Preparation; Complex

Semen Analysis

Special Treatments

Sample Preparation

Analysis Witnessing 1

Insemination; Standard

Record Review/Pt ID

Oocyte Preparation

Insemination Drop Prep

Insemination Witnessing 2

ICSI; Simple; 12 Oocytes

Record Review/Pt ID

Oocyte Preparation

Dish Preparation

Microinjection Witnessing 4

ICSI; Complex; 12 Oocytes

Record Review/Pt ID

Oocyte Preparation

Dish Preparation

Sperm Search Microinjection Witnessing 4

PICSI Record Review/Pt ID

Oocyte Preparation

Dish Preparation

Sperm Search Microinjection Witnessing 4

Insemination; Split ICSI/Standard

Record Review/Pt ID

Oocyte Preparation

Insemination Dish Preparation

Microinjection Witnessing 4

Fertilization Check; Standard

Oocyte denuding

PN Assessment

Zygote Culture Witnessing

2

Fertilization Check; ICSI

Pronucleus Assessment

Zygote Culture

Witnessing

1

Staffing: Two components

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

0 200 400 600 800 1000 1200 1400 1600 1800 2000

Number of IVFs per Year

Nu

mb

er

of

pe

rso

nn

el r

eq

uir

ed

Based on Procedure Performance

Fixed number to staff the lab

Greater of the two methods

ASRM Guidelines

Calculation of Staffing Requirements for an Embryology Laboratory

Minimum Personnel Required (MPR)

MPR(procedures) =

# IVF’s (without PGS/PGD) X 0.00672

+ # IVF’s (with PGS/PGD) X 0.01077

+ # FET’s X 0.00192

+ # Oocyte Freezes X 0.00299

+ # Oocyte Thaws X 0.00686

+ Days of Operation (Days of QC activity) X {(# Incubators QC’d X 0.0000347) + 0.000267}

MPR(2 per day) =

Days of Procedures X 0.008

MPR = Maximum(MPR(procedures), MPR(2 per day))

Load the Staffing Program, Dave….

But there is a problem….

• The Calculator really works only for full time labs

• We need a correction for part-time facilities

• Must divide the Total Person Years required

– By the staffing hours of operation per year

(hours of operation in one year)

= Results X (hours/person-year) / (hours of operation)

Clinical Relevance vs. Statistical Significance

• Problem

– Statistical significance is generally based on determinining if there is significance between mean values

• The best estimate of the mean value is the mean with a vaariability of +/- Standard Error of the Mean

• The best estimate of a single variate is the mean with a variability of +/- Standard Deviation

– Usually, a patient will provide an N of 1 to the outcome

List of Examples: Are these Clinically Relevant Effects?

• hCG on day 28 (14 days after retrieval) predicts outcome (beating fetal heart)

• Premature Births comparing Day 3 with Day 5 Transfer

• Incubator parameters (QC params) predict pregnancy loss (live birth vs. preg. loss)

An Analytical Trap

• Pregnancy Test (hCG) values predict outcome?

– Comparison of predictor values between two groups segregated by outcome

– Instead, Receiver Operator Characteristic curve should be used

[hCG] on Day 28 (619 delivered and 1299 did not deliver)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 10 100 1000

Cu

mu

lati

ve I

nci

de

nce

[hCG] on Day 28 (mIU/mL)

Delivered

Did Not Deliver

1347 1040

20

40

60

80

100

120

140

160

180

200

Yes No

hC

G (

mIU

/m

L)

on

da

y 2

8

Fetal Heart Beat Observed with Ultrasound?

hCG is Significantly Different for Patients with/without FHB (p ~0.00000046)

But…. Can we use hCG to predict whether an FHB will be seen?

But…. Can we use hCG to predict whether an FHB will be seen?

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Tru

e P

os

itiv

e (

Se

ns

itiv

ity)

False Positives (1 - Specificity)

Low Day 28 hCG Predicts NO Fetal Heartbeat (AUC = 0.673)

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Tru

e P

osi

tiv

es

(Se

nsi

tiv

ity

)

False Positives (1 - Specificity)

0.5

0.7

0.8

0.9

0.95

Incidence of Loss after +β

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 10 100 1000

Inci

de

nce

of P

regn

ancy

Lo

ss

(Fra

ctio

n o

f P

osi

tive

hC

Gs)

[hCG] (mIU/mL on day 28)

Does [hCG] Predict Outcome?

• Despite the highly significant difference between [hCG]’s comparing

– Fetal Cardiac Activity seen

– Fetal Cardiac Activity NOT seen

• [hCG] only poorly predicts whether FCA will or will not be seen

Adverse Outcomes: Day 3 vs. Day 5

Ob. Gyn. (2012) 120(1): 69-75

Ob. Gyn. (2012) 120(1): 69-75

Kalra et al. Data

• Using a large data set from SART spanning 3 years (69,039 live births in 2004 - 2006)

• Concluded that culture to day 5 (relative to day 3) resulted in higher incidence of premature delivery

• Close examination of their data reveals that the mean delivery date was 2.8 days earlier for day 5 embryos (37.8 versus 38.2 weeks)

• Mean birth mass at delivery was 29.2 grams lower for day 5 than for day 3 (3219.7 g versus 3,248.9 g)

P > 0.75

Comparison of Gestational Age at Birth: Day 3 vs. Day 5 at NYUFC

Comparison of Birth Weight: Day 3 vs. Day 5 at NYUFC

P > 0.2

Using the data

• Is this significantly different? – 2.8 day / 0.14 (SE) yes, indeed 2.8 days is significant

• A 2.8 day difference in the means

• But is 2.8 days clinically relevant? – 2.8 days within a 70 day range (4 X S.D.)

• Is this significantly different? – 29.2 g / 2.8 g (SE) yes, indeed 29.2 g is significant

• A 29.2 g difference in the means

• But is 29.2 g clinically relevant? – 29.2 g within a 2448 g range (4 X S.D.)

Incubator QC Parameters • Analysis of quality control data

– Fluctuations – are there trends with QC values?

• Multiple regression

• Can we find laboratory parameters that are associated with outcomes?

– Clinical Pregnancy

– Live Birth

– Live Birth per Clinical Pregnancy

• The inverse of clinical pregnancy loss…

Table I: Values of Daily Laboratory Parameters1

1 Displaying the mean + standard deviations (n) for the parameters examined for association with outcomes (Clinical Pregnancy, Live Birth, or Live

Birth per Clinical Pregnancy). 2 the day of embryo culture where 0 is the day of oocyte retrieval. 3 age (years) of the female patient on the day that gondatotropin injections were initiated. 4 temperature (oC) 5 percentage of the incubator gaseous environment that comprised carbon dioxide (%) 6 relative humidity (%)

Culture

Day2

Female

Patient

Age3

Incubator

Temperature4

Incubator

CO25

Incubator

Humidity6

Room

Temperature4

Room

Humidity5

Inverted

Microscope

Stage

Temperature4

Dissecting

Microscope

Stage

Temperature4

FIV

Temperature4

Number of

Patients in

Incubator

0 35.9

+ 4.6

(1503)

37.07

+ 0.13

(1503)

5.39

+ 0.29

(1503)

88.2

+ 5.8

(1503)

21.7

+ 1.1

(1503)

36

+ 14

(1503)

36.80

+ 0.39

(1503)

36.75

+ 0.42

(1503)

37.00

+ 0.10

(1503)

2.16

+ 0.99

(1503)

1 35.9

+ 4.6

(1508)

37.08

+ 0.13

(1508)

5.39

+ 0.29

(1508)

88.1

+ 6.5

(1508)

21.6

+ 1.0

(1508)

36

+ 14

(1508)

36.79

+ 0.38

(1508)

36.76

+ 0 36

(1508)

37.00

+ 0.07

(1508)

2.17

+ 0.99

(1508)

2 35.9

+ 4.6

(1498)

37.07

+ 0.15

(1498)

5.39

+ 0.30

(1498)

88.1

+ 6.2

(1498)

21.6

+ 1.1

(1498)

36

+ 14

(1498)

36.80

+ 0.41

(1498)

36.75

+ 0.37

(1498)

37.00

+ 0.07

(1498)

2.20

+ 1.01

(1498)

3 35.8

+ 4.6

(1513)

37.07

+ 0.14

(1513)

5.39

+ 0.29

(1513)

88.1

+ 6.0

(1513)

21.6

+ 1.0

(1513)

36

+ 14

(1513)

36.80

+ 0.41

(1513)

36.72

+ 0.38

(1513)

37.00

+ 0.07

(1513)

2.19

+ 1.01

(1513)

4 36.7

+ 3.9

(343)

37.07

+ 0.12

(343)

5.42

+ 0.29

(343)

87.7

+ 5.1

(343)

21.6

+ 1.0

(343)

35

+ 14

(343)

36.86

+ 0.42

(343)

36.75

+ 0.40

(343)

37.00

+ 0.02

(343)

2.15

+ 0.98

(343)

5 36.6

+ 4.0

(345)

37.07

+ 0.20

(345)

5.41

+ 0.29

(345)

87.7

+ 5.8

(345)

21.6

+ 1.0

(345)

35

+ 14

(345)

36.83

+ 0.37

(345)

36.7

+ 0.40

(345)

37.00

+ 0.02

(345)

2.11

+ 1.04

(345)

Table IV: Multiple Logisitic Regression Coefficients – Live Birth per Clinical Pregnancy1

Cultur

e

Day2

Femal

e

Patient

Age3

Incubator

Temperatur

e4

Incubato

r CO25

Incubato

r

Humidity6

Room

Temperatur

e4

Room

Humidity6

Inverted

Microscope

Stage

Temperatur

e4

Dissecting

Microscope

Stage

Temperatur

e4

FIV

Temperatur

e4

Number

of

Patients

in

Incubato

r

Offset

0 -0.140 0.0281 3.607

1 -0.131 -0.739 0.0362 6.550

2 -0.135 -0.929 10.886

3 -0.128 -0.466 0.0303 5.454

4 -0.164 -1.345 56.688

5 -0.175 -0.778 36.091

1 Displaying only those coefficients for parameters found to be associated with the outcome (Live Birth per Clinical Pregnancy) when the Akaike

Information Criterion was applied following maximum likelihood fitting. Blank cells were retained to indicate parameters that were examined but

found not to be associated with Live Births per Clinical Pregnancy. 2 the day of embryo culture where 0 is the day of oocyte retrieval. 3 age (years-1) of the female patient on the day that gondatotropin injections were initiated. 4 temperature (oC)-1 5 percentage of the incubator gaseous environment that comprised carbon dioxide (%)-1 6 relative humidity (%)-1

Lab Parameters: Multiple Regression

• Several Parameters Examined Simultaneously – Some params only significant when adjustments made

• Sensitivity of embryos may vary with stage – More sensitive prior to compaction?

• Surprising sensitivity of clinical pregnancy loss – increased losses with – Lower Humidity - Days 0 – 3

– Higher CO2 - Days 1,2,3

– This sensitivity is important to consider!

• Must be cautious of multicollinearity

Regression Equation for Day 3:

• Ln(ORlivebirth/clinical pregnancy) =

-0.128 X Female Patient Age (years)

-0.466 X Incubator [CO2] (%)

+0.0303 X Incubator Humidity (%)

+ 5.454

(from multiple logisitic regression; N = 1498)

Relative effects on 73.7% LB/CP • Age: 35.8 + 4.6 years

+ 2 std dev (26.6 – 45 years)

Change of live birth rate -43.7% (90.0 – 46.2%)

• CO2: 5.39 + 0.29 %

+ 2 std dev (4.81 – 5.97% CO2)

Change of live birth rate -10.4% (78.6 – 68.1%)

• Humidity: 88.1 + 6.0 %

+ 2 std dev (76.1 – 100.1% humidity)

Change of live birth rate +14.1% (66.1 – 80.1%)

Relative Contributions: Age, Humidity

7577

7981

8385

8789

9193

9597

99

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

Patient Age (years)

Humidity (%)

Live

bir

th /

clin

ical

pre

gnan

cy

Clinical Relevance vs. Statistical Significance

• Approaches to the solution

– Using the Standard Deviation as a measure of variability

– Comparisons of the change in outcome compared to the Standard Deviation are more realistic estimates of the effect – than –

– The change in outcome compared to the Standard Error of the Mean

Others have thought about this

• Effect Size: (M1 – M2)/Std Dev.

< 0.2 - small effect

0.2 – 0.7 - moderate effect

> 0.7 - large effect

Kazis et al., (1989); Cohen (1977)

Outcome Independent variable

difference Std. Dev. Effect Size

Fetal Cardiac Activity

hCG on day 28

50 mIU/ml 92 mIU/ml 0.54

Gest. Age Xfer Day (3/5) 2.8 days 17.5 days 0.16

Birth Weight Xfer Day (3/5) 29.2 grams 612 grams 0.048

Preg Losses Incubator param

10.4% (CO2) 14.1% (humid) (+/- 2 Std Dev)

~45% 0.23 (CO2)

0.31 (humid.)

Table of Effect Sizes

Using the data

• Is this significantly different? – 2.8 day / 0.14 (SE) yes, indeed 2.8 days is significant

• A 2.8 day difference in the means

• But is 2.8 days clinically relevant? – 2.8 days / 17.5 days (SD) is a very small effect (0.16)

• 70 day range (4 X S.D.)

• Is this significantly different? – 29.2 g / 2.8 g (SE) yes, indeed 29.2 g is significant

• A 29.2 g difference in the means

• But is 29.2 g clinically relevant? – 29.2 g / 612 g (SD) is a very small effect (0.04)

• 2448 g range (4 X S.D.)

Statistics, hmmmph! • Definition of Statistics: The science of producing

unreliable facts from reliable figures. Evan Esar

• There are lies, damned lies and statistics. Mark Twain

• Statistics are no substitute for judgment. Henry Clay

• Facts are stubborn, but statistics are more pliable. Mark Twain

• Statistics: the mathematical theory of ignorance. Morris Kline

• Like dreams, statistics are a form of wish fulfillment. Jean Baudrillard

• If your experiment needs statistics, you ought to have done a better experiment. Ernest Rutherford

Thank you!