Modelli basati su alberi e - loro interpretazione

Modelli basati su alberi e - loro interpretazione -

Anna Gottard 29 maggio 2019

1

Outline

2

Decision trees

Regression and classification trees

Bagging and Random forest

Conditional inference trees and forest

Bart

Interpreting and understanding

Remember: as these algorithms are not set

according to theoretical assumptions, you

need to use

- training/testing sets

- cross-validation

General setting

3

- A vector of explanatory variables X = (X1, … Xp) and a response Y are observed

on a sample of iid statistical units

- We want to predict Y assuming that

f(X) = E[Y | X]<latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit><latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit><latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit><latexit sha1_base64="N9gOGcforqnnaj+7PFn5/jhED50=">AAACrHicfVFNTxsxEHW2LR+h0FCOXFwCElyiXVQJLkgIVKlHQASCsqvI9s6ChT9WtheIVnvgl/Ta/qT+G7zJChFSdUaWn968mfF4aC64dWH4txV8+PhpYXFpub3yeXXtS2f965XVhWHQZ1poM6DEguAK+o47AYPcAJFUwDW9P63j1w9gLNfq0o1zSCS5VTzjjDhPjTob2e5gDx/hH0N8g2PJUzxIRp1u2AsnhudB1IAuauxstN56jlPNCgnKMUGsHUZh7pKSGMeZgKodFxZywu7JLQw9VESCTcrJ6yu845kUZ9r4oxyesG8zSiKtHUvqlZK4O/s+VpP/jFHpaapFOtvfZYdJyVVeOFBs2j4rBHYa1x+EU26AOTH2gDDD/QSY3RFDmPPf2I4VPDItJVFpGXOVVmUsqX4qt+McTB5/m3gNt6tqVqz+p27g1C04yVVh50pcABGvJfztJ6YUX1S10C8ter+ieXC134vCXnT+vXt80qxvCW2iLbSLInSAjtFPdIb6iKEx+oV+oz9BL7gMhkEylQatJmcDzViQvQCcLNIK</latexit>

- The best predictor is the function that minimises among all the possible functions

g(X) a loss function, such as for instance the mean squared error (if Y continuous)

- Let’s call such function [f(X)

<latexit sha1_base64="zNZ//GeNWKi9QLtZL09xwL6oDbA=">AAACqHicfVFLbxMxEJ4srxIeTemRi2mLVECKdrm0xwouPYaKNEF1FNne2caqHyvb2zZa7YHfwYUD/Cj+Dd4kQqRBzMjyp2++mfF4eKmkD2n6q5Pcu//g4aOtx90nT5893+7tvDj3tnICh8Iq68aceVTS4DDIoHBcOmSaKxzxq49tfHSNzktrPod5iRPNLo0spGAhUtPeNr2ROc5YqIvD8ZuGTHv7aT9dGNkE2Qrsn+zRd98BYDDd6XyluRWVRhOEYt5fZGkZJjVzQQqFTZdWHksmrtglXkRomEY/qRcvb8jryOSksC4eE8iC/TujZtr7ueZRqVmY+buxlvxnjOtIc6vy9f6hOJ7U0pRVQCOW7YtKkWBJ+zkklw5FUPMImHAyTkDEjDkmQvzCLjV4I6zWzOQ1lSZvaqq5va0PaImupK8W3sKDplkXm/+pV3DpHoOWpvIbJc6QqT8l4h0n5pycNa0wLi27u6JNcP6+n6X97FPc3gdY2ha8hD04hAyO4AROYQBDEFDBN/gBP5O3ySAZJV+W0qSzytmFNUv4b6BU06A=</latexit><latexit sha1_base64="ZT8RpmwreS7ZYZhnB5ThZbfD7mw=">AAACqHicfVFLbxMxEHaWVwmPpnDkYtoiFZCi3V7oMYILx1CRJqiOIts721j1Y2XPtkSrPfAfuHGFH9V/gzeJEGkQM7L86ZtvZjweUWoVME1vOsmdu/fuP9h52H30+MnT3d7es7PgKi9hJJ12fiJ4AK0sjFChhknpgRuhYSwuP7Tx8RX4oJz9jIsSpoZfWFUoyTFSs94uu1Y5zDnWxdHkdUNnvYO0ny6NboNsDQ4G++zt95vBYjjb63xjuZOVAYtS8xDOs7TEac09Kqmh6bIqQMnlJb+A8wgtNxCm9fLlDX0VmZwWzsdjkS7ZvzNqbkJYGBGVhuM83I615D9jwkRaOJ1v9sfiZForW1YIVq7aF5Wm6Gj7OTRXHiTqRQRcehUnoHLOPZcYv7DLLFxLZwy3ec2UzZuaGeG+1oesBF+yl0tv4WHTbIrt/9RruPIAaJStwlaJU+D6T4l4x4mFoKdNK4xLy26vaBucHfeztJ99itt7T1a2Q16QfXJEMvKODMhHMiQjIklFfpCf5FfyJhkm4+TLSpp01jnPyYYl4je17dUm</latexit><latexit sha1_base64="ZT8RpmwreS7ZYZhnB5ThZbfD7mw=">AAACqHicfVFLbxMxEHaWVwmPpnDkYtoiFZCi3V7oMYILx1CRJqiOIts721j1Y2XPtkSrPfAfuHGFH9V/gzeJEGkQM7L86ZtvZjweUWoVME1vOsmdu/fuP9h52H30+MnT3d7es7PgKi9hJJ12fiJ4AK0sjFChhknpgRuhYSwuP7Tx8RX4oJz9jIsSpoZfWFUoyTFSs94uu1Y5zDnWxdHkdUNnvYO0ny6NboNsDQ4G++zt95vBYjjb63xjuZOVAYtS8xDOs7TEac09Kqmh6bIqQMnlJb+A8wgtNxCm9fLlDX0VmZwWzsdjkS7ZvzNqbkJYGBGVhuM83I615D9jwkRaOJ1v9sfiZForW1YIVq7aF5Wm6Gj7OTRXHiTqRQRcehUnoHLOPZcYv7DLLFxLZwy3ec2UzZuaGeG+1oesBF+yl0tv4WHTbIrt/9RruPIAaJStwlaJU+D6T4l4x4mFoKdNK4xLy26vaBucHfeztJ99itt7T1a2Q16QfXJEMvKODMhHMiQjIklFfpCf5FfyJhkm4+TLSpp01jnPyYYl4je17dUm</latexit><latexit sha1_base64="vEeKIGcWKlkqjcY/p73KIenx9iY=">AAACqHicfVFNbxMxEHWWQkv4aApHLi4pUuEQ7fbSHiu49Bgq0qTqRtHYO9tY9cfK9rZEqz3wO7jSH8W/wZusEGlQZ2T56c2bGY+HFVI4H8e/O9GTrafPtneed1+8fPV6t7f35sKZ0nIccSONnTBwKIXGkRde4qSwCIpJHLObL018fIvWCaO/+UWBUwXXWuSCgw/UrLeb3okM5+Cr/HDysaazXj8exEujmyBpQZ+0NpztdX6kmeGlQu25BOeukrjw0wqsF1xi3U1LhwXwG7jGqwA1KHTTavnymn4ITEZzY8PRni7ZfzMqUM4tFAtKBX7uHsYa8r8xpgLNjMzW+/v8ZFoJXZQeNV+1z0tJvaHN59BMWOReLgIAbkWYgPI5WOA+fGE31XjHjVKgsyoVOqurVDHzvTpIC7RFur/0Bh7U9bpYP6Zu4codeiV06TZKnCPIvyXCHSZmjJ7XjTAsLXm4ok1wcTRI4kHyNe6ffm7Xt0PekffkkCTkmJySMzIkI8JJSX6SX+Q++hQNo3F0uZJGnTbnLVmziP0Bf5bSFw==</latexit>

Mean

⇣Y � g(X)

⌘2| X = x

�

<latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit><latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit><latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit><latexit sha1_base64="dZwMprb1ep0YLx4HRur6d8i0HnQ=">AAAC13icfVFNbxMxEHWWrxK+UjhyMaRI6YFot0Kil6KqXLgglYq0qeIlsr2zG6u2d2V7IdFqhbggrvwHfg1XuPBv8CYrRBrEjCw/v3kzo/GwQgrrwvBXJ7hy9dr1G1s3u7du37l7r7d9/9TmpeEw4rnMzZhRC1JoGDnhJIwLA1QxCWfs4mUTP3sPxopcv3WLAmJFMy1Swanz1LT3gjiYu+o1UF0TCanDE0yORIYH+Bw/xdlgvLt6777bw0SJBI8P5hgTI7KZw/G01w+H4dLwJoha0EetHU+3O59IkvNSgXZcUmsnUVi4uKLGCS6h7pLSQkH5Bc1g4qGmCmxcLQet8RPPJDjNjT/a4SX7d0ZFlbULxbxSUTezl2MN+c8YU55muUzW+7t0P66ELkoHmq/ap6XELsfNX+JEGOBOLjyg3Ag/AeYzaih3/se7RMMHnitFdVIRoZO6Iorl82qHFGAK8mjpDdyp63Wx/p+6hSu34JTQpd0ocQJU/inhbz8xY/ikboR+adHlFW2C071hFA6jN8/6h0ft+rbQQ/QYDVCEnqND9AodoxHi6Bv6jn6gn8F58DH4HHxZSYNOm/MArVnw9TeXmOHx</latexit>

Example : Multiple linear regression

4

MSE =RSS

n<latexit sha1_base64="KhtGCUkucezpZCk8hgu1Rm15wEs=">AAACdXicfVFdaxQxFM2OH63rR7f2UYToruLTNrNabZFCUQQRhOq6bWGzLJnMnTY0H0OSUZcwv6Wv+pP8Jb6a2Y5iRb2XwLnnnptcTrJSCucJ+dZJLl2+cnVl9Vr3+o2bt9Z667cPnKkshwk30tijjDmQQsPECy/hqLTAVCbhMDt92fQPP4J1wugPflHCTLFjLQrBmY/UvLfxdvwK72JaWMbD+/G4Drqe9/pkuEXSnacpJkOyjAhGW2Rnm+C0Zfqojf35eucNzQ2vFGjPJXNumpLSzwKzXnAJdZdWDkrGT9kxTCPUTIGbheX2NX4QmRwXxsajPV6yv08EppxbqCwqFfMn7s9eQ/6tN618sT0LQpeVB83PHyoqib3BjRU4Fxa4l4sIGLci7or5CYtG+GhYl2r4xI1STOeBCp3XgarMfA4DWoIt6b02m2JQ1xfl+v/6X0VMTB14JXTlMH3eXBTd/2kx/jc4GA3Tx8PRuyf9vRftP6yiO+g+eoRS9AztoddoH00QRwt0hr6gr53vyd1kkDw8lyaddmYDXYhk8wcVZL0x</latexit>

Decision trees

5

X1<c

X2<r

Yes

Yes

No

No

—> IDEA: To find a piecewise-constant

approximation of f(X) to predict Y, chosen

in a way that mimics how decisions are actually taken

—> EXAMPLE:

Y = treatment

X1 = Fever

X2 = Pain

Example with only 1 covariate

6

First split

Terminal nodes / leaves

Internal/decision nodes

Induced partitioning

Example : simulated data, 2 covariates

A reg

7

A decision tree is a structure organised hierarchically

The tree structure is equivalent to partition the joint space of the explanatory variables into M

(= n. leaves) subspaces

The number in each leaf is the mean of the response for the observations that fall there.

Y = sin(X1) · sin(X2) + ✏, ✏ ⇠ N(0, 1)<latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit><latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit><latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit><latexit sha1_base64="vmF9ZPX75PLTPRkAIqTloVLBW9Q=">AAAC33icfVFLbxMxEHaWVwmvFI5cDClSIqpot0KiF6QKJMQJlYq0QXUU2d5Ja9Wv2t5CtNoDJ26IK/+BX8MNwY/BmywVaRAzsvz5m29mNB5mpfAhTX+0kkuXr1y9tna9fePmrdt3Out3970pHIchN9K4EaMepNAwDCJIGFkHVDEJB+zkRR0/OAPnhdFvw8zCWNEjLaaC0xCpSeflO/wMEy90bzTJ+pjw3IQ/760+fowJWC+k0ZuYnJ4WNMfnTC1T+HUv3cz6k043HaRzw6sga0AXNbY7WW99JLnhhQIduKTeH2apDeOSuiC4hKpNCg+W8hN6BIcRaqrAj8v5wBV+FJkcT42LRwc8Z//OKKnyfqZYVCoajv3FWE3+M8ZUpJmR+XL/MN0el0LbIoDmi/bTQuJgcP2nOBcOeJCzCCh3Ik6A+TF1lIf4822i4T03SlGdl0TovCqJYuZDuUEsOEsezL2GG1W1LNb/Uzdw4R6CErrwKyX2gMrzEvGOEzOG96paGJeWXVzRKtjfGmTpIHvzpLvzvFnfGrqPHqIeytBTtINeoV00RBx9Q9/RT/Qrocmn5HPyZSFNWk3OPbRkydffFtfk3w==</latexit>

X1, X2 ⇠ N(0, 1)<latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit><latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit><latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit><latexit sha1_base64="vuZjzxA94BrYvpAa8+VvAliLuLg=">AAACrXicfVFNTxsxEHWW0tL0K9DeuBhCJZBQuosqlSMql54QRQQi4bCyvZNg4Y+V7S2E1R76T3ql/6j/pt5kVTUEMSPLT2/ezHg8LJfC+Tj+04qWni0/f7Hysv3q9Zu37zqra2fOFJZDnxtp7IBRB1Jo6HvhJQxyC1QxCefs+rCOn/8A64TRp36Sw1DRsRYjwakPVNr5MEiTXTxI9zBxQuGj7Xg32Uk73bgXTw0vgqQBXdTYcbra+kkywwsF2nNJnbtI4twPS2q94BKqNikc5JRf0zFcBKipAjcsp8+v8MfAZHhkbDja4yn7f0ZJlXMTxYJSUX/lHsZq8tEYU4FmRmbz/f1of1gKnRceNJ+1HxUSe4PrH8KZsMC9nARAuRVhAsyvqKXch39sEw033ChFdVYSobOqJIqZ23KL5GBzsjH1Gm5V1bxYP6Vu4MwdeCV04RZKnACV/0qEO0zMGD6pamFYWvJwRYvgbK+XxL3k++fuwddmfStoHW2ibZSgL+gAfUPHqI84ukO/0D36HX2K+hGJLmfSqNXkvEdzFo3/Asbg0hI=</latexit>

X1<c

X2<r

Yes

Yes

No

No

Regression trees

8

—> We need to learn:

- structure of the tree

- which variable splits and where

- predictions in each leaf/region

Recursive binary splitting: a top-down, greedy approach

The approach is top-down because it begins at the top of the tree (at which

point all observations belong to a single region) and then successively splits the

covariate space

Binary: Each split is indicated via two new branches further down on the tree

It is greedy because at each step of the tree-building process, the best split is

made at that particular step, rather than looking ahead and picking a split that

will lead to a better tree in some future step.

9

The tree-building process examples

10

Regression trees from a regression perspective

11

(1) Start with M = 1, R1 = Rp<latexit sha1_base64="2MPgHA2f9xx4CwF6K56FayMie/Q=">AAACgHicfVFdTxQxFO2OqLh+LfBoYqqLCQ9mnSIgakhIeCEkJEhcINmum07nLjT0i7aDbibz5q/xFf6M/8bOMhIw6r1pcu655/bjNLNS+JCmP1vJnZm79+7PPmg/fPT4ydPO3PyBN4Xj0OdGGneUMQ9SaOgHESQcWQdMZRIOs9Otun94Ds4Loz+HiYWhYsdajAVnIVKjzvPdDfIa07OzguV4f0TwBlUsnGRZuV99sXjU6aa9dBr4BlhNyfs1gknDdFETe6O51g7NDS8U6MAl835AUhuGJXNBcAlVmxYeLOOn7BgGEWqmwA/L6UMq/CoyOR4bF5cOeMrenCiZ8n6isqisb+n/7NXk33qDIozXh6XQtgig+dVB40LiYHDtCs6FAx7kJALGnYh3xfyEOcZD9K5NNXzlRimm85IKnVclVZn5Vi5SC87SF03WxWJV3Zbr/+uvi5iYeghK6MJj+rHeKLr/22L8b3Cw3CNve8ufVrqb680/zKJn6CVaQgS9Q5toG+2hPuLoO/qBLtBlkiRLyZuEXEmTVjOzgG5F8uEXAzS/rg==</latexit>

(2) Search for the first split:

(j, s1) : R1 = {(X1, . . . , Xp) 2 X : Xj s1}, R2 = {(X1, . . . , Xp) 2 X : Xj > s1}<latexit sha1_base64="C+13GHF5DEj49Ec1VS3s/3CjM6o=">AAAC9XicjVFNb9QwEHXCV1k+uoUjBwa2SFtptUqWFkoRqBIXxKkgto20XkWO423d2k5qO5SVlV/BmRviyu/hp3DD2aaoRYCYka2ZN2+erZmsFNzYKPoehJcuX7l6bel658bNW7eXuyt3dk1RacrGtBCFTjJimOCKjS23giWlZkRmgu1lR6+a+t4Hpg0v1Hs7L9lUkn3FZ5wS66G0+6l/OACTxmuAB7DVXO/SGF4Adv0kjQeARV5YAwNI0tJzuAIsiT2gRLik3vKoO6w9iR03Irj2RHxckdyrjP5XBc5kXrYaabcXDaOFwblgI4qfPYkhbpEeam0nXQne4LyglWTKUkGMmcRRaaeOaMupYHUHV4aVhB6RfTbxoSKSmalbTK+GRx7JYVZof5SFBXq+wxFpzFxmntl82vxea8A/1SaVnW1OHVdlZZmipw/NKgG2gGYVkHPNqBVzHxCquf8r0AOiCbV+YR2s2AktpCQqd35iee2wzIqPbhWXTJf4QetNslrXF+nq3/xfiXfAhlnJVWUAP2+E/PTPRgx/D3ZHw/jxcPR2vbe92e5hCd1DD1Efxegp2kav0Q4aI4p+BPeDfrAWnoSfwy/h11NqGLQ9d9EFC7/9BBXm5Jc=</latexit>

minj,s1

2

4minµ1

X

i:xij2R1

(yi � µ1)2 + min

µ2

X

i:xij2R2

(yi � µ2)2

3

5

<latexit sha1_base64="lUOUG9ySK6cckuindVfiSRiqeS8=">AAADEHicfVHLbtQwFHXCqwyvKbBjc2GKVASMkvAqZVOpLBCrgpi20niIHMeZcWs7ke1AR1F+gg9hzQ6x5Q/4G5xMoJ2Kci1b5557fG3dkxSCGxsEvzz/3PkLFy+tXO5duXrt+o3+6s1dk5eashHNRa73E2KY4IqNLLeC7ReaEZkItpccbjf1vU9MG56rD3ZesIkkU8UzTol1VNz/iiVXcXXwCEwc1oAFyyyMYcFiWbakKWVc8c0jdxy4lCt4fyxeh3nM4XGrBaz5dGbhwccI8Ct42BzHraIzWkWnWkErXuq1gBOI+4NgGLQBJ8CzIHz5PISwYwaoi5141XuL05yWkilLBTFmHAaFnVREW04Fq3u4NKwg9JBM2dhBRSQzk6odbA33HZNClmu3lYWWPXmjItKYuUycUhI7M6drDfmv2ri02cak4qooLVN08VBWCrA5NC5ByjWjVswdIFRz91egM6IJtc7LHlbsM82lJCqt3AjT2o03yY+qNVwwXeC73WqStbpelqv/6/8mbjm3mHXulcb52DRy0/8zYjgb7EbD8Mkwevd0sLXR+bCC7qB7aB2F6AXaQm/QDhoh6t32Nr1t77X/xf/mf/d/LKS+1925hZbC//kbHALx4Q==</latexit>

This corresponds to finding j and s1 that minimise the MSE of the one-factor regression model

Yi = µ1 I{Xijsj} + µ2 I{Xij>sj}"i<latexit sha1_base64="3jqSnVhYBaqamChltnq39FcWWtA=">AAAC1XicfVFNbxMxEPUuX2n4aApHLgMpEhIo2g0tFCFQJS7AqUikCYqjldfrtG5t72J7C5HlG+LK7+uvAW+yVC0qzMjS85v3/DGTV4IbmySnUXzl6rXrNzpr3Zu3bt9Z723c3TdlrSkb0VKUepITwwRXbGS5FWxSaUZkLtg4P37b1McnTBteqk92UbGZJAeKzzklNlBZz3zOOLwGACzrzKUe8NMGE3uY5+69zxx2MMkcPwoVwb6AyY4Aew/wZOUYLh2XG96cqfEJ0awyXIQrOWS9fjJIlgHnwHaSvnyeQtoyfdTGXrYRfcBFSWvJlKWCGDNNk8rOHNGWU8F8F9eGVYQekwM2DVARyczMLbvj4VFgCpiXOixlYcmedzgijVnIPCibb5i/aw15WW1a2/nOzHFV1ZYpurpoXguwJTSthoJrRq1YBECo5uGtQA+JJtSGgXSxYl9pKSVRhcNcFd5hmZff3CaumK7wgzabzab3F+Xq//qzTUjAhlnJVW0Av2oOCt3/02L4N9gfDtJng+HHrf7uTjuHDrqPHqLHKEUv0C56h/bQCFF0in5FnWgtHsc+/h7/WEnjqPXcQxci/vkb27re6w==</latexit>

) bµm = yRm m = 1, 2<latexit sha1_base64="hn++SOZYSWZuxqw8MnslktjmHxc=">AAACnnicfVFdaxNBFJ1s/ajxo2l97MtoKvggYTdam2ILBUFEEGsxbSATlruzN8nQnZntzKwxLPtn/DW+6pv/xtl0FSvqvQyce+6583EmyTNhXRh+bwVr167fuLl+q337zt17G53NrVOrC8NxyHWmzSgBi5lQOHTCZTjKDYJMMjxLzl/W/bOPaKzQ6oNb5jiRMFNiKjg4T8WdA3YiZnMHxugFZRcFpJQtRIpzcCWTRRVLekhZAoYu4/IkllUjkofRk37c6Ya93TDafx7RsBeuwoP+brg/CGnUMF3SxHG82XrDUs0LicrxDKwdR2HuJiUYJ3iGVZsVFnPg5zDDsYcKJNpJuXpmRR95JqVTbfxSjq7Y3ydKkNYuZeKVEtzc/tmryb/1xoWbDialUHnhUPHLg6ZFRp2mtWc0FQa5y5YeADfC35XyORjgzjvbZgoXXEsJKi2ZUGnljUv0p3KH5Why9qDJutipqqty9X/9r8InZRadFKqwlL2oN/Lu/7SY/huc9nvR017//bPu0aD5h3WyTR6SxyQie+SIvCbHZEg4+Uy+kK/kW0CDV8Hb4N2lNGg1M/fJlQhGPwAwC8y+</latexit>

Estimate/Prediction

j=1, …, p

+

Regression trees from a regression perspective

12

(3) Search for the second split: repeat the procedure (2) within R1 or R2

This corresponds to finding k and s2 that minimise the MSE in one of the one-factor

regression models

Estimate/Prediction

R1 , R2 —> R1 , R2 , R3 minimising the loss function (MSE)

Yi = µ1 I{Xijs1} + µ2 I{Xij>s1}I{Xiks2} + µ3 I{Xij>s1}I{Xik>s2} + "i<latexit sha1_base64="aWhe8QCQb//B3Q++djGFB60LeAY=">AAADQ3icnVFdb9MwFHXC1yhfHTzycqFDQgJVScfGEGKaxAMfT0OiW1FdRY7jbqa2E2yno7Ly8/gRPPLMG+IVCafNpg4VHrhWpONzzj2OfdNCcGOj6GsQXrh46fKVtauta9dv3LzVXr99YPJSU9anucj1ICWGCa5Y33Ir2KDQjMhUsMN08rLWD6dMG56r93ZWsJEkR4qPOSXWU0n724eEwwsALMvExRXgxzUm9jhN3ZsqcdjBIHH8o1cE+wQmiQFXFcCjRUdv3rG6YffUvUKenOX15nlN3Ob/x+0uZ02JZoXhwt+QJ+1O1I3mBUtgK4qfbccQN0wHNbWfrAdvcZbTUjJlqSDGDOOosCNHtOVUsKqFS8MKQifkiA09VEQyM3LzWVTwwDMZjHPtP2Vhzi53OCKNmcnUO+t7mD+1mlylDUs73hk5rorSMkUXB41LATaHerCQcc2oFTMPCNXc/yvQY6IJtX78LazYCc2lJCpzmKusclim+We3gQumC3yvWfVmo6rO29W//WcbvwAbZiVXpQH8vA7yr3/6xPB3cNDrxpvd3rsnnb2dZg5r6C66jx6iGD1Fe+g12kd9RINXgQymwUn4Jfwe/gh/Lqxh0PTcQecq/PUb/yEIPw==</latexit>

Yi = µ1 I{Xijs1}I{Xiks2} + µ2 I{Xijs1}I{Xik>s2} + µ3 I{Xij>s1} + "i,<latexit sha1_base64="ZwSsHDzZp1hhnluWoRq1433gYPs=">AAADSHicjVLLbhMxFPVMeZTwaApLNhdSJCSqaCZtoQhRVWIDiEWRSBsURyOPx2lNbM9gewqRmR/kD/gB1uwQOzzJNGpRQFzL0vG55x4/rtNCcGOj6FsQrly6fOXq6rXW9Rs3b621128fmrzUlPVpLnI9SIlhgivWt9wKNig0IzIV7CidvKjzR6dMG56rd3ZasJEkx4qPOSXWU0n7+/uEw3MALMvExRXgzRoTe5Km7lWVOOxgkDj+wWcE+wgmiQFX1VLNZKHp1ZpHc8/e/3oud9w7s4PGb2vmt9xub3E+Lz4lmhWGC39Lvpm0O1E3mgWcAztR/PRxDHHDdFATB8l68BpnOS0lU5YKYswwjgo7ckRbTgWrWrg0rCB0Qo7Z0ENFJDMjN2tIBQ88k8E4134qCzP2fIUj0pipTL2yvoj5M1eTy3LD0o53R46rorRM0flG41KAzaHuLmRcM2rF1ANCNfdnBXpCNKHW/4EWVuwTzaUkKnOYq6xyWKb5Z7eBC6YLfK8Z9WKjqi7K1b/1i4UfgA2zkqvSAH5WG/nXP3ti+Ds47HXjrW7v7XZnf7fpwyq6i+6jhyhGT9A+eokOUB/R4E2gAxd8Cb+GP8Kf4a+5NAyamjvoQqyEvwH1nQke</latexit>

) bµm = yRm m = 1, . . . , 3<latexit sha1_base64="Icuy3eKqWhfY/RiR2q0UYDs/ZO4=">AAACpXicfVHbbhMxEHWWWwmXpvDIiyFF8FBFuymFVKhSpb4gJKRSNWmlOFrNep3Eqi+LPUuIVvtBfA2v9G9w0gVRBMzI0pkzZ3w5zgolPcbxZSu6cfPW7Tsbd9v37j94uNnZejTytnRcDLlV1p1n4IWSRgxRohLnhROgMyXOsoujVf/ss3BeWnOKy0JMNMyMnEoOGKi0c8RO5GyO4JxdUPaphJyyhczFHLBiuqxTTQ8oy8DRZVqdpLpuRPog2WEqt+h3dtNON+7txcn+64TGvXgdAfT34v1BTJOG6ZImjtOt1nuWW15qYZAr8H6cxAVOKnAouRJ1m5VeFMAvYCbGARrQwk+q9Wtr+jwwOZ1aF5ZBumZ/n6hAe7/UWVBqwLn/s7ci/9YblzgdTCppihKF4VcHTUtF0dKVdTSXTnBUywCAOxnuSvkcHHAMBreZEQtutQaTV0yavA7+ZfZLtc0K4Qr2tMlVsV3X1+Xm//pfRUjKvEAtTekpe7vaKLj/02L6bzDq95LdXv/jq+7hoPmHDfKEPCMvSULekEPyjhyTIeHkK/lGvpPL6EX0ITqNRlfSqNXMPCbXIkp/ADGvz7M=</latexit>

The tree model : assigning a value to each terminal node

13

conditional means

E[Y | X = x] =MX

m=1

µmI{x2Rm}<latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit>

R1 : x1 < 3 ^ x2 < 1.5 ) µ1 = 60<latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit><latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit><latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit><latexit sha1_base64="KnSd0jQICzsmRk5PD+V+o7aoKY0=">AAACnnicfVHbbhMxEHWWS9twS+GxLwMpEg9otdvSGwSpAgkhJESpSBspjlZe7ySxurZXtpc0Wu3P8DW8wht/gzcNiHKbkaVzZs74cpwWubAuir61gitXr11fWV1r37h56/adzvrdE6tLw7HPda7NIGUWc6Gw74TLcVAYZDLN8TQ9e9n0Tz+isUKrD25e4EiyiRJjwZnzpaTTO05ieAr0GZwncQ+2gc4wm6BnWz2Iwx2gj4Eei8nUMWP0bEFl6Weew26UdLpRGB3sR9sH8CeIw2gRXbKMo2S99YZmmpcSleM5s3YYR4UbVcw4wXOs27S0WDB+xiY49FAxiXZULZ5Zw0NfyWCsjV/KwaL660TFpLVzmXqlZG5qf+81xb/1hqUb748qoYrSoeIXB43LHJyGxjPIhEHu8rkHjBvh7wp8ygzjzjvbpgpnXEvJVFZRobK6ojLV59UmLdAU9P4yG7JZ15fl6v/6n8QnUItOClVa/1vNRt79HxbDv8HJVhhHYfz+SffwxfIfVskGeUAekZjskUPymhyRPuHkE/lMvpCvAQSvgrfBuwtp0FrO3COXIhh8B6ajyFI=</latexit>

R2 : x1 < 3 ^ x2 � 1.5 ) µ2 = 100<latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit><latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit><latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit><latexit sha1_base64="zJUHIZd6JAKg/Lb2eZxwJ5zcyyk=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YBWuymoLRepwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/gTQOi3GZkac6ZM74cp2UhrIuib53g0uUrV6+tXe/euHnr9p3e+t1DqyvDccR1oc1xyiwWQuHICVfgcWmQybTAo/TkTds/+ojGCq0O3KLERLJciangzHlq0ns1nAzgGdDncDaJX8AW0DlmOXo0AJrjKcThU6CPgQ5FPnPMGD1fQll5wUuIo2jS60dhtLsTbe3Cn0UcRsvok1XsT9Y772imeSVROV4wa8dxVLqkZsYJXmDTpZXFkvETluPYl4pJtEm9fGsDm57JYKqNX8rBkv11ombS2oVMvVIyN7O/91ryb71x5aY7SS1UWTlU/PygaVWA09AaB5kwyF2x8AXjRvi7Ap8xw7jz9napwjnXUjKV1VSorKmpTPVZvUFLNCV9sMoWbDTNRbn6v/4n8AnUopNCVdZ/WbuRd/+HxfDv4nAQxlEYf3jS33u9+oc1cp88JI9ITLbJHnlL9smIcPKJfCZfyNdgM3gfDIODc2nQWc3cIxciSL4DF9bKLg==</latexit>

R4 : x1 � 3 ^ x1 � 4 ) µ4 = 30<latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit><latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit><latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit><latexit sha1_base64="eEqsyyfxJKZHKOvUf2Pin8/BR04=">AAACo3icfVHbbhMxEHWWWwm3FB55GUgr8YCiXRKJtgipwAuClxI1baV4FXm9k43Vtb21vaTRar+Hr+EVxN/g3QZEuc3I0pkzZ3w5TopcWBeG3zrBlavXrt/YuNm9dfvO3Xu9zftHVpeG44TrXJuThFnMhcKJEy7Hk8Igk0mOx8npm6Z//BGNFVodulWBsWSZEnPBmfPUrPdqPBvBHtAXcD6LaIZnMAS6xDTDhoCWGQF9CnQssoVjxuhlW8rSD76EYTjr9cNBuLsTDnfhTxANwjb6ZB0Hs83OO5pqXkpUjufM2mkUFi6umHGC51h3aWmxYPyUZTj1UDGJNq7at9aw7ZkU5tr4pRy07K8TFZPWrmTilZK5hf2915B/601LN9+JK6GK0qHiFwfNyxychsY4SIVB7vKVB4wb4e8KfMEM487b26UKl1xLyVRaUaHSuqIy0efVFi3QFPTROptiq64vy9X/9T8Ln0AtOilUaf2XNRt5939YDP8GR88GUTiIPoz6+6/X/7BBHpLH5AmJyHOyT96SAzIhnHwin8kX8jXYDt4H4+DwQhp01jMPyKUI4u/X1ssA</latexit>

45 30R3 : x1 � 3 ^ x1 < 4 ) µ3 = 45

<latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit><latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit><latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit><latexit sha1_base64="W4+yVGs8uGqm30UKpoJrTZcJsxs=">AAACoHicfVHLbhMxFHWmQEt4pWXJ5tIUiQWKZkgqWl6qYFOxoa1IWxFHI4/nZmJ1bA+2hzQazd/wNWxhw9/gSQOivO6VpXPPPdeP46TIhXVh+K0VrFy5em117Xr7xs1bt+901jeOrS4NxyHXuTanCbOYC4VDJ1yOp4VBJpMcT5Kz103/5CMaK7R65+YFjiXLlJgIzpyn4s7Lo7gPT4E+g/M4ohl+gD7QGaYZNgQ8hwHQR0CPRDZ1zBg9W5Sy9FMvYLAdd7phL9zdCfu78CeIeuEiumQZB/F66w1NNS8lKsdzZu0oCgs3rphxgudYt2lpsWD8jGU48lAxiXZcLR5awwPPpDDRxi/lYMH+OlExae1cJl4pmZva33sN+bfeqHSTnXElVFE6VPzioEmZg9PQuAapMMhdPveAcSP8XYFPmWHceW/bVOGMaymZSisqVFpXVCb6vNqiBZqC3l9mU2zV9WW5+r/+Z+ETqEUnhSqt/69mI+/+D4vh3+D4cS8Ke9HhoLv3avkPa+Qe2SQPSUSekD2yTw7IkHDyiXwmX8jXYDPYD94GhxfSoLWcuUsuRfD+O3B1yYk=</latexit>

The tree model

14

E[Y | X = x] =MX

m=1

µmI{x2Rm}<latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit><latexit sha1_base64="Y2jmsOr295e4AWR92gXAVO8I2Uo=">AAACqXicfVFtixMxEE7Xt7O+9fSjX0Z7giAs2dLqFSkcivgCwp3Yu2rTW7LZ9C7cJrskWW0J+Un+Gj8J+l/M9nriiTpD4JmZZ5LJM1lVCGMx/taKLly8dPnKxtX2tes3bt7qbN7eN2WtGR+zsij1JKOGF0LxsRW24JNKcyqzgh9kJ8+b+sEnro0o1Xu7rPhM0iMl5oJRG1Jp5yWR1B5nmXvhpx+ASJHDZLSAGYyAmFqmTo4Sf/g2VOpUwhn5tU8dcQsgQsG7VBLv004Xx7i3Pej3AMe9AR4mwwAGOBk+7kMS45V10dp2083WG5KXrJZcWVZQY6YJruzMUW0FK7hvk9rwirITesSnASoquZm51Y89PAiZHOalDkdZWGV/73BUGrOUWWA2I5s/a03yb7VpbefbMydUVVuu2OlD87oAW0IjH+RCc2aLZQCUaRFmBXZMNWU2iNwmin9mpZRU5S5ok3tHZFYu3BapuK7IvbU3wZb35+nq//xfQfCwGm6lULUB8rS5KKh/JjH8G+z34gTHyV6/u/NsvYcNdBfdRw9Rgp6gHfQK7aIxYugL+oq+ox/Ro2gvmkQfT6lRa91zB52ziP0Ehs7RKw==</latexit>

The tree-building algorithm: stopping rule

STOPPING RULE

On the training data:

To avoid to have leaves with only one unit - perfect (over)fitting

- stop if the node contains less than a pre-specified minimum node size (usually 5, but depends on n)

- stop if the pre-specified maximum tree depth limit is reached

15

Stopping rule : Example with only 1 covariate

16

Stopping rule : Example with 2 covariate2

17

Pruning a tree

This process may produce good predictions on the training set, but is

likely to overfit the data, leading to poor test set performance.

A smaller tree with fewer splits (fewer regions R1 … RJ) might lead to

lower variance and better interpretation at the cost of a little bias.

Good strategy: grow a very large tree then prune it back

18

Pruning a tree

19

Pruning a tree : cost-complexity measure

20

Cost-complexity pruning :

Construct a sequence of sub-trees, pruned at different depth d, having

numbers of nodes varying from 1 to |Td|

Compute the cost complexity measure for the tree, which is based on

CP (d) =

|Td|X

m=1

X

i:xi2Rm

(yi � µm)2 + ↵|Td|<latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit><latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit><latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit><latexit sha1_base64="X34oKPUyfzHHZcEkmlVzLHMwL5M=">AAACtHicfVHbbhMxEHWWWwmXpu0jLwMpUipEtFuqXkCVKvqCeApV01aK05XXO2ms2t6V7aWNtvtf/AovvMJn4E0XRLmNZenMmTO+nElyKawLw8+t4NbtO3fvLdxvP3j46PFiZ2n5yGaF4TjkmczMScIsSqFx6ISTeJIbZCqReJyc79f1449orMj0oZvlOFbsTIuJ4Mx5Ku4c7A966RrsArWFiku1G1Wn5dVhnF5VQN80rHgNl7EAKjQcxKqC3sxnL4GqIlZrp+vwAiiT+ZTBvDHudMN+uLMdvtqBP0HUD+fRJU0M4qXWe5pmvFCoHZfM2lEU5m5cMuMEl1i1aWExZ/ycneHIQ80U2nE5/3wFzz2TwiQzfmsHc/bXjpIpa2cq8UrF3NT+XqvJv9VGhZtsj0uh88Kh5tcXTQoJLoPaSUiFQe7kzAPGjfBvBT5lhnHn/W5TjRc8U4rptPTGpVVJVZJdlqs0R5PTp82qk9WquinX/9f/TPzyE0KnhC6sH1d9kHf/h8Xwb3C03o/CfvRho7v3tpnDAnlCnpEeicgW2SPvyIAMCSefyBfylXwLNgMa8ACvpUGr6VkhNyLQ3wF0t9NS</latexit>

where α is a non-negative regularization parameter controlling the trade-off

between the tree complexity and its fitting

Pruning a tree : Choosing the best subtree

21

Choose 𝜶:

- 𝜶 controls the trade-off between complexity and fitting

- optimal value chosen using cross-validation

- Then fit the tree on full data using the chosen optimal value

Classification trees

X1<c

X2<r

Yes

Yes

No

No

Classification

Very similar to a regression tree,

For qualitative responses rather than a

quantitative one

Response classes: 1, …, K

23

Prediction at each node: the most

common class in the corresponding

Need to change the loss function

IDEA: the more homogeneous the units in

the leaves the better

24

Splitting Rule : purity/ impurity measures

pmk m = 1, . . .M k = 1, . . .K<latexit sha1_base64="Mxci4lmwtiFpdOLDwyyw88dpO/k=">AAACznicfVHbbhMxEHWWWwm3FB55MaRIPKBoNxBoH5AqeEFCSKEibaU6imzvpLHiW21vIVqtEG/8A1/DK3wBf4M32aKG24wsH505M6OZYVYKH9L0Ryu5cPHS5SsbV9vXrt+4eauzeXvfm8JxGHEjjTtk1IMUGkZBBAmH1gFVTMIBm7+s4wen4Lww+l1YWBgreqzFVHAaIjXpDMiMBmwnpZpXmJycFDTH6nn2CBOZm+DxmzNyfo58Pel0094gzXaeZjjtpUuLoD9Id7ZTnDVMFzU2nGy2PpHc8EKBDlxS74+y1IZxSV0QXELVJoUHS/mcHsNRhJoq8ONyOV+FH0Qmx1Pj4tMBL9nzGSVV3i8Ui0pFw8z/HqvJv8aYijQzMl/vH6bb41JoWwTQfNV+WkgcDK5XiHPhgAe5iIByJ+IEmM+oozzERbeJhvfcKEV1XhKh86okipkP5Rax4Cy5t/QablXVulj/T93AlXsISujC/1FiD6j8VSL+cWLG8F5VC+PRzi6D/w32+73sca//9kl390Vzvg10F91HD1GGnqFd9AoN0Qhx9AV9Rd/Q92SYnCZV8nElTVpNzh20Zsnnn9qt4FA=</latexit>

-> Proportion of units in node m having Y = k

-> Gini index for node mVariance of a Bernoulli distribution

Total Variance

-> Cross entropy for node m

Gm =KX

k=1

pmk(1� pmk)<latexit sha1_base64="VXdJ14C2DvmF+wdsqMqFufYH4Q8=">AAACzHicfVHbbhMxEHWWWwm3FB55MaRI5YFotxcoEpUq8QASEioVaSt1w8r2ThorvqxsbyGy/MAL/8DX8AqfwN/gTRbUcJuR5aMzZ2Y0M7QS3Lo0/d5JLly8dPnKytXutes3bt7qrd4+tLo2DIZMC22OKbEguIKh407AcWWASCrgiE6fN/GjMzCWa/XWzSoYSXKq+Jgz4iJV9DZfFBLv4tzWsvDT3Sy8868CzifE4arwchrwOs4eLREPi14/HaRzw+fAdpo9fZzhrGX6qLX9YrXzMS81qyUoxwSx9iRLKzfyxDjOBIRuXluoCJuSUziJUBEJduTn0wX8IDIlHmsTn3J4zp7P8ERaO5M0KiVxE/t7rCH/GqMy0lSLcrm/G++MPFdV7UCxRftxLbDTuFkgLrkB5sQsAsIMjxNgNiGGMBfX3M0VvGdaSqJKn3NVBp9Lqj/4tbwCU+X35t7AtRCWxep/6hYu3IKTXNX2jxIHQMSvEvGPE1OKD0IjjEf7eRn8b3C4Mcg2Bxtvtvp7z9rzraC76D5aRxl6gvbQS7SPhoihz+gL+oq+Ja8Tl/gkLKRJp825g5Ys+fQDOyTfPw==</latexit>

Dm = �KX

k=1

pmk log pmk

<latexit sha1_base64="iDIg4964Ahn9Kk+LukO1kZ9qMSs=">AAACzHicfVHbbhMxEHWWWwm3FB55MaRIvBDtttCCRKVK8ICEhEpF2krdsLK9k8SKLyvbC0SWH3jhH/gaXuET+Bu8yYISbjOyfHTmzIxmhlaCW5em3zvJufMXLl7auNy9cvXa9Ru9zZvHVteGwZBpoc0pJRYEVzB03Ak4rQwQSQWc0NmzJn7yDozlWr1x8wpGkkwUH3NGXKSK3s7zQuJ9/ADntpaFn+1n4a1/GXA+JQ5XhZeziIWerBJFr58O0oXhFfAozZ7sZjhrmT5q7bDY7HzMS81qCcoxQaw9y9LKjTwxjjMBoZvXFirCZmQCZxEqIsGO/GK6gO9FpsRjbeJTDi/Y1QxPpLVzSaNSEje1v8ca8q8xKiNNtSjX+7vx45HnqqodKLZsP64Fdho3C8QlN8CcmEdAmOFxAsymxBDm4pq7uYL3TEtJVOlzrsrgc0n1B7+VV2Cq/M7CG7gVwrpY/U/dwqVbcJKr2v5R4giI+FUi/nFiSvFRaITxaD8vg/8NjrcH2c5g+/XD/sHT9nwb6Da6i+6jDO2hA/QCHaIhYugz+oK+om/Jq8QlPglLadJpc26hNUs+/QBa3uA4</latexit>

-> Misclassification error for node m Em = 1�maxk

pmk<latexit sha1_base64="fapc2jHpwRwqoRiroIUJeBqFGPk=">AAACuHicfVFLbxMxEHaWVwmvtBw5YEiRuBDtFvoAgVQJIXEsFWkr1dFieyeJFT9Wtpc2svbAgV/DFX4M/wZvsqCG14wsf/rmmxnNDCulcD5Nv3eSS5evXL22dr174+at23d66xtHzlSWw5AbaewJow6k0DD0wks4KS1QxSQcs9nrJn78EawTRr/38xJGik60GAtOfaTy3v03ucKvcIafYKLoeR5mNSZT6nGZBxVx3uung3Rh+ALYTrPnOxnOWqaPWjvI1zufSGF4pUB7Lqlzp1la+lGg1gsuoe6SykFJ+YxO4DRCTRW4UVhMUuNHkSnw2Nj4tMcL9mJGoMq5uWJRqaifut9jDfnXGFORZkYWq/39eG8UhC4rD5ov248rib3BzbJwISxwL+cRUG5FnADzKbWU+7jSLtFwxo1SVBeBCF3UgShmzsMmKcGW5MHCG7hZ16ti/T91C5fuwCuhK/dHiUOg8leJ+MeJGcOHdSOMR/t5GfxvcLQ1yJ4Ott496++/bM+3hu6hh+gxytAu2kdv0QEaIo4+oy/oK/qWvEg+JJNELKVJp825i1YssT8A48fXjg==</latexit>

Splitting Rules : impurity measures

25

Splitting and stopping rules

Gini index and cross-entropy are quite similar numerically

Cross-entropy and the Gini index are more sensitive to changes in the node probabilities than the misclassification rate

Gini index or the cross-entropy are typically used to evaluate the quality of a particular split

Any of these three approaches might be used when pruning the tree but Classification error rate is preferable for comparing the prediction accuracy of the final pruned tree

26

Ensamble methods

VS

Many beats one

Trees are someway easier to explain to people than other predictive procedures : the tree plot makes them easy to understand

Trees can naturally deal with interactions and non-linearities, continuous and continuous predictors and responses

But they cannot boast a great predictive performance

Can we use more trees?

28

Bagging (Bootstrap aggregation)

Bagging

BAGGING = Bootstrap AGGregation

To have more trees we need to introduce some variability among the trees: grow each tree on a different bootstrap sample

Averaging many trees reduces the variability of the prediction

Bagging grows B trees, taking advantage of resampling techniques

30

Many beat one

Trees are someway easier to explain to people than other predictive procedures : The tree plot makes them easy to interprete

They can naturally deal with interactions and non-linearities, continuous and continuous predictors and responses

But they cannot boast a great predictive performance

Can we have more trees?

31

Choice of B

32

Out-of-Bag prediction

On average, each bagged tree uses of around 2/3 of the observations

The remaining 1/3 of the units, not used to fit a bagged tree, are called the out-of-

bag (OOB) sample

We can predict the Yi using each of the trees in which unit i was OOB

This yields around B/3 predictions for the i-th unit

To obtain a single prediction for the ith observation we average those predicted

responses (quantitative), or take a majority vote (qualitative)

33

OOB error estimate

Since an OOB prediction can be computed for all n units, we can

compute an overall OOB MSE or classification error rate

It can be shown that with B sufficiently large, OOB error is virtually

equivalent to leave-one-out cross-validation error

Price to pay for bagging : interpretation

34

Random forests

From bagging to Random Forest

The bagged trees based on the bootstrapped samples often look quite similar to

each other. They are therefore often highly correlated

Averaging uncorrelated trees can lead to a larger reduction in variance

To de-correlate the trees, random forests build a number of trees on

bootstrapped data using a random sample of mtry predictors

36

Random Forest

Two parameters to be tuned:

37

(1) B = number of trees

Random Forest

Two parameters to be tuned:

38

(2) mtry = n. variables sampled at each split

BAGGING

NB: It depends on the unknown number of good predictors

Conditional inference trees and forests

Conditional inference trees

In conditional inference trees (CTREE), we perform a Fisher permutation test for

independence between the response and each predictor

A split is possible only if the p-value (adjusted for multiple comparisons) is

smaller than a pre-specified nominal level

No need to prune the tree!

40

(1) Perform all the independence tests

(2) Choose the variable with lowest p-value and split maximising the contrast

(3) Stop when no adjusted p-values are below the threshold

BART = Bayesian Additive Regression Trees

Bayesian “almost" nonparametric

Sum of tree model, no bagging, no variable sampling, no pruning

The trees are grown via MCMC and regularised by ad hoc priors

Each tree is evaluated in its entirety via the leaves parameters

Very good performance

42

E[Y | X = x] =⌧X

t=1

T (X;Rt,�t) =⌧X

t=1

MtX

m=1

µmtI{x2Rmt}<latexit sha1_base64="EtHyiQQzNe3VUJpcE4DNNilbO8s=">AAADPXicfVHdbtMwGHXC3yh/HVxy80GHNCRUJeN3QKVJCAkukEa1bkV1FzmO21mznSh2oJXlC96F9+A5eADuELfc4qYBrTD4rEjH5zvHdr6TFoJrE0VfgvDM2XPnL6xdbF26fOXqtfb69X2dVyVlA5qLvBymRDPBFRsYbgQbFiUjMhXsID1+segfvGel5rnaM/OCjSWZKj7hlBhPJe1PWBJzlKb2pRu9Ayx5BjiVMOzNxtADrCuZWNOL3aHFhlQOYG+z7j+D2kiJsH2XmHu1C0+JlCQxd0+1LhlZM28S4wlZ+X0Nlm947RKLLcwAcwX9puccJO1O1I3qghPgYRRvP4ohbpgOamo3WQ8+4iynlWTKUEG0HsVRYcaWlIZTwVwLV5oVhB6TKRt5qIhkemzrcTq445kMJnnpP2WgZk86LJFaz2XqlYuH6z97C/LUXio9neYiW73fTJ6MLVdFZZiiy+snlQCTwyIxyHjJqBFzDwgtuf8DoEekJNT4XFtYsQ8092NXmfVjy5zFMs1ndgMXrCzwrXot4IZzq2L1P3UDl0szI7mq9F9H9BkRv4+wTYzQdwuhD+1XMvBvsL/Vje93t94+6Ow8b+JbQzfRbbSJYvQY7aBXaBcNEA1aQRRsB0/Dz+HX8Fv4fSkNg8ZzA61U+OMnFHoJzA==</latexit>

BART = Bayesian Additive Regression Trees

No greedy search, but Backfitting MCMC algorithm

At each step, we sample from the full-conditional using the residuals given the other trees

A move in the tree structure consists of Growing, Pruning, Changing

43

E[Y | X = x] =⌧X

t=1

T (X;Rt,�t) =⌧X

t=1

MtX

m=1

µmtI{x2Rmt}<latexit sha1_base64="EtHyiQQzNe3VUJpcE4DNNilbO8s=">AAADPXicfVHdbtMwGHXC3yh/HVxy80GHNCRUJeN3QKVJCAkukEa1bkV1FzmO21mznSh2oJXlC96F9+A5eADuELfc4qYBrTD4rEjH5zvHdr6TFoJrE0VfgvDM2XPnL6xdbF26fOXqtfb69X2dVyVlA5qLvBymRDPBFRsYbgQbFiUjMhXsID1+segfvGel5rnaM/OCjSWZKj7hlBhPJe1PWBJzlKb2pRu9Ayx5BjiVMOzNxtADrCuZWNOL3aHFhlQOYG+z7j+D2kiJsH2XmHu1C0+JlCQxd0+1LhlZM28S4wlZ+X0Nlm947RKLLcwAcwX9puccJO1O1I3qghPgYRRvP4ohbpgOamo3WQ8+4iynlWTKUEG0HsVRYcaWlIZTwVwLV5oVhB6TKRt5qIhkemzrcTq445kMJnnpP2WgZk86LJFaz2XqlYuH6z97C/LUXio9neYiW73fTJ6MLVdFZZiiy+snlQCTwyIxyHjJqBFzDwgtuf8DoEekJNT4XFtYsQ8092NXmfVjy5zFMs1ndgMXrCzwrXot4IZzq2L1P3UDl0szI7mq9F9H9BkRv4+wTYzQdwuhD+1XMvBvsL/Vje93t94+6Ow8b+JbQzfRbbSJYvQY7aBXaBcNEA1aQRRsB0/Dz+HX8Fv4fSkNg8ZzA61U+OMnFHoJzA==</latexit>

On the interpretation

Interpreting and understanding

45

Interpreting tree-based models : longing transparency

46

Black box

X y = f(x) + ✏<latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit>

Accurate predictions No (few) assumptions

by = df(x)<latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit>

by<latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit><latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit><latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit><latexit sha1_base64="FnE4oyBTKg2X6CtyIjxvJ9uvArY=">AAAConicfVFLbxMxEHaWVwmvFo5cDAkSp2oXIcGxggvi1BbSVoqjauydNFb9WNmzlGi1B34EV/hd/Bu8yQqRBjEjy5+++WbG45GV0ZHy/Ncgu3Hz1u07O3eH9+4/ePhod+/xSfR1UDhR3vhwJiGi0Q4npMngWRUQrDR4Ki/fd/HTLxii9u4zLSucWbhweq4VUKKm4kqXuADiS36+O8r385XxbVD0YMR6OzzfG3wTpVe1RUfKQIzTIq9o1kAgrQy2Q1FHrEBdwgVOE3RgMc6a1Ztb/iIxJZ/7kI4jvmL/zmjAxri0Mikt0CJej3XkP2PSJlp6U272p/nbWaNdVRM6tW4/rw0nz7tv4aUOqMgsEwAVdJqAqwUEUJQ+bygcXilvLbiyEdqVbSOs9F+bsagwVOLZyjs4bttNsfufuodrj0hWuzpulThGMH9KpDtNLCU/bjthWlpxfUXb4OTVfpHw0evRwbt+fTvsKXvOXrKCvWEH7AM7ZBOmmGff2Q/2MxtnH7Oj7NNamg36nCdswzLxGyLt0FQ=</latexit>X

X y = f(x) + ✏<latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit><latexit sha1_base64="lmdalk2ezbeczQt3wGyxgXAH/pk=">AAACrHicfVFLT9wwEPamL9i+oBy5uF0qUVVaOagUOFRC9NIjRSwgbSJkOxOw8Eu2Q4miHPpLem1/Ev+mzm5adfuakeVP33wz4/EwK4UPhNwOkjt3791/sLQ8fPjo8ZOnK6vPTrypHIcJN9K4M0Y9SKFhEkSQcGYdUMUknLKr91389BqcF0Yfh9pCruiFFqXgNETqfGWtxu9wuXnzCr/GGVgvZMeOyHibpHtvU0zGZGYRbG2TvV2C054Zod4Oz1cHn7PC8EqBDlxS76cpsSFvqAuCS2iHWeXBUn5FL2AaoaYKfN7MXt/il5EpcGlcPDrgGftrRkOV97ViUalouPS/xzryrzGmIs2MLBb7h3I3b4S2VQDN5+3LSuJgcPdBuBAOeJB1BJQ7ESfA/JI6ykP8xmGm4RM3SlFdNJnQRdtkipmbZiOz4Gz2fOYd3GjbRbH+n7qHc/cQlNCV/6PEEVD5s0S848SM4aO2E8al/dgM/jc42RqnEX98M9o/6Ne3hNbRC7SJUrSD9tEHdIgmiKMafUFf0bdknBwn0ySfS5NBn7OGFiwpvwOSwNLu</latexit>

by = df(x)<latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit><latexit sha1_base64="jRiB5LFxU/DlwBcnk0Mzicvc6Mc=">AAACtHicfVFLTxsxEHa2L0gfhPbIxSVUopfIi4DCoRIqF44UNYCEo8j2zhILP1a2txCt9sCv6bX9Of03eJOlalraGVn+/M03MxoPL5T0gZCfneTR4ydPny0td5+/ePlqpbf6+tTb0gkYCqusO+fMg5IGhkEGBeeFA6a5gjN+ddjEz76C89KaL2FawEizSyNzKViI1Li3Rq9lBhMW8BR/xPePKt+8eV+Pe30y2CHp/m6KyYDMLIKtHbK/R3DaMn3U2vF4tXNLMytKDSYIxby/SEkRRhVzQQoFdZeWHgomrtglXERomAY/qmZT1PhdZDKcWxePCXjG/p5RMe39VPOo1CxM/J+xhnwwxnWkuVXZYv+Q740qaYoygBHz9nmpcLC4+SicSQciqGkETDgZJ8BiwhwTIX5nlxq4FlZrZrKKSpPVFdXc3lQbtABX0Lczb+BGXS+Kzf/ULZy7h6ClKf1fJU6AqV8l4h0n5hyf1I0wLu1+M/jf4HRrkEb8ebt/8Kld3xJaQ+toE6XoAzpAR+gYDZFAt+gb+o5+JLsJTUQCc2nSaXPeoAVLzB2RgtcZ</latexit>

Accurate estimates Assumptions needed

by = f(x, b✓)<latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit><latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit><latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit><latexit sha1_base64="f8PoFNp6AX+VUnUNbcIJligTQxY=">AAACu3icfVFLb9NAEN6YR0t4pXDksjRFKhKK7KrPQ6UKLhxLRdpKdRTtrsfN0n1Yu2PayPKht/4arvBX+DesExcRXjNa7advvpnRzPBCSY9x/L0T3bl77/7S8oPuw0ePnzztrTw79rZ0AobCKutOOfOgpIEhSlRwWjhgmis44RfvmvjJZ3BeWvMRpwWMNDs3MpeCYaDGvdX0UmYwYUindJ/m61dvbokqxQkgq1+Pe/14sBUne9sJjQfxzALY2Ir3dmOatEyftHY4Xulcp5kVpQaDQjHvz5K4wFHFHEqhoO6mpYeCiQt2DmcBGqbBj6rZMDV9FZiM5taFZ5DO2F8zKqa9n2oelJrhxP8ea8i/xrgONLcqW+yP+e6okqYoEYyYt89LRdHSZl80kw4EqmkATDgZJqBiwhwTGLbaTQ1cCqs1M1mVSpPVVaq5varW0gJckb6ceQPX6npRbP6nbuHcPaCWpvR/lDgCpn6WCH+YmHN6VDfCcLTby9B/g+ONQRLwh83+wdv2fMvkBVkl6yQhO+SAvCeHZEgEuSFfyFfyLdqPRPQpUnNp1GlznpMFi8ofp7PZ/Q==</latexit>

...<latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit><latexit sha1_base64="DS2CiyTtBeyi7XrtuJnrYrmOpLs=">AAACnnicfVFLbxMxEHaWAiW8WjhycUmROEXeqoX2VoGEekG0VdNE6kaV7Z20VvxY2bOFaLUHfgJX+GX9N3iTpWp4zcjyp2++mfF4RKFVQMauO8mdlbv37q8+6D589PjJ07X1Z6fBlV7CQDrt/EjwAFpZGKBCDaPCAzdCw1BM3zfx4RX4oJw9wVkBY8MvrJooyTFSw+wqdxi652s91t9h6d6blLI+m1sEWztsb5fRtGV6pLXD8/XO1yx3sjRgUWoewlnKChxX3KOSGupuVgYouJzyCziL0HIDYVzN31vTV5HJ6cT5eCzSOXs7o+ImhJkRUWk4XobfYw3515gwkRZO58v9cbI7rpQtSgQrF+0npaboaPMlNFceJOpZBFx6FSeg8pJ7LjF+XDez8Fk6Y7jNq0zZvK4yI9yXajMrwBfZxtwbuFnXy2L7P3ULFx4AjbJl+KPEMXB9UyLecWIh6HHdCOPSfm2G/hucbvXTiI+2e/vv2vWtkhfkJXlNUvKW7JMDckgGRJIp+Ua+kx8JTT4kH5NPC2nSaXOekyVLRj8Bzp/O9Q==</latexit>




… Using machine learning for making decisions…

47

- Out of the Artificial Intelligence/technological framework …

- Sometimes accurate predictions are not enough for making good decisions

- To understand whether the algorithm is working in a sensible way, the black box has to be whitened

- It is not a matter of exactly understanding every bit and bytes of the model for all data points

- It is a matter of exactly understanding what drives the prediction, which are the discriminative predictors … are they reasonable?

How to interpret Trees and Forests?

One tree -> the tree plot makes clear how predictions have been made

Forests are less transparent -> Variable importance measures

Variable importance is a measure of the importance of each variable in predicting

the response

Several way to compute variable importance: the best is based on permutation of

the variable -> Gain in prediction

Importance in predicting is not importance in explaining or causing

48

How to interpret Variable importance?

49

50

- Think about Personalised medicine

- Or algorithms for banks to give a loan

- It is not just a matter of “to know how” but also of “Is it fair?”

Y

- Variable importance in predicting sometimes deviates from true causal mechanism/direct association

Tricky example

Generative/explanatory vs Predictive models

51

- Statistical/Machine learning is focused on predicting

- Computer science, text or image processing rely on predictive modelling: the

focus is on new/future observation

- Human sciences usually require generative/explanatory modelling: observed

data are used to assess causal/explanatory hypotheses

- Predicting is different from explaining

- Lack of understanding in many disciplines of this distinction

Thanks for your attention!Some references

- Breiman, L. (2001). Random forests. Machine Learning, 45:5–32.

- Chipman, H. A., George, E. I., and McCulloch, R. E. (2010). BART: Bayesian additive regression trees. Annals of Applied Statistics, 4:266–298.

- Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. (1984). Classification and Regression Trees. Chapman & Hall/CRC.

- Hothorn, T., Hornik, K., and Zeileis, A. (2006). Unbiased recursive partitioning: a conditional inference framework. Journal of Computational and Graphical Statistics, 15:651–674

- Berk, R. A. (2008). Statistical learning from a regression perspective (2nd edition). New York: Springer.

- Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.

Flexibility - interpretability trade-off

55

Flexibility/Accuracy

Inte

rpre

tabi

lity/

Tran

spar

ency

Linear/logistic regression

Deep learning

Random Forest

BART

Bagging

CART

SVM

Boosting

GAM

Cross-validation

Training Error versus Test error

57

— The training and the test errors can be quietly different, and can vary a lot among

different partitions of the data

— Training error : it is the value of the loss measure computed on the training data

— Test error : it is the value of the loss measure computed predicting the statistical

learning method on the test data

Training- versus Test-Set Performance

58

K-fold Cross-validation

59

Com

plet

e da

ta

Test

Test

Test

Test

TestTrain

Train

Train

Train

Train

Train

Train Train Train Train

Train

Train

Train

Train

Train

Train

Train

Train

Train

Train

Test

err

or

K=5

Modelli basati su alberi e - loro interpretazione

Documents

Transcript of Modelli basati su alberi e - loro interpretazione