Deep Learning for generative modeling - MLVU Learning2.annotated.pdfPos from X Neg Generate...

Deep generative models Part 1: Generator networks

Machine Learning mlvu.github.io

Vrije Universiteit Amsterdam

Inthislecture,we’lllookatgenerativemodeling,Thebusinessoftrainingprobabilitymodelsthataretoocomplextogiveusanexplicitdensityfunctionoverourfeaturespace,butthatdoallowustosamplepoints.Ifwetrainthemwell,wegetpointsthatlooklikethoseinourdataset.

generative models

source: A Style-Based Generator Architecture for Generative Adversarial Networks, Karras et al.

Hereistheexample,wegaveinthe?irstlecture.Adeepneuralnetworkfromwhichwecansamplehighlyrealisticimagesofhumanfaces.


visual shorthand

3

any neural networka (standard)

multivariate normal distribution

input

output

Intherestofthelecture,wewillusethefollowingvisualshorthand.Thediagramontheleftrepresentsanykindofneuralnetwork.Wedon’;tcareabouttheprecisearchitecture,whetherithasoneorahundredhiddenlayersandwhetheritusesfullyconnectedlayersofconvolutions,wejustcareabouttheshapeoftheinputandtheoutput.

Theimageontherightrepresentsamultivariatenormaldistribution.Thinkofthisasacontourlineforthedensityfunction.We’vedrawnitina2Dspace,butwe’lluseitasarepresentationforMVNsinhigherdimensionalspacesaswell.IftheMVNisnonstandard,we’lldrawitasanellipsesomewhereelseinspace.

how to turn a NN into a probability distribution

option 1:

4

µ ⌃<latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit><latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit><latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit><latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit>

<latexit sha1_base64="1xMfvi+K7RtUooBB3m5QI4uNvLI=">AAAHZXicfVXbbtNAEHULNCVQaAHxwgMWERJCUWWn9MJDpdKktAJ6oWqSSk1UrTeTxMr6ot11EnflX+AVfo0v4DdYOxfZWcO+eDTnnPHM2dWu5RObccP4vbR87/6DlcLqw+Kjx2tPnq5vPGswL6AY6tgjHr22EANiu1DnNidw7VNAjkWgaQ2qMd4cAmW2517x0Ie2g3qu3bUx4nGqNS4Wb9dLxqaRLF0NzGlQ0qbr4nZj5Wur4+HAAZdjghi7MQ2ftwWi3MYEomIrYOAjPEA9uAl4d68tbNcPOLg40t9KrBsQnXt63I/esSlgTkIZIExtWUHHfUQR5rLrYrYUAxc5wMqdoe2zSciGvUnAkRy5LcaJJdFaRil6FPl9G48zrQnkMAfxvpJkoWNlkxAQoEMnm4zblE0uMMdAsc1iEy6kM+d+bDO78i6meD/0++CySASURGmhBIBS6EphEjLggS+SaeTeDtg+pwGU4zDJ7dcQHVxCpyzrZBLZdrrEQzybsuQY0h0XRthzHOR2RMuPRIvDmItWeTNKvEujl5EQrdgoy9IvYziDnqXQsyjKgkcp8EiCWbQ+R7t6fVHaSIEN5a/NFNpclFpBCg0UdJhCh0pla5SCRwo8TqFjBQ1TaKigdyn0TvUZyWNxU2mLyV4kmyrOiT2EYwrgRqJUiRZnoXK/b8ysJD4DomRGid0d6MqLYQI4YUwXJ1en3yJR3atsGzvRIsMiAcwoxtbOdtVQKL1JN1OOsbdXOVQ4HkVub16odrTzyVQL+QH1yZy0u7v1+aNaKQRCvNG8UvWwVtlaPEcUKyZMZ9VLpq6Y1sujT6fKFVh5golTufyByj+mKPwH28urPjMwV+HnKWZu5irCPMXM2pliYQg/Pq0DLH8X3+uITCg1kDc+hVN5is/lLYW4R9/Lo0t7ji3tk99WOY7+R0TjGVFGyfNjLj42atCobJo7m8b3D6WDL9OHaFV7pb3R3mmmtqsdaCfahVbXsNbXfmg/tV8rfwprhReFlxPq8tJU81zLrMLrv8TYrL4=</latexit>x


<latexit sha1_base64="4+Iz0rR5dS3H7wE4nLPxkEVgf8A=">AAAH23ichVVdb9s2FJW7tm69pk27x74QMwakhRFITptkGAJ0TbIW25pkWZwUiIyAommZMCURJGVLJfi0t2Gv+3F72W8Z9eFMstSNfvDlPedckYcXpMcoEdK2/+rc+eLuvfvdBw97Xz7aePxk8+mzSxHFHOERimjEP3pQYEpCPJJEUvyRcQwDj+Irb36Y4VcLzAWJwguZMjwOoB+SKUFQmtTNZsK23BS4AZkAN3kBDsDJv/MIKdcLzCTWA+B65exX4gdQvwDud/lP4kQqAJZEzgDQYOt/VAdgumU+1OvdbPbtbTsfoBk4ZdC3ynF28/T+T+4kQnGAQ4koFOLasZkcK8glQRTrnhsLzCCaQx9fx3K6P1YkZLHEIdLgG4NNYwpkBDIXwIRwjCRNTQARJ6YCQDPIIZLGq169lMAhDLAYTBaEiSIUC78IJDRGj1WSH4TeqCmVzyGbEZTUlqZgIAIoZ42kSAOvnsQxxXwR1JPZMs0i15gJ5oiIzIQz48wpyw5XXERnJT5L2QyHQquYU10VGgBzjqdGmIcCy5ipfDemo+biQPIYD7Iwzx0cQT4/x5OBqVNL1JczpRGU9ZRntmHcCfESRUEAw4lymVZF97iDbZ17V0XPtVJuZpTngfMMrqEnFfRE6zp4XAGPDVhHR7foFIzWpZcV8LLx1asKerUu9eIKGjfQRQVdNCp7ywq8bMBJBU0aaFpB0wb6qYJ+avoMTVtcD8eqOIv8UNUpJQv8jmMcatUf6vW9cHPe105dkvWA6js6t3uCp+Y6KoAgzejq/cWHn7U63B++tnf1OsOjMV5R7J3d14d2g+IXqyk59v7+8G2DE3EY+reFjo53v3eahVjMGb0l7e3t/PBts1KKKY2Wt5UO3x4Nd9b7iKOGCeVeQd8BDdP8Nnq5q1aB1yYonGrlz5v8dxymn2FHbdVXBrYqWJti5WarIm1TrKxdKdY2wbJunZtHg2X3OqQF5QibG5/jD6aLT80tBWXEX5rW5X5AjH3m3x1k0X8RYbIimih/fpz1x6YZXA63nd1t+5dX/Tc/lg/RA+u59bW1ZTnWnvXGem+dWSMLWX937nY2Oo+74+5v3d+7fxTUO51S85VVG90//wFhWNO5</latexit>

p(y | x) = N(y | µ,⌃) with (µ,⌃) = f(x)<latexit sha1_base64="4+Iz0rR5dS3H7wE4nLPxkEVgf8A=">AAAH23ichVVdb9s2FJW7tm69pk27x74QMwakhRFITptkGAJ0TbIW25pkWZwUiIyAommZMCURJGVLJfi0t2Gv+3F72W8Z9eFMstSNfvDlPedckYcXpMcoEdK2/+rc+eLuvfvdBw97Xz7aePxk8+mzSxHFHOERimjEP3pQYEpCPJJEUvyRcQwDj+Irb36Y4VcLzAWJwguZMjwOoB+SKUFQmtTNZsK23BS4AZkAN3kBDsDJv/MIKdcLzCTWA+B65exX4gdQvwDud/lP4kQqAJZEzgDQYOt/VAdgumU+1OvdbPbtbTsfoBk4ZdC3ynF28/T+T+4kQnGAQ4koFOLasZkcK8glQRTrnhsLzCCaQx9fx3K6P1YkZLHEIdLgG4NNYwpkBDIXwIRwjCRNTQARJ6YCQDPIIZLGq169lMAhDLAYTBaEiSIUC78IJDRGj1WSH4TeqCmVzyGbEZTUlqZgIAIoZ42kSAOvnsQxxXwR1JPZMs0i15gJ5oiIzIQz48wpyw5XXERnJT5L2QyHQquYU10VGgBzjqdGmIcCy5ipfDemo+biQPIYD7Iwzx0cQT4/x5OBqVNL1JczpRGU9ZRntmHcCfESRUEAw4lymVZF97iDbZ17V0XPtVJuZpTngfMMrqEnFfRE6zp4XAGPDVhHR7foFIzWpZcV8LLx1asKerUu9eIKGjfQRQVdNCp7ywq8bMBJBU0aaFpB0wb6qYJ+avoMTVtcD8eqOIv8UNUpJQv8jmMcatUf6vW9cHPe105dkvWA6js6t3uCp+Y6KoAgzejq/cWHn7U63B++tnf1OsOjMV5R7J3d14d2g+IXqyk59v7+8G2DE3EY+reFjo53v3eahVjMGb0l7e3t/PBts1KKKY2Wt5UO3x4Nd9b7iKOGCeVeQd8BDdP8Nnq5q1aB1yYonGrlz5v8dxymn2FHbdVXBrYqWJti5WarIm1TrKxdKdY2wbJunZtHg2X3OqQF5QibG5/jD6aLT80tBWXEX5rW5X5AjH3m3x1k0X8RYbIimih/fpz1x6YZXA63nd1t+5dX/Tc/lg/RA+u59bW1ZTnWnvXGem+dWSMLWX937nY2Oo+74+5v3d+7fxTUO51S85VVG90//wFhWNO5</latexit>

p(y | x) = N(y | µ,⌃) with (µ,⌃) = f(x)

Aplainneuralnetworkispurelydeterministic.Ittranslatesaninputtoanoutputanddoesthesamethingeverytimewithnorandomness.Howtoweturnthisintoaprobabilitydistribution?

The?irstoptionistotaketheoutputandtointerpretitastheparametersofamultivariatenormal(aμandaΣ).Ifthedimensionalityoftheoutputislarge,we’lloftentakeΣtobeadiagonalmatrix(whichiswhyit’sthesamesizeasμinthisimage).

Thedistributionwerepresentinthiswayisalwaysaconditionaldistribution.Wegetaprobabilityonouroutputspace,conditionedontheinputvaluex.Ifwechangex,wegetadifferentdistributionontheoutput

space.

categorical

5

s ss

0.1 0.6 0.3

softmax

p(C=Pos|x)

binary classification

0.6

Thisprincipleisnotnew.Inneuralnetworkclassi?ication,thestandardapproacheswe’veseen(alinearregressionlayerandasoftmaxoutput)theneuralnetworkalsoproducestheparametersforaprobabilitydistributionontheoutputspace(aBernoullidistributionontheleftandaCategoricalontheright).

Evenlinearregression,aswesawinthepreviouslecture,canbethoughtofasprovidingthemeanforanerrordistribution,andinthatcasethemaximumlikelihoodobjectiveisequivalenttotrainingbysquarederrors.Wejustdon’tprovidethevarianceoftheoutputdistributioninthatcase,onlythemean.

6



⌃ = diag(�)

<latexit sha1_base64="zBAH71B/0T/jJqhx8nA4nbfHaBk=">AAAHdXicfVXbbttGEGUutRI1aZz0sShAVE2QFoJCyvElDwbS2G2Coold17IDmIKxXI2ohZYX7C5lMYv9jHxNX5OP6Jf0tcuLDFLLlC8czDlnMHN2sOsnlHDhOP/cuHnr9lcbnTt3u1/fu//Ng82Hj854nDIMIxzTmL33EQdKIhgJIii8Txig0Kdw7s8Pcvx8AYyTODoVWQLjEAURmRKMhE5dbj7zfCw9P5TeXyQIkVL2vu0JWAo5IShQTyvY9ngB/3S52XMGTvHZZuBWQc+qvuPLhxsDbxLjNIRIYIo4v3CdRIwlYoJgCqrrpRwShOcogItUTPfGkkRJKiDCyn6ssWlKbRHbefP2hDDAgmY6QJgRXcHGM8QQFnrEbrMUhwiFwPuTBUl4GfJFUAYCaX/Gcln4p+43lDJgKJkRvGy0JlHIQyRmRpJnod9MQkqBLcJmMm9TN7nGXALDhOcmHGtnjpL8TPhpfFzhsyyZQcSVTBlVdaEGgDGYamERchBpIotp9CLM+b5gKfTzsMjtHyI2P4FJX9dpJJrtTGmMRDPl6zG0OxFc4TgMUTSRXqJkuSBef6AK7+roiZLSy43yffskhxvouxr6Tqk17egandojjTbAsxp4ZhQ+r6Hn61I/raGpgS5q6MKo7F/V4CsDXtbQpYFmNTQz0A819INpJdInfzEcy9Lu4tzkESULeM0AIiV7Q7U+C9NHeuE2Jfkxy56rCrsnMNUXRQmEWU6Xb07f/qHkwd5w29lR6wyfprCiOFs72weOQQnKbiqOs7c3fGVwYoai4LrQ4a87v7hmoSRlCb0m7e5u/fbCrJQBpfHVdaWDV4fDrfU9YtgwoZrV7rm2YVrQRq+mahX4bYLSqVb+3OS/Zij7Ajtuq74ysFWRtClWbrYqsjbFytqVYm2IJN/WuX4NkvzqRrSkHIK+1Bm81Vt8pC8iJGL2s15dFoRE26f/Xj+P/o+Iliuijrpd/cK46++JGZwNB+7zwfafz3svf6/emjvWd9YP1lPLtXatl9Yb69gaWdj6aP1tfbI+b/zb+b7zY+dJSb15o9J8azW+zrP/AFfws5k=</latexit>

<- exp(x) activationµ ⌃<latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit><latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit><latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit><latexit sha1_base64="UrcwEiWOoygl4iFhaGReX1Ss3as=">AAAHanicfVXbbtNAEDW3BsKtwBPixSIgIRRFdgptEUICWi5CQEtp2kp1VK03E8fK2l7trtOY1X4BX8MrfAkfwT+wTpzKzhqch4zmnDOaObPa9SkJuXCc3+fOX7h4aaVx+Urz6rXrN26u3rp9wJOUYejhhCTsyEccSBhDT4SCwBFlgCKfwKE/3srxwwkwHibxvsgo9CMUxOEwxEjo1MnqQy/B0vMj6UWpUrb3PP/Znl8kv4ZBhJQ6WW05HWf22WbgFkHLKr7dk1srHW+Q4DSCWGCCOD92HSr6EjERYgKq6aUcKMJjFMBxKoabfRnGNBUQY2U/1NgwJbZI7LxlexAywIJkOkCYhbqCjUeIISz0YM1qKQ4xioC3B5OQ8nnIJ8E8EEi70pfTmWvqekUpA4boKMTTSmsSRTxCYmQkeRb51SSkBNgkqibzNnWTS8wpMBzy3IRd7cwOzTfB95PdAh9ldAQxVzJlRJWFGgDGYKiFs5CDSKmcTaPXP+YvBEuhnYez3IttxMZ7MGjrOpVEtZ0hSZCopnw9hnYnhlOcRBGKB9KjSnoCpkJ67Y6aeVdG95TUB0gb5fv2Xg5X0M8l9LNSS9reGTq0exqtgAcl8MAofFhCD5elflpCUwOdlNCJUdk/LcGnBjwtoVMDzUpoZqDfSug300qkN3/c7cu53bO9yR0STuAdA4iVbHXV8ixMr/TYrUryNcuWq2Z2D2Cor4c5EGU5Xb7f//RRya3N7lNnXS0zfJLCguKsrT/dcgxKMO+m4Dibm93XBidhKA7OCm2/WX/lmoVoyig5I21srL19ZlbKgJDk9KzS1uvt7tryOWLYMKGY1W65tmFaUEcvpqoV+HWCuVO1/LHJf8dQ9g92Uld9YWCtgtYpFm7WKrI6xcLahWJpCJqf1rF+DWh+dSMyp2yDvtQZfNKneEdfREgk7LE+uiyIQm2f/vfaefQ/IpouiDpqNvUL4y6/J2bQ63aeddwvT1ovPxRPzWXrnnXfemS51ob10npv7Vo9C1vfrR/WT+vXyp/Gncbdxr059fy5QnPHqnyNB38BffSvQA==</latexit>

or softplus: ln(1+ex)

Ifwedoprovidethemeanandthevarianceofanoutputdistribution,itlookslikethis(forann-dimensionaloutputspace).Wesimplysplittheoutputlayerintwoparts,andinterpretonepartasthemeanandtheotherasthecovariancematrix.

Sincerepresentingafullcovariancematrixwouldgrowverybig,weusuallyassumethatthecovariancematrixonlyhasnonzerovaluesonthediagonal.Thatwaytherepresentationofthecovariancerequiresasmanyargumentsastherepresentationsofthemean,andwecansimplysplittheoutputintotwohalves.

Equivalently,wecanthinkoftheoutputdistributionasputtinganindependent1DGaussianoneachdimension,withameanandvarianceprovidedforeach.

Forthemean,wecanusealinearactivation,sinceitcanhaveanyvalue,includingnegativevalues.However,thevaluesofthecovariancematrixneedtobepositive.Toachievethis,weoftenexponentiatethem.We’llcallthisanexponentialactivation.Analternativeoptionisthesoftplusfunctionln(1+ex),whichgrowslessexplosively.

for images

7


0 1µ87r,�87r

<latexit sha1_base64="Tuex34KXG/9ARK+hXtnE4kGLDrs=">AAAH33icfVVNj9s2EFU+GqdO02ySYy5qjAJFYSwkb3btHAIk+9HkkGS3i/VugJWxoOixLJiSCJLyWiF4zq3oNX+gt16b35J/E8qWXUlUq4vH894bDd+Qok9JyIXjfL1x89bt7+607n7fvvfD/R8fbD18dM6TlGEY4oQk7IOPOJAwhqEIBYEPlAGKfAIX/uwgxy/mwHiYxGciozCKUBCHkxAjoVNXWz95CZZelKorOegz1bU9X//nYRChInW11XG2neVjm4FbBB2reE6uHt75yxsnOI0gFpggzi9dh4qRREyEmIBqeykHivAMBXCZislgJMOYpgJirOyfNTZJiS0SO2/XHocMsCCZDhBmoa5g4yliCAu9qHa1FIcYRcC743lI+Srk82AVCKQdGcnF0jF1v6KUAUN0GuJFpTWJIh4hMTWSPIv8ahJSAmweVZN5m7rJGnMBDIc8N+FEO3NM8ynws+SkwKcZnULMlUwZUWWhBoAxmGjhMuQgUiqXq9Gjn/EXgqXQzcNl7sUhYrNTGHd1nUqi2s6EJEhUU75ehnYnhmucRBGKx9KjSnoCFkJ63W219K6MniqpN5A2yvft0xyuoO9L6HulquBRCTzSYBUdbtCJPaxLz0vgufHWixJ6UZf6aQlNDXReQudGZf+6BF8b8KKELgw0K6GZgX4soR9Nn5HeFpe9kVzNYjlUeUzCObxmALGSnZ6qr4XpeV+6VUm+B2THVUu7xzDR340VEGU5Xb45e/dWyYNBb9fZU3WGT1JYU5ydvd0Dx6AEq24KjjMY9PYNTsJQHGwKHR7tvXLNQjRllGxI/f7Ob8/NShkQklxvKh3sH/Z26gvTjlSbcvuu49R3G8OGVYUjdse1DWuDJnrxmkaB3yRY+dnIn5n81wxl/8FOmqqvbW5U0CbF2vNGRdakWA9grahK4gab/h3HRlNbOc0PwkxfQzS/MhBZ1T0EfZkweKcPyLH+ACKRsF/1qWBBFOpa+tfr5tH/EdFiTdRRu61vNrd+j5nBeW/bfba9+/uzzsv94o67az2xnlq/WK7Vt15ab6wTa2hh65P1t/WP9aWFWp9af7T+XFFv3ig0j63K0/r8Daz23K8=</latexit>

µ87r,�87r


µ87r,�87r


µ87r,�87r


�

<latexit sha1_base64="GD9afcuMm0lKVCYncR3B4YPFvW8=">AAAHzXicfVXLbtQwFA3lMWV4tIUlm4gREkKjKpk+WVQqfQALoKXqtJWaUeV47mSicRLLdqaTmrDlI9gC/8Tf4MyLJA54k6t7zrm599iJXUp8Lizr962F23fu3qst3q8/ePjo8dLyypMzHsUMQxtHJGIXLuJA/BDawhcELigDFLgEzt3BfoafD4FxPwpPRUKhEyAv9Hs+RkKlrpaXHBdLxw1Mh/tegNKr5Ya1ao2XqQf2NGgY03V8tXLvl9ONcBxAKDBBnF/aFhUdiZjwMYG07sQcKMID5MFlLHrbHemHNBYQ4tR8obBeTEwRmVlzZtdngAVJVIAw81UFE/cRQ1ioEerFUhxCFABvdoc+5ZOQD71JIJCavyNHY3/SRwWl9BiifR+PCq1JFPAAib6W5EngFpMQE2DDoJjM2lRNlpgjYNjnmQnHypkjmnnOT6PjKd5PaB9CnsqYkTQvVAAwBj0lHIccREzleBq10QO+I1gMzSwc53YOEBucQLep6hQSxXZ6JEKimHLVGMqdEK5xFAQo7EqHptIRMBLSaa6mY+/y6EkqpZMZ5brmSQYX0E859FOaFsHDHHiowCLanqM9s12WnuXAM+2t5zn0vCx14xwaa+gwhw61yu51Dr7W4FEOHWlokkMTDb3JoTe6z0gdi8tWR072Yryp8oj4Q3jHAMJUNlppeRam9vvSLkqyMyAbdjq2uws99ZeYAEGS0eX7048fUrm/3dqwNtMywyUxzCjW2ubGvqVRvEk3U461vd3a0zgRQ6E3L3RwuPnG1gvRmFEyJ21trb19rVdKgJDoel5pf++gtVYeTDlSbMresi2rfNoY1qyaOmI2bFOz1quiT19TKXCrBBM/K/kDnf+OoeQf7Kiq+szmSgWtUsw8r1QkVYrZBswURUlYYdPf7ZhrSpPT7EMYqEuIZlcGIpO6B6AuEwYf1QdypH6ASETslfoqmBf4qpZ6Os0s+h8RjWZEFdXr6mazy/eYHpy1Vu311Y3P643dvekdt2g8M54bLw3b2DJ2jffGsdE2sBEb340fxs/aUS2ufal9nVAXbk01T43Cqn37A8zl1SY=</latexit>

Here’swhatthatlookslikewhenwegenerateanimage.Theoutputdistributiongiveusameanvalueforeverychannelofeverypixel(a3-tensor)andacorrespondingvarianceforeverymean.Ifwelookatwhatthattellsusabouttheredvalueofthepixelatcoordinate(8,7)weseethatwegetaunivariateGaussianwithaparticularmeanandavariance.Themeantellsusthenetwork’sbestguessforthatvalue,andthevariancetellsushowcertainthenetworkisaboutthisoutput.Weshouldnotethatneuralnetworksbydefaultareterribleatestimatinghowcertaintheyshouldbe,soweshouldtakethisvaluewithagrainofsalt.

8

output distribution maximum log likelihood loss

Bernoulli

Categorical

Normal (mean only)

Normal (mean and variance)

Binary cross-entropy

Categorical cross-entropy

Mean squared error loss

Laplace (median only) Mean absolute error loss

<latexit sha1_base64="PV7rQ6v2HlZxoi2S/Ldy8SWHmQ4=">AAAHlXicfVVdb9s2FFW7ru68dku3hz30RZgxoNtcQ1LWJHsI0DZeWwxtnQVxUiByDYq+lglTEkFSjlWCP6s/Ztjr9j9G+SOQTG160cU951zce3hBRowSIT3vz1u3P7vz+d3WvS/aX95/8NXXew+/uRBZzjEMcUYz/j5CAihJYSiJpPCecUBJROEymp+U+OUCuCBZei4LBqMExSmZEoykSY33Bk/cUOTJmLghTd0wwioUJE6QNpmf3XDKEVa+VoGqQfpDoB+HyzF5EmYmneQm9+OHIAzb472O1/NWn2sH/iboOJvvdPzwbi+cZDhPIJWYIiGufI/JkUJcEkxBt8NcAEN4jmK4yuX0aKRIynIJKdbuDwab5tSVmVsO504IByxpYQKEOTEVXDxDZgZpLGjXSwlIUQKiO1kQJtahWMTrQCLj30gtV/7qBzWlijliM4KXtdYUSkSC5MxKiiKJ6knIKfBFUk+WbZomd5hL4JiI0oRT48yAlWcmzrPTDT4r2AxSoVXOqa4KDQCcw9QIV6EAmTO1msYsylwcS55DtwxXueM+4vMzmHRNnVqi3s6UZkjWU5EZw7iTwjXOkgSlExUyrUIJS6nCbk+vvKuiZ1qZfTFGRZF7VsI19F0Ffaf1jnZ4g07doUFr4EUFvLAKX1bQy11plFfQ3EIXFXRhVY6uK/C1BS8r6NJCiwpaWOjHCvrRthKZk78KRmpt9+rc1ICSBbziAKlWnUDvzsLNkV75dUl5zKrj65XdE5iai2QNJEVJV6/P377R6uQoeOod6F1GRHPYUrz9g6cnnkWJ191sON7RUfDC4mQcpfFNof5vB899uxDLOaM3pMPD/Ze/2pUKoDS7vql08qIf7O/uEceWCZtZ3Y7vWqbFTfTNVI2CqEmwdqqRP7f5rzgq/oOdNVXfGtioYE2KrZuNiqJJsbV2q9gZgpXbOjevASuvbkTXlD6YS53DW7PFA3MRIZnxn8zq8jghxj7zD7tl9H9EtNwSTdQuXxh/9z2xg4ug5x/0vD9+6Tz7ffPW3HMeOd87jx3fOXSeOa+dU2foYOeT85fzt/NP67vWcavfermm3r610Xzr1L7W4F/kt76v</latexit>

-X

i

ln�i +1

2�i2(xi - µi)

2

Formanyoutputdistributions,maximizingtheloglikelihoodofourdataleadstolossfunctionsthatwe’veseenalready.

Thelossfunctionforannormaloutputdistributionwithameanandvariance,isamodi?iedsquarederror.Wecansetthevariancelargertoreducetheimpactofthesquarederrors,butwepayapenaltyofthelogarithmofsigma.Ifweknowwearegoingtogettheoutputvalueforinstanceiexactlyright,thenwewillgetalmostnosquarederrorandwecansetthevarianceverysmall,payinglittlepenalty.Ifwewe’relesssure,thenweexpectasizablesquarederrorandweshouldincreasethevariancetoreducetheimpactalittle.

mixture density networks

9

<latexit sha1_base64="1xMfvi+K7RtUooBB3m5QI4uNvLI=">AAAHZXicfVXbbtNAEHULNCVQaAHxwgMWERJCUWWn9MJDpdKktAJ6oWqSSk1UrTeTxMr6ot11EnflX+AVfo0v4DdYOxfZWcO+eDTnnPHM2dWu5RObccP4vbR87/6DlcLqw+Kjx2tPnq5vPGswL6AY6tgjHr22EANiu1DnNidw7VNAjkWgaQ2qMd4cAmW2517x0Ie2g3qu3bUx4nGqNS4Wb9dLxqaRLF0NzGlQ0qbr4nZj5Wur4+HAAZdjghi7MQ2ftwWi3MYEomIrYOAjPEA9uAl4d68tbNcPOLg40t9KrBsQnXt63I/esSlgTkIZIExtWUHHfUQR5rLrYrYUAxc5wMqdoe2zSciGvUnAkRy5LcaJJdFaRil6FPl9G48zrQnkMAfxvpJkoWNlkxAQoEMnm4zblE0uMMdAsc1iEy6kM+d+bDO78i6meD/0++CySASURGmhBIBS6EphEjLggS+SaeTeDtg+pwGU4zDJ7dcQHVxCpyzrZBLZdrrEQzybsuQY0h0XRthzHOR2RMuPRIvDmItWeTNKvEujl5EQrdgoy9IvYziDnqXQsyjKgkcp8EiCWbQ+R7t6fVHaSIEN5a/NFNpclFpBCg0UdJhCh0pla5SCRwo8TqFjBQ1TaKigdyn0TvUZyWNxU2mLyV4kmyrOiT2EYwrgRqJUiRZnoXK/b8ysJD4DomRGid0d6MqLYQI4YUwXJ1en3yJR3atsGzvRIsMiAcwoxtbOdtVQKL1JN1OOsbdXOVQ4HkVub16odrTzyVQL+QH1yZy0u7v1+aNaKQRCvNG8UvWwVtlaPEcUKyZMZ9VLpq6Y1sujT6fKFVh5golTufyByj+mKPwH28urPjMwV+HnKWZu5irCPMXM2pliYQg/Pq0DLH8X3+uITCg1kDc+hVN5is/lLYW4R9/Lo0t7ji3tk99WOY7+R0TjGVFGyfNjLj42atCobJo7m8b3D6WDL9OHaFV7pb3R3mmmtqsdaCfahVbXsNbXfmg/tV8rfwprhReFlxPq8tJU81zLrMLrv8TYrL4=</latexit>x

<latexit sha1_base64="wcXGcPIzBGpY3wBCiKM0F1a+Ef0=">AAAH03icfVVdb9NIFHX5aNjwVZbHfbE2QkIoqmwH2u4DEkvbBfHVUpq2Uh1F48mNY2Vsj2bGSczILyte+XtI/BceGCdOZXsMlqJc33PO9dwzVzMeJQEXlvV949r1Gzc3W7f+aN++c/fe/a0Hf57xOGEY+jgmMbvwEAcSRNAXgSBwQRmg0CNw7k33c/x8BowHcXQqUgqDEPlRMA4wEio13KLzoW3Oh4769Uw3xtL1QtMNk0ylq+9O7V3RveL1U+CHKFfUM46W6Q23Ota2tXxMPbCLoGMUz/HwweZbdxTjJIRIYII4v7QtKgYSMRFgAlnbTThQhKfIh8tEjPcGMohoIiDCmflIYeOEmCI28+7NUcAAC5KqAGEWqAomniCGsFAetaulOEQoBN4dzQLKVyGf+atAIGXwQC6WG5DdrSilzxCdBHhRWZpEIQ+RmGhJnoZeNQkJATYLq8l8mWqRNeYCGA54bsKxcuaI5pvKT+PjAp+kdAIRz2TCSFYWKgAYg7ESLkMOIqFy2Y2apCl/LlgC3Txc5p4fIDY9gVFX1akkqssZkxiJaspTbSh3IpjjOAxRNJIuzaQrYCGk293Olt6V0ZNMSjc3yvPMkxyuoB9K6Icsq4KHJfBQgVW0f4WOzX5delYCz7SvnpfQ87rUS0pooqGzEjrTKnvzEjzX4EUJXWhoWkJTDf1cQj/rPiM1FpfOQK72Yrmp8ogEM3jFAKJMdpys3gtT+31pVyX5DMiOnS3tHsFYHUMrIExzunx9+v5dJvf3nGfWTlZneCSBNcXq7TzbtzSKv1pNwbH29pyXGidmKPKvCh0c7vxr64Vowii5Iu3u9v77R6+UAiHx/KrS/ssDp1efI4Y1E4pezY5taqb5TfSiq0aB1yRYOdXIn+r8Vwylv2DHTdXXBjYqaJNi7WajIm1SrK1dK2pN0Hxap+quoPm5jsiKcgDqxGfwXk3xkTqlkIjZEzW6zA8DZZ/6d7t59DsiWqyJKmq31fVj1y8bPThztu2dbevj086LN8VFdMv4y/jbeGzYxq7xwnhtHBt9AxvfjB8bNzc2W/2WbP3f+rKiXtsoNA+NytP6+hNJddQg</latexit>

w1w2w3µ1µ2µ3⌃1⌃2⌃3<latexit sha1_base64="wcXGcPIzBGpY3wBCiKM0F1a+Ef0=">AAAH03icfVVdb9NIFHX5aNjwVZbHfbE2QkIoqmwH2u4DEkvbBfHVUpq2Uh1F48mNY2Vsj2bGSczILyte+XtI/BceGCdOZXsMlqJc33PO9dwzVzMeJQEXlvV949r1Gzc3W7f+aN++c/fe/a0Hf57xOGEY+jgmMbvwEAcSRNAXgSBwQRmg0CNw7k33c/x8BowHcXQqUgqDEPlRMA4wEio13KLzoW3Oh4769Uw3xtL1QtMNk0ylq+9O7V3RveL1U+CHKFfUM46W6Q23Ota2tXxMPbCLoGMUz/HwweZbdxTjJIRIYII4v7QtKgYSMRFgAlnbTThQhKfIh8tEjPcGMohoIiDCmflIYeOEmCI28+7NUcAAC5KqAGEWqAomniCGsFAetaulOEQoBN4dzQLKVyGf+atAIGXwQC6WG5DdrSilzxCdBHhRWZpEIQ+RmGhJnoZeNQkJATYLq8l8mWqRNeYCGA54bsKxcuaI5pvKT+PjAp+kdAIRz2TCSFYWKgAYg7ESLkMOIqFy2Y2apCl/LlgC3Txc5p4fIDY9gVFX1akkqssZkxiJaspTbSh3IpjjOAxRNJIuzaQrYCGk293Olt6V0ZNMSjc3yvPMkxyuoB9K6Icsq4KHJfBQgVW0f4WOzX5delYCz7SvnpfQ87rUS0pooqGzEjrTKnvzEjzX4EUJXWhoWkJTDf1cQj/rPiM1FpfOQK72Yrmp8ogEM3jFAKJMdpys3gtT+31pVyX5DMiOnS3tHsFYHUMrIExzunx9+v5dJvf3nGfWTlZneCSBNcXq7TzbtzSKv1pNwbH29pyXGidmKPKvCh0c7vxr64Vowii5Iu3u9v77R6+UAiHx/KrS/ssDp1efI4Y1E4pezY5taqb5TfSiq0aB1yRYOdXIn+r8Vwylv2DHTdXXBjYqaJNi7WajIm1SrK1dK2pN0Hxap+quoPm5jsiKcgDqxGfwXk3xkTqlkIjZEzW6zA8DZZ/6d7t59DsiWqyJKmq31fVj1y8bPThztu2dbevj086LN8VFdMv4y/jbeGzYxq7xwnhtHBt9AxvfjB8bNzc2W/2WbP3f+rKiXtsoNA+NytP6+hNJddQg</latexit>


w1w2w3µ1µ2µ3⌃1⌃2⌃3

<latexit sha1_base64="wcXGcPIzBGpY3wBCiKM0F1a+Ef0=">AAAH03icfVVdb9NIFHX5aNjwVZbHfbE2QkIoqmwH2u4DEkvbBfHVUpq2Uh1F48mNY2Vsj2bGSczILyte+XtI/BceGCdOZXsMlqJc33PO9dwzVzMeJQEXlvV949r1Gzc3W7f+aN++c/fe/a0Hf57xOGEY+jgmMbvwEAcSRNAXgSBwQRmg0CNw7k33c/x8BowHcXQqUgqDEPlRMA4wEio13KLzoW3Oh4769Uw3xtL1QtMNk0ylq+9O7V3RveL1U+CHKFfUM46W6Q23Ota2tXxMPbCLoGMUz/HwweZbdxTjJIRIYII4v7QtKgYSMRFgAlnbTThQhKfIh8tEjPcGMohoIiDCmflIYeOEmCI28+7NUcAAC5KqAGEWqAomniCGsFAetaulOEQoBN4dzQLKVyGf+atAIGXwQC6WG5DdrSilzxCdBHhRWZpEIQ+RmGhJnoZeNQkJATYLq8l8mWqRNeYCGA54bsKxcuaI5pvKT+PjAp+kdAIRz2TCSFYWKgAYg7ESLkMOIqFy2Y2apCl/LlgC3Txc5p4fIDY9gVFX1akkqssZkxiJaspTbSh3IpjjOAxRNJIuzaQrYCGk293Olt6V0ZNMSjc3yvPMkxyuoB9K6Icsq4KHJfBQgVW0f4WOzX5delYCz7SvnpfQ87rUS0pooqGzEjrTKnvzEjzX4EUJXWhoWkJTDf1cQj/rPiM1FpfOQK72Yrmp8ogEM3jFAKJMdpys3gtT+31pVyX5DMiOnS3tHsFYHUMrIExzunx9+v5dJvf3nGfWTlZneCSBNcXq7TzbtzSKv1pNwbH29pyXGidmKPKvCh0c7vxr64Vowii5Iu3u9v77R6+UAiHx/KrS/ssDp1efI4Y1E4pezY5taqb5TfSiq0aB1yRYOdXIn+r8Vwylv2DHTdXXBjYqaJNi7WajIm1SrK1dK2pN0Hxap+quoPm5jsiKcgDqxGfwXk3xkTqlkIjZEzW6zA8DZZ/6d7t59DsiWqyJKmq31fVj1y8bPThztu2dbevj086LN8VFdMv4y/jbeGzYxq7xwnhtHBt9AxvfjB8bNzc2W/2WbP3f+rKiXtsoNA+NytP6+hNJddQg</latexit>






w1w2w3µ1µ2µ3⌃1⌃2⌃3

exp activationlinear activationsoftmax

argmax{wi},{µi},{⌃i}

lnX

i

wiN(x | µi,⌃i)<latexit sha1_base64="R+r+uFM4XDwwbxdd0Q33M+W5fzI=">AAAHwnicfVVbb9s2FJa7rW7ddUvbx74INQp0g2HIzpqkDwXaJl2LYU2yLE4KRIZB0cc2YeoykrKlsvyhe94fGXVxIIna9KKD813A85EgvYgSLhzn786db7797m733v3eg+8f/vDj3qPHVzyMGYYJDmnIPnuIAyUBTAQRFD5HDJDvUbj21scZfr0BxkkYXIo0gqmPlgFZEIyEbs32lIvY0kfJTEpXbmfEVQPblW6Ipev50vVjNSOqbHpl80+iFUVfKdulge3y2J8RW+vt0xeJ7fpkbjc8tIOh/2m213eGTv7ZZjEqi75VfuezR3eH7jzEsQ+BwBRxfjNyIjGViAmCKaieG3OIEF6jJdzEYnE0lSSIYgEBVvZzjS1iaovQzqKw54QBFjTVBcKMaAcbrxBDWOjAenUrDgHygQ/mGxLxouSbZVEIpNOeyiTfDfWwppRLhqIVwUltaRL53EdiZTR56nv1JsQU2MavN7Nl6kU2mAkwTHgWwrlO5izKdphfhuclvkqjFQRcyZhRVRVqABiDhRbmJQcRRzKfRh+rNX8tWAyDrMx7r08QW1/AfKB9ao36chY0RKLe8vQYOp0Atjj0fRTMpRsp6QpIhHQHQ5VnV0UvlD6WWVCeZ19kcA09raCnSjW0k1t0YU80WgOvKuCVYXxdQa+bUi+uoLGBbiroxnD2thV4a8BJBU0MNK2gqYF+qaBfzCiR3vmb8VQWcef7Js8o2cAHBhAo2R+r5ixMb+nNqC7Jtln2RyqPew4Lfe0UgJ9mdPnx8tPvSh4fjV86B6rJ8GgMO4qzf/Dy2DEoy2I1Jcc5Ohq/MzghQ8Hy1ujk/cHbkWkUxSyit6TDw/1fX5lOKVAabm+djt+djPeb54hhI4RyVrs/so3Qlm30cqpWgdcmKJJq5a9N/geG0v9gh23uuwBbFVGbYpdmqyJtU+yi3SkaQ0TZaV3rByHKrm5EC8oJ6EudwSd9is/0RYREyH6W+QtFdHz67w6y6v+IKNkRddXr6Rdm1HxPzGIyHr4aOn/80n/zW/nU3LOeWs+sF9bIOrTeWB+tc2tiYeufzv3O486T7vvuuvtXlxfUO51S88Sqfd2v/wJ20tCX</latexit><latexit sha1_base64="R+r+uFM4XDwwbxdd0Q33M+W5fzI=">AAAHwnicfVVbb9s2FJa7rW7ddUvbx74INQp0g2HIzpqkDwXaJl2LYU2yLE4KRIZB0cc2YeoykrKlsvyhe94fGXVxIIna9KKD813A85EgvYgSLhzn786db7797m733v3eg+8f/vDj3qPHVzyMGYYJDmnIPnuIAyUBTAQRFD5HDJDvUbj21scZfr0BxkkYXIo0gqmPlgFZEIyEbs32lIvY0kfJTEpXbmfEVQPblW6Ipev50vVjNSOqbHpl80+iFUVfKdulge3y2J8RW+vt0xeJ7fpkbjc8tIOh/2m213eGTv7ZZjEqi75VfuezR3eH7jzEsQ+BwBRxfjNyIjGViAmCKaieG3OIEF6jJdzEYnE0lSSIYgEBVvZzjS1iaovQzqKw54QBFjTVBcKMaAcbrxBDWOjAenUrDgHygQ/mGxLxouSbZVEIpNOeyiTfDfWwppRLhqIVwUltaRL53EdiZTR56nv1JsQU2MavN7Nl6kU2mAkwTHgWwrlO5izKdphfhuclvkqjFQRcyZhRVRVqABiDhRbmJQcRRzKfRh+rNX8tWAyDrMx7r08QW1/AfKB9ao36chY0RKLe8vQYOp0Atjj0fRTMpRsp6QpIhHQHQ5VnV0UvlD6WWVCeZ19kcA09raCnSjW0k1t0YU80WgOvKuCVYXxdQa+bUi+uoLGBbiroxnD2thV4a8BJBU0MNK2gqYF+qaBfzCiR3vmb8VQWcef7Js8o2cAHBhAo2R+r5ixMb+nNqC7Jtln2RyqPew4Lfe0UgJ9mdPnx8tPvSh4fjV86B6rJ8GgMO4qzf/Dy2DEoy2I1Jcc5Ohq/MzghQ8Hy1ujk/cHbkWkUxSyit6TDw/1fX5lOKVAabm+djt+djPeb54hhI4RyVrs/so3Qlm30cqpWgdcmKJJq5a9N/geG0v9gh23uuwBbFVGbYpdmqyJtU+yi3SkaQ0TZaV3rByHKrm5EC8oJ6EudwSd9is/0RYREyH6W+QtFdHz67w6y6v+IKNkRddXr6Rdm1HxPzGIyHr4aOn/80n/zW/nU3LOeWs+sF9bIOrTeWB+tc2tiYeufzv3O486T7vvuuvtXlxfUO51S88Sqfd2v/wJ20tCX</latexit><latexit sha1_base64="R+r+uFM4XDwwbxdd0Q33M+W5fzI=">AAAHwnicfVVbb9s2FJa7rW7ddUvbx74INQp0g2HIzpqkDwXaJl2LYU2yLE4KRIZB0cc2YeoykrKlsvyhe94fGXVxIIna9KKD813A85EgvYgSLhzn786db7797m733v3eg+8f/vDj3qPHVzyMGYYJDmnIPnuIAyUBTAQRFD5HDJDvUbj21scZfr0BxkkYXIo0gqmPlgFZEIyEbs32lIvY0kfJTEpXbmfEVQPblW6Ipev50vVjNSOqbHpl80+iFUVfKdulge3y2J8RW+vt0xeJ7fpkbjc8tIOh/2m213eGTv7ZZjEqi75VfuezR3eH7jzEsQ+BwBRxfjNyIjGViAmCKaieG3OIEF6jJdzEYnE0lSSIYgEBVvZzjS1iaovQzqKw54QBFjTVBcKMaAcbrxBDWOjAenUrDgHygQ/mGxLxouSbZVEIpNOeyiTfDfWwppRLhqIVwUltaRL53EdiZTR56nv1JsQU2MavN7Nl6kU2mAkwTHgWwrlO5izKdphfhuclvkqjFQRcyZhRVRVqABiDhRbmJQcRRzKfRh+rNX8tWAyDrMx7r08QW1/AfKB9ao36chY0RKLe8vQYOp0Atjj0fRTMpRsp6QpIhHQHQ5VnV0UvlD6WWVCeZ19kcA09raCnSjW0k1t0YU80WgOvKuCVYXxdQa+bUi+uoLGBbiroxnD2thV4a8BJBU0MNK2gqYF+qaBfzCiR3vmb8VQWcef7Js8o2cAHBhAo2R+r5ixMb+nNqC7Jtln2RyqPew4Lfe0UgJ9mdPnx8tPvSh4fjV86B6rJ8GgMO4qzf/Dy2DEoy2I1Jcc5Ohq/MzghQ8Hy1ujk/cHbkWkUxSyit6TDw/1fX5lOKVAabm+djt+djPeb54hhI4RyVrs/so3Qlm30cqpWgdcmKJJq5a9N/geG0v9gh23uuwBbFVGbYpdmqyJtU+yi3SkaQ0TZaV3rByHKrm5EC8oJ6EudwSd9is/0RYREyH6W+QtFdHz67w6y6v+IKNkRddXr6Rdm1HxPzGIyHr4aOn/80n/zW/nU3LOeWs+sF9bIOrTeWB+tc2tiYeufzv3O486T7vvuuvtXlxfUO51S88Sqfd2v/wJ20tCX</latexit>

Ifwewanttogoallout,wecanevenmakeourneuralnetworkoutputtheparametersofaGaussianmixture.Thisiscalledamixturedensitynetwork.

Allweneedtodoismakesurethatwehaveoneoutputnodeforeveryparameterrequired,andapplydifferentactivationstothedifferentkindsofparameters.Themeansgetalinearactivationandthecovariancesgetanexponentialactivationasbefore.Theweightsneedtosumtoone,soweneedtogivethemasoftmaxactivation(overjustthesethreeweights).

Ifwewanttotrainwithmaximumlikelihood,weencounterthissum-inside-a-logarithmfunctionagain,whichisdif?iculttodealwith.Butthistime,it’snotsuchaheadache.Aswenotedinthelastlecturewecanworkoutthegradientforthisloss,it’sjustaveryuglyfunction.Sinceweareusingbackpropagationanyway,thatneedn’tworryushere.Allweneedtoworkoutadthelocalderivativesforfunctionslikethelogarithmandthesum,andthoseareusuallyprovidedbysystemslikePytorchandKerasanyway.


10

L1

L2

θ1

θ2

(x1 , x2 ) (x1 , x2 )

elbowup

elbowdown

0.0

0.5

1.0

0.0 0.5 1.0

A

B

C

x1

x2elbowdown

elbowup

image source: Mixture Density Networks, Christopher Bishop, 1994

Themixturedensitynetworkmayseemlikeoverkill,butit’sactuallyveryusefulincaseswherewehavemultiplevalidanswers.

Considertheproblemofinversekinematicsinrobotics.Wehavearobotarmwithtwojoints,andweknowwhereinspacewewantthehandofthearmtobe.Whatanglesshouldwesetthetwojointsto?Thisisagreatapplicationformachinelearning:it’sarelativelysimple,smoothfunction.It’seasytogenerateexamples,andexplicitsolutionsareapaintowrite,andnotrobusttonoiseinthecontroloftherobot.Sowecansolveitwithaneuralnet.

Theinputsarethetwocoordinateswherewewantthehandtobe(x1,x2),andtheoutputsarethetwoanglesofthejoints(θ1,θ2).Theproblemweruninto,isthatforeveryinput,therearetwosolutions.Onewiththeelbowup,andonewiththeelbowdown.AnormalneuralnetworktrainedwithanMSElosswouldnotpickoneortheother,butitwouldaveragebetweenthetwo.

Amixturedensitynetworkwithtwocomponentscan?ixthisproblem.Foreachinput,itcansimplyputitscomponentsonthetwooutputs

0.0 0.5 1.00.0

0.5

1.0

x

t

0.0 0.5 1.00.0

0.5

1.0

x

t


11

0.0 0.5 1.00.0

0.5

1.0

x

t

0.0 0.5 1.00.0

0.5

1.0

x

t

0.0 0.5 1.0

0.5

1.0

0.0x

t

mode collapse

image source: Mixture Density Networks, Christopher Bishop, 1994

Theproblemwiththerobotarmisthatthetaskisuniquelydeterminedinonedirection—everycon?igurationoftherobotarmsleadstoauniquepointinspaceforthehand—butnotwhenwetrytoreasonbackwardfromthehandtothecon?igurationofthejoints.

Thisoftenhappenswhenyoutakeatrivialregressionproblemand?liparoundtheinputsandtheoutputs.Ontheleft,weseeaverysimpleregressionproblem.foreveryinputx,thereisauniqueoutputt,ifwithalittlenoise.Asmallnetworkeasily?indsanice?it.If,however,we?liparoundthevaluesoftandx,theproblembecomesreallydif?icult.Foraninputlike0.4,therearetwodistinctregionsthatbothhaveahighdensityofpoints.

Thenetworkcanonlypredictoneoutputvalue,soitendsupaveragingbetweenthetwo,puttingtheoutputinapartofspacewheretherearenodataatall.Ifwearetrainingwithmeansquarederrorloss,wecanthisoftheoutputas?ittinghemeanofanormaldistributiontotheoutput.Wecangivetheneuralnetworkcontroloverthevarianceofthisdistributionaswell,butallthatachievesisthatthevariancegrowstocoverbothgroupsofpoints.Themeanstaysinthesameplace.

Bycontrast,themixturedensitynetworkcanoutputadistributionwithtwopeaks,knownasitsmodes.Thisallowsittocoverthetwogroupsofpointsinthe

output,andsosolvetheprobleminamuchmoreusefulway

Thegeneralprobleminthemiddlepictureiscalledmodecollapse.

mode collapse

12

Ifourdataisspreadoutinspaceinacomplex,clusteredpattern,and?itasimpleunimodeldistributiontoit,thatis,adistributionwithonepeak.Theresultisadistributionthatputsalltheprobabilitymassontheaverageofourpoints,butverylittleprobabilitymasswherethepointsactuallyare.

13

Mixturedensitynetworksgosomewaytowardslettinguscapturemorecomplexdistributionsinourneuralnetworks,butwhenwewanttocapturesomethingascomplexasthedistributiononimagesrepresentinghumanfaces,they’restillinsuf?icient.

Amixturemodelwithkcomponentsgivesuskmodes.Sointhespaceofimages,wecanpickkimagestogivehighprobabilityandtherestisjustasimplegaussianshapearoundthosekpoints.Thedistributiononhumanfaceshasin?initelymanymodesinthisspacethatshouldallbeequallylikely.Toachieveadistributionthiscomplex,weneedtousethepoweroftheneuralnet,notjusttochoosea?initesetofmodes,buttocontrolthewholeshapeoftheprobabilityfunction.


option 2:

14

generator network

Here’samorepowerfulapproach:weputthenormaldistributionatthestartofthenetworkinsteadofattheendofit.Wesampleapointfromsomestraightforwarddistribution,usuallyastandardnormaldistribution,andwefeedthatpointtoaneuralnet.Theresultofthesetwostepsisarandompoint,sowe’vede?inedanotherprobabilitydistribution.Wecallthisconstructionageneratornetwork.

Comparethistohowwede?inedparametrizedmultivariatenormalsinthepreviouslecture:westartedwithastandardnormaldistribution,andweappliedalineartransformation.Thisisthesamething,butwe’vereplacedthelineartransformationbyanonlinearone.

a small experiment

15

100 nodes

2 nodes

2 nodes

Toseewhatkindofdistributionswemightgetwhenwedothis,let’stryalittleexperiment.

Wewireuparandomnetworkasshown:atwo-nodeinputlayer,followedby12,100-nodefully-connectedhiddenlayerswithReLUactivations.,anda?inaltransformationbacktotwopoints.Wedon’ttrainthenetwork.WejustuseGlorotinitialisationtopicktheparameters,andthensamplesomepoints.Sincetheoutputis2D,wecaneasilyscatter-plotit.

16

Here’saplotof100kpointssampledinthisway.Clearly,we’vede?ineahighlycomplexdistribution.Insteadofhavinga?initesetofsinglepointsasmodes,wegetstrandsofhighprobabilityinspace,andsheetsoflower,butnonzeroprobability.Remember,thisnetworkhasn’tbeentrained,soit’snotrepresentinganythingmeaningful,butitshowsthatthekindsofdistributionswecanrepresentinthiswayisahighlycomplexfamily.

NB:Notethatthevariancehasshrunk,andthemeanhasdriftedawayfrom(0,0);apparentlytheweightinitialisationisnotquiteperfect.

17

fully connected

upsample

convolution

upsample

convolution

upsample

Wecanalsousethistricktogenerateimages.Anormalconvolutionalnetstartswithalow-channel,highresolutionimage,andslowlydecreasestheresolutionbymaxpooling,whileincreasingthenumberofchannels.Here,wereversetheprocess.Weshapeourinputintoalowresolutionimagewithalargenumberofchannels.Weslowlyincreasetheresolutionbyupsamplinglayers,anddeacreasethenumberofchannels.

Wecanuseregularconvolutionlayers,ordeconvolutions,whichareconvolutionswhereonepatchintheoutputiswiredtoasinglenodeintheinput.bothapproachesgiveuseffectivegeneratornetworksforimages.

NB:I’veenhancedthecontrastandsaturationtoensurethatthecolorsshowuponaprojector.


option 3: both

18



<- latent space

z

Ofcourse,wecanalsodoboth:wesampletheinputfromastandardMVN,interprettheoutputasanotherMVN,andthensamplefromthat.

Inthesekindsofgeneratornetworks,theinputisoftencalledz,andthespaceofinputsisoftencalledthelatentspace.

training a generator: the naive approach

loop:

Generate a random output y

Sample a random instance x

Compute loss(x, y) and backpropagate

loss: mean-squared error, binary cross-entropy, L1, etc.

19

Sothebigquestionis,howdowetrainageneratornetwork?Givensomedata,howdowesettheweightsofthenetworksothattheoutputstartstolooklikeourexamples?

We’llstartwithwhatdoesn’twork.Hereisanaiveapproach:wesimplysamplearandompoint(e.g.apicture)fromthedata,andsampleapointfromthemodelandtrainonhowclosetheyare.

Thelosscouldbeanydistancefunctionbetweentwotensors.Themean-squarederrorovertheelementsisasimpleapproach.Iftheelementsarevaluesbetween0and1(likeinimages),thebinarycross-entropymakessensetoo.Forimagestheabsolutevalueoftheerror(alsocalledL1loss)isalsopopular.

mode collapse

20

Ifweimplementthenaiveapproach,wegetsomethingcalledmodecollapse.Hereisanexample:thebluepointsrepresentthemodes(likelypoints)ofthedata.Thegreenpointisgeneratedbythemodel.It’sclosetooneofthebluepoints,sothemodelshouldberewarded,butit’salsofarawayfromsomeoftheotherpoints.Duringtraining,there’snoguaranteethatwepairitupwiththecorrectpoint,andwearelikelytocomputethelosstoacompletelydifferentpoint.

Onaveragethemodelispunishedforgeneratingsuchpointsasmuchasitisrewarded.Themodelthatgeneratesonlytheopenpointinthemiddlegetsthesameloss(andlessvariance).Underbackpropagation,neuralnetworkstendtoconvergetoadistributionthatgeneratesonlytheopenpointoverandoveragain.

Inotherwords,themanydifferentmodes(areasofhighprobability)ofthedatadistributionendupbeingaveraged(“collapsing”)intoasinglepoint.

mode collapse

21

Sowe’rebacktomodecollapse.Eventhoughwehaveaprobabilitydistributionthatisabletorepresenthighlycomplex,multi-modaloutputs,ifwetrainitlikethis,westillendupproducingaunimodaloutputcenteredonthemeanofourdata.

Howdowegetthethenetworktoimaginedetails,insteadofaverageingoverallpossibilities?

training generator networks

Generative Adversarial Networks Train an adversary to tell fake data from real data.

Variational Autoencoders Train an encoder to tell us which data point a generated point should be compared to.

22

Therearetwomainapproaches,whichwe’lldiscussintherestofthelecture.

Deep generative models Part 2: Generative Adversarial Networks



Inthelastvideo,wede?inedgeneratornetworks,andwesawthattheyrepresentaveryrichfamilyofprobabilitydistributions.Wealsosaw,thattrainingthemcanbeatrickybusiness.

Inthisvideowe’lllookonewayoftrainingsuchnetworks:themethodofgenerativeadversarialnetworks.

adversarial examples (2014)

24

“this is definitely a bus”

bus

cat

chai

r

bus

cat

chai

r

<- optimize these values

keep the weights fixed

<-maximize

this value

bus

cat

chai

r

“this is definitely a bus”

GANsoriginatedjustafterConvNetswerebreakingnewground,showingspectacular,sometimessuper-human,performanceinimagelabeling.Thesuggestionaroethatconvnetsweredoingmoreorlessasthesameaswhathumansdowhentheylookatsomething.

Toverifythis,researchersdecidedtostartinvestigatingwhatkindofinputswouldmakeatrainedConvNetgiveacertainoutput.Thisiseasytodo,youjustcomputethegradientwithrespecttotheinputofthenetwork,andtraintheinputtomaximisetheactivationofaparticularlabel.Thisissimilartothefeaturevisualizingapproachwesawbefore,butyoudoitontheoutputnodesinsteadofthehiddennodes.

Youwouldexpectthatifyoustartwitharandomimage,andfollowthegradienttomaximizetheactivationoftheoutputnodecorrespondingtothelabel“bus”,you’dgetapictureofabus.Oratleastsomethingthatlooksalittlebitlikeabus.Whatyouactuallygetissomethingthatisindistinguishablefromthenoiseyoustartedwith.Onlytheverytiniestofchangesisrequiretomakethenetworkseeabus.

Thesearecalledadversarialexamples.instancesthatarespeci?icallycraftedtotripupagivenmodel.

adversarial examples (2014)

25source: http://karpathy.github.io/2015/03/30/breaking-convnets/

Theresearchersalsofoundthatiftheystartedthesearchnotatarandomimage,butatanimageofanotherclass,allthatwasneededtoturnitintoanotherclasswasaverysmalldistortion.Sosmall,thattoustheimagelookedunchanged.Inshort,atinybitofdistortionisenoughtomakeaCNNthinkthatapictureofabusisapictureofanostrich.

Adversarialexamplesareanactiveareaofresearch(bothhowtogeneratethemandmakemodelsmorerobustagainstthem).

26

source: Eykholt, Kevin, et al. "Robust physical-world attacks on deep learning visual classification." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

Evenmanipulatingobjectsinthephysicalworldcanhavethiseffect.Astopsigncanbemadetolooklikeadifferenttraf?icsignbythesimpleadditionofsomestickers.

generative adversarial networks

loop:

train a classifier to tell XPos from XNeg

Generate adversarial examples Clearly not Pos, but the classifier thinks so anyway

Add the adversarial examples to XNeg

The classifier (discriminator) gets more robust, the generator gets more realistic.

27

Prettysoon,thisbadnewswasturnedintogoodnewsbyrealisingthatifyoucangenerateadversarialexamplesautomatically,youcanalsoaddthemtothedatasetasnegativesandretrainyournetworktomakeitmorerobust.Youcansimplytellyournetworkthatthesethingsarenotstopsigns,andtolearnagain.

Wecanthinkofthisasakindofiterated2playergame(oranarmsrace).TheGenerator(theprocessthatgeneratestheadversarialexamples)triestogetgoodenoughtofooltheclassi?ierandtheclassi?iertriestogetgoodenoughtotellthefakesfromthetrueexamples.

Thisisthebasicideaofthegenerativeadversarial

GANs

• Vanilla GANs

• Conditional GANs

• CycleGAN

• StyleGAN

28

We’lllookatfourdifferentexamplesofGANs.We’llcallthebasicapproachthe“vanillaGAN”

end-to-end GANs

29

generator: G discriminator: D

Generatingadversarialexamplesbygradientdescentispossible,butit’smuchnicerifbothourgeneratorandourdiscriminatorareseparateneuralnetworks.ThiswillleadtoamuchcleanerapproachfortrainingGANs.

Wewilldrawthetwocomponentslikethis.ThegeneratortakesaninputsampledfromastandardMVNandproducesanimage.Thisisageneratornetworkaredescribedinthepreviousvideo(withnoouptudistribution).Thediscriminatortakesanimageandclassi?iesitasPos(arealimage)orNeg(afakeimagesampledfromthegenerator).

Ifwehaveotherimagesthatarenotofthetargetclass,wecanaddthosetothenegativeexamplesaswell,but

often,thepositiveclassisjustourdataset(likeacollectionofhumanfaces),andthenegativeclassisjustthefakeimagescreatedbythegenerator.

training the discriminator

30

real examplePos

D

fake example

Neg

update

freeze

D

G

target:

Totrainthediscriminator,wefeeditexamplesfromthepositiveclass,andtrainittoclassifytheseasPos.

Wealsosampleimagesfromthegenerator(whoseweightswekeep?ixed)andtrainthediscriminatortomakethesenegative.At?irstthesewilljustberandomnoise,butthere’slittleharmintellingournetworkthatsuchimagesarenotbusses(orwhateverourpositiveclassis).

Notethatsincethegeneratorisaneuralnetwork,wedon’tneedtocollectadatasetoffakeimages.Wecanjuststickthediscriminatorontopofthegenerator,andtrainit,bygradientdescenttoclassifytheresultarenegative.Wejustneedtomakesuretofreezethe

training the generator

31

Pos

update

freezeD

G

Then,totrainthegenerator,wefreezethediscriminatorandtraintheweightsofthegeneratortoproduceimagesthatcausethediscriminatortolabelthemasPositive.

Wedon’tneedtowaitforeithersteptoconverge.Wecanjusttrainathediscriminatorforonebatch(i.e.onestepofgradientdescent)andthentrainthegeneratorforonebatch,andsoon.

Andthisiswhatwe’llcallthevanillaGAN.

conditional GAN (2016)

32

Sometimeswewanttotrainthenetworktotakeaninput,buttogeneratetheoutputprobabilistically.Forinstance,whenwetrainanetworktocolorinablack-and-whitephotographofa?lower,itcouldchoosemanycolorsforthe?lower.Wewanttoavoidmodecollapsehere:insteadofaveragingoverallpossiblecolors,givingusabrownorgray?lower,wewantittopickonecolor,fromallthepossibilities.

AconditionalGANletsustraingeneratornetworksthatcandothis.

generator is now a function

33

goes into designing effective losses. In other words, we stillhave to tell the CNN what we wish it to minimize. But, justlike King Midas, we must be careful what we wish for! Ifwe take a naive approach and ask the CNN to minimize theEuclidean distance between predicted and ground truth pix-els, it will tend to produce blurry results [43, 62]. This isbecause Euclidean distance is minimized by averaging allplausible outputs, which causes blurring. Coming up withloss functions that force the CNN to do what we really want– e.g., output sharp, realistic images – is an open problemand generally requires expert knowledge.

It would be highly desirable if we could instead specifyonly a high-level goal, like “make the output indistinguish-able from reality”, and then automatically learn a loss func-tion appropriate for satisfying this goal. Fortunately, this isexactly what is done by the recently proposed GenerativeAdversarial Networks (GANs) [24, 13, 44, 52, 63]. GANslearn a loss that tries to classify if the output image is realor fake, while simultaneously training a generative modelto minimize this loss. Blurry images will not be toleratedsince they look obviously fake. Because GANs learn a lossthat adapts to the data, they can be applied to a multitude oftasks that traditionally would require very different kinds ofloss functions.

In this paper, we explore GANs in the conditional set-ting. Just as GANs learn a generative model of data, condi-tional GANs (cGANs) learn a conditional generative model[24]. This makes cGANs suitable for image-to-image trans-lation tasks, where we condition on an input image and gen-erate a corresponding output image.

GANs have been vigorously studied in the last twoyears and many of the techniques we explore in this pa-per have been previously proposed. Nonetheless, ear-lier papers have focused on specific applications, andit has remained unclear how effective image-conditionalGANs can be as a general-purpose solution for image-to-image translation. Our primary contribution is to demon-strate that on a wide variety of problems, conditionalGANs produce reasonable results. Our second contri-bution is to present a simple framework sufficient toachieve good results, and to analyze the effects of sev-eral important architectural choices. Code is available athttps://github.com/phillipi/pix2pix.

2. Related workStructured losses for image modeling Image-to-image

translation problems are often formulated as per-pixel clas-sification or regression (e.g., [39, 58, 28, 35, 62]). Theseformulations treat the output space as “unstructured” in thesense that each output pixel is considered conditionally in-dependent from all others given the input image. Condi-tional GANs instead learn a structured loss. Structuredlosses penalize the joint configuration of the output. A

fake

G(x)

x

D

real

D

Gx y

x

Figure 2: Training a conditional GAN to map edges!photo. Thediscriminator, D, learns to classify between fake (synthesized bythe generator) and real {edge, photo} tuples. The generator, G,learns to fool the discriminator. Unlike an unconditional GAN,both the generator and discriminator observe the input edge map.

large body of literature has considered losses of this kind,with methods including conditional random fields [10], theSSIM metric [56], feature matching [15], nonparametriclosses [37], the convolutional pseudo-prior [57], and lossesbased on matching covariance statistics [30]. The condi-tional GAN is different in that the loss is learned, and can, intheory, penalize any possible structure that differs betweenoutput and target.

Conditional GANs We are not the first to apply GANsin the conditional setting. Prior and concurrent works haveconditioned GANs on discrete labels [41, 23, 13], text [46],and, indeed, images. The image-conditional models havetackled image prediction from a normal map [55], futureframe prediction [40], product photo generation [59], andimage generation from sparse annotations [31, 48] (c.f. [47]for an autoregressive approach to the same problem). Sev-eral other papers have also used GANs for image-to-imagemappings, but only applied the GAN unconditionally, re-lying on other terms (such as L2 regression) to force theoutput to be conditioned on the input. These papers haveachieved impressive results on inpainting [43], future stateprediction [64], image manipulation guided by user con-straints [65], style transfer [38], and superresolution [36].Each of the methods was tailored for a specific applica-tion. Our framework differs in that nothing is application-specific. This makes our setup considerably simpler thanmost others.

Our method also differs from the prior works in severalarchitectural choices for the generator and discriminator.Unlike past work, for our generator we use a “U-Net”-basedarchitecture [50], and for our discriminator we use a convo-lutional “PatchGAN” classifier, which only penalizes struc-ture at the scale of image patches. A similar PatchGAN ar-chitecture was previously proposed in [38] to capture localstyle statistics. Here we show that this approach is effectiveon a wider range of problems, and we investigate the effectof changing the patch size.

3. MethodGANs are generative models that learn a mapping from

random noise vector z to output image y, G : z ! y [24]. In

source: Image-to-Image Translation with Conditional Adversarial Networks (2016), Phillip Isola Jun-Yan Zhu et al.







fake

G(x)

x

D

real

D

Gx y

x







G

zx

G(x)

InaconditionalGAN,thegeneratorisafunctionwithanimageinput,whichitmapsittoanimageoutput.However,itusesrandomnesstoimaginespeci?icdetailsintheoutput.runningthisgeneratortwicewouldresultindifferentshoesthatareboth“correct”instantiationsoftheinputlinedrawing.

cGAN

34source: Image-to-Image Translation with Conditional Adversarial Networks (2016), Phillip Isola Jun-Yan Zhu et al.







fake

G(x)

x

D

real

D

Gx y

x













fake

G(x)

x

D

real

D

Gx y

x







Thediscriminatortakespairsofinputsandoutputs.Ifthesecomefromthegenerator,theyshouldbeclassi?iedasfake(negative)andiftheycomefromthedata,theyshouldbeclassi?iedasreal(positive).

Thegeneratoristrainedintwoways.

1. Wefreezetheweightsofthediscriminator,asbefore,andtrainthegeneratortoproducethinsthatthediscriminatorwillthinkarereal.

2. Wefeeditandinputfromthedata,andbackpropagateonthecorrespondingoutput(usingL1loss).

for unpaired images

35source: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (2017) Zhu et al

TheconditionalGANworksreallywell,butonlyifwehaveanexampleofaspeci?icoutputfortheinput.Forsometasks,wedon’thavepairedimages.Weonlyhaveunmatchedbagsofimagesintwodomains.

Ifwerandomlymatchoneimageinonedomaintoanimageinanotherdomain,wegetmodecollapseagain.

CycleGAN (2017)

Transformations are always learned both ways. Ingredients:

• Horse discriminator

• Zebra discriminator

• Horse-to-zebra generator (G)

• Zebra-to-horse generator (F)

36

CycleGANsachievethisbyaddinga“cycleconsistencyterm”tothelossfunction.Forinstance,inthehorse-to-zebraexample,ifwetransformahorsetoazebraandbackagain,theresultshouldbeclosetotheoriginalimage.Thus,theobjectivebecomestotrainahorse-to-zebratransformerandazebra-to-horsetransformertogetherinsuchawaythat:

• ahorse-discriminatorcan’ttellthegeneratedhorsesfromrealones(andlikewiseforazebradiscriminator)

• thecycleconsistencylossforbothcombinedislow.

Aftersometrainingofthegenerators,thediscriminatorsareretrainedwithwiththeoriginal

dataandsomegeneratedhorsesandzebras.

CycleGAN (2017)

Think of the generators as practicing steganography. I.e. hiding a horse picture inside a zebra picture.

37source: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (2017) Zhu et al

CycleGANsachievethisbyaddinga“cycleconsistencyterm”tothelossfunction.Forinstance,inthehorse-to-zebraexample,ifwetransformahorsetoazebraandbackagain,theresultshouldbeclosetotheoriginalimage.Thus,theobjectivebecomestotrainahorse-to-zebratransformerandazebra-to-horsetransformertogetherinsuchawaythat:

• ahorse-discriminatorcan’ttellthegeneratedhorsesfromrealones(andlikewiseforazebradiscriminator)

• thecycleconsistencylossforbothcombinedislow.

Aftersometrainingofthegenerators,thediscriminatorsareretrainedwithwiththeoriginaldataandsomegeneratedhorsesandzebras.

38

TheCycleGANworkssurprisinglywell.Here’showitmapsphotographstoimpressionistpaintingsandviceversa.

Itdoesn’talwaysworkperfectly,though.


StyleGAN (2018)

40

Az+ t<latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit><latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit><latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit><latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit>Az+ t

<latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit><latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit><latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit><latexit sha1_base64="lYKiJEcKA3jv+DiZnR7yWBIYPeM=">AAAH1XicfVXLbtNAFHV5BcKrhQULNhYREoKoslNoy6JSn7QL+qBqGqQmqsaTm8TK+KGZcRp3NDvElo9AYgUfxN8wdh7YHsNsfHXPOdf3nhl7nJC4jFvW74UbN2/dvlO5e696/8HDR48Xl56csyCiGJo4IAH97CAGxPWhyV1O4HNIAXkOgZYz3Enw1ggocwP/jMchdDzU992eixFXqcvFZ22KhWg7PXPrWppvzDTkUl4u1qxlK12mHtjToGZM18nl0p0f7W6AIw98jgli7MK2Qt4RiHIXE5DVdsQgRHiI+nAR8d56R7h+GHHwsTRfKqwXEZMHZtKk2XUpYE5iFSBMXVXBxANEEeZqlGq+FAMfecDq3ZEbsknIRv1JwJHyoSPGqU/yYU4p+hSFAxePc60J5DEP8YGWZLHn5JMQEaAjL59M2lRNFphjoNhliQknypnjMPGenQUnU3wQhwPwmRQRJTIrVABQCj0lTEMGPApFOo3a8CHb4DSCehKmuY1dRIen0K2rOrlEvp0eCRDPpxw1hnLHhysceB7yu6IdStHmMOaiXV+WqXdZ9FSqU5MY5TjmaQLn0KMMeiRlHtzLgHsKzKPNOdozm0XpeQY8197ayqCtotSJMmikoaMMOtIqO1cZ+EqDxxl0rKFxBo019DqDXus+I3UsLhodMdmLdFPFMXFHsE8BfClqDVmchar9vrDzkuQMiJotU7u70FN/iwngxQldHJwdfpRiZ73xzlqVRYZDIphRrJXVdzuWRulPuplyrPX1xrbGCSjy+/NCu3urW7ZeKIxoSOaktbWVD+/1SjEQElzNK+1s7zZWioMpR/JN2Wu2ZRVPG8WaVVNHzJptatb2y+jT15QKnDLBxM9S/lDn71MU/4MdlFWf2VyqCMsUM89LFXGZYrYBM0Ve4pfY9Hc75prC5GHyIQyx6jG5MhCZ1N0FdZlQOFQfyLH6ASIe0Nfqq6B9z1W11LNdT6L/EdF4RlRRtapuNrt4j+nBeWPZtpbtT29rm9vTO+6u8dx4YbwybGPN2DQOjBOjaWBDGt+Nn8avSqsiK18qXyfUGwtTzVMjtyrf/gAUHdfY</latexit>

latent vector per-layer noise

Finally,let’stakealookattheStyleGAN,thenetworkthatgeneratedthefaceswe?irstsawintheintroduction.ThisisbasicallyaVanillaGAN,withallmostofthespecialtricksinthewaythegeneratorisconstructed.Itusestoomanytrickstodiscusshereindetail,sowe’lljustfocusononeaspect:theideathatthelatentvectorisfedtothenetworkateachstageofitsforwardpass.

Sinceanimagegeneratorstartswithacoarse(lowresolution),highleveldescriptionofanimage,andslowly?illsinthedetails,feedingitthelatentvectorateverylayer(transformedbyanaf?inetransformationto?itittotheshapeofthedataatthatstage),allowsittousedifferentpartsofthelatentvectortodescribedifferentaspectsoftheimage(theauthorscallthese“styles”).

Thenetworkalsoreceivesseparateextrarandomnoiseperlayer,thatallowsittomakerandomchoices.Withoutthis,allrandomnesswouldhavetocomefromthelatentvector.

changing the latent vector

destination

sour

ce

Coa

rse

styl

esco

pied

Mid

dle

styl

esco

pied

Fine

styl

es

Figure 3. Visualizing the effect of styles in the generator by having the styles produced by one latent code (source) override a subset of thestyles of another one (destination). Overriding the styles of layers corresponding to coarse spatial resolutions (42 – 82), high-level aspectssuch as pose, general hair style, face shape, and eyeglasses get copied from the source, while all colors (eyes, hair, lighting) and finer facialfeatures of the destination are retained. If we instead copy the styles of middle layers (162 – 322), we inherit smaller scale facial features,hair style, eyes open/closed from the source, while the pose, general face shape, and eyeglasses from the destination are preserved. Finally,copying the styles corresponding to fine resolutions (642 – 10242) brings mainly the color scheme and microstructure from the source.

4


destination

sour

ce

destination

sour

ce

Coa

rse

styl

esco

pied

Mid

dle

styl

esco

pied

Fine

styl

es


4

source inserted for lower layers

destination

sour

ce

Coa

rse

styl

esco

pied

Mid

dle

styl

esco

pied

Fine

styl

es


4

source inserted for middle layers

destination

sour

ce

Coa

rse

styl

esco

pied

Mid

dle

styl

esco

pied

Fine

styl

es


4

source inserted for top layers

dest

inat

ion

sour

ce

destination

sour

ce

Coa

rse

styl

esco

pied

Mid

dle

styl

esco

pied

Fine

styl

es


4

destination

sour

ce

Coa

rse

styl

esco

pied

Mid

dle

styl

esco

pied

Fine

styl

es


4

destination

sour

ce

Coa

rse

styl

esco

pied

Mid

dle

styl

esco

pied

Fine

styl

es


4

Toseehowthisworks,wecantrytomanipulatethenetwork,bychangingthelatentvectortoanotherforsomeofthelayers.Inthisexampleallimagesonthemarginsarepeoplethataregeneratedforaparticularsinglelatentvector.

Wethenre-generatetheimageforthedestination,exceptthatforafewlayers(atthebottom,middleortop),weusethesourcelatentvectorinstead.

Aswesee,overridingthebottomlayerschangesthingslikegender,ageandhairlength,butnotethnicity.Forthemiddlelayer,theageislargelytakenfromthedestinationimage,buttheethnicityisnowoverridebythesource.Finallyforthetoplayers,onlysurface

detailsarechanged.

Thiskindofmanipulationwasdoneduringtrainingaswell,toensurethatitwouldleadtofacesthatfoolthediscriminator.

changing the noise

42source: A Style-Based Generator Architecture for Generative Adversarial Networks, Karras et al.

per-layer noise

Let’slookattheothersideofthenetwork:thenoiseinputs.

Ifwekeepallthelatentandnoiseinputsthesameexceptfortheverylastnoiseinput,wecanseewhatthenoiseachieves:theman’shairisequallymessyineachgeneratedexample,butexactlyinwhatwayit’smessychangespersample.Thenetworkusesthenoisetodeterminethepreciseorientationoftheindividual“hairs”.

GANs: not discussed

• Wasserstein distance

• Evaluation: Inception score, FID score

• Batch normalisation

• Relativistic GANs

43

We’vegivenyouahighleveloverviewofGANs,whichwillhopefullygiveyouanintuitivegraspofhowtheywork.However,GANsarenotoriouslydif?iculttotrain,andmanyothertricksarerequiredtogetthemtowork.HerearesomephrasesyoushouldGoogleifyoudecidetotryimplementingyourownGAN.

Inthenextvideo,we’lllookatacompletelydifferentapproachtotraininggeneratornetworks:autoencoders.

Deep generative models Part 3: Autoencoders



What can we do with a generator?

• Generate “new” data

• Interpolation

• Data manipulation

• Dimensionality reduction

45

Inthelastvideo,wesawour?irststrategyfortraininggeneratornetworks.Whatcanwedooncewehaveawell-trainedgeneratornetwork?We’lllookatfourusecases.

The?irst,ofcourseisthatwecangeneratedatathatlookslikeitcamefromthesamedistributionasours.

interpolation

46

gene

rato

r

source: Sampling Generative Networks, Tom White

Ifwetaketwopointsinthelatentspace,anddrawalinebetweenthem,wecanpickevenlyspacedpointsonthatlineanddecodethem.Ifthegeneratorisgood,thisshouldgiveusasmoothtransitionfromonepointtotheother,andeachpointshouldresultinaconvincingexampleofouroutputdomain.

interpolation grid

47source: Sampling Generative Networks, Tom White

Wecanalsodrawaninterpolationgrid;wejustmapthecornersofasquarelatticeofequallyspacedpointstofourpointsinourlatentspace.

spherical linear interpolation

48source: Sampling Generative Networks, Tom White

Ifthelatentspaceishighdimensional,mostoftheprobabilityofthestandardMVNisneartheedgesoftheradius-1hypersphere(notinthecentreasitisin1,2and3-dimensionalMVNs).High-dimensionalMVNslookmorelikeasoapbubblethanthedensepointcloudwe’reusedtoseeinginlow-dimensionalvisualizations.

Forthatreason,wegetbetterresultsifweinterpolatealonganarcinsteadofalongastraightline.Thisiscalledsphericallinearinterpolation.

Interpolationexperimentsareagreatwaytohelpusgettogripswiththestructureofourlatentspace.

interpolation on real data

49source: Abdal, Rameen, Yipeng Qin, and Peter Wonka. "Image2stylegan: How to embed images into the stylegan latent space?." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

Whatifwewanttointerpolatebetweenpointsinourdataset?It’spossibletodothiswithaGANtrainedgenerator,buttomakethiswork,we?irsthaveto7indourdatapointsinthegenerator.Remember,duringtrainingthedsicriminatoristheonlynetworkthatgetstoseetheactualdata.Weneverexplicitlymapthedatatothelatentspace.

Wecantackamappingfromdatatolatentspaceontothenetworkafetrtraining(aswasdonefortheseimages),butwecanalsolearnsuchamappingdirectly.Asithappens,thiscanhelpustotrainthegeneratorinamuchmoredirectway.

What can we do with a generator?

• Generative modelling

• Interpolation

• Data manipulation autoencoders only

• Dimensionality reduction autoencoders only

50

requires mapping into the latent space

Notethatsuchamappingwouldalsogiveusadimensionsalityreduction.Wecanseethelatentspacerepresentationofthedataasareduceddimensionalityrepresentationoftheinput.

We’llfocusontheprespectiveofdimensionalityreductionfortherestofthisvideo,tosetupbasicautoencoders.Wecangetageneratornetworkoutofthese,butit’sabitofanafterthought.Inthenextvideo,we’llseehowtotraingeneratornetworkswithadata-to-latent-spacemappinginamoreprincipledway.

autoencoders

bottleneck architecture for dimensionality reduction.

input should be as close as possible to the output

but: must pass through a small representation.

51

output

input

Here’swhatasimpleautoencoderlookslike.It’sisaparticulartypeofneuralnetwork,shapedlikeanhourglass.Itsjobisjusttomaketheoutputasclosetotheinputaspossible,butsomewhereinthenetworkthereisasmalllayerthatfunctionsasabottleneck.

Afterthenetworkistrained,thissmalllayerbecomesacompressedlow-dimensionalrepresentationoftheinput.

52

enco

der

deco

der

h1 h2

h1

h2

latent space

loss

Here’sthepictureindetail.Wecallthebottomhalfofthenetworktheencoderandthetophalfthedecoder.

Wecallthebluelayerthelatentrepresentationoftheinput.Iftrainanautoencoderwithjusttwonodesonthelatentrepresentation,wecanplotwhatlatentrepresentationeachinputisassigned.Iftheautoencoderworkswell,weexpecttoseesimilarimagesclusteredtogether(forinstancesmilingpeoplevsfrowningpeople,menvswomen,etc).

Ina2Dspace,wecan’tclustertoomanyattributestogether,butinhigherdimensionsit’seasier.ToquoteGeoffHinton:“Iftherewasa30dimensionalsupermarket,[theanchovies]couldbeclosetothe

pizzatoppingsandclosetothesardines.”

after 5 epochs (256 hidden units)

53

Herearethereconstructionsonaverysimplenetwork(just4fullyconnectedReLUlayersfortheencoderandthedecoder),withMSElossontheoutput.

after 25 epochs

54

after 100 epochs

55

after 300 epochs

56data reconstructions

find the smiling vector

57

latent space (256 dim)

smiling

nonsmiling

smilin

g vect

or

Onethingwecannowdoistostudythelatentspacebasedontheexmaplesthatwehave.Forinstance,wecanseetosmilingandnon-smilingpeopleendupindistinctpartsofthelatentspace.

Wejustlabelasmallamountofinstancesassmilingandnonsmiling.Ifwecomputethemeansofthetworesultingclustersinlatentspace,wecandrawavectorbetweenthem.Wecanthinkofthisasa“smiling”vector.Thefurtherwepushpeoplealongthisline,themorethedecodedpointwillsmile.

Thisisonebigbene?itofautoencoders:wecantrainthemonunlabeleddata(whichischeap)andthenuseonlyaverysmallnumberoflabeledexamplesto

make someone smile/frown

encode to the latent space: z = encode(x)

add/subtract some proportion of the smiling vector: zsmile = z + vsmile * 0.2

decode to a smiling face: xsmile = decode(zsmile)

58

59

60source: https://blogs.nvidia.com/blog/2016/12/23/ai-flips-kanye-wests-frown-upside-down/

Withabitmorepowerfulmodel,andsomefacedetection,wecanseewhatsomefamouslymoodycelebritiesmightlooklikeiftheysmiled.

autoencoders

keep the encoder and decoder: data manipulator.

keep the encoder, ditch the decoder: dimensionality reduction.

ditch the encoder, keep the decoder: generator network.

61

Ifwekeeptheencoderandthedecoder,wegetanetworkthatcanhelpusmanipulatedatainthisway.

Ifwekeepjusttheencoder,wegetapowerfuldimensionalityreductionmethod.Wecanusethelatentspacerepresentationasthefeaturesforamodelthatdoesnotscalewelltomanyfeatures(likeanon-naiveBayesianclassi?ier).

Butthislecturewasaboutgeneratornetworks.Howdowegetageneratoroutofatrainedautoencoder?Itturnsoutwecandothisbykeepingjustthedecoder.

turning an autoencoder into a generator

train an autoencoder

encode the data to latent variables Z

fit an MVN to Z

sample from the MVN

“decode” the sample

62

Wedon’tbeforehandknowwherethedatawillendupinlatentspace,butaftertrainingwecanjustcheck.Weencodethetrainingdata,?itadistributiontothispointcloudinourlatentspace,andthenjustusethisdistributionastheinputtoourdecodertocreateageneratornetwork.

63z

Thisisthepointcloudofthelatentrepresentationsinourexample.Weplotthe?irsttwoofthe128dimensions,resultinginthebluepointcloud.

Tothesepointswe,we?itanMVN(in128dimensions),andwesample400pointsfromit,thereddots.

generated64

Ifwefeedthesepointstothedecoder,thisiswhatweget.

How to control the shape of the latent space?

What are we optimizing? Can we optimize maximum likelihood directly?

Can we optimize for better interpolation directly?

65

Thishasgivenusagenerator,butwehavelittlecontroloverwhatthelatentspacelookslike.WejusthavetohopethatitlooksenoughlikeanormaldistributionthatourMVNmakesagood?it.IntheGAN,wehaveperfectcontroloverwhatourdistributiononthelatentspacelookslike;wecanfreelysetittoanything.However,there,wehaveto?itamappingfromdatatolatentspaceafterthefact.

We’vealsoseenthatthisinterpolationworkswell,butit’snotsomethingwe’vespeci?icallytrainedthenetworktodo.IntheGAN,weshouldexpectalllatentspacepointstodecodetosomethingthatfoolsthedecoder,butintheautoencoder,thereisnothingthatstopsthepointsinbetweenthedatapointsfromdecodingtogarbage.

Moreover,neithertheGANnortheautoencoderisaveryprincipledwayofoptimizing.Isthereawaytotrainformaximumlikelihooddirectly?

Theanswertoallofthesequestionsisthevariationalautoencoder,whichwe’lldiscussinthenextvideo.

Deep generative models Part 4: Variational autoencoders



variational autoencoders

• Force the decoder to also decode points near z correctly

• Forces the latent distribution of the data towards N(0, I)

• Can be derived from first principles maximum likelihood

67

maximum (log) likelihood objective

68

argmax✓

lnp(x | ✓)<latexit sha1_base64="faqvOQML4UreBqR++MufzXpRVak=">AAAH5nicfVVdb9NIFDWfgQC7BR55sYiQCooqO13aIoRU+gE8LLRUTYtUR9V4cpNYGdujmXFqd+S/wBvidZ/2F/AK/4N/w9hOsrbH7OQhV/ecc33nzNjXpcTjwrJ+Xrl67fqNm61bt9t37t7748+V+w9OeBgxDH0ckpB9chEH4gXQF54g8IkyQL5L4NSd7mb46QwY98LgWCQUBj4aB97Iw0io1PnKqoPY2EfxuSMmIJDpvFQ/0yGBSVdjx/eGZgE8PV/pWGtWvkw9sOdBx5ivw/P7N/91hiGOfAgEJojzM9uiYiAREx4mkLadiANFeIrGcBaJ0dZAegGNBAQ4NZ8obBQRU4Rm1rU59BhgQRIVIMw8VcHEE8QQFmpv7WopDgHygXeHM4/yIuSzcREIpIwZyDg3Lr1XUcoxQ3Ti4bjSmkQ+95GYaEme+G41CREBNvOryaxN1WSNGQPDHs9MOFTOHNDsMPhxeDjHJwmdQMBTGTGSloUKAMZgpIR5yEFEVOa7UTdgyl8JFkE3C/Pcqz3Epkcw7Ko6lUS1nREJkaimXLUN5U4AFzj0fRQMpUNT6QiIhXS6a2nuXRk9SqV0MqNc1zzK4Ar6oYR+SNMquF8C9xVYRftLdGT269KTEniiPfW0hJ7WpW5UQiMNnZXQmVbZvSjBFxocl9BYQ5MSmmjoZQm91H1G6lqc9QayOIv8UOUB8WbwlgEEqez00vpemDrvM7sqye6A7NhpbvcQRurzUQB+ktHlu+P3f6dyd6v33NpI6wyXRLCgWOsbz3ctjTIuuplzrK2t3o7GCRkKxstCe/sbr229EI0YJUvS5ub6mxd6pQQICS+WlXZ39nrr9Y0pR6pN2Zu2ZdVvG8OaVXNHzI5tataOm+jzxzQK3CZB4Wcjf6rz3zKU/IYdNlVf2NyooE2KheeNiqRJsTiAhaIqCRps+u84lprazmn2Ikyx6jEbGYgUdfdADRMG79ULcqA+gEiE7JnMh5inaql/p5tF/0dE8YKoonZbTTa7Psf04KS3Zltr9se/Ots78xl3y3hkPDZWDdvYNLaNd8ah0Tew8dn4Znw3frQmrc+tL62vBfXqlbnmoVFZrX9+AWoZ3nU=</latexit><latexit sha1_base64="faqvOQML4UreBqR++MufzXpRVak=">AAAH5nicfVVdb9NIFDWfgQC7BR55sYiQCooqO13aIoRU+gE8LLRUTYtUR9V4cpNYGdujmXFqd+S/wBvidZ/2F/AK/4N/w9hOsrbH7OQhV/ecc33nzNjXpcTjwrJ+Xrl67fqNm61bt9t37t7748+V+w9OeBgxDH0ckpB9chEH4gXQF54g8IkyQL5L4NSd7mb46QwY98LgWCQUBj4aB97Iw0io1PnKqoPY2EfxuSMmIJDpvFQ/0yGBSVdjx/eGZgE8PV/pWGtWvkw9sOdBx5ivw/P7N/91hiGOfAgEJojzM9uiYiAREx4mkLadiANFeIrGcBaJ0dZAegGNBAQ4NZ8obBQRU4Rm1rU59BhgQRIVIMw8VcHEE8QQFmpv7WopDgHygXeHM4/yIuSzcREIpIwZyDg3Lr1XUcoxQ3Ti4bjSmkQ+95GYaEme+G41CREBNvOryaxN1WSNGQPDHs9MOFTOHNDsMPhxeDjHJwmdQMBTGTGSloUKAMZgpIR5yEFEVOa7UTdgyl8JFkE3C/Pcqz3Epkcw7Ko6lUS1nREJkaimXLUN5U4AFzj0fRQMpUNT6QiIhXS6a2nuXRk9SqV0MqNc1zzK4Ar6oYR+SNMquF8C9xVYRftLdGT269KTEniiPfW0hJ7WpW5UQiMNnZXQmVbZvSjBFxocl9BYQ5MSmmjoZQm91H1G6lqc9QayOIv8UOUB8WbwlgEEqez00vpemDrvM7sqye6A7NhpbvcQRurzUQB+ktHlu+P3f6dyd6v33NpI6wyXRLCgWOsbz3ctjTIuuplzrK2t3o7GCRkKxstCe/sbr229EI0YJUvS5ub6mxd6pQQICS+WlXZ39nrr9Y0pR6pN2Zu2ZdVvG8OaVXNHzI5tataOm+jzxzQK3CZB4Wcjf6rz3zKU/IYdNlVf2NyooE2KheeNiqRJsTiAhaIqCRps+u84lprazmn2Ikyx6jEbGYgUdfdADRMG79ULcqA+gEiE7JnMh5inaql/p5tF/0dE8YKoonZbTTa7Psf04KS3Zltr9se/Ots78xl3y3hkPDZWDdvYNLaNd8ah0Tew8dn4Znw3frQmrc+tL62vBfXqlbnmoVFZrX9+AWoZ3nU=</latexit><latexit sha1_base64="faqvOQML4UreBqR++MufzXpRVak=">AAAH5nicfVVdb9NIFDWfgQC7BR55sYiQCooqO13aIoRU+gE8LLRUTYtUR9V4cpNYGdujmXFqd+S/wBvidZ/2F/AK/4N/w9hOsrbH7OQhV/ecc33nzNjXpcTjwrJ+Xrl67fqNm61bt9t37t7748+V+w9OeBgxDH0ckpB9chEH4gXQF54g8IkyQL5L4NSd7mb46QwY98LgWCQUBj4aB97Iw0io1PnKqoPY2EfxuSMmIJDpvFQ/0yGBSVdjx/eGZgE8PV/pWGtWvkw9sOdBx5ivw/P7N/91hiGOfAgEJojzM9uiYiAREx4mkLadiANFeIrGcBaJ0dZAegGNBAQ4NZ8obBQRU4Rm1rU59BhgQRIVIMw8VcHEE8QQFmpv7WopDgHygXeHM4/yIuSzcREIpIwZyDg3Lr1XUcoxQ3Ti4bjSmkQ+95GYaEme+G41CREBNvOryaxN1WSNGQPDHs9MOFTOHNDsMPhxeDjHJwmdQMBTGTGSloUKAMZgpIR5yEFEVOa7UTdgyl8JFkE3C/Pcqz3Epkcw7Ko6lUS1nREJkaimXLUN5U4AFzj0fRQMpUNT6QiIhXS6a2nuXRk9SqV0MqNc1zzK4Ar6oYR+SNMquF8C9xVYRftLdGT269KTEniiPfW0hJ7WpW5UQiMNnZXQmVbZvSjBFxocl9BYQ5MSmmjoZQm91H1G6lqc9QayOIv8UOUB8WbwlgEEqez00vpemDrvM7sqye6A7NhpbvcQRurzUQB+ktHlu+P3f6dyd6v33NpI6wyXRLCgWOsbz3ctjTIuuplzrK2t3o7GCRkKxstCe/sbr229EI0YJUvS5ub6mxd6pQQICS+WlXZ39nrr9Y0pR6pN2Zu2ZdVvG8OaVXNHzI5tataOm+jzxzQK3CZB4Wcjf6rz3zKU/IYdNlVf2NyooE2KheeNiqRJsTiAhaIqCRps+u84lprazmn2Ikyx6jEbGYgUdfdADRMG79ULcqA+gEiE7JnMh5inaql/p5tF/0dE8YKoonZbTTa7Psf04KS3Zltr9se/Ots78xl3y3hkPDZWDdvYNLaNd8ah0Tew8dn4Znw3frQmrc+tL62vBfXqlbnmoVFZrX9+AWoZ3nU=</latexit><latexit sha1_base64="faqvOQML4UreBqR++MufzXpRVak=">AAAH5nicfVVdb9NIFDWfgQC7BR55sYiQCooqO13aIoRU+gE8LLRUTYtUR9V4cpNYGdujmXFqd+S/wBvidZ/2F/AK/4N/w9hOsrbH7OQhV/ecc33nzNjXpcTjwrJ+Xrl67fqNm61bt9t37t7748+V+w9OeBgxDH0ckpB9chEH4gXQF54g8IkyQL5L4NSd7mb46QwY98LgWCQUBj4aB97Iw0io1PnKqoPY2EfxuSMmIJDpvFQ/0yGBSVdjx/eGZgE8PV/pWGtWvkw9sOdBx5ivw/P7N/91hiGOfAgEJojzM9uiYiAREx4mkLadiANFeIrGcBaJ0dZAegGNBAQ4NZ8obBQRU4Rm1rU59BhgQRIVIMw8VcHEE8QQFmpv7WopDgHygXeHM4/yIuSzcREIpIwZyDg3Lr1XUcoxQ3Ti4bjSmkQ+95GYaEme+G41CREBNvOryaxN1WSNGQPDHs9MOFTOHNDsMPhxeDjHJwmdQMBTGTGSloUKAMZgpIR5yEFEVOa7UTdgyl8JFkE3C/Pcqz3Epkcw7Ko6lUS1nREJkaimXLUN5U4AFzj0fRQMpUNT6QiIhXS6a2nuXRk9SqV0MqNc1zzK4Ar6oYR+SNMquF8C9xVYRftLdGT269KTEniiPfW0hJ7WpW5UQiMNnZXQmVbZvSjBFxocl9BYQ5MSmmjoZQm91H1G6lqc9QayOIv8UOUB8WbwlgEEqez00vpemDrvM7sqye6A7NhpbvcQRurzUQB+ktHlu+P3f6dyd6v33NpI6wyXRLCgWOsbz3ctjTIuuplzrK2t3o7GCRkKxstCe/sbr229EI0YJUvS5ub6mxd6pQQICS+WlXZ39nrr9Y0pR6pN2Zu2ZdVvG8OaVXNHzI5tataOm+jzxzQK3CZB4Wcjf6rz3zKU/IYdNlVf2NyooE2KheeNiqRJsTiAhaIqCRps+u84lprazmn2Ikyx6jEbGYgUdfdADRMG79ULcqA+gEiE7JnMh5inaql/p5tF/0dE8YKoonZbTTa7Psf04KS3Zltr9se/Ots78xl3y3hkPDZWDdvYNLaNd8ah0Tew8dn4Znw3frQmrc+tL62vBfXqlbnmoVFZrX9+AWoZ3nU=</latexit>

We’llstartwiththemaximumlog-likelihoodobjective.Wewanttochooseourparametersθ(theweightsoftheneuralnetwork)tomaximisetheloglikelihoodofthedata.Wewillwritethisobjectivestepbystepuntilweendupwithanautoencoder.

hidden variable model

69



z

x

✓<latexit sha1_base64="CZ9E8E1A0SjQmmVxEiAYLD4Wobg=">AAAHSXicfVVdb9MwFM026EZhsMEjLxEVEkJVlXRs6x4mjbWwCbGtTGuHtFaT4962oc6HbKdtZuU/8Aq/iT/A3+AJ8YTTLyV1wC+9uuecm3uPXdvyic24YfxcWV27dz+3vvEg//DR5uMnW9tPm8wLKIYG9ohHP1uIAbFdaHCbE/jsU0COReDaGlRj/HoIlNmee8VDH9oO6rl218aIy1SzxfvA0e1WwSgZk6WrgTkLCtps1W+3c6VWx8OBAy7HBDF2Yxo+bwtEuY0JRPlWwMBHeIB6cBPwbqUtbNcPOLg40l9KrBsQnXt63JHesSlgTkIZIExtWUHHfUQR5rLvfLoUAxc5wIqdoe2zaciGvWnAkRy6LcYTU6LNlFL0KPL7Nh6nWhPIYQ7ifSXJQsdKJyEgQIdOOhm3KZtcYo6BYpvFJtSlMxd+bDS78uozvB/6fXBZJAJKoqRQAkApdKVwEjLggS8m08jdHbBDTgMoxuEkd1hDdHAJnaKsk0qk2+kSD/F0ypJjSHdcGGHPcZDbES0/Ei0OYy5axVI08S6JXkZCtGKjLEu/jOEUep5Az6NoSdtYoF29IdEU2EyATaXwdQK9XpZaQQINFHSYQIdKZWuUgEcKPE6gYwUNE2iooHcJ9E61Esmdvym3xdTuyb6JC2IP4YQCuJEolKPlWajc0hszLYm3WRTMaGJ3B7ry3z8FnDCmi9Ors4+RqFbKu8ZetMywSABzirGzt1s1FEpv2s2MY1Qq5WOF41Hk9haFau/23ppqIT+gPlmQ9vd33h+olUIgxBstKlWPa+Wd5XNEsWLCbFa9YOqKab0s+myqTIGVJZg6lckfqPwTisJ/sL2s6nMDMxV+lmLuZqYizFLMrZ0rlobw49M6wPJz8dWNyJRSA3mpUziTp/hCXkSIe/S1PLq059jSPvnbKsbR/4hoPCfKKJ+XL4y5/J6oQaNcOiiZn94Ujj7MnpoN7bn2Qnulmdq+dqSdanWtoWHti/ZV+6Z9z/3I/cr9zv2ZUldXZppnWmqtr/0Fp/+lMw==</latexit><latexit sha1_base64="CZ9E8E1A0SjQmmVxEiAYLD4Wobg=">AAAHSXicfVVdb9MwFM026EZhsMEjLxEVEkJVlXRs6x4mjbWwCbGtTGuHtFaT4962oc6HbKdtZuU/8Aq/iT/A3+AJ8YTTLyV1wC+9uuecm3uPXdvyic24YfxcWV27dz+3vvEg//DR5uMnW9tPm8wLKIYG9ohHP1uIAbFdaHCbE/jsU0COReDaGlRj/HoIlNmee8VDH9oO6rl218aIy1SzxfvA0e1WwSgZk6WrgTkLCtps1W+3c6VWx8OBAy7HBDF2Yxo+bwtEuY0JRPlWwMBHeIB6cBPwbqUtbNcPOLg40l9KrBsQnXt63JHesSlgTkIZIExtWUHHfUQR5rLvfLoUAxc5wIqdoe2zaciGvWnAkRy6LcYTU6LNlFL0KPL7Nh6nWhPIYQ7ifSXJQsdKJyEgQIdOOhm3KZtcYo6BYpvFJtSlMxd+bDS78uozvB/6fXBZJAJKoqRQAkApdKVwEjLggS8m08jdHbBDTgMoxuEkd1hDdHAJnaKsk0qk2+kSD/F0ypJjSHdcGGHPcZDbES0/Ei0OYy5axVI08S6JXkZCtGKjLEu/jOEUep5Az6NoSdtYoF29IdEU2EyATaXwdQK9XpZaQQINFHSYQIdKZWuUgEcKPE6gYwUNE2iooHcJ9E61Esmdvym3xdTuyb6JC2IP4YQCuJEolKPlWajc0hszLYm3WRTMaGJ3B7ry3z8FnDCmi9Ors4+RqFbKu8ZetMywSABzirGzt1s1FEpv2s2MY1Qq5WOF41Hk9haFau/23ppqIT+gPlmQ9vd33h+olUIgxBstKlWPa+Wd5XNEsWLCbFa9YOqKab0s+myqTIGVJZg6lckfqPwTisJ/sL2s6nMDMxV+lmLuZqYizFLMrZ0rlobw49M6wPJz8dWNyJRSA3mpUziTp/hCXkSIe/S1PLq059jSPvnbKsbR/4hoPCfKKJ+XL4y5/J6oQaNcOiiZn94Ujj7MnpoN7bn2Qnulmdq+dqSdanWtoWHti/ZV+6Z9z/3I/cr9zv2ZUldXZppnWmqtr/0Fp/+lMw==</latexit><latexit sha1_base64="CZ9E8E1A0SjQmmVxEiAYLD4Wobg=">AAAHSXicfVVdb9MwFM026EZhsMEjLxEVEkJVlXRs6x4mjbWwCbGtTGuHtFaT4962oc6HbKdtZuU/8Aq/iT/A3+AJ8YTTLyV1wC+9uuecm3uPXdvyic24YfxcWV27dz+3vvEg//DR5uMnW9tPm8wLKIYG9ohHP1uIAbFdaHCbE/jsU0COReDaGlRj/HoIlNmee8VDH9oO6rl218aIy1SzxfvA0e1WwSgZk6WrgTkLCtps1W+3c6VWx8OBAy7HBDF2Yxo+bwtEuY0JRPlWwMBHeIB6cBPwbqUtbNcPOLg40l9KrBsQnXt63JHesSlgTkIZIExtWUHHfUQR5rLvfLoUAxc5wIqdoe2zaciGvWnAkRy6LcYTU6LNlFL0KPL7Nh6nWhPIYQ7ifSXJQsdKJyEgQIdOOhm3KZtcYo6BYpvFJtSlMxd+bDS78uozvB/6fXBZJAJKoqRQAkApdKVwEjLggS8m08jdHbBDTgMoxuEkd1hDdHAJnaKsk0qk2+kSD/F0ypJjSHdcGGHPcZDbES0/Ei0OYy5axVI08S6JXkZCtGKjLEu/jOEUep5Az6NoSdtYoF29IdEU2EyATaXwdQK9XpZaQQINFHSYQIdKZWuUgEcKPE6gYwUNE2iooHcJ9E61Esmdvym3xdTuyb6JC2IP4YQCuJEolKPlWajc0hszLYm3WRTMaGJ3B7ry3z8FnDCmi9Ors4+RqFbKu8ZetMywSABzirGzt1s1FEpv2s2MY1Qq5WOF41Hk9haFau/23ppqIT+gPlmQ9vd33h+olUIgxBstKlWPa+Wd5XNEsWLCbFa9YOqKab0s+myqTIGVJZg6lckfqPwTisJ/sL2s6nMDMxV+lmLuZqYizFLMrZ0rlobw49M6wPJz8dWNyJRSA3mpUziTp/hCXkSIe/S1PLq059jSPvnbKsbR/4hoPCfKKJ+XL4y5/J6oQaNcOiiZn94Ujj7MnpoN7bn2Qnulmdq+dqSdanWtoWHti/ZV+6Z9z/3I/cr9zv2ZUldXZppnWmqtr/0Fp/+lMw==</latexit><latexit sha1_base64="CZ9E8E1A0SjQmmVxEiAYLD4Wobg=">AAAHSXicfVVdb9MwFM026EZhsMEjLxEVEkJVlXRs6x4mjbWwCbGtTGuHtFaT4962oc6HbKdtZuU/8Aq/iT/A3+AJ8YTTLyV1wC+9uuecm3uPXdvyic24YfxcWV27dz+3vvEg//DR5uMnW9tPm8wLKIYG9ohHP1uIAbFdaHCbE/jsU0COReDaGlRj/HoIlNmee8VDH9oO6rl218aIy1SzxfvA0e1WwSgZk6WrgTkLCtps1W+3c6VWx8OBAy7HBDF2Yxo+bwtEuY0JRPlWwMBHeIB6cBPwbqUtbNcPOLg40l9KrBsQnXt63JHesSlgTkIZIExtWUHHfUQR5rLvfLoUAxc5wIqdoe2zaciGvWnAkRy6LcYTU6LNlFL0KPL7Nh6nWhPIYQ7ifSXJQsdKJyEgQIdOOhm3KZtcYo6BYpvFJtSlMxd+bDS78uozvB/6fXBZJAJKoqRQAkApdKVwEjLggS8m08jdHbBDTgMoxuEkd1hDdHAJnaKsk0qk2+kSD/F0ypJjSHdcGGHPcZDbES0/Ei0OYy5axVI08S6JXkZCtGKjLEu/jOEUep5Az6NoSdtYoF29IdEU2EyATaXwdQK9XpZaQQINFHSYQIdKZWuUgEcKPE6gYwUNE2iooHcJ9E61Esmdvym3xdTuyb6JC2IP4YQCuJEolKPlWajc0hszLYm3WRTMaGJ3B7ry3z8FnDCmi9Ors4+RqFbKu8ZetMywSABzirGzt1s1FEpv2s2MY1Qq5WOF41Hk9haFau/23ppqIT+gPlmQ9vd33h+olUIgxBstKlWPa+Wd5XNEsWLCbFa9YOqKab0s+myqTIGVJZg6lckfqPwTisJ/sL2s6nMDMxV+lmLuZqYizFLMrZ0rlobw49M6wPJz8dWNyJRSA3mpUziTp/hCXkSIe/S1PLq059jSPvnbKsbR/4hoPCfKKJ+XL4y5/J6oQaNcOiiZn94Ujj7MnpoN7bn2Qnulmdq+dqSdanWtoWHti/ZV+6Z9z/3I/cr9zv2ZUldXZppnWmqtr/0Fp/+lMw==</latexit>

<latexit sha1_base64="+Xxi6XOiYJkqCTElci17fupV9aQ=">AAAH5XicfVXLbtNAFDXPQHgVWLKxiHgqquwU2rKoBH0AC6ClalqkOqrGkxvHytgezYzTuCN/AjvElh0/wBY+hL9hHCfB9hhm46t7zrm+98zY41Lic2FZv8+dv3Dx0uXGlavNa9dv3Ly1dPvOIY9ihqGLIxKxTy7iQPwQusIXBD5RBihwCRy5o60MPxoD434UHoiEQi9AXugPfIyESp0sPXI8LGn6cMPMg8fOxHQCv286Z23TEUMQ6InpOM2TpZa1bE2XqQf2LGgZs7V3cvvyd6cf4TiAUGCCOD+2LSp6EjHhYwJp04k5UIRHyIPjWAzWe9IPaSwgxKn5QGGDmJgiMrOmzb7PAAuSqABh5qsKJh4ihrBQozXLpTiEKADe7o99yvOQj708EEj50pOTqW/pjZJSegzRoY8npdYkCniAxFBL8iRwy0mICbBxUE5mbaomK8wJMOzzzIQ95cwuzfaCH0R7M3yY0CGEPJUxI2lRqABgDAZKOA05iJjK6TTqAIz4hmAxtLNwmtvYRmy0D/22qlNKlNsZkAiJcspVYyh3QjjFURCgsC8dmkpHwERIp72cTr0rovuplE5mlOua+xlcQj8U0A9pWgZ3CuCOAstod4EOzG5VelgAD7W3HhXQo6rUjQtorKHjAjrWKrunBfhUgycFdKKhSQFNNPSsgJ7pPiN1LI47PZnvxXRT5S7xx/CGAYSpbHXS6ixM7fexXZZkZ0C27HRqdx8G6u+RA0GS0eXbg/fvUrm13nluraZVhktimFOsldXnW5ZG8fJuZhxrfb2zqXEihkJvUWh7Z/WVrReiMaNkQVpbW3n9Qq+UACHR6aLS1uZ2Z6U6mHKk3JS9ZltW9bQxrFk1c8Rs2aZmrVdHn72mVuDWCXI/a/kjnf+GoeQf7Kiu+tzmWgWtU8w9r1UkdYr5BswVZUlYY9Pf7VhoKpPT7EMYYdVjdmUgktfdBnWZMHivPpBd9QNEImJP1VfBvMBXtdTTaWfR/4hoMieqqJndbHb1HtODw86yvbpsfXzWerk5u+OuGPeM+8ZjwzbWjJfGW2PP6BrY+Gz8MH4avxpe43PjS+NrTj1/bqa5a5RW49sf9HjdUg==</latexit>

p = p(x | z, ✓)

The?irstinsightisthatwecanviewourgeneratorasahiddenvariablemodel.Wehaveahiddenvariablez,whichisstandardnormallydistributedandthenfedtoaneuralnetworktoproduceavariablex,whichweobserve.

thenetworkcomputestheconditionaldistributionofxgivenz.

mode collapse

70

z

Hereweseehowthehiddenvariableproblemcausesourmodecollapse.Ifweknewwhichzwassupposedtoproducewhichx,wecouldfeedthatztothenetworkcomputethelossbetweentheoutputandxandoptimizebybackpropagationandgradientdescent.

Theproblem,asinthepreviouslecture,isthatwedon’thavethecompletedata.Wedon’tknowthevaluesofz,onlythevaluesofx.

Thesolutionistointroduce

introducing q

71

<latexit sha1_base64="AlcKA5/XMCkhkac7SBHMaI4x41c=">AAAHWHicfVVbb9MwFPYGrFu5beORl4gKCVBVJR278DBpbINNCLYxrRtSU02Oe9pGdS7YTpfMyl/hFf4S/BqcXqakDvglR+f7vpNzPlu2E1KXC9P8vbB47/6DpcrySvXho8dPnq6urV/yIGIEWiSgAfvmYA7U9aElXEHhW8gAew6FK2d4kOFXI2DcDfwLkYTQ8XDfd3suwUKlrlfXbUbk9/SVfWvYnts17Pj19WrNbJjjZeiBNQ1qaLrOrteWGnY3IJEHviAUc962zFB0JGbCJRTSqh1xCDEZ4j60I9Hb6UjXDyMBPkmNlwrrRdQQgZH1Z3RdBkTQRAWYMFdVMMgAM0yEmqJaLMXBxx7wenfkhnwS8lF/EgisLOjIeGxR+riglH2Gw4FL4kJrEnvcw2KgJXniOcUkRBTYyCsmszZVk3PMGBhxeWbCmXLmNMxs5xfB2RQfJOEAfJ7KiNE0L1QAMAY9JRyHHEQUyvE0aq+HfFewCOpZOM7tHmI2PIduXdUpJIrt9GiARTHlqDGUOz7ckMDzsN+VdphKW0AspF1vpGPv8uh5KqWdGeU4xnkGF9CTHHqSpnPa1h3aM1oKLYCXOfBSK3yVQ6/mpU6UQyMNHeXQkVbZucnBNxoc59BYQ5McmmjobQ691a3EaufbzY6c2D3eN3lK3REcMQA/lbVmOj8LU1vatoqSbJtlzUrHdnehp+6CCeAlGV0eX3z5nMqDneamuZXOMxwawYxibmxtHpgapT/pZsoxd3aa+xonYNjv3xU6/LD13tILhREL6R1pe3vj4zu9UgKUBjd3lQ72D5sb8+eIEc2E6axGzTI00/pl9OlUpQKnTDBxqpQ/1PlHDCf/YAdl1WcGlirCMsXMzVJFUqaYWTtTzA0RZqd1SNTvsqsb0wnlENSlzuCLOsWn6iLCImBv1NFlfc9V9qmvXc+i/xFxPCOqqFpVL4w1/57owWWzYW01zK9va3ufpm/NMnqOXqBXyELbaA8dozPUQgTF6Af6iX4t/amgSqWyMqEuLkw1z1BhVdb/Ao4Ypwk=</latexit>

q(z | x)

OursolutionintheEMalgorithmwastointroduceanewdistributionq,whichtellsusforagivenx,whichvaluesofzaremostlikely.We’llfollowthesamestrategyhere.Whatwouldbeagoodq?

inverting p?

72



<latexit sha1_base64="G2awwzlDJORZFTWsznQPYNpO9jA=">AAAHYHicfVVbb9MwFM64rKMw2OBtvERUSANVVdKxCw+TxjbYhNiFad2Qlmpy3NM0qpNYttM1syLxa3iFv8MrvwSnlympA37J0fm+7+Scz5btUuJzYVm/5+7df/BwvrLwqPr4yeLTZ0vLzy94FDMMLRyRiH1zEQfih9ASviDwjTJAgUvg0u3vZfjlABj3o/BcJBTaAfJCv+tjJFTqemnF8bCk6apzazqB3zGdYd10RA8EenO9VLMa1miZemBPgpoxWafXy/MNpxPhOIBQYII4v7ItKtoSMeFjAmnViTlQhPvIg6tYdLfa0g9pLCDEqflaYd2YmCIysz7Njs8AC5KoAGHmqwom7iGGsFDTVIulOIQoAF7vDHzKxyEfeONAIGVFWw5HVqWLBaX0GKI9Hw8LrUkU8ACJnpbkSeAWkxATYIOgmMzaVE3OMIfAsM8zE06VMyc0s5+fR6cTvJfQHoQ8lTEjaV6oAGAMuko4CjmImMrRNGrP+3xbsBjqWTjKbe8j1j+DTl3VKSSK7XRJhEQx5aoxlDsh3OAoCFDYkQ5NpSNgKKRTb6Qj7/LoWSqlkxnluuZZBhfQ4xx6nKYz2tYd2jVbCi2AFznwQit8mUMvZ6VunENjDR3k0IFW2b3JwTcaPMyhQw1Ncmiiobc59Fa3Eqmdv2q25dju0b7JE+IP4IABhKmsNdPZWZja0iu7KMm2WdbsdGR3B7rqThgDQZLR5eH50ZdU7m01162NdJbhkhimFGttY33P0ijeuJsJx9raau5qnIih0LsrtP9x44OtF6Ixo+SOtLm59um9XikBQqKbu0p7u/vNtdlzxLBmwmRWs2abmmleGX0yVanALROMnSrl93X+AUPJP9hRWfWpgaUKWqaYulmqSMoUU2unipkhaHZa+1j9Lru6ERlT9kFd6gyO1Ck+URcREhF7q44u8wJf2ae+Tj2L/kdEwylRRdWqemHs2fdEDy6aDXujYX19V9v5PHlrFoyXxitj1bCNTWPHODROjZaBje/GD+On8Wv+T2Wh8qyyPKbem5toXhiFVVn5C6m1qgs=</latexit>

p(z | x, ✓)

IntheEMalgorithm,foragivenchoiceofparametersforp,wecoudleasilyworkoutthedistribtiononz,conditionedonx.Thisgaveusagoodchoiceforq.

Here,thingsaren’tsoeasy.Toworkoutp(z|x,theta)wewouldneedtoinverttheneuralnetwork:workoutforaparticularoutputx,whichinputvalueszarelikelytohavecausedthatoutput.

Thisisnotimpossibletodo(wesawsomethingsimilarinatthestartofthelecture),butitsacostlyandimprecisebusiness.JustlikewedidwiththeGANs,it’sbesttointroduceanetworkthatwilllearntheinversionforus.

introducing q

73

x<latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit><latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit><latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit><latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit>

µz ⌃z<latexit sha1_base64="cERa83nXYH1MYuJDKbOwBjsolO4=">AAAHcHicfVXdbts2FFb/4sxt13S9GnoxtUaBojMMyVl+iqJAl2RrL9omy+KkQGQEFH0sC6YkgqQcKwQfok/T2+4x9hx7gVGyHEimWvnCB+f7voNzPh6QPiUhF47z742bt27fWWut/9C+e+/+jw82Hv50ypOUYRjghCTsk484kDCGgQgFgU+UAYp8Amf+dD/Hz2bAeJjEJyKjMIxQEIfjECOhUxcbv3oJlp4fSS9KlbrwrmzvVf6zPb/M/x0GESqgi42O03OKzzYDtww6VvkdXTxc63mjBKcRxAITxPm561AxlIiJEBNQbS/lQBGeogDOUzHeHcowpqmAGCv7mcbGKbFFYueN26OQARYk0wHCLNQVbDxBDGGhx2vXS3GIUQS8O5qFlC9CPgsWgUDam6GcF96p+zWlDBiikxDPa61JFPEIiYmR5Fnk15OQEmCzqJ7M29RNrjDnwHDIcxOOtDOHND8PfpIclfgkoxOIuZIpI6oq1AAwBmMtLEIOIqWymEYvwZS/FiyFbh4WudcHiE2PYdTVdWqJejtjkiBRT/l6DO1ODJc4iSIUj6RHlfQEzIX0uj1VeFdFj5XUa6SN8n37OIdr6McK+lGpFe3gGh3bA43WwNMKeGoUPqugZ6tSP62gqYHOKujMqOxfVuBLA55X0LmBZhU0M9CrCnplWon0yZ/3h3Jhd3Fu8pCEM3jLAGIlO321OgvTR3ru1iX5McuOqwq7RzDWl8QCiLKcLt+dfHiv5P5uf8vZVqsMn6SwpDib21v7jkEJFt2UHGd3t79ncBKG4uC60MEf27+7ZiGaMkquSTs7m3++NCtlQEhyeV1pf++gv7m6RwwbJpSz2h3XNkwLmujlVI0Cv0mwcKqRPzX5bxnKvsFOmqovDWxU0CbF0s1GRdakWFq7VKwMQfNtneoHgeZXNyILygHoS53BB73Fh/oiQiJhL/TqsiAKtX363+vm0feIaL4k6qjd1i+Mu/qemMGg33vZc//6rfNmr3xq1q3H1lPrueVaO9Yb6511ZA0sbH22vlhfrX/W/mv93Pql9WRBvXmj1Dyyal/rxf9zAbHe</latexit><latexit sha1_base64="cERa83nXYH1MYuJDKbOwBjsolO4=">AAAHcHicfVXdbts2FFb/4sxt13S9GnoxtUaBojMMyVl+iqJAl2RrL9omy+KkQGQEFH0sC6YkgqQcKwQfok/T2+4x9hx7gVGyHEimWvnCB+f7voNzPh6QPiUhF47z742bt27fWWut/9C+e+/+jw82Hv50ypOUYRjghCTsk484kDCGgQgFgU+UAYp8Amf+dD/Hz2bAeJjEJyKjMIxQEIfjECOhUxcbv3oJlp4fSS9KlbrwrmzvVf6zPb/M/x0GESqgi42O03OKzzYDtww6VvkdXTxc63mjBKcRxAITxPm561AxlIiJEBNQbS/lQBGeogDOUzHeHcowpqmAGCv7mcbGKbFFYueN26OQARYk0wHCLNQVbDxBDGGhx2vXS3GIUQS8O5qFlC9CPgsWgUDam6GcF96p+zWlDBiikxDPa61JFPEIiYmR5Fnk15OQEmCzqJ7M29RNrjDnwHDIcxOOtDOHND8PfpIclfgkoxOIuZIpI6oq1AAwBmMtLEIOIqWymEYvwZS/FiyFbh4WudcHiE2PYdTVdWqJejtjkiBRT/l6DO1ODJc4iSIUj6RHlfQEzIX0uj1VeFdFj5XUa6SN8n37OIdr6McK+lGpFe3gGh3bA43WwNMKeGoUPqugZ6tSP62gqYHOKujMqOxfVuBLA55X0LmBZhU0M9CrCnplWon0yZ/3h3Jhd3Fu8pCEM3jLAGIlO321OgvTR3ru1iX5McuOqwq7RzDWl8QCiLKcLt+dfHiv5P5uf8vZVqsMn6SwpDib21v7jkEJFt2UHGd3t79ncBKG4uC60MEf27+7ZiGaMkquSTs7m3++NCtlQEhyeV1pf++gv7m6RwwbJpSz2h3XNkwLmujlVI0Cv0mwcKqRPzX5bxnKvsFOmqovDWxU0CbF0s1GRdakWFq7VKwMQfNtneoHgeZXNyILygHoS53BB73Fh/oiQiJhL/TqsiAKtX363+vm0feIaL4k6qjd1i+Mu/qemMGg33vZc//6rfNmr3xq1q3H1lPrueVaO9Yb6511ZA0sbH22vlhfrX/W/mv93Pql9WRBvXmj1Dyyal/rxf9zAbHe</latexit><latexit sha1_base64="cERa83nXYH1MYuJDKbOwBjsolO4=">AAAHcHicfVXdbts2FFb/4sxt13S9GnoxtUaBojMMyVl+iqJAl2RrL9omy+KkQGQEFH0sC6YkgqQcKwQfok/T2+4x9hx7gVGyHEimWvnCB+f7voNzPh6QPiUhF47z742bt27fWWut/9C+e+/+jw82Hv50ypOUYRjghCTsk484kDCGgQgFgU+UAYp8Amf+dD/Hz2bAeJjEJyKjMIxQEIfjECOhUxcbv3oJlp4fSS9KlbrwrmzvVf6zPb/M/x0GESqgi42O03OKzzYDtww6VvkdXTxc63mjBKcRxAITxPm561AxlIiJEBNQbS/lQBGeogDOUzHeHcowpqmAGCv7mcbGKbFFYueN26OQARYk0wHCLNQVbDxBDGGhx2vXS3GIUQS8O5qFlC9CPgsWgUDam6GcF96p+zWlDBiikxDPa61JFPEIiYmR5Fnk15OQEmCzqJ7M29RNrjDnwHDIcxOOtDOHND8PfpIclfgkoxOIuZIpI6oq1AAwBmMtLEIOIqWymEYvwZS/FiyFbh4WudcHiE2PYdTVdWqJejtjkiBRT/l6DO1ODJc4iSIUj6RHlfQEzIX0uj1VeFdFj5XUa6SN8n37OIdr6McK+lGpFe3gGh3bA43WwNMKeGoUPqugZ6tSP62gqYHOKujMqOxfVuBLA55X0LmBZhU0M9CrCnplWon0yZ/3h3Jhd3Fu8pCEM3jLAGIlO321OgvTR3ru1iX5McuOqwq7RzDWl8QCiLKcLt+dfHiv5P5uf8vZVqsMn6SwpDib21v7jkEJFt2UHGd3t79ncBKG4uC60MEf27+7ZiGaMkquSTs7m3++NCtlQEhyeV1pf++gv7m6RwwbJpSz2h3XNkwLmujlVI0Cv0mwcKqRPzX5bxnKvsFOmqovDWxU0CbF0s1GRdakWFq7VKwMQfNtneoHgeZXNyILygHoS53BB73Fh/oiQiJhL/TqsiAKtX363+vm0feIaL4k6qjd1i+Mu/qemMGg33vZc//6rfNmr3xq1q3H1lPrueVaO9Yb6511ZA0sbH22vlhfrX/W/mv93Pql9WRBvXmj1Dyyal/rxf9zAbHe</latexit><latexit sha1_base64="cERa83nXYH1MYuJDKbOwBjsolO4=">AAAHcHicfVXdbts2FFb/4sxt13S9GnoxtUaBojMMyVl+iqJAl2RrL9omy+KkQGQEFH0sC6YkgqQcKwQfok/T2+4x9hx7gVGyHEimWvnCB+f7voNzPh6QPiUhF47z742bt27fWWut/9C+e+/+jw82Hv50ypOUYRjghCTsk484kDCGgQgFgU+UAYp8Amf+dD/Hz2bAeJjEJyKjMIxQEIfjECOhUxcbv3oJlp4fSS9KlbrwrmzvVf6zPb/M/x0GESqgi42O03OKzzYDtww6VvkdXTxc63mjBKcRxAITxPm561AxlIiJEBNQbS/lQBGeogDOUzHeHcowpqmAGCv7mcbGKbFFYueN26OQARYk0wHCLNQVbDxBDGGhx2vXS3GIUQS8O5qFlC9CPgsWgUDam6GcF96p+zWlDBiikxDPa61JFPEIiYmR5Fnk15OQEmCzqJ7M29RNrjDnwHDIcxOOtDOHND8PfpIclfgkoxOIuZIpI6oq1AAwBmMtLEIOIqWymEYvwZS/FiyFbh4WudcHiE2PYdTVdWqJejtjkiBRT/l6DO1ODJc4iSIUj6RHlfQEzIX0uj1VeFdFj5XUa6SN8n37OIdr6McK+lGpFe3gGh3bA43WwNMKeGoUPqugZ6tSP62gqYHOKujMqOxfVuBLA55X0LmBZhU0M9CrCnplWon0yZ/3h3Jhd3Fu8pCEM3jLAGIlO321OgvTR3ru1iX5McuOqwq7RzDWl8QCiLKcLt+dfHiv5P5uf8vZVqsMn6SwpDib21v7jkEJFt2UHGd3t79ncBKG4uC60MEf27+7ZiGaMkquSTs7m3++NCtlQEhyeV1pf++gv7m6RwwbJpSz2h3XNkwLmujlVI0Cv0mwcKqRPzX5bxnKvsFOmqovDWxU0CbF0s1GRdakWFq7VKwMQfNtneoHgeZXNyILygHoS53BB73Fh/oiQiJhL/TqsiAKtX363+vm0feIaL4k6qjd1i+Mu/qemMGg33vZc//6rfNmr3xq1q3H1lPrueVaO9Yb6511ZA0sbH22vlhfrX/W/mv93Pql9WRBvXmj1Dyyal/rxf9zAbHe</latexit>

<latexit sha1_base64="wSSk9dIAFQ2isW2SZNrZrSqdtHw=">AAAHXnicfVVdb9MwFM0GrKMwtsHLJF4iKqSBqirpWFceJo1tsAmxD6Z1m9RUk+PetlGdD2yna2aFX8Mr/B/e+Ck4/VJSB/ySq3vOubn32LLtgDiMG8bvhcUHDx8tFZYfF588XXm2urb+/Ir5IcXQwD7x6Y2NGBDHgwZ3OIGbgAJybQLXdv8gwa8HQJnje5c8CqDloq7ndByMuEzdrm1YFItv8aZ1r1uu09atYVm3gp7z5natZFSM0dLVwJwEJW2yzm/XlypW28ehCx7HBDHWNI2AtwSi3MEE4qIVMggQ7qMuNEPeqbeE4wUhBw/H+muJdUKic19PutTbDgXMSSQDhKkjK+i4hyjCXM5SzJZi4CEXWLk9cAI2DtmgOw44kka0xHBkVLySUYouRXJSPMy0JpDLXMR7SpJFrp1NQkiADtxsMmlTNjnHHALFDktMOJfOnAWJ+ezSP5/gvSjogcdiEVISp4USAEqhI4WjkAEPAzGaRu54n+1yGkI5CUe53UNE+xfQLss6mUS2nQ7xEc+mbDmGdMeDO+y7LvLawgpiYXEYcmGVK/HIuzR6EQthJUbZtn6RwBn0NIWexvGctjFDO3pDohnwKgVeKYWvU+j1vNQOU2iooIMUOlAq23cp+E6Bhyl0qKBRCo0U9D6F3qtWIrnzzWpLjO0e7Zs4I84AjiiAF4tSNZ6fhcotbZpZSbLNomTGI7vb0JE3whhwo4Quji9PvsTioF7dNmrxPMMmIUwpxlZt+8BQKN1xNxOOUa9X9xWOT5HXnRU6/Fj7YKqFgpAGZEba2dn69F6tFAEh/t2s0sH+YXVr/hxRrJgwmVUvmbpiWjePPpkqV2DnCcZO5fL7Kv+IougfbD+v+tTAXEWQp5i6mauI8hRTa6eKuSGC5LT2sfxdcnUjMqYcgrzUKZzIU3wmLyLEffpWHl3adR1pn/xa5ST6HxENp0QZFYvyhTHn3xM1uKpWzFrF+PqutPd58tYsay+1V9qmZmo72p52rJ1rDQ1r37Uf2k/t19KfwlJhpbA6pi4uTDQvtMwqbPwF3QqpLg==</latexit>

q(z | x,�)

z

x

x,�)

<latexit sha1_base64="G2awwzlDJORZFTWsznQPYNpO9jA=">AAAHYHicfVVbb9MwFM64rKMw2OBtvERUSANVVdKxCw+TxjbYhNiFad2Qlmpy3NM0qpNYttM1syLxa3iFv8MrvwSnlympA37J0fm+7+Scz5btUuJzYVm/5+7df/BwvrLwqPr4yeLTZ0vLzy94FDMMLRyRiH1zEQfih9ASviDwjTJAgUvg0u3vZfjlABj3o/BcJBTaAfJCv+tjJFTqemnF8bCk6apzazqB3zGdYd10RA8EenO9VLMa1miZemBPgpoxWafXy/MNpxPhOIBQYII4v7ItKtoSMeFjAmnViTlQhPvIg6tYdLfa0g9pLCDEqflaYd2YmCIysz7Njs8AC5KoAGHmqwom7iGGsFDTVIulOIQoAF7vDHzKxyEfeONAIGVFWw5HVqWLBaX0GKI9Hw8LrUkU8ACJnpbkSeAWkxATYIOgmMzaVE3OMIfAsM8zE06VMyc0s5+fR6cTvJfQHoQ8lTEjaV6oAGAMuko4CjmImMrRNGrP+3xbsBjqWTjKbe8j1j+DTl3VKSSK7XRJhEQx5aoxlDsh3OAoCFDYkQ5NpSNgKKRTb6Qj7/LoWSqlkxnluuZZBhfQ4xx6nKYz2tYd2jVbCi2AFznwQit8mUMvZ6VunENjDR3k0IFW2b3JwTcaPMyhQw1Ncmiiobc59Fa3Eqmdv2q25dju0b7JE+IP4IABhKmsNdPZWZja0iu7KMm2WdbsdGR3B7rqThgDQZLR5eH50ZdU7m01162NdJbhkhimFGttY33P0ijeuJsJx9raau5qnIih0LsrtP9x44OtF6Ixo+SOtLm59um9XikBQqKbu0p7u/vNtdlzxLBmwmRWs2abmmleGX0yVanALROMnSrl93X+AUPJP9hRWfWpgaUKWqaYulmqSMoUU2unipkhaHZa+1j9Lru6ERlT9kFd6gyO1Ck+URcREhF7q44u8wJf2ae+Tj2L/kdEwylRRdWqemHs2fdEDy6aDXujYX19V9v5PHlrFoyXxitj1bCNTWPHODROjZaBje/GD+On8Wv+T2Wh8qyyPKbem5toXhiFVVn5C6m1qgs=</latexit>

p(z | x, ✓)approximation for

Thisisthenetworkwe’lluse.Itconsumesxanditproducesanormaldistributiononz.Ithasitsownparameters,phi,whicharenotrelatedtotheparametersthetaofp.

We’llnow?igureoutawaytotrainpandqinconcert.We’lltrytoupdatetheparametersofpto?itthedata,andtrywe’llupdatetheparametersofqtokeepitagoodapproximationtotheinversionofp.

simplify our notation

74

<latexit sha1_base64="a9dQY+m05eGRc9MDacxWsvx6QbM=">AAAIZ3icfVVrb+NEFHUW2A2BpV1WWiHxZSACbVFU2Sl98KGw9MEuEtuWqmlXqqNqPLlxrPjFzDixM5q/g8RP4ifwLxg/kvUL/MVX95xz5865Y48Vug7juv5359EHH370+En3494nnz79bGv72ee3LIgogREJ3IC+szAD1/FhxB3uwruQAvYsF+6s+WmK3y2AMifwb3gSwtjDtu9MHYK5Sj1s/2XaRITypRmbnjNB5mqATD4DjnfQt8coAx+EaU3RUqYkVLB2kGn2TErEHyq7KrKx0oYzJ1emWK5cyDInVxaLrvLk+wUvNkwl9ZAuB3nwq9x52O7ru3r2oGZgFEFfK56rh2eP/zQnAYk88DlxMWP3hh7yscCUO8QF2TMjBiEmc2zDfcSnR2Ph+GHEwScSfaOwaeQiHqDUNjRxKBDuJirAhDqqAiIzTDHhytxetRQDH3vABpOFE7I8ZAs7DzhWkxmLOJucfFpRCpti5R+JK60J7DEP81kjyRLPqiYhcoEuvGoybVM1WWPGQInDUhOulDOXYXoa2E1wVeCzJJyBz6SIqCvLQgUApTBVwixkwKNQZLtRR3DOjjmNYJCGWe74DNP5NUwGqk4lUW1n6gaYV1OW2oZyx4clCTwP+xNhhlKYHGIuzMGuzLwro9dSCDM1yrLQdQpX0IsSeiFlFTwvgecKrKKjDTpFo7r0tgTeNla9K6F3dakVldCogS5K6KJR2VqW4GUDjkto3ECTEpo00FUJXTV9xupY3A/HIp9FNlRx6ToLeE0BfCn6Q1nfC1XzvjeqkvQMiL4hM7snMFX/rxzwkpQu3ty8/U2K06Phvn4g6wzLjWBN0fcO9k/1BsXOuyk4+tHR8KTBCSj27U2hs/ODn41moTCiobshHR7u/fJDs1ICrhssN5VOT86Ge/WNKUeqTRmHhq7XTxslDasKR1DfQA1r7TZ6sUyrwGoT5H628udN/muKk/9gB23V1za3KsI2xdrzVkXSplgPYK2oSvwWm96PY6Op7TxMP4Q5UT2mVwZ287pnoC4TCm/VB3KpfoCYB/Q79VVQ23NULfU2B2n0f0Qcr4kq6vXUzWbU77FmcDvcNQ529d+/7786Ke64rval9rX2UjO0Q+2V9ka70kYa6Wx19js/dn568k93q/ui+0VOfdQpNM+1ytP96l8fdAXL</latexit>

p(x | z, ✓) = pw(x | z)

q(z | x,�) = qv(z | x)

p(z | ✓) = N(z | 0, I)

<latexit sha1_base64="a9dQY+m05eGRc9MDacxWsvx6QbM=">AAAIZ3icfVVrb+NEFHUW2A2BpV1WWiHxZSACbVFU2Sl98KGw9MEuEtuWqmlXqqNqPLlxrPjFzDixM5q/g8RP4ifwLxg/kvUL/MVX95xz5865Y48Vug7juv5359EHH370+En3494nnz79bGv72ee3LIgogREJ3IC+szAD1/FhxB3uwruQAvYsF+6s+WmK3y2AMifwb3gSwtjDtu9MHYK5Sj1s/2XaRITypRmbnjNB5mqATD4DjnfQt8coAx+EaU3RUqYkVLB2kGn2TErEHyq7KrKx0oYzJ1emWK5cyDInVxaLrvLk+wUvNkwl9ZAuB3nwq9x52O7ru3r2oGZgFEFfK56rh2eP/zQnAYk88DlxMWP3hh7yscCUO8QF2TMjBiEmc2zDfcSnR2Ph+GHEwScSfaOwaeQiHqDUNjRxKBDuJirAhDqqAiIzTDHhytxetRQDH3vABpOFE7I8ZAs7DzhWkxmLOJucfFpRCpti5R+JK60J7DEP81kjyRLPqiYhcoEuvGoybVM1WWPGQInDUhOulDOXYXoa2E1wVeCzJJyBz6SIqCvLQgUApTBVwixkwKNQZLtRR3DOjjmNYJCGWe74DNP5NUwGqk4lUW1n6gaYV1OW2oZyx4clCTwP+xNhhlKYHGIuzMGuzLwro9dSCDM1yrLQdQpX0IsSeiFlFTwvgecKrKKjDTpFo7r0tgTeNla9K6F3dakVldCogS5K6KJR2VqW4GUDjkto3ECTEpo00FUJXTV9xupY3A/HIp9FNlRx6ToLeE0BfCn6Q1nfC1XzvjeqkvQMiL4hM7snMFX/rxzwkpQu3ty8/U2K06Phvn4g6wzLjWBN0fcO9k/1BsXOuyk4+tHR8KTBCSj27U2hs/ODn41moTCiobshHR7u/fJDs1ICrhssN5VOT86Ge/WNKUeqTRmHhq7XTxslDasKR1DfQA1r7TZ6sUyrwGoT5H628udN/muKk/9gB23V1za3KsI2xdrzVkXSplgPYK2oSvwWm96PY6Op7TxMP4Q5UT2mVwZ287pnoC4TCm/VB3KpfoCYB/Q79VVQ23NULfU2B2n0f0Qcr4kq6vXUzWbU77FmcDvcNQ529d+/7786Ke64rval9rX2UjO0Q+2V9ka70kYa6Wx19js/dn568k93q/ui+0VOfdQpNM+1ytP96l8fdAXL</latexit>

p(x | z, ✓) = pw(x | z)

q(z | x,�) = qv(z | x)

p(z | ✓) = N(z | 0, I)

Sincetheparametersofourmodelaretheneuralnetworkweights,we’llsimplifyournotationlikethis.

Thisempasizesthateventhoughtheseareprobabilities,theconditionalprobabilitiesshownherearetheonlyprobabilitydensitiesthatwecanef?icientlycompute.Wecan’treversetheconditional,ormaginalizeanythingout.Thepricewepayforusingthepowerofneuralnetworksisthatwearestuckwiththesefunctionsandhavebuildanalgorithmonjustthese.

Withoneexception.Wedoknowthemarginaldistributiononz,sincethatiswhatwede?inedasthedistributionontheinputofourgeneratorneuralnet:

it’sasimplestandardnormalMVN.

75


µx ⌃x<latexit sha1_base64="Hl68MPLI79R3yTLxDS2GN37K3tc=">AAAHcHicfVXdbtNIFDY/20BYlgJXiAvMRkgrNorslP4JIUFbFi4W2u02LVIdVePJiWNlbI9mxmnMaB6Cp9nb5TF4Dl5gx45T2RmzzkWOzvd9R+d8czTjUxJy4Tjfrl2/cfOntdat2+07P9/95d76/QenPEkZhgFOSMI++YgDCWMYiFAQ+EQZoMgncOZP93P8bAaMh0l8IjIKwwgFcTgOMRI6dbH+u5dg6fmR9KJUqQtvbnsv85/t+WX+7zCIUAFdrHecnlN8thm4ZdCxyu/o4v5azxslOI0gFpggzs9dh4qhREyEmIBqeykHivAUBXCeivHOUIYxTQXEWNnPNDZOiS0SO2/cHoUMsCCZDhBmoa5g4wliCAs9XrteikOMIuDd0SykfBHyWbAIBNLeDOW88E7drSllwBCdhHhea02iiEdITIwkzyK/noSUAJtF9WTepm5yhTkHhkOem3CknTmk+Xnwk+SoxCcZnUDMlUwZUVWhBoAxGGthEXIQKZXFNHoJpvyVYCl087DIvTpAbHoMo66uU0vU2xmTBIl6ytdjaHdiuMRJFKF4JD2qpCdgLqTX7anCuyp6rKReI22U79vHOVxDP1bQj0qtaAdX6NgeaLQGnlbAU6PwWQU9W5X6aQVNDXRWQWdGZf+yAl8a8LyCzg00q6CZgX6uoJ9NK5E++fP+UC7sLs5NHpJwBu8YQKxkp69WZ2H6SM/duiQ/ZtlxVWH3CMb6klgAUZbT5fuTD38qub/T33S21CrDJyksKc7G1ua+Y1CCRTclx9nZ6e8ZnIShOLgqdPB2641rFqIpo+SKtL298ceuWSkDQpLLq0r7ewf9jdU9YtgwoZzV7ri2YVrQRC+nahT4TYKFU438qcl/x1D2A3bSVH1pYKOCNimWbjYqsibF0tqlYmUImm/rVD8INL+6EVlQDkBf6gw+6C0+1BcREgl7rleXBVGo7dP/XjeP/o+I5kuijtpt/cK4q++JGQz6vd2e+9eLzuu98qm5ZT22frV+s1xr23ptvbeOrIGFrS/WP9a/1te1761HrSetpwvq9Wul5qFV+1rP/wNXS7Ha</latexit><latexit sha1_base64="Hl68MPLI79R3yTLxDS2GN37K3tc=">AAAHcHicfVXdbtNIFDY/20BYlgJXiAvMRkgrNorslP4JIUFbFi4W2u02LVIdVePJiWNlbI9mxmnMaB6Cp9nb5TF4Dl5gx45T2RmzzkWOzvd9R+d8czTjUxJy4Tjfrl2/cfOntdat2+07P9/95d76/QenPEkZhgFOSMI++YgDCWMYiFAQ+EQZoMgncOZP93P8bAaMh0l8IjIKwwgFcTgOMRI6dbH+u5dg6fmR9KJUqQtvbnsv85/t+WX+7zCIUAFdrHecnlN8thm4ZdCxyu/o4v5azxslOI0gFpggzs9dh4qhREyEmIBqeykHivAUBXCeivHOUIYxTQXEWNnPNDZOiS0SO2/cHoUMsCCZDhBmoa5g4wliCAs9XrteikOMIuDd0SykfBHyWbAIBNLeDOW88E7drSllwBCdhHhea02iiEdITIwkzyK/noSUAJtF9WTepm5yhTkHhkOem3CknTmk+Xnwk+SoxCcZnUDMlUwZUVWhBoAxGGthEXIQKZXFNHoJpvyVYCl087DIvTpAbHoMo66uU0vU2xmTBIl6ytdjaHdiuMRJFKF4JD2qpCdgLqTX7anCuyp6rKReI22U79vHOVxDP1bQj0qtaAdX6NgeaLQGnlbAU6PwWQU9W5X6aQVNDXRWQWdGZf+yAl8a8LyCzg00q6CZgX6uoJ9NK5E++fP+UC7sLs5NHpJwBu8YQKxkp69WZ2H6SM/duiQ/ZtlxVWH3CMb6klgAUZbT5fuTD38qub/T33S21CrDJyksKc7G1ua+Y1CCRTclx9nZ6e8ZnIShOLgqdPB2641rFqIpo+SKtL298ceuWSkDQpLLq0r7ewf9jdU9YtgwoZzV7ri2YVrQRC+nahT4TYKFU438qcl/x1D2A3bSVH1pYKOCNimWbjYqsibF0tqlYmUImm/rVD8INL+6EVlQDkBf6gw+6C0+1BcREgl7rleXBVGo7dP/XjeP/o+I5kuijtpt/cK4q++JGQz6vd2e+9eLzuu98qm5ZT22frV+s1xr23ptvbeOrIGFrS/WP9a/1te1761HrSetpwvq9Wul5qFV+1rP/wNXS7Ha</latexit><latexit sha1_base64="Hl68MPLI79R3yTLxDS2GN37K3tc=">AAAHcHicfVXdbtNIFDY/20BYlgJXiAvMRkgrNorslP4JIUFbFi4W2u02LVIdVePJiWNlbI9mxmnMaB6Cp9nb5TF4Dl5gx45T2RmzzkWOzvd9R+d8czTjUxJy4Tjfrl2/cfOntdat2+07P9/95d76/QenPEkZhgFOSMI++YgDCWMYiFAQ+EQZoMgncOZP93P8bAaMh0l8IjIKwwgFcTgOMRI6dbH+u5dg6fmR9KJUqQtvbnsv85/t+WX+7zCIUAFdrHecnlN8thm4ZdCxyu/o4v5azxslOI0gFpggzs9dh4qhREyEmIBqeykHivAUBXCeivHOUIYxTQXEWNnPNDZOiS0SO2/cHoUMsCCZDhBmoa5g4wliCAs9XrteikOMIuDd0SykfBHyWbAIBNLeDOW88E7drSllwBCdhHhea02iiEdITIwkzyK/noSUAJtF9WTepm5yhTkHhkOem3CknTmk+Xnwk+SoxCcZnUDMlUwZUVWhBoAxGGthEXIQKZXFNHoJpvyVYCl087DIvTpAbHoMo66uU0vU2xmTBIl6ytdjaHdiuMRJFKF4JD2qpCdgLqTX7anCuyp6rKReI22U79vHOVxDP1bQj0qtaAdX6NgeaLQGnlbAU6PwWQU9W5X6aQVNDXRWQWdGZf+yAl8a8LyCzg00q6CZgX6uoJ9NK5E++fP+UC7sLs5NHpJwBu8YQKxkp69WZ2H6SM/duiQ/ZtlxVWH3CMb6klgAUZbT5fuTD38qub/T33S21CrDJyksKc7G1ua+Y1CCRTclx9nZ6e8ZnIShOLgqdPB2641rFqIpo+SKtL298ceuWSkDQpLLq0r7ewf9jdU9YtgwoZzV7ri2YVrQRC+nahT4TYKFU438qcl/x1D2A3bSVH1pYKOCNimWbjYqsibF0tqlYmUImm/rVD8INL+6EVlQDkBf6gw+6C0+1BcREgl7rleXBVGo7dP/XjeP/o+I5kuijtpt/cK4q++JGQz6vd2e+9eLzuu98qm5ZT22frV+s1xr23ptvbeOrIGFrS/WP9a/1te1761HrSetpwvq9Wul5qFV+1rP/wNXS7Ha</latexit><latexit sha1_base64="Hl68MPLI79R3yTLxDS2GN37K3tc=">AAAHcHicfVXdbtNIFDY/20BYlgJXiAvMRkgrNorslP4JIUFbFi4W2u02LVIdVePJiWNlbI9mxmnMaB6Cp9nb5TF4Dl5gx45T2RmzzkWOzvd9R+d8czTjUxJy4Tjfrl2/cfOntdat2+07P9/95d76/QenPEkZhgFOSMI++YgDCWMYiFAQ+EQZoMgncOZP93P8bAaMh0l8IjIKwwgFcTgOMRI6dbH+u5dg6fmR9KJUqQtvbnsv85/t+WX+7zCIUAFdrHecnlN8thm4ZdCxyu/o4v5azxslOI0gFpggzs9dh4qhREyEmIBqeykHivAUBXCeivHOUIYxTQXEWNnPNDZOiS0SO2/cHoUMsCCZDhBmoa5g4wliCAs9XrteikOMIuDd0SykfBHyWbAIBNLeDOW88E7drSllwBCdhHhea02iiEdITIwkzyK/noSUAJtF9WTepm5yhTkHhkOem3CknTmk+Xnwk+SoxCcZnUDMlUwZUVWhBoAxGGthEXIQKZXFNHoJpvyVYCl087DIvTpAbHoMo66uU0vU2xmTBIl6ytdjaHdiuMRJFKF4JD2qpCdgLqTX7anCuyp6rKReI22U79vHOVxDP1bQj0qtaAdX6NgeaLQGnlbAU6PwWQU9W5X6aQVNDXRWQWdGZf+yAl8a8LyCzg00q6CZgX6uoJ9NK5E++fP+UC7sLs5NHpJwBu8YQKxkp69WZ2H6SM/duiQ/ZtlxVWH3CMb6klgAUZbT5fuTD38qub/T33S21CrDJyksKc7G1ua+Y1CCRTclx9nZ6e8ZnIShOLgqdPB2641rFqIpo+SKtL298ceuWSkDQpLLq0r7ewf9jdU9YtgwoZzV7ri2YVrQRC+nahT4TYKFU438qcl/x1D2A3bSVH1pYKOCNimWbjYqsibF0tqlYmUImm/rVD8INL+6EVlQDkBf6gw+6C0+1BcREgl7rleXBVGo7dP/XjeP/o+I5kuijtpt/cK4q++JGQz6vd2e+9eLzuu98qm5ZT22frV+s1xr23ptvbeOrIGFrS/WP9a/1te1761HrSetpwvq9Wul5qFV+1rP/wNXS7Ha</latexit>


z<latexit sha1_base64="fzVeLgBDrGdiJ0cciYKccClUWxs=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lG++52q2CUjOnS1cCcBwVtvuq327lSu+vhwAGXY4IYuzENn3cEotzGBKJ8O2DgIzxEfbgJeK/SEbbrBxxcHOmvJdYLiM49Pe5G79oUMCehDBCmtqyg4wGiCHPZcz5dioGLHGDF7sj22Sxko/4s4EgO3BGTqSHRZkop+hT5AxtPUq0J5DAH8YGSZKFjpZMQEKAjJ52M25RNrjAnQLHNYhPq0pkLPzaZXXr1OT4I/QG4LBIBJVFSKAGgFHpSOA0Z8MAX02nkzg7ZIacBFONwmjusITpsQLco66QS6XZ6xEM8nbLkGNIdF8bYcxzkdkXbj0Sbw4SLdrEUTb1Loo1IiHZslGXpjRhOoecJ9DyKVrTNJdrTmxJNgVcJ8Eop3EqgrVWpFSTQQEFHCXSkVLbGCXiswJMEOlHQMIGGCnqXQO9UK5Hc+ZtyR8zsnu6buCD2CE4ogBuJQjlanYXKLb0x05J4m0XBjKZ2d6En//kzwAljuji9PPsUiWqlvGvsRasMiwSwoBg7e7tVQ6H0Z93MOUalUj5WOB5Fbn9ZqPZh772pFvID6pMlaX9/5+OBWikEQrzxslL1uFbeWT1HFCsmzGfVC6aumNbPos+nyhRYWYKZU5n8oco/oSj8B9vLqr4wMFPhZykWbmYqwizFwtqFYmUIPz6tQyw/F1/diMwoNZCXOoUzeYov5EWEuEffyqNL+44t7ZO/7WIc/Y+IJguijPJ5+cKYq++JGjTLpYOS+fld4eh4/tRsaC+1V9obzdT2tSPtVKtrTQ1rPe2L9lX7lvue+5n7lfs9o66vzTUvtNTK/fkL5pekZg==</latexit><latexit sha1_base64="fzVeLgBDrGdiJ0cciYKccClUWxs=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lG++52q2CUjOnS1cCcBwVtvuq327lSu+vhwAGXY4IYuzENn3cEotzGBKJ8O2DgIzxEfbgJeK/SEbbrBxxcHOmvJdYLiM49Pe5G79oUMCehDBCmtqyg4wGiCHPZcz5dioGLHGDF7sj22Sxko/4s4EgO3BGTqSHRZkop+hT5AxtPUq0J5DAH8YGSZKFjpZMQEKAjJ52M25RNrjAnQLHNYhPq0pkLPzaZXXr1OT4I/QG4LBIBJVFSKAGgFHpSOA0Z8MAX02nkzg7ZIacBFONwmjusITpsQLco66QS6XZ6xEM8nbLkGNIdF8bYcxzkdkXbj0Sbw4SLdrEUTb1Loo1IiHZslGXpjRhOoecJ9DyKVrTNJdrTmxJNgVcJ8Eop3EqgrVWpFSTQQEFHCXSkVLbGCXiswJMEOlHQMIGGCnqXQO9UK5Hc+ZtyR8zsnu6buCD2CE4ogBuJQjlanYXKLb0x05J4m0XBjKZ2d6En//kzwAljuji9PPsUiWqlvGvsRasMiwSwoBg7e7tVQ6H0Z93MOUalUj5WOB5Fbn9ZqPZh772pFvID6pMlaX9/5+OBWikEQrzxslL1uFbeWT1HFCsmzGfVC6aumNbPos+nyhRYWYKZU5n8oco/oSj8B9vLqr4wMFPhZykWbmYqwizFwtqFYmUIPz6tQyw/F1/diMwoNZCXOoUzeYov5EWEuEffyqNL+44t7ZO/7WIc/Y+IJguijPJ5+cKYq++JGjTLpYOS+fld4eh4/tRsaC+1V9obzdT2tSPtVKtrTQ1rPe2L9lX7lvue+5n7lfs9o66vzTUvtNTK/fkL5pekZg==</latexit><latexit sha1_base64="fzVeLgBDrGdiJ0cciYKccClUWxs=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lG++52q2CUjOnS1cCcBwVtvuq327lSu+vhwAGXY4IYuzENn3cEotzGBKJ8O2DgIzxEfbgJeK/SEbbrBxxcHOmvJdYLiM49Pe5G79oUMCehDBCmtqyg4wGiCHPZcz5dioGLHGDF7sj22Sxko/4s4EgO3BGTqSHRZkop+hT5AxtPUq0J5DAH8YGSZKFjpZMQEKAjJ52M25RNrjAnQLHNYhPq0pkLPzaZXXr1OT4I/QG4LBIBJVFSKAGgFHpSOA0Z8MAX02nkzg7ZIacBFONwmjusITpsQLco66QS6XZ6xEM8nbLkGNIdF8bYcxzkdkXbj0Sbw4SLdrEUTb1Loo1IiHZslGXpjRhOoecJ9DyKVrTNJdrTmxJNgVcJ8Eop3EqgrVWpFSTQQEFHCXSkVLbGCXiswJMEOlHQMIGGCnqXQO9UK5Hc+ZtyR8zsnu6buCD2CE4ogBuJQjlanYXKLb0x05J4m0XBjKZ2d6En//kzwAljuji9PPsUiWqlvGvsRasMiwSwoBg7e7tVQ6H0Z93MOUalUj5WOB5Fbn9ZqPZh772pFvID6pMlaX9/5+OBWikEQrzxslL1uFbeWT1HFCsmzGfVC6aumNbPos+nyhRYWYKZU5n8oco/oSj8B9vLqr4wMFPhZykWbmYqwizFwtqFYmUIPz6tQyw/F1/diMwoNZCXOoUzeYov5EWEuEffyqNL+44t7ZO/7WIc/Y+IJguijPJ5+cKYq++JGjTLpYOS+fld4eh4/tRsaC+1V9obzdT2tSPtVKtrTQ1rPe2L9lX7lvue+5n7lfs9o66vzTUvtNTK/fkL5pekZg==</latexit><latexit sha1_base64="fzVeLgBDrGdiJ0cciYKccClUWxs=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lG++52q2CUjOnS1cCcBwVtvuq327lSu+vhwAGXY4IYuzENn3cEotzGBKJ8O2DgIzxEfbgJeK/SEbbrBxxcHOmvJdYLiM49Pe5G79oUMCehDBCmtqyg4wGiCHPZcz5dioGLHGDF7sj22Sxko/4s4EgO3BGTqSHRZkop+hT5AxtPUq0J5DAH8YGSZKFjpZMQEKAjJ52M25RNrjAnQLHNYhPq0pkLPzaZXXr1OT4I/QG4LBIBJVFSKAGgFHpSOA0Z8MAX02nkzg7ZIacBFONwmjusITpsQLco66QS6XZ6xEM8nbLkGNIdF8bYcxzkdkXbj0Sbw4SLdrEUTb1Loo1IiHZslGXpjRhOoecJ9DyKVrTNJdrTmxJNgVcJ8Eop3EqgrVWpFSTQQEFHCXSkVLbGCXiswJMEOlHQMIGGCnqXQO9UK5Hc+ZtyR8zsnu6buCD2CE4ogBuJQjlanYXKLb0x05J4m0XBjKZ2d6En//kzwAljuji9PPsUiWqlvGvsRasMiwSwoBg7e7tVQ6H0Z93MOUalUj5WOB5Fbn9ZqPZh772pFvID6pMlaX9/5+OBWikEQrzxslL1uFbeWT1HFCsmzGfVC6aumNbPos+nyhRYWYKZU5n8oco/oSj8B9vLqr4wMFPhZykWbmYqwizFwtqFYmUIPz6tQyw/F1/diMwoNZCXOoUzeYov5EWEuEffyqNL+44t7ZO/7WIc/Y+IJguijPJ5+cKYq++JGjTLpYOS+fld4eh4/tRsaC+1V9obzdT2tSPtVKtrTQ1rPe2L9lX7lvue+5n7lfs9o66vzTUvtNTK/fkL5pekZg==</latexit>


qv(z|x)

pw(x|z) w v<latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit>

w v<latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit>

Puttingeverythingtogether,thisisourmodel.Ifwefeedqvaninstancefromourdatax,wegetanormaldistributiononthelatentspace.Ifwesampleapointzfromthisdistribution,andfeedittopwwegetadistributiononx.Ifthenetworksarebothwelltrained,thisshouldgiveusagoodreconstructionofx.

Theneuralnetworkpwisourprobabilitydistributionconditionalonthelatentvector.qvisourapproximationoftheconditionaldistributiononz.

a very useful decomposition

76

<latexit sha1_base64="r5L60W+u9WNEq7I05RcUg7d2s1o=">AAAJGXicfVVdc9tEFJULFGMoNOWRlx1cMg6YjOS0SYDJTMkH7UzTJmTipEPkyazktazxSlpWK0fKzv4S3vkffWN45YlfwCv8A64+7EiWwC++c889R/eevdJajLqh0PU/W/feefe9+++3P+h8+NGDjz95uPboIgwibpOhHdCAv7FwSKjrk6FwBSVvGCfYsyi5tGYHKX45Jzx0A/9cJIyMPOz47sS1sYDU9VrrJ5P6iPXM2PTcMTLFlAi8gdb30HHP5Lb8WfWXya/Qy1LSsSVTG8g0O6YgsZA3rpiq9W+zRIaBRh70zFuUq8d3YmlZJgVoAW6sm9+hXAz7CcKM8SB2vaxTJAL0eCm3IDxWqU69qzuhlxGlFrZnXx8TFzzhaOyCHQ7xbaKAWh8y7RoeObUseaSucxSlHpkTjkG8lw2xGKggKbmcZTHphupcP+zqm3r2Q/XAKIKuVvxOr9fu/2qOAzvyiC9sisPwytCZGEnMhWtTojpmFBIG02CHXEVisjuSrs8iAdMo9AVgk4imRqUHDYNyYguaQIBt7oICsqcYZhCwDp2qVEh87JGwP567LMzDcO7kgcDg20jG2a6pBxWmdDhmU9eOK61J7IWphbVkmHhWNUkiOJO5V02mbUKTK5Ux4bYbpiacgjMnLN2K8Dw4LfBpwqbED5WMOFVlIgCEczIBYhaGRERMZtPASzML9wSPSD8Ns9zeIeazMzLug04lUW1nQgMsqikLxgB3fHJjB56H/bE0GSxGtoZmf1Nl3pXRMyVlsWvoLIUr6OsS+lqpKnhUAo8ArKLDJTpBw1XqRQm8qD31soRerlKtqIRGNXReQuc1ZeumBN/U4LiExjU0KaFJDb0tobd1nzGsxdVgJPOzyA5VnlD4DjznhPhKdgdqdRYO531lVCnpDsiuoTK7x2QCX9wc8JK0XL44f3Ws5MHu4Km+rVYrLBqRRYm+tf30QK+VOHk3RY2+uzvYr9UEHPvOUujwaPt7oy7EIs7osmhnZ+uHb+pKCaE0uFkqHewfDrZWBwNHqk0ZO4aur24bt2tWFY6groFq1jpN5cVjGglWEyH3s7F+Vq9/znHyH9VBk/rC5kYGa2IsPG9kJE2MxQEsGFWK32DT3XEsOSuTs/RFmNnQY3plYJrrHhK4TDh5BS/ICXwAsQj4l/BWcMdzQQv+zX4a/V8hjheFEHXSm81YvcfqwcVg09je1H980n22X9xxbe0z7XOtpxnajvZMe6GdakPNbr1t/dX6u/VP+5f22/Zv7d/z0nutgvOpVvm1//gXge1KjQ==</latexit>

lnp(x | ✓) = L(q, ✓) + KL(q,p)

with :

p = p(z | x, ✓)

q(z | x) any approximation to p(z | x)

KL(q,p) Kullback-Leibler divergence

L(q, ✓) = Eq lnp(x, z | ✓)

q(z | x)

Wecannowapplythedecompositionweprovedinthelastlecture.Welookatthemarginaldistributiononx,andbreakitupinto

Before,wehadadiscretehiddenvariable,andnowwehaveacontinuousone,buttheproofstillworks(sincewewroteeverythingintermsofexpectations).

a very useful decomposition

77

argmaxw

X

x

lnpw(x)

lnpw(x) = L(qv,pw) + KL(qv,pw)

with:qv(z | x) : any approximation to pw(z | x)

L(qv,pw) = Eqpw(x, z)

qv(z | x)<latexit sha1_base64="z844AIV/5n/nhKBQhhWBpGAZC9I=">AAAJI3icfVVbb9s2FJbbrfO8dWvWx70QM1YkmxFIzppkBQJ0uawFljZZkEuB0DAomrYFUxJHUrYUgj9n+zN9G/ayh/2IvW4vo2TZlixlggEfnO+icw5J0WXUE9K2/2w8ePjBh48+an7c+uTTx599/mTji2sRRhyTKxzSkL9zkSDUC8iV9CQl7xgnyHcpuXEnRyl+MyVceGFwKRNGej4aBd7Qw0iaVH+jMXgGER/5KO4rOMIKukMw0xpAEfl9GANIA5DmWX8BbcJ4C0DYMr9n96AH4HQTcqx+6UN3qjs5BbozvQW+BVCSWKqfTnXOyaRTveQtrFZvyQQzT45faJAlSrpNeAeg7w2AefWL3B0FCUCM8TD2/KxTIEOgK7WuhJnv/VUfAHjSz0BjMuQIqxVqWjbkuy2tVmpjvXBOS+4/advbdvaAauDkQdvKn/P+xqPf4CDEkU8CiSkS4taxmewpxKWHKdEtGAnCEJ6gEbmN5HC/p7yARZIEWIOvDTaMaNpyuuRg4HGCJU1MgDD3jAPAY2SakGZjtMpWggTIJ6IzmHpMzEMxHc0Dicyu6qk423X6cUmpRhyxsYfjUmkK+cLMf1xJisR3y0kSUcKnfjmZlmmKXGPGhGNPpEM4N5M5Y+n6isvwPMfHCRuTQGgVcaqLQgMQzsnQCLNQEBkxlXVjjs9EHEgekU4aZrmDY8QnF2TQMT6lRLmcIQ2RLKdc04aZTkBmOPR9FAwUZGZ3ZDsTdrZ1NrsieqGVgumgXBdcpHAJfVtA32pdBk8K4IkBy+jVEh2Cq3XpdQG8rrz1poDerEvdqIBGFXRaQKcVZ3NmVvCsAscFNK6gSQFNKuhdAb2rzhmZbXHb7an5WmSLqs6oNyWvOCGBVu2uXu+Fm/W+dcqSdA+otqOzcQ/I0Hx754CfpHT1+vLNqVZH+93n9q5eZ7g0IguKvbP7/MiuUEbzanKOvb/fPaxwQo6C0dLo+GT3B6dqxCLO6JK0t7fz4/dVp4RQGs6WTkeHx92d9cbMRMpFOXuOba/vNo4ro8onAtoOqIx2VEfPX1MrcOsE83nW8idV/iuOknvYYZ37Ysy1ClanWMy8VpHUKRYLsFCUJUHNmFbLsdSsdc7SgzAx1xRLrwxE577HxFwmnLwxB+TMfACRDPk3Krv/PeNl/mEnjf6PiOIF0UStlrnZnPV7rBpcd7cde9v5+bv2y8P8jmtaX1pfWZuWY+1ZL63X1rl1ZeHG+8bfjX8a/zZ/bb5v/t78Y0590Mg1T63S0/zrP2umUcU=</latexit><latexit sha1_base64="z844AIV/5n/nhKBQhhWBpGAZC9I=">AAAJI3icfVVbb9s2FJbbrfO8dWvWx70QM1YkmxFIzppkBQJ0uawFljZZkEuB0DAomrYFUxJHUrYUgj9n+zN9G/ayh/2IvW4vo2TZlixlggEfnO+icw5J0WXUE9K2/2w8ePjBh48+an7c+uTTx599/mTji2sRRhyTKxzSkL9zkSDUC8iV9CQl7xgnyHcpuXEnRyl+MyVceGFwKRNGej4aBd7Qw0iaVH+jMXgGER/5KO4rOMIKukMw0xpAEfl9GANIA5DmWX8BbcJ4C0DYMr9n96AH4HQTcqx+6UN3qjs5BbozvQW+BVCSWKqfTnXOyaRTveQtrFZvyQQzT45faJAlSrpNeAeg7w2AefWL3B0FCUCM8TD2/KxTIEOgK7WuhJnv/VUfAHjSz0BjMuQIqxVqWjbkuy2tVmpjvXBOS+4/advbdvaAauDkQdvKn/P+xqPf4CDEkU8CiSkS4taxmewpxKWHKdEtGAnCEJ6gEbmN5HC/p7yARZIEWIOvDTaMaNpyuuRg4HGCJU1MgDD3jAPAY2SakGZjtMpWggTIJ6IzmHpMzEMxHc0Dicyu6qk423X6cUmpRhyxsYfjUmkK+cLMf1xJisR3y0kSUcKnfjmZlmmKXGPGhGNPpEM4N5M5Y+n6isvwPMfHCRuTQGgVcaqLQgMQzsnQCLNQEBkxlXVjjs9EHEgekU4aZrmDY8QnF2TQMT6lRLmcIQ2RLKdc04aZTkBmOPR9FAwUZGZ3ZDsTdrZ1NrsieqGVgumgXBdcpHAJfVtA32pdBk8K4IkBy+jVEh2Cq3XpdQG8rrz1poDerEvdqIBGFXRaQKcVZ3NmVvCsAscFNK6gSQFNKuhdAb2rzhmZbXHb7an5WmSLqs6oNyWvOCGBVu2uXu+Fm/W+dcqSdA+otqOzcQ/I0Hx754CfpHT1+vLNqVZH+93n9q5eZ7g0IguKvbP7/MiuUEbzanKOvb/fPaxwQo6C0dLo+GT3B6dqxCLO6JK0t7fz4/dVp4RQGs6WTkeHx92d9cbMRMpFOXuOba/vNo4ro8onAtoOqIx2VEfPX1MrcOsE83nW8idV/iuOknvYYZ37Ysy1ClanWMy8VpHUKRYLsFCUJUHNmFbLsdSsdc7SgzAx1xRLrwxE577HxFwmnLwxB+TMfACRDPk3Krv/PeNl/mEnjf6PiOIF0UStlrnZnPV7rBpcd7cde9v5+bv2y8P8jmtaX1pfWZuWY+1ZL63X1rl1ZeHG+8bfjX8a/zZ/bb5v/t78Y0590Mg1T63S0/zrP2umUcU=</latexit><latexit sha1_base64="z844AIV/5n/nhKBQhhWBpGAZC9I=">AAAJI3icfVVbb9s2FJbbrfO8dWvWx70QM1YkmxFIzppkBQJ0uawFljZZkEuB0DAomrYFUxJHUrYUgj9n+zN9G/ayh/2IvW4vo2TZlixlggEfnO+icw5J0WXUE9K2/2w8ePjBh48+an7c+uTTx599/mTji2sRRhyTKxzSkL9zkSDUC8iV9CQl7xgnyHcpuXEnRyl+MyVceGFwKRNGej4aBd7Qw0iaVH+jMXgGER/5KO4rOMIKukMw0xpAEfl9GANIA5DmWX8BbcJ4C0DYMr9n96AH4HQTcqx+6UN3qjs5BbozvQW+BVCSWKqfTnXOyaRTveQtrFZvyQQzT45faJAlSrpNeAeg7w2AefWL3B0FCUCM8TD2/KxTIEOgK7WuhJnv/VUfAHjSz0BjMuQIqxVqWjbkuy2tVmpjvXBOS+4/advbdvaAauDkQdvKn/P+xqPf4CDEkU8CiSkS4taxmewpxKWHKdEtGAnCEJ6gEbmN5HC/p7yARZIEWIOvDTaMaNpyuuRg4HGCJU1MgDD3jAPAY2SakGZjtMpWggTIJ6IzmHpMzEMxHc0Dicyu6qk423X6cUmpRhyxsYfjUmkK+cLMf1xJisR3y0kSUcKnfjmZlmmKXGPGhGNPpEM4N5M5Y+n6isvwPMfHCRuTQGgVcaqLQgMQzsnQCLNQEBkxlXVjjs9EHEgekU4aZrmDY8QnF2TQMT6lRLmcIQ2RLKdc04aZTkBmOPR9FAwUZGZ3ZDsTdrZ1NrsieqGVgumgXBdcpHAJfVtA32pdBk8K4IkBy+jVEh2Cq3XpdQG8rrz1poDerEvdqIBGFXRaQKcVZ3NmVvCsAscFNK6gSQFNKuhdAb2rzhmZbXHb7an5WmSLqs6oNyWvOCGBVu2uXu+Fm/W+dcqSdA+otqOzcQ/I0Hx754CfpHT1+vLNqVZH+93n9q5eZ7g0IguKvbP7/MiuUEbzanKOvb/fPaxwQo6C0dLo+GT3B6dqxCLO6JK0t7fz4/dVp4RQGs6WTkeHx92d9cbMRMpFOXuOba/vNo4ro8onAtoOqIx2VEfPX1MrcOsE83nW8idV/iuOknvYYZ37Ysy1ClanWMy8VpHUKRYLsFCUJUHNmFbLsdSsdc7SgzAx1xRLrwxE577HxFwmnLwxB+TMfACRDPk3Krv/PeNl/mEnjf6PiOIF0UStlrnZnPV7rBpcd7cde9v5+bv2y8P8jmtaX1pfWZuWY+1ZL63X1rl1ZeHG+8bfjX8a/zZ/bb5v/t78Y0590Mg1T63S0/zrP2umUcU=</latexit><latexit sha1_base64="z844AIV/5n/nhKBQhhWBpGAZC9I=">AAAJI3icfVVbb9s2FJbbrfO8dWvWx70QM1YkmxFIzppkBQJ0uawFljZZkEuB0DAomrYFUxJHUrYUgj9n+zN9G/ayh/2IvW4vo2TZlixlggEfnO+icw5J0WXUE9K2/2w8ePjBh48+an7c+uTTx599/mTji2sRRhyTKxzSkL9zkSDUC8iV9CQl7xgnyHcpuXEnRyl+MyVceGFwKRNGej4aBd7Qw0iaVH+jMXgGER/5KO4rOMIKukMw0xpAEfl9GANIA5DmWX8BbcJ4C0DYMr9n96AH4HQTcqx+6UN3qjs5BbozvQW+BVCSWKqfTnXOyaRTveQtrFZvyQQzT45faJAlSrpNeAeg7w2AefWL3B0FCUCM8TD2/KxTIEOgK7WuhJnv/VUfAHjSz0BjMuQIqxVqWjbkuy2tVmpjvXBOS+4/advbdvaAauDkQdvKn/P+xqPf4CDEkU8CiSkS4taxmewpxKWHKdEtGAnCEJ6gEbmN5HC/p7yARZIEWIOvDTaMaNpyuuRg4HGCJU1MgDD3jAPAY2SakGZjtMpWggTIJ6IzmHpMzEMxHc0Dicyu6qk423X6cUmpRhyxsYfjUmkK+cLMf1xJisR3y0kSUcKnfjmZlmmKXGPGhGNPpEM4N5M5Y+n6isvwPMfHCRuTQGgVcaqLQgMQzsnQCLNQEBkxlXVjjs9EHEgekU4aZrmDY8QnF2TQMT6lRLmcIQ2RLKdc04aZTkBmOPR9FAwUZGZ3ZDsTdrZ1NrsieqGVgumgXBdcpHAJfVtA32pdBk8K4IkBy+jVEh2Cq3XpdQG8rrz1poDerEvdqIBGFXRaQKcVZ3NmVvCsAscFNK6gSQFNKuhdAb2rzhmZbXHb7an5WmSLqs6oNyWvOCGBVu2uXu+Fm/W+dcqSdA+otqOzcQ/I0Hx754CfpHT1+vLNqVZH+93n9q5eZ7g0IguKvbP7/MiuUEbzanKOvb/fPaxwQo6C0dLo+GT3B6dqxCLO6JK0t7fz4/dVp4RQGs6WTkeHx92d9cbMRMpFOXuOba/vNo4ro8onAtoOqIx2VEfPX1MrcOsE83nW8idV/iuOknvYYZ37Ysy1ClanWMy8VpHUKRYLsFCUJUHNmFbLsdSsdc7SgzAx1xRLrwxE577HxFwmnLwxB+TMfACRDPk3Krv/PeNl/mEnjf6PiOIF0UStlrnZnPV7rBpcd7cde9v5+bv2y8P8jmtaX1pfWZuWY+1ZL63X1rl1ZeHG+8bfjX8a/zZ/bb5v/t78Y0590Mg1T63S0/zrP2umUcU=</latexit>

argmaxw

X

x

lnpw(x)





argmaxw

X

x

lnpw(x)





<latexit sha1_base64="nU9B6EQ3wtWmsQJb95lKNXLOCnI=">AAAIHHicfVXLbtw2FFXSJpNOktZplt0IGTSwg4EhjRvbXRhI/GiySGLH8NgBrOmA4tzRCENJDEmNJRP8lS77JdkV3RboL3SbfkApzcN6pdro4p5zrngPSV2XEp8Ly/r71u2vvr5zt3Xvm/b9Bw+//W7t0ffnPIoZhj6OSMQ+uIgD8UPoC18Q+EAZoMAlcOFODzL8YgaM+1F4JlIKgwB5oT/2MRI6NVz79embdYdh+XHouDPVNR0PS6rjK7Vh7pnO0TAHlemQ0HTGDGF5w1h3Ei243lDypsK6c+0E/sh0kg0tcoZrHWvTyh+zHtiLoGMsnpPho7u/O6MIxwGEAhPE+aVtUTGQiAkfE1BtJ+ZAEZ4iDy5jMd4dSD+ksYAQK/NHjY1jYorIzHo1Rz4DLEiqA4SZryuYeIJ0E0I70i6X4hCiAHh3NPMpn4d85s0DgbSdA5nkdquHJaX0GKITHyelpUkU8ACJSS3J08AtJyEmwGZBOZktUy+ywkyAYZ9nJpxoZ45ptoX8LDpZ4JOUTiDkSsaMqKJQA8AYjLUwDzmImMq8G31upnxPsBi6WZjn9g4Rm57CqKvrlBLl5YxJhEQ55eo2tDshXOEoCFA4kg7Vp0NAIqTT3VS5d0X0VEnpZEa5rnmawSX0XQF9p1QZPCqARxoso/0VOjb7Vel5ATyvffWigF5UpW5cQOMaOiugs1plfWdu4KsanBTQpIamBTStodcF9LruM9LH4rI3kPO9yDdVHhN/Bq8YQKhkp6eqvTC935d2WZKdAdmxVW73CMb6pzMHgjSjy9dnb98oebDbe25tqyrDJTEsKdbW9vMDq0bx5qtZcKzd3d5+jRMxFHqrQodH2y/teiEaM0pWpJ2drV9+rldKgZDoalXpYP+wt1VtTDtSXpS9Y1tW9bQxXLNq4YjZsc2atV4TffGZRoHbJJj72cif1vmvGEq/wI6aqi9tblTQJsXS80ZF2qRYbsBSUZaEDTbdbMdKU+mcZhdhqscUzUYGIvO6h6CHCYO3+oIc6x8gEhF7pm8F8wJf19Jvp5tF/0dEyZKoo3ZbTza7OsfqwXlv097etN7/1Hmxv5hx94wfjCfGumEbO8YL47VxYvQNbHwy/jE+G/+2fmt9av3R+nNOvX1roXlslJ7WX/8B1OH0Wg==</latexit>

L(qv,pw) = Eq lnpw(x, z)

qv(z | x)

Hereisthedecomposition,rewritteninournewnotation.

IntheEMalgorithm,weusedthistosetupanalternatingoptimizationalgorithm:?irstoptimizingonetermandthentheother.Todothathere,we’dneedtominimizetheKLdivergencebetweenqandtheinverseofp.Thisispossible,butexpensive.Instead,we’llfollowacheapertrack.

what’s our loss?

78

argmaxw

X

x

lnpw(x)




qv(z)<latexit sha1_base64="TOTrmSgCR7aQgKgP+cLRA0oTIjU=">AAAJHHicfVXdbts2FJbbrfO8dWvWy90QM1Y4mxFIzppkBQJ0+VkLLG2yIE4KhIZB0bQsmJI4irKlEHyVXe5JejfsdsBeYbfbA4ySZVuylAkGfHy+Hx0eHpo2o24oTPOvxoOHH3z46KPmx61PPn382edPtr64DoOIY9LHAQ34OxuFhLo+6QtXUPKOcYI8m5Ibe3qc4jczwkM38K9EwsjAQ47vjl2MhE4NtxqDZxBxx0PxUEIHS2iPwVwpAMPIG8IYQOqDNM+GS6gD420AYUt/nt2DHoKzDuRY/jKE9kx1cwq052obfAugILGQP52pnJNJZ2rFW1qt35IJ5q6YvFAgS5R0HXgHoOeOgH71i9wd+QlAjPEgdr1spUAEQFVqXQsz3/urPgTwdJiB2mTMEZZrVC9Zk++2lVyrO+lv7Tl80jZ3zOwB1cDKg7aRPxfDrUe/wVGAI4/4AlMUhreWycRAIi5cTIlqwSgkDOEpcshtJMYHA+n6LBLExwp8rbFxRNO1pnsNRi4nWNBEBwhzVzsAPEG6eqEnolW2ComPPBJ2RzOXhYswnDmLQCA9TgMZZ+OmHpeU0uGITVwcl0qTyAt14yeVZJh4djlJIkr4zCsn0zJ1kRvMmHDshmkTLnRnzlm6seFVcJHjk4RNiB8qGXGqikINEM7JWAuzMCQiYjJbjT430/BQ8Ih00zDLHZ4gPr0ko672KSXK5YxpgEQ5Zetl6O74ZI4Dz0P+SEKmxyIbSdjdUVnviuilkhKmjbJtcJnCJfRtAX2rVBk8LYCnGiyj/RU6Bv1N6XUBvK689aaA3mxK7aiARhV0VkBnFWd9WNbwvALHBTSuoEkBTSroXQG9q/YZ6bG47Q3kYi+yTZXn1J2RV5wQX8l2T22uhev9vrXKknQGZNtSWbtHZKz/dBeAl6R0+frqzZmSxwe95+ae2mTYNCJLirm79/zYrFCcRTU5xzw46B1VOAFHvrMyOjnd+8GqGrGIM7oi7e/v/vh91SkhlAbzldPx0Ulvd3NhuiPloqx9yzQ3p43jSqvyjoC2BSqtdero+WtqBXadYNHPWv60yn/FUXIPO6hzX7a5VsHqFMue1yqSOsVyA5aKssSvadN6O1aajZWz9CBM9f3E0isD0YXvCdGXCSdv9AE513+ASAT8G5ld/K720t+wm0b/R0TxkqijVkvfbNbmPVYNrns7lrlj/fxd++VRfsc1jS+Nr4yOYRn7xkvjtXFh9A3ceN/4u/FP49/mr833zd+bfyyoDxq55qlRepp//gcY5071</latexit><latexit sha1_base64="TOTrmSgCR7aQgKgP+cLRA0oTIjU=">AAAJHHicfVXdbts2FJbbrfO8dWvWy90QM1Y4mxFIzppkBQJ0+VkLLG2yIE4KhIZB0bQsmJI4irKlEHyVXe5JejfsdsBeYbfbA4ySZVuylAkGfHy+Hx0eHpo2o24oTPOvxoOHH3z46KPmx61PPn382edPtr64DoOIY9LHAQ34OxuFhLo+6QtXUPKOcYI8m5Ibe3qc4jczwkM38K9EwsjAQ47vjl2MhE4NtxqDZxBxx0PxUEIHS2iPwVwpAMPIG8IYQOqDNM+GS6gD420AYUt/nt2DHoKzDuRY/jKE9kx1cwq052obfAugILGQP52pnJNJZ2rFW1qt35IJ5q6YvFAgS5R0HXgHoOeOgH71i9wd+QlAjPEgdr1spUAEQFVqXQsz3/urPgTwdJiB2mTMEZZrVC9Zk++2lVyrO+lv7Tl80jZ3zOwB1cDKg7aRPxfDrUe/wVGAI4/4AlMUhreWycRAIi5cTIlqwSgkDOEpcshtJMYHA+n6LBLExwp8rbFxRNO1pnsNRi4nWNBEBwhzVzsAPEG6eqEnolW2ComPPBJ2RzOXhYswnDmLQCA9TgMZZ+OmHpeU0uGITVwcl0qTyAt14yeVZJh4djlJIkr4zCsn0zJ1kRvMmHDshmkTLnRnzlm6seFVcJHjk4RNiB8qGXGqikINEM7JWAuzMCQiYjJbjT430/BQ8Ih00zDLHZ4gPr0ko672KSXK5YxpgEQ5Zetl6O74ZI4Dz0P+SEKmxyIbSdjdUVnviuilkhKmjbJtcJnCJfRtAX2rVBk8LYCnGiyj/RU6Bv1N6XUBvK689aaA3mxK7aiARhV0VkBnFWd9WNbwvALHBTSuoEkBTSroXQG9q/YZ6bG47Q3kYi+yTZXn1J2RV5wQX8l2T22uhev9vrXKknQGZNtSWbtHZKz/dBeAl6R0+frqzZmSxwe95+ae2mTYNCJLirm79/zYrFCcRTU5xzw46B1VOAFHvrMyOjnd+8GqGrGIM7oi7e/v/vh91SkhlAbzldPx0Ulvd3NhuiPloqx9yzQ3p43jSqvyjoC2BSqtdero+WtqBXadYNHPWv60yn/FUXIPO6hzX7a5VsHqFMue1yqSOsVyA5aKssSvadN6O1aajZWz9CBM9f3E0isD0YXvCdGXCSdv9AE513+ASAT8G5ld/K720t+wm0b/R0TxkqijVkvfbNbmPVYNrns7lrlj/fxd++VRfsc1jS+Nr4yOYRn7xkvjtXFh9A3ceN/4u/FP49/mr833zd+bfyyoDxq55qlRepp//gcY5071</latexit><latexit sha1_base64="TOTrmSgCR7aQgKgP+cLRA0oTIjU=">AAAJHHicfVXdbts2FJbbrfO8dWvWy90QM1Y4mxFIzppkBQJ0+VkLLG2yIE4KhIZB0bQsmJI4irKlEHyVXe5JejfsdsBeYbfbA4ySZVuylAkGfHy+Hx0eHpo2o24oTPOvxoOHH3z46KPmx61PPn382edPtr64DoOIY9LHAQ34OxuFhLo+6QtXUPKOcYI8m5Ibe3qc4jczwkM38K9EwsjAQ47vjl2MhE4NtxqDZxBxx0PxUEIHS2iPwVwpAMPIG8IYQOqDNM+GS6gD420AYUt/nt2DHoKzDuRY/jKE9kx1cwq052obfAugILGQP52pnJNJZ2rFW1qt35IJ5q6YvFAgS5R0HXgHoOeOgH71i9wd+QlAjPEgdr1spUAEQFVqXQsz3/urPgTwdJiB2mTMEZZrVC9Zk++2lVyrO+lv7Tl80jZ3zOwB1cDKg7aRPxfDrUe/wVGAI4/4AlMUhreWycRAIi5cTIlqwSgkDOEpcshtJMYHA+n6LBLExwp8rbFxRNO1pnsNRi4nWNBEBwhzVzsAPEG6eqEnolW2ComPPBJ2RzOXhYswnDmLQCA9TgMZZ+OmHpeU0uGITVwcl0qTyAt14yeVZJh4djlJIkr4zCsn0zJ1kRvMmHDshmkTLnRnzlm6seFVcJHjk4RNiB8qGXGqikINEM7JWAuzMCQiYjJbjT430/BQ8Ih00zDLHZ4gPr0ko672KSXK5YxpgEQ5Zetl6O74ZI4Dz0P+SEKmxyIbSdjdUVnviuilkhKmjbJtcJnCJfRtAX2rVBk8LYCnGiyj/RU6Bv1N6XUBvK689aaA3mxK7aiARhV0VkBnFWd9WNbwvALHBTSuoEkBTSroXQG9q/YZ6bG47Q3kYi+yTZXn1J2RV5wQX8l2T22uhev9vrXKknQGZNtSWbtHZKz/dBeAl6R0+frqzZmSxwe95+ae2mTYNCJLirm79/zYrFCcRTU5xzw46B1VOAFHvrMyOjnd+8GqGrGIM7oi7e/v/vh91SkhlAbzldPx0Ulvd3NhuiPloqx9yzQ3p43jSqvyjoC2BSqtdero+WtqBXadYNHPWv60yn/FUXIPO6hzX7a5VsHqFMue1yqSOsVyA5aKssSvadN6O1aajZWz9CBM9f3E0isD0YXvCdGXCSdv9AE513+ASAT8G5ld/K720t+wm0b/R0TxkqijVkvfbNbmPVYNrns7lrlj/fxd++VRfsc1jS+Nr4yOYRn7xkvjtXFh9A3ceN/4u/FP49/mr833zd+bfyyoDxq55qlRepp//gcY5071</latexit><latexit sha1_base64="TOTrmSgCR7aQgKgP+cLRA0oTIjU=">AAAJHHicfVXdbts2FJbbrfO8dWvWy90QM1Y4mxFIzppkBQJ0+VkLLG2yIE4KhIZB0bQsmJI4irKlEHyVXe5JejfsdsBeYbfbA4ySZVuylAkGfHy+Hx0eHpo2o24oTPOvxoOHH3z46KPmx61PPn382edPtr64DoOIY9LHAQ34OxuFhLo+6QtXUPKOcYI8m5Ibe3qc4jczwkM38K9EwsjAQ47vjl2MhE4NtxqDZxBxx0PxUEIHS2iPwVwpAMPIG8IYQOqDNM+GS6gD420AYUt/nt2DHoKzDuRY/jKE9kx1cwq052obfAugILGQP52pnJNJZ2rFW1qt35IJ5q6YvFAgS5R0HXgHoOeOgH71i9wd+QlAjPEgdr1spUAEQFVqXQsz3/urPgTwdJiB2mTMEZZrVC9Zk++2lVyrO+lv7Tl80jZ3zOwB1cDKg7aRPxfDrUe/wVGAI4/4AlMUhreWycRAIi5cTIlqwSgkDOEpcshtJMYHA+n6LBLExwp8rbFxRNO1pnsNRi4nWNBEBwhzVzsAPEG6eqEnolW2ComPPBJ2RzOXhYswnDmLQCA9TgMZZ+OmHpeU0uGITVwcl0qTyAt14yeVZJh4djlJIkr4zCsn0zJ1kRvMmHDshmkTLnRnzlm6seFVcJHjk4RNiB8qGXGqikINEM7JWAuzMCQiYjJbjT430/BQ8Ih00zDLHZ4gPr0ko672KSXK5YxpgEQ5Zetl6O74ZI4Dz0P+SEKmxyIbSdjdUVnviuilkhKmjbJtcJnCJfRtAX2rVBk8LYCnGiyj/RU6Bv1N6XUBvK689aaA3mxK7aiARhV0VkBnFWd9WNbwvALHBTSuoEkBTSroXQG9q/YZ6bG47Q3kYi+yTZXn1J2RV5wQX8l2T22uhev9vrXKknQGZNtSWbtHZKz/dBeAl6R0+frqzZmSxwe95+ae2mTYNCJLirm79/zYrFCcRTU5xzw46B1VOAFHvrMyOjnd+8GqGrGIM7oi7e/v/vh91SkhlAbzldPx0Ulvd3NhuiPloqx9yzQ3p43jSqvyjoC2BSqtdero+WtqBXadYNHPWv60yn/FUXIPO6hzX7a5VsHqFMue1yqSOsVyA5aKssSvadN6O1aajZWz9CBM9f3E0isD0YXvCdGXCSdv9AE513+ASAT8G5ld/K720t+wm0b/R0TxkqijVkvfbNbmPVYNrns7lrlj/fxd++VRfsc1jS+Nr4yOYRn7xkvjtXFh9A3ceN/4u/FP49/mr833zd+bfyyoDxq55qlRepp//gcY5071</latexit>

variational lower bound or evidence lower bound (ELBO)

<latexit sha1_base64="nU9B6EQ3wtWmsQJb95lKNXLOCnI=">AAAIHHicfVXLbtw2FFXSJpNOktZplt0IGTSwg4EhjRvbXRhI/GiySGLH8NgBrOmA4tzRCENJDEmNJRP8lS77JdkV3RboL3SbfkApzcN6pdro4p5zrngPSV2XEp8Ly/r71u2vvr5zt3Xvm/b9Bw+//W7t0ffnPIoZhj6OSMQ+uIgD8UPoC18Q+EAZoMAlcOFODzL8YgaM+1F4JlIKgwB5oT/2MRI6NVz79embdYdh+XHouDPVNR0PS6rjK7Vh7pnO0TAHlemQ0HTGDGF5w1h3Ei243lDypsK6c+0E/sh0kg0tcoZrHWvTyh+zHtiLoGMsnpPho7u/O6MIxwGEAhPE+aVtUTGQiAkfE1BtJ+ZAEZ4iDy5jMd4dSD+ksYAQK/NHjY1jYorIzHo1Rz4DLEiqA4SZryuYeIJ0E0I70i6X4hCiAHh3NPMpn4d85s0DgbSdA5nkdquHJaX0GKITHyelpUkU8ACJSS3J08AtJyEmwGZBOZktUy+ywkyAYZ9nJpxoZ45ptoX8LDpZ4JOUTiDkSsaMqKJQA8AYjLUwDzmImMq8G31upnxPsBi6WZjn9g4Rm57CqKvrlBLl5YxJhEQ55eo2tDshXOEoCFA4kg7Vp0NAIqTT3VS5d0X0VEnpZEa5rnmawSX0XQF9p1QZPCqARxoso/0VOjb7Vel5ATyvffWigF5UpW5cQOMaOiugs1plfWdu4KsanBTQpIamBTStodcF9LruM9LH4rI3kPO9yDdVHhN/Bq8YQKhkp6eqvTC935d2WZKdAdmxVW73CMb6pzMHgjSjy9dnb98oebDbe25tqyrDJTEsKdbW9vMDq0bx5qtZcKzd3d5+jRMxFHqrQodH2y/teiEaM0pWpJ2drV9+rldKgZDoalXpYP+wt1VtTDtSXpS9Y1tW9bQxXLNq4YjZsc2atV4TffGZRoHbJJj72cif1vmvGEq/wI6aqi9tblTQJsXS80ZF2qRYbsBSUZaEDTbdbMdKU+mcZhdhqscUzUYGIvO6h6CHCYO3+oIc6x8gEhF7pm8F8wJf19Jvp5tF/0dEyZKoo3ZbTza7OsfqwXlv097etN7/1Hmxv5hx94wfjCfGumEbO8YL47VxYvQNbHwy/jE+G/+2fmt9av3R+nNOvX1roXlslJ7WX/8B1OH0Wg==</latexit>

L(qv,pw) = Eq lnpw(x, z)

qv(z | x)

Becausewecannotinvertpw,wecannoteasilycomputetheKLterm(letaloneoptimiseqvtominimiseit).

Instead,wefocusentirelyontheLterm.Sinceit’salowerboundonthequantitywe’retryingtomaximize,anythingthatincreasesLwillhelpourmodel.ThebetterwemaximiseL,thebetterourmodelwilldo.

lower bound loss

79


<latexit sha1_base64="3XRj/pFskvBJRF4gPTtIOoP8T+Q=">AAAIGXicfZXLjts2FIaVNI1Tt2kn7bIbIUaLSWsMJE/m0sUAyVyaAJ1kpoPxTADLMCj62BZMSSxJyVYIPkn2fY/sim676hN0275BKcl2dGu18QG//z8mf1KiS4nHhWX9eefuR/c+vt968En7088efv7F1qMvb3gYMQx9HJKQvXERB+IF0BeeIPCGMkC+S+DWnZ+k/DYGxr0wuBYJhaGPpoE38TASemi0NfjWcbF0SGBKOpKOOzEXSm07yyfKPDKdEMvzbfnLyHFj1U0VjrtQGn1vOgKWQv50rlKc+WKVS9ZNnpijrY61Y2WPWS/sVdExVs/l6NH9X51xiCMfAoEJ4nxgW1QMJWLCwwRU24k4UITnaAqDSEwOh9ILaCQgwMr8RrNJREwRmulCzbHHAAuS6AJh5ukOJp4hhrDQcbTLrTgEyAfeHcce5XnJ42leCKSzHMpllrV6WHLKKUN05uFlaWoS+dxHYlYb5InvlgchIsBivzyYTlNPsqJcAsMeT0O41Mlc0HT/+HV4ueKzhM4g4EpGjKiiUQNgDCbamJUcRERlthp9aOb8SLAIummZjR2dIja/gnFX9ykNlKczISES5SFXL0OnE8ACh76PgrF0qJL5QXG6OyrLrkivlJROGpTrmlcpLtHXBfpaqTI8K8AzDcu0v6ETs1+13hTgTe1fbwv0tmp1owKNajQu0LjWWb85H/CihpcFuqzRpECTGn1boG/rOSN9LAa9ocz3IttUeUG8GF4wgEDJTk9V18L0fg/ssiU9A7JjqyzuMUz0FycHfpLK5cvrV+dKnhz29qx9VVW4JIK1xNrd3zuxapJpPpuVxjo87B3XNCFDwXTT6PRs/7ldb0QjRslGdHCw++MP9U4JEBIuNp1Ojk97u9WF6UTKk7IPbMuqnjaGa1GtEjE7tlmLdtokX/1No8FtMuR5Nurndf0LhpL/UIdN3dcxNzpok2OdeaMjaXKsN2DtKFuChpg+bMfGU1k5TV+Eub7PaHplIJL3PQV9mTB4pV+QC/0BRCJk3+m3gk19T/fSv043rf5PiJZroa7abX2z2dV7rF7c9Hbspzt7Pz/tPDte3XEPjK+Nx8a2YRsHxjPjpXFp9A1svDf+Mv42/mm9a71v/db6PZfevbPyfGWUntYf/wJsvPLR</latexit>







model space

Here’savisualisationofhowalowerboundobjectiveworks.We’reinterestedin?indingthehighestpointoftheblueline(themaximumlikelihoodsolution),butthat’sdif?iculttocompute.Instead,wemaximisetheorangeline(theevidencelowerbound).Becauseit’sguaranteedtobebelowthebluelineeverywhere,wemayexpecttobe?indingahighvalueforthebluelineaswell.Tosomeextent,pushinguptheorangeline,pushesupthebluelineaswell.

Howwellwedoonthebluelinedependsalotonhowtightthelowerboundis.ThedistancebetweenthelowerboundandtheloglikelihoodisexpressedbytheKLdivergencebetweenpw(z|x)andqv(z|x).Thatis,becausewecannoteasilycomputepw(z|x),weintroducedanapproximationqv(z|x).Thebetterthisapproximation,thelowertheKLdivergence,andthetighterthelowerbound.

minimize -L(v, w)

80

-L(v,w) = -Eq lnpw(x, z)

qv(z | x)

= -Eq lnpw(x | z)pw(z)

qv(z | x)

= -Eq lnpw(x | z)- Eq lnpw(z) + Eq lnqv(x | z)

= KL(qv(z | x),pw(z))- Eq lnpw(x | z)<latexit sha1_base64="8vpYl0cEYpC2j7kTO7U7D7Wvuz0=">AAAJQnicjVVPb+NEFHcDLCGw0MIJuIyIQA1kKztl23KotPQPuxK721I17UqdKBo7E8fK2B7G49TOaE58Fz4JX4KvsDfElQNjO8naHi/Clzy935958+ZNxqbEi7hp/rnVeufd9x683/6g8+FHDz/+ZHvn05sojJmDh05IQvbKRhEmXoCH3OMEv6IMI98m+Naen2b47QKzyAuDa55SPPKRG3hTz0FcpcY7W789As93IXMEtBeyD6CbRfeyB745Bo8APB+LDPxVSgBJAOCUIUVQJDrOaLsw6cNlTxakceaxC5cA+t4EwKSnRLCzNip83mZTSJY9UM7+T+dqiWVXsLGtEjWmYnxXL7K0cF5eXl2+6s9FyzZl5WjS69cse/rGG4sbb3fNPTP/gB5Yq6BrrL7L8c6D3+EkdGIfB9whKIruLJPykUCMew7BsgPjCFPkzJGL72I+PRoJL6Axx4EjwdcKm8YE8BBkEwEmHsMOJ6kKkMM85QCcGVIHxNXcdKpWEQ6Qj6P+ZOHRqAijhVsEHKmhG4kkH0r5sKIULkN05jlJpTSB/MhHfKYlo9S3q0kcE8wWfjWZlamKrDETzBwvyppwqTpzQbNBj67DyxU+S+kMB5EUMSOyLFQAZgxPlTAPI8xjKvLdqNs1j445i3E/C/Pc8Rli8ys86SufSqJazpSEiFdTttqG6k6A753Q91EwEZCqOec44QL292TeuzJ6JYWAWaNsG1xlcAV9WUJfSlkFz0vguQKr6HCDTsGwLr0pgTfaqrcl9LYuteMSGmvoooQuNGd1Od7A9xqclNBEQ9MSmmrosoQu9T4jNRZ3g5EoziI/VHFBvAV+yjAOpOgOZH0vTJ33nVWVZDMgupbM2z3BU/XXXAB+mtHFs+sXz6U4PRo8Ng9knWGTGK8p5v7B41NTo7hFNSuOeXQ0ONE4IUOBuzE6Oz/40dKNaMwo2ZAOD/d/+kF3SjEh4f3G6fTkbLBf35jqSLUo69Ayzfq0MUdr1aojoGsBrbVuE321TKPAbhIU/Wzkz3X+U4bSt7DDJvd1mxsVtEmx7nmjIm1SrA9grahKgoY2vTmOjaa2c5pdhLl6gmn2ZCBS+J5h9Zgw/EJdkAv1B4h4yL5Vt4K5vqe81C/sZ9F/EVGyJqqo01Evm1V/x/TgZrBnmXvWL993n5ys3ri28aXxlbFrWMah8cR4ZlwaQ8PZet3abn3e+qL9R/t1+6/23wW1tbXSfGZUvvY//wJYblfA</latexit><latexit sha1_base64="8vpYl0cEYpC2j7kTO7U7D7Wvuz0=">AAAJQnicjVVPb+NEFHcDLCGw0MIJuIyIQA1kKztl23KotPQPuxK721I17UqdKBo7E8fK2B7G49TOaE58Fz4JX4KvsDfElQNjO8naHi/Clzy935958+ZNxqbEi7hp/rnVeufd9x683/6g8+FHDz/+ZHvn05sojJmDh05IQvbKRhEmXoCH3OMEv6IMI98m+Naen2b47QKzyAuDa55SPPKRG3hTz0FcpcY7W789As93IXMEtBeyD6CbRfeyB745Bo8APB+LDPxVSgBJAOCUIUVQJDrOaLsw6cNlTxakceaxC5cA+t4EwKSnRLCzNip83mZTSJY9UM7+T+dqiWVXsLGtEjWmYnxXL7K0cF5eXl2+6s9FyzZl5WjS69cse/rGG4sbb3fNPTP/gB5Yq6BrrL7L8c6D3+EkdGIfB9whKIruLJPykUCMew7BsgPjCFPkzJGL72I+PRoJL6Axx4EjwdcKm8YE8BBkEwEmHsMOJ6kKkMM85QCcGVIHxNXcdKpWEQ6Qj6P+ZOHRqAijhVsEHKmhG4kkH0r5sKIULkN05jlJpTSB/MhHfKYlo9S3q0kcE8wWfjWZlamKrDETzBwvyppwqTpzQbNBj67DyxU+S+kMB5EUMSOyLFQAZgxPlTAPI8xjKvLdqNs1j445i3E/C/Pc8Rli8ys86SufSqJazpSEiFdTttqG6k6A753Q91EwEZCqOec44QL292TeuzJ6JYWAWaNsG1xlcAV9WUJfSlkFz0vguQKr6HCDTsGwLr0pgTfaqrcl9LYuteMSGmvoooQuNGd1Od7A9xqclNBEQ9MSmmrosoQu9T4jNRZ3g5EoziI/VHFBvAV+yjAOpOgOZH0vTJ33nVWVZDMgupbM2z3BU/XXXAB+mtHFs+sXz6U4PRo8Ng9knWGTGK8p5v7B41NTo7hFNSuOeXQ0ONE4IUOBuzE6Oz/40dKNaMwo2ZAOD/d/+kF3SjEh4f3G6fTkbLBf35jqSLUo69Ayzfq0MUdr1aojoGsBrbVuE321TKPAbhIU/Wzkz3X+U4bSt7DDJvd1mxsVtEmx7nmjIm1SrA9grahKgoY2vTmOjaa2c5pdhLl6gmn2ZCBS+J5h9Zgw/EJdkAv1B4h4yL5Vt4K5vqe81C/sZ9F/EVGyJqqo01Evm1V/x/TgZrBnmXvWL993n5ys3ri28aXxlbFrWMah8cR4ZlwaQ8PZet3abn3e+qL9R/t1+6/23wW1tbXSfGZUvvY//wJYblfA</latexit><latexit sha1_base64="8vpYl0cEYpC2j7kTO7U7D7Wvuz0=">AAAJQnicjVVPb+NEFHcDLCGw0MIJuIyIQA1kKztl23KotPQPuxK721I17UqdKBo7E8fK2B7G49TOaE58Fz4JX4KvsDfElQNjO8naHi/Clzy935958+ZNxqbEi7hp/rnVeufd9x683/6g8+FHDz/+ZHvn05sojJmDh05IQvbKRhEmXoCH3OMEv6IMI98m+Naen2b47QKzyAuDa55SPPKRG3hTz0FcpcY7W789As93IXMEtBeyD6CbRfeyB745Bo8APB+LDPxVSgBJAOCUIUVQJDrOaLsw6cNlTxakceaxC5cA+t4EwKSnRLCzNip83mZTSJY9UM7+T+dqiWVXsLGtEjWmYnxXL7K0cF5eXl2+6s9FyzZl5WjS69cse/rGG4sbb3fNPTP/gB5Yq6BrrL7L8c6D3+EkdGIfB9whKIruLJPykUCMew7BsgPjCFPkzJGL72I+PRoJL6Axx4EjwdcKm8YE8BBkEwEmHsMOJ6kKkMM85QCcGVIHxNXcdKpWEQ6Qj6P+ZOHRqAijhVsEHKmhG4kkH0r5sKIULkN05jlJpTSB/MhHfKYlo9S3q0kcE8wWfjWZlamKrDETzBwvyppwqTpzQbNBj67DyxU+S+kMB5EUMSOyLFQAZgxPlTAPI8xjKvLdqNs1j445i3E/C/Pc8Rli8ys86SufSqJazpSEiFdTttqG6k6A753Q91EwEZCqOec44QL292TeuzJ6JYWAWaNsG1xlcAV9WUJfSlkFz0vguQKr6HCDTsGwLr0pgTfaqrcl9LYuteMSGmvoooQuNGd1Od7A9xqclNBEQ9MSmmrosoQu9T4jNRZ3g5EoziI/VHFBvAV+yjAOpOgOZH0vTJ33nVWVZDMgupbM2z3BU/XXXAB+mtHFs+sXz6U4PRo8Ng9knWGTGK8p5v7B41NTo7hFNSuOeXQ0ONE4IUOBuzE6Oz/40dKNaMwo2ZAOD/d/+kF3SjEh4f3G6fTkbLBf35jqSLUo69Ayzfq0MUdr1aojoGsBrbVuE321TKPAbhIU/Wzkz3X+U4bSt7DDJvd1mxsVtEmx7nmjIm1SrA9grahKgoY2vTmOjaa2c5pdhLl6gmn2ZCBS+J5h9Zgw/EJdkAv1B4h4yL5Vt4K5vqe81C/sZ9F/EVGyJqqo01Evm1V/x/TgZrBnmXvWL993n5ys3ri28aXxlbFrWMah8cR4ZlwaQ8PZet3abn3e+qL9R/t1+6/23wW1tbXSfGZUvvY//wJYblfA</latexit><latexit sha1_base64="8vpYl0cEYpC2j7kTO7U7D7Wvuz0=">AAAJQnicjVVPb+NEFHcDLCGw0MIJuIyIQA1kKztl23KotPQPuxK721I17UqdKBo7E8fK2B7G49TOaE58Fz4JX4KvsDfElQNjO8naHi/Clzy935958+ZNxqbEi7hp/rnVeufd9x683/6g8+FHDz/+ZHvn05sojJmDh05IQvbKRhEmXoCH3OMEv6IMI98m+Naen2b47QKzyAuDa55SPPKRG3hTz0FcpcY7W789As93IXMEtBeyD6CbRfeyB745Bo8APB+LDPxVSgBJAOCUIUVQJDrOaLsw6cNlTxakceaxC5cA+t4EwKSnRLCzNip83mZTSJY9UM7+T+dqiWVXsLGtEjWmYnxXL7K0cF5eXl2+6s9FyzZl5WjS69cse/rGG4sbb3fNPTP/gB5Yq6BrrL7L8c6D3+EkdGIfB9whKIruLJPykUCMew7BsgPjCFPkzJGL72I+PRoJL6Axx4EjwdcKm8YE8BBkEwEmHsMOJ6kKkMM85QCcGVIHxNXcdKpWEQ6Qj6P+ZOHRqAijhVsEHKmhG4kkH0r5sKIULkN05jlJpTSB/MhHfKYlo9S3q0kcE8wWfjWZlamKrDETzBwvyppwqTpzQbNBj67DyxU+S+kMB5EUMSOyLFQAZgxPlTAPI8xjKvLdqNs1j445i3E/C/Pc8Rli8ys86SufSqJazpSEiFdTttqG6k6A753Q91EwEZCqOec44QL292TeuzJ6JYWAWaNsG1xlcAV9WUJfSlkFz0vguQKr6HCDTsGwLr0pgTfaqrcl9LYuteMSGmvoooQuNGd1Od7A9xqclNBEQ9MSmmrosoQu9T4jNRZ3g5EoziI/VHFBvAV+yjAOpOgOZH0vTJ33nVWVZDMgupbM2z3BU/XXXAB+mtHFs+sXz6U4PRo8Ng9knWGTGK8p5v7B41NTo7hFNSuOeXQ0ONE4IUOBuzE6Oz/40dKNaMwo2ZAOD/d/+kF3SjEh4f3G6fTkbLBf35jqSLUo69Ayzfq0MUdr1aojoGsBrbVuE321TKPAbhIU/Wzkz3X+U4bSt7DDJvd1mxsVtEmx7nmjIm1SrA9grahKgoY2vTmOjaa2c5pdhLl6gmn2ZCBS+J5h9Zgw/EJdkAv1B4h4yL5Vt4K5vqe81C/sZ9F/EVGyJqqo01Evm1V/x/TgZrBnmXvWL993n5ys3ri28aXxlbFrWMah8cR4ZlwaQ8PZet3abn3e+qL9R/t1+6/23wW1tbXSfGZUvvY//wJYblfA</latexit>


qv(z | x)


qv(z | x)




qv(z | x)


qv(z | x)



pw(z) = N(0, I)<latexit sha1_base64="dJbgrQHMFLiJtBnvMqNa4YibEgs=">AAAH4nicfVVNb9w2EFXSNJtsm8ZJj70I2RRwioUhrRPbPRhI/NGkQGO7htcOYC0MijurFZaSCJJar0zwD+QW5Npzz70m/yT/ppT2I5KolhcN5r03Gr4RRZ+SkAvH+XLr9jd3vr3bune//d33D354uPbo8TlPUoahjxOSsHc+4kDCGPoiFATeUQYo8glc+JP9HL+YAuNhEp+JjMIgQkEcjkKMhE5drT31Aizpledfq3Xv5pm9ax+tS88f2Y7q2kXwu3p2tdZxNpxi2WbgLoKOtVgnV4/u/u0NE5xGEAtMEOeXrkPFQCImQkxAtb2UA0V4ggK4TMVoZyDDmKYCYqzsnzU2SoktEjvv2B6GDLAgmQ4QZqGuYOMxYggLva92tRSHGEXAu8NpSPk85NNgHgikTRnIWWGaelBRyoAhOg7xrNKaRBGPkBgbSZ5FfjUJKQE2jarJvE3dZI05A4ZDnptwop05pvkg+FlyssDHGR1DzJVMGVFloQaAMRhpYRFyECmVxW709Cd8V7AUunlY5HYPEJucwrCr61QS1XZGJEGimvL1NrQ7MVzjJIpQPJQeVdITMBPS626owrsyeqqk9HKjfN8+zeEKelRCj5Sqgocl8FCDVbS/Qkd2vy49L4HnxlsvSuhFXeqnJTQ10GkJnRqV9VH5Cl8b8KyEzgw0K6GZgd6U0BvTZ6Q/i8veQM5nUQxVHpNwCq8ZQKxkp6fqe2F63pduVZJ/A7LjqsLuIYz0r2MORFlOl2/O3v6h5P5O74WzpeoMn6SwpDibWy/2HYMSzLtZcJydnd6ewUkYioNVoYPDrVeuWYimjJIVaXt787dfzUoZEJJcryrt7x30Nusb045Um3K3Xcepf20MG1YtHLE7rm1YGzTRF69pFPhNgrmfjfyJyX/NUPYf7KSp+tLmRgVtUiw9b1RkTYrlAJaKqiRusOnrOFaa2s5pfhAmWPeYXxmIzOsegL5MGLzVB+RY/wCRSNgv+lSwIAp1Lf30unn0f0Q0WxJ11G7rm82t32NmcN7bcJ0N98/nnZd7izvunvWT9cRat1xr23ppvbFOrL6FrffWP9Yn63Nr2Hrf+tD6OKfevrXQ/GhVVuuvfwHBG9vB</latexit><latexit sha1_base64="dJbgrQHMFLiJtBnvMqNa4YibEgs=">AAAH4nicfVVNb9w2EFXSNJtsm8ZJj70I2RRwioUhrRPbPRhI/NGkQGO7htcOYC0MijurFZaSCJJar0zwD+QW5Npzz70m/yT/ppT2I5KolhcN5r03Gr4RRZ+SkAvH+XLr9jd3vr3bune//d33D354uPbo8TlPUoahjxOSsHc+4kDCGPoiFATeUQYo8glc+JP9HL+YAuNhEp+JjMIgQkEcjkKMhE5drT31Aizpledfq3Xv5pm9ax+tS88f2Y7q2kXwu3p2tdZxNpxi2WbgLoKOtVgnV4/u/u0NE5xGEAtMEOeXrkPFQCImQkxAtb2UA0V4ggK4TMVoZyDDmKYCYqzsnzU2SoktEjvv2B6GDLAgmQ4QZqGuYOMxYggLva92tRSHGEXAu8NpSPk85NNgHgikTRnIWWGaelBRyoAhOg7xrNKaRBGPkBgbSZ5FfjUJKQE2jarJvE3dZI05A4ZDnptwop05pvkg+FlyssDHGR1DzJVMGVFloQaAMRhpYRFyECmVxW709Cd8V7AUunlY5HYPEJucwrCr61QS1XZGJEGimvL1NrQ7MVzjJIpQPJQeVdITMBPS626owrsyeqqk9HKjfN8+zeEKelRCj5Sqgocl8FCDVbS/Qkd2vy49L4HnxlsvSuhFXeqnJTQ10GkJnRqV9VH5Cl8b8KyEzgw0K6GZgd6U0BvTZ6Q/i8veQM5nUQxVHpNwCq8ZQKxkp6fqe2F63pduVZJ/A7LjqsLuIYz0r2MORFlOl2/O3v6h5P5O74WzpeoMn6SwpDibWy/2HYMSzLtZcJydnd6ewUkYioNVoYPDrVeuWYimjJIVaXt787dfzUoZEJJcryrt7x30Nusb045Um3K3Xcepf20MG1YtHLE7rm1YGzTRF69pFPhNgrmfjfyJyX/NUPYf7KSp+tLmRgVtUiw9b1RkTYrlAJaKqiRusOnrOFaa2s5pfhAmWPeYXxmIzOsegL5MGLzVB+RY/wCRSNgv+lSwIAp1Lf30unn0f0Q0WxJ11G7rm82t32NmcN7bcJ0N98/nnZd7izvunvWT9cRat1xr23ppvbFOrL6FrffWP9Yn63Nr2Hrf+tD6OKfevrXQ/GhVVuuvfwHBG9vB</latexit><latexit sha1_base64="dJbgrQHMFLiJtBnvMqNa4YibEgs=">AAAH4nicfVVNb9w2EFXSNJtsm8ZJj70I2RRwioUhrRPbPRhI/NGkQGO7htcOYC0MijurFZaSCJJar0zwD+QW5Npzz70m/yT/ppT2I5KolhcN5r03Gr4RRZ+SkAvH+XLr9jd3vr3bune//d33D354uPbo8TlPUoahjxOSsHc+4kDCGPoiFATeUQYo8glc+JP9HL+YAuNhEp+JjMIgQkEcjkKMhE5drT31Aizpledfq3Xv5pm9ax+tS88f2Y7q2kXwu3p2tdZxNpxi2WbgLoKOtVgnV4/u/u0NE5xGEAtMEOeXrkPFQCImQkxAtb2UA0V4ggK4TMVoZyDDmKYCYqzsnzU2SoktEjvv2B6GDLAgmQ4QZqGuYOMxYggLva92tRSHGEXAu8NpSPk85NNgHgikTRnIWWGaelBRyoAhOg7xrNKaRBGPkBgbSZ5FfjUJKQE2jarJvE3dZI05A4ZDnptwop05pvkg+FlyssDHGR1DzJVMGVFloQaAMRhpYRFyECmVxW709Cd8V7AUunlY5HYPEJucwrCr61QS1XZGJEGimvL1NrQ7MVzjJIpQPJQeVdITMBPS626owrsyeqqk9HKjfN8+zeEKelRCj5Sqgocl8FCDVbS/Qkd2vy49L4HnxlsvSuhFXeqnJTQ10GkJnRqV9VH5Cl8b8KyEzgw0K6GZgd6U0BvTZ6Q/i8veQM5nUQxVHpNwCq8ZQKxkp6fqe2F63pduVZJ/A7LjqsLuIYz0r2MORFlOl2/O3v6h5P5O74WzpeoMn6SwpDibWy/2HYMSzLtZcJydnd6ewUkYioNVoYPDrVeuWYimjJIVaXt787dfzUoZEJJcryrt7x30Nusb045Um3K3Xcepf20MG1YtHLE7rm1YGzTRF69pFPhNgrmfjfyJyX/NUPYf7KSp+tLmRgVtUiw9b1RkTYrlAJaKqiRusOnrOFaa2s5pfhAmWPeYXxmIzOsegL5MGLzVB+RY/wCRSNgv+lSwIAp1Lf30unn0f0Q0WxJ11G7rm82t32NmcN7bcJ0N98/nnZd7izvunvWT9cRat1xr23ppvbFOrL6FrffWP9Yn63Nr2Hrf+tD6OKfevrXQ/GhVVuuvfwHBG9vB</latexit><latexit sha1_base64="dJbgrQHMFLiJtBnvMqNa4YibEgs=">AAAH4nicfVVNb9w2EFXSNJtsm8ZJj70I2RRwioUhrRPbPRhI/NGkQGO7htcOYC0MijurFZaSCJJar0zwD+QW5Npzz70m/yT/ppT2I5KolhcN5r03Gr4RRZ+SkAvH+XLr9jd3vr3bune//d33D354uPbo8TlPUoahjxOSsHc+4kDCGPoiFATeUQYo8glc+JP9HL+YAuNhEp+JjMIgQkEcjkKMhE5drT31Aizpledfq3Xv5pm9ax+tS88f2Y7q2kXwu3p2tdZxNpxi2WbgLoKOtVgnV4/u/u0NE5xGEAtMEOeXrkPFQCImQkxAtb2UA0V4ggK4TMVoZyDDmKYCYqzsnzU2SoktEjvv2B6GDLAgmQ4QZqGuYOMxYggLva92tRSHGEXAu8NpSPk85NNgHgikTRnIWWGaelBRyoAhOg7xrNKaRBGPkBgbSZ5FfjUJKQE2jarJvE3dZI05A4ZDnptwop05pvkg+FlyssDHGR1DzJVMGVFloQaAMRhpYRFyECmVxW709Cd8V7AUunlY5HYPEJucwrCr61QS1XZGJEGimvL1NrQ7MVzjJIpQPJQeVdITMBPS626owrsyeqqk9HKjfN8+zeEKelRCj5Sqgocl8FCDVbS/Qkd2vy49L4HnxlsvSuhFXeqnJTQ10GkJnRqV9VH5Cl8b8KyEzgw0K6GZgd6U0BvTZ6Q/i8veQM5nUQxVHpNwCq8ZQKxkp6fqe2F63pduVZJ/A7LjqsLuIYz0r2MORFlOl2/O3v6h5P5O74WzpeoMn6SwpDibWy/2HYMSzLtZcJydnd6ewUkYioNVoYPDrVeuWYimjJIVaXt787dfzUoZEJJcryrt7x30Nusb045Um3K3Xcepf20MG1YtHLE7rm1YGzTRF69pFPhNgrmfjfyJyX/NUPYf7KSp+tLmRgVtUiw9b1RkTYrlAJaKqiRusOnrOFaa2s5pfhAmWPeYXxmIzOsegL5MGLzVB+RY/wCRSNgv+lSwIAp1Lf30unn0f0Q0WxJ11G7rm82t32NmcN7bcJ0N98/nnZd7izvunvWT9cRat1xr23ppvbFOrL6FrffWP9Yn63Nr2Hrf+tD6OKfevrXQ/GhVVuuvfwHBG9vB</latexit>

= -Eq lnpw(x | z)- Eq lnpw(z) + Eq lnqv(z | x)

<latexit sha1_base64="Fm7f0965uFsBBOPCo6nrjHgJJao=">AAAIMnicfZXdbts2GIbldqtbb93S9XAnwowN7eYGktP87CBAl5+1B2uTBXFSIDIMiv4sC6Z+RlKOFYK3syvYzaxnw053ESMt25VEbTrxBz7v+5l8SYl+SkLGHefP1r37n3z6oP3wUeezzx9/8eXWk6+uWJJRDAOckIS+9xEDEsYw4CEn8D6lgCKfwLU/O9b8eg6UhUl8yfMUhhEK4nASYsTV0GiLf3dov7C905HwKBa/SWl7JLa9AIt05Pm38pm3sL0oHNve3fOa0FAqxQ9aUQgKrEuF5xrrPt7iud0ZbXWdbWf52GbhroqutXrOR08e/O6NE5xFEHNMEGM3rpPyoUCUh5iA7HgZgxThGQrgJuOTg6EI4zTjEGNpf6vYJCM2T2ydgD0OKWBOclUgTEPVwcZTRBHmKqdOtRWDGEXAeuN5mLKiZPOgKDhSIQ/FYrkJ8nHFKQKK0mmIF5WpCRSxCPGpMcjyyK8OQkaAzqPqoJ6mmmRNuQCKQ6ZDOFfJnKV6Y9llcr7i0zydQsykyCiRZaMCQClMlHFZMuBZKparUadpxg45zaCny+XY4QmiswsY91SfykB1OhOSIF4d8tUyVDox3OIkilA8Fl4qhcdhwYXX25bL7Mr0Qgrh6aB8377QuELfleg7KavwtARPFazSwYZO7EHdelWCV8a/Xpfodd3qZyWaGXReonOjs3p3PuJbAy9KdGHQvERzg96V6J2ZM1LH4qY/FMVeLDdVnJFwDq8pQCxFty/ra6Fqv2/cqkWfAdF15TLuMUzUp6gAUa7l4s3l21+kOD7o7zp7sq7wSQZribOzt3vsGJKgmM1K4xwc9I8MTUJRHGwanZzu/eSajdKMpmQj2t/f+flHs1MOhCS3m07HRyf9nfrCVCLVSbn7ruPUTxvFRlSrROyuaxvRBk3y1d80GvwmQ5Fno35m6l9TlP+HOmnqvo650ZE2OdaZNzryJsd6A9aOqiVuiOnjdmw8tZWn+kWYYTVHfWUgUvQ9AXWZUHirXpAz9QFEPKHfq7eCBlGoeqlfr6er/xOixVqoqo6+2dz6PWYWV/1t9+X27q8vu6+OVnfcQ+tr6xvrmeVa+9Yr6411bg0sbH1oWa1HrU77j/aH9l/tvwvpvdbK89SqPO1//gW92PhW</latexit>

<latexit sha1_base64="W1wlM+nQu/nKa4RSkP1wZGgLqm8=">AAAJnXicjVXdcttEFFYMlNpQ2sAlF+zggYnBzUgOTcJFZkoS0zKkScjEcWcij2clr22NVz+sVo7knX0d3okbnoUjyXYlrQrVjc+c72fPnrPrtQLqhFzX/95pfPTxJ48+fdxsffb5ky+ePtv98i70I2aTge1Tn721cEio45EBdzglbwNGsGtRMrQWZyk+XBIWOr53y5OAjFw885ypY2MOqfHuzj/P0cWeyWxhWkvZReYsjR5kB31/gp4jsz8WKfinlMikHjKnDAMBSME4pe2ZcddcdWROGqcee+YKma4zQWbcAZHZ2hjlPu+zySWrDipmP9C5XGLRFW1ty0SFCYwfq0UWFs7Ky6rLVv09b9m2rAyNO92KZUfdeG1x/2N6uQcjcZEO48mC32Tng63Hz9r6vp59SA2MddDW1t/1ePfRX+bEtyOXeNymOAzvDT3gI4EZd2xKZMuMQhJge4Fn5D7i0+ORcLwg4sSzJfoOsGlEEfdRetjQxGHE5jSBANvMAQdkzzHMnsORbJWtQuJhl4TdydIJwjwMl7M84BjO80jE2XmXT0pKMWM4mDt2XCpNYDd0MZ8ryTBxrXKSRJSwpVtOpmVCkRVmTJjthGkTrqEzV0F6h8Jb/3qNz5NgTrxQiohRWRQCQBgjUxBmYUh4FIhsN3BxF+EJZxHppmGWOznHbHFDJl3wKSXK5Uypj3k5ZcE2oDseebB918XeRJgBXCFOYi7M7r7MeldEb6QQZtooy0I3KVxCLwvopZRlsF8A+wCW0cEWnaJBVXpXAO+UVYcFdFiVWlEBjRR0WUCXijNcjnfwgwLHBTRW0KSAJgq6KqArtc8YjsV9byTyWWRDFVfUWZJXjBBPinZPVvfCYN73RlmSngHRNmTW7gmZwr9+DrhJShevb99cSHF23HuhH8oqw6IR2VD0g8MXZ7pCmeXVrDn68XHvVOH4DHuzrdF5//AXQzUKIhbQLeno6ODXn1WnhFDqP2ydzk7PewfVjUFHykUZR4auV08bs5VWrTuC2gZSWjuro6+XqRVYdYK8n7X8hcp/xXDyHrZf575pc60iqFNsel6rSOoUmwFsFGWJV9Omd+PYaio7D9KLsIDXPUifDExz33MCjwkjb+CCXMEfIOY++wFuBZu5DnjBr9lNo/8i4nhDhKjVgpfNqL5janDX2zcO9/U/fmq/PF2/cY+1r7VvtT3N0I60l9pr7VobaHaj31g0eCNqftPsNy+alzm1sbPWfKWVvubwXyx+cxM=</latexit>


qv(z | x)


qv(z | x)


= KL(qv(z | x),pw(z))- Eq lnpw(x | z)

= KL(qv(z | x),N(0, I)))- Eq lnpw(x | z)

Rightnow,wecan’tuseLasalossfunctiondirectly,sinceitcontainsfunctionslikep(x,z)thatarenoteasilycomputedandbecauseit’sanexpectation.We’llrewriteitstepbystepuntilit’salossfunctionthatcandedirectlyused,heaplyandeasilyinadeeplearningsystem.

Sincewewanttoimplementalossfunction,wewillminimizethenegativeoftheLfunction.Wenowneedtoworkoutwhatthatmeansforourneuralnet.

Allthreeprobabilityfunctionsareknown:q(z|x)istheencodernetwork,p(x|z)isthedecodernetwork,andp(z)waschosenwhenwede?ined(backinslide9)howthegeneratorworks,ifweignorex,thedistributionon

zisastandardmultivariatenormal.

NB:Sincewenowtaketheexpectationoveracontinuousdistribution,allsumsbecomeintegralsand(insideallexpectationsandinsidetheKLdivergence).Bykeepingeverythingwrittenasexpectations,weensurethatwedon’thavetodealwithintegrals.

loss function

81

loss(v,w) = KL(qv(z | x),pw(z))- Eq lnpw(x | z)<latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit><latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit><latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit><latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit>


qv(z|x) w v<latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit><latexit sha1_base64="p7xf3bGFsx4zGm4eABZ66MyF7RI=">AAAHWXicfVVdb9MwFA0D1lEYjI83XiIqJISqKulgG0KTxjY+HmAb07ohLdXkuLdpVCexbKdrZvm/8Ar/CIkfg9OPkdQB5yFX95xzde+xZfuUhFw4zq8bSzdv3V6urdyp3723ev/B2sNHpzxJGYYOTkjCvvmIAwlj6IhQEPhGGaDIJ3DmD/dy/GwEjIdJfCIyCt0IBXHYDzESOnWx9sQLsPT8S+W91R/L45G6WGs4LWeybDNwZ0HDmq2ji4fLLa+X4DSCWGCCOD93HSq6EjERYgKq7qUcKMJDFMB5KvpbXRnGNBUQY2U/11g/JbZI7LxDuxcywIJkOkCYhbqCjQeIISz0HPVyKQ4xioA3e6OQ8mnIR8E0EEib0JXjiUlqtaSUAUN0EOJxqTWJIh4hMTCSPIv8chJSAmwUlZN5m7rJBeYYGA55bsKRduaQ5sbzk+Rohg8yOoCYK5kyoopCDQBj0NfCSchBpFROptG7PeTbgqXQzMNJbnsfseEx9Jq6TilRbqdPEiTKKV+Pod2J4RInUYTinvSokp6AsZBes6Um3hXRYyWllxvl+/ZxDpfQgwJ6oNSCtnON9u2ORkvgaQE8NQqfFdCzRamfFtDUQEcFdGRU1uf/L3xpwOMCOjbQrIBmBnpVQK9MK5He+fN2V07tnuybPCThCD4ygFjJRlstzsL0lp67ZUm+zbLhqondPejr22AKRFlOl59OvnxWcm+r/drZUIsMn6QwpzjrG6/3HIMSTLuZcZytrfauwUkYioPrQvvvN965ZiGaMkquSZub6x/emJUyICS5vK60t7vfXl88RwwbJsxmtRuubZgWVNFnU1UK/CrB1KlK/tDkf2Qo+wc7qao+N7BSQasUczcrFVmVYm7tXLEwBM1P61C/AjS/uhGZUvZBX+oMvuhTfKgvIiQS9lIfXRZEobZP/71mHv2PiMZzoo7qdf3CuIvviRl02q03Lffrq8bO7uypWbGeWs+sF5ZrbVo71ifryOpY2Lqyvls/rJ/Lv2tLtZVafUpdujHTPLZKq/b4Dzi5qFc=</latexit>

N(0, I)µz �z

<latexit sha1_base64="AQPHvLMJL7+fz4fFfohCuzRIjg4=">AAAHcXicfVXbbttGEKWT1kqVpHGSp6IvhIUAQSMIpBxfgiJAajuXh/hSw7IDmIKxXI0oQktysbuURS/2I/o1fW0/I9+RH8iSogxSy5Z60GDOOYOZs4Ndn5KQC8f5unbv/g8/rrce/NR++Ojxz082nj674EnKMAxwQhL2xUccSBjDQISCwBfKAEU+gUt/epDjlzNgPEzic5FRGEYoiMNxiJHQqeuNV16CpedH0otSpa69W9v7Pf/Znl/meRhEqICuNzpOzyk+2wzcMuhY5Xd6/XS9540SnEYQC0wQ51euQ8VQIiZCTEC1vZQDRXiKArhKxXhvKMOYpgJirOwXGhunxBaJnTduj0IGWJBMBwizUFew8QQxhIUer10vxSFGEfDuaBZSvgj5LFgEAmlvhnJeeKce15QyYIhOQjyvtSZRxCMkJkaSZ5FfT0JKgM2iejJvUze5wpwDwyHPTTjVzpzQ/Dz4eXJa4pOMTiDmSqaMqKpQA8AYjLWwCDmIlMpiGr0EU/5WsBS6eVjk3h4iNj2DUVfXqSXq7YxJgkQ95esxtDsx3OAkilA8kh5V0hMwF9Lr9lThXRU9U1KvkTbK9+2zHK6hxxX0WKkV7eAOHdsDjdbAiwp4YRS+rKCXq1I/raCpgc4q6Myo7N9U4BsDnlfQuYFmFTQz0NsKemtaifTJX/WHcmF3cW7yhIQz+MgAYiU7fbU6C9NHeuXWJfkxy46rCrtHMNaXxAKIspwuP50ffVbyYK+/7eyoVYZPUlhSnK2d7QPHoASLbkqOs7fX3zc4CUNxcFfo8P3OH65ZiKaMkjvS7u7WhzdmpQwISW7uKh3sH/a3VveIYcOEcla749qGaUETvZyqUeA3CRZONfKnJv8jQ9l/sJOm6ksDGxW0SbF0s1GRNSmW1i4VK0PQfFun+kGg+dWNyIJyCPpSZ3Ckt/hEX0RIJOw3vbosiEJtn/73unn0f0Q0XxJ11G7rF8ZdfU/M4KLfc1/3tv983Xm3X741D6xfrU3rpeVau9Y765N1ag0sbP1l/W39Y/27/q31S8tubS6o99ZKzXOr9rVefQcKJbIy</latexit>

TheKLtermisjusttheKLdivergencebetweentheMVNthattheencoderproduceforxandthestandardnormalMVN.Thisworksoutasarelativelysimpledifferentiablefunctionofmuandsigma,sowecanuseitdirectlyinalossfunction.

RememberthatSigma_zisusuallyrestrictedtoadiagonalmatrix,sothenetworkjustoutputsavectorofthesamesizeasmu_z,whichwetaketobethediagonalofthecovariancematrix.

82




�x

<latexit sha1_base64="fS8LkWLR7/gt5QWBKM8fawfsBzk=">AAAHWXicfVVdb9MwFM0GrKN8DfbIS0SFhFBVJR374GHS2AbjAdiY1m3SUk2Oe5tGdRLLdrpmln8Lr/CTEH8Gpx9TUgfykqt7zrm691zL9ikJuXCc30vL9+4/WKmtPqw/evzk6bO15y/OeZIyDB2ckIRd+ogDCWPoiFAQuKQMUOQTuPCHBzl+MQLGwyQ+ExmFboSCOOyHGAmdul5b93wsPT+SHg+DCCl17Y2v1xpOy5l8thm4s6Bhzb6T6+crLa+X4DSCWGCCOL9yHSq6EjERYgKq7qUcKMJDFMBVKvo7XRnGNBUQY2W/1lg/JbZI7LxBuxcywIJkOkCYhbqCjQeIISz0GPVyKQ4xioA3e6OQ8mnIR8E0EEh70JXjiUfqSUkpA4boIMTjUmsSRTxCYmAkeRb55SSkBNgoKifzNnWTC8wxMBzy3IQT7cwxzX3nZ8nJDB9kdAAxVzJlRBWFGgDGoK+Fk5CDSKmcTKOXPeS7gqXQzMNJbvcQseEp9Jq6TilRbqdPEiTKKV+Pod2J4QYnUYTinvSokp6AsZBes6Um3hXRUyWllxvl+/ZpDpfQbwX0m1IL2s4d2rc7Gi2B5wXw3Ch8UUAvFqV+WkBTAx0V0JFR2b8pwDcGPC6gYwPNCmhmoLcF9Na0EunNX7W7cmr3ZG/ymIQjOGIAsZKNtlqchemVXrllSb5m2XDVxO4e9PVlMAWiLKfLz2dfvyh5sNPedLbUIsMnKcwpzsbW5oFjUIJpNzOOs7PT3jc4CUNxcFfo8OPWB9csRFNGyR1pe3vj03uzUgaEJDd3lQ72D9sbi+eIYcOE2ax2w7UN04Iq+myqSoFfJZg6VckfmvwjhrJ/sJOq6nMDKxW0SjF3s1KRVSnm1s4VC0PQ/LQO9YNA86sbkSnlEPSlzuCrPsXH+iJCImFv9dFlQRRq+/Tfa+bR/4hoPCfqqF7XL4y7+J6YwXm75b5rbX5/19jbn701q9ZL65X1xnKtbWvP+mydWB0LW5n1w/pp/Vr5U1uqrdbqU+ry0kyzbpW+2vpfrVCofw==</latexit>

take L samples {zi} from q(z|x).

approximate as:

keep things simple: L=1

loss(v,w) = KL(qv(z | x),pw(z))- Eq lnpw(x | z)<latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit><latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit><latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit><latexit sha1_base64="4S8b9OtyuxCWD2+8dzTuSyWDzB4=">AAAIMHicfZXdbts2GIaVdqszb13T9XAnxIwNyeAFkrMm2UGALj9rgbVNFsRJgcgwKJqWBVMSR1KOFIK3swvY1XRHw053EcNIyXb1t+nEH/i872fyJSV6lARc2PYfGw8efvTxo87mJ91PP3v8+ZOtp19c8zhhCA9RTGL2zoMckyDCQxEIgt9RhmHoEXzjzU8Mv1lgxoM4uhIZxaMQ+lEwDRAUemi8xVyBUyFJzLnadhmSrrdQfeD6prpTO+CbI/Dz65z8OjZs2713w2DipjuFio6NTo/u7IDvgHs2zqUKuCSq8BQYG9C68VbP3rXzBzQLZ1n0rOVzMX766Dd3EqMkxJFABHJ+69hUjCRkIkAEq66bcEwhmkMf3yZiejiSQUQTgSOkwNeaTRMCRAzM+sEkYBgJkukCIhboDgDNIINI6JS61VYcRzDEvD9ZBJQXJV/4RSGgjngk03wL1OOKU/oM0lmA0srUJAx5CMWsMciz0KsO4oRgtgirg2aaepI1ZYoZCrgJ4UInc07NtvKr+GLJZxmd4YgrmTCiykYNMGN4qo15ybFIqMxXo8/SnB8JluC+KfOxo1PI5pd40td9KgPV6UxJDEV1yNPL0OlE+A7FYQijiXSpksWxc/u7Ks+uTC+VlK4JyvPApcEV+rZE3ypVhWcleKZhlQ7XdAqGdet1CV43/vWmRG/qVi8p0aRBFyW6aHTWL8cHfNfAaYmmDZqVaNag9yV638wZ6mNxOxjJYi/yTZXnJFjglwzjSMneQNXXwvR+3zpVizkDsueoPO4JnuoPUQHCzMjlq6s3r5U8ORw8t/dVXeGRBK8k9t7+8xO7IfGL2Sw19uHh4LihiRmM/HWj07P9H51mI5owStaig4O9n35odsowIfHdutPJ8elgr74wnUh1Us6BY9v108ZQI6plIqDngEa0fpt8+TetBq/NUOTZqp839S8ZzP5DHbd1X8Xc6qBtjlXmrY6szbHagJWjaolaYvqwHWtPbeXUvAhzfaNRc2VAUvQ9xfoyYfiNfkHO9QcQiph9q98K5oeB7qV/3b6p/k8I05VQV92uvtmc+j3WLK4Hu4696/zyfe/F8fKO27S+tL6yti3HOrBeWK+sC2toIeu99c9GZ2Oz83vnfefPzl+F9MHG0vPMqjydv/8FyGX5og==</latexit>

1

L

X

i

lnpw(x | zi)<latexit sha1_base64="MSxqYZTcp4AoJQhHXaxJzxtG1Ww=">AAAH7HicfVVdb9s2FFW7ru68dU23x70INQZ0g2FIzppkDwW6fKx9aJssiJMCkWFQ9LVMmJIIknKsEPwXfSv62h+x1+1P7N+M8lclUZtedHHPOVf3HpJiyCgR0vP+uXP3i3tf3m89+Kr99TcPv3208/i7S5FmHMMApzTl70IkgJIEBpJICu8YBxSHFK7C2VGBX82BC5ImFzJnMIxRlJAJwUia1GinF0w4wsrX6rV2A5HFI+IGNHGDCCs2CsIb/TRYBDEZu8HtiPw02ul4PW/5uHbgr4OOs37ORo/vfwrGKc5iSCSmSIhr32NyqBCXBFPQ7SATwBCeoQiuMzk5GCqSsExCgrX7o8EmGXVl6ha9u2PCAUuamwBhTkwFF0+R6V+aCdvVUgISFIPojueEiVUo5tEqkMjYM1SLpX36YUWpIo7YlOBFpTWFYhEjObWSIo/DahIyCnweV5NFm6bJGnMBHBNRmHBmnDllxZKIi/RsjU9zNoVEaJVxqstCAwDnMDHCZShAZkwtpzH7YCaeS55BtwiXuefHiM/OYdw1dSqJajsTmiJZTYVmDONOAjc4jWOUjFXAtAokLKQKuj299K6MnmulgsKoMHTPC7iCvi2hb7Wugicl8MSAVXSwRSfuoC69LIGX1levSuhVXRpmJTSz0HkJnVuVzen4DN9Y8KKELiw0L6G5hd6W0FvbZ2S2xXV/qFZrsVxUdUrJHF5ygESrTl/XZ+Fmva/9qqTYA6rj66XdY5iYn8gKiPOCrl5dvHmt1dFB/5m3p+uMkGawoXi7e8+OPIsSrbpZc7yDg/6hxUk5SqJtoeOTvd98uxDLOKNb0v7+7u+/2pVyoDS92VY6Ojzu79YHM45Um/L3fc+r7zaOLavWjrgd37WsjZro6880CsImwcrPRv7M5r/kKP8PdtpUfWNzo4I1KTaeNyryJsVmATaKqiRpsOnzcmw1tclZcRBm2PRYXBmIruoeg7lMOLwxB+TU/ACRTPnP5lTwKCamlnkH3SL6PyJabIgmarfNzebX7zE7uOz3fK/n//FL58Xh+o574PzgPHGeOr6z77xwXjlnzsDBznvnT+cv5+9W0nrf+tD6uKLevbPWfO9UntanfwH/0uGS</latexit><latexit sha1_base64="MSxqYZTcp4AoJQhHXaxJzxtG1Ww=">AAAH7HicfVVdb9s2FFW7ru68dU23x70INQZ0g2FIzppkDwW6fKx9aJssiJMCkWFQ9LVMmJIIknKsEPwXfSv62h+x1+1P7N+M8lclUZtedHHPOVf3HpJiyCgR0vP+uXP3i3tf3m89+Kr99TcPv3208/i7S5FmHMMApzTl70IkgJIEBpJICu8YBxSHFK7C2VGBX82BC5ImFzJnMIxRlJAJwUia1GinF0w4wsrX6rV2A5HFI+IGNHGDCCs2CsIb/TRYBDEZu8HtiPw02ul4PW/5uHbgr4OOs37ORo/vfwrGKc5iSCSmSIhr32NyqBCXBFPQ7SATwBCeoQiuMzk5GCqSsExCgrX7o8EmGXVl6ha9u2PCAUuamwBhTkwFF0+R6V+aCdvVUgISFIPojueEiVUo5tEqkMjYM1SLpX36YUWpIo7YlOBFpTWFYhEjObWSIo/DahIyCnweV5NFm6bJGnMBHBNRmHBmnDllxZKIi/RsjU9zNoVEaJVxqstCAwDnMDHCZShAZkwtpzH7YCaeS55BtwiXuefHiM/OYdw1dSqJajsTmiJZTYVmDONOAjc4jWOUjFXAtAokLKQKuj299K6MnmulgsKoMHTPC7iCvi2hb7Wugicl8MSAVXSwRSfuoC69LIGX1levSuhVXRpmJTSz0HkJnVuVzen4DN9Y8KKELiw0L6G5hd6W0FvbZ2S2xXV/qFZrsVxUdUrJHF5ygESrTl/XZ+Fmva/9qqTYA6rj66XdY5iYn8gKiPOCrl5dvHmt1dFB/5m3p+uMkGawoXi7e8+OPIsSrbpZc7yDg/6hxUk5SqJtoeOTvd98uxDLOKNb0v7+7u+/2pVyoDS92VY6Ojzu79YHM45Um/L3fc+r7zaOLavWjrgd37WsjZro6880CsImwcrPRv7M5r/kKP8PdtpUfWNzo4I1KTaeNyryJsVmATaKqiRpsOnzcmw1tclZcRBm2PRYXBmIruoeg7lMOLwxB+TU/ACRTPnP5lTwKCamlnkH3SL6PyJabIgmarfNzebX7zE7uOz3fK/n//FL58Xh+o574PzgPHGeOr6z77xwXjlnzsDBznvnT+cv5+9W0nrf+tD6uKLevbPWfO9UntanfwH/0uGS</latexit><latexit sha1_base64="MSxqYZTcp4AoJQhHXaxJzxtG1Ww=">AAAH7HicfVVdb9s2FFW7ru68dU23x70INQZ0g2FIzppkDwW6fKx9aJssiJMCkWFQ9LVMmJIIknKsEPwXfSv62h+x1+1P7N+M8lclUZtedHHPOVf3HpJiyCgR0vP+uXP3i3tf3m89+Kr99TcPv3208/i7S5FmHMMApzTl70IkgJIEBpJICu8YBxSHFK7C2VGBX82BC5ImFzJnMIxRlJAJwUia1GinF0w4wsrX6rV2A5HFI+IGNHGDCCs2CsIb/TRYBDEZu8HtiPw02ul4PW/5uHbgr4OOs37ORo/vfwrGKc5iSCSmSIhr32NyqBCXBFPQ7SATwBCeoQiuMzk5GCqSsExCgrX7o8EmGXVl6ha9u2PCAUuamwBhTkwFF0+R6V+aCdvVUgISFIPojueEiVUo5tEqkMjYM1SLpX36YUWpIo7YlOBFpTWFYhEjObWSIo/DahIyCnweV5NFm6bJGnMBHBNRmHBmnDllxZKIi/RsjU9zNoVEaJVxqstCAwDnMDHCZShAZkwtpzH7YCaeS55BtwiXuefHiM/OYdw1dSqJajsTmiJZTYVmDONOAjc4jWOUjFXAtAokLKQKuj299K6MnmulgsKoMHTPC7iCvi2hb7Wugicl8MSAVXSwRSfuoC69LIGX1levSuhVXRpmJTSz0HkJnVuVzen4DN9Y8KKELiw0L6G5hd6W0FvbZ2S2xXV/qFZrsVxUdUrJHF5ygESrTl/XZ+Fmva/9qqTYA6rj66XdY5iYn8gKiPOCrl5dvHmt1dFB/5m3p+uMkGawoXi7e8+OPIsSrbpZc7yDg/6hxUk5SqJtoeOTvd98uxDLOKNb0v7+7u+/2pVyoDS92VY6Ojzu79YHM45Um/L3fc+r7zaOLavWjrgd37WsjZro6880CsImwcrPRv7M5r/kKP8PdtpUfWNzo4I1KTaeNyryJsVmATaKqiRpsOnzcmw1tclZcRBm2PRYXBmIruoeg7lMOLwxB+TU/ACRTPnP5lTwKCamlnkH3SL6PyJabIgmarfNzebX7zE7uOz3fK/n//FL58Xh+o574PzgPHGeOr6z77xwXjlnzsDBznvnT+cv5+9W0nrf+tD6uKLevbPWfO9UntanfwH/0uGS</latexit><latexit sha1_base64="MSxqYZTcp4AoJQhHXaxJzxtG1Ww=">AAAH7HicfVVdb9s2FFW7ru68dU23x70INQZ0g2FIzppkDwW6fKx9aJssiJMCkWFQ9LVMmJIIknKsEPwXfSv62h+x1+1P7N+M8lclUZtedHHPOVf3HpJiyCgR0vP+uXP3i3tf3m89+Kr99TcPv3208/i7S5FmHMMApzTl70IkgJIEBpJICu8YBxSHFK7C2VGBX82BC5ImFzJnMIxRlJAJwUia1GinF0w4wsrX6rV2A5HFI+IGNHGDCCs2CsIb/TRYBDEZu8HtiPw02ul4PW/5uHbgr4OOs37ORo/vfwrGKc5iSCSmSIhr32NyqBCXBFPQ7SATwBCeoQiuMzk5GCqSsExCgrX7o8EmGXVl6ha9u2PCAUuamwBhTkwFF0+R6V+aCdvVUgISFIPojueEiVUo5tEqkMjYM1SLpX36YUWpIo7YlOBFpTWFYhEjObWSIo/DahIyCnweV5NFm6bJGnMBHBNRmHBmnDllxZKIi/RsjU9zNoVEaJVxqstCAwDnMDHCZShAZkwtpzH7YCaeS55BtwiXuefHiM/OYdw1dSqJajsTmiJZTYVmDONOAjc4jWOUjFXAtAokLKQKuj299K6MnmulgsKoMHTPC7iCvi2hb7Wugicl8MSAVXSwRSfuoC69LIGX1levSuhVXRpmJTSz0HkJnVuVzen4DN9Y8KKELiw0L6G5hd6W0FvbZ2S2xXV/qFZrsVxUdUrJHF5ygESrTl/XZ+Fmva/9qqTYA6rj66XdY5iYn8gKiPOCrl5dvHmt1dFB/5m3p+uMkGawoXi7e8+OPIsSrbpZc7yDg/6hxUk5SqJtoeOTvd98uxDLOKNb0v7+7u+/2pVyoDS92VY6Ojzu79YHM45Um/L3fc+r7zaOLavWjrgd37WsjZro6880CsImwcrPRv7M5r/kKP8PdtpUfWNzo4I1KTaeNyryJsVmATaKqiRpsOnzcmw1tclZcRBm2PRYXBmIruoeg7lMOLwxB+TU/ACRTPnP5lTwKCamlnkH3SL6PyJabIgmarfNzebX7zE7uOz3fK/n//FL58Xh+o574PzgPHGeOr6z77xwXjlnzsDBznvnT+cv5+9W0nrf+tD6uKLevbPWfO9UntanfwH/0uGS</latexit>

Thesecondpartofourlossfunctionrequiresalittlemorework.It’sanexpectationforwhichwedon’thaveaclosedformexpression.Instead,wecanapproximateitbytakingsomesamples,andaveraging.Tokeepthingssimple,wejusttakeasinglesample(we’llbecomputingthenetworklotsoftimesduringtraining,sooverall,we’llbetakinglotsofsamples).

83x<latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit><latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit><latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit><latexit sha1_base64="pFjF/HIzHRPyWesj5UJlDoT5xfA=">AAAHRXicfVXLbtNAFHULhBIotLBkYxEhIRRFdkrbdFGpNIF2QdsQNU2lJqrGk5vEyvihmXESd+Q/YAvfxC/wE6wQWxjnJTtjmE2u7jnn+t4zkxnLJzbjhvFjbf3e/Qe5hxuP8o+fbD59trX9/Ip5AcXQxB7x6LWFGBDbhSa3OYFrnwJyLAIta1iN8dYIKLM995KHPnQc1Hftno0Rl6lGe3K7VTBKxnTpamDOg4I2X/Xb7Vyp3fVw4IDLMUGM3ZiGzzsCUW5jAlG+HTDwER6iPtwEvFfpCNv1Aw4ujvTXEusFROeeHnejd20KmJNQBghTW1bQ8QBRhLnsOZ8uxcBFDrBid2T7bBayUX8WcCQH7ojJ1JBoM6UUfYr8gY0nqdYEcpiD+EBJstCx0kkICNCRk07GbcomV5gToNhmsQl16cyFH5vMLr36HB+E/gBcFomAkigplABQCj0pnIYMeOCL6TRyZ4fskNMAinE4zR3WEB02oFuUdVKJdDs94iGeTllyDOmOC2PsOQ5yu6LtR6LNYcJFu1iKpt4l0UYkRDs2yrL0Rgyn0PMEeh5FK9rmEu3pTYmmwKsEeKUUbiXQ1qrUChJooKCjBDpSKlvjBDxW4EkCnShomEBDBb1LoHeqlUju/E25I2Z2T/dNXBB7BCcUwI1EoRytzkLllt6YaUm8zaJgRlO7u9CT//wZ4IQxXZxenn2KRLVS3jX2olWGRQJYUIydvd2qoVD6s27mHKNSKR8rHI8it78sVPuw995UC/kB9cmStL+/8/FArRQCId54Wal6XCvvrJ4jihUT5rPqBVNXTOtn0edTZQqsLMHMqUz+UOWfUBT+g+1lVV8YmKnwsxQLNzMVYZZiYe1CsTKEH5/WIZafi69uRGaUGshLncKZPMUX8iJC3KNv5dGlfceW9snfdjGO/kdEkwVRRvm8fGHM1fdEDZrl0kHJ/PyucHQ8f2o2tJfaK+2NZmr72pF2qtW1poa1nvZF+6p9y33P/cz9yv2eUdfX5poXWmrl/vwF2NekZA==</latexit>




loss(v,w) = KL(qv(z | x),N(0, I))- lnpw(x | z 0)<latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="RL/hKxJj4Z/yZFeX46MiyZeaFXs=">AAAHNXicfVXdbtowFE73wzrWbu31bqKhSdOEUELVll5U2gpbe7G2rCqlUkGVYw4Q4fzIdoDUygvsds+0h9gz7GraC8xJACU4m284Ot/3nZzz2diWT2zGDePnxqPHT56Wnm0+L7/YKm+/fLWzdcO8gGLoYI949NZCDIjtQofbnMCtTwE5FoGuNWnGeHcKlNmee81DH/oOGrn20MaIy1T7fqdi1Ixk6WpgLoKKtlj3u6Vab+DhwAGXY4IYuzMNn/cFotzGBKJyL2DgIzxBI7gL+LDRF7brBxxcHOlvJTYMiM49PW5FH9gUMCehDBCmtqyg4zGiCHPZcDlfioGLHGDVwdT2WRqy6SgNOJLT9sU8cSPazinFiCJ/bON5rjWBHOYgPlaSLHSsfBICAnTq5JNxm7LJNeYcKLZZbEJbOnPpxw6za6+9wMehPwaXRSKgJMoKJQCUwlAKk5ABD3yRTCO3dcKOOQ2gGodJ7riF6OQKBlVZJ5fItzMkHuL5lCXHkO64MMOe4yB3IHp+JHoc5lz0qrUo8S6LXkVC9GKjLEu/iuEcepFBL6JoTdtZoUO9I9EceJMBb5TC3QzaXZdaQQYNFHSaQadKZWuWgWcKPM+gcwUNM2iooA8Z9EG1Esmdv6v3RWp3sm/ikthTOKUAbiQq9Wh9Fiq39M7MS+JtFhUzSuwewFD+7VPACWO6OLs+/xKJZqO+bxxE6wyLBLCkGHsH+01DoYzSbhYco9GonygcjyJ3tCrU+nTw0VQL+QH1yYp0eLj3+UitFAIh3mxVqXnSqu+tnyOKFRMWs+oVU1dMGxXRF1MVCqwiQepUIX+i8k8pCv/B9oqqLw0sVPhFiqWbhYqwSLG0dqlYG8KPT+sEy8/FVzciKaUF8lKncC5P8aW8iBD36Ht5dOnIsaV98rdXjaP/EdF8SZRRuSwfGHP9OVGDTr12VDO/Gtqm9lp7o73TTO1Q+6CdaW2to2FtoH3Tvpd+lH6Vfqfv0KONxYO0q+VW6c9f66ig2Q==</latexit><latexit sha1_base64="pss90MrmT1oIOUXrNgXHBYe1cU0=">AAAHqXicfVXRbts2FFW6tu68dk33uhdixjB78AzJQZP0IUDbpGuHtUkWxEmByDAo+loWTEkcSTlSCH5mH/Ytexll2ZlkaeOLLu4554L38Ir0GA2EtO2/dh589fDR49aTr9vfPH327fPdF0+vRJxwAiMS05h/9rAAGkQwkoGk8JlxwKFH4dpbHOf49RK4COLoUmYMxiH2o2AWECxNarIbuxJSqWgshO66nCjXW+o+cv08utU9dIR+/9j9c7KBuu6dGwZT5Ka9Pjrtul6o7Jxvvr/pXg/9glwaITbZFOi6acG/+6mHkOtOdjv2wF4tVA+cddCx1ut88uLxwJ3GJAkhkoRiIW4cm8mxwlwGhIJuu4kAhskC+3CTyNnhWAURSyRERKMfDTZLKJIxyptH04ADkTQzASY8MBUQmWOOiTQWtaulBEQ4BNGfLgMmilAs/SKQ2Pg7VunKf/2solQ+x2wekLSyNYVDEWI5ryVFFnrVJCQU+DKsJvNtmk1uMVPgJBC5CefGmTOWn6m4jM/X+Dxjc4iEVgmnuiw0AHAOMyNchQJkwtSqGzNIC3EkeQL9PFzljk4wX1zAtG/qVBLV7cxojGU15Zk2jDsR3JI4DHE0VS7Tqpg5tz/QK+/K6IVWys2N8jx0kcMV9LSEnmq9pR3dozM0MmgFvCqBV7XC1yX0elvqJSU0qaHLErqsVTb/wL/wbQ1OS2haQ7MSmtXQuxJ6V7cSm5O/GY5VYffq3NQZDZbwngNEWnWGersXbo70xqlK8mNWHUev7J7CzFw0BRBmOV19uPz0Uavjw+FLe19vMzyawIZi7+2/PLZrFL/YzZpjHx4O39Y4MceRf1/o5N3+G6deiCWc0XvSwcHer6/qlTKgNL69r3T89mS4tz1HnNRMWPeKOg6qmeY30dddNQq8JkHhVCN/Uee/5zj7D3bcVH1jYKOCNSk2bjYqsibFxtqNYqsJlk/rwrwKLL+6MS0oJ2AudQ6fzBSfmYsIy5j/bEaX+2Fg7DNft59H/0fE6YZoonbbvDDO9ntSD0bDwauB84dtPbG+t36wupZjHVivrQ/WuTWyiPXF+nvn4c6j1kFr3CLFU/RgZ/0mfWdVVgv+AUfIxfo=</latexit><latexit sha1_base64="pss90MrmT1oIOUXrNgXHBYe1cU0=">AAAHqXicfVXRbts2FFW6tu68dk33uhdixjB78AzJQZP0IUDbpGuHtUkWxEmByDAo+loWTEkcSTlSCH5mH/Ytexll2ZlkaeOLLu4554L38Ir0GA2EtO2/dh589fDR49aTr9vfPH327fPdF0+vRJxwAiMS05h/9rAAGkQwkoGk8JlxwKFH4dpbHOf49RK4COLoUmYMxiH2o2AWECxNarIbuxJSqWgshO66nCjXW+o+cv08utU9dIR+/9j9c7KBuu6dGwZT5Ka9Pjrtul6o7Jxvvr/pXg/9glwaITbZFOi6acG/+6mHkOtOdjv2wF4tVA+cddCx1ut88uLxwJ3GJAkhkoRiIW4cm8mxwlwGhIJuu4kAhskC+3CTyNnhWAURSyRERKMfDTZLKJIxyptH04ADkTQzASY8MBUQmWOOiTQWtaulBEQ4BNGfLgMmilAs/SKQ2Pg7VunKf/2solQ+x2wekLSyNYVDEWI5ryVFFnrVJCQU+DKsJvNtmk1uMVPgJBC5CefGmTOWn6m4jM/X+Dxjc4iEVgmnuiw0AHAOMyNchQJkwtSqGzNIC3EkeQL9PFzljk4wX1zAtG/qVBLV7cxojGU15Zk2jDsR3JI4DHE0VS7Tqpg5tz/QK+/K6IVWys2N8jx0kcMV9LSEnmq9pR3dozM0MmgFvCqBV7XC1yX0elvqJSU0qaHLErqsVTb/wL/wbQ1OS2haQ7MSmtXQuxJ6V7cSm5O/GY5VYffq3NQZDZbwngNEWnWGersXbo70xqlK8mNWHUev7J7CzFw0BRBmOV19uPz0Uavjw+FLe19vMzyawIZi7+2/PLZrFL/YzZpjHx4O39Y4MceRf1/o5N3+G6deiCWc0XvSwcHer6/qlTKgNL69r3T89mS4tz1HnNRMWPeKOg6qmeY30dddNQq8JkHhVCN/Uee/5zj7D3bcVH1jYKOCNSk2bjYqsibFxtqNYqsJlk/rwrwKLL+6MS0oJ2AudQ6fzBSfmYsIy5j/bEaX+2Fg7DNft59H/0fE6YZoonbbvDDO9ntSD0bDwauB84dtPbG+t36wupZjHVivrQ/WuTWyiPXF+nvn4c6j1kFr3CLFU/RgZ/0mfWdVVgv+AUfIxfo=</latexit><latexit sha1_base64="B09YcjGXJK00Mxk3KlsjUC/GcaE=">AAAHtHicfVVdb9s2FFX65c5bt3R93AsxY5g9eIbkoEn6EKBLsrXD2iQL4qRAaBgUfS0Lpj5GUo4Ugj9zD/stexll2ZlkaeOLLu4554L38Ip0Y+YLadt/7Tx6/OTps9bzz9qff/Hiy692X359LaKEUxjRiEX8k0sEMD+EkfQlg08xBxK4DG7cxUmO3yyBCz8Kr2QWwzggXujPfEqkSU12IywhlYpFQugu5lRhd6n7CHt5dKd76Aj99qH7x2QDdfE9Dvwpwmmvj8662A2UnfPN91fd66EfEWYhiiebAl2cFvz773sIYTzZ7dgDe7VQPXDWQcdar4vJy2cDPI1oEkAoKSNC3Dp2LMeKcOlTBrqNEwExoQviwW0iZ4dj5YdxIiGkGn1nsFnCkIxQ3jya+hyoZJkJCOW+qYDonHBCpbGoXS0lICQBiP506ceiCMXSKwJJjL9jla781y8qSuVxEs99mla2pkggAiLntaTIAreahIQBXwbVZL5Ns8ktZgqc+iI34cI4cx7nZyquoos1Ps/iOYRCq4QzXRYaADiHmRGuQgEyidWqGzNIC3EkeQL9PFzljk4JX1zCtG/qVBLV7cxYRGQ15Zo2jDsh3NEoCEg4VTjWqpg53B/olXdl9FIrhXOjXBdd5nAFPSuhZ1pvaUcP6AyNDFoBr0vgda3wTQm92Za6SQlNauiyhC5rlc0/8C98V4PTEprW0KyEZjX0voTe160k5uRvh2NV2L06N3XO/CW84wChVp2h3u6FmyO9daqS/JhVx9Eru6cwMxdNAQRZTlfvrz5+0OrkcPja3tfbDJclsKHYe/uvT+waxSt2s+bYh4fD4xon4iT0Hgqd/rz/k1MvFCc8Zg+kg4O9X97UK2XAWHT3UOnk+HS4tz1HnNZMWPeKOg6qmeY10dddNQrcJkHhVCN/Uee/4yT7D3bUVH1jYKMiblJs3GxUZE2KjbUbxVYTcT6tC/MqxPnVTVhBOQVzqXP4aKb43FxEREb8BzO63At8Y5/54n4e/R+RpBuiidpt88I42+9JPRgNB28Gzu925+3x+ql5bn1jfWt1Lcc6sN5a760La2RR60/r750nO09bB61xi7agoD7aWWteWZXVCv8BAqbHGw==</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit><latexit sha1_base64="JqXPSQgjngnK/z0GIEdqHZMp/mw=">AAAHtHicfVVdb9s2FFW6tu68dU23x70QM4bZg2dIzpqkDwG6JFs7rE2yIE4KhIZB0deyYOpjJOVIIfgz97DfspdRlp1Jlja+6OKecy54D69IN2a+kLb9186jTx4/edp69mn7s8+ff/Fi9+WX1yJKOIURjVjEP7pEAPNDGElfMvgYcyCBy+DGXZzk+M0SuPCj8EpmMYwD4oX+zKdEmtRkN8ISUqlYJITuYk4Vdpe6j7CXR3e6h47Qb++7f0w2UBff48CfIpz2+uisi91A2TnffH/VvR76AWEWoniyKdDFacG//66HEMaT3Y49sFcL1QNnHXSs9bqYvHw6wNOIJgGEkjIixK1jx3KsCJc+ZaDbOBEQE7ogHtwmcnY4Vn4YJxJCqtG3BpslDMkI5c2jqc+BSpaZgFDumwqIzgknVBqL2tVSAkISgOhPl34silAsvSKQxPg7VunKf/28olQeJ/Hcp2lla4oEIiByXkuKLHCrSUgY8GVQTebbNJvcYqbAqS9yEy6MM+dxfqbiKrpY4/MsnkMotEo402WhAYBzmBnhKhQgk1itujGDtBBHkifQz8NV7uiU8MUlTPumTiVR3c6MRURWU65pw7gTwh2NgoCEU4VjrYqZw/2BXnlXRi+1Ujg3ynXRZQ5X0LMSeqb1lnb0gM7QyKAV8LoEXtcK35TQm22pm5TQpIYuS+iyVtn8A//CdzU4LaFpDc1KaFZD70vofd1KYk7+djhWhd2rc1PnzF/CWw4QatUZ6u1euDnSW6cqyY9ZdRy9snsKM3PRFECQ5XT17urDe61ODoev7H29zXBZAhuKvbf/6sSuUbxiN2uOfXg4PK5xIk5C76HQ6c/7Pzn1QnHCY/ZAOjjY++V1vVIGjEV3D5VOjk+He9tzxGnNhHWvqOOgmmleE33dVaPAbRIUTjXyF3X+W06y/2BHTdU3BjYq4ibFxs1GRdak2Fi7UWw1EefTujCvQpxf3YQVlFMwlzqHD2aKz81FRGTEvzejy73AN/aZL+7n0f8RSbohmqjdNi+Ms/2e1IPRcPB64Pz+Y+fN8fqpeWZ9bX1jdS3HOrDeWO+sC2tkUetP6++dxztPWgetcYu2oKA+2llrvrIqqxX+AwPmxx8=</latexit>

z 0<latexit sha1_base64="id9tsSEiRI4UfTtEnc4DrwlN2pc=">AAAHRnicfVXLbtNAFJ0WSEug0MKSjUWEQCiK7JS+FpVKW2gX9EFp2kpJVI0nN46V8UMz4zTuyJ/AFr6JT+ArWCG2jPOo7IxhNrm655zre89MZuyQulyY5s+5+Xv3H5QWFh+WHz1eevJ0eeXZBQ8iRqBBAhqwKxtzoK4PDeEKClchA+zZFC7t/l6KXw6AcTfwz0UcQtvDju92XYKFSn1p3b6+Xq6YNXO0DD2wJkEFTdbp9Uqp1uoEJPLAF4RizpuWGYq2xEy4hEJSbkUcQkz62IFmJLqbben6YSTAJ4nxSmHdiBoiMNJ2jI7LgAgaqwAT5qoKBulhholQTZfzpTj42ANe7QzckI9DPnDGgcBq4rYcjhxJlnJK6TAc9lwyzLUmscc9LHpakseenU9CRIENvHwybVM1OcMcAiMuT004Vc6chKnL/Dw4neC9OOyBzxMZMZpkhQoAxqCrhKOQg4hCOZpGbW2fbwsWQTUNR7ntfcz6Z9Cpqjq5RL6dLg2wyKdsNYZyx4cbEnge9juyFSayJWAoZKtaS0beZdGzRMpWapRtG2cpnEOPM+hxksxoG3do12goNAdeZMALrfBlBr2cldpRBo00dJBBB1pl+yYD32jwMIMONTTOoLGG3mbQW91KrHa+WW/Lsd2jfZMn1B3AAQPwE1mpJ7OzMLWlTSsvSbdZVqxkZHcHuuqvPwa8OKXLw/OjT4nc26yvmevJLMOmEUwp5ur62p6pUZxxNxOOublZ39U4AcO+c1do/8P6e0svFEYspHekjY3Vj1t6pRgoDW7uKu3t7tdXZ88RI5oJk1mNimVopjlF9MlUhQK7SDB2qpDf1/kHDMf/YAdF1acGFirCIsXUzUJFXKSYWjtVzAwRpqe1T9Tn0qsb0zFlH9SlzuBIneITdRFhEbC36ugyx3OVfeq3VU2j/xHxcEpUUbmsXhhr9j3Rg0a9tlWzPr+r7OxOnppF9AK9RG+QhTbQDjpEp6iBCHLQV/QNfS/9KP0q/S79GVPn5yaa5yi3FtBfHO+jmA==</latexit><latexit sha1_base64="id9tsSEiRI4UfTtEnc4DrwlN2pc=">AAAHRnicfVXLbtNAFJ0WSEug0MKSjUWEQCiK7JS+FpVKW2gX9EFp2kpJVI0nN46V8UMz4zTuyJ/AFr6JT+ArWCG2jPOo7IxhNrm655zre89MZuyQulyY5s+5+Xv3H5QWFh+WHz1eevJ0eeXZBQ8iRqBBAhqwKxtzoK4PDeEKClchA+zZFC7t/l6KXw6AcTfwz0UcQtvDju92XYKFSn1p3b6+Xq6YNXO0DD2wJkEFTdbp9Uqp1uoEJPLAF4RizpuWGYq2xEy4hEJSbkUcQkz62IFmJLqbben6YSTAJ4nxSmHdiBoiMNJ2jI7LgAgaqwAT5qoKBulhholQTZfzpTj42ANe7QzckI9DPnDGgcBq4rYcjhxJlnJK6TAc9lwyzLUmscc9LHpakseenU9CRIENvHwybVM1OcMcAiMuT004Vc6chKnL/Dw4neC9OOyBzxMZMZpkhQoAxqCrhKOQg4hCOZpGbW2fbwsWQTUNR7ntfcz6Z9Cpqjq5RL6dLg2wyKdsNYZyx4cbEnge9juyFSayJWAoZKtaS0beZdGzRMpWapRtG2cpnEOPM+hxksxoG3do12goNAdeZMALrfBlBr2cldpRBo00dJBBB1pl+yYD32jwMIMONTTOoLGG3mbQW91KrHa+WW/Lsd2jfZMn1B3AAQPwE1mpJ7OzMLWlTSsvSbdZVqxkZHcHuuqvPwa8OKXLw/OjT4nc26yvmevJLMOmEUwp5ur62p6pUZxxNxOOublZ39U4AcO+c1do/8P6e0svFEYspHekjY3Vj1t6pRgoDW7uKu3t7tdXZ88RI5oJk1mNimVopjlF9MlUhQK7SDB2qpDf1/kHDMf/YAdF1acGFirCIsXUzUJFXKSYWjtVzAwRpqe1T9Tn0qsb0zFlH9SlzuBIneITdRFhEbC36ugyx3OVfeq3VU2j/xHxcEpUUbmsXhhr9j3Rg0a9tlWzPr+r7OxOnppF9AK9RG+QhTbQDjpEp6iBCHLQV/QNfS/9KP0q/S79GVPn5yaa5yi3FtBfHO+jmA==</latexit><latexit sha1_base64="id9tsSEiRI4UfTtEnc4DrwlN2pc=">AAAHRnicfVXLbtNAFJ0WSEug0MKSjUWEQCiK7JS+FpVKW2gX9EFp2kpJVI0nN46V8UMz4zTuyJ/AFr6JT+ArWCG2jPOo7IxhNrm655zre89MZuyQulyY5s+5+Xv3H5QWFh+WHz1eevJ0eeXZBQ8iRqBBAhqwKxtzoK4PDeEKClchA+zZFC7t/l6KXw6AcTfwz0UcQtvDju92XYKFSn1p3b6+Xq6YNXO0DD2wJkEFTdbp9Uqp1uoEJPLAF4RizpuWGYq2xEy4hEJSbkUcQkz62IFmJLqbben6YSTAJ4nxSmHdiBoiMNJ2jI7LgAgaqwAT5qoKBulhholQTZfzpTj42ANe7QzckI9DPnDGgcBq4rYcjhxJlnJK6TAc9lwyzLUmscc9LHpakseenU9CRIENvHwybVM1OcMcAiMuT004Vc6chKnL/Dw4neC9OOyBzxMZMZpkhQoAxqCrhKOQg4hCOZpGbW2fbwsWQTUNR7ntfcz6Z9Cpqjq5RL6dLg2wyKdsNYZyx4cbEnge9juyFSayJWAoZKtaS0beZdGzRMpWapRtG2cpnEOPM+hxksxoG3do12goNAdeZMALrfBlBr2cldpRBo00dJBBB1pl+yYD32jwMIMONTTOoLGG3mbQW91KrHa+WW/Lsd2jfZMn1B3AAQPwE1mpJ7OzMLWlTSsvSbdZVqxkZHcHuuqvPwa8OKXLw/OjT4nc26yvmevJLMOmEUwp5ur62p6pUZxxNxOOublZ39U4AcO+c1do/8P6e0svFEYspHekjY3Vj1t6pRgoDW7uKu3t7tdXZ88RI5oJk1mNimVopjlF9MlUhQK7SDB2qpDf1/kHDMf/YAdF1acGFirCIsXUzUJFXKSYWjtVzAwRpqe1T9Tn0qsb0zFlH9SlzuBIneITdRFhEbC36ugyx3OVfeq3VU2j/xHxcEpUUbmsXhhr9j3Rg0a9tlWzPr+r7OxOnppF9AK9RG+QhTbQDjpEp6iBCHLQV/QNfS/9KP0q/S79GVPn5yaa5yi3FtBfHO+jmA==</latexit><latexit sha1_base64="id9tsSEiRI4UfTtEnc4DrwlN2pc=">AAAHRnicfVXLbtNAFJ0WSEug0MKSjUWEQCiK7JS+FpVKW2gX9EFp2kpJVI0nN46V8UMz4zTuyJ/AFr6JT+ArWCG2jPOo7IxhNrm655zre89MZuyQulyY5s+5+Xv3H5QWFh+WHz1eevJ0eeXZBQ8iRqBBAhqwKxtzoK4PDeEKClchA+zZFC7t/l6KXw6AcTfwz0UcQtvDju92XYKFSn1p3b6+Xq6YNXO0DD2wJkEFTdbp9Uqp1uoEJPLAF4RizpuWGYq2xEy4hEJSbkUcQkz62IFmJLqbben6YSTAJ4nxSmHdiBoiMNJ2jI7LgAgaqwAT5qoKBulhholQTZfzpTj42ANe7QzckI9DPnDGgcBq4rYcjhxJlnJK6TAc9lwyzLUmscc9LHpakseenU9CRIENvHwybVM1OcMcAiMuT004Vc6chKnL/Dw4neC9OOyBzxMZMZpkhQoAxqCrhKOQg4hCOZpGbW2fbwsWQTUNR7ntfcz6Z9Cpqjq5RL6dLg2wyKdsNYZyx4cbEnge9juyFSayJWAoZKtaS0beZdGzRMpWapRtG2cpnEOPM+hxksxoG3do12goNAdeZMALrfBlBr2cldpRBo00dJBBB1pl+yYD32jwMIMONTTOoLGG3mbQW91KrHa+WW/Lsd2jfZMn1B3AAQPwE1mpJ7OzMLWlTSsvSbdZVqxkZHcHuuqvPwa8OKXLw/OjT4nc26yvmevJLMOmEUwp5ur62p6pUZxxNxOOublZ39U4AcO+c1do/8P6e0svFEYspHekjY3Vj1t6pRgoDW7uKu3t7tdXZ88RI5oJk1mNimVopjlF9MlUhQK7SDB2qpDf1/kHDMf/YAdF1acGFirCIsXUzUJFXKSYWjtVzAwRpqe1T9Tn0qsb0zFlH9SlzuBIneITdRFhEbC36ugyx3OVfeq3VU2j/xHxcEpUUbmsXhhr9j3Rg0a9tlWzPr+r7OxOnppF9AK9RG+QhTbQDjpEp6iBCHLQV/QNfS/9KP0q/S79GVPn5yaa5yi3FtBfHO+jmA==</latexit>

µz �z


�x

<latexit sha1_base64="fS8LkWLR7/gt5QWBKM8fawfsBzk=">AAAHWXicfVVdb9MwFM0GrKN8DfbIS0SFhFBVJR374GHS2AbjAdiY1m3SUk2Oe5tGdRLLdrpmln8Lr/CTEH8Gpx9TUgfykqt7zrm691zL9ikJuXCc30vL9+4/WKmtPqw/evzk6bO15y/OeZIyDB2ckIRd+ogDCWPoiFAQuKQMUOQTuPCHBzl+MQLGwyQ+ExmFboSCOOyHGAmdul5b93wsPT+SHg+DCCl17Y2v1xpOy5l8thm4s6Bhzb6T6+crLa+X4DSCWGCCOL9yHSq6EjERYgKq7qUcKMJDFMBVKvo7XRnGNBUQY2W/1lg/JbZI7LxBuxcywIJkOkCYhbqCjQeIISz0GPVyKQ4xioA3e6OQ8mnIR8E0EEh70JXjiUfqSUkpA4boIMTjUmsSRTxCYmAkeRb55SSkBNgoKifzNnWTC8wxMBzy3IQT7cwxzX3nZ8nJDB9kdAAxVzJlRBWFGgDGoK+Fk5CDSKmcTKOXPeS7gqXQzMNJbvcQseEp9Jq6TilRbqdPEiTKKV+Pod2J4QYnUYTinvSokp6AsZBes6Um3hXRUyWllxvl+/ZpDpfQbwX0m1IL2s4d2rc7Gi2B5wXw3Ch8UUAvFqV+WkBTAx0V0JFR2b8pwDcGPC6gYwPNCmhmoLcF9Na0EunNX7W7cmr3ZG/ymIQjOGIAsZKNtlqchemVXrllSb5m2XDVxO4e9PVlMAWiLKfLz2dfvyh5sNPedLbUIsMnKcwpzsbW5oFjUIJpNzOOs7PT3jc4CUNxcFfo8OPWB9csRFNGyR1pe3vj03uzUgaEJDd3lQ72D9sbi+eIYcOE2ax2w7UN04Iq+myqSoFfJZg6VckfmvwjhrJ/sJOq6nMDKxW0SjF3s1KRVSnm1s4VC0PQ/LQO9YNA86sbkSnlEPSlzuCrPsXH+iJCImFv9dFlQRRq+/Tfa+bR/4hoPCfqqF7XL4y7+J6YwXm75b5rbX5/19jbn701q9ZL65X1xnKtbWvP+mydWB0LW5n1w/pp/Vr5U1uqrdbqU+ry0kyzbpW+2vpfrVCofw==</latexit>

Wealmosthaveafullydifferentiablemodel.Unfortunately,westillhaveasamplingstepinthemiddle(andsamplingisnotadifferentiableoperation).

: see numpy.random.randn

:

:

:

N(0, 1)<latexit sha1_base64="zlRO7ntmw2EnnU/16dSETWA53KM=">AAAHSnicfVVdb9MwFM3G6EZhsMEjLxEV0kBVlXRs6x4mjW2wCbEPpnWd1FaT4962UZ0P2U7XzMqP4BV+Ez+A38ET4gWnaaukDvilV/ecc3PvsWtbPrEZN4yfC4sPlh4WllceFR8/WX36bG39+TXzAoqhjj3i0RsLMSC2C3VucwI3PgXkWAQa1uAwxhtDoMz23Cse+tB2UM+1uzZGXKYaZxtGWTff3K6VjIoxXroamJOgpE3Wxe16odLqeDhwwOWYIMaapuHztkCU25hAVGwFDHyEB6gHzYB3a21hu37AwcWR/lpi3YDo3NPjlvSOTQFzEsoAYWrLCjruI4owl40Xs6UYuMgBVu4MbZ8lIRv2koAjOXVbjMauRKsZpehR5PdtPMq0JpDDHMT7SpKFjpVNQkCADp1sMm5TNjnHHAHFNotNuJDOnPux0+zKu5jg/dDvg8siEVASpYUSAEqhK4XjkAEPfDGeRm7vgO1xGkA5Dse5vSNEB5fQKcs6mUS2nS7xEM+mLDmGdMeFO+w5DnI7ouVHosVhxEWrXInG3qXRy0iIVmyUZemXMZxBz1LoWRTNaesztKvXJZoBr1PgtVK4kUIb81IrSKGBgg5T6FCpbN2l4DsFHqXQkYKGKTRU0PsUeq9aieTON6ttkdg93jdxTuwhHFMANxKlajQ/C5Vb2jSzknibRcmMxnZ3oCv//gnghDFdnFydfo7EYa26ZWxH8wyLBDClGJvbW4eGQukl3Uw4Rq1WPVA4HkVub1bo6MP2e1Mt5AfUJzPSzs7mx121UgiEeHezSocHR9XN+XNEsWLCZFa9ZOqKab08+mSqXIGVJ0icyuUPVP4xReE/2F5e9amBuQo/TzF1M1cR5imm1k4Vc0P48WkdYPm5+OpGJKEcgbzUKZzKU3wuLyLEPfpWHl3ac2xpn/xtlePof0Q0mhJlVCzKF8acf0/UoF6t7FaML+9K+58mT82K9lJ7pW1opraj7Wsn2oVW17A20L5q37TvhR+FX4XfhT8JdXFhonmhZdby0l/qfqQW</latexit><latexit sha1_base64="zlRO7ntmw2EnnU/16dSETWA53KM=">AAAHSnicfVVdb9MwFM3G6EZhsMEjLxEV0kBVlXRs6x4mjW2wCbEPpnWd1FaT4962UZ0P2U7XzMqP4BV+Ez+A38ET4gWnaaukDvilV/ecc3PvsWtbPrEZN4yfC4sPlh4WllceFR8/WX36bG39+TXzAoqhjj3i0RsLMSC2C3VucwI3PgXkWAQa1uAwxhtDoMz23Cse+tB2UM+1uzZGXKYaZxtGWTff3K6VjIoxXroamJOgpE3Wxe16odLqeDhwwOWYIMaapuHztkCU25hAVGwFDHyEB6gHzYB3a21hu37AwcWR/lpi3YDo3NPjlvSOTQFzEsoAYWrLCjruI4owl40Xs6UYuMgBVu4MbZ8lIRv2koAjOXVbjMauRKsZpehR5PdtPMq0JpDDHMT7SpKFjpVNQkCADp1sMm5TNjnHHAHFNotNuJDOnPux0+zKu5jg/dDvg8siEVASpYUSAEqhK4XjkAEPfDGeRm7vgO1xGkA5Dse5vSNEB5fQKcs6mUS2nS7xEM+mLDmGdMeFO+w5DnI7ouVHosVhxEWrXInG3qXRy0iIVmyUZemXMZxBz1LoWRTNaesztKvXJZoBr1PgtVK4kUIb81IrSKGBgg5T6FCpbN2l4DsFHqXQkYKGKTRU0PsUeq9aieTON6ttkdg93jdxTuwhHFMANxKlajQ/C5Vb2jSzknibRcmMxnZ3oCv//gnghDFdnFydfo7EYa26ZWxH8wyLBDClGJvbW4eGQukl3Uw4Rq1WPVA4HkVub1bo6MP2e1Mt5AfUJzPSzs7mx121UgiEeHezSocHR9XN+XNEsWLCZFa9ZOqKab08+mSqXIGVJ0icyuUPVP4xReE/2F5e9amBuQo/TzF1M1cR5imm1k4Vc0P48WkdYPm5+OpGJKEcgbzUKZzKU3wuLyLEPfpWHl3ac2xpn/xtlePof0Q0mhJlVCzKF8acf0/UoF6t7FaML+9K+58mT82K9lJ7pW1opraj7Wsn2oVW17A20L5q37TvhR+FX4XfhT8JdXFhonmhZdby0l/qfqQW</latexit><latexit sha1_base64="zlRO7ntmw2EnnU/16dSETWA53KM=">AAAHSnicfVVdb9MwFM3G6EZhsMEjLxEV0kBVlXRs6x4mjW2wCbEPpnWd1FaT4962UZ0P2U7XzMqP4BV+Ez+A38ET4gWnaaukDvilV/ecc3PvsWtbPrEZN4yfC4sPlh4WllceFR8/WX36bG39+TXzAoqhjj3i0RsLMSC2C3VucwI3PgXkWAQa1uAwxhtDoMz23Cse+tB2UM+1uzZGXKYaZxtGWTff3K6VjIoxXroamJOgpE3Wxe16odLqeDhwwOWYIMaapuHztkCU25hAVGwFDHyEB6gHzYB3a21hu37AwcWR/lpi3YDo3NPjlvSOTQFzEsoAYWrLCjruI4owl40Xs6UYuMgBVu4MbZ8lIRv2koAjOXVbjMauRKsZpehR5PdtPMq0JpDDHMT7SpKFjpVNQkCADp1sMm5TNjnHHAHFNotNuJDOnPux0+zKu5jg/dDvg8siEVASpYUSAEqhK4XjkAEPfDGeRm7vgO1xGkA5Dse5vSNEB5fQKcs6mUS2nS7xEM+mLDmGdMeFO+w5DnI7ouVHosVhxEWrXInG3qXRy0iIVmyUZemXMZxBz1LoWRTNaesztKvXJZoBr1PgtVK4kUIb81IrSKGBgg5T6FCpbN2l4DsFHqXQkYKGKTRU0PsUeq9aieTON6ttkdg93jdxTuwhHFMANxKlajQ/C5Vb2jSzknibRcmMxnZ3oCv//gnghDFdnFydfo7EYa26ZWxH8wyLBDClGJvbW4eGQukl3Uw4Rq1WPVA4HkVub1bo6MP2e1Mt5AfUJzPSzs7mx121UgiEeHezSocHR9XN+XNEsWLCZFa9ZOqKab08+mSqXIGVJ0icyuUPVP4xReE/2F5e9amBuQo/TzF1M1cR5imm1k4Vc0P48WkdYPm5+OpGJKEcgbzUKZzKU3wuLyLEPfpWHl3ac2xpn/xtlePof0Q0mhJlVCzKF8acf0/UoF6t7FaML+9K+58mT82K9lJ7pW1opraj7Wsn2oVW17A20L5q37TvhR+FX4XfhT8JdXFhonmhZdby0l/qfqQW</latexit>

N(µ,⌃)<latexit sha1_base64="sZC27btAVKX3ytxl3fwFrSmc2Bw=">AAAHXXicfVVdb9MwFA0DVihjbPAAEi8RFRKgqko72MYDErDxIcQ+GOs6aakmx71NozqJZTtdg2V+Da/wf3jjp+D0Y0rqQF5ydc85V/eea9keJQEXjvP7ytLVa9eXKzduVm+t3F69s7Z+94THCcPQxjGJ2amHOJAggrYIBIFTygCFHoGON9zJ8M4IGA/i6FikFLoh8qOgH2AkdOp87cH+EzfG0g0TVbddT0dfAz9E6un5Ws1pOJPPNoPmLKhZs+/wfH254fZinIQQCUwQ52dNh4quREwEmICqugkHivAQ+XCWiP52VwYRTQREWNmPNdZPiC1iO+vS7gUMsCCpDhBmga5g4wFiCAs9S7VYikOEQuD13iigfBrykT8NBNJGdOV4YpS6XVBKnyE6CPC40JpEIQ+RGBhJnoZeMQkJATYKi8msTd3kAnMMDAc8M+FQO3NAM/P5cXw4wwcpHUDElUwYUXmhBoAx6GvhJOQgEion0+iND/krwRKoZ+Ek92oXseER9Oq6TiFRbKdPYiSKKU+Pod2J4ALHYYiinnSpkq6AsZBuvaEm3uXRIyX1qdFGeZ59lMEFdD+H7iu1oG1fon27rdECeJIDT4zCnRzaWZR6SQ5NDHSUQ0dGZe8iB18Y8DiHjg00zaGpgX7Lod9MK5He/FmrK6d2T/YmD0gwgg8MIFKy1lKLszC90rNmUZKtWdaaamJ3D/r6RpgCYZrR5cfjvc9K7my3XjibapHhkQTmFGdj88WOY1D8aTczjrO93XprcGKGIv+y0O67zTdNsxBNGCWXpK2tjfcvzUopEBJfXFbaebvb2lg8RwwbJsxmtWtN2zDNL6PPpioVeGWCqVOl/KHJ/8BQ+g92XFZ9bmCpgpYp5m6WKtIyxdzauWJhCJqd1qF+CWh2dSMypeyCvtQZ7OlTfKAvIiRi9kwfXeaHgbZP/916Fv2PiMZzoo6qVf3CNBffEzNotxovG86X57XXn2ZPzQ3rofXIemI1rS3rtfXROrTaFra+Wz+sn9av5T+V5cpKZXVKXboy09yzCl/l/l/EgKld</latexit><latexit sha1_base64="sZC27btAVKX3ytxl3fwFrSmc2Bw=">AAAHXXicfVVdb9MwFA0DVihjbPAAEi8RFRKgqko72MYDErDxIcQ+GOs6aakmx71NozqJZTtdg2V+Da/wf3jjp+D0Y0rqQF5ydc85V/eea9keJQEXjvP7ytLVa9eXKzduVm+t3F69s7Z+94THCcPQxjGJ2amHOJAggrYIBIFTygCFHoGON9zJ8M4IGA/i6FikFLoh8qOgH2AkdOp87cH+EzfG0g0TVbddT0dfAz9E6un5Ws1pOJPPNoPmLKhZs+/wfH254fZinIQQCUwQ52dNh4quREwEmICqugkHivAQ+XCWiP52VwYRTQREWNmPNdZPiC1iO+vS7gUMsCCpDhBmga5g4wFiCAs9S7VYikOEQuD13iigfBrykT8NBNJGdOV4YpS6XVBKnyE6CPC40JpEIQ+RGBhJnoZeMQkJATYKi8msTd3kAnMMDAc8M+FQO3NAM/P5cXw4wwcpHUDElUwYUXmhBoAx6GvhJOQgEion0+iND/krwRKoZ+Ek92oXseER9Oq6TiFRbKdPYiSKKU+Pod2J4ALHYYiinnSpkq6AsZBuvaEm3uXRIyX1qdFGeZ59lMEFdD+H7iu1oG1fon27rdECeJIDT4zCnRzaWZR6SQ5NDHSUQ0dGZe8iB18Y8DiHjg00zaGpgX7Lod9MK5He/FmrK6d2T/YmD0gwgg8MIFKy1lKLszC90rNmUZKtWdaaamJ3D/r6RpgCYZrR5cfjvc9K7my3XjibapHhkQTmFGdj88WOY1D8aTczjrO93XprcGKGIv+y0O67zTdNsxBNGCWXpK2tjfcvzUopEBJfXFbaebvb2lg8RwwbJsxmtWtN2zDNL6PPpioVeGWCqVOl/KHJ/8BQ+g92XFZ9bmCpgpYp5m6WKtIyxdzauWJhCJqd1qF+CWh2dSMypeyCvtQZ7OlTfKAvIiRi9kwfXeaHgbZP/916Fv2PiMZzoo6qVf3CNBffEzNotxovG86X57XXn2ZPzQ3rofXIemI1rS3rtfXROrTaFra+Wz+sn9av5T+V5cpKZXVKXboy09yzCl/l/l/EgKld</latexit><latexit sha1_base64="sZC27btAVKX3ytxl3fwFrSmc2Bw=">AAAHXXicfVVdb9MwFA0DVihjbPAAEi8RFRKgqko72MYDErDxIcQ+GOs6aakmx71NozqJZTtdg2V+Da/wf3jjp+D0Y0rqQF5ydc85V/eea9keJQEXjvP7ytLVa9eXKzduVm+t3F69s7Z+94THCcPQxjGJ2amHOJAggrYIBIFTygCFHoGON9zJ8M4IGA/i6FikFLoh8qOgH2AkdOp87cH+EzfG0g0TVbddT0dfAz9E6un5Ws1pOJPPNoPmLKhZs+/wfH254fZinIQQCUwQ52dNh4quREwEmICqugkHivAQ+XCWiP52VwYRTQREWNmPNdZPiC1iO+vS7gUMsCCpDhBmga5g4wFiCAs9S7VYikOEQuD13iigfBrykT8NBNJGdOV4YpS6XVBKnyE6CPC40JpEIQ+RGBhJnoZeMQkJATYKi8msTd3kAnMMDAc8M+FQO3NAM/P5cXw4wwcpHUDElUwYUXmhBoAx6GvhJOQgEion0+iND/krwRKoZ+Ek92oXseER9Oq6TiFRbKdPYiSKKU+Pod2J4ALHYYiinnSpkq6AsZBuvaEm3uXRIyX1qdFGeZ59lMEFdD+H7iu1oG1fon27rdECeJIDT4zCnRzaWZR6SQ5NDHSUQ0dGZe8iB18Y8DiHjg00zaGpgX7Lod9MK5He/FmrK6d2T/YmD0gwgg8MIFKy1lKLszC90rNmUZKtWdaaamJ3D/r6RpgCYZrR5cfjvc9K7my3XjibapHhkQTmFGdj88WOY1D8aTczjrO93XprcGKGIv+y0O67zTdNsxBNGCWXpK2tjfcvzUopEBJfXFbaebvb2lg8RwwbJsxmtWtN2zDNL6PPpioVeGWCqVOl/KHJ/8BQ+g92XFZ9bmCpgpYp5m6WKtIyxdzauWJhCJqd1qF+CWh2dSMypeyCvtQZ7OlTfKAvIiRi9kwfXeaHgbZP/916Fv2PiMZzoo6qVf3CNBffEzNotxovG86X57XXn2ZPzQ3rofXIemI1rS3rtfXROrTaFra+Wz+sn9av5T+V5cpKZXVKXboy09yzCl/l/l/EgKld</latexit>

Nd(0, 1)<latexit sha1_base64="POx+1ATID4h2VJT2d22zPTfrX6o=">AAAHWHicfVVbb9MwFM4GrFthsMsjLxEV0kBVlXTs9jBpbIVNiF2Y1m3SWibHPW2jOhfZTtfM8m/hFX4S4s/g9KakDvglR+f7vpNzPlu2ExKXccv6PTf/5OmzhcLiUvH5i+WXr1ZW165ZEFEMdRyQgN46iAFxfahzlxO4DSkgzyFw4/SOEvymD5S5gX/F4xCaHur4btvFiKvU/cr62ffWRsPxhCXLZvK15bv7lZJVsYbL1AN7HJSM8bq4X12oNFoBjjzwOSaIsTvbCnlTIMpdTEAWGxGDEOEe6sBdxNu7TeH6YcTBx9J8q7B2REwemEmDZsulgDmJVYAwdVUFE3cRRZirMYrZUgx85AErt/puyEYh63dGAUfKg6YYDD2Syxml6FAUdl08yLQmkMc8xLtaksWek01CRID2vWwyaVM1OcMcAMUuS0y4UM6ch4nv7Cq4GOPdOOyCz6SIKJFpoQKAUmgr4TBkwKNQDKdRm91j+5xGUE7CYW6/hmjvElplVSeTyLbTJgHi2ZSjxlDu+PCAA89Dfks0QikaHAZcNMoVOfQujV5KIRqJUY5jXiZwBj1LoWdSzmjrU7Rt1hWaAa9T4LVW+CaF3sxKnSiFRhraT6F9rbLzkIIfNHiQQgcaGqfQWEMfU+ijbiVSO39XbYqR3cN9E+fE7cMxBfClKFXl7CxUbemdnZUk2yxKthza3YK2ugxGgBcndHFydfpViqPd6pa1LWcZDolgQrE2t7eOLI3SGXUz5li7u9VDjRNQ5HemhWqftj/aeqEwoiGZknZ2Nj/v6ZViICR4mFY6OqxVN2fPEcWaCeNZzZJtaqZ18ujjqXIFTp5g5FQuv6fzjymK/8EO8qpPDMxVhHmKiZu5ijhPMbF2opgZIkxOaw+r3yVXNyIjSg3UpU7hVJ3ic3URIR7Q9+ro0o7nKvvUt1FOov8R0WBCVFGxqF4Ye/Y90YN6tbJXsb59KB18GT81i8Zr442xYdjGjnFgnBgXRt3ARmz8MH4avxb+FOYKhcLSiDo/N9asG5lVWPsLapCmyg==</latexit><latexit sha1_base64="POx+1ATID4h2VJT2d22zPTfrX6o=">AAAHWHicfVVbb9MwFM4GrFthsMsjLxEV0kBVlXTs9jBpbIVNiF2Y1m3SWibHPW2jOhfZTtfM8m/hFX4S4s/g9KakDvglR+f7vpNzPlu2ExKXccv6PTf/5OmzhcLiUvH5i+WXr1ZW165ZEFEMdRyQgN46iAFxfahzlxO4DSkgzyFw4/SOEvymD5S5gX/F4xCaHur4btvFiKvU/cr62ffWRsPxhCXLZvK15bv7lZJVsYbL1AN7HJSM8bq4X12oNFoBjjzwOSaIsTvbCnlTIMpdTEAWGxGDEOEe6sBdxNu7TeH6YcTBx9J8q7B2REwemEmDZsulgDmJVYAwdVUFE3cRRZirMYrZUgx85AErt/puyEYh63dGAUfKg6YYDD2Syxml6FAUdl08yLQmkMc8xLtaksWek01CRID2vWwyaVM1OcMcAMUuS0y4UM6ch4nv7Cq4GOPdOOyCz6SIKJFpoQKAUmgr4TBkwKNQDKdRm91j+5xGUE7CYW6/hmjvElplVSeTyLbTJgHi2ZSjxlDu+PCAA89Dfks0QikaHAZcNMoVOfQujV5KIRqJUY5jXiZwBj1LoWdSzmjrU7Rt1hWaAa9T4LVW+CaF3sxKnSiFRhraT6F9rbLzkIIfNHiQQgcaGqfQWEMfU+ijbiVSO39XbYqR3cN9E+fE7cMxBfClKFXl7CxUbemdnZUk2yxKthza3YK2ugxGgBcndHFydfpViqPd6pa1LWcZDolgQrE2t7eOLI3SGXUz5li7u9VDjRNQ5HemhWqftj/aeqEwoiGZknZ2Nj/v6ZViICR4mFY6OqxVN2fPEcWaCeNZzZJtaqZ18ujjqXIFTp5g5FQuv6fzjymK/8EO8qpPDMxVhHmKiZu5ijhPMbF2opgZIkxOaw+r3yVXNyIjSg3UpU7hVJ3ic3URIR7Q9+ro0o7nKvvUt1FOov8R0WBCVFGxqF4Ye/Y90YN6tbJXsb59KB18GT81i8Zr442xYdjGjnFgnBgXRt3ARmz8MH4avxb+FOYKhcLSiDo/N9asG5lVWPsLapCmyg==</latexit><latexit sha1_base64="POx+1ATID4h2VJT2d22zPTfrX6o=">AAAHWHicfVVbb9MwFM4GrFthsMsjLxEV0kBVlXTs9jBpbIVNiF2Y1m3SWibHPW2jOhfZTtfM8m/hFX4S4s/g9KakDvglR+f7vpNzPlu2ExKXccv6PTf/5OmzhcLiUvH5i+WXr1ZW165ZEFEMdRyQgN46iAFxfahzlxO4DSkgzyFw4/SOEvymD5S5gX/F4xCaHur4btvFiKvU/cr62ffWRsPxhCXLZvK15bv7lZJVsYbL1AN7HJSM8bq4X12oNFoBjjzwOSaIsTvbCnlTIMpdTEAWGxGDEOEe6sBdxNu7TeH6YcTBx9J8q7B2REwemEmDZsulgDmJVYAwdVUFE3cRRZirMYrZUgx85AErt/puyEYh63dGAUfKg6YYDD2Syxml6FAUdl08yLQmkMc8xLtaksWek01CRID2vWwyaVM1OcMcAMUuS0y4UM6ch4nv7Cq4GOPdOOyCz6SIKJFpoQKAUmgr4TBkwKNQDKdRm91j+5xGUE7CYW6/hmjvElplVSeTyLbTJgHi2ZSjxlDu+PCAA89Dfks0QikaHAZcNMoVOfQujV5KIRqJUY5jXiZwBj1LoWdSzmjrU7Rt1hWaAa9T4LVW+CaF3sxKnSiFRhraT6F9rbLzkIIfNHiQQgcaGqfQWEMfU+ijbiVSO39XbYqR3cN9E+fE7cMxBfClKFXl7CxUbemdnZUk2yxKthza3YK2ugxGgBcndHFydfpViqPd6pa1LWcZDolgQrE2t7eOLI3SGXUz5li7u9VDjRNQ5HemhWqftj/aeqEwoiGZknZ2Nj/v6ZViICR4mFY6OqxVN2fPEcWaCeNZzZJtaqZ18ujjqXIFTp5g5FQuv6fzjymK/8EO8qpPDMxVhHmKiZu5ijhPMbF2opgZIkxOaw+r3yVXNyIjSg3UpU7hVJ3ic3URIR7Q9+ro0o7nKvvUt1FOov8R0WBCVFGxqF4Ye/Y90YN6tbJXsb59KB18GT81i8Zr442xYdjGjnFgnBgXRt3ARmz8MH4avxb+FOYKhcLSiDo/N9asG5lVWPsLapCmyg==</latexit>

Nd(µ,⌃)<latexit sha1_base64="P07RnGEzYTKTM8RlcZ0G8GPAWCw=">AAAHaXicfVXbbtNAEDW3BsKtlBcELxYBCVAUOSm98IAEtFyEoC2laZHqUK03E8fK2l7trtOY1f4AX8Mr/An/wEewTpzKzhr8ksmcc0YzZ1a7HiUBF47z+9z5CxcvLdUuX6lfvXb9xs3lWyuHPE4Yhi6OScy+eIgDCSLoikAQ+EIZoNAjcOSNtjL8aAyMB3F0IFIKvRD5UTAIMBI6dbL8YOdr/5EbY+l6oXTDRKmm7Xr538+BHyKlHp8sN5yWM/1sM2jnQcPKv72TW0sttx/jJIRIYII4P247VPQkYiLABFTdTThQhEfIh+NEDDZ7MohoIiDCyn6osUFCbBHbWcd2P2CABUl1gDALdAUbDxFDWOi56uVSHCIUAm/2xwHls5CP/VkgkDalJydT09T1klL6DNFhgCel1iQKeYjE0EjyNPTKSUgIsHFYTmZt6iYXmBNgOOCZCXvamV2aLYIfxHs5PkzpECKuZMKIKgo1AIzBQAunIQeRUDmdRm9/xJ8LlkAzC6e559uIjfah39R1SolyOwMSI1FOeXoM7U4EpzgOQxT1pUuVdAVMhHSbLTX1rojuK6nPjjbK8+z9DC6hOwV0R6kFbfcMHdhdjZbAwwJ4aBQ+KqBHi1IvKaCJgY4L6Nio7J0W4FMDnhTQiYGmBTQ10G8F9JtpJdKbP+705Mzu6d7kLgnG8JYBREo2OmpxFqZXetwuS7I1y0ZbTe3uw0DfDjMgTDO6fHfw8YOSW5udNWddLTI8ksCc4qyur205BsWfdZNznM3NziuDEzMU+WeFtl+vv2ybhWjCKDkjbWysvnlmVkqBkPj0rNLWq+3O6uI5YtgwIZ/VbrRtwzS/ip5PVSnwqgQzpyr5I5P/lqH0H+y4qvrcwEoFrVLM3axUpFWKubVzxcIQNDutI/0c0OzqRmRG2QZ9qTP4qE/xrr6IkIjZE310mR8G2j796zaz6H9ENJkTdVSv6xemvfiemEG303rWcj49bbx4nz81l6171n3rkdW2NqwX1jtrz+pa2Ppu/bB+Wr+W/tRWandqd2fU8+dyzW2r9NUafwHnZa7d</latexit><latexit sha1_base64="P07RnGEzYTKTM8RlcZ0G8GPAWCw=">AAAHaXicfVXbbtNAEDW3BsKtlBcELxYBCVAUOSm98IAEtFyEoC2laZHqUK03E8fK2l7trtOY1f4AX8Mr/An/wEewTpzKzhr8ksmcc0YzZ1a7HiUBF47z+9z5CxcvLdUuX6lfvXb9xs3lWyuHPE4Yhi6OScy+eIgDCSLoikAQ+EIZoNAjcOSNtjL8aAyMB3F0IFIKvRD5UTAIMBI6dbL8YOdr/5EbY+l6oXTDRKmm7Xr538+BHyKlHp8sN5yWM/1sM2jnQcPKv72TW0sttx/jJIRIYII4P247VPQkYiLABFTdTThQhEfIh+NEDDZ7MohoIiDCyn6osUFCbBHbWcd2P2CABUl1gDALdAUbDxFDWOi56uVSHCIUAm/2xwHls5CP/VkgkDalJydT09T1klL6DNFhgCel1iQKeYjE0EjyNPTKSUgIsHFYTmZt6iYXmBNgOOCZCXvamV2aLYIfxHs5PkzpECKuZMKIKgo1AIzBQAunIQeRUDmdRm9/xJ8LlkAzC6e559uIjfah39R1SolyOwMSI1FOeXoM7U4EpzgOQxT1pUuVdAVMhHSbLTX1rojuK6nPjjbK8+z9DC6hOwV0R6kFbfcMHdhdjZbAwwJ4aBQ+KqBHi1IvKaCJgY4L6Nio7J0W4FMDnhTQiYGmBTQ10G8F9JtpJdKbP+705Mzu6d7kLgnG8JYBREo2OmpxFqZXetwuS7I1y0ZbTe3uw0DfDjMgTDO6fHfw8YOSW5udNWddLTI8ksCc4qyur205BsWfdZNznM3NziuDEzMU+WeFtl+vv2ybhWjCKDkjbWysvnlmVkqBkPj0rNLWq+3O6uI5YtgwIZ/VbrRtwzS/ip5PVSnwqgQzpyr5I5P/lqH0H+y4qvrcwEoFrVLM3axUpFWKubVzxcIQNDutI/0c0OzqRmRG2QZ9qTP4qE/xrr6IkIjZE310mR8G2j796zaz6H9ENJkTdVSv6xemvfiemEG303rWcj49bbx4nz81l6171n3rkdW2NqwX1jtrz+pa2Ppu/bB+Wr+W/tRWandqd2fU8+dyzW2r9NUafwHnZa7d</latexit><latexit sha1_base64="P07RnGEzYTKTM8RlcZ0G8GPAWCw=">AAAHaXicfVXbbtNAEDW3BsKtlBcELxYBCVAUOSm98IAEtFyEoC2laZHqUK03E8fK2l7trtOY1f4AX8Mr/An/wEewTpzKzhr8ksmcc0YzZ1a7HiUBF47z+9z5CxcvLdUuX6lfvXb9xs3lWyuHPE4Yhi6OScy+eIgDCSLoikAQ+EIZoNAjcOSNtjL8aAyMB3F0IFIKvRD5UTAIMBI6dbL8YOdr/5EbY+l6oXTDRKmm7Xr538+BHyKlHp8sN5yWM/1sM2jnQcPKv72TW0sttx/jJIRIYII4P247VPQkYiLABFTdTThQhEfIh+NEDDZ7MohoIiDCyn6osUFCbBHbWcd2P2CABUl1gDALdAUbDxFDWOi56uVSHCIUAm/2xwHls5CP/VkgkDalJydT09T1klL6DNFhgCel1iQKeYjE0EjyNPTKSUgIsHFYTmZt6iYXmBNgOOCZCXvamV2aLYIfxHs5PkzpECKuZMKIKgo1AIzBQAunIQeRUDmdRm9/xJ8LlkAzC6e559uIjfah39R1SolyOwMSI1FOeXoM7U4EpzgOQxT1pUuVdAVMhHSbLTX1rojuK6nPjjbK8+z9DC6hOwV0R6kFbfcMHdhdjZbAwwJ4aBQ+KqBHi1IvKaCJgY4L6Nio7J0W4FMDnhTQiYGmBTQ10G8F9JtpJdKbP+705Mzu6d7kLgnG8JYBREo2OmpxFqZXetwuS7I1y0ZbTe3uw0DfDjMgTDO6fHfw8YOSW5udNWddLTI8ksCc4qyur205BsWfdZNznM3NziuDEzMU+WeFtl+vv2ybhWjCKDkjbWysvnlmVkqBkPj0rNLWq+3O6uI5YtgwIZ/VbrRtwzS/ip5PVSnwqgQzpyr5I5P/lqH0H+y4qvrcwEoFrVLM3axUpFWKubVzxcIQNDutI/0c0OzqRmRG2QZ9qTP4qE/xrr6IkIjZE310mR8G2j796zaz6H9ENJkTdVSv6xemvfiemEG303rWcj49bbx4nz81l6171n3rkdW2NqwX1jtrz+pa2Ppu/bB+Wr+W/tRWandqd2fU8+dyzW2r9NUafwHnZa7d</latexit>

X�+ µ with X ⇠ N(0, 1)<latexit sha1_base64="2iOtzUeLho9/5pCnYsMyLF050Ic=">AAAHgHicfVVdb9s2FFXbte68dU23x70IMwa0neFKTpukCwp0TbYWQ5tkQZwEiIyAoq9lwpREkJRtleCP6a/p6/a4fzPKkgN9dXzR1T3nXPAeXpA+o0RIx/n31u07X92917n/dfebbx9893Dr0ffnIk44hhGOacwvfSSAkghGkkgKl4wDCn0KF/78IMMvFsAFiaMzmTIYhyiIyJRgJE3qemv/0vOx8gQJQqTtX2wvNn9hom1v39u3PQkrqZZEzorEpW2ooX302Om7T663es7AWS+7GbhF0LOKdXL96N7Am8Q4CSGSmCIhrlyHybFCXBJMQXe9RABDeI4CuErkdG+sSMQSCRHW9s8GmybUlrGdNWJPCAcsaWoChDkxFWw8QxxhadrtVksJiFAIoj9ZECbyUCyCPJDIeDVWq7WX+kFFqQKO2IzgVWVrCoUiRMaTelKkoV9NQkKBL8JqMtum2WSNuQKOichMODHOHLPsfMRZfFLgs5TNIBJaJZzqstAAwDlMjXAdCpAJU+tuzFDMxSvJE+hn4Tr36hDx+SlM+qZOJVHdzpTGSFZTvmnDuBPBEsdhiKKJ8phW+Yh4/YFee1dGT7Uys2SM8n37NIMr6FEJPdK6ph3doFN7ZNAKeF4CzxuFL0roRV3qJyU0aaCLErpoVPaXJXjZgFcldNVA0xKaNtCPJfRj00pkTv5qOFa53etzU8eULOAtB4i06g11vRdujvTKrUqyY1Y9V6/tnsDUXBo5EKYZXb07+/Beq4O94QtnR9cZPk1gQ3G2d14cOA1KkO+m4Dh7e8M3DU7MURTcFDr8fec3t1mIJZzRG9Lu7vYfL5uVUqA0Xt5UOnhzONyuzxHHDROKXu2eazdMC9roRVetAr9NkDvVyp83+W85Sr/AjtuqbwxsVbA2xcbNVkXapthYu1HUmmDZtM7NS8GyqxvRnHII5lLn8MFM8bG5iJCM+VMzujwIibHPfL1+Fv0fEa02RBN1u+aFcevvSTMYDQcvB85fz3uv/yyemvvWj9ZP1mPLtXat19Y768QaWdj6ZH22/rb+6dzpPOk867g59fatQvODVVmdX/8Dl/a0zg==</latexit><latexit sha1_base64="2iOtzUeLho9/5pCnYsMyLF050Ic=">AAAHgHicfVVdb9s2FFXbte68dU23x70IMwa0neFKTpukCwp0TbYWQ5tkQZwEiIyAoq9lwpREkJRtleCP6a/p6/a4fzPKkgN9dXzR1T3nXPAeXpA+o0RIx/n31u07X92917n/dfebbx9893Dr0ffnIk44hhGOacwvfSSAkghGkkgKl4wDCn0KF/78IMMvFsAFiaMzmTIYhyiIyJRgJE3qemv/0vOx8gQJQqTtX2wvNn9hom1v39u3PQkrqZZEzorEpW2ooX302Om7T663es7AWS+7GbhF0LOKdXL96N7Am8Q4CSGSmCIhrlyHybFCXBJMQXe9RABDeI4CuErkdG+sSMQSCRHW9s8GmybUlrGdNWJPCAcsaWoChDkxFWw8QxxhadrtVksJiFAIoj9ZECbyUCyCPJDIeDVWq7WX+kFFqQKO2IzgVWVrCoUiRMaTelKkoV9NQkKBL8JqMtum2WSNuQKOichMODHOHLPsfMRZfFLgs5TNIBJaJZzqstAAwDlMjXAdCpAJU+tuzFDMxSvJE+hn4Tr36hDx+SlM+qZOJVHdzpTGSFZTvmnDuBPBEsdhiKKJ8phW+Yh4/YFee1dGT7Uys2SM8n37NIMr6FEJPdK6ph3doFN7ZNAKeF4CzxuFL0roRV3qJyU0aaCLErpoVPaXJXjZgFcldNVA0xKaNtCPJfRj00pkTv5qOFa53etzU8eULOAtB4i06g11vRdujvTKrUqyY1Y9V6/tnsDUXBo5EKYZXb07+/Beq4O94QtnR9cZPk1gQ3G2d14cOA1KkO+m4Dh7e8M3DU7MURTcFDr8fec3t1mIJZzRG9Lu7vYfL5uVUqA0Xt5UOnhzONyuzxHHDROKXu2eazdMC9roRVetAr9NkDvVyp83+W85Sr/AjtuqbwxsVbA2xcbNVkXapthYu1HUmmDZtM7NS8GyqxvRnHII5lLn8MFM8bG5iJCM+VMzujwIibHPfL1+Fv0fEa02RBN1u+aFcevvSTMYDQcvB85fz3uv/yyemvvWj9ZP1mPLtXat19Y768QaWdj6ZH22/rb+6dzpPOk867g59fatQvODVVmdX/8Dl/a0zg==</latexit><latexit sha1_base64="2iOtzUeLho9/5pCnYsMyLF050Ic=">AAAHgHicfVVdb9s2FFXbte68dU23x70IMwa0neFKTpukCwp0TbYWQ5tkQZwEiIyAoq9lwpREkJRtleCP6a/p6/a4fzPKkgN9dXzR1T3nXPAeXpA+o0RIx/n31u07X92917n/dfebbx9893Dr0ffnIk44hhGOacwvfSSAkghGkkgKl4wDCn0KF/78IMMvFsAFiaMzmTIYhyiIyJRgJE3qemv/0vOx8gQJQqTtX2wvNn9hom1v39u3PQkrqZZEzorEpW2ooX302Om7T663es7AWS+7GbhF0LOKdXL96N7Am8Q4CSGSmCIhrlyHybFCXBJMQXe9RABDeI4CuErkdG+sSMQSCRHW9s8GmybUlrGdNWJPCAcsaWoChDkxFWw8QxxhadrtVksJiFAIoj9ZECbyUCyCPJDIeDVWq7WX+kFFqQKO2IzgVWVrCoUiRMaTelKkoV9NQkKBL8JqMtum2WSNuQKOichMODHOHLPsfMRZfFLgs5TNIBJaJZzqstAAwDlMjXAdCpAJU+tuzFDMxSvJE+hn4Tr36hDx+SlM+qZOJVHdzpTGSFZTvmnDuBPBEsdhiKKJ8phW+Yh4/YFee1dGT7Uys2SM8n37NIMr6FEJPdK6ph3doFN7ZNAKeF4CzxuFL0roRV3qJyU0aaCLErpoVPaXJXjZgFcldNVA0xKaNtCPJfRj00pkTv5qOFa53etzU8eULOAtB4i06g11vRdujvTKrUqyY1Y9V6/tnsDUXBo5EKYZXb07+/Beq4O94QtnR9cZPk1gQ3G2d14cOA1KkO+m4Dh7e8M3DU7MURTcFDr8fec3t1mIJZzRG9Lu7vYfL5uVUqA0Xt5UOnhzONyuzxHHDROKXu2eazdMC9roRVetAr9NkDvVyp83+W85Sr/AjtuqbwxsVbA2xcbNVkXapthYu1HUmmDZtM7NS8GyqxvRnHII5lLn8MFM8bG5iJCM+VMzujwIibHPfL1+Fv0fEa02RBN1u+aFcevvSTMYDQcvB85fz3uv/yyemvvWj9ZP1mPLtXat19Y768QaWdj6ZH22/rb+6dzpPOk867g59fatQvODVVmdX/8Dl/a0zg==</latexit>

0

[email protected]

1

CA with Xi ⇠ N(0, 1)

<latexit sha1_base64="eDs13Md4zQPJymcbvQxjpvCoxn8=">AAAHmnicfVVdb9s2FFXbre68dUu7x2GAMGNANxiG5KxJiqFAV2dr99EkC+IkQGQYFH0tE6YkgqRsqwR/2X7JHve6/YlRlm1Ipjb6wdf3nHPNe3hBhowSIT3vz3v3H3zw4cPWo4/aH3/y+NPPDp48vRZpxjEMcUpTfhsiAZQkMJREUrhlHFAcUrgJ54MCv1kAFyRNrmTOYBSjKCFTgpE0qfHBMAghIoliMZKcrPTt2HeDwA0Wk1SKIrodTwJIJjuCG3xvPhJWUi2JnOnipyERNxAkds+eeV3X/2Z80PF63nq5duBvgo6zWRfjJw97wSTFWQyJxBQJced7TI4U4pJgCrodZAIYwnMUwV0mpycjRRKWSUiwdr822DSjrkzdokV3QjhgSXMTIMyJqeDiGeIIS2NEu15KQIJiEN3JgjBRhmIRlYFExsWRWq1d1o9rShVxxGYEr2pbUygWxqeZlRR5HNaTkFHgi7ieLLZpNrnHXAHHRBQmXBhnzllxcuIqvdjgs5zNIBFaZZzqqtAAwDlMjXAdCpAZU+tuzLjMxUvJM+gW4Tr38hTx+SVMuqZOLVHfzpSmSNZToWnDuJPAEqdxjMy0BEyrckiCbk+vvauil1qpoDAqDN3LAq6hZxX0TOs97XCHTt2hQWvgdQW8tgrfVNCbfWmYVdDMQhcVdGFVDpcVeGnBqwq6stC8guYW+r6CvretRObk7/ojVdq9Pjd1TskC3nCARKtOX+/3ws2R3vl1SXHMquPrtd0TmJrrpATivKCrt1fvftNqcNJ/7h3pfUZIM9hSvMOj5wPPokTlbjYc7+Sk/9ripBwl0a7Q6Y9HP/h2IZZxRnek4+PDn17YlXKgNF3uKg1en/YP9+eIY8uETa9ux3ct06Im+qarRkHYJCidauTPbf4bjvL/YKdN1bcGNipYk2LrZqMib1Jsrd0q9ppgxbTOsfm74upGtKScgrnUObwzU3xuLiIkU/6tGV0excTYZ76DbhH9HxGttkQTtdvmhfH33xM7GPZ7L3re7991Xv2yeWoeOV84XznPHN85dl45b50LZ+hg5w/nL+dv55/Wl61B6+fWryX1/r2N5nOntlpX/wJbnsA6</latexit><latexit sha1_base64="eDs13Md4zQPJymcbvQxjpvCoxn8=">AAAHmnicfVVdb9s2FFXbre68dUu7x2GAMGNANxiG5KxJiqFAV2dr99EkC+IkQGQYFH0tE6YkgqRsqwR/2X7JHve6/YlRlm1Ipjb6wdf3nHPNe3hBhowSIT3vz3v3H3zw4cPWo4/aH3/y+NPPDp48vRZpxjEMcUpTfhsiAZQkMJREUrhlHFAcUrgJ54MCv1kAFyRNrmTOYBSjKCFTgpE0qfHBMAghIoliMZKcrPTt2HeDwA0Wk1SKIrodTwJIJjuCG3xvPhJWUi2JnOnipyERNxAkds+eeV3X/2Z80PF63nq5duBvgo6zWRfjJw97wSTFWQyJxBQJced7TI4U4pJgCrodZAIYwnMUwV0mpycjRRKWSUiwdr822DSjrkzdokV3QjhgSXMTIMyJqeDiGeIIS2NEu15KQIJiEN3JgjBRhmIRlYFExsWRWq1d1o9rShVxxGYEr2pbUygWxqeZlRR5HNaTkFHgi7ieLLZpNrnHXAHHRBQmXBhnzllxcuIqvdjgs5zNIBFaZZzqqtAAwDlMjXAdCpAZU+tuzLjMxUvJM+gW4Tr38hTx+SVMuqZOLVHfzpSmSNZToWnDuJPAEqdxjMy0BEyrckiCbk+vvauil1qpoDAqDN3LAq6hZxX0TOs97XCHTt2hQWvgdQW8tgrfVNCbfWmYVdDMQhcVdGFVDpcVeGnBqwq6stC8guYW+r6CvretRObk7/ojVdq9Pjd1TskC3nCARKtOX+/3ws2R3vl1SXHMquPrtd0TmJrrpATivKCrt1fvftNqcNJ/7h3pfUZIM9hSvMOj5wPPokTlbjYc7+Sk/9ripBwl0a7Q6Y9HP/h2IZZxRnek4+PDn17YlXKgNF3uKg1en/YP9+eIY8uETa9ux3ct06Im+qarRkHYJCidauTPbf4bjvL/YKdN1bcGNipYk2LrZqMib1Jsrd0q9ppgxbTOsfm74upGtKScgrnUObwzU3xuLiIkU/6tGV0excTYZ76DbhH9HxGttkQTtdvmhfH33xM7GPZ7L3re7991Xv2yeWoeOV84XznPHN85dl45b50LZ+hg5w/nL+dv55/Wl61B6+fWryX1/r2N5nOntlpX/wJbnsA6</latexit><latexit sha1_base64="eDs13Md4zQPJymcbvQxjpvCoxn8=">AAAHmnicfVVdb9s2FFXbre68dUu7x2GAMGNANxiG5KxJiqFAV2dr99EkC+IkQGQYFH0tE6YkgqRsqwR/2X7JHve6/YlRlm1Ipjb6wdf3nHPNe3hBhowSIT3vz3v3H3zw4cPWo4/aH3/y+NPPDp48vRZpxjEMcUpTfhsiAZQkMJREUrhlHFAcUrgJ54MCv1kAFyRNrmTOYBSjKCFTgpE0qfHBMAghIoliMZKcrPTt2HeDwA0Wk1SKIrodTwJIJjuCG3xvPhJWUi2JnOnipyERNxAkds+eeV3X/2Z80PF63nq5duBvgo6zWRfjJw97wSTFWQyJxBQJced7TI4U4pJgCrodZAIYwnMUwV0mpycjRRKWSUiwdr822DSjrkzdokV3QjhgSXMTIMyJqeDiGeIIS2NEu15KQIJiEN3JgjBRhmIRlYFExsWRWq1d1o9rShVxxGYEr2pbUygWxqeZlRR5HNaTkFHgi7ieLLZpNrnHXAHHRBQmXBhnzllxcuIqvdjgs5zNIBFaZZzqqtAAwDlMjXAdCpAZU+tuzLjMxUvJM+gW4Tr38hTx+SVMuqZOLVHfzpSmSNZToWnDuJPAEqdxjMy0BEyrckiCbk+vvauil1qpoDAqDN3LAq6hZxX0TOs97XCHTt2hQWvgdQW8tgrfVNCbfWmYVdDMQhcVdGFVDpcVeGnBqwq6stC8guYW+r6CvretRObk7/ojVdq9Pjd1TskC3nCARKtOX+/3ws2R3vl1SXHMquPrtd0TmJrrpATivKCrt1fvftNqcNJ/7h3pfUZIM9hSvMOj5wPPokTlbjYc7+Sk/9ripBwl0a7Q6Y9HP/h2IZZxRnek4+PDn17YlXKgNF3uKg1en/YP9+eIY8uETa9ux3ct06Im+qarRkHYJCidauTPbf4bjvL/YKdN1bcGNipYk2LrZqMib1Jsrd0q9ppgxbTOsfm74upGtKScgrnUObwzU3xuLiIkU/6tGV0excTYZ76DbhH9HxGttkQTtdvmhfH33xM7GPZ7L3re7991Xv2yeWoeOV84XznPHN85dl45b50LZ+hg5w/nL+dv55/Wl61B6+fWryX1/r2N5nOntlpX/wJbnsA6</latexit>

AX+ µ with X ⇠ Nd(0, 1),

⌃ = AAT<latexit sha1_base64="oAHtmKYlOTIQ5cjGCdQkoBf5QYo=">AAAHpHicfVVdbxtFFN0WqIuhNIVHXkZYQAHL2nVokgpFappAK0Q+SOMkUtaJZsfX65FnPzQza3s7mp/Hj+A38ArvzHrX0X7B+GGv7znnau6ZqxkvZlRI2/7zwcMPPvzoUefxx91PPn3y2dOtZ59fiijhBEYkYhG/9rAARkMYSSoZXMcccOAxuPLmhxl+tQAuaBReyDSGcYD9kE4pwdKk7rbuDq7RD8iNiHK9QLlBojVyfzI/CSupllTOdPYXXX+DXEEDdHI7eZ4xbd1H2dfR3/Vdt2tQryjxjvoBNlX20cHB7cXdVs8e2OuFmoFTBD2rWGd3zx4N3ElEkgBCSRgW4saxYzlWmEtKGOiumwiIMZljH24SOd0bKxrGiYSQaPS1waYJQzJCWbNoQjkQyVITYMKpqYDIDHNMpLGkWy0lIMQBiP5kQWORh2Lh54HExs+xWq391k8qSuVzHM8oWVW2pnAgAmzcqydFGnjVJCQM+CKoJrNtmk3WmCvghIrMhDPjzGmcnaG4iM4KfJbGMwiFVglnuiw0AHAOUyNchwJkEqt1N2Zw5mJf8gT6WbjO7R9hPj+HSd/UqSSq25myCMtqyjNtGHdCWJIoCHA4UW6sVT5Mbn+g196V0XOtzNQZozwPnWdwBT0poSda17Sje3SKRgatgJcl8LJR+KqEXtWlXlJCkwa6KKGLRmVvWYKXDXhVQlcNNC2haQN9X0LfN63E5uRvhmOV270+N3XK6ALecIBQq95Q13vh5khvnKokO2bVc/Ta7glMzcWSA0Ga0dXbi+PftDrcG76wd3Sd4bEENhR7e+fFod2g+PluCo69tzd83eBEHIf+faGjn3cOnGahOOExuyft7m7/8rJZKQXGouV9pcPXR8Pt+hxx0jCh6BX1HNQwzW+jF121Crw2Qe5UK3/e5L/hOP0PdtRWfWNgqyJuU2zcbFWkbYqNtRtFrYk4m9a5eRTi7OrGLKccgbnUORybKT41FxGWEf/ejC73A2rsM1+3n0X/R8SrDdFE3a55YZz6e9IMRsPBy4H9+4+9V78WT81j60vrK+u55Vi71ivrrXVmjSxi/WH9Zf1t/dP5tnPcedcZ5dSHDwrNF1ZldW7/BSrAwpA=</latexit><latexit sha1_base64="oAHtmKYlOTIQ5cjGCdQkoBf5QYo=">AAAHpHicfVVdbxtFFN0WqIuhNIVHXkZYQAHL2nVokgpFappAK0Q+SOMkUtaJZsfX65FnPzQza3s7mp/Hj+A38ArvzHrX0X7B+GGv7znnau6ZqxkvZlRI2/7zwcMPPvzoUefxx91PPn3y2dOtZ59fiijhBEYkYhG/9rAARkMYSSoZXMcccOAxuPLmhxl+tQAuaBReyDSGcYD9kE4pwdKk7rbuDq7RD8iNiHK9QLlBojVyfzI/CSupllTOdPYXXX+DXEEDdHI7eZ4xbd1H2dfR3/Vdt2tQryjxjvoBNlX20cHB7cXdVs8e2OuFmoFTBD2rWGd3zx4N3ElEkgBCSRgW4saxYzlWmEtKGOiumwiIMZljH24SOd0bKxrGiYSQaPS1waYJQzJCWbNoQjkQyVITYMKpqYDIDHNMpLGkWy0lIMQBiP5kQWORh2Lh54HExs+xWq391k8qSuVzHM8oWVW2pnAgAmzcqydFGnjVJCQM+CKoJrNtmk3WmCvghIrMhDPjzGmcnaG4iM4KfJbGMwiFVglnuiw0AHAOUyNchwJkEqt1N2Zw5mJf8gT6WbjO7R9hPj+HSd/UqSSq25myCMtqyjNtGHdCWJIoCHA4UW6sVT5Mbn+g196V0XOtzNQZozwPnWdwBT0poSda17Sje3SKRgatgJcl8LJR+KqEXtWlXlJCkwa6KKGLRmVvWYKXDXhVQlcNNC2haQN9X0LfN63E5uRvhmOV270+N3XK6ALecIBQq95Q13vh5khvnKokO2bVc/Ta7glMzcWSA0Ga0dXbi+PftDrcG76wd3Sd4bEENhR7e+fFod2g+PluCo69tzd83eBEHIf+faGjn3cOnGahOOExuyft7m7/8rJZKQXGouV9pcPXR8Pt+hxx0jCh6BX1HNQwzW+jF121Crw2Qe5UK3/e5L/hOP0PdtRWfWNgqyJuU2zcbFWkbYqNtRtFrYk4m9a5eRTi7OrGLKccgbnUORybKT41FxGWEf/ejC73A2rsM1+3n0X/R8SrDdFE3a55YZz6e9IMRsPBy4H9+4+9V78WT81j60vrK+u55Vi71ivrrXVmjSxi/WH9Zf1t/dP5tnPcedcZ5dSHDwrNF1ZldW7/BSrAwpA=</latexit><latexit sha1_base64="oAHtmKYlOTIQ5cjGCdQkoBf5QYo=">AAAHpHicfVVdbxtFFN0WqIuhNIVHXkZYQAHL2nVokgpFappAK0Q+SOMkUtaJZsfX65FnPzQza3s7mp/Hj+A38ArvzHrX0X7B+GGv7znnau6ZqxkvZlRI2/7zwcMPPvzoUefxx91PPn3y2dOtZ59fiijhBEYkYhG/9rAARkMYSSoZXMcccOAxuPLmhxl+tQAuaBReyDSGcYD9kE4pwdKk7rbuDq7RD8iNiHK9QLlBojVyfzI/CSupllTOdPYXXX+DXEEDdHI7eZ4xbd1H2dfR3/Vdt2tQryjxjvoBNlX20cHB7cXdVs8e2OuFmoFTBD2rWGd3zx4N3ElEkgBCSRgW4saxYzlWmEtKGOiumwiIMZljH24SOd0bKxrGiYSQaPS1waYJQzJCWbNoQjkQyVITYMKpqYDIDHNMpLGkWy0lIMQBiP5kQWORh2Lh54HExs+xWq391k8qSuVzHM8oWVW2pnAgAmzcqydFGnjVJCQM+CKoJrNtmk3WmCvghIrMhDPjzGmcnaG4iM4KfJbGMwiFVglnuiw0AHAOUyNchwJkEqt1N2Zw5mJf8gT6WbjO7R9hPj+HSd/UqSSq25myCMtqyjNtGHdCWJIoCHA4UW6sVT5Mbn+g196V0XOtzNQZozwPnWdwBT0poSda17Sje3SKRgatgJcl8LJR+KqEXtWlXlJCkwa6KKGLRmVvWYKXDXhVQlcNNC2haQN9X0LfN63E5uRvhmOV270+N3XK6ALecIBQq95Q13vh5khvnKokO2bVc/Ta7glMzcWSA0Ga0dXbi+PftDrcG76wd3Sd4bEENhR7e+fFod2g+PluCo69tzd83eBEHIf+faGjn3cOnGahOOExuyft7m7/8rJZKQXGouV9pcPXR8Pt+hxx0jCh6BX1HNQwzW+jF121Crw2Qe5UK3/e5L/hOP0PdtRWfWNgqyJuU2zcbFWkbYqNtRtFrYk4m9a5eRTi7OrGLKccgbnUORybKT41FxGWEf/ejC73A2rsM1+3n0X/R8SrDdFE3a55YZz6e9IMRsPBy4H9+4+9V78WT81j60vrK+u55Vi71ivrrXVmjSxi/WH9Zf1t/dP5tnPcedcZ5dSHDwrNF1ZldW7/BSrAwpA=</latexit>

sampling WecangetgetridofitbyrememberingthealgorithmforsamplingfromagivenMVN.WesamplefromthestandardMVN,andtransformthesampleusingtheparametersoftherequiredMVN.

85



<latexit sha1_base64="gLVbQjX9K5UEC47PhCywHjbG58I=">AAAHvHicfVVdb9s2FJWzru7cr3R77AsxY0C7GobkrElWoEPXZGs3rEmaxkmAyAso+lomTH2ApBSrBH9ogf2YUbIcSJY2vfj6nnMueM+9IL2YUSFt+0tn66s7X9/t3vumd//Bw0ePt598ey6ihBMYk4hF/NLDAhgNYSypZHAZc8CBx+DCWxzk+EUKXNAoPJNZDJMA+yGdUYKlSV1vp5fIFTRAR39PnynXC5CtB6gI/tDPkev2csCNSJ5SbpBoA7te+fcT9QOs9fNX6NJoJA0z5EaSBiD0mpRXz0noBSqroLzK9XbfHtrFh5qBUwZ9q/xOrp/cHbrTiCQBhJIwLMSVY8dyojCXlDDQPTcREGOywD5cJXK2P1E0jBMJIdHoB4PNEoZkhHIP0JRyIJJlJsCEU1MBkTnmmEjjVK9eSkCITUODaUpjsQpF6q8CiY3NE7UsxqAf1pTK5zieU7KsHU3hQARYzhtJkQVePQkJA54G9WR+THPIDeYSOKEiN+HEOHMc56MVZ9FJic+zeA6h0CrhTFeFBgDOYWaERShAJrEqujH7tBCvJU9gkIdF7vUh5otTmA5MnVqifpwZi7CspzzThnEnhBsSBQEOp8qNtdkYWErlDoa68K6Knmplts0Y5XnoNIdr6FEFPdJ6Qzu+RWdobNAaeF4BzxuFLyroxabUSypo0kDTCpo2Kns3FfimAS8r6LKBZhU0a6CfK+jnppXYTP5qNFEru4u5qWNGU3jHAUKt+iO92Qs3I71y6pJ8zKrv6MLuKczMfbMCgiynq/dnH/7S6mB/9NLe1ZsMjyWwptg7uy8P7AbFX52m5Nj7+6O3DU7EcejfFjr8bfdXp1koTnjMbkl7ezu//9yslAFj0c1tpYO3h6OdzT3ipGFC2SvqO6hhmt9GL7tqFXhtgpVTrfxFk/+O4+w/2FFb9bWBrYq4TbF2s1WRtSnW1q4VG03E+bYuzEsQ51c3ZivKIZhLncMHs8XH5iLCMuI/mtXlfkCNfebXHeTR/xHxck00Ua9nXhhn8z1pBuejobM7tD/+1H/zZ/nW3LOeWt9bzyzH2rPeWO+tE2tsEeufzlbnfudB95futLvoBivqVqfUfGfVvm76L00DyhQ=</latexit>

X ⇠ Nd(0, I)

Nd(µ,⌃) : X⌦�+ µ<latexit sha1_base64="or2iBGmQ/NX9OnlGuZSo9QOAY1E=">AAAHcXicfVXdbtNIFDawNBD+Clyt9sYiQkIQRXYKbdEKCdrycwG0VE1bqY6q8eTEsTK2RzPjNO5oHmKfZm/hMXiOfYEdO05lZwzORY7O931H53xzNONTEnLhOD+vXb/xx8211q3b7Tt3791/sP7w0TFPUoZhgBOSsFMfcSBhDAMRCgKnlAGKfAIn/nQ3x09mwHiYxEciozCMUBCH4xAjoVPn6y+8BEvPj6QXpUqde5e293f+sz2/zPMwiFABna93nJ5TfLYZuGXQscrv4PzhWs8bJTiNIBaYIM7PXIeKoURMhJiAanspB4rwFAVwlorx9lCGMU0FxFjZTzU2ToktEjtv3B6FDLAgmQ4QZqGuYOMJYggLPV67XopDjCLg3dEspHwR8lmwCATS3gzlvPBO3aspZcAQnYR4XmtNoohHSEyMJM8iv56ElACbRfVk3qZucoU5B4ZDnptwoJ3Zp/l58KPkoMQnGZ1AzJVMGVFVoQaAMRhrYRFyECmVxTR6Cab8jWApdPOwyL3ZQ2x6CKOurlNL1NsZkwSJesrXY2h3YrjASRSheCQ9qqQnYC6k1+2pwrsqeqikXiNtlO/bhzlcQ79W0K9KrWgHV+jYHmi0Bh5XwGOj8EkFPVmV+mkFTQ10VkFnRmX/ogJfGPC8gs4NNKugmYFeVtBL00qkT/6sP5QLu4tzk/sknMFHBhAr2emr1VmYPtIzty7Jj1l2XFXYPYKxviQWQJTldPnp6MtnJXe3+6+cTbXK8EkKS4qzsflq1zEowaKbkuNsb/d3DE7CUBxcFdp7v/nONQvRlFFyRdra2vjw2qyUASHJxVWl3Z29/sbqHjFsmFDOandc2zAtaKKXUzUK/CbBwqlG/tTkf2Qo+wU7aaq+NLBRQZsUSzcbFVmTYmntUrEyBM23daofBJpf3YgsKHugL3UGX/QW7+uLCImEPdery4Io1Pbpf6+bR78jovmSqKN2W78w7up7YgbH/Z672XO+vey83SnfmlvWX9YT65nlWlvWW+uTdWANLGz9Y/1rfbd+rP3X+rNlt54sqNevlZrHVu1rvfgfCTOyLw==</latexit>µz �z

86







e ⇠ N(0, I)<latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit><latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit><latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit><latexit sha1_base64="jQBWECCUluNooIjfpHsA/uNTPLQ=">AAAHYnicfVVbT9swFA67UFbGBuNxmxStmsSmqkrKuD0gMWCDSeMyRAGJVMhxT9uozkW2UxosP+3X7HX7NfsB+x9zekFJnc0vOT3f952e89my3Yh4jFvW75kHDx89ni3NPSnPP1149nxx6cUFC2OKoYFDEtIrFzEgXgAN7nECVxEF5LsELt3eXopf9oEyLwzOeRJB00edwGt7GHGVull87bi+AGk6zPPN45X0lyWrZvr9It/dLFasmjVcph7Y46BijNfpzdJszWmFOPYh4Jggxq5tK+JNgSj3MAFZdmIGEcI91IHrmLc3m8ILophDgKX5VmHtmJg8NNNWzZZHAXOSqABh6qkKJu4iijBXA5XzpRgEyAdWbfW9iI1C1u+MAo6UG00xGLolF3JK0aEo6np4kGtNIJ/5iHe1JEt8N5+EmADt+/lk2qZqcoo5AIo9lppwqpw5idIdYOfh6RjvJlEXAiZFTInMChUAlEJbCYchAx5HYjiN2vYe2+Y0hmoaDnPb+4j2zqBVVXVyiXw7bRIink+5agzlTgC3OPR9FLSEE0nhcBhw4VRrcuhdFj2TQjipUa5rnqVwDj3OoMdSTmkb92jbbCg0B15kwAut8GUGvZyWunEGjTW0n0H7WmX3NgPfavAggw40NMmgiYbeZdA73Uqkdv663hQju4f7Jk6I14cDChBIUanL6Vmo2tJrOy9Jt1lUbDm0uwVtdS2MAD9J6eLw/OirFHub9TVrXU4zXBLDhGKtrq/tWRqlM+pmzLE2N+u7GiekKOjcF9r/tP7R1gtFMY3IPWljY/Xzll4pAULC2/tKe7v79dXpc0SxZsJ4VrNim5ppnSL6eKpCgVskGDlVyO/p/AOKkn+ww6LqEwMLFVGRYuJmoSIpUkysnSimhojS09rD6u/SqxuREWUf1KVO4Uid4hN1ESEe0vfq6NKO7yn71NepptH/iGgwIaqoXFYvjD39nuhBo17bqtnfPlR2dsdPzZzx0nhjrBi2sWHsGIfGqdEwsPHd+GH8NH7N/inNl5ZKyyPqg5mxZtnIrdKrv0hXquo=</latexit>


<latexit sha1_base64="or2iBGmQ/NX9OnlGuZSo9QOAY1E=">AAAHcXicfVXdbtNIFDawNBD+Clyt9sYiQkIQRXYKbdEKCdrycwG0VE1bqY6q8eTEsTK2RzPjNO5oHmKfZm/hMXiOfYEdO05lZwzORY7O931H53xzNONTEnLhOD+vXb/xx8211q3b7Tt3791/sP7w0TFPUoZhgBOSsFMfcSBhDAMRCgKnlAGKfAIn/nQ3x09mwHiYxEciozCMUBCH4xAjoVPn6y+8BEvPj6QXpUqde5e293f+sz2/zPMwiFABna93nJ5TfLYZuGXQscrv4PzhWs8bJTiNIBaYIM7PXIeKoURMhJiAanspB4rwFAVwlorx9lCGMU0FxFjZTzU2ToktEjtv3B6FDLAgmQ4QZqGuYOMJYggLPV67XopDjCLg3dEspHwR8lmwCATS3gzlvPBO3aspZcAQnYR4XmtNoohHSEyMJM8iv56ElACbRfVk3qZucoU5B4ZDnptwoJ3Zp/l58KPkoMQnGZ1AzJVMGVFVoQaAMRhrYRFyECmVxTR6Cab8jWApdPOwyL3ZQ2x6CKOurlNL1NsZkwSJesrXY2h3YrjASRSheCQ9qqQnYC6k1+2pwrsqeqikXiNtlO/bhzlcQ79W0K9KrWgHV+jYHmi0Bh5XwGOj8EkFPVmV+mkFTQ10VkFnRmX/ogJfGPC8gs4NNKugmYFeVtBL00qkT/6sP5QLu4tzk/sknMFHBhAr2emr1VmYPtIzty7Jj1l2XFXYPYKxviQWQJTldPnp6MtnJXe3+6+cTbXK8EkKS4qzsflq1zEowaKbkuNsb/d3DE7CUBxcFdp7v/nONQvRlFFyRdra2vjw2qyUASHJxVWl3Z29/sbqHjFsmFDOandc2zAtaKKXUzUK/CbBwqlG/tTkf2Qo+wU7aaq+NLBRQZsUSzcbFVmTYmntUrEyBM23daofBJpf3YgsKHugL3UGX/QW7+uLCImEPdery4Io1Pbpf6+bR78jovmSqKN2W78w7up7YgbH/Z672XO+vey83SnfmlvWX9YT65nlWlvWW+uTdWANLGz9Y/1rfbd+rP3X+rNlt54sqNevlZrHVu1rvfgfCTOyLw==</latexit>µz �z

<latexit sha1_base64="jnG4CR7vkkR04c36Y8P+MKH/oJg=">AAAH43icfVVdb9s2FJXbru68dUu3x70QNQbYm2dIzppkDwHaJFs7rE2yIE4KhIZBydeyYOpjJOVIIfgL+jbstT9s/2aUJWeSpY0vurjnnAvewyvSjqjHhWn+3Xrw8NEnj9tPPu189vnTL77cefbVFQ9j5sDYCWnI3tuEA/UCGAtPUHgfMSC+TeHaXh5n+PUKGPfC4FKkEUx84gbe3HOI0KnpzkcsIBGShpyrHmaOxPZKDRB2s+hW9dEh+u1t74/pBurhO+x7M4ST/gCd9rDtSzPj6++vqt9HPyBMAxRNNwV6OMn5mgAKh8LzgWM7A32Euef6RE3xHfoe4bBI+nGW6SOE8XSnaw7N9UL1wCqCrlGs8+mzx0M8C53Yh0A4lHB+Y5mRmEjChOdQUB0cc4iIsyQu3MRifjCRXhDFAgJHoW81No8pEiHKrEIzj4EjaKoD4jBPV0DOgjDiCG1op1qKQ0B0Z4PZyot4HvKVmweC6NOYyGR9WuppRSldRqKF5ySVrUnic5+IRS3JU9+uJiGmwFZ+NZltU29yi5kAczyemXCunTmLsgngl+F5gS/SaAEBVzJmVJWFGgDGYK6F65CDiCO57kaP3ZIfChbDIAvXucMTwpYXMBvoOpVEdTtzGhJRTdm6De1OALdO6PskmEkcKZlPKB4M1dq7MnqhpMSZUbaNLjK4gp6W0FOltrTje3SOxhqtgFcl8KpW+LqEXm9L7biExjV0VUJXtcr6j/kXvq3BSQlNamhaQtMaeldC7+pWEn3yN6OJzO1en5s8o94KXjOAQMnuSG33wvSR3lhVSXbMsmuptd0zmOtrKQf8NKPLN5fv3ip5fDB6Ye6pbYZNY9hQzN29F8dmjeLmuyk45sHB6KjGCRkJ3PtCJz/vvbLqhaKYRfSetL+/+8tP9UopUBre3lc6PjoZ7W7PEXNqJhS9oq6Faqa5TfSiq0aB3STInWrkL+v814yk/8EOm6pvDGxURE2KjZuNirRJsbF2o9hqIsqmdalfhCi7ugnNKSegL3UG7/QUn+mLiIiQfadHl7m+p+3TXzzIov8jkmRD1FGno18Ya/s9qQdXo6G1NzR//7H78qh4a54Y3xjPjZ5hGfvGS+ONcW6MDaf1sNVvjVq7bWh/aP/Z/iunPmgVmq+Nymp//AdaD9jr</latexit>

loss(v,w) = KL(qv(z | x),N(0, I))- lnpw(x | e⌦ �z + µz)Wecanworkthissamplingintothearchitecture.Weprovidethenetworkwithanextrainput:asamplefromthestandardMVN.Wealsoslightlychangetheqnetwork.

InsteadofmakingthenetworkproduceSigma,wemakeitproduceA,sincethenetworkistrainedanyway,wecanjustinterprettheoutputasAinsteadofSigma.SincebotharediagonalmatricesAjustcontainsthesquarerootsofSigma.

Whydoesthishelpus?We’restillsampling,butwe’vemovedtherandomsamplingoutofthewayofthebackpropagation.Thegradientcannowpropagatedowntotheweightsoftheqfunction,andtheactualrandomnessistreatedasaninput,ratherthanacomputation.

<latexit sha1_base64="or2iBGmQ/NX9OnlGuZSo9QOAY1E=">AAAHcXicfVXdbtNIFDawNBD+Clyt9sYiQkIQRXYKbdEKCdrycwG0VE1bqY6q8eTEsTK2RzPjNO5oHmKfZm/hMXiOfYEdO05lZwzORY7O931H53xzNONTEnLhOD+vXb/xx8211q3b7Tt3791/sP7w0TFPUoZhgBOSsFMfcSBhDAMRCgKnlAGKfAIn/nQ3x09mwHiYxEciozCMUBCH4xAjoVPn6y+8BEvPj6QXpUqde5e293f+sz2/zPMwiFABna93nJ5TfLYZuGXQscrv4PzhWs8bJTiNIBaYIM7PXIeKoURMhJiAanspB4rwFAVwlorx9lCGMU0FxFjZTzU2ToktEjtv3B6FDLAgmQ4QZqGuYOMJYggLPV67XopDjCLg3dEspHwR8lmwCATS3gzlvPBO3aspZcAQnYR4XmtNoohHSEyMJM8iv56ElACbRfVk3qZucoU5B4ZDnptwoJ3Zp/l58KPkoMQnGZ1AzJVMGVFVoQaAMRhrYRFyECmVxTR6Cab8jWApdPOwyL3ZQ2x6CKOurlNL1NsZkwSJesrXY2h3YrjASRSheCQ9qqQnYC6k1+2pwrsqeqikXiNtlO/bhzlcQ79W0K9KrWgHV+jYHmi0Bh5XwGOj8EkFPVmV+mkFTQ10VkFnRmX/ogJfGPC8gs4NNKugmYFeVtBL00qkT/6sP5QLu4tzk/sknMFHBhAr2emr1VmYPtIzty7Jj1l2XFXYPYKxviQWQJTldPnp6MtnJXe3+6+cTbXK8EkKS4qzsflq1zEowaKbkuNsb/d3DE7CUBxcFdp7v/nONQvRlFFyRdra2vjw2qyUASHJxVWl3Z29/sbqHjFsmFDOandc2zAtaKKXUzUK/CbBwqlG/tTkf2Qo+wU7aaq+NLBRQZsUSzcbFVmTYmntUrEyBM23daofBJpf3YgsKHugL3UGX/QW7+uLCImEPdery4Io1Pbpf6+bR78jovmSqKN2W78w7up7YgbH/Z672XO+vey83SnfmlvWX9YT65nlWlvWW+uTdWANLGz9Y/1rfbd+rP3X+rNlt54sqNevlZrHVu1rvfgfCTOyLw==</latexit>µz �z

87









reconstruction lossKL loss


loss(v,w) = KL(qv(z | x),N(0, I))- lnpw(x | e⌦ �z + µz)ThetwotermsofthelossfunctionareusuallycalledKLlossandreconstructionloss.

Thereconstructionlossmaximisestheprobabilityofthecurrentinstances.Thisisbasicallythesamelossweusedfortheregularauto-encoder:wewanttheoutputofthedecodertolookliketheinput.

TheKLlossensuresthatthelatentdistributionsareclusteredaroundtheorigin,withvariance1.Overthewholedataset,itensuresthatthelatentdistributionlookslikeastandardnormaldistribution.

three forces

88


µz �z


reconstruction loss

KL loss

N(0, I)

TheformulationoftheVAEhasthreeforcesactingonthelatentspace.Thereconstructionlosspullsthelatentdistributionasmuchaspossibletowardsasinglepointthatgivesthebestreconstruction.Meanwhile,theKLloss,pullsthelatentdistribution(forallpoints)towardsthestandardnormaldistribution,actingasaregularizer.Finally,thesamplingstepensuresthatnotjustasinglepointreturnsagoodreconstruction,butawholeneighbourhoodofpointsdoes.Theeffectcanbesummarizedasfollows:

Thereconstructionlossensuresthattherearepointsinthelatentspacethatdecodetothedata.

TheKLlossensuresthatallthesepointstogetherarelaidoutlikeastandardnormaldistribution.

Thesamplingstepensurethatpointsinbetweenthesealsodecodetopointsthatresemblethedata.

choosing rec. loss

89



squared error

absolute error sharper images Laplace distribution

cross-entropy fast conv, obscure distribution


loss(v,w) = KL(qv(z | x),N(0, I))- lnpw(x | e⌦ �z + µz)

ThetwotermsofthelossfunctionareusuallycalledKLlossandreconstructionloss.

Thereconstructionlossmaximisestheprobabilityofthecurrentinstances.Thisisbasicallythesamelossweusedfortheregularauto-encoder:wewanttheoutputofthedecodertolookliketheinput.

TheKLlossensuresthatthelatentdistributionsareclusteredaroundtheorigin,withvariance1.Overthewholedataset,itensuresthatthelatentdistributionlookslikeastandardnormaldistribution.

after 300 epochs

90data ae vae

generated91

ae vae

92

with VAEs

93source: Deep Feature Consistent Variational Autoencoder by Xianxu Hou, Linlin Shen, Ke Sun, Guoping Qiu

HerearesomeexamplesfromamoreelaborateVAE

source: Deep Feature Consistent Variational Autoencoder by Xianxu Hou, Linlin Shen, Ke Sun, Guoping Qiu

with VAEs

94source: https://houxianxu.github.io/assets/project/dfcvae

source: https://houxianxu.github.io/assets/project/dfcvae

from worksheet 5

95https://github.com/mlvu/worksheets/blob/master/Worksheet%205%2C%20Pytorch.ipynb

Hereiswhatthealgorithmlookslikeinpytorch.Loadthe5thworksheettogiveitatry.

https://github.com/mlvu/worksheets/blob/master/Worksheet%205%2C%20Pytorch.ipynb

96

Inthisworksheet,theVAEistrainedonMNISTdata,witha2Dlatentspace.Hereistheoriginaldata,plottedbytheirlatentscoordinates.Thecolorsrepresentclasses,towhichtheVAEdidnothaveaccess.

Ifyouruentheworksheet,you’llendupwiththispicture.

interpolation

97

VAE

AE

WhiletheaddedvalueoftheVAEisabitdif?iculttodetectinourexample,inotherdomainsit’smoreclear.

Hereisanexampleofinterpolationonsentences.Firstusingaregularautoencoder,andthenusingaVAE.NotethattheintermediatesentencesfortheAEarenon-grammatical,buttheintermediatesentencesfortheVAEareallgrammatical.

source: Generating Sentences from a Continuous Space by Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio https://arxiv.org/abs/1511.06349

summary: generative modeling

98

D

G

GANs VAEs







WeseethatGANsareinmanywaystheinverseofautoencoders,inthatGANShavethedataspaceastheinsideofthenetwork,andVAEshaveitastheoutside.

GANs and VAEs

Better for images, often poor in other domains.

Ad-hoc model, difficult to establish what is being optimized.

Can’t handle discrete data easily.

Derived by inverting a discriminator.

99

GANs VAEsWork for language, music, etc.

Derived from first principles.

Allow mapping from data to latent space.

Can’t handle discrete latent variables easily.

Derived by inverting a generator.

Allow us to train generator networks, avoiding mode collapse

VAEs and PCA

Linear transformation

Analytical solution

Principal components often meaningful, ordered by impact.

100

Nonlinear transformation

GD required

Latent dimensions not usually meaningful. Directions in latent space meaningful.

PCA VAEs

Mapping to low-dimensional whitened latent space (zero, mean, decorrelated).

[email protected]

Deep Learning for generative modeling - MLVU Learning2.annotated.pdfPos from X Neg Generate...

Documents

Transcript of Deep Learning for generative modeling - MLVU Learning2.annotated.pdfPos from X Neg Generate...