A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit...
Transcript of A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit...
![Page 1: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/1.jpg)
A Hybrid DL and GM (cont’d)+
Applications in Computer Vision
Kayhan BatmanghelichMingming Gong
1
![Page 2: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/2.jpg)
Outlines
• Joint deep feature extraction and learning
• Variational Auto Encoder (VAE)
• Generative Adversarial Networks (GANs)
2
![Page 3: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/3.jpg)
Recap: Variational Auto Encoder (VAE)
3
Data code-length Hypothesis codeStochastic encoder
Encoder!"($|&)
Data x
Amortization(sharing parameters)
max�,✓
F(x, q�) = Eq�(z) [log p✓(x|z)]�KL [q�(z)||p(z)]<latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit><latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit><latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit><latexit sha1_base64="6LBvx+YoSQ/o0mq+5Y6l5LB5K9A=">AAACh3icbVHdbtMwFHbCgK38dXDJzREVUopGSdgEu0EaIBASXAyJbpOaKHJcp7HmJJ59gtqleRUeijveBqdNBWwcyfKn78f2OU6UFAZ9/5fj3ti6eev29k7vzt179x/0dx+emLLSjI9ZKUt9llDDpSj4GAVKfqY0p3ki+Wly/r7VT79zbURZfMOF4lFOZ4VIBaNoqbj/I8zpPK5DlYm9EDOOtAFLYcaorD823nzvwqoborU1zRDerD1JUn9o4vpinW+8y6ENS57ixG7lDJQVujO9+fJyCKEWswwjeA6fv2ycmzTYOCyXoLw/xl7cH/gjf1VwHQQdGJCujuP+z3BasirnBTJJjZkEvsKophoFk7zphZXhirJzOuMTCwuacxPVqzk28NQyU0hLbVeBsGL/TtQ0N2aRJ9bZtm+uai35P21SYXoY1aJQFfKCrS9KKwlYQvspMBWaM5QLCyjTwr4VWEY1ZWi/rh1CcLXl6+Dk5SjwR8HXg8HRu24c2+QxeUI8EpDX5Ih8IsdkTJiz5Txz9p0Dd8d94b5yD9dW1+kyj8g/5b79DWPgwxQ=</latexit>
z ⇠ q�(z|x)<latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit><latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit><latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit><latexit sha1_base64="kSwGc8xw1Nm9fCA2TEmgm0faw0g=">AAACnXicbVFba9swFJa9W5fdsu1te9hhoeBAF+wxWF8KoaNjlzI6WJpCbIysyLGofKl0XJI4/lf7JXvbv5mcOHRrd0Do47voSEdRIYVG1/1t2bdu37l7b+d+58HDR4+fdJ8+O9V5qRgfsVzm6iyimkuR8REKlPysUJymkeTj6PxDo48vudIiz37gouBBSmeZiAWjaKiw+3PXT+k8rPwiEXs+JhxpDYbChFFZfayd+d6FUbdEY6vrPhxsPFFUHdVhdbHJ186yb8KSxzgxWz6Dwgjtmc58teyDr8QswQDewNfjrXObBhOH1QoK58rYWYKvRQpXHVbzfifs9tyBuy64CbwW9EhbJ2H3lz/NWZnyDJmkWk88t8CgogoFk7zu+KXmBWXndMYnBmY05Tqo1tOtYdcwU4hzZVaGsGb/TlQ01XqRRsbZzERf1xryf9qkxHg/qERWlMgztmkUlxIwh+arYCoUZygXBlCmhLkrsIQqytB8aDME7/qTb4LTtwPPHXjf3/WGh+04dshL8po4xCPvyZB8IidkRJj1whpan60v9iv7yD62v22sttVmnpN/yh7/AXMFy6I=</latexit>
Example:
![Page 4: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/4.jpg)
Recap: Mapping the Distributions
4
• VAE learns a mapping that transforms a simple distribution, q(z), to a complicate distribution, q(z|x) so that the empirical distribution (in the x-space) are matched.
Fig Credit: Kingma’s thesis
![Page 5: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/5.jpg)
Recap: Reparametrization Trick
5
Can we improve the estimation of gradient?
(reduce the variance)
g(·)<latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit><latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit><latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit><latexit sha1_base64="NqFKqTagOiOKxD8J2RuIgerdfII=">AAACv3icbVFbi9QwFE7rba23UR99CQ4DLaxDuwjuwwqrogj6sIKzuzAtJc2kbZj0ssmpdKbTP+mD4L8xnXYY3fVAyMd3OUlOolJwBa772zBv3b5z997BfevBw0ePn4yePjtXRSUpm9FCFPIyIooJnrMZcBDsspSMZJFgF9HyQ6df/GBS8SL/DquSBRlJch5zSkBT4ejXxM9IHTZ+mfJDH1IGpMWagpQS0Xxq7frwSqs7orO1rYPf9p4oaj62YXPV51t77eiwYDHM9VYkuNTC0NOuN2sH+5InKQT4Ff7ydefcpbGO480Gl/beaE3W2Fc8w/sjNrVjTeqe3ffftrcS26eLAhwrHI3dqbstfBN4Axijoc7C0U9/UdAqYzlQQZSae24JQUMkcCpYa/mVYiWhS5KwuYY5yZgKmu38WzzRzALHhdQrB7xl/040JFNqlUXa2U1NXdc68n/avIL4OGh4XlbActofFFcCQ4G7z8QLLhkFsdKAUMn1XTFNiSQU9Jd3Q/CuP/kmOD+aeu7U+/Z6fPp+GMcBeoFeIht56A06RZ/RGZohapwYkbE0hPnOTMzcLHuraQyZ5+ifMld/AEQ22Gc=</latexit>
z ⇠ p�(z)<latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit><latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit><latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit><latexit sha1_base64="z+58agIDONqdNqB0Sr1jIXx7N3U=">AAACyHicbVFbi9NAFJ7E2xpvVR99GSyFFNaSLIK+CIuyIurDCnZ3oQlhMp00QyfJ7MyJpk3z4k/0zSf/ipMmpbrrgSEf3+XM5JxYCq7B835Z9o2bt27fObjr3Lv/4OGjweMnZ7ooFWVTWohCXcREM8FzNgUOgl1IxUgWC3YeL9+1+vk3pjQv8q+wkizMyCLnCacEDBUNfo+CjFRRHciUHwaQMiANNhSklIj6feNWh5dG3RGtrWnG+E3nieP6pInqyy7fuOuxCQuWwMx8igWWRuh7utVmPcaB4osUQvwCf/q8c+7S2MTxZoOluzc6ozUONM/w/opNNXZGVcfu+2/bO+s93b8nGgy9ibctfB34PRiivk6jwc9gXtAyYzlQQbSe+Z6EsCYKOBWscYJSM0nokizYzMCcZEyH9XYRDR4ZZo6TQpmTA96yfydqkmm9ymLjbMenr2ot+T9tVkLyOqx5LktgOe0uSkqBocDtVvGcK0ZBrAwgVHHzVkxToggFs3vHDMG/+svXwdnRxPcm/peXw+O3/TgO0DP0HLnIR6/QMfqATtEUUevEWlpglfZHW9rf7VVnta0+8xT9U/aPPxss3Ic=</latexit>
Let’s revisit the change of variables:
![Page 6: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/6.jpg)
Recap: How to use this technique in a Graphical Model?
6
Example: Mixture Model
M. Johnson, Composing graphical models with neural networks for structured representations and fast inference, https://arxiv.org/pdf/1603.06277.pdf
✓<latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit><latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit><latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit><latexit sha1_base64="3mclGziwalKoiRKKpEfy5DrS5pY=">AAACvHicbVFbb9MwFHbCZSPcCjzyYlFVSqVRJQiJ8QCaQCAkeBgS3SY1UeS4TmPVSTz7ZEqb5k/CE/8Gp0lV2DiSlc/f5dg5jqXgGjzvt2Xfun3n7sHhPef+g4ePHg+ePD3TRakom9JCFOoiJpoJnrMpcBDsQipGsliw83j5sdXPr5jSvMh/wEqyMCOLnCecEjBUNPg1CjJSRXUgU34UQMqANNhQkFIi6s+NWx1dGnVHtLamGeN3nSeO609NVF92+cZdj01YsARm5lMssDRC39OtNusxDhRfpBDil/jrt51zl8YmjjcbLN290RmtcaB5hvdHbKqxM6o6dt9/297pNtFg6E28beGbwO/BEPV1Gg1+BvOClhnLgQqi9cz3JIQ1UcCpYI0TlJpJQpdkwWYG5iRjOqy3w2/wyDBznBTKrBzwlv07UZNM61UWG2c7Mn1da8n/abMSkuOw5rksgeW0OygpBYYCty+J51wxCmJlAKGKm7timhJFKJj3dswQ/Ou/fBOcvZr43sT//np48qEfxyF6jl4gF/noDTpBX9ApmiJqvbUiK7W4/d6e20s766y21WeeoX/KvvoDatXX8w==</latexit>
yn<latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit><latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit><latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit><latexit sha1_base64="gFFKtqjEn37Pbp4cGW+uwbmKO1s=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnNzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBOodap</latexit>
zn<latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit><latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit><latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit><latexit sha1_base64="pDIyV9KUtQ3ixZMU0R5Ay3U2agU=">AAACuXicbVFbi9NAFJ7E2xpvVR99OVgKKawlEVFBhEVRBH1Ywe4uNKFMppNm3MllZ04kbZr/KL75b5w0KdVdDwzz8V3OzJyJCik0et5vy752/cbNWwe3nTt3791/MHj46ETnpWJ8ynKZq7OIai5FxqcoUPKzQnGaRpKfRufvW/30B1da5Nk3XBU8TOkyE7FgFA01H/wcBSmt5nVQJOIwwIQjbcBQmDAq64+NWx1eGHVHtLamGcPbzhNF9YdmXl90+cZdj01Y8hhnZsuXUBih7+lWm/UYAiWWCYbwDD5/2Tl3aTBx2GygcPdGZ7SGQIsU9kdsqrEzqjp233/b3lnPzZOG3sTbFlwFfg+GpK/j+eBXsMhZmfIMmaRaz3yvwLCmCgWTvHGCUvOCsnO65DMDM5pyHdbbyTcwMswC4lyZlSFs2b8TNU21XqWRcbbz0pe1lvyfNisxfh3WIitK5BnrDopLCZhD+42wEIozlCsDKFPC3BVYQhVlaD7bMUPwLz/5Kjh5PvG9if/1xfDoXT+OA/KEPCUu8ckrckQ+kWMyJcx6aQUWt2L7jU3txP7eWW2rzzwm/5St/wBQJ9aq</latexit>
�<latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit><latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit><latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit><latexit sha1_base64="SApNCgqVwNq2hovs7dKXK2lLlRw=">AAACvHicbVFbb9MwFHbCZSPcCjzyckRVKZVGlSAkxgNoAoGQ4GFIdJvUVJHjOolVO8lsZ0qb5k/CE/8Gp0lV2DiS5U/f5dg+jgrOlPa835Z96/aduweH95z7Dx4+ejx48vRM5aUkdEpynsuLCCvKWUanmmlOLwpJsYg4PY+WH1v9/IpKxfLsh14VdC5wkrGYEawNFQ5+jQKBq7AOipQdBTqlGjdgKJ0SzOvPjVsdXRp1R7S2phnDu84TRfWnJqwvu3zjrscmzGmsZ2bLEyiM0Pd0q816DIFkSarn8BK+fts5d2kwcdhsoHD3Rme0hkAxAfsjNtXYGVUdu++/be8ECRYCh4OhN/G2BTeB34Mh6us0HPwMFjkpBc004Vipme8Vel5jqRnhtHGCUtECkyVO6MzADAuq5vV2+A2MDLOAOJdmZRq27N+JGgulViIyznZk6rrWkv/TZqWOj+c1y4pS04x0B8UlB51D+5OwYJISzVcGYCKZuSuQFEtMtPlvxwzBv/7km+Ds1cT3Jv7318OTD/04DtFz9AK5yEdv0An6gk7RFBHrrRVaqcXs9/bCXtqis9pWn3mG/in76g9Nydfg</latexit>
![Page 7: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/7.jpg)
Limitations of VAE• When applied to image data, it results into
blurry images.• For images, it is sensitive to irrelevant
variance, e.g., translations.• Not applicable to discrete latent variables.• Limited flexibility: converting the data
distribution to fixed, single-mode prior distribution
7
![Page 8: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/8.jpg)
GAN: GENERATIVE ADVERSARIAL NETWORK
8
![Page 9: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/9.jpg)
GANs
• Introduced by Goodfellow et al., 2014• Assumes implicit generative model• Asymptotically consistent (unlike
variational methods)• No Markov chains needed• Often regarded as producing the best
samples (but,… no good way to quantify it)
9
![Page 10: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/10.jpg)
Generator Network
10
z ⇠ N (0, I)<latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit><latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit><latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit><latexit sha1_base64="yqFcrQprV4aweSd5C/hTed5XlQU=">AAACznicbVJbi9NAFJ7Ey67xVvXRl4OlkEAtiQjui7AoirIiFezuQhPKZDpphp1cOjNZ0qbBV3+fb/4A/4eTJrW664EhH9/lzJmZhDlnUrnuT8O8cfPW7YPDO9bde/cfPOw9enwqs0IQOiEZz8R5iCXlLKUTxRSn57mgOAk5PQsv3jb62SUVkmXpV7XKaZDgRcoiRrDS1Kz3a+AnuJxVfh6zoa9iqnANmlIxwbx6X9vlcKnVHdHY6tqB160nDKt39axatvnaXjs6zGmkpvqTLSDXQtfTLjdrB3zBFrEK4DmcfNo5d2nQcdhsILf3RmuwBl+yBPZbbErHGpQtu++/bW915j/jfq5td/jRgVmv747cbcF14HWgj7oaz3o//HlGioSminAs5dRzcxVUWChGOK0tv5A0x+QCL+hUwxQnVAbV9jlqGGhmDlEm9EoVbNm/ExVOpFwloXY2g8qrWkP+T5sWKjoKKpbmhaIpaTeKCg4qg+ZtYc4EJYqvNMBEMD0rkBgLTJT+Ayx9Cd7VI18Hpy9GnjvyvrzsH7/pruMQPUXPkI089Aodow9ojCaIGCfG0lgblTk2L83a/NZaTaPLPEH/lPn9NwLK3iY=</latexit>
• Maps noise variable to the data space• Must be differentiable but no invertibility is
required
![Page 11: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/11.jpg)
1-D Example
11Figure: Roger Grosse
![Page 12: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/12.jpg)
Learning
• Train ! to discriminate between training examples and generated samples
• Train " to fool the discriminator
12Figure courtesy: Kim’s slides
![Page 13: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/13.jpg)
Minimax Game
Fixing G
13
Optimal D:
Taking derivative of !log % + 'log(1 − %)
![Page 14: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/14.jpg)
Discriminator Strategy
14
![Page 15: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/15.jpg)
Minimax Game
Substituting
15
GANs minimizes the Jensen–Shannon divergence between two distributions.
![Page 16: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/16.jpg)
More Stable Version
16Redford et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”
• DCGAN: Most “deconvs” are batch normalized
![Page 17: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/17.jpg)
17Redford et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”
![Page 18: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/18.jpg)
Practical Issues
18
Strong intra-batch correlation
![Page 19: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/19.jpg)
Practical Issues
19
Mode Collapse
![Page 20: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/20.jpg)
Practical Issues
20
![Page 21: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/21.jpg)
Summary• Does not need a specific likelihood function at the last layer to
train a latent-variable model• Similar move from Exact inference to MCMC to var. inf: Don’t
restrict model to allow easy inference - just let a neural network clean up after.
• Topics of research:– How to compute P(z|x), like VAE, without compromising the
quality of samples?– How to make GANs more stable: various divergences, … ?– How to handle discrete inputs ?– How to generalize to idea to GM. Notable works ?
• “Averserial Variational Bayes”, Sebastian Nowozin, Andreas Geiger, 2017 • “Learning in Implicit Generative Models,” Shakir Mohamed, Balaji
Lakshminarayanan, 2016 • “Variational Inference using Implicit Distributions,” Ferenc Huszar, 2017• “Deep and Hierarchical Implicit Models.” Dustin Tran, Rajesh Ranganath,
David Blei, 2017
21
![Page 22: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/22.jpg)
Applications in Vision
Mingming Gong
22
![Page 23: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/23.jpg)
�������
• Semantic Segmentation
• Image-to-Image Translation
23
![Page 24: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/24.jpg)
24
• Partition an image into semantically meaningful regions
![Page 25: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/25.jpg)
25
![Page 26: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/26.jpg)
26
![Page 27: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/27.jpg)
27
Krähenbühl, Philipp, and Vladlen Koltun. "Efficient inference in fully connected crfs with gaussian edge potentials." Advances in neural information processing systems. 2011.
![Page 28: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/28.jpg)
28
• Model long-range connections
• Inference is computationally hard
![Page 29: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/29.jpg)
29
• Label ! = X$,… , X'
• Image ( = I$, … , I'• Gibbs distribution
• Energy function
![Page 30: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/30.jpg)
30
• Label ! = X$,… , X'
• Image ( = I$, … , I'• Gibbs distribution
• Energy function
Unary Pairwise
![Page 31: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/31.jpg)
31
• Unary !"($%)• Computed independently for each pixel• Multinomial logistic regression, boosting
![Page 32: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/32.jpg)
32
• Pairwise potential
!" #$, #& = ((#$, #&)∑,-./ 0 , 1 , (2$, 2&)
• ( - label compatibility function
• k – linear combination of Gaussian kernels
1 , 3$, 3& = exp(−12 2$ − 2& Σ , 2$ − 2& )
![Page 33: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/33.jpg)
33
!" #$, #& = ((#$, #&)∑,-./ 0 , 1 , (2$, 2&)
• ( - label compatibility function
• k – Appearance and local smoothness
( #$, #& = [#$ ≠ #&]Potts Model:
![Page 34: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/34.jpg)
34
https://www.cs.cmu.edu/~efros/courses/LBMV12/crf_deconstruction.pdf
![Page 35: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/35.jpg)
35
• Mean field approximation• Minimize KL divergence !(#||%),
s.t. # ' =)*#*('*)
• Iterative update equation
• Message passing from all '+ to all '*
![Page 36: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/36.jpg)
36
http://vladlen.info/papers/densecrf-slides.pdf
![Page 37: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/37.jpg)
37
• Downsample
![Page 38: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/38.jpg)
38
• Downsample
• Blur the sampled signal
![Page 39: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/39.jpg)
39
• Downsample
• Blur the sampled signal
• Upsample to reconstruct
![Page 40: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/40.jpg)
40
�����������
![Page 41: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/41.jpg)
41
• MSRC dataset• 591 images• 21 classes
![Page 42: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/42.jpg)
42
• PASCAL dataset• 1928 images• 21 classes
![Page 43: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/43.jpg)
43
• CRF can refine the results from unary classifiers
• Limitation: the unary classifiers trained on handcrafted features
• Solution: Deep convolutional nets for dense feature extraction
![Page 44: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/44.jpg)
44
��������
![Page 45: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/45.jpg)
45
![Page 46: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/46.jpg)
46
���� ���������
![Page 47: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/47.jpg)
47
![Page 48: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/48.jpg)
48
• PASCAL dataset• 10, 582 images• 21 classes
![Page 49: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/49.jpg)
�������
• Semantic Segmentation
• Image-to-Image Translation
49
![Page 50: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/50.jpg)
50
![Page 51: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/51.jpg)
51
• Multi-output regression
• ! " − $ %
?
![Page 52: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/52.jpg)
52
z
Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." arXiv preprint (2017).
![Page 53: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/53.jpg)
53
���z
![Page 54: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/54.jpg)
54
![Page 55: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/55.jpg)
55
![Page 56: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/56.jpg)
56
Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arXiv preprint arXiv:1511.06434 (2015).
![Page 57: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/57.jpg)
57
����������
![Page 58: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/58.jpg)
58
![Page 59: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/59.jpg)
59
![Page 60: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/60.jpg)
60
P(X, Y) P(X) P(Y)
P(Y|X) Infinite solutions!!
![Page 61: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/61.jpg)
… Discriminator D": #$%& ' ( , *Real zebras vs. generated zebras ……
http://people.eecs.berkeley.edu/~junyanz/talks/pix2pix_cyclegan.pptx
Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint arXiv:1703.10593 (2017).
![Page 62: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/62.jpg)
Discriminator D": #$%& ' ( , *Real horses vs. generated horses
Discriminator D+: #$%& , * , (Real zebras vs. generated zebras ……
![Page 63: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/63.jpg)
mode collapse!
![Page 64: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/64.jpg)
Discriminator
x G(x)
DGenerator
G Real!
![Page 65: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/65.jpg)
Discriminator
x G(x)
DGenerator
G Real too!
![Page 66: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/66.jpg)
x G(x)
Generator
GD
No input-output pairs!
![Page 67: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/67.jpg)
Cycle-consistency LossForward cycle loss: F G x − x %G(x) F(G x )x
![Page 68: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/68.jpg)
Cycle-consistency LossLarge cycle lossForward cycle loss: F G x − x %G(x) F(G x )x Small cycle loss
![Page 69: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/69.jpg)
Cycle-consistency LossBackward cycle loss: ! " # − # %Forward cycle loss: F G x − x %G(x) F(G x )x F(y) G(F x )#
![Page 70: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/70.jpg)
���������
GAN loss:
Cycle consistent loss:
Autoencoder
![Page 71: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/71.jpg)
Cezanne Ukiyo-eMonetInput Van Gogh
![Page 72: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/72.jpg)
![Page 73: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/73.jpg)
Failure cases
![Page 74: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/74.jpg)
74
……
Constrain !" and !#
M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. In NIPS, pages 469–477, 2016. 3, 5
![Page 75: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/75.jpg)
75
![Page 76: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/76.jpg)
����
76
……
Infer z from x, VAE, ALI
Ming-Yu Liu, Thomas Breuel, Jan Kautz, "Unsupervised Image-to-Image Translation Networks" NIPS 2017 Spotlight, arXiv:1703.00848 2017
![Page 77: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/77.jpg)
����
77
![Page 78: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/78.jpg)
����
78
!"##→% = '%()# ∼ +# )# "# )
![Page 79: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/79.jpg)
����
79
![Page 80: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/80.jpg)
����
80
![Page 81: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/81.jpg)
����
81
![Page 82: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/82.jpg)
82
�������
![Page 83: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/83.jpg)
83
�������
![Page 84: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/84.jpg)
84
�������
![Page 85: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/85.jpg)
85
Hoffman, Judy, et al. "CyCADA: Cycle-Consistent Adversarial Domain Adaptation." arXivpreprint arXiv:1711.03213 (2017).
![Page 86: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/86.jpg)
86
![Page 87: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/87.jpg)
87
![Page 88: A Hybrid DL and GM (cont’d) Applications inComputerVision · •“Learning in Implicit Generative Models,” ShakirMohamed, Balaji Lakshminarayanan, 2016 •“VariationalInference](https://reader034.fdocuments.in/reader034/viewer/2022042308/5ed40f5a8d46b66d226360e3/html5/thumbnails/88.jpg)
88
�������