CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning...

38
CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised Learning of Structure Graham Neubig Site https://phontron.com/class/nn4nlp2019/

Transcript of CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning...

Page 1: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

CS11-747 Neural Networks for NLP Unsupervised and Semi-supervised

Learning of StructureGraham Neubig

Sitehttps://phontron.com/class/nn4nlp2019/

Page 2: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Supervised, Unsupervised, Semi-supervised

• Most models handled here are supervised learning

• Model P(Y|X), at training time given both

• Sometimes we are interested in unsupervised learning

• Model P(Y|X), at training time given only X

• Or semi-supervised learning

• Model P(Y|X), at training time given both or only X

Page 3: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning Features vs. Learning Structure

Page 4: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning Features vs. Learning Discrete Structure

• Learning features, e.g. word/sentence embeddings:

this is an example

• Learning discrete structure:

• Why discrete structure? • We may want to model information flow differently • More interpretable than features?

this is an example this is an examplethis is an example this is an example

Page 5: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Unsupervised Feature Learning (Review)

• When learning embeddings, we have an objective and use the intermediate states of this objective

• CBOW

• Skip-gram

• Sentence-level auto-encoder

• Skip-thought vectors

• Variational auto-encoder

Page 6: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

How do we Use Learned Features?

• To solve tasks directly (Mikolov et al. 2013)

• And by proxy, knowledge base completion, etc., to be covered in a few classes

• To initialize downstream models

Page 7: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

What About Discrete Structure?

• We can cluster words

• We can cluster words in context (POS/NER)

• We can learn structure

Page 8: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

What is our Objective?• Basically, a generative model of the data X

• Sometimes factorized P(X|Y)P(Y), a traditional generative model

• Sometimes factorized P(X|Y)P(Y|X), an auto-encoder

• This can be made mathematically correct through variational autoencoder P(X|Y)Q(Y|X)

Page 9: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Clustering Words in Context

Page 10: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

A Simple First Attempt• Train word embeddings

• Perform k-means clustering on them

• Implemented in word2vec (-classes option)

• But what if we want single words to appear in different classes (same surface form, different values)

Page 11: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Hidden Markov Models• Factored model of P(X|Y)P(Y) • State→state transition probabilities • State→word emission probabilities

<s> JJ NN NN LRB NN RRB … </s>

Natural Language Processing ( NLP ) …

PE(Natural|JJ) * PE(Language|JJ) * PE(Processing|JJ) * …

PT(JJ|<s>) * PT(NN|JJ) * PT(NN|NN) * PT(NN|LRB) * …

Page 12: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Unsupervised Hidden Markov Models

• Change label states to unlabeled numbers

0 13 17 17 6 12 6 … 0

Natural Language Processing ( NLP ) …

PE(Natural|13) * PE(Language|17) * PE(Processing|17) * …

PT(13|0) * PT(17|13) * PT(17|17) * PT(6|17) * …

• Can be trained with forward-backward algorithm

Page 13: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Hidden Markov Models w/ Gaussian Emissions

• Instead of parameterizing each state with a categorical distribution, we can use a Gaussian (or Gaussian mixture)!

• Long the defacto-standard for speech • Applied to POS tagging by training to emit word embeddings by

Lin et al. (2015)

0 13 17 17 6 12 6 … 0

Page 14: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

A Simple Approximation: State Clustering (Giles et al. 1992)

• Simply train an RNN according to a standard loss function (e.g. language model)

• Then cluster the hidden states according to k-means, etc.

Page 15: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Featurized Hidden Markov Models (Tran et al. 2016)

• Calculate the transition/emission probabilities with neural networks! • Emission: Calculate representation of each word in vocabulary w/

CNN, dot product with tag representation and softmax to calculate emission prob

• Transition Matrix: Calculate w/ LSTMs (breaks Markov assumption)

Page 16: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Problem: Embeddings May Not be Indicative of Syntax

(He et al. 2018)

noun proper

verb base

cardinal number

verb gerund

adjective

noun singular

noun plural

adverb

verb past tenseverb past participleverb 3rd singular

Page 17: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning POS Taggersw/ Latent Embeddings

(He et al. 2018)

z1<latexit sha1_base64="jbtr+W9vpRfg2saI7YImOblkNik=">AAACCHicZVDLSgMxFM3UV62vqks3g0VwUcqMCOqu6MZlRccW2qFk0kwbmkyG5I5Qh36BuNXvcCVu/Qs/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7GN9m8/USVZjJ6gElMfYGHEQsZwWCo++e+26/WnIaTl70M3ALUUFGtfvW3N5AkETQCwrHWXdeJwU+xAkY4nVZ6iaYxJmM8pF0DIyyo9tPc6tQ+MczADqUyJwI7Z/9vpFhogWFklFnTc7OMASm5rhsVjETWsmfyu56IoB6IeiZSOtQLRiC89FMWxQnQiMx8hAm3QdpZLPaAKUqATwzARDHzFZuMsMIETHgVk5G7mMgy8M4aVw3n7rzWvC7CKqMjdIxOkYsuUBPdohbyEEFD9Ire0Lv1Yn1Yn9bXTFqyip1DNFfW9x/uTZqw</latexit><latexit sha1_base64="jbtr+W9vpRfg2saI7YImOblkNik=">AAACCHicZVDLSgMxFM3UV62vqks3g0VwUcqMCOqu6MZlRccW2qFk0kwbmkyG5I5Qh36BuNXvcCVu/Qs/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7GN9m8/USVZjJ6gElMfYGHEQsZwWCo++e+26/WnIaTl70M3ALUUFGtfvW3N5AkETQCwrHWXdeJwU+xAkY4nVZ6iaYxJmM8pF0DIyyo9tPc6tQ+MczADqUyJwI7Z/9vpFhogWFklFnTc7OMASm5rhsVjETWsmfyu56IoB6IeiZSOtQLRiC89FMWxQnQiMx8hAm3QdpZLPaAKUqATwzARDHzFZuMsMIETHgVk5G7mMgy8M4aVw3n7rzWvC7CKqMjdIxOkYsuUBPdohbyEEFD9Ire0Lv1Yn1Yn9bXTFqyip1DNFfW9x/uTZqw</latexit><latexit sha1_base64="jbtr+W9vpRfg2saI7YImOblkNik=">AAACCHicZVDLSgMxFM3UV62vqks3g0VwUcqMCOqu6MZlRccW2qFk0kwbmkyG5I5Qh36BuNXvcCVu/Qs/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7GN9m8/USVZjJ6gElMfYGHEQsZwWCo++e+26/WnIaTl70M3ALUUFGtfvW3N5AkETQCwrHWXdeJwU+xAkY4nVZ6iaYxJmM8pF0DIyyo9tPc6tQ+MczADqUyJwI7Z/9vpFhogWFklFnTc7OMASm5rhsVjETWsmfyu56IoB6IeiZSOtQLRiC89FMWxQnQiMx8hAm3QdpZLPaAKUqATwzARDHzFZuMsMIETHgVk5G7mMgy8M4aVw3n7rzWvC7CKqMjdIxOkYsuUBPdohbyEEFD9Ire0Lv1Yn1Yn9bXTFqyip1DNFfW9x/uTZqw</latexit>

z2<latexit sha1_base64="ZNKuRe23lzCJMNesh6cRgweKXJI=">AAACCHicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3Lio4W2qFk0kwbmkyG5I5Qh36BuNXvcCVu/Qs/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DBy0TRahHJJeqE2BNOYuoBww47cSKYhFw+hiMr7P54xNVmsnoHiYx9QUeRixkBIOh7p77zX615jScvOxl4Baghopq96u/vYEkiaAREI617rpODH6KFTDC6bTSSzSNMRnjIe0aGGFBtZ/mVqf2iWEGdiiVORHYOft/I8VCCwwjo8yanptlDEjJdd2oYCSylj2T3/VEBPVA1DOR0qFeMALhhZ+yKE6ARmTmI0y4DdLOYrEHTFECfGIAJoqZr9hkhBUmYMKrmIzcxUSWgddsXDac27Na66oIq4yO0DE6RS46Ry10g9rIQwQN0St6Q+/Wi/VhfVpfM2nJKnYO0VxZ33/v75qx</latexit><latexit sha1_base64="ZNKuRe23lzCJMNesh6cRgweKXJI=">AAACCHicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3Lio4W2qFk0kwbmkyG5I5Qh36BuNXvcCVu/Qs/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DBy0TRahHJJeqE2BNOYuoBww47cSKYhFw+hiMr7P54xNVmsnoHiYx9QUeRixkBIOh7p77zX615jScvOxl4Baghopq96u/vYEkiaAREI617rpODH6KFTDC6bTSSzSNMRnjIe0aGGFBtZ/mVqf2iWEGdiiVORHYOft/I8VCCwwjo8yanptlDEjJdd2oYCSylj2T3/VEBPVA1DOR0qFeMALhhZ+yKE6ARmTmI0y4DdLOYrEHTFECfGIAJoqZr9hkhBUmYMKrmIzcxUSWgddsXDac27Na66oIq4yO0DE6RS46Ry10g9rIQwQN0St6Q+/Wi/VhfVpfM2nJKnYO0VxZ33/v75qx</latexit><latexit sha1_base64="ZNKuRe23lzCJMNesh6cRgweKXJI=">AAACCHicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3Lio4W2qFk0kwbmkyG5I5Qh36BuNXvcCVu/Qs/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DBy0TRahHJJeqE2BNOYuoBww47cSKYhFw+hiMr7P54xNVmsnoHiYx9QUeRixkBIOh7p77zX615jScvOxl4Baghopq96u/vYEkiaAREI617rpODH6KFTDC6bTSSzSNMRnjIe0aGGFBtZ/mVqf2iWEGdiiVORHYOft/I8VCCwwjo8yanptlDEjJdd2oYCSylj2T3/VEBPVA1DOR0qFeMALhhZ+yKE6ARmTmI0y4DdLOYrEHTFECfGIAJoqZr9hkhBUmYMKrmIzcxUSWgddsXDac27Na66oIq4yO0DE6RS46Ry10g9rIQwQN0St6Q+/Wi/VhfVpfM2nJKnYO0VxZ33/v75qx</latexit>

z3<latexit sha1_base64="VwYPdoeVjXkUU912kwmjdymaS+E=">AAACCHicZVBLTsMwFHT4lvIrsGQTUSGxqKoEkIBdBRuWRRBaqY0qx3Vaq3Yc2S9IJeoJEFs4ByvElltwDG6Ak2ZB2ydZbzxvnjWeIOZMg+P8WEvLK6tr66WN8ubW9s5uZW//UctEEeoRyaVqB1hTziLqAQNO27GiWASctoLRTTZvPVGlmYweYBxTX+BBxEJGMBjq/rl31qtUnbqTl70I3AJUUVHNXuW325ckETQCwrHWHdeJwU+xAkY4nZS7iaYxJiM8oB0DIyyo9tPc6sQ+NkzfDqUyJwI7Z/9vpFhogWFolFnTM7OMASm5rhkVDEXWsmfyux6LoBaIWiZSOtRzRiC89FMWxQnQiEx9hAm3QdpZLHafKUqAjw3ARDHzFZsMscIETHhlk5E7n8gi8E7rV3Xn7rzauC7CKqFDdIROkIsuUAPdoibyEEED9Ire0Lv1Yn1Yn9bXVLpkFTsHaKas7z/xkZqy</latexit><latexit sha1_base64="VwYPdoeVjXkUU912kwmjdymaS+E=">AAACCHicZVBLTsMwFHT4lvIrsGQTUSGxqKoEkIBdBRuWRRBaqY0qx3Vaq3Yc2S9IJeoJEFs4ByvElltwDG6Ak2ZB2ydZbzxvnjWeIOZMg+P8WEvLK6tr66WN8ubW9s5uZW//UctEEeoRyaVqB1hTziLqAQNO27GiWASctoLRTTZvPVGlmYweYBxTX+BBxEJGMBjq/rl31qtUnbqTl70I3AJUUVHNXuW325ckETQCwrHWHdeJwU+xAkY4nZS7iaYxJiM8oB0DIyyo9tPc6sQ+NkzfDqUyJwI7Z/9vpFhogWFolFnTM7OMASm5rhkVDEXWsmfyux6LoBaIWiZSOtRzRiC89FMWxQnQiEx9hAm3QdpZLHafKUqAjw3ARDHzFZsMscIETHhlk5E7n8gi8E7rV3Xn7rzauC7CKqFDdIROkIsuUAPdoibyEEED9Ire0Lv1Yn1Yn9bXVLpkFTsHaKas7z/xkZqy</latexit><latexit sha1_base64="VwYPdoeVjXkUU912kwmjdymaS+E=">AAACCHicZVBLTsMwFHT4lvIrsGQTUSGxqKoEkIBdBRuWRRBaqY0qx3Vaq3Yc2S9IJeoJEFs4ByvElltwDG6Ak2ZB2ydZbzxvnjWeIOZMg+P8WEvLK6tr66WN8ubW9s5uZW//UctEEeoRyaVqB1hTziLqAQNO27GiWASctoLRTTZvPVGlmYweYBxTX+BBxEJGMBjq/rl31qtUnbqTl70I3AJUUVHNXuW325ckETQCwrHWHdeJwU+xAkY4nZS7iaYxJiM8oB0DIyyo9tPc6sQ+NkzfDqUyJwI7Z/9vpFhogWFolFnTM7OMASm5rhkVDEXWsmfyux6LoBaIWiZSOtRzRiC89FMWxQnQiEx9hAm3QdpZLHafKUqAjw3ARDHzFZsMscIETHhlk5E7n8gi8E7rV3Xn7rzauC7CKqFDdIROkIsuUAPdoibyEEED9Ire0Lv1Yn1Yn9bXVLpkFTsHaKas7z/xkZqy</latexit>

x3<latexit sha1_base64="r7RTsVKDbYXIRCyPj9szuInKMVA=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcqMCuqu6MZlBccW2qFk0kwbmkyG5I5Yhv6DuNXvcCVu/QY/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7Gt9m8/USVZjJ6gElMfYGHEQsZwWCoTi8Q6fO0f96v1pyGk5e9DNwC1FBRrX71tzeQJBE0AsKx1l3XicFPsQJGOJ1WeommMSZjPKRdAyMsqPbT3O/UPjHMwA6lMicCO2f/b6RYaIFhZJRZ03OzjAEpua4bFYxE1rJn8rueiKAeiHomUjrUC0YgvPJTFsUJ0IjMfIQJt0HaWTb2gClKgE8MwEQx8xWbjLDCBEyCFZORu5jIMvDOGtcN5/6i1rwpwiqjI3SMTpGLLlET3aEW8hBBHL2iN/RuvVgf1qf1NZOWrGLnEM2V9f0HLzudBQ==</latexit><latexit sha1_base64="r7RTsVKDbYXIRCyPj9szuInKMVA=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcqMCuqu6MZlBccW2qFk0kwbmkyG5I5Yhv6DuNXvcCVu/QY/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7Gt9m8/USVZjJ6gElMfYGHEQsZwWCoTi8Q6fO0f96v1pyGk5e9DNwC1FBRrX71tzeQJBE0AsKx1l3XicFPsQJGOJ1WeommMSZjPKRdAyMsqPbT3O/UPjHMwA6lMicCO2f/b6RYaIFhZJRZ03OzjAEpua4bFYxE1rJn8rueiKAeiHomUjrUC0YgvPJTFsUJ0IjMfIQJt0HaWTb2gClKgE8MwEQx8xWbjLDCBEyCFZORu5jIMvDOGtcN5/6i1rwpwiqjI3SMTpGLLlET3aEW8hBBHL2iN/RuvVgf1qf1NZOWrGLnEM2V9f0HLzudBQ==</latexit><latexit sha1_base64="r7RTsVKDbYXIRCyPj9szuInKMVA=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcqMCuqu6MZlBccW2qFk0kwbmkyG5I5Yhv6DuNXvcCVu/QY/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7Gt9m8/USVZjJ6gElMfYGHEQsZwWCoTi8Q6fO0f96v1pyGk5e9DNwC1FBRrX71tzeQJBE0AsKx1l3XicFPsQJGOJ1WeommMSZjPKRdAyMsqPbT3O/UPjHMwA6lMicCO2f/b6RYaIFhZJRZ03OzjAEpua4bFYxE1rJn8rueiKAeiHomUjrUC0YgvPJTFsUJ0IjMfIQJt0HaWTb2gClKgE8MwEQx8xWbjLDCBEyCFZORu5jIMvDOGtcN5/6i1rwpwiqjI3SMTpGLLlET3aEW8hBBHL2iN/RuvVgf1qf1NZOWrGLnEM2V9f0HLzudBQ==</latexit>

x2<latexit sha1_base64="OUe9NQ3ny9IilmHPU3ONQ4TtfRo=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3LCo4ttEPJpJk2NJkMyR2xDP0Hcavf4Urc+g1+hn9gZjoL214I9+Tcc8PJCWLONDjOj1VaW9/Y3CpvV3Z29/YPqodHj1omilCPSC5VN8CachZRDxhw2o0VxSLgtBNMbrN554kqzWT0ANOY+gKPIhYygsFQ3X4g0ufZoDmo1pyGk5e9CtwC1FBR7UH1tz+UJBE0AsKx1j3XicFPsQJGOJ1V+ommMSYTPKI9AyMsqPbT3O/MPjPM0A6lMicCO2f/b6RYaIFhbJRZ0wuzjAEpua4bFYxF1rJn8rueiqAeiHomUjrUS0YgvPJTFsUJ0IjMfYQJt0HaWTb2kClKgE8NwEQx8xWbjLHCBEyCFZORu5zIKvCajeuGc39Ra90UYZXRCTpF58hFl6iF7lAbeYggjl7RG3q3XqwP69P6mktLVrFzjBbK+v4DLZmdBA==</latexit><latexit sha1_base64="OUe9NQ3ny9IilmHPU3ONQ4TtfRo=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3LCo4ttEPJpJk2NJkMyR2xDP0Hcavf4Urc+g1+hn9gZjoL214I9+Tcc8PJCWLONDjOj1VaW9/Y3CpvV3Z29/YPqodHj1omilCPSC5VN8CachZRDxhw2o0VxSLgtBNMbrN554kqzWT0ANOY+gKPIhYygsFQ3X4g0ufZoDmo1pyGk5e9CtwC1FBR7UH1tz+UJBE0AsKx1j3XicFPsQJGOJ1V+ommMSYTPKI9AyMsqPbT3O/MPjPM0A6lMicCO2f/b6RYaIFhbJRZ0wuzjAEpua4bFYxF1rJn8rueiqAeiHomUjrUS0YgvPJTFsUJ0IjMfYQJt0HaWTb2kClKgE8NwEQx8xWbjLHCBEyCFZORu5zIKvCajeuGc39Ra90UYZXRCTpF58hFl6iF7lAbeYggjl7RG3q3XqwP69P6mktLVrFzjBbK+v4DLZmdBA==</latexit><latexit sha1_base64="OUe9NQ3ny9IilmHPU3ONQ4TtfRo=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3LCo4ttEPJpJk2NJkMyR2xDP0Hcavf4Urc+g1+hn9gZjoL214I9+Tcc8PJCWLONDjOj1VaW9/Y3CpvV3Z29/YPqodHj1omilCPSC5VN8CachZRDxhw2o0VxSLgtBNMbrN554kqzWT0ANOY+gKPIhYygsFQ3X4g0ufZoDmo1pyGk5e9CtwC1FBR7UH1tz+UJBE0AsKx1j3XicFPsQJGOJ1V+ommMSYTPKI9AyMsqPbT3O/MPjPM0A6lMicCO2f/b6RYaIFhbJRZ0wuzjAEpua4bFYxF1rJn8rueiqAeiHomUjrUS0YgvPJTFsUJ0IjMfYQJt0HaWTb2kClKgE8NwEQx8xWbjLHCBEyCFZORu5zIKvCajeuGc39Ra90UYZXRCTpF58hFl6iF7lAbeYggjl7RG3q3XqwP69P6mktLVrFzjBbK+v4DLZmdBA==</latexit>

x1<latexit sha1_base64="wPlw8y/0IqfHJrazcSQaHuwQ07k=">AAACDXicZVDLSgMxFM34rPVVdekmWAQXpcyIoO6KblxWcGyhLSWTZtrQZDIkd8Qy9B/ErX6HK3HrN/gZ/oGZ6SxseyHck3PPDScniAU34Lo/zsrq2vrGZmmrvL2zu7dfOTh8NCrRlPlUCaXbATFM8Ij5wEGwdqwZkYFgrWB8m81bT0wbrqIHmMSsJ8kw4iGnBCzV7gYyfZ72vX6l6tbdvPAy8ApQRUU1+5Xf7kDRRLIIqCDGdDw3hl5KNHAq2LTcTQyLCR2TIetYGBHJTC/N/U7xqWUGOFTanghwzv7fSIk0ksDIKrNm5mYZA0oJU7MqGMmsZc/kdzORQS2QtUykTWgWjEB41Ut5FCfAIjrzESYCg8JZNnjANaMgJhYQqrn9CqYjogkFm2DZZuQtJrIM/PP6dd29v6g2boqwSugYnaAz5KFL1EB3qIl8RJFAr+gNvTsvzofz6XzNpCtOsXOE5sr5/gMr950D</latexit><latexit sha1_base64="wPlw8y/0IqfHJrazcSQaHuwQ07k=">AAACDXicZVDLSgMxFM34rPVVdekmWAQXpcyIoO6KblxWcGyhLSWTZtrQZDIkd8Qy9B/ErX6HK3HrN/gZ/oGZ6SxseyHck3PPDScniAU34Lo/zsrq2vrGZmmrvL2zu7dfOTh8NCrRlPlUCaXbATFM8Ij5wEGwdqwZkYFgrWB8m81bT0wbrqIHmMSsJ8kw4iGnBCzV7gYyfZ72vX6l6tbdvPAy8ApQRUU1+5Xf7kDRRLIIqCDGdDw3hl5KNHAq2LTcTQyLCR2TIetYGBHJTC/N/U7xqWUGOFTanghwzv7fSIk0ksDIKrNm5mYZA0oJU7MqGMmsZc/kdzORQS2QtUykTWgWjEB41Ut5FCfAIjrzESYCg8JZNnjANaMgJhYQqrn9CqYjogkFm2DZZuQtJrIM/PP6dd29v6g2boqwSugYnaAz5KFL1EB3qIl8RJFAr+gNvTsvzofz6XzNpCtOsXOE5sr5/gMr950D</latexit><latexit sha1_base64="wPlw8y/0IqfHJrazcSQaHuwQ07k=">AAACDXicZVDLSgMxFM34rPVVdekmWAQXpcyIoO6KblxWcGyhLSWTZtrQZDIkd8Qy9B/ErX6HK3HrN/gZ/oGZ6SxseyHck3PPDScniAU34Lo/zsrq2vrGZmmrvL2zu7dfOTh8NCrRlPlUCaXbATFM8Ij5wEGwdqwZkYFgrWB8m81bT0wbrqIHmMSsJ8kw4iGnBCzV7gYyfZ72vX6l6tbdvPAy8ApQRUU1+5Xf7kDRRLIIqCDGdDw3hl5KNHAq2LTcTQyLCR2TIetYGBHJTC/N/U7xqWUGOFTanghwzv7fSIk0ksDIKrNm5mYZA0oJU7MqGMmsZc/kdzORQS2QtUykTWgWjEB41Ut5FCfAIjrzESYCg8JZNnjANaMgJhYQqrn9CqYjogkFm2DZZuQtJrIM/PP6dd29v6g2boqwSugYnaAz5KFL1EB3qIl8RJFAr+gNvTsvzofz6XzNpCtOsXOE5sr5/gMr950D</latexit>

e1<latexit sha1_base64="WgitCLnES1r4TeJXeVDWd7cTvpM=">AAACDXicZVDLSgMxFM3UV62vqks3wSK4KGVGBHVXdOOygmMLbSmZNNOGJpMhuSOUof8gbvU7XIlbv8HP8A/MTGdh2wvhnpx7bjg5QSy4Adf9cUpr6xubW+Xtys7u3v5B9fDoyahEU+ZTJZTuBMQwwSPmAwfBOrFmRAaCtYPJXTZvPzNtuIoeYRqzviSjiIecErBUpxfIlM0G3qBacxtuXngVeAWooaJag+pvb6hoIlkEVBBjup4bQz8lGjgVbFbpJYbFhE7IiHUtjIhkpp/mfmf4zDJDHCptTwQ4Z/9vpEQaSWBslVkzC7OMAaWEqVsVjGXWsmfyu5nKoB7IeibSJjRLRiC87qc8ihNgEZ37CBOBQeEsGzzkmlEQUwsI1dx+BdMx0YSCTbBiM/KWE1kF/kXjpuE+XNaat0VYZXSCTtE58tAVaqJ71EI+okigV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAMuJzw</latexit><latexit sha1_base64="WgitCLnES1r4TeJXeVDWd7cTvpM=">AAACDXicZVDLSgMxFM3UV62vqks3wSK4KGVGBHVXdOOygmMLbSmZNNOGJpMhuSOUof8gbvU7XIlbv8HP8A/MTGdh2wvhnpx7bjg5QSy4Adf9cUpr6xubW+Xtys7u3v5B9fDoyahEU+ZTJZTuBMQwwSPmAwfBOrFmRAaCtYPJXTZvPzNtuIoeYRqzviSjiIecErBUpxfIlM0G3qBacxtuXngVeAWooaJag+pvb6hoIlkEVBBjup4bQz8lGjgVbFbpJYbFhE7IiHUtjIhkpp/mfmf4zDJDHCptTwQ4Z/9vpEQaSWBslVkzC7OMAaWEqVsVjGXWsmfyu5nKoB7IeibSJjRLRiC87qc8ihNgEZ37CBOBQeEsGzzkmlEQUwsI1dx+BdMx0YSCTbBiM/KWE1kF/kXjpuE+XNaat0VYZXSCTtE58tAVaqJ71EI+okigV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAMuJzw</latexit><latexit sha1_base64="WgitCLnES1r4TeJXeVDWd7cTvpM=">AAACDXicZVDLSgMxFM3UV62vqks3wSK4KGVGBHVXdOOygmMLbSmZNNOGJpMhuSOUof8gbvU7XIlbv8HP8A/MTGdh2wvhnpx7bjg5QSy4Adf9cUpr6xubW+Xtys7u3v5B9fDoyahEU+ZTJZTuBMQwwSPmAwfBOrFmRAaCtYPJXTZvPzNtuIoeYRqzviSjiIecErBUpxfIlM0G3qBacxtuXngVeAWooaJag+pvb6hoIlkEVBBjup4bQz8lGjgVbFbpJYbFhE7IiHUtjIhkpp/mfmf4zDJDHCptTwQ4Z/9vpEQaSWBslVkzC7OMAaWEqVsVjGXWsmfyu5nKoB7IeibSJjRLRiC87qc8ihNgEZ37CBOBQeEsGzzkmlEQUwsI1dx+BdMx0YSCTbBiM/KWE1kF/kXjpuE+XNaat0VYZXSCTtE58tAVaqJ71EI+okigV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAMuJzw</latexit>

e2<latexit sha1_base64="wBcNk6euWGMrg+Da+mnoLSs6/Fg=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcq0COqu6MZlBUcL7VAyaaYNzWNI7ghl6D+IW/0OV+LWb/Az/AMz01nY9kK4J+eeG05OGHNmwPN+nNLa+sbmVnm7srO7t39QPTx6NCrRhPpEcaW7ITaUM0l9YMBpN9YUi5DTp3Bym82fnqk2TMkHmMY0EHgkWcQIBkt1+6FI6WzQGlRrXsPLy10FzQLUUFGdQfW3P1QkEVQC4diYXtOLIUixBkY4nVX6iaExJhM8oj0LJRbUBGnud+aeWWboRkrbI8HN2f8bKRZGYBhbZdbMwixjQClu6lYFY5G17Jn8bqYirIeinom0icySEYiugpTJOAEqydxHlHAXlJtl4w6ZpgT41AJMNLNfcckYa0zAJlixGTWXE1kFfqtx3fDuL2rtmyKsMjpBp+gcNdElaqM71EE+IoijV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAOWpzx</latexit><latexit sha1_base64="wBcNk6euWGMrg+Da+mnoLSs6/Fg=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcq0COqu6MZlBUcL7VAyaaYNzWNI7ghl6D+IW/0OV+LWb/Az/AMz01nY9kK4J+eeG05OGHNmwPN+nNLa+sbmVnm7srO7t39QPTx6NCrRhPpEcaW7ITaUM0l9YMBpN9YUi5DTp3Bym82fnqk2TMkHmMY0EHgkWcQIBkt1+6FI6WzQGlRrXsPLy10FzQLUUFGdQfW3P1QkEVQC4diYXtOLIUixBkY4nVX6iaExJhM8oj0LJRbUBGnud+aeWWboRkrbI8HN2f8bKRZGYBhbZdbMwixjQClu6lYFY5G17Jn8bqYirIeinom0icySEYiugpTJOAEqydxHlHAXlJtl4w6ZpgT41AJMNLNfcckYa0zAJlixGTWXE1kFfqtx3fDuL2rtmyKsMjpBp+gcNdElaqM71EE+IoijV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAOWpzx</latexit><latexit sha1_base64="wBcNk6euWGMrg+Da+mnoLSs6/Fg=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcq0COqu6MZlBUcL7VAyaaYNzWNI7ghl6D+IW/0OV+LWb/Az/AMz01nY9kK4J+eeG05OGHNmwPN+nNLa+sbmVnm7srO7t39QPTx6NCrRhPpEcaW7ITaUM0l9YMBpN9YUi5DTp3Bym82fnqk2TMkHmMY0EHgkWcQIBkt1+6FI6WzQGlRrXsPLy10FzQLUUFGdQfW3P1QkEVQC4diYXtOLIUixBkY4nVX6iaExJhM8oj0LJRbUBGnud+aeWWboRkrbI8HN2f8bKRZGYBhbZdbMwixjQClu6lYFY5G17Jn8bqYirIeinom0icySEYiugpTJOAEqydxHlHAXlJtl4w6ZpgT41AJMNLNfcckYa0zAJlixGTWXE1kFfqtx3fDuL2rtmyKsMjpBp+gcNdElaqM71EE+IoijV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAOWpzx</latexit>

e3<latexit sha1_base64="LgiizHRmhHOKEEbwqVR4tWBS3lw=">AAACDXicZVDLTgIxFL2DL8QXauLGzURi4oKQQRfqjujGJSQiJECwUzrQ0E4n7R0TMuEfjFv9DlfGrd/gxn/wD+wAC4GbNPf03HOb0+NHghv0vG8ns7K6tr6R3cxtbe/s7uX3Dx6MijVldaqE0k2fGCZ4yOrIUbBmpBmRvmANf3ibzhtPTBuuwnscRawjST/kAacELdVs+zJh4+5FN1/wSt6k3GVQnoFC5aj28wgA1W7+t91TNJYsRCqIMa2yF2EnIRo5FWyca8eGRYQOSZ+1LAyJZKaTTPyO3VPL9NxAaXtCdCfs/42ESCMJDqwybWZuljKolDBFq8KBTFv6zORuRtIv+rKYirQJzIIRDK46CQ+jGFlIpz6CWLio3DQbt8c1oyhGFhCquf2KSwdEE4o2wZzNqLyYyDKon5euS17NZnUD08rCMZzAGZThEipwB1WoAwUBL/AKb86z8+58OJ9TacaZ7RzCXDlff9YqntM=</latexit><latexit sha1_base64="RMNkU799kfoHwXHS6vi9XiRQ2tw=">AAACDXicZVDLSgMxFM3UV62vquDGzWARXJQy1YW6K3XjsgXHFtqhZNJMG5pMhuSOUIb+g7jVX3DrSlwJfoMb/8E/MDPtwrYXwj0599xwcvyIMw2O823lVlbX1jfym4Wt7Z3dveL+wb2WsSLUJZJL1faxppyF1AUGnLYjRbHwOW35o5t03nqgSjMZ3sE4op7Ag5AFjGAwVLvri4ROehe9YsmpOFnZy6A6A6XaUfOHvdY/G73ib7cvSSxoCIRjrTtVJwIvwQoY4XRS6MaaRpiM8IB2DAyxoNpLMr8T+9QwfTuQypwQ7Iz9v5FgoQWGoVGmTc/NUgak5LpsVDAUaUufye56LPyyL8qpSOlALxiB4MpLWBjFQEMy9RHE3AZpp9nYfaYoAT42ABPFzFdsMsQKEzAJFkxG1cVEloF7XrmuOE2TVR1NK4+O0Qk6Q1V0iWroFjWQiwji6Ak9oxfr0Xqz3q2PqTRnzXYO0VxZX39bgqCP</latexit><latexit sha1_base64="RMNkU799kfoHwXHS6vi9XiRQ2tw=">AAACDXicZVDLSgMxFM3UV62vquDGzWARXJQy1YW6K3XjsgXHFtqhZNJMG5pMhuSOUIb+g7jVX3DrSlwJfoMb/8E/MDPtwrYXwj0599xwcvyIMw2O823lVlbX1jfym4Wt7Z3dveL+wb2WsSLUJZJL1faxppyF1AUGnLYjRbHwOW35o5t03nqgSjMZ3sE4op7Ag5AFjGAwVLvri4ROehe9YsmpOFnZy6A6A6XaUfOHvdY/G73ib7cvSSxoCIRjrTtVJwIvwQoY4XRS6MaaRpiM8IB2DAyxoNpLMr8T+9QwfTuQypwQ7Iz9v5FgoQWGoVGmTc/NUgak5LpsVDAUaUufye56LPyyL8qpSOlALxiB4MpLWBjFQEMy9RHE3AZpp9nYfaYoAT42ABPFzFdsMsQKEzAJFkxG1cVEloF7XrmuOE2TVR1NK4+O0Qk6Q1V0iWroFjWQiwji6Ak9oxfr0Xqz3q2PqTRnzXYO0VxZX39bgqCP</latexit>

f�(e)<latexit sha1_base64="8oxgnabmEamXCM6ELWdhWTrrt8c=">AAACHHicZVA9SwNBEJ3z2/gVFSubwyhECOFio3ZBG0sFzwgmhL3NnlncvT1254Rw3G8RW+2tbazEVvBnWNi7d0lh4sAyb9+8Wd6+IBbcoOd9OVPTM7Nz8wuLpaXlldW18vrGlVGJpsynSih9HRDDBI+YjxwFu441IzIQrBXcnebz1j3ThqvoEgcx60hyG/GQU4KW6pa3wm7aDmTajvs8y6o5ZNl+t1zx6l5R7n/QGIFKc/fn5RUAzrvl73ZP0USyCKkgxtw0vBg7KdHIqWBZqZ0YFhN6R27ZjYURkcx00sJ+5u5ZpueGStsToVuwfzdSIo0k2LfKvJmxWc6gUsLUrAr7Mm/5M8XdDGRQC2QtF2kTmgkjGB51Uh7FCbKIDn2EiXBRuXlUbo9rRlEMLCBUc/sVl/aJJhRtoCWbUWMykf/AP6gf170Lm9UJDGsBtmEHqtCAQ2jCGZyDDxRSeIQneHYenDfn3fkYSqec0c4mjJXz+QtiiKWs</latexit><latexit sha1_base64="JbCPJ2fHAGZa8Ebw6DTj/vvFjWE=">AAACHHicZVDLSgMxFM34rK2PqrhyM1iFCqXMuFF3RTcuK1hbaEvJpJk2NJkMyZ1CGeZbxK3u/QFxJW4FP8OFrs20Xdj2Qrgn554bTo4XcqbBcb6speWV1bX1zEY2t7m1vZPf3bvXMlKE1ojkUjU8rClnAa0BA04boaJYeJzWvcF1Oq8PqdJMBncwCmlb4F7AfEYwGKqTP/A7ccsTcSvssyQpppAmp518wSk747IXgTsFhcrxz8vrMPdb7eS/W11JIkEDIBxr3XSdENoxVsAIp0m2FWkaYjLAPdo0MMCC6nY8tp/YJ4bp2r5U5gRgj9n/GzEWWmDoG2Xa9MwsZUBKrktGBX2RtvSZ8V2PhFfyRCkVKe3rOSPgX7RjFoQR0IBMfPgRt0HaaVR2lylKgI8MwEQx8xWb9LHCBEygWZORO5/IIqidlS/Lzq3J6gpNKoMO0REqIhedowq6QVVUQwTF6BE9oWfrwXqz3q2PiXTJmu7so5myPv8Ah3SnJg==</latexit><latexit sha1_base64="JbCPJ2fHAGZa8Ebw6DTj/vvFjWE=">AAACHHicZVDLSgMxFM34rK2PqrhyM1iFCqXMuFF3RTcuK1hbaEvJpJk2NJkMyZ1CGeZbxK3u/QFxJW4FP8OFrs20Xdj2Qrgn554bTo4XcqbBcb6speWV1bX1zEY2t7m1vZPf3bvXMlKE1ojkUjU8rClnAa0BA04boaJYeJzWvcF1Oq8PqdJMBncwCmlb4F7AfEYwGKqTP/A7ccsTcSvssyQpppAmp518wSk747IXgTsFhcrxz8vrMPdb7eS/W11JIkEDIBxr3XSdENoxVsAIp0m2FWkaYjLAPdo0MMCC6nY8tp/YJ4bp2r5U5gRgj9n/GzEWWmDoG2Xa9MwsZUBKrktGBX2RtvSZ8V2PhFfyRCkVKe3rOSPgX7RjFoQR0IBMfPgRt0HaaVR2lylKgI8MwEQx8xWb9LHCBEygWZORO5/IIqidlS/Lzq3J6gpNKoMO0REqIhedowq6QVVUQwTF6BE9oWfrwXqz3q2PiXTJmu7so5myPv8Ah3SnJg==</latexit>

ei ⇠ N (µzi ,⌃zi)<latexit sha1_base64="8CLMr+o0dZvNiJKf88rFwAP//jQ=">AAACQ3icZVBNa9tAEB0laZO4H3GaYy9LQiGlxki5JIUcQnvJKaS0bgyWEKv1yl6yqxW7o4Ir9F9Kfkzptb31P+TS0Guhu7IPiT2wzJs3b5aZl5VSWAzD38Ha+sajx5tb250nT5893+nuvvhsdWUYHzAttRlm1HIpCj5AgZIPS8OpyiS/yq7f+/7VF26s0MUnnJU8UXRSiFwwio5Ku6dxpmrepILEVigSK4pTRmV90Rz6TqyqJq2/pqLpkbb+KCaKLqjXafcg7IdtkFUQLcDB2X785hsAXKbdP/FYs0rxApmk1o6isMSkpgYFk7zpxJXlJWXXdMJHDhZUcZvU7ZUNeeWYMcm1ca9A0rL3J2qqrF/fKX2yD3qeQa2l7TkVTpVP/pu2tjOV9TLV8yJjc7u0COYnSS2KskJesPkeeSUJauIdJWNhOEM5c4AyI9wphE2poQyd7x3nUbTsyCoYHPXf9sMPzqt3MI8teAn7cAgRHMMZnMMlDIDBDfyAn/Ar+B7cBnfB37l0LVjM7MGDCP79BwcMtOc=</latexit><latexit sha1_base64="N3CijJIqFBowULdbrh+WNrhPgX8=">AAACQ3icZVBNSwMxEM36bf2qevQSFEGxlK0XFTwUvXgSRauCW5Zsmm2DyWZJZoW67L/wF/hjxKve/A9eFK+CybYHPwbCvHnzJsy8KBXcgO+/eiOjY+MTk1PTlZnZufmF6uLShVGZpqxFlVD6KiKGCZ6wFnAQ7CrVjMhIsMvo5tD1L2+ZNlwl59BPWVuSbsJjTglYKqzuB5HMWRFyHBgucSAJ9CgR+XGx4TqBzIowvwt5UcNlfca7kgypzbC65tf9MvB/0BiCteZqsHX/2uyfhNX3oKNoJlkCVBBjrht+Cu2caOBUsKISZIalhN6QLru2MCGSmXZeXlngdct0cKy0fQngkv05kRNp3PpW6ZL51XMMKCVMzaqgJ11y35S16cuoFsmaE2kTmz+LQLzbznmSZsASOtgjzgQGhZ2juMM1oyD6FhCquT0F0x7RhIL1vWI9avx15D9obdf36v6p9eoADWIKraBVtIEaaAc10RE6QS1E0QN6Qs/oxXv03rwP73MgHfGGM8voV3hf3z4qtm0=</latexit><latexit sha1_base64="N3CijJIqFBowULdbrh+WNrhPgX8=">AAACQ3icZVBNSwMxEM36bf2qevQSFEGxlK0XFTwUvXgSRauCW5Zsmm2DyWZJZoW67L/wF/hjxKve/A9eFK+CybYHPwbCvHnzJsy8KBXcgO+/eiOjY+MTk1PTlZnZufmF6uLShVGZpqxFlVD6KiKGCZ6wFnAQ7CrVjMhIsMvo5tD1L2+ZNlwl59BPWVuSbsJjTglYKqzuB5HMWRFyHBgucSAJ9CgR+XGx4TqBzIowvwt5UcNlfca7kgypzbC65tf9MvB/0BiCteZqsHX/2uyfhNX3oKNoJlkCVBBjrht+Cu2caOBUsKISZIalhN6QLru2MCGSmXZeXlngdct0cKy0fQngkv05kRNp3PpW6ZL51XMMKCVMzaqgJ11y35S16cuoFsmaE2kTmz+LQLzbznmSZsASOtgjzgQGhZ2juMM1oyD6FhCquT0F0x7RhIL1vWI9avx15D9obdf36v6p9eoADWIKraBVtIEaaAc10RE6QS1E0QN6Qs/oxXv03rwP73MgHfGGM8voV3hf3z4qtm0=</latexit>

zi ⇠<latexit sha1_base64="A0vAn9BPw9kDXHS9rkuB77rpTAE=">AAACDnicZVDLTgIxFO3gC/GFunTTSExcEDJjTNQd0Y1LTEQwMCGd0oGGdjpp75jghI8wbvU7XBm3/oKf4R/YgVkI3KS5p+ee25yeIBbcgOv+OIWV1bX1jeJmaWt7Z3evvH/wYFSiKWtSJZRuB8QwwSPWBA6CtWPNiAwEawWjm2zeemLacBXdwzhmviSDiIecErDU43OP467hEvfKFbfmTgsvAy8HFZRXo1f+7fYVTSSLgApiTMdzY/BTooFTwSalbmJYTOiIDFjHwohIZvx0aniCTyzTx6HS9kSAp+z/jZRIIwkMrTJrZm6WMaCUMFWrgqHMWvbM9G7GMqgGspqJtAnNghEIL/2UR3ECLKIzH2EiMCichYP7XDMKYmwBoZrbr2A6JJpQsBGWbEbeYiLLoHlWu6q5d+eV+nUeVhEdoWN0ijx0geroFjVQE1Ek0St6Q+/Oi/PhfDpfM2nByXcO0Vw5339MaJ0J</latexit><latexit sha1_base64="A0vAn9BPw9kDXHS9rkuB77rpTAE=">AAACDnicZVDLTgIxFO3gC/GFunTTSExcEDJjTNQd0Y1LTEQwMCGd0oGGdjpp75jghI8wbvU7XBm3/oKf4R/YgVkI3KS5p+ee25yeIBbcgOv+OIWV1bX1jeJmaWt7Z3evvH/wYFSiKWtSJZRuB8QwwSPWBA6CtWPNiAwEawWjm2zeemLacBXdwzhmviSDiIecErDU43OP467hEvfKFbfmTgsvAy8HFZRXo1f+7fYVTSSLgApiTMdzY/BTooFTwSalbmJYTOiIDFjHwohIZvx0aniCTyzTx6HS9kSAp+z/jZRIIwkMrTJrZm6WMaCUMFWrgqHMWvbM9G7GMqgGspqJtAnNghEIL/2UR3ECLKIzH2EiMCichYP7XDMKYmwBoZrbr2A6JJpQsBGWbEbeYiLLoHlWu6q5d+eV+nUeVhEdoWN0ijx0geroFjVQE1Ek0St6Q+/Oi/PhfDpfM2nByXcO0Vw5339MaJ0J</latexit><latexit sha1_base64="A0vAn9BPw9kDXHS9rkuB77rpTAE=">AAACDnicZVDLTgIxFO3gC/GFunTTSExcEDJjTNQd0Y1LTEQwMCGd0oGGdjpp75jghI8wbvU7XBm3/oKf4R/YgVkI3KS5p+ee25yeIBbcgOv+OIWV1bX1jeJmaWt7Z3evvH/wYFSiKWtSJZRuB8QwwSPWBA6CtWPNiAwEawWjm2zeemLacBXdwzhmviSDiIecErDU43OP467hEvfKFbfmTgsvAy8HFZRXo1f+7fYVTSSLgApiTMdzY/BTooFTwSalbmJYTOiIDFjHwohIZvx0aniCTyzTx6HS9kSAp+z/jZRIIwkMrTJrZm6WMaCUMFWrgqHMWvbM9G7GMqgGspqJtAnNghEIL/2UR3ECLKIzH2EiMCichYP7XDMKYmwBoZrbr2A6JJpQsBGWbEbeYiLLoHlWu6q5d+eV+nUeVhEdoWN0ijx0geroFjVQE1Ek0St6Q+/Oi/PhfDpfM2nByXcO0Vw5339MaJ0J</latexit>

Markov Structure

xi ⇠<latexit sha1_base64="4XmkaafD4EWeU6wrjePkY8qE6No=">AAACE3icZVDLTgIxFO3gC/GFunTTSExcEDIYE3VHdOMSE0dIGCSd0oGGdjpp7xjJhN8wbvU7XBm3foCf4R/YGVgI3KS5p+ee25yeIBbcgOv+OIWV1bX1jeJmaWt7Z3evvH/wYFSiKfOoEkq3A2KY4BHzgINg7VgzIgPBWsHoJpu3npg2XEX3MI5ZV5JBxENOCVjq0Q9k+jzpcewbLnGvXHFrbl54GdRnoIJm1eyVf/2+oolkEVBBjOnU3Ri6KdHAqWCTkp8YFhM6IgPWsTAikplumrue4BPL9HGotD0R4Jz9v5ESaSSBoVVmzczNMgaUEqZqVTCUWcueye9mLINqIKuZSJvQLBiB8LKb8ihOgEV06iNMBAaFs4Rwn2tGQYwtIFRz+xVMh0QTCjbHks2ovpjIMvDOalc19+680riehVVER+gYnaI6ukANdIuayEMUafSK3tC78+J8OJ/O11RacGY7h2iunO8/lwWfXA==</latexit><latexit sha1_base64="4XmkaafD4EWeU6wrjePkY8qE6No=">AAACE3icZVDLTgIxFO3gC/GFunTTSExcEDIYE3VHdOMSE0dIGCSd0oGGdjpp7xjJhN8wbvU7XBm3foCf4R/YGVgI3KS5p+ee25yeIBbcgOv+OIWV1bX1jeJmaWt7Z3evvH/wYFSiKfOoEkq3A2KY4BHzgINg7VgzIgPBWsHoJpu3npg2XEX3MI5ZV5JBxENOCVjq0Q9k+jzpcewbLnGvXHFrbl54GdRnoIJm1eyVf/2+oolkEVBBjOnU3Ri6KdHAqWCTkp8YFhM6IgPWsTAikplumrue4BPL9HGotD0R4Jz9v5ESaSSBoVVmzczNMgaUEqZqVTCUWcueye9mLINqIKuZSJvQLBiB8LKb8ihOgEV06iNMBAaFs4Rwn2tGQYwtIFRz+xVMh0QTCjbHks2ovpjIMvDOalc19+680riehVVER+gYnaI6ukANdIuayEMUafSK3tC78+J8OJ/O11RacGY7h2iunO8/lwWfXA==</latexit><latexit sha1_base64="4XmkaafD4EWeU6wrjePkY8qE6No=">AAACE3icZVDLTgIxFO3gC/GFunTTSExcEDIYE3VHdOMSE0dIGCSd0oGGdjpp7xjJhN8wbvU7XBm3foCf4R/YGVgI3KS5p+ee25yeIBbcgOv+OIWV1bX1jeJmaWt7Z3evvH/wYFSiKfOoEkq3A2KY4BHzgINg7VgzIgPBWsHoJpu3npg2XEX3MI5ZV5JBxENOCVjq0Q9k+jzpcewbLnGvXHFrbl54GdRnoIJm1eyVf/2+oolkEVBBjOnU3Ri6KdHAqWCTkp8YFhM6IgPWsTAikplumrue4BPL9HGotD0R4Jz9v5ESaSSBoVVmzczNMgaUEqZqVTCUWcueye9mLINqIKuZSJvQLBiB8LKb8ihOgEV06iNMBAaFs4Rwn2tGQYwtIFRz+xVMh0QTCjbHks2ovpjIMvDOalc19+680riehVVER+gYnaI6ukANdIuayEMUafSK3tC78+J8OJ/O11RacGY7h2iunO8/lwWfXA==</latexit>

f�(ei)<latexit sha1_base64="ISFMuiBWVv5rfvJnqq9uM8Osfvo=">AAACHnicZVBNS8NAEN34WetX1YMHL8EiVCglEUG9Fb14rGBsoS1hs920S3ezYXcilJAfI171d3gSr/oz/Adu0h5sO7DM2zdvhpkXxJxpcJwfa2V1bX1js7RV3t7Z3duvHBw+aZkoQj0iuVSdAGvKWUQ9YMBpJ1YUi4DTdjC+y+vtZ6o0k9EjTGLaF3gYsZARDIbyK8ehn/YCkfbiEcuyWg5p5rNzv1J1Gk4R9jJwZ6CKZtHyK7+9gSSJoBEQjrXuuk4M/RQrYITTrNxLNI0xGeMh7RoYYUF1Py0OyOwzwwzsUCrzIrAL9n9HioUWGEZGmSc9V8sZkJLrulHBSOQpH1P89UQE9UDUc5HSoV5YBMLrfsqiOAEakekeYcJtkHZulj1gihLgEwMwUcycYpMRVpiAsbRsPHIXHVkG3kXjpuE8XFabtzOzSugEnaIactEVaqJ71EIeIihDr+gNvVsv1of1aX1NpSvWrOcIzYX1/QfnyqO9</latexit><latexit sha1_base64="ISFMuiBWVv5rfvJnqq9uM8Osfvo=">AAACHnicZVBNS8NAEN34WetX1YMHL8EiVCglEUG9Fb14rGBsoS1hs920S3ezYXcilJAfI171d3gSr/oz/Adu0h5sO7DM2zdvhpkXxJxpcJwfa2V1bX1js7RV3t7Z3duvHBw+aZkoQj0iuVSdAGvKWUQ9YMBpJ1YUi4DTdjC+y+vtZ6o0k9EjTGLaF3gYsZARDIbyK8ehn/YCkfbiEcuyWg5p5rNzv1J1Gk4R9jJwZ6CKZtHyK7+9gSSJoBEQjrXuuk4M/RQrYITTrNxLNI0xGeMh7RoYYUF1Py0OyOwzwwzsUCrzIrAL9n9HioUWGEZGmSc9V8sZkJLrulHBSOQpH1P89UQE9UDUc5HSoV5YBMLrfsqiOAEakekeYcJtkHZulj1gihLgEwMwUcycYpMRVpiAsbRsPHIXHVkG3kXjpuE8XFabtzOzSugEnaIactEVaqJ71EIeIihDr+gNvVsv1of1aX1NpSvWrOcIzYX1/QfnyqO9</latexit><latexit sha1_base64="ISFMuiBWVv5rfvJnqq9uM8Osfvo=">AAACHnicZVBNS8NAEN34WetX1YMHL8EiVCglEUG9Fb14rGBsoS1hs920S3ezYXcilJAfI171d3gSr/oz/Adu0h5sO7DM2zdvhpkXxJxpcJwfa2V1bX1js7RV3t7Z3duvHBw+aZkoQj0iuVSdAGvKWUQ9YMBpJ1YUi4DTdjC+y+vtZ6o0k9EjTGLaF3gYsZARDIbyK8ehn/YCkfbiEcuyWg5p5rNzv1J1Gk4R9jJwZ6CKZtHyK7+9gSSJoBEQjrXuuk4M/RQrYITTrNxLNI0xGeMh7RoYYUF1Py0OyOwzwwzsUCrzIrAL9n9HioUWGEZGmSc9V8sZkJLrulHBSOQpH1P89UQE9UDUc5HSoV5YBMLrfsqiOAEakekeYcJtkHZulj1gihLgEwMwUcycYpMRVpiAsbRsPHIXHVkG3kXjpuE8XFabtzOzSugEnaIactEVaqJ71EIeIihDr+gNvVsv1of1aX1NpSvWrOcIzYX1/QfnyqO9</latexit>

Point mass at

xi = f�(ei)<latexit sha1_base64="270VJLD7gt0vH7AtNf4cVJpk6VA=">AAACKXicZVC7TisxEJ3lTXgFKGksHhJIUbShgVtcKYKGEiQCSCRaeR0vsbDXK3v26kar7fkYRAmUfAMV0PIFFPTYCQWPkaw5c+aMNXPiTAqLYfgcjIyOjU9MTk1XZmbn5heqi0snVueG8RbTUpuzmFouRcpbKFDys8xwqmLJT+PLfd8//ceNFTo9xn7GO4pepCIRjKKjoupqO1bF/zIS5C9JosJX7awnynLTQ+4aW1F1LayHgyC/QeMTrDXX328fAOAwqr61u5rliqfIJLX2vBFm2CmoQcEkLyvt3PKMskt6wc8dTKnitlMMbinJhmO6JNHGvRTJgP06UVBlFcWeU/pkv/U8g1pLW3Mq7Cmf/DeD2vZVXItVzYuMTeyPRTDZ7RQizXLkKRvukeSSoCbeN9IVhjOUfQcoM8KdQliPGsrQuVtxHjV+OvIbtLbrf+rhkfNqD4YxBSuwCpvQgB1owgEcQgsYXMEN3MF9cB08Bk/By1A6EnzOLMO3CF4/AGOkqtY=</latexit><latexit sha1_base64="PuNsspGjpcc2SeVmkDWfr5ab2W0=">AAACKXicZVBNSwMxEM36bf2qevQS/IAKpWy9qAeh6MWjgrUFW5ZsmrXBZLMks8Wy7N0fIx7Vu//Ak3r1F3jQs0nrQetAmDdv3oSZFyaCG/D9V29sfGJyanpmtjA3v7C4VFxeOTcq1ZTVqRJKN0NimOAxqwMHwZqJZkSGgjXCqyPXb/SYNlzFZ9BPWFuSy5hHnBKwVFBcb4Uyu84Djg9wFGSuaiVdnuclB5ltbAfFDb/iDwL/B9UfsFHb/Lx/6s19nQTFj1ZH0VSyGKggxlxU/QTaGdHAqWB5oZUalhB6RS7ZhYUxkcy0s8EtOd6yTAdHStsXAx6wvycyIo0k0LVKl8yfnmNAKWHKVgVd6ZL7ZlCbvgzLoSw7kTaRGVkEor12xuMkBRbT4R5RKjAo7HzDHa4ZBdG3gFDN7SmYdokmFKy7BetRddSR/6C+U9mv+KfWq0M0jBm0htZRCVXRLqqhY3SC6oiiG3SHHtCjd+s9ey/e21A65v3MrKI/4b1/A4iQrFA=</latexit><latexit sha1_base64="PuNsspGjpcc2SeVmkDWfr5ab2W0=">AAACKXicZVBNSwMxEM36bf2qevQS/IAKpWy9qAeh6MWjgrUFW5ZsmrXBZLMks8Wy7N0fIx7Vu//Ak3r1F3jQs0nrQetAmDdv3oSZFyaCG/D9V29sfGJyanpmtjA3v7C4VFxeOTcq1ZTVqRJKN0NimOAxqwMHwZqJZkSGgjXCqyPXb/SYNlzFZ9BPWFuSy5hHnBKwVFBcb4Uyu84Djg9wFGSuaiVdnuclB5ltbAfFDb/iDwL/B9UfsFHb/Lx/6s19nQTFj1ZH0VSyGKggxlxU/QTaGdHAqWB5oZUalhB6RS7ZhYUxkcy0s8EtOd6yTAdHStsXAx6wvycyIo0k0LVKl8yfnmNAKWHKVgVd6ZL7ZlCbvgzLoSw7kTaRGVkEor12xuMkBRbT4R5RKjAo7HzDHa4ZBdG3gFDN7SmYdokmFKy7BetRddSR/6C+U9mv+KfWq0M0jBm0htZRCVXRLqqhY3SC6oiiG3SHHtCjd+s9ey/e21A65v3MrKI/4b1/A4iQrFA=</latexit>

NeuralProjector

Page 18: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

A Simpler Method: Map Directly to Space of Prior

Example of Markov prior

xf(e;�)

<latexit sha1_base64="FsElmiZYiYnGj5o7UHjArdk6Bk8=">AAACUXicZVBNT9tAEB2blo9AIWmPvViNkIIURU6E1KJeUHvpkUoEEDiK1psxWbEf1u4YiCz/i17pr+LCX+HEOsmBwEirefvmzertS3MpHMXxUxCuffi4vrG51dje+bS712x9PnOmsByH3EhjL1LmUAqNQxIk8SK3yFQq8Ty9+V3Pz2/ROmH0Kc1yHCl2rUUmOCNPXWadBH8m+VQcjJvtuBfPK3oP+kvQhmWdjFvBIJkYXijUxCVz7qof5zQqmSXBJVaNpHCYM37DrvHKQ80UulE5t1xF+56ZRJmx/miK5uzrjZIppxhNvbJubmVWM2SMdF2voqmqW/3M/O5mKu2mqluLrMvcGyOU/RiVQucFoeYLH1khIzJRHU80ERY5yZkHjFvhvxLxKbOMkw9xxUSqqkYj0XjHjVJMT8rkviqTVJX3VbXK44JHz/uU+28zfQ+Gg95RL/572D7+tYx7E77CN+hAH77DMfyBExgCBw3/4AH+B4/BcwhhuJCGwXLnC6xUuP0C/9e00g==</latexit><latexit sha1_base64="FsElmiZYiYnGj5o7UHjArdk6Bk8=">AAACUXicZVBNT9tAEB2blo9AIWmPvViNkIIURU6E1KJeUHvpkUoEEDiK1psxWbEf1u4YiCz/i17pr+LCX+HEOsmBwEirefvmzertS3MpHMXxUxCuffi4vrG51dje+bS712x9PnOmsByH3EhjL1LmUAqNQxIk8SK3yFQq8Ty9+V3Pz2/ROmH0Kc1yHCl2rUUmOCNPXWadBH8m+VQcjJvtuBfPK3oP+kvQhmWdjFvBIJkYXijUxCVz7qof5zQqmSXBJVaNpHCYM37DrvHKQ80UulE5t1xF+56ZRJmx/miK5uzrjZIppxhNvbJubmVWM2SMdF2voqmqW/3M/O5mKu2mqluLrMvcGyOU/RiVQucFoeYLH1khIzJRHU80ERY5yZkHjFvhvxLxKbOMkw9xxUSqqkYj0XjHjVJMT8rkviqTVJX3VbXK44JHz/uU+28zfQ+Gg95RL/572D7+tYx7E77CN+hAH77DMfyBExgCBw3/4AH+B4/BcwhhuJCGwXLnC6xUuP0C/9e00g==</latexit><latexit sha1_base64="FsElmiZYiYnGj5o7UHjArdk6Bk8=">AAACUXicZVBNT9tAEB2blo9AIWmPvViNkIIURU6E1KJeUHvpkUoEEDiK1psxWbEf1u4YiCz/i17pr+LCX+HEOsmBwEirefvmzertS3MpHMXxUxCuffi4vrG51dje+bS712x9PnOmsByH3EhjL1LmUAqNQxIk8SK3yFQq8Ty9+V3Pz2/ROmH0Kc1yHCl2rUUmOCNPXWadBH8m+VQcjJvtuBfPK3oP+kvQhmWdjFvBIJkYXijUxCVz7qof5zQqmSXBJVaNpHCYM37DrvHKQ80UulE5t1xF+56ZRJmx/miK5uzrjZIppxhNvbJubmVWM2SMdF2voqmqW/3M/O5mKu2mqluLrMvcGyOU/RiVQucFoeYLH1khIzJRHU80ERY5yZkHjFvhvxLxKbOMkw9xxUSqqkYj0XjHjVJMT8rkviqTVJX3VbXK44JHz/uU+28zfQ+Gg95RL/572D7+tYx7E77CN+hAH77DMfyBExgCBw3/4AH+B4/BcwhhuJCGwXLnC6xUuP0C/9e00g==</latexit>

?

max log pGHMM(f�(x))<latexit sha1_base64="QrUGXd8IoThzpkp7A2ek+zngVOU=">AAACcnicZVFLaxsxEJa3r9R9OekxF7WmYIMx61BoegvpobkEUqibQGSMVp61RfRYpNl0jdjf0F/Ta/s78kNyr9bZQ50MiPnmmwczn7JCSY9petNJHj1+8vTZzvPui5evXr/p7e798LZ0AqbCKusuMu5BSQNTlKjgonDAdabgPLv60uTPr8F5ac13XBcw03xpZC4Fx0jNe0OmeUUZZcouaTEPDKHC8PXk9LSuB3mMi5WsB6waDue9fjpON0YfgkkL+qS1s/lu54AtrCg1GBSKe385SQucBe5QCgV1l5UeCi6u+BIuIzRcg5+FzU01/RCZBc2ti88g3bD/dwSuvea4ipWN81u5hkFrlR/FKlzpxjVjNrFf62yU6VFT5Hzu7y2C+eEsSFOUCEbc7ZGXiqKljX50IR0IVOsIuHAynkLFijsuMKq8tUSm626XGfgprNbcLAKr6sAyHaq63ubhjofIR5Un9zV9CKYH48/j9NvH/tFxK/cO2SfvyYBMyCdyRE7IGZkSQX6R3+QP+du5TfaTd0n7N0mn7XlLtiwZ/QPP2MC/</latexit><latexit sha1_base64="QrUGXd8IoThzpkp7A2ek+zngVOU=">AAACcnicZVFLaxsxEJa3r9R9OekxF7WmYIMx61BoegvpobkEUqibQGSMVp61RfRYpNl0jdjf0F/Ta/s78kNyr9bZQ50MiPnmmwczn7JCSY9petNJHj1+8vTZzvPui5evXr/p7e798LZ0AqbCKusuMu5BSQNTlKjgonDAdabgPLv60uTPr8F5ac13XBcw03xpZC4Fx0jNe0OmeUUZZcouaTEPDKHC8PXk9LSuB3mMi5WsB6waDue9fjpON0YfgkkL+qS1s/lu54AtrCg1GBSKe385SQucBe5QCgV1l5UeCi6u+BIuIzRcg5+FzU01/RCZBc2ti88g3bD/dwSuvea4ipWN81u5hkFrlR/FKlzpxjVjNrFf62yU6VFT5Hzu7y2C+eEsSFOUCEbc7ZGXiqKljX50IR0IVOsIuHAynkLFijsuMKq8tUSm626XGfgprNbcLAKr6sAyHaq63ubhjofIR5Un9zV9CKYH48/j9NvH/tFxK/cO2SfvyYBMyCdyRE7IGZkSQX6R3+QP+du5TfaTd0n7N0mn7XlLtiwZ/QPP2MC/</latexit><latexit sha1_base64="QrUGXd8IoThzpkp7A2ek+zngVOU=">AAACcnicZVFLaxsxEJa3r9R9OekxF7WmYIMx61BoegvpobkEUqibQGSMVp61RfRYpNl0jdjf0F/Ta/s78kNyr9bZQ50MiPnmmwczn7JCSY9petNJHj1+8vTZzvPui5evXr/p7e798LZ0AqbCKusuMu5BSQNTlKjgonDAdabgPLv60uTPr8F5ac13XBcw03xpZC4Fx0jNe0OmeUUZZcouaTEPDKHC8PXk9LSuB3mMi5WsB6waDue9fjpON0YfgkkL+qS1s/lu54AtrCg1GBSKe385SQucBe5QCgV1l5UeCi6u+BIuIzRcg5+FzU01/RCZBc2ti88g3bD/dwSuvea4ipWN81u5hkFrlR/FKlzpxjVjNrFf62yU6VFT5Hzu7y2C+eEsSFOUCEbc7ZGXiqKljX50IR0IVOsIuHAynkLFijsuMKq8tUSm626XGfgprNbcLAKr6sAyHaq63ubhjofIR5Un9zV9CKYH48/j9NvH/tFxK/cO2SfvyYBMyCdyRE7IGZkSQX6R3+QP+du5TfaTd0n7N0mn7XlLtiwZ/QPP2MC/</latexit>

Information Loss

Page 19: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Normalizing Flow (Rezende and Mohamed 2015)

• Basic idea: a way to map from one probability distribution to another in an invertible manner

ei ⇠ N (µzi ,⌃zi)<latexit sha1_base64="8CLMr+o0dZvNiJKf88rFwAP//jQ=">AAACQ3icZVBNa9tAEB0laZO4H3GaYy9LQiGlxki5JIUcQnvJKaS0bgyWEKv1yl6yqxW7o4Ir9F9Kfkzptb31P+TS0Guhu7IPiT2wzJs3b5aZl5VSWAzD38Ha+sajx5tb250nT5893+nuvvhsdWUYHzAttRlm1HIpCj5AgZIPS8OpyiS/yq7f+/7VF26s0MUnnJU8UXRSiFwwio5Ku6dxpmrepILEVigSK4pTRmV90Rz6TqyqJq2/pqLpkbb+KCaKLqjXafcg7IdtkFUQLcDB2X785hsAXKbdP/FYs0rxApmk1o6isMSkpgYFk7zpxJXlJWXXdMJHDhZUcZvU7ZUNeeWYMcm1ca9A0rL3J2qqrF/fKX2yD3qeQa2l7TkVTpVP/pu2tjOV9TLV8yJjc7u0COYnSS2KskJesPkeeSUJauIdJWNhOEM5c4AyI9wphE2poQyd7x3nUbTsyCoYHPXf9sMPzqt3MI8teAn7cAgRHMMZnMMlDIDBDfyAn/Ar+B7cBnfB37l0LVjM7MGDCP79BwcMtOc=</latexit><latexit sha1_base64="N3CijJIqFBowULdbrh+WNrhPgX8=">AAACQ3icZVBNSwMxEM36bf2qevQSFEGxlK0XFTwUvXgSRauCW5Zsmm2DyWZJZoW67L/wF/hjxKve/A9eFK+CybYHPwbCvHnzJsy8KBXcgO+/eiOjY+MTk1PTlZnZufmF6uLShVGZpqxFlVD6KiKGCZ6wFnAQ7CrVjMhIsMvo5tD1L2+ZNlwl59BPWVuSbsJjTglYKqzuB5HMWRFyHBgucSAJ9CgR+XGx4TqBzIowvwt5UcNlfca7kgypzbC65tf9MvB/0BiCteZqsHX/2uyfhNX3oKNoJlkCVBBjrht+Cu2caOBUsKISZIalhN6QLru2MCGSmXZeXlngdct0cKy0fQngkv05kRNp3PpW6ZL51XMMKCVMzaqgJ11y35S16cuoFsmaE2kTmz+LQLzbznmSZsASOtgjzgQGhZ2juMM1oyD6FhCquT0F0x7RhIL1vWI9avx15D9obdf36v6p9eoADWIKraBVtIEaaAc10RE6QS1E0QN6Qs/oxXv03rwP73MgHfGGM8voV3hf3z4qtm0=</latexit><latexit sha1_base64="N3CijJIqFBowULdbrh+WNrhPgX8=">AAACQ3icZVBNSwMxEM36bf2qevQSFEGxlK0XFTwUvXgSRauCW5Zsmm2DyWZJZoW67L/wF/hjxKve/A9eFK+CybYHPwbCvHnzJsy8KBXcgO+/eiOjY+MTk1PTlZnZufmF6uLShVGZpqxFlVD6KiKGCZ6wFnAQ7CrVjMhIsMvo5tD1L2+ZNlwl59BPWVuSbsJjTglYKqzuB5HMWRFyHBgucSAJ9CgR+XGx4TqBzIowvwt5UcNlfca7kgypzbC65tf9MvB/0BiCteZqsHX/2uyfhNX3oKNoJlkCVBBjrht+Cu2caOBUsKISZIalhN6QLru2MCGSmXZeXlngdct0cKy0fQngkv05kRNp3PpW6ZL51XMMKCVMzaqgJ11y35S16cuoFsmaE2kTmz+LQLzbznmSZsASOtgjzgQGhZ2juMM1oyD6FhCquT0F0x7RhIL1vWI9avx15D9obdf36v6p9eoADWIKraBVtIEaaAc10RE6QS1E0QN6Qs/oxXv03rwP73MgHfGGM8voV3hf3z4qtm0=</latexit>

xi = f�(ei)<latexit sha1_base64="270VJLD7gt0vH7AtNf4cVJpk6VA=">AAACKXicZVC7TisxEJ3lTXgFKGksHhJIUbShgVtcKYKGEiQCSCRaeR0vsbDXK3v26kar7fkYRAmUfAMV0PIFFPTYCQWPkaw5c+aMNXPiTAqLYfgcjIyOjU9MTk1XZmbn5heqi0snVueG8RbTUpuzmFouRcpbKFDys8xwqmLJT+PLfd8//ceNFTo9xn7GO4pepCIRjKKjoupqO1bF/zIS5C9JosJX7awnynLTQ+4aW1F1LayHgyC/QeMTrDXX328fAOAwqr61u5rliqfIJLX2vBFm2CmoQcEkLyvt3PKMskt6wc8dTKnitlMMbinJhmO6JNHGvRTJgP06UVBlFcWeU/pkv/U8g1pLW3Mq7Cmf/DeD2vZVXItVzYuMTeyPRTDZ7RQizXLkKRvukeSSoCbeN9IVhjOUfQcoM8KdQliPGsrQuVtxHjV+OvIbtLbrf+rhkfNqD4YxBSuwCpvQgB1owgEcQgsYXMEN3MF9cB08Bk/By1A6EnzOLMO3CF4/AGOkqtY=</latexit><latexit sha1_base64="PuNsspGjpcc2SeVmkDWfr5ab2W0=">AAACKXicZVBNSwMxEM36bf2qevQS/IAKpWy9qAeh6MWjgrUFW5ZsmrXBZLMks8Wy7N0fIx7Vu//Ak3r1F3jQs0nrQetAmDdv3oSZFyaCG/D9V29sfGJyanpmtjA3v7C4VFxeOTcq1ZTVqRJKN0NimOAxqwMHwZqJZkSGgjXCqyPXb/SYNlzFZ9BPWFuSy5hHnBKwVFBcb4Uyu84Djg9wFGSuaiVdnuclB5ltbAfFDb/iDwL/B9UfsFHb/Lx/6s19nQTFj1ZH0VSyGKggxlxU/QTaGdHAqWB5oZUalhB6RS7ZhYUxkcy0s8EtOd6yTAdHStsXAx6wvycyIo0k0LVKl8yfnmNAKWHKVgVd6ZL7ZlCbvgzLoSw7kTaRGVkEor12xuMkBRbT4R5RKjAo7HzDHa4ZBdG3gFDN7SmYdokmFKy7BetRddSR/6C+U9mv+KfWq0M0jBm0htZRCVXRLqqhY3SC6oiiG3SHHtCjd+s9ey/e21A65v3MrKI/4b1/A4iQrFA=</latexit><latexit sha1_base64="PuNsspGjpcc2SeVmkDWfr5ab2W0=">AAACKXicZVBNSwMxEM36bf2qevQS/IAKpWy9qAeh6MWjgrUFW5ZsmrXBZLMks8Wy7N0fIx7Vu//Ak3r1F3jQs0nrQetAmDdv3oSZFyaCG/D9V29sfGJyanpmtjA3v7C4VFxeOTcq1ZTVqRJKN0NimOAxqwMHwZqJZkSGgjXCqyPXb/SYNlzFZ9BPWFuSy5hHnBKwVFBcb4Uyu84Djg9wFGSuaiVdnuclB5ltbAfFDb/iDwL/B9UfsFHb/Lx/6s19nQTFj1ZH0VSyGKggxlxU/QTaGdHAqWB5oZUalhB6RS7ZhYUxkcy0s8EtOd6yTAdHStsXAx6wvycyIo0k0LVKl8yfnmNAKWHKVgVd6ZL7ZlCbvgzLoSw7kTaRGVkEor12xuMkBRbT4R5RKjAo7HzDHa4ZBdG3gFDN7SmYdokmFKy7BetRddSR/6C+U9mv+KfWq0M0jBm0htZRCVXRLqqhY3SC6oiiG3SHHtCjd+s9ey/e21A65v3MrKI/4b1/A4iQrFA=</latexit>

NeuralProjector

xf�1(e;�)

<latexit sha1_base64="0TvUX8MzRC5izm1cmvW9PX4J8iQ=">AAACWXicZVBNSyNBEO2M7hrHdY169NIYBBdimJGFVbyIXjwqGBWcbOjp1JjG/hi6a9QwzD/xqv9J8MfYk+RgtKCp169eNa9fmkvhMIreGsHC4o+fS83lcOXX6u+11vrGlTOF5dDjRhp7kzIHUmjooUAJN7kFplIJ1+n9aT2/fgDrhNGXOM6hr9idFpngDD01aLWy/+VeXO0mcESTfCT+DFrtqBtNin4H8Qy0yazOB+uN/WRoeKFAI5fMuds4yrFfMouCS6jCpHCQM37P7uDWQ80UuH45sV7RHc8MaWasPxrphP28UTLlFMORV9bNzc1qBo2RruNVOFJ1q5+Z3N1YpZ1UdWqRdZn7YgSzg34pdF4gaD71kRWSoqF1THQoLHCUYw8Yt8J/hfIRs4yjD3PORKqqMEw0PHKjFNPDMnmqyiRV5VNVzfMw5cHzPuX4a6bfQW+/e9iNLv62j09mcTfJFtkmuyQm/8gxOSPnpEc4eSDP5IW8Nt6DIGgG4VQaNGY7m2Sugs0PzFK2FA==</latexit><latexit sha1_base64="0TvUX8MzRC5izm1cmvW9PX4J8iQ=">AAACWXicZVBNSyNBEO2M7hrHdY169NIYBBdimJGFVbyIXjwqGBWcbOjp1JjG/hi6a9QwzD/xqv9J8MfYk+RgtKCp169eNa9fmkvhMIreGsHC4o+fS83lcOXX6u+11vrGlTOF5dDjRhp7kzIHUmjooUAJN7kFplIJ1+n9aT2/fgDrhNGXOM6hr9idFpngDD01aLWy/+VeXO0mcESTfCT+DFrtqBtNin4H8Qy0yazOB+uN/WRoeKFAI5fMuds4yrFfMouCS6jCpHCQM37P7uDWQ80UuH45sV7RHc8MaWasPxrphP28UTLlFMORV9bNzc1qBo2RruNVOFJ1q5+Z3N1YpZ1UdWqRdZn7YgSzg34pdF4gaD71kRWSoqF1THQoLHCUYw8Yt8J/hfIRs4yjD3PORKqqMEw0PHKjFNPDMnmqyiRV5VNVzfMw5cHzPuX4a6bfQW+/e9iNLv62j09mcTfJFtkmuyQm/8gxOSPnpEc4eSDP5IW8Nt6DIGgG4VQaNGY7m2Sugs0PzFK2FA==</latexit><latexit sha1_base64="0TvUX8MzRC5izm1cmvW9PX4J8iQ=">AAACWXicZVBNSyNBEO2M7hrHdY169NIYBBdimJGFVbyIXjwqGBWcbOjp1JjG/hi6a9QwzD/xqv9J8MfYk+RgtKCp169eNa9fmkvhMIreGsHC4o+fS83lcOXX6u+11vrGlTOF5dDjRhp7kzIHUmjooUAJN7kFplIJ1+n9aT2/fgDrhNGXOM6hr9idFpngDD01aLWy/+VeXO0mcESTfCT+DFrtqBtNin4H8Qy0yazOB+uN/WRoeKFAI5fMuds4yrFfMouCS6jCpHCQM37P7uDWQ80UuH45sV7RHc8MaWasPxrphP28UTLlFMORV9bNzc1qBo2RruNVOFJ1q5+Z3N1YpZ1UdWqRdZn7YgSzg34pdF4gaD71kRWSoqF1THQoLHCUYw8Yt8J/hfIRs4yjD3PORKqqMEw0PHKjFNPDMnmqyiRV5VNVzfMw5cHzPuX4a6bfQW+/e9iNLv62j09mcTfJFtkmuyQm/8gxOSPnpEc4eSDP5IW8Nt6DIGgG4VQaNGY7m2Sugs0PzFK2FA==</latexit>

• Advantage: optimizing probability can be shown to be possible by following equation

�1<latexit sha1_base64="whzOwvpgac/k7KeUY3/DRCrw/7o=">AAACTnicdVBNbxMxEPUGSkMotIUjF4sIiUOIdrdR096qcuHYSg2p1I0qrzPbWPHHyp4tjVb7I7jCr+qRX8IJtXYSJILESNY8v3ljPb+8lMJhHP+MWk+ebj3bbj/vvNh5+Wp3b//1F2cqy2HEjTT2MmcOpNAwQoESLksLTOUSxvn8U5iPb8E6YfQFLkqYKHajRSE4Q0+NP2ZCF7i43uvG/eOjw3RwSON+HA+TNAkgHQ4OBjTxTKguWdfZ9X6UZlPDKwUauWTOXSVxiZOaWRRcQtPJKgcl43N2A1ceaqbATeql34a+98yUFsb6o5Eu2b83aqacYjjzytDcxiwwaIx0Pa/CmQotPLO8u4XKe7nqBZF1hfvHCBZHk1roskLQfOWjqCRFQ0M2dCoscJQLDxi3wn+F8hmzjKNPcMNErppOJ9PwlRulmJ7W2V1TZ7mq75pmk4cVD573Kf+Jkv4fjNL+cT8+H3RPTtdxt8lb8o58IAkZkhPymZyREeFkTr6R7+RHdB/9in5HDytpK1rvvCEb1Wo/AhvGtlc=</latexit><latexit sha1_base64="whzOwvpgac/k7KeUY3/DRCrw/7o=">AAACTnicdVBNbxMxEPUGSkMotIUjF4sIiUOIdrdR096qcuHYSg2p1I0qrzPbWPHHyp4tjVb7I7jCr+qRX8IJtXYSJILESNY8v3ljPb+8lMJhHP+MWk+ebj3bbj/vvNh5+Wp3b//1F2cqy2HEjTT2MmcOpNAwQoESLksLTOUSxvn8U5iPb8E6YfQFLkqYKHajRSE4Q0+NP2ZCF7i43uvG/eOjw3RwSON+HA+TNAkgHQ4OBjTxTKguWdfZ9X6UZlPDKwUauWTOXSVxiZOaWRRcQtPJKgcl43N2A1ceaqbATeql34a+98yUFsb6o5Eu2b83aqacYjjzytDcxiwwaIx0Pa/CmQotPLO8u4XKe7nqBZF1hfvHCBZHk1roskLQfOWjqCRFQ0M2dCoscJQLDxi3wn+F8hmzjKNPcMNErppOJ9PwlRulmJ7W2V1TZ7mq75pmk4cVD573Kf+Jkv4fjNL+cT8+H3RPTtdxt8lb8o58IAkZkhPymZyREeFkTr6R7+RHdB/9in5HDytpK1rvvCEb1Wo/AhvGtlc=</latexit><latexit sha1_base64="whzOwvpgac/k7KeUY3/DRCrw/7o=">AAACTnicdVBNbxMxEPUGSkMotIUjF4sIiUOIdrdR096qcuHYSg2p1I0qrzPbWPHHyp4tjVb7I7jCr+qRX8IJtXYSJILESNY8v3ljPb+8lMJhHP+MWk+ebj3bbj/vvNh5+Wp3b//1F2cqy2HEjTT2MmcOpNAwQoESLksLTOUSxvn8U5iPb8E6YfQFLkqYKHajRSE4Q0+NP2ZCF7i43uvG/eOjw3RwSON+HA+TNAkgHQ4OBjTxTKguWdfZ9X6UZlPDKwUauWTOXSVxiZOaWRRcQtPJKgcl43N2A1ceaqbATeql34a+98yUFsb6o5Eu2b83aqacYjjzytDcxiwwaIx0Pa/CmQotPLO8u4XKe7nqBZF1hfvHCBZHk1roskLQfOWjqCRFQ0M2dCoscJQLDxi3wn+F8hmzjKNPcMNErppOJ9PwlRulmJ7W2V1TZ7mq75pmk4cVD573Kf+Jkv4fjNL+cT8+H3RPTtdxt8lb8o58IAkZkhPymZyREeFkTr6R7+RHdB/9in5HDytpK1rvvCEb1Wo/AhvGtlc=</latexit>

+X

log���det

@f�1�

@xi

���<latexit sha1_base64="SkZYIwUJm36ifl+bt7GkjNGFpcg=">AAAClHicdZFbixMxFMfTWS9rvXX1wQdfgkUQrGVmLNvdB2HZRRAEWcG6C5s6ZDKZNmwuQ3JmbRnzofw04pt+EzNtRSt4IOTkdy6c809eSeEgjr93op1r12/c3L3VvX3n7r37vb0HH52pLeMTZqSx5zl1XArNJyBA8vPKcqpyyc/yy5M2fnbFrRNGf4BlxaeKzrQoBaMQUNZ7i59j4mpFpJmRYzH7QoAvoCk4eFJayhpSUQuCSlx+al4kPgtgLrz/w8kiE35VmvX68fDwYD8d7eN4GMfjJE1aJx2PXo5wEkhrfbSx02yvk5LCsFpxDUxS5y6SuIJp03ZmkvsuqR2vKLukM34RXE0Vd9NmtbXHTwMpcGlsOBrwiv5d0VDlFIV5yGwvtxVrCRgj3SBkwVy1V9tm9XZLlQ9yNWiTrCvdP4NAeTBthK5q4Jqt5yhricHgVmFcCMsZyGVwKLMirILZnAYtIfzD1hC58t0u0fwzM0pRXTRkEYTNVbPwfpvzNeeBB5V/S4n/70zS4eEwfj/qHx1v5N5Fj9ET9AwlaIyO0Bt0iiaIoa/oG/qBfkaPolfRSfR6nRp1NjUP0ZZF734BeF7PyQ==</latexit><latexit sha1_base64="SkZYIwUJm36ifl+bt7GkjNGFpcg=">AAAClHicdZFbixMxFMfTWS9rvXX1wQdfgkUQrGVmLNvdB2HZRRAEWcG6C5s6ZDKZNmwuQ3JmbRnzofw04pt+EzNtRSt4IOTkdy6c809eSeEgjr93op1r12/c3L3VvX3n7r37vb0HH52pLeMTZqSx5zl1XArNJyBA8vPKcqpyyc/yy5M2fnbFrRNGf4BlxaeKzrQoBaMQUNZ7i59j4mpFpJmRYzH7QoAvoCk4eFJayhpSUQuCSlx+al4kPgtgLrz/w8kiE35VmvX68fDwYD8d7eN4GMfjJE1aJx2PXo5wEkhrfbSx02yvk5LCsFpxDUxS5y6SuIJp03ZmkvsuqR2vKLukM34RXE0Vd9NmtbXHTwMpcGlsOBrwiv5d0VDlFIV5yGwvtxVrCRgj3SBkwVy1V9tm9XZLlQ9yNWiTrCvdP4NAeTBthK5q4Jqt5yhricHgVmFcCMsZyGVwKLMirILZnAYtIfzD1hC58t0u0fwzM0pRXTRkEYTNVbPwfpvzNeeBB5V/S4n/70zS4eEwfj/qHx1v5N5Fj9ET9AwlaIyO0Bt0iiaIoa/oG/qBfkaPolfRSfR6nRp1NjUP0ZZF734BeF7PyQ==</latexit><latexit sha1_base64="SkZYIwUJm36ifl+bt7GkjNGFpcg=">AAAClHicdZFbixMxFMfTWS9rvXX1wQdfgkUQrGVmLNvdB2HZRRAEWcG6C5s6ZDKZNmwuQ3JmbRnzofw04pt+EzNtRSt4IOTkdy6c809eSeEgjr93op1r12/c3L3VvX3n7r37vb0HH52pLeMTZqSx5zl1XArNJyBA8vPKcqpyyc/yy5M2fnbFrRNGf4BlxaeKzrQoBaMQUNZ7i59j4mpFpJmRYzH7QoAvoCk4eFJayhpSUQuCSlx+al4kPgtgLrz/w8kiE35VmvX68fDwYD8d7eN4GMfjJE1aJx2PXo5wEkhrfbSx02yvk5LCsFpxDUxS5y6SuIJp03ZmkvsuqR2vKLukM34RXE0Vd9NmtbXHTwMpcGlsOBrwiv5d0VDlFIV5yGwvtxVrCRgj3SBkwVy1V9tm9XZLlQ9yNWiTrCvdP4NAeTBthK5q4Jqt5yhricHgVmFcCMsZyGVwKLMirILZnAYtIfzD1hC58t0u0fwzM0pRXTRkEYTNVbPwfpvzNeeBB5V/S4n/70zS4eEwfj/qHx1v5N5Fj9ET9AwlaIyO0Bt0iiaIoa/oG/qBfkaPolfRSfR6nRp1NjUP0ZZF734BeF7PyQ==</latexit>

when is not invertiblef<latexit sha1_base64="mEEGp2ktLOyduAzcmi2vdTnECNI=">AAACSHicdVBNbxMxEPUGWkqgpaVHLhYREoco2l2iprlVcOGYSIRE6kaV15ltrPpjZc9CotX+Aq7wq/gF/AxOFTfsJEgEiZGseX7zxnp+eSmFwzj+EbUePDw4fHT0uP3k6fHJs9Oz5x+dqSyHCTfS2FnOHEihYYICJcxKC0zlEqb53bswn34C64TRH3BdwlyxWy0KwRl6alzcnHbi3vDyIu1f0LgXx4MkTQJIB/03fZp4JlSH7Gp0cxal2cLwSoFGLplz10lc4rxmFgWX0LSzykHJ+B27hWsPNVPg5vXGaUNfeWZBC2P90Ug37N8bNVNOMVx6ZWhubxYYNEa6rlfhUoUWntnc3Vrl3Vx1g8i6wv1jBIvLeS10WSFovvVRVJKioSEVuhAWOMq1B4xb4b9C+ZJZxtFnt2ciV027nWn4zI1STC/qbNXUWa7qVdPs87DlwfM+5T9R0v+DSdob9uJxv3P1dhf3EXlBXpLXJCEDckXekxGZEE6AfCFfybfoe/Qzuo9+baWtaLdzTvaq1foN2xqzzg==</latexit><latexit sha1_base64="mEEGp2ktLOyduAzcmi2vdTnECNI=">AAACSHicdVBNbxMxEPUGWkqgpaVHLhYREoco2l2iprlVcOGYSIRE6kaV15ltrPpjZc9CotX+Aq7wq/gF/AxOFTfsJEgEiZGseX7zxnp+eSmFwzj+EbUePDw4fHT0uP3k6fHJs9Oz5x+dqSyHCTfS2FnOHEihYYICJcxKC0zlEqb53bswn34C64TRH3BdwlyxWy0KwRl6alzcnHbi3vDyIu1f0LgXx4MkTQJIB/03fZp4JlSH7Gp0cxal2cLwSoFGLplz10lc4rxmFgWX0LSzykHJ+B27hWsPNVPg5vXGaUNfeWZBC2P90Ug37N8bNVNOMVx6ZWhubxYYNEa6rlfhUoUWntnc3Vrl3Vx1g8i6wv1jBIvLeS10WSFovvVRVJKioSEVuhAWOMq1B4xb4b9C+ZJZxtFnt2ciV027nWn4zI1STC/qbNXUWa7qVdPs87DlwfM+5T9R0v+DSdob9uJxv3P1dhf3EXlBXpLXJCEDckXekxGZEE6AfCFfybfoe/Qzuo9+baWtaLdzTvaq1foN2xqzzg==</latexit><latexit sha1_base64="mEEGp2ktLOyduAzcmi2vdTnECNI=">AAACSHicdVBNbxMxEPUGWkqgpaVHLhYREoco2l2iprlVcOGYSIRE6kaV15ltrPpjZc9CotX+Aq7wq/gF/AxOFTfsJEgEiZGseX7zxnp+eSmFwzj+EbUePDw4fHT0uP3k6fHJs9Oz5x+dqSyHCTfS2FnOHEihYYICJcxKC0zlEqb53bswn34C64TRH3BdwlyxWy0KwRl6alzcnHbi3vDyIu1f0LgXx4MkTQJIB/03fZp4JlSH7Gp0cxal2cLwSoFGLplz10lc4rxmFgWX0LSzykHJ+B27hWsPNVPg5vXGaUNfeWZBC2P90Ug37N8bNVNOMVx6ZWhubxYYNEa6rlfhUoUWntnc3Vrl3Vx1g8i6wv1jBIvLeS10WSFovvVRVJKioSEVuhAWOMq1B4xb4b9C+ZJZxtFnt2ciV027nWn4zI1STC/qbNXUWa7qVdPs87DlwfM+5T9R0v+DSdob9uJxv3P1dhf3EXlBXpLXJCEDckXekxGZEE6AfCFfybfoe/Qzuo9+baWtaLdzTvaq1foN2xqzzg==</latexit>

log p(x) = log pGHMM(f�1

� (x))<latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit><latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit><latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit><latexit sha1_base64="jBAzhNAJ26J1i9dTzZ4SO1PLZqY=">AAACOXicZVHLTsMwEHTKq4Ty6JlLBELiUFVpL8ANiQtHkAhUIhVynA216kdkb6BVlB/gylfxCXwFJ8QVp+2BlpWsHc/uWuPZJBfcYhh+eo219Y3Nrea2v9Pyd/f2D1r3VheGQcS00GaQUAuCK4iQo4BBboDKRMBDMr6q6w8vYCzX6g6nOQwlfVY844yio26eDo7DbjiL4D/oLcAxWcRT2+vHqWaFBIVMUGsfe2GOw5Ia5ExA5ceFhZyyMX2GRwcVlWCH5UxnFZw4Jg0ybdxRGMzYvxMllVZSHLnOOtmlWs2g1sJ2XBeOZJ3qZ2Z3O5VJJ5GdusnYzK4Iwex8WHKVFwiKzXVkhQhQB7UnQcoNMBRTBygz3H0lYCNqKEPn3JKIRFa+Hyt4ZVpKqtIynlRlnMhyUlXLPMx5cLwzubdq6X8Q9bsX3fA2JE1ySI7IKemRM3JJrskNiQgjKXkj796H9+V9z3fR8BZLaZOl8H5+AY4YsXk=</latexit><latexit sha1_base64="Ku8E5xMhx2yjUIS3NBNvbWk1Bus=">AAACcnicZVFNbxMxEJ0sXyUUmnLlYlGBUilEmwoJOCAhcaCXSkUitFI3RF5nNrHqj5U9C4ks/xF+DVf4B/wbvEkOpB3J8ps3z9bMm7JW0lOe/+1kd+7eu/9g72H30f7jJwe9w/2v3jZO4FhYZd1lyT0qaXBMkhRe1g65LhVelNcf2/rFd3ReWvOFVjVONJ8bWUnBKVHT3utC2Tmr+8XymL1nm2QaCsIlhU+nZ2cx9quU1wsZv4VXo9gKj6e9o3yYr4PdBqMtOIJtnE8POyfFzIpGoyGhuPdXo7ymSeCOpFAYu0Xjsebims/xKkHDNfpJWI8X2YvEzFhlXTqG2Jr9/0Xg2mtOi6RsL79TaxmyVvlBUtFCt1f7zTr3K10OSj1oRc5X/kYjVL2dBGnqhtCITR9VoxhZ1lrJZtKhILVKgAsn0yhMLLjjgpLhO02UOna7hcEfwmrNzSwUyxiKUodljLs8bnhMfHJ5dNPT22B8Mnw3zD/nsAfP4Dn0YQRv4AOcwjmMQcBP+AW/4U8G2ctssFlH1tnu5SnsRDb8B6n/wos=</latexit><latexit sha1_base64="Ku8E5xMhx2yjUIS3NBNvbWk1Bus=">AAACcnicZVFNbxMxEJ0sXyUUmnLlYlGBUilEmwoJOCAhcaCXSkUitFI3RF5nNrHqj5U9C4ks/xF+DVf4B/wbvEkOpB3J8ps3z9bMm7JW0lOe/+1kd+7eu/9g72H30f7jJwe9w/2v3jZO4FhYZd1lyT0qaXBMkhRe1g65LhVelNcf2/rFd3ReWvOFVjVONJ8bWUnBKVHT3utC2Tmr+8XymL1nm2QaCsIlhU+nZ2cx9quU1wsZv4VXo9gKj6e9o3yYr4PdBqMtOIJtnE8POyfFzIpGoyGhuPdXo7ymSeCOpFAYu0Xjsebims/xKkHDNfpJWI8X2YvEzFhlXTqG2Jr9/0Xg2mtOi6RsL79TaxmyVvlBUtFCt1f7zTr3K10OSj1oRc5X/kYjVL2dBGnqhtCITR9VoxhZ1lrJZtKhILVKgAsn0yhMLLjjgpLhO02UOna7hcEfwmrNzSwUyxiKUodljLs8bnhMfHJ5dNPT22B8Mnw3zD/nsAfP4Dn0YQRv4AOcwjmMQcBP+AW/4U8G2ctssFlH1tnu5SnsRDb8B6n/wos=</latexit><latexit sha1_base64="xChZ9DUVMtsn1KFXp2HSXCyWRic=">AAACfXicZVDbahsxEJW3t9S9Oc1jX0RNiwOu2Q2FtA+B0Dw0L4EU4iSQdY1WnrVFdFmk2cZG6Ef6NXlN/6B/U63thzoZEHPmzBkxc4pKCodp+reVPHr85OmzreftFy9fvX7T2X577kxtOQy5kcZeFsyBFBqGKFDCZWWBqULCRXF91PQvfoF1wugzXFQwUmyqRSk4w0iNO59zaaa06uXzXXpAV8XY5whz9N+PT05C6JWxrmYi/PSfstAId8edbjpIl0EfgmwNumQdp+Pt1l4+MbxWoJFL5txVllY48syi4BJCO68dVIxfsylcRaiZAjfyy/MC/RCZCS2NjU8jXbL/T3imnGI4i8omuY1ew6Ax0vWjCmeqSc03y9otVNEvVL8RWVe6e4tg+WXkha5qBM1Xe5S1pGhoYyWdCAsc5SICxq2Ip1A+Y5ZxjIZvLFGo0G7nGm64UYrpic/nweeF8vMQNnlY8RD56HJ239OHYLg3+DpIf6Tdw29ru7fIO/Ke9EhG9skhOSanZEg4+U1uyR35k5DkY9JPBitp0lrP7JCNSPb/ARdTw3A=</latexit><latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit><latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit><latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit><latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit><latexit sha1_base64="oq/g0iP5izTeEta3oHw43r5mrTc=">AAACfXicZVDbahsxEJW3t9S9xGkf+yJqWhxwzW4IpHkIhOaheQmkUDeBrGu08qwtossizSY2Qj/Sr+lr+gf9m2ptP9TJgJgzZ86ImVNUUjhM07+t5NHjJ0+fbT1vv3j56vV2Z+fND2dqy2HIjTT2smAOpNAwRIESLisLTBUSLorrk6Z/cQPWCaO/46KCkWJTLUrBGUZq3NnPpZnSqpfPd+kRXRVjnyPM0X89PTsLoVfGupqJ8NN/ykIj3B13uukgXQZ9CLI16JJ1nI93Wnv5xPBagUYumXNXWVrhyDOLgksI7bx2UDF+zaZwFaFmCtzIL88L9ENkJrQ0Nj6NdMn+P+GZcorhLCqb5DZ6DYPGSNePKpypJjXfLGu3UEW/UP1GZF3p7i2C5eeRF7qqETRf7VHWkqKhjZV0IixwlIsIGLcinkL5jFnGMRq+sUShQruda7jlRimmJz6fB58Xys9D2ORhxUPko8vZfU8fguHe4HCQftvvHn9Z271F3pH3pEcyckCOySk5J0PCyS/ym9yRPwlJPib9ZLCSJq31zFuyEcnBPxiTw3Q=</latexit>

Page 20: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Results of Learning w/ HMM (He et al. 2018)

noun proper

verb base

cardinal number

verb gerund

adjective

noun singular

noun plural

adverb

verb past tenseverb past participleverb 3rd singular

Page 21: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Unsupervised Phrase-structured Composition Functions

Page 22: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Soft vs. Hard Tree Structure• Soft tree structure: use a differentiable gating function

• Hard tree structure: non-differentiable, but allows for more complicated composition methods

x1 x2

x1,2

x3

x2,3

x1,30.2 0.8

Soft

x1 x2 x3

x2,3

x1,3Hard

Page 23: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

One Other Paradigm: Weak Supervision

• Supervised: given X,Y to model P(Y|X)

• Unsupervised: given X to model P(Y|X)

• Weakly Supervised: given X and V to model P(Y|X), under assumption that Y and V are correlated

• Note: different from multi-task or transfer learning because we are given no Y

• Note: different from supervised learning with latent variables, because we care about Y, not V

Page 24: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

• Can choose whether to use left node, right node, or combination of both

Gated Convolution (Cho et al. 2014)

• Trained using MT loss

Page 25: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning with RL (Yogatama et al. 2016)

• Intermediate tree-structured representation for language modeling

• Predict that tree using shift-reduce parsing, sentence representation composed in tree-structured manner

• Reinforcement learning with supervised loss, prediction loss

Page 26: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning w/ Layer-wise Reductions (Choi et al. 2017)

• Choose one parent at each layer, reducing size by one

• Train using Gumbel-straighthrough reparameterization trick

• Faster and more effective than RL?

• Williams et al. (2017) find that this gives less trivial trees as well

Page 27: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Difficulties in Learning Latent Structure

(Williams et al. 2018)• Unfortunately, many models learn trivial structure...

• e.g. balanced binary, left/right branching

• Why? One explanation: tension between (untrained) parser, and (untrained) downstream task model

Page 28: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning Dependencies

Page 29: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Phrase Structure vs. Dependency Structure

• Previous methods attempt to learn representations of phrases in tree-structured manner

• We might also want to learn dependencies, that tell which words depend on others

Page 30: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Dependency Model w/ Valence (Klein and Manning 2004)

• Basic idea: top-down dependency based language model that generates left and right sides, then stops

I saw a girl with a telescopeROOT• For both the right and left side, calculate whether to

continue generating words, and if yes generatee.g., a slightly simplified view for word “saw”Pd(<cont> | saw, ←, false) * Pw(I | saw, ←, false) *Pd(<stop> | saw, ←, true) * Pd(<cont> | saw, →, false) * Pw(girl | saw, ←, false) * Pd(<cont> | saw, →, true) * Pw(with | saw, ←, true) *Pd(<stop> | saw, ←, true)

Page 31: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Unsupervised Dependency Induction w/ Neural Nets (Jiang et al. 2016)

• Simple: parameterize the decision with neural nets instead of with count-based distributions

• Like DMV, train with EM algorithm

Page 32: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Invertible Projections for DMV (He et al. 2018)

z1<latexit sha1_base64="kbJpa6Sam6jLYh1k9CLbrRqm2Rk=">AAACMnicZVDLSsNAFJ34Nr5a3ekmWAQXpSQiqLuiG5eK1gqmlMl00g7OI8zcaGvIJ7jVL/FjxJW49SOcpF1YvTDcM+eeO5w5UcKZAd9/d2Zm5+YXFpeW3ZXVtfWNSnXzxqhUE9oiiit9G2FDOZO0BQw4vU00xSLitB3dnxXz9gPVhil5DaOEdgTuSxYzgsFSV0/doFup+Q2/LO8/CCaghiZ10a0622FPkVRQCYRjY+4CP4FOhjUwwmnuhqmhCSb3uE/vLJRYUNPJSq+5t2eZnhcrbY8Er2R/b2RYGIFhYJVFM1OzggGluKlbFQxE0YpnyrsZiageiXoh0iY2f4xAfNzJmExSoJKMfcQp90B5RS5ej2lKgI8swEQz+xWPDLDGBGx6UyYikbtuKOkjUUJg2cvCYZ6FkciGee7aNIO/2f0HrYPGScO/PKw1TyexLqEdtIv2UYCOUBOdowvUQgT10TN6Qa/Om/PhfDpfY+mMM9nZQlPlfP8AG16q6w==</latexit><latexit sha1_base64="kbJpa6Sam6jLYh1k9CLbrRqm2Rk=">AAACMnicZVDLSsNAFJ34Nr5a3ekmWAQXpSQiqLuiG5eK1gqmlMl00g7OI8zcaGvIJ7jVL/FjxJW49SOcpF1YvTDcM+eeO5w5UcKZAd9/d2Zm5+YXFpeW3ZXVtfWNSnXzxqhUE9oiiit9G2FDOZO0BQw4vU00xSLitB3dnxXz9gPVhil5DaOEdgTuSxYzgsFSV0/doFup+Q2/LO8/CCaghiZ10a0622FPkVRQCYRjY+4CP4FOhjUwwmnuhqmhCSb3uE/vLJRYUNPJSq+5t2eZnhcrbY8Er2R/b2RYGIFhYJVFM1OzggGluKlbFQxE0YpnyrsZiageiXoh0iY2f4xAfNzJmExSoJKMfcQp90B5RS5ej2lKgI8swEQz+xWPDLDGBGx6UyYikbtuKOkjUUJg2cvCYZ6FkciGee7aNIO/2f0HrYPGScO/PKw1TyexLqEdtIv2UYCOUBOdowvUQgT10TN6Qa/Om/PhfDpfY+mMM9nZQlPlfP8AG16q6w==</latexit><latexit sha1_base64="kbJpa6Sam6jLYh1k9CLbrRqm2Rk=">AAACMnicZVDLSsNAFJ34Nr5a3ekmWAQXpSQiqLuiG5eK1gqmlMl00g7OI8zcaGvIJ7jVL/FjxJW49SOcpF1YvTDcM+eeO5w5UcKZAd9/d2Zm5+YXFpeW3ZXVtfWNSnXzxqhUE9oiiit9G2FDOZO0BQw4vU00xSLitB3dnxXz9gPVhil5DaOEdgTuSxYzgsFSV0/doFup+Q2/LO8/CCaghiZ10a0622FPkVRQCYRjY+4CP4FOhjUwwmnuhqmhCSb3uE/vLJRYUNPJSq+5t2eZnhcrbY8Er2R/b2RYGIFhYJVFM1OzggGluKlbFQxE0YpnyrsZiageiXoh0iY2f4xAfNzJmExSoJKMfcQp90B5RS5ej2lKgI8swEQz+xWPDLDGBGx6UyYikbtuKOkjUUJg2cvCYZ6FkciGee7aNIO/2f0HrYPGScO/PKw1TyexLqEdtIv2UYCOUBOdowvUQgT10TN6Qa/Om/PhfDpfY+mMM9nZQlPlfP8AG16q6w==</latexit>

z2<latexit sha1_base64="k9/8EMjjLfM8r2mBl7kTM6haUuo=">AAACMnicZVDLSgMxFM3UVx1fVXe6GSyCi1KmRVB3RTcuFa0tdErJpJk2NI8huaPWYT7BrX6JHyOuxK0fYaZ2YeuFcE/OPTecnDDmzIDvvzuFhcWl5ZXiqru2vrG5VdreuTMq0YQ2ieJKt0NsKGeSNoEBp+1YUyxCTlvh6CKft+6pNkzJWxjHtCvwQLKIEQyWunnq1Xulsl/1J+X9B7UpKKNpXfW2nb2gr0giqATCsTGdmh9DN8UaGOE0c4PE0BiTER7QjoUSC2q66cRr5h1apu9FStsjwZuwfzdSLIzAMLTKvJmZWc6AUtxUrAqGIm/5M5O7GYuwEopKLtImMnNGIDrtpkzGCVBJfn1ECfdAeXkuXp9pSoCPLcBEM/sVjwyxxgRsejMmQpG5biDpA1FCYNlPg8csDUKRPmaZa9OszWf3HzTr1bOqf31cbpxPYy2ifXSAjlANnaAGukRXqIkIGqBn9IJenTfnw/l0vn6lBWe6s4tmyvn+AR0qquw=</latexit><latexit sha1_base64="k9/8EMjjLfM8r2mBl7kTM6haUuo=">AAACMnicZVDLSgMxFM3UVx1fVXe6GSyCi1KmRVB3RTcuFa0tdErJpJk2NI8huaPWYT7BrX6JHyOuxK0fYaZ2YeuFcE/OPTecnDDmzIDvvzuFhcWl5ZXiqru2vrG5VdreuTMq0YQ2ieJKt0NsKGeSNoEBp+1YUyxCTlvh6CKft+6pNkzJWxjHtCvwQLKIEQyWunnq1Xulsl/1J+X9B7UpKKNpXfW2nb2gr0giqATCsTGdmh9DN8UaGOE0c4PE0BiTER7QjoUSC2q66cRr5h1apu9FStsjwZuwfzdSLIzAMLTKvJmZWc6AUtxUrAqGIm/5M5O7GYuwEopKLtImMnNGIDrtpkzGCVBJfn1ECfdAeXkuXp9pSoCPLcBEM/sVjwyxxgRsejMmQpG5biDpA1FCYNlPg8csDUKRPmaZa9OszWf3HzTr1bOqf31cbpxPYy2ifXSAjlANnaAGukRXqIkIGqBn9IJenTfnw/l0vn6lBWe6s4tmyvn+AR0qquw=</latexit><latexit sha1_base64="k9/8EMjjLfM8r2mBl7kTM6haUuo=">AAACMnicZVDLSgMxFM3UVx1fVXe6GSyCi1KmRVB3RTcuFa0tdErJpJk2NI8huaPWYT7BrX6JHyOuxK0fYaZ2YeuFcE/OPTecnDDmzIDvvzuFhcWl5ZXiqru2vrG5VdreuTMq0YQ2ieJKt0NsKGeSNoEBp+1YUyxCTlvh6CKft+6pNkzJWxjHtCvwQLKIEQyWunnq1Xulsl/1J+X9B7UpKKNpXfW2nb2gr0giqATCsTGdmh9DN8UaGOE0c4PE0BiTER7QjoUSC2q66cRr5h1apu9FStsjwZuwfzdSLIzAMLTKvJmZWc6AUtxUrAqGIm/5M5O7GYuwEopKLtImMnNGIDrtpkzGCVBJfn1ECfdAeXkuXp9pSoCPLcBEM/sVjwyxxgRsejMmQpG5biDpA1FCYNlPg8csDUKRPmaZa9OszWf3HzTr1bOqf31cbpxPYy2ifXSAjlANnaAGukRXqIkIGqBn9IJenTfnw/l0vn6lBWe6s4tmyvn+AR0qquw=</latexit>

?

z3<latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit><latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit><latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit><latexit sha1_base64="bAUTGLr9EgIAz6GcyetjDEUesRE=">AAACIXicZVDLTsJAFJ3iCysqbHXTSExcEFLcqDsTNy4xESGhhEynU5gwj2bmViFNf8CtX+LXGBfGX3FaWAjcZHLPnHvu5MwJE84M+P63U9nZ3ds/qB66RzX3+OS0XnsxKtWE9ojiSg9CbChnkvaAAaeDRFMsQk774eyhmPdfqTZMyWdYJHQk8ESymBEMluqO602/7ZflbYPOCjTRqsYN5yyIFEkFlUA4NmbY8RMYZVgDI5zmbpAammAywxM6tFBiQc0oK33m3qVlIi9W2h4JXsn+38iwMALD1CqLZtZmBQNKcdOyKpiKohXPlHezEGErFK1CpE1sNoxAfDvKmExSoJIsfcQp90B5RSZexDQlwBcWYKKZ/YpHplhjAja5NROhyF03kPSNKCGwjLJgnmdBKLJ5nrs2zM5mdNugd92+a/tPPqqic3SBrlAH3aB79Ii6qIcIitA7+nA+nS/nZ5l5xVmF30Br5fz+AWoppzk=</latexit><latexit sha1_base64="cHNML6NFzNaVJV0shKLwGy2tbDA=">AAACJ3icZVDLTsMwENzwJpTnES4RFRKHqkrhANyQuHAEQWmlpqoc12kt/IjsDVCifAJX+BI+BnFC/AhO6YHSlawdz85a44lTwS2G4Yc3N7+wuLS8suqvVdY3Nre2K3dWZ4ayJtVCm3ZMLBNcsSZyFKydGkZkLFgrvr8o560HZizX6hZHKetKMlA84ZSgo26ee8e9rWpYD8cVzILGBFRhUle9bW836muaSaaQCmJtpxGm2M2JQU4FK/wosywl9J4MWMdBRSSz3XzstQgOHNMPEm3cURiM2b8bOZFWEhw6Zdns1KxkUGtha06FQ1m28pnx3Y5kXItlrRQZm9h/RjA57eZcpRkyRX99JJkIUAdlLkGfG0ZRjBwg1HD3lYAOiSEUXXpTJmJZ+H6k2CPVUhLVz6OnIo9imT8Vhe/SbPzPbhY0j+pn9fA6hBXYg304hAacwDlcwhU0gcIAXuAV3rx379P7+o19zpvkvwNT5X3/AMQVqVA=</latexit><latexit sha1_base64="cHNML6NFzNaVJV0shKLwGy2tbDA=">AAACJ3icZVDLTsMwENzwJpTnES4RFRKHqkrhANyQuHAEQWmlpqoc12kt/IjsDVCifAJX+BI+BnFC/AhO6YHSlawdz85a44lTwS2G4Yc3N7+wuLS8suqvVdY3Nre2K3dWZ4ayJtVCm3ZMLBNcsSZyFKydGkZkLFgrvr8o560HZizX6hZHKetKMlA84ZSgo26ee8e9rWpYD8cVzILGBFRhUle9bW836muaSaaQCmJtpxGm2M2JQU4FK/wosywl9J4MWMdBRSSz3XzstQgOHNMPEm3cURiM2b8bOZFWEhw6Zdns1KxkUGtha06FQ1m28pnx3Y5kXItlrRQZm9h/RjA57eZcpRkyRX99JJkIUAdlLkGfG0ZRjBwg1HD3lYAOiSEUXXpTJmJZ+H6k2CPVUhLVz6OnIo9imT8Vhe/SbPzPbhY0j+pn9fA6hBXYg304hAacwDlcwhU0gcIAXuAV3rx379P7+o19zpvkvwNT5X3/AMQVqVA=</latexit><latexit sha1_base64="RLvPvxg5oSd4xWwIdnMbsHCGlB0=">AAACMnicZVDLTgIxFO3gC8e37nQzkZi4IGTQhbojunGJUYSEIaRTOtDYx6S9o+BkPsGtfokfY1wZt36EHWAhcpPmnp57bnN6wpgzA77/4RQWFpeWV4qr7tr6xubW9s7uvVGJJrRBFFe6FWJDOZO0AQw4bcWaYhFy2gwfrvJ585Fqw5S8g1FMOwL3JYsYwWCp2+fuaXe75Ff8cXnzoDoFJTStenfH2Q96iiSCSiAcG9Ou+jF0UqyBEU4zN0gMjTF5wH3atlBiQU0nHXvNvCPL9LxIaXskeGP270aKhREYBlaZNzMzyxlQipuyVcFA5C1/Znw3IxGWQ1HORdpE5p8RiM47KZNxAlSSiY8o4R4oL8/F6zFNCfCRBZhoZr/ikQHWmIBNb8ZEKDLXDSR9IkoILHtpMMzSIBTpMMtcm2b1f3bzoHFSuaj4N36pdjmNtYgO0CE6RlV0hmroGtVRAxHURy/oFb05786n8+V8T6QFZ7qzh2bK+fkFHbaq6Q==</latexit><latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit><latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit><latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit><latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit><latexit sha1_base64="GoWTV2IhQuk8x82feQPyqOSmaP4=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSQAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx197vVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwzw9qrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fHvaq7Q==</latexit>

z4<latexit sha1_base64="qRJO9a4HbiuS4+vcPiAoCJbeVKg=">AAACMnicZVDLSgMxFM3UVx1fVXe6GSyCi1KmUlB3RTcuFa0tdErJpJk2NI8huaPWYT7BrX6JHyOuxK0fYaZ2YeuFcE/OPTecnDDmzIDvvzuFhcWl5ZXiqru2vrG5VdreuTMq0YQ2ieJKt0NsKGeSNoEBp+1YUyxCTlvh6CKft+6pNkzJWxjHtCvwQLKIEQyWunnq1Xulsl/1J+X9B7UpKKNpXfW2nb2gr0giqATCsTGdmh9DN8UaGOE0c4PE0BiTER7QjoUSC2q66cRr5h1apu9FStsjwZuwfzdSLIzAMLTKvJmZWc6AUtxUrAqGIm/5M5O7GYuwEopKLtImMnNGIDrtpkzGCVBJfn1ECfdAeXkuXp9pSoCPLcBEM/sVjwyxxgRsejMmQpG5biDpA1FCYNlPg8csDUKRPmaZa9OszWf3HzSPq2dV/7pebpxPYy2ifXSAjlANnaAGukRXqIkIGqBn9IJenTfnw/l0vn6lBWe6s4tmyvn+ASDCqu4=</latexit><latexit sha1_base64="qRJO9a4HbiuS4+vcPiAoCJbeVKg=">AAACMnicZVDLSgMxFM3UVx1fVXe6GSyCi1KmUlB3RTcuFa0tdErJpJk2NI8huaPWYT7BrX6JHyOuxK0fYaZ2YeuFcE/OPTecnDDmzIDvvzuFhcWl5ZXiqru2vrG5VdreuTMq0YQ2ieJKt0NsKGeSNoEBp+1YUyxCTlvh6CKft+6pNkzJWxjHtCvwQLKIEQyWunnq1Xulsl/1J+X9B7UpKKNpXfW2nb2gr0giqATCsTGdmh9DN8UaGOE0c4PE0BiTER7QjoUSC2q66cRr5h1apu9FStsjwZuwfzdSLIzAMLTKvJmZWc6AUtxUrAqGIm/5M5O7GYuwEopKLtImMnNGIDrtpkzGCVBJfn1ECfdAeXkuXp9pSoCPLcBEM/sVjwyxxgRsejMmQpG5biDpA1FCYNlPg8csDUKRPmaZa9OszWf3HzSPq2dV/7pebpxPYy2ifXSAjlANnaAGukRXqIkIGqBn9IJenTfnw/l0vn6lBWe6s4tmyvn+ASDCqu4=</latexit><latexit sha1_base64="qRJO9a4HbiuS4+vcPiAoCJbeVKg=">AAACMnicZVDLSgMxFM3UVx1fVXe6GSyCi1KmUlB3RTcuFa0tdErJpJk2NI8huaPWYT7BrX6JHyOuxK0fYaZ2YeuFcE/OPTecnDDmzIDvvzuFhcWl5ZXiqru2vrG5VdreuTMq0YQ2ieJKt0NsKGeSNoEBp+1YUyxCTlvh6CKft+6pNkzJWxjHtCvwQLKIEQyWunnq1Xulsl/1J+X9B7UpKKNpXfW2nb2gr0giqATCsTGdmh9DN8UaGOE0c4PE0BiTER7QjoUSC2q66cRr5h1apu9FStsjwZuwfzdSLIzAMLTKvJmZWc6AUtxUrAqGIm/5M5O7GYuwEopKLtImMnNGIDrtpkzGCVBJfn1ECfdAeXkuXp9pSoCPLcBEM/sVjwyxxgRsejMmQpG5biDpA1FCYNlPg8csDUKRPmaZa9OszWf3HzSPq2dV/7pebpxPYy2ifXSAjlANnaAGukRXqIkIGqBn9IJenTfnw/l0vn6lBWe6s4tmyvn+ASDCqu4=</latexit>

z5<latexit sha1_base64="7v8NYCmbh9REcawh1qKc3rzsbb8=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSBAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx1D7rVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwz/drrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fIo6q7w==</latexit><latexit sha1_base64="7v8NYCmbh9REcawh1qKc3rzsbb8=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSBAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx1D7rVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwz/drrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fIo6q7w==</latexit><latexit sha1_base64="7v8NYCmbh9REcawh1qKc3rzsbb8=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapSBAJuFVw4FkEBiUSV4zithR+RvQFKlE/gCl/CxyBOiCsfgdP2QGEla8ezs9Z4opQzA77/7kxNz8zOzVcW3MWl5ZXV6tr6lVGZJrRDFFf6JsKGciZpBxhwepNqikXE6XV0d1rOr++pNkzJSxikNBS4J1nCCAZLXTx1D7rVmt/wh+X9B80xqKFxtbtrzmYQK5IJKoFwbMxt008hzLEGRjgt3CAzNMXkDvforYUSC2rCfOi18HYsE3uJ0vZI8Ibs740cCyMw9K2ybGZiVjKgFDd1q4K+KFv5zPBuBiKqR6JeirRJzB8jkByFOZNpBlSSkY8k4x4or8zFi5mmBPjAAkw0s1/xSB9rTMCmN2EiEoXrBpI+ECUElnEePBZ5EIn8sShcm2bzb3b/QWevcdzwz/drrZNxrBW0hbbRLmqiQ9RCZ6iNOoigHnpGL+jVeXM+nE/naySdcsY7G2iinO8fIo6q7w==</latexit>

z6<latexit sha1_base64="pGOJAC+5XIxMYgPoGWdA3fbkKOc=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapShHjcKrhwLIICEokqx3FaCz8iewOUKJ/AFb6Ej0GcEFc+AqftgcJK1o5nZ63xRClnBnz/3Zmanpmdm68suItLyyur1bX1K6MyTWiHKK70TYQN5UzSDjDg9CbVFIuI0+vo7rScX99TbZiSlzBIaShwT7KEEQyWunjqHnSrNb/hD8v7D5pjUEPjanfXnM0gViQTVALh2Jjbpp9CmGMNjHBauEFmaIrJHe7RWwslFtSE+dBr4e1YJvYSpe2R4A3Z3xs5FkZg6Ftl2czErGRAKW7qVgV9UbbymeHdDERUj0S9FGmTmD9GIDkKcybTDKgkIx9Jxj1QXpmLFzNNCfCBBZhoZr/ikT7WmIBNb8JEJArXDSR9IEoILOM8eCzyIBL5Y1G4Ns3m3+z+g85e47jhn+/XWifjWCtoC22jXdREh6iFzlAbdRBBPfSMXtCr8+Z8OJ/O10g65Yx3NtBEOd8/JFqq8A==</latexit><latexit sha1_base64="pGOJAC+5XIxMYgPoGWdA3fbkKOc=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapShHjcKrhwLIICEokqx3FaCz8iewOUKJ/AFb6Ej0GcEFc+AqftgcJK1o5nZ63xRClnBnz/3Zmanpmdm68suItLyyur1bX1K6MyTWiHKK70TYQN5UzSDjDg9CbVFIuI0+vo7rScX99TbZiSlzBIaShwT7KEEQyWunjqHnSrNb/hD8v7D5pjUEPjanfXnM0gViQTVALh2Jjbpp9CmGMNjHBauEFmaIrJHe7RWwslFtSE+dBr4e1YJvYSpe2R4A3Z3xs5FkZg6Ftl2czErGRAKW7qVgV9UbbymeHdDERUj0S9FGmTmD9GIDkKcybTDKgkIx9Jxj1QXpmLFzNNCfCBBZhoZr/ikT7WmIBNb8JEJArXDSR9IEoILOM8eCzyIBL5Y1G4Ns3m3+z+g85e47jhn+/XWifjWCtoC22jXdREh6iFzlAbdRBBPfSMXtCr8+Z8OJ/O10g65Yx3NtBEOd8/JFqq8A==</latexit><latexit sha1_base64="pGOJAC+5XIxMYgPoGWdA3fbkKOc=">AAACMnicZVDLTsMwEHR4lvAqcINLRIXEoapShHjcKrhwLIICEokqx3FaCz8iewOUKJ/AFb6Ej0GcEFc+AqftgcJK1o5nZ63xRClnBnz/3Zmanpmdm68suItLyyur1bX1K6MyTWiHKK70TYQN5UzSDjDg9CbVFIuI0+vo7rScX99TbZiSlzBIaShwT7KEEQyWunjqHnSrNb/hD8v7D5pjUEPjanfXnM0gViQTVALh2Jjbpp9CmGMNjHBauEFmaIrJHe7RWwslFtSE+dBr4e1YJvYSpe2R4A3Z3xs5FkZg6Ftl2czErGRAKW7qVgV9UbbymeHdDERUj0S9FGmTmD9GIDkKcybTDKgkIx9Jxj1QXpmLFzNNCfCBBZhoZr/ikT7WmIBNb8JEJArXDSR9IEoILOM8eCzyIBL5Y1G4Ns3m3+z+g85e47jhn+/XWifjWCtoC22jXdREh6iFzlAbdRBBPfSMXtCr8+Z8OJ/O10g65Yx3NtBEOd8/JFqq8A==</latexit>

z7<latexit sha1_base64="UaMLlKfO/9t66m7p0UvzLTfZft8=">AAACMnicZVDLSsNAFJ34rPFZ3ekmWAQXpaQiqLuiG5cVjQomlMl00g7OI8zcaGvIJ7jVL/FjxJW49SOctF1YvTDcM+eeO5w5ccqZAd9/d2Zm5+YXFitL7vLK6tr6RnXz2qhMExoQxZW+jbGhnEkaAANOb1NNsYg5vYnvz8r5zQPVhil5BcOURgL3JEsYwWCpy6fOUWej5jf8UXn/QXMCamhS7U7V2Q67imSCSiAcG3PX9FOIcqyBEU4LN8wMTTG5xz16Z6HEgpooH3ktvD3LdL1EaXskeCP290aOhREY+lZZNjM1KxlQipu6VUFflK18ZnQ3QxHXY1EvRdok5o8RSI6jnMk0AyrJ2EeScQ+UV+bidZmmBPjQAkw0s1/xSB9rTMCmN2UiFoXrhpI+EiUElt08HBR5GIt8UBSuTbP5N7v/IDhonDT8i8Na63QSawXtoF20j5roCLXQOWqjABHUQ8/oBb06b86H8+l8jaUzzmRnC02V8/0DJiaq8Q==</latexit><latexit sha1_base64="UaMLlKfO/9t66m7p0UvzLTfZft8=">AAACMnicZVDLSsNAFJ34rPFZ3ekmWAQXpaQiqLuiG5cVjQomlMl00g7OI8zcaGvIJ7jVL/FjxJW49SOctF1YvTDcM+eeO5w5ccqZAd9/d2Zm5+YXFitL7vLK6tr6RnXz2qhMExoQxZW+jbGhnEkaAANOb1NNsYg5vYnvz8r5zQPVhil5BcOURgL3JEsYwWCpy6fOUWej5jf8UXn/QXMCamhS7U7V2Q67imSCSiAcG3PX9FOIcqyBEU4LN8wMTTG5xz16Z6HEgpooH3ktvD3LdL1EaXskeCP290aOhREY+lZZNjM1KxlQipu6VUFflK18ZnQ3QxHXY1EvRdok5o8RSI6jnMk0AyrJ2EeScQ+UV+bidZmmBPjQAkw0s1/xSB9rTMCmN2UiFoXrhpI+EiUElt08HBR5GIt8UBSuTbP5N7v/IDhonDT8i8Na63QSawXtoF20j5roCLXQOWqjABHUQ8/oBb06b86H8+l8jaUzzmRnC02V8/0DJiaq8Q==</latexit><latexit sha1_base64="UaMLlKfO/9t66m7p0UvzLTfZft8=">AAACMnicZVDLSsNAFJ34rPFZ3ekmWAQXpaQiqLuiG5cVjQomlMl00g7OI8zcaGvIJ7jVL/FjxJW49SOctF1YvTDcM+eeO5w5ccqZAd9/d2Zm5+YXFitL7vLK6tr6RnXz2qhMExoQxZW+jbGhnEkaAANOb1NNsYg5vYnvz8r5zQPVhil5BcOURgL3JEsYwWCpy6fOUWej5jf8UXn/QXMCamhS7U7V2Q67imSCSiAcG3PX9FOIcqyBEU4LN8wMTTG5xz16Z6HEgpooH3ktvD3LdL1EaXskeCP290aOhREY+lZZNjM1KxlQipu6VUFflK18ZnQ3QxHXY1EvRdok5o8RSI6jnMk0AyrJ2EeScQ+UV+bidZmmBPjQAkw0s1/xSB9rTMCmN2UiFoXrhpI+EiUElt08HBR5GIt8UBSuTbP5N7v/IDhonDT8i8Na63QSawXtoF20j5roCLXQOWqjABHUQ8/oBb06b86H8+l8jaUzzmRnC02V8/0DJiaq8Q==</latexit>

x1<latexit sha1_base64="wPlw8y/0IqfHJrazcSQaHuwQ07k=">AAACDXicZVDLSgMxFM34rPVVdekmWAQXpcyIoO6KblxWcGyhLSWTZtrQZDIkd8Qy9B/ErX6HK3HrN/gZ/oGZ6SxseyHck3PPDScniAU34Lo/zsrq2vrGZmmrvL2zu7dfOTh8NCrRlPlUCaXbATFM8Ij5wEGwdqwZkYFgrWB8m81bT0wbrqIHmMSsJ8kw4iGnBCzV7gYyfZ72vX6l6tbdvPAy8ApQRUU1+5Xf7kDRRLIIqCDGdDw3hl5KNHAq2LTcTQyLCR2TIetYGBHJTC/N/U7xqWUGOFTanghwzv7fSIk0ksDIKrNm5mYZA0oJU7MqGMmsZc/kdzORQS2QtUykTWgWjEB41Ut5FCfAIjrzESYCg8JZNnjANaMgJhYQqrn9CqYjogkFm2DZZuQtJrIM/PP6dd29v6g2boqwSugYnaAz5KFL1EB3qIl8RJFAr+gNvTsvzofz6XzNpCtOsXOE5sr5/gMr950D</latexit><latexit sha1_base64="wPlw8y/0IqfHJrazcSQaHuwQ07k=">AAACDXicZVDLSgMxFM34rPVVdekmWAQXpcyIoO6KblxWcGyhLSWTZtrQZDIkd8Qy9B/ErX6HK3HrN/gZ/oGZ6SxseyHck3PPDScniAU34Lo/zsrq2vrGZmmrvL2zu7dfOTh8NCrRlPlUCaXbATFM8Ij5wEGwdqwZkYFgrWB8m81bT0wbrqIHmMSsJ8kw4iGnBCzV7gYyfZ72vX6l6tbdvPAy8ApQRUU1+5Xf7kDRRLIIqCDGdDw3hl5KNHAq2LTcTQyLCR2TIetYGBHJTC/N/U7xqWUGOFTanghwzv7fSIk0ksDIKrNm5mYZA0oJU7MqGMmsZc/kdzORQS2QtUykTWgWjEB41Ut5FCfAIjrzESYCg8JZNnjANaMgJhYQqrn9CqYjogkFm2DZZuQtJrIM/PP6dd29v6g2boqwSugYnaAz5KFL1EB3qIl8RJFAr+gNvTsvzofz6XzNpCtOsXOE5sr5/gMr950D</latexit><latexit sha1_base64="wPlw8y/0IqfHJrazcSQaHuwQ07k=">AAACDXicZVDLSgMxFM34rPVVdekmWAQXpcyIoO6KblxWcGyhLSWTZtrQZDIkd8Qy9B/ErX6HK3HrN/gZ/oGZ6SxseyHck3PPDScniAU34Lo/zsrq2vrGZmmrvL2zu7dfOTh8NCrRlPlUCaXbATFM8Ij5wEGwdqwZkYFgrWB8m81bT0wbrqIHmMSsJ8kw4iGnBCzV7gYyfZ72vX6l6tbdvPAy8ApQRUU1+5Xf7kDRRLIIqCDGdDw3hl5KNHAq2LTcTQyLCR2TIetYGBHJTC/N/U7xqWUGOFTanghwzv7fSIk0ksDIKrNm5mYZA0oJU7MqGMmsZc/kdzORQS2QtUykTWgWjEB41Ut5FCfAIjrzESYCg8JZNnjANaMgJhYQqrn9CqYjogkFm2DZZuQtJrIM/PP6dd29v6g2boqwSugYnaAz5KFL1EB3qIl8RJFAr+gNvTsvzofz6XzNpCtOsXOE5sr5/gMr950D</latexit>

e2<latexit sha1_base64="wBcNk6euWGMrg+Da+mnoLSs6/Fg=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcq0COqu6MZlBUcL7VAyaaYNzWNI7ghl6D+IW/0OV+LWb/Az/AMz01nY9kK4J+eeG05OGHNmwPN+nNLa+sbmVnm7srO7t39QPTx6NCrRhPpEcaW7ITaUM0l9YMBpN9YUi5DTp3Bym82fnqk2TMkHmMY0EHgkWcQIBkt1+6FI6WzQGlRrXsPLy10FzQLUUFGdQfW3P1QkEVQC4diYXtOLIUixBkY4nVX6iaExJhM8oj0LJRbUBGnud+aeWWboRkrbI8HN2f8bKRZGYBhbZdbMwixjQClu6lYFY5G17Jn8bqYirIeinom0icySEYiugpTJOAEqydxHlHAXlJtl4w6ZpgT41AJMNLNfcckYa0zAJlixGTWXE1kFfqtx3fDuL2rtmyKsMjpBp+gcNdElaqM71EE+IoijV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAOWpzx</latexit><latexit sha1_base64="wBcNk6euWGMrg+Da+mnoLSs6/Fg=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcq0COqu6MZlBUcL7VAyaaYNzWNI7ghl6D+IW/0OV+LWb/Az/AMz01nY9kK4J+eeG05OGHNmwPN+nNLa+sbmVnm7srO7t39QPTx6NCrRhPpEcaW7ITaUM0l9YMBpN9YUi5DTp3Bym82fnqk2TMkHmMY0EHgkWcQIBkt1+6FI6WzQGlRrXsPLy10FzQLUUFGdQfW3P1QkEVQC4diYXtOLIUixBkY4nVX6iaExJhM8oj0LJRbUBGnud+aeWWboRkrbI8HN2f8bKRZGYBhbZdbMwixjQClu6lYFY5G17Jn8bqYirIeinom0icySEYiugpTJOAEqydxHlHAXlJtl4w6ZpgT41AJMNLNfcckYa0zAJlixGTWXE1kFfqtx3fDuL2rtmyKsMjpBp+gcNdElaqM71EE+IoijV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAOWpzx</latexit><latexit sha1_base64="wBcNk6euWGMrg+Da+mnoLSs6/Fg=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcq0COqu6MZlBUcL7VAyaaYNzWNI7ghl6D+IW/0OV+LWb/Az/AMz01nY9kK4J+eeG05OGHNmwPN+nNLa+sbmVnm7srO7t39QPTx6NCrRhPpEcaW7ITaUM0l9YMBpN9YUi5DTp3Bym82fnqk2TMkHmMY0EHgkWcQIBkt1+6FI6WzQGlRrXsPLy10FzQLUUFGdQfW3P1QkEVQC4diYXtOLIUixBkY4nVX6iaExJhM8oj0LJRbUBGnud+aeWWboRkrbI8HN2f8bKRZGYBhbZdbMwixjQClu6lYFY5G17Jn8bqYirIeinom0icySEYiugpTJOAEqydxHlHAXlJtl4w6ZpgT41AJMNLNfcckYa0zAJlixGTWXE1kFfqtx3fDuL2rtmyKsMjpBp+gcNdElaqM71EE+IoijV/SG3p0X58P5dL7m0pJT7ByjhXK+/wAOWpzx</latexit>

e1<latexit sha1_base64="5hfya15Z2veNhmcIusOWF706ODA=">AAACS3icZVDLSgMxFM3Ud31VXboZLIKLUmZEUHdFNy4VrBY6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmH1Qrgn554bTk6cCm4xCD68ysLi0vLK6lp1fWNza7u2s3tndWYYtJkW2nRiakFwBW3kKKCTGqAyFnAfP16W8/snMJZrdYvjFHqSPiiecEaxpCLoh/1aPWgGk/L/g3AG6mRW1/0d7zgaaJZJUMgEtbYbBin2cmqQMwFFNcospJQ90gfoOqioBNvLJ2YL/9AxAz/Rxh2F/oT9vZFTaSXFoVOWzc7NSga1FrbhVDiUZSufmdztWMaNWDZKkbGJ/WMEk7NezlWaISg29ZFkwkftl8H4A26AoRg7QJnh7is+G1JDGbr45kzEsqhWIwXPTEtJ1SCPRkUexTIfFcU8D1MeHO9SDv9m+h+0j5vnzeDmpN66mMW9SvbJATkiITklLXJFrkmbMDIkL+SVvHnv3qf35X1PpRVvtrNH5qqy9AOQxLSV</latexit><latexit sha1_base64="5hfya15Z2veNhmcIusOWF706ODA=">AAACS3icZVDLSgMxFM3Ud31VXboZLIKLUmZEUHdFNy4VrBY6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmH1Qrgn554bTk6cCm4xCD68ysLi0vLK6lp1fWNza7u2s3tndWYYtJkW2nRiakFwBW3kKKCTGqAyFnAfP16W8/snMJZrdYvjFHqSPiiecEaxpCLoh/1aPWgGk/L/g3AG6mRW1/0d7zgaaJZJUMgEtbYbBin2cmqQMwFFNcospJQ90gfoOqioBNvLJ2YL/9AxAz/Rxh2F/oT9vZFTaSXFoVOWzc7NSga1FrbhVDiUZSufmdztWMaNWDZKkbGJ/WMEk7NezlWaISg29ZFkwkftl8H4A26AoRg7QJnh7is+G1JDGbr45kzEsqhWIwXPTEtJ1SCPRkUexTIfFcU8D1MeHO9SDv9m+h+0j5vnzeDmpN66mMW9SvbJATkiITklLXJFrkmbMDIkL+SVvHnv3qf35X1PpRVvtrNH5qqy9AOQxLSV</latexit><latexit sha1_base64="5hfya15Z2veNhmcIusOWF706ODA=">AAACS3icZVDLSgMxFM3Ud31VXboZLIKLUmZEUHdFNy4VrBY6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmH1Qrgn554bTk6cCm4xCD68ysLi0vLK6lp1fWNza7u2s3tndWYYtJkW2nRiakFwBW3kKKCTGqAyFnAfP16W8/snMJZrdYvjFHqSPiiecEaxpCLoh/1aPWgGk/L/g3AG6mRW1/0d7zgaaJZJUMgEtbYbBin2cmqQMwFFNcospJQ90gfoOqioBNvLJ2YL/9AxAz/Rxh2F/oT9vZFTaSXFoVOWzc7NSga1FrbhVDiUZSufmdztWMaNWDZKkbGJ/WMEk7NezlWaISg29ZFkwkftl8H4A26AoRg7QJnh7is+G1JDGbr45kzEsqhWIwXPTEtJ1SCPRkUexTIfFcU8D1MeHO9SDv9m+h+0j5vnzeDmpN66mMW9SvbJATkiITklLXJFrkmbMDIkL+SVvHnv3qf35X1PpRVvtrNH5qqy9AOQxLSV</latexit>

e3<latexit sha1_base64="Yo9t2xRNhbcucp40+9En1vW30ug=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqYqqLuiG5cVrBWcUjLpHRvMY0juaMswv+BWv8q9/+FKXJhpu7D2Qrgn554bTk6UCG4xCD690sLi0vJKebWytr6xubW9s3tndWoYtJkW2txH1ILgCtrIUcB9YoDKSEAneroq5p1nMJZrdYujBLqSPioec0axoELonfS2q0E9GJc/DxpTUCXTavV2vOOwr1kqQSET1NqHRpBgN6MGOROQV8LUQkLZE32EBwcVlWC72dhs7h86pu/H2rij0B+zfzcyKq2kOHDKotmZWcGg1sLWnAoHsmjFM+O7HcmoFslaITI2tv+MYHzezbhKUgTFJj7iVPio/SIYv88NMBQjBygz3H3FZwNqKEMX34yJSOaVSqjghWkpqepn4TDPwkhmwzyf5WHCg+Ndyo3/mc6D9nH9oh7cnFabl9O4y2SfHJAj0iBnpEmuSYu0CSMD8kreyLv34X15397PRFrypjt7ZKZKS7+UjLSX</latexit><latexit sha1_base64="Yo9t2xRNhbcucp40+9En1vW30ug=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqYqqLuiG5cVrBWcUjLpHRvMY0juaMswv+BWv8q9/+FKXJhpu7D2Qrgn554bTk6UCG4xCD690sLi0vJKebWytr6xubW9s3tndWoYtJkW2txH1ILgCtrIUcB9YoDKSEAneroq5p1nMJZrdYujBLqSPioec0axoELonfS2q0E9GJc/DxpTUCXTavV2vOOwr1kqQSET1NqHRpBgN6MGOROQV8LUQkLZE32EBwcVlWC72dhs7h86pu/H2rij0B+zfzcyKq2kOHDKotmZWcGg1sLWnAoHsmjFM+O7HcmoFslaITI2tv+MYHzezbhKUgTFJj7iVPio/SIYv88NMBQjBygz3H3FZwNqKEMX34yJSOaVSqjghWkpqepn4TDPwkhmwzyf5WHCg+Ndyo3/mc6D9nH9oh7cnFabl9O4y2SfHJAj0iBnpEmuSYu0CSMD8kreyLv34X15397PRFrypjt7ZKZKS7+UjLSX</latexit><latexit sha1_base64="Yo9t2xRNhbcucp40+9En1vW30ug=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqYqqLuiG5cVrBWcUjLpHRvMY0juaMswv+BWv8q9/+FKXJhpu7D2Qrgn554bTk6UCG4xCD690sLi0vJKebWytr6xubW9s3tndWoYtJkW2txH1ILgCtrIUcB9YoDKSEAneroq5p1nMJZrdYujBLqSPioec0axoELonfS2q0E9GJc/DxpTUCXTavV2vOOwr1kqQSET1NqHRpBgN6MGOROQV8LUQkLZE32EBwcVlWC72dhs7h86pu/H2rij0B+zfzcyKq2kOHDKotmZWcGg1sLWnAoHsmjFM+O7HcmoFslaITI2tv+MYHzezbhKUgTFJj7iVPio/SIYv88NMBQjBygz3H3FZwNqKEMX34yJSOaVSqjghWkpqepn4TDPwkhmwzyf5WHCg+Ndyo3/mc6D9nH9oh7cnFabl9O4y2SfHJAj0iBnpEmuSYu0CSMD8kreyLv34X15397PRFrypjt7ZKZKS7+UjLSX</latexit>

e4<latexit sha1_base64="pOqccpiEKJr77/8/ytM6r0hBvEQ=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUyloO6KblwqWBU6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW19Y3y5tb2zm6lundndWoYdJgW2jxE1ILgCjrIUcBDYoDKSMB99HRZzO+fwViu1S2OE+hJ+qh4zBnFggqh3+pXakEjmJT/HzRnoEZmdd2veifhQLNUgkImqLXdZpBgL6MGOROQl8PUQkLZE32EroOKSrC9bGI2948cM/BjbdxR6E/YvxsZlVZSHDpl0ezcrGBQa2HrToVDWbTimcndjmVUj2S9EBkb2wUjGJ/1Mq6SFEGxqY84FT5qvwjGH3ADDMXYAcoMd1/x2ZAaytDFN2ciknm5HCp4YVpKqgZZOMqzMJLZKM/neZjy4HiXcnMx0/+gc9I4bwQ3rVr7Yhb3Ojkgh+SYNMkpaZMrck06hJEheSVv5N378L68b+9nKi15s519MlellV+WcLSY</latexit><latexit sha1_base64="pOqccpiEKJr77/8/ytM6r0hBvEQ=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUyloO6KblwqWBU6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW19Y3y5tb2zm6lundndWoYdJgW2jxE1ILgCjrIUcBDYoDKSMB99HRZzO+fwViu1S2OE+hJ+qh4zBnFggqh3+pXakEjmJT/HzRnoEZmdd2veifhQLNUgkImqLXdZpBgL6MGOROQl8PUQkLZE32EroOKSrC9bGI2948cM/BjbdxR6E/YvxsZlVZSHDpl0ezcrGBQa2HrToVDWbTimcndjmVUj2S9EBkb2wUjGJ/1Mq6SFEGxqY84FT5qvwjGH3ADDMXYAcoMd1/x2ZAaytDFN2ciknm5HCp4YVpKqgZZOMqzMJLZKM/neZjy4HiXcnMx0/+gc9I4bwQ3rVr7Yhb3Ojkgh+SYNMkpaZMrck06hJEheSVv5N378L68b+9nKi15s519MlellV+WcLSY</latexit><latexit sha1_base64="pOqccpiEKJr77/8/ytM6r0hBvEQ=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUyloO6KblwqWBU6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW19Y3y5tb2zm6lundndWoYdJgW2jxE1ILgCjrIUcBDYoDKSMB99HRZzO+fwViu1S2OE+hJ+qh4zBnFggqh3+pXakEjmJT/HzRnoEZmdd2veifhQLNUgkImqLXdZpBgL6MGOROQl8PUQkLZE32EroOKSrC9bGI2948cM/BjbdxR6E/YvxsZlVZSHDpl0ezcrGBQa2HrToVDWbTimcndjmVUj2S9EBkb2wUjGJ/1Mq6SFEGxqY84FT5qvwjGH3ADDMXYAcoMd1/x2ZAaytDFN2ciknm5HCp4YVpKqgZZOMqzMJLZKM/neZjy4HiXcnMx0/+gc9I4bwQ3rVr7Yhb3Ojkgh+SYNMkpaZMrck06hJEheSVv5N378L68b+9nKi15s519MlellV+WcLSY</latexit>

e5<latexit sha1_base64="MjmPb/ACrHEtZbpogr0zpFexMKk=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqaiqLuiG5cVrBWcUjLpHRvMY0juaMswv+BWv8q9/+FKXJhpu7D2Qrgn554bTk6UCG4xCD690sLi0vJKebWytr6xubW9s3tndWoYtJkW2txH1ILgCtrIUcB9YoDKSEAneroq5p1nMJZrdYujBLqSPioec0axoELonfa2q0E9GJc/DxpTUCXTavV2vOOwr1kqQSET1NqHRpBgN6MGOROQV8LUQkLZE32EBwcVlWC72dhs7h86pu/H2rij0B+zfzcyKq2kOHDKotmZWcGg1sLWnAoHsmjFM+O7HcmoFslaITI2tv+MYHzezbhKUgTFJj7iVPio/SIYv88NMBQjBygz3H3FZwNqKEMX34yJSOaVSqjghWkpqepn4TDPwkhmwzyf5WHCg+Ndyo3/mc6D9nH9oh7cnFSbl9O4y2SfHJAj0iBnpEmuSYu0CSMD8kreyLv34X15397PRFrypjt7ZKZKS7+YVLSZ</latexit><latexit sha1_base64="MjmPb/ACrHEtZbpogr0zpFexMKk=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqaiqLuiG5cVrBWcUjLpHRvMY0juaMswv+BWv8q9/+FKXJhpu7D2Qrgn554bTk6UCG4xCD690sLi0vJKebWytr6xubW9s3tndWoYtJkW2txH1ILgCtrIUcB9YoDKSEAneroq5p1nMJZrdYujBLqSPioec0axoELonfa2q0E9GJc/DxpTUCXTavV2vOOwr1kqQSET1NqHRpBgN6MGOROQV8LUQkLZE32EBwcVlWC72dhs7h86pu/H2rij0B+zfzcyKq2kOHDKotmZWcGg1sLWnAoHsmjFM+O7HcmoFslaITI2tv+MYHzezbhKUgTFJj7iVPio/SIYv88NMBQjBygz3H3FZwNqKEMX34yJSOaVSqjghWkpqepn4TDPwkhmwzyf5WHCg+Ndyo3/mc6D9nH9oh7cnFSbl9O4y2SfHJAj0iBnpEmuSYu0CSMD8kreyLv34X15397PRFrypjt7ZKZKS7+YVLSZ</latexit><latexit sha1_base64="MjmPb/ACrHEtZbpogr0zpFexMKk=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqaiqLuiG5cVrBWcUjLpHRvMY0juaMswv+BWv8q9/+FKXJhpu7D2Qrgn554bTk6UCG4xCD690sLi0vJKebWytr6xubW9s3tndWoYtJkW2txH1ILgCtrIUcB9YoDKSEAneroq5p1nMJZrdYujBLqSPioec0axoELonfa2q0E9GJc/DxpTUCXTavV2vOOwr1kqQSET1NqHRpBgN6MGOROQV8LUQkLZE32EBwcVlWC72dhs7h86pu/H2rij0B+zfzcyKq2kOHDKotmZWcGg1sLWnAoHsmjFM+O7HcmoFslaITI2tv+MYHzezbhKUgTFJj7iVPio/SIYv88NMBQjBygz3H3FZwNqKEMX34yJSOaVSqjghWkpqepn4TDPwkhmwzyf5WHCg+Ndyo3/mc6D9nH9oh7cnFSbl9O4y2SfHJAj0iBnpEmuSYu0CSMD8kreyLv34X15397PRFrypjt7ZKZKS7+YVLSZ</latexit>

e6<latexit sha1_base64="MmCjOdrAP44kfWvHfghiTcRDIeQ=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqYiPnZFNy4rWCs4pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHthXBPzj03nJwoEdxiEHx6pYXFpeWV8mplbX1jc2t7Z/fO6tQwaDMttLmPqAXBFbSRo4D7xACVkYBO9HRVzDvPYCzX6hZHCXQlfVQ85oxiQYXQO+1tV4N6MC5/HjSmoEqm1erteMdhX7NUgkImqLUPjSDBbkYNciYgr4SphYSyJ/oIDw4qKsF2s7HZ3D90TN+PtXFHoT9m/25kVFpJceCURbMzs4JBrYWtORUOZNGKZ8Z3O5JRLZK1QmRsbP8Zwfi8m3GVpAiKTXzEqfBR+0Uwfp8bYChGDlBmuPuKzwbUUIYuvhkTkcwrlVDBC9NSUtXPwmGehZHMhnk+y8OEB8e7lBv/M50H7eP6RT24Oak2L6dxl8k+OSBHpEHOSJNckxZpE0YG5JW8kXfvw/vyvr2fibTkTXf2yEyVln4Bmji0mg==</latexit><latexit sha1_base64="MmCjOdrAP44kfWvHfghiTcRDIeQ=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqYiPnZFNy4rWCs4pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHthXBPzj03nJwoEdxiEHx6pYXFpeWV8mplbX1jc2t7Z/fO6tQwaDMttLmPqAXBFbSRo4D7xACVkYBO9HRVzDvPYCzX6hZHCXQlfVQ85oxiQYXQO+1tV4N6MC5/HjSmoEqm1erteMdhX7NUgkImqLUPjSDBbkYNciYgr4SphYSyJ/oIDw4qKsF2s7HZ3D90TN+PtXFHoT9m/25kVFpJceCURbMzs4JBrYWtORUOZNGKZ8Z3O5JRLZK1QmRsbP8Zwfi8m3GVpAiKTXzEqfBR+0Uwfp8bYChGDlBmuPuKzwbUUIYuvhkTkcwrlVDBC9NSUtXPwmGehZHMhnk+y8OEB8e7lBv/M50H7eP6RT24Oak2L6dxl8k+OSBHpEHOSJNckxZpE0YG5JW8kXfvw/vyvr2fibTkTXf2yEyVln4Bmji0mg==</latexit><latexit sha1_base64="MmCjOdrAP44kfWvHfghiTcRDIeQ=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqYiPnZFNy4rWCs4pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHthXBPzj03nJwoEdxiEHx6pYXFpeWV8mplbX1jc2t7Z/fO6tQwaDMttLmPqAXBFbSRo4D7xACVkYBO9HRVzDvPYCzX6hZHCXQlfVQ85oxiQYXQO+1tV4N6MC5/HjSmoEqm1erteMdhX7NUgkImqLUPjSDBbkYNciYgr4SphYSyJ/oIDw4qKsF2s7HZ3D90TN+PtXFHoT9m/25kVFpJceCURbMzs4JBrYWtORUOZNGKZ8Z3O5JRLZK1QmRsbP8Zwfi8m3GVpAiKTXzEqfBR+0Uwfp8bYChGDlBmuPuKzwbUUIYuvhkTkcwrlVDBC9NSUtXPwmGehZHMhnk+y8OEB8e7lBv/M50H7eP6RT24Oak2L6dxl8k+OSBHpEHOSJNckxZpE0YG5JW8kXfvw/vyvr2fibTkTXf2yEyVln4Bmji0mg==</latexit>

e7<latexit sha1_base64="+2fC0KPrqycQirOuKDePs7MeCTY=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFqO6KblwqWBU6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW19Y3y5tb2zm6lundndWoYdJgW2jxE1ILgCjrIUcBDYoDKSMB99HRZzO+fwViu1S2OE+hJ+qh4zBnFggqh3+pXakEjmJT/HzRnoEZmdd2veifhQLNUgkImqLXdZpBgL6MGOROQl8PUQkLZE32EroOKSrC9bGI2948cM/BjbdxR6E/YvxsZlVZSHDpl0ezcrGBQa2HrToVDWbTimcndjmVUj2S9EBkb2wUjGJ/1Mq6SFEGxqY84FT5qvwjGH3ADDMXYAcoMd1/x2ZAaytDFN2ciknm5HCp4YVpKqgZZOMqzMJLZKM/neZjy4HiXcnMx0/+gc9I4bwQ3p7X2xSzudXJADskxaZIWaZMrck06hJEheSVv5N378L68b+9nKi15s519MlellV+cHLSb</latexit><latexit sha1_base64="+2fC0KPrqycQirOuKDePs7MeCTY=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFqO6KblwqWBU6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW19Y3y5tb2zm6lundndWoYdJgW2jxE1ILgCjrIUcBDYoDKSMB99HRZzO+fwViu1S2OE+hJ+qh4zBnFggqh3+pXakEjmJT/HzRnoEZmdd2veifhQLNUgkImqLXdZpBgL6MGOROQl8PUQkLZE32EroOKSrC9bGI2948cM/BjbdxR6E/YvxsZlVZSHDpl0ezcrGBQa2HrToVDWbTimcndjmVUj2S9EBkb2wUjGJ/1Mq6SFEGxqY84FT5qvwjGH3ADDMXYAcoMd1/x2ZAaytDFN2ciknm5HCp4YVpKqgZZOMqzMJLZKM/neZjy4HiXcnMx0/+gc9I4bwQ3p7X2xSzudXJADskxaZIWaZMrck06hJEheSVv5N378L68b+9nKi15s519MlellV+cHLSb</latexit><latexit sha1_base64="+2fC0KPrqycQirOuKDePs7MeCTY=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFqO6KblwqWBU6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW19Y3y5tb2zm6lundndWoYdJgW2jxE1ILgCjrIUcBDYoDKSMB99HRZzO+fwViu1S2OE+hJ+qh4zBnFggqh3+pXakEjmJT/HzRnoEZmdd2veifhQLNUgkImqLXdZpBgL6MGOROQl8PUQkLZE32EroOKSrC9bGI2948cM/BjbdxR6E/YvxsZlVZSHDpl0ezcrGBQa2HrToVDWbTimcndjmVUj2S9EBkb2wUjGJ/1Mq6SFEGxqY84FT5qvwjGH3ADDMXYAcoMd1/x2ZAaytDFN2ciknm5HCp4YVpKqgZZOMqzMJLZKM/neZjy4HiXcnMx0/+gc9I4bwQ3p7X2xSzudXJADskxaZIWaZMrck06hJEheSVv5N378L68b+9nKi15s519MlellV+cHLSb</latexit>

x4<latexit sha1_base64="9frhfHTAh/tVZBbTFF8T6hotkFU=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqZFUHeiG5cVrAqdUjLpHRuax5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW18nplY3Nre2d3b//e6tQwaDMttHmMqAXBFbSRo4DHxACVkYCHaHhdzB+ewViu1R2OE+hK+qR4zBnFggpHvdPebjWoB5Py/4PGDFTJrFq9Pa8Z9jVLJShkglrbaQQJdjNqkDMBeSVMLSSUDekTdBxUVILtZhOzuX/smL4fa+OOQn/C/t3IqLSS4sApi2bnZgWDWgtbcyocyKIVz0zudiyjWiRrhcjY2C4Ywfi8m3GVpAiKTX3EqfBR+0Uwfp8bYCjGDlBmuPuKzwbUUIYuvjkTkcwrlVDBC9NSUtXPwlGehZHMRnk+z8OUB8e7lBuLmf4H7Wb9oh7cnlYvr2Zxl8khOSInpEHOyCW5IS3SJowMyCt5I+/eh/flfXs/U2nJm+0ckLkqrfwCuoK0qw==</latexit><latexit sha1_base64="9frhfHTAh/tVZBbTFF8T6hotkFU=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqZFUHeiG5cVrAqdUjLpHRuax5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW18nplY3Nre2d3b//e6tQwaDMttHmMqAXBFbSRo4DHxACVkYCHaHhdzB+ewViu1R2OE+hK+qR4zBnFggpHvdPebjWoB5Py/4PGDFTJrFq9Pa8Z9jVLJShkglrbaQQJdjNqkDMBeSVMLSSUDekTdBxUVILtZhOzuX/smL4fa+OOQn/C/t3IqLSS4sApi2bnZgWDWgtbcyocyKIVz0zudiyjWiRrhcjY2C4Ywfi8m3GVpAiKTX3EqfBR+0Uwfp8bYCjGDlBmuPuKzwbUUIYuvjkTkcwrlVDBC9NSUtXPwlGehZHMRnk+z8OUB8e7lBuLmf4H7Wb9oh7cnlYvr2Zxl8khOSInpEHOyCW5IS3SJowMyCt5I+/eh/flfXs/U2nJm+0ckLkqrfwCuoK0qw==</latexit><latexit sha1_base64="9frhfHTAh/tVZBbTFF8T6hotkFU=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqZFUHeiG5cVrAqdUjLpHRuax5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW18nplY3Nre2d3b//e6tQwaDMttHmMqAXBFbSRo4DHxACVkYCHaHhdzB+ewViu1R2OE+hK+qR4zBnFggpHvdPebjWoB5Py/4PGDFTJrFq9Pa8Z9jVLJShkglrbaQQJdjNqkDMBeSVMLSSUDekTdBxUVILtZhOzuX/smL4fa+OOQn/C/t3IqLSS4sApi2bnZgWDWgtbcyocyKIVz0zudiyjWiRrhcjY2C4Ywfi8m3GVpAiKTX3EqfBR+0Uwfp8bYCjGDlBmuPuKzwbUUIYuvjkTkcwrlVDBC9NSUtXPwlGehZHMRnk+z8OUB8e7lBuLmf4H7Wb9oh7cnlYvr2Zxl8khOSInpEHOyCW5IS3SJowMyCt5I+/eh/flfXs/U2nJm+0ckLkqrfwCuoK0qw==</latexit>

x5<latexit sha1_base64="Iqq29VCa1Eput5eq9gbubvV+118=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFUXdFNy4VrBY6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmH1Qrgn554bTk6UCG4xCD680sLi0vLK6lp5fWNza7tS3bmzOjUM2kwLbToRtSC4gjZyFNBJDFAZCbiPHi+L+f0TGMu1usVxAj1JHxSPOaNYUOGof9Kv1IJGMCn/P2jOQI3M6rpf9Y7CgWapBIVMUGu7zSDBXkYNciYgL4ephYSyR/oAXQcVlWB72cRs7h84ZuDH2rij0J+wvzcyKq2kOHTKotm5WcGg1sLWnQqHsmjFM5O7HcuoHsl6ITI2tn+MYHzWy7hKUgTFpj7iVPio/SIYf8ANMBRjBygz3H3FZ0NqKEMX35yJSOblcqjgmWkpqRpk4SjPwkhmozyf52HKg+Ndys2/mf4H7aPGeSO4Oa61LmZxr5I9sk8OSZOckha5ItekTRgZkhfySt68d+/T+/K+p9KSN9vZJXNVWvoBvGa0rA==</latexit><latexit sha1_base64="Iqq29VCa1Eput5eq9gbubvV+118=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFUXdFNy4VrBY6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmH1Qrgn554bTk6UCG4xCD680sLi0vLK6lp5fWNza7tS3bmzOjUM2kwLbToRtSC4gjZyFNBJDFAZCbiPHi+L+f0TGMu1usVxAj1JHxSPOaNYUOGof9Kv1IJGMCn/P2jOQI3M6rpf9Y7CgWapBIVMUGu7zSDBXkYNciYgL4ephYSyR/oAXQcVlWB72cRs7h84ZuDH2rij0J+wvzcyKq2kOHTKotm5WcGg1sLWnQqHsmjFM5O7HcuoHsl6ITI2tn+MYHzWy7hKUgTFpj7iVPio/SIYf8ANMBRjBygz3H3FZ0NqKEMX35yJSOblcqjgmWkpqRpk4SjPwkhmozyf52HKg+Ndys2/mf4H7aPGeSO4Oa61LmZxr5I9sk8OSZOckha5ItekTRgZkhfySt68d+/T+/K+p9KSN9vZJXNVWvoBvGa0rA==</latexit><latexit sha1_base64="Iqq29VCa1Eput5eq9gbubvV+118=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFUXdFNy4VrBY6pWTSOzaYx5Dc0ZZhfsGtfpV7/8OVuDDTdmH1Qrgn554bTk6UCG4xCD680sLi0vLK6lp5fWNza7tS3bmzOjUM2kwLbToRtSC4gjZyFNBJDFAZCbiPHi+L+f0TGMu1usVxAj1JHxSPOaNYUOGof9Kv1IJGMCn/P2jOQI3M6rpf9Y7CgWapBIVMUGu7zSDBXkYNciYgL4ephYSyR/oAXQcVlWB72cRs7h84ZuDH2rij0J+wvzcyKq2kOHTKotm5WcGg1sLWnQqHsmjFM5O7HcuoHsl6ITI2tn+MYHzWy7hKUgTFpj7iVPio/SIYf8ANMBRjBygz3H3FZ0NqKEMX35yJSOblcqjgmWkpqRpk4SjPwkhmozyf52HKg+Ndys2/mf4H7aPGeSO4Oa61LmZxr5I9sk8OSZOckha5ItekTRgZkhfySt68d+/T+/K+p9KSN9vZJXNVWvoBvGa0rA==</latexit>

x6<latexit sha1_base64="KY5/j4VzeKfzF+AkT011ZxYsr7Q=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFfOyKblwqWC10Ssmkd2wwjyG5oy3D/IJb/Sr3/ocrcWGm7cLqhXBPzj03nJwoEdxiEHx4pYXFpeWV1bXy+sbm1nalunNndWoYtJkW2nQiakFwBW3kKKCTGKAyEnAfPV4W8/snMJZrdYvjBHqSPigec0axoMJR/6RfqQWNYFL+f9CcgRqZ1XW/6h2FA81SCQqZoNZ2m0GCvYwa5ExAXg5TCwllj/QBug4qKsH2sonZ3D9wzMCPtXFHoT9hf29kVFpJceiURbNzs4JBrYWtOxUOZdGKZyZ3O5ZRPZL1QmRsbP8Ywfisl3GVpAiKTX3EqfBR+0Uw/oAbYCjGDlBmuPuKz4bUUIYuvjkTkczL5VDBM9NSUjXIwlGehZHMRnk+z8OUB8e7lJt/M/0P2keN80Zwc1xrXcziXiV7ZJ8ckiY5JS1yRa5JmzAyJC/klbx5796n9+V9T6Ulb7azS+aqtPQDvkq0rQ==</latexit><latexit sha1_base64="KY5/j4VzeKfzF+AkT011ZxYsr7Q=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFfOyKblwqWC10Ssmkd2wwjyG5oy3D/IJb/Sr3/ocrcWGm7cLqhXBPzj03nJwoEdxiEHx4pYXFpeWV1bXy+sbm1nalunNndWoYtJkW2nQiakFwBW3kKKCTGKAyEnAfPV4W8/snMJZrdYvjBHqSPigec0axoMJR/6RfqQWNYFL+f9CcgRqZ1XW/6h2FA81SCQqZoNZ2m0GCvYwa5ExAXg5TCwllj/QBug4qKsH2sonZ3D9wzMCPtXFHoT9hf29kVFpJceiURbNzs4JBrYWtOxUOZdGKZyZ3O5ZRPZL1QmRsbP8Ywfisl3GVpAiKTX3EqfBR+0Uw/oAbYCjGDlBmuPuKz4bUUIYuvjkTkczL5VDBM9NSUjXIwlGehZHMRnk+z8OUB8e7lJt/M/0P2keN80Zwc1xrXcziXiV7ZJ8ckiY5JS1yRa5JmzAyJC/klbx5796n9+V9T6Ulb7azS+aqtPQDvkq0rQ==</latexit><latexit sha1_base64="KY5/j4VzeKfzF+AkT011ZxYsr7Q=">AAACS3icZVDLSgMxFM3Ud321unQzWAQXpUxFfOyKblwqWC10Ssmkd2wwjyG5oy3D/IJb/Sr3/ocrcWGm7cLqhXBPzj03nJwoEdxiEHx4pYXFpeWV1bXy+sbm1nalunNndWoYtJkW2nQiakFwBW3kKKCTGKAyEnAfPV4W8/snMJZrdYvjBHqSPigec0axoMJR/6RfqQWNYFL+f9CcgRqZ1XW/6h2FA81SCQqZoNZ2m0GCvYwa5ExAXg5TCwllj/QBug4qKsH2sonZ3D9wzMCPtXFHoT9hf29kVFpJceiURbNzs4JBrYWtOxUOZdGKZyZ3O5ZRPZL1QmRsbP8Ywfisl3GVpAiKTX3EqfBR+0Uw/oAbYCjGDlBmuPuKz4bUUIYuvjkTkczL5VDBM9NSUjXIwlGehZHMRnk+z8OUB8e7lJt/M/0P2keN80Zwc1xrXcziXiV7ZJ8ckiY5JS1yRa5JmzAyJC/klbx5796n9+V9T6Ulb7azS+aqtPQDvkq0rQ==</latexit>

x7<latexit sha1_base64="QuysJyVG4LzFzaADM3QOo12uCc8=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqZFUHeiG5cVrAqdUjLpHRuax5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW18nplY3Nre2d3b//e6tQwaDMttHmMqAXBFbSRo4DHxACVkYCHaHhdzB+ewViu1R2OE+hK+qR4zBnFggpHvbPebjWoB5Py/4PGDFTJrFq9Pa8Z9jVLJShkglrbaQQJdjNqkDMBeSVMLSSUDekTdBxUVILtZhOzuX/smL4fa+OOQn/C/t3IqLSS4sApi2bnZgWDWgtbcyocyKIVz0zudiyjWiRrhcjY2C4Ywfi8m3GVpAiKTX3EqfBR+0Uwfp8bYCjGDlBmuPuKzwbUUIYuvjkTkcwrlVDBC9NSUtXPwlGehZHMRnk+z8OUB8e7lBuLmf4H7Wb9oh7cnlYvr2Zxl8khOSInpEHOyCW5IS3SJowMyCt5I+/eh/flfXs/U2nJm+0ckLkqrfwCwC60rg==</latexit><latexit sha1_base64="QuysJyVG4LzFzaADM3QOo12uCc8=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqZFUHeiG5cVrAqdUjLpHRuax5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW18nplY3Nre2d3b//e6tQwaDMttHmMqAXBFbSRo4DHxACVkYCHaHhdzB+ewViu1R2OE+hK+qR4zBnFggpHvbPebjWoB5Py/4PGDFTJrFq9Pa8Z9jVLJShkglrbaQQJdjNqkDMBeSVMLSSUDekTdBxUVILtZhOzuX/smL4fa+OOQn/C/t3IqLSS4sApi2bnZgWDWgtbcyocyKIVz0zudiyjWiRrhcjY2C4Ywfi8m3GVpAiKTX3EqfBR+0Uwfp8bYCjGDlBmuPuKzwbUUIYuvjkTkcwrlVDBC9NSUtXPwlGehZHMRnk+z8OUB8e7lBuLmf4H7Wb9oh7cnlYvr2Zxl8khOSInpEHOyCW5IS3SJowMyCt5I+/eh/flfXs/U2nJm+0ckLkqrfwCwC60rg==</latexit><latexit sha1_base64="QuysJyVG4LzFzaADM3QOo12uCc8=">AAACS3icZVDLSgMxFM3UV61vXboZLIKLUqZFUHeiG5cVrAqdUjLpHRuax5Dc0ZZhfsGtfpV7/8OVuDDTdmHrhXBPzj03nJwoEdxiEHx6paXlldW18nplY3Nre2d3b//e6tQwaDMttHmMqAXBFbSRo4DHxACVkYCHaHhdzB+ewViu1R2OE+hK+qR4zBnFggpHvbPebjWoB5Py/4PGDFTJrFq9Pa8Z9jVLJShkglrbaQQJdjNqkDMBeSVMLSSUDekTdBxUVILtZhOzuX/smL4fa+OOQn/C/t3IqLSS4sApi2bnZgWDWgtbcyocyKIVz0zudiyjWiRrhcjY2C4Ywfi8m3GVpAiKTX3EqfBR+0Uwfp8bYCjGDlBmuPuKzwbUUIYuvjkTkcwrlVDBC9NSUtXPwlGehZHMRnk+z8OUB8e7lBuLmf4H7Wb9oh7cnlYvr2Zxl8khOSInpEHOyCW5IS3SJowMyCt5I+/eh/flfXs/U2nJm+0ckLkqrfwCwC60rg==</latexit>

cat stopped dog inThe a Paris

x3<latexit sha1_base64="r7RTsVKDbYXIRCyPj9szuInKMVA=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcqMCuqu6MZlBccW2qFk0kwbmkyG5I5Yhv6DuNXvcCVu/QY/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7Gt9m8/USVZjJ6gElMfYGHEQsZwWCoTi8Q6fO0f96v1pyGk5e9DNwC1FBRrX71tzeQJBE0AsKx1l3XicFPsQJGOJ1WeommMSZjPKRdAyMsqPbT3O/UPjHMwA6lMicCO2f/b6RYaIFhZJRZ03OzjAEpua4bFYxE1rJn8rueiKAeiHomUjrUC0YgvPJTFsUJ0IjMfIQJt0HaWTb2gClKgE8MwEQx8xWbjLDCBEyCFZORu5jIMvDOGtcN5/6i1rwpwiqjI3SMTpGLLlET3aEW8hBBHL2iN/RuvVgf1qf1NZOWrGLnEM2V9f0HLzudBQ==</latexit><latexit sha1_base64="r7RTsVKDbYXIRCyPj9szuInKMVA=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcqMCuqu6MZlBccW2qFk0kwbmkyG5I5Yhv6DuNXvcCVu/QY/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7Gt9m8/USVZjJ6gElMfYGHEQsZwWCoTi8Q6fO0f96v1pyGk5e9DNwC1FBRrX71tzeQJBE0AsKx1l3XicFPsQJGOJ1WeommMSZjPKRdAyMsqPbT3O/UPjHMwA6lMicCO2f/b6RYaIFhZJRZ03OzjAEpua4bFYxE1rJn8rueiKAeiHomUjrUC0YgvPJTFsUJ0IjMfIQJt0HaWTb2gClKgE8MwEQx8xWbjLDCBEyCFZORu5jIMvDOGtcN5/6i1rwpwiqjI3SMTpGLLlET3aEW8hBBHL2iN/RuvVgf1qf1NZOWrGLnEM2V9f0HLzudBQ==</latexit><latexit sha1_base64="r7RTsVKDbYXIRCyPj9szuInKMVA=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcqMCuqu6MZlBccW2qFk0kwbmkyG5I5Yhv6DuNXvcCVu/QY/wz8wM52FbS+Ee3LuueHkBDFnGhznxyqtrK6tb5Q3K1vbO7t71f2DRy0TRahHJJeqE2BNOYuoBww47cSKYhFw2g7Gt9m8/USVZjJ6gElMfYGHEQsZwWCoTi8Q6fO0f96v1pyGk5e9DNwC1FBRrX71tzeQJBE0AsKx1l3XicFPsQJGOJ1WeommMSZjPKRdAyMsqPbT3O/UPjHMwA6lMicCO2f/b6RYaIFhZJRZ03OzjAEpua4bFYxE1rJn8rueiKAeiHomUjrUC0YgvPJTFsUJ0IjMfIQJt0HaWTb2gClKgE8MwEQx8xWbjLDCBEyCFZORu5jIMvDOGtcN5/6i1rwpwiqjI3SMTpGLLlET3aEW8hBBHL2iN/RuvVgf1qf1NZOWrGLnEM2V9f0HLzudBQ==</latexit>

x2<latexit sha1_base64="OUe9NQ3ny9IilmHPU3ONQ4TtfRo=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3LCo4ttEPJpJk2NJkMyR2xDP0Hcavf4Urc+g1+hn9gZjoL214I9+Tcc8PJCWLONDjOj1VaW9/Y3CpvV3Z29/YPqodHj1omilCPSC5VN8CachZRDxhw2o0VxSLgtBNMbrN554kqzWT0ANOY+gKPIhYygsFQ3X4g0ufZoDmo1pyGk5e9CtwC1FBR7UH1tz+UJBE0AsKx1j3XicFPsQJGOJ1V+ommMSYTPKI9AyMsqPbT3O/MPjPM0A6lMicCO2f/b6RYaIFhbJRZ0wuzjAEpua4bFYxF1rJn8rueiqAeiHomUjrUS0YgvPJTFsUJ0IjMfYQJt0HaWTb2kClKgE8NwEQx8xWbjLHCBEyCFZORu5zIKvCajeuGc39Ra90UYZXRCTpF58hFl6iF7lAbeYggjl7RG3q3XqwP69P6mktLVrFzjBbK+v4DLZmdBA==</latexit><latexit sha1_base64="OUe9NQ3ny9IilmHPU3ONQ4TtfRo=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3LCo4ttEPJpJk2NJkMyR2xDP0Hcavf4Urc+g1+hn9gZjoL214I9+Tcc8PJCWLONDjOj1VaW9/Y3CpvV3Z29/YPqodHj1omilCPSC5VN8CachZRDxhw2o0VxSLgtBNMbrN554kqzWT0ANOY+gKPIhYygsFQ3X4g0ufZoDmo1pyGk5e9CtwC1FBR7UH1tz+UJBE0AsKx1j3XicFPsQJGOJ1V+ommMSYTPKI9AyMsqPbT3O/MPjPM0A6lMicCO2f/b6RYaIFhbJRZ0wuzjAEpua4bFYxF1rJn8rueiqAeiHomUjrUS0YgvPJTFsUJ0IjMfYQJt0HaWTb2kClKgE8NwEQx8xWbjLHCBEyCFZORu5zIKvCajeuGc39Ra90UYZXRCTpF58hFl6iF7lAbeYggjl7RG3q3XqwP69P6mktLVrFzjBbK+v4DLZmdBA==</latexit><latexit sha1_base64="OUe9NQ3ny9IilmHPU3ONQ4TtfRo=">AAACDXicZVDLSgMxFM3UV62vqks3g0VwUcpMEdRd0Y3LCo4ttEPJpJk2NJkMyR2xDP0Hcavf4Urc+g1+hn9gZjoL214I9+Tcc8PJCWLONDjOj1VaW9/Y3CpvV3Z29/YPqodHj1omilCPSC5VN8CachZRDxhw2o0VxSLgtBNMbrN554kqzWT0ANOY+gKPIhYygsFQ3X4g0ufZoDmo1pyGk5e9CtwC1FBR7UH1tz+UJBE0AsKx1j3XicFPsQJGOJ1V+ommMSYTPKI9AyMsqPbT3O/MPjPM0A6lMicCO2f/b6RYaIFhbJRZ0wuzjAEpua4bFYxF1rJn8rueiqAeiHomUjrUS0YgvPJTFsUJ0IjMfYQJt0HaWTb2kClKgE8NwEQx8xWbjLHCBEyCFZORu5zIKvCajeuGc39Ra90UYZXRCTpF58hFl6iF7lAbeYggjl7RG3q3XqwP69P6mktLVrFzjBbK+v4DLZmdBA==</latexit>

Page 33: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning Dependency Heads w/ Attention (Kuncoro et al. 2017)

• Given a phrase structure tree, what child is the head word, the most important word in the phrase?

• Idea: create a phrase composition function that uses attention: examine if attention weights follow heads defined by linguistics

Page 34: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Other Examples

Page 35: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning about Word Segmentation from Attention (Boito et al. 2017)

• We want to learn word segmentation in an unsegmented language

• Simple idea: we can inspect the attention matrices from a neural MT system to extract words

Page 36: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning Segmentations w/ Reconstruction Loss (Elsner and Shain 2017)• Learn segmentations of speech/text that allow for easy re-

construction of the original • Idea: consistent segmentation should result in easier-to-

reconstruct segments • Train segmentation using policy gradient

Page 37: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Learning Language-level Features (Malaviya et al. 2017)

• All previous work learned features of a single sentence

• Can we learn features of the whole language? e.g. Typology: what is the canonical word order, etc.

• A simple method: train a neural MT system on 1017 languages, and extract its representations

Page 38: CS11-747 Neural Networks for NLP Unsupervised and Semi ... · Learning Features vs. Learning Discrete Structure • Learning features, e.g. word/sentence embeddings: this is an example

Questions?