Structure and Interpretation of Neural Codes

Jacob Andreas

Translating Neuralese

Jacob Andreas, Anca Drăgan and Dan Klein

Learning to Communicate

33[Wagner et al. 03, Sukhbaatar et al. 16, Foerster et al. 16]

Learning to Communicate

Neuralese

1.02.3-0.30.4-1.21.1

Translating neuralese

1.02.3-0.30.4-1.21.1

all clear

• Interoperate with autonomous systems

• Diagnose errors

• Learn from solutions

Translating neuralese

1.02.3-0.30.4-1.21.1

all clear

[Lazaridou et al. 16] 7

Outline

Natural language & neuralese

Statistical machine translation

Semantic machine translation

Implementation details

Evaluation8

Outline

Evaluation9

Outline

Evaluation10

Outline

Evaluation1111

Outline

Evaluation12

Outline

Evaluation13

A statistical MT problem

max p( | ) p( )0 a aa

1.02.3-0.30.4-1.21.1

all clear

[e.g. Koehn 10]

How do we induce a translation model?

max p( | ) p( )0 a aa

max Σ p( | ) p( | ) p( )0 aa

Strategy mismatch

�(s) =1

�(s)

� �

ex � 1xs dx

Strategy mismatch

not sure

�(s) =1

�(s)

� �

ex � 1xs dx

Strategy mismatch

not sure

Strategy mismatch

not sure

Strategy mismatch

not sure

Strategy mismatch

not sure yes

Σ p( , | not sure ) p( not sure )0

Stat MT criterion doesn’t capture meaning

moving(0,3)→(1,4)In the

intersection

Outline

Evaluation

A “semantic MT” problem

I’m going north

The meaning of an utterance is given by its truth conditions

[Davidson 67]

26✔ ✔ ✘

I’m going north

[Davidson 67]

27(loc(goalblue)north)

I’m going north

280.4 0.2 0.001

I’m going north

the distribution over states in which it is uttered

[Beltagy et al. 14]

290.4 0.2 0.001

I’m going north

or equivalently, the belief it induces in listeners

[Frank et al. 09, A & Klein 16]

Representing meaning

This distribution is well-defined even if the “utterance” is a vector rather than a sequence of tokens.

Translating with meaning

1.02.3-0.30.4-1.21.1

In the intersection

1.02.3-0.30.4-1.21.1

I’m going north

1.02.3-0.30.4-1.21.1

I’m going north

p( | )0p( | )a

1.02.3-0.30.4-1.21.1

I’m going north

β( )0β( )a

Interlingua!

source text target text

β( )0 β( )a

KL( || )

Translation criterion

argmin

β( )0 β( )a

KL( || )

argmin

β( )0 β( )a

KL( || )

argmin

β( )0 β( )a

KL( || )

argmin

β( )0 β( )a

Computing representations

argmina

KL( || )β( ) β( )a0

Computing representations: sparsity

argmina

KL( || )β( ) β( )a0

p( | )ap( | )0

Computing representations: smoothing

argmina

KL( || )β( ) β( )a0

actions &messages

agentpolicy

argmina

KL( || )β( ) β( )a0

actions &messages

agentpolicy

agentmodel

argmina

KL( || )β( ) β( )a0

actions &messages

argmina

KL( || )β( ) β( )a0

actions &messages

humanpolicy

humanmodel

argmina

KL( || )β( ) β( )a0

Computing KL

argmina

KL( || )β( ) β( )a0

Computing KL

argmina

KL( || )β( ) β( )a0

KL(p || q) = Ep( )

Computing KL: sampling

argmina

KL( || )β( ) β( )a0

KL(p || q) = Σ p( ) logp( )

Finding translations

argmina

KL( || )β( ) β( )a0

Finding translations: brute force

argmina

KL( || )β( ) β( )a0

going north

crossing the intersection

I’m done

after you

argmina

KL( || )β( ) β( )a0

going north

crossing the intersection

I’m done

after you

Finding translations: brute force

KL( || )

Finding translations

argmin

β( )0 β( )a

Outline

Evaluation56

Referring expression games

1.02.3-0.30.4-1.21.1

orange bird with black

Evaluation: translator-in-the-loop

1.02.3-0.30.4-1.21.1

Evaluation: translator-in-the-loop

Experiment: color references

1.00Neuralese → Neuralese

English → English*0.83

Neuralese → English* English → Neuralese

Neuralese → Neuralese

English → English*Statistical MT 0.83

0.72 0.70

English → English*Statistical MT

Semantic MT

0.72 0.70

translation), we focus on developing automaticmeasures of system performance. We use the avail-able training data to develop simulated models ofhuman decisions; by first showing that these mod-els track well with human judgments, we can beconfident that their use in evaluations will corre-late with human understanding. We employ thefollowing two metrics:

Belief evaluation This evaluation focuses on thedenotational perspective in semantics that moti-vated the initial development of our model. Wehave successfully understood the semantics of amessage z

if, after translating z

, a humanlistener can form a correct belief about the statein which z

was produced. We construct a simplestate-guessing game where the listener is presentedwith a translated message and two state observa-tions, and must guess which state the speaker wasin when the message was emitted.

When translating from natural language to neu-ralese, we use the learned agent model to directlyguess the hidden state. For neuralese to naturallanguage we must first construct a “model humanlistener” to map from strings back to state repre-sentations; we do this by using the training data tofit a simple regression model that scores (state, sen-tence) pairs using a bag-of-words sentence repre-sentation. We find that our “model human” matchesthe judgments of real humans 83% of the time onthe colors task, 77% of the time on the birds task,and 77% of the time on the driving task. This givesus confidence that the model human gives a reason-ably accurate proxy for human interpretation.

Behavior evaluation This evaluation focuses onthe cooperative aspects of interpretability: we mea-sure the extent to which learned models are ableto interoperate with each other by way of a transla-tion layer. In the case of reference games, the goalof this semantic evaluation is identical to the goalof the game itself (to identify the hidden state ofthe speaker), so we perform this additional prag-matic evaluation only for the driving game. Wefound that the most data-efficient and reliable wayto make use of human game traces was to constructa “deaf” model human. The evaluation selects afull game trace from a human player, and replaysboth the human’s actions and messages exactly (dis-regarding any incoming messages); the evaluationmeasures the quality of the natural-language-to-neuralese translator, and the extent to which the

as speakerR H

r R 1.000.50 random0.70 direct0.73 belief (ours)

H*0.50

0.830.720.86

as speakerR H

H*0.50

0.770.570.75

Table 1: Evaluation results for reference games. (a) The colorstask. (b) The birds task. Whether the model human is in alistener or speaker role, translation based on belief matchingoutperforms both random and machine translation baselines.

learned agent model can accommodate a (real) hu-man given translations of the human’s messages.

Baselines We compare our approach to two base-lines: a random baseline that chooses a translationof each input uniformly from messages observedduring training, and a direct baseline that directlymaximizes p(z

0|z) (by analogy to a conventionalmachine translation system). This is accomplishedby sampling from a DCP speaker in training stateslabeled with natural language strings.

8 Results

In all below, “R” indicates a DCP agent, “H” in-dicates a real human, and “H*” indicates a modelhuman player.

Reference games Results for the two referencegames are shown in Table 1. The end-to-end trainedmodel achieves nearly perfect accuracy in both

magenta, hot, rose, violet, purple

magenta, hot, violet, rose, purple

olive, puke, pea, grey, brown

pinkish, grey, dull, pale, light

Figure 7: Best-scoring translations generated for color task.

magenta, hot, rose

magenta, hot, violet

olive, puke, pea

pinkish, grey, dull

as speakerR H

H*0.50

0.830.720.86

as speakerR H

H*0.50

0.770.570.75

8 Results

magenta, hot, rose

olive, puke, pea

pinkish, grey, dull

as speakerR H

H*0.50

0.830.720.86

as speakerR H

H*0.50

0.770.570.75

8 Results

magenta, hot, rose

olive, puke, pea

pinkish, grey, dull

as speakerR H

H*0.50

0.830.720.86

as speakerR H

H*0.50

0.770.570.75

8 Results

magenta, hot, rose

olive, puke, pea

pinkish, grey, dull

as speakerR H

H*0.50

0.830.720.86

as speakerR H

H*0.50

0.770.570.75

8 Results

magenta, hot, rose

olive, puke, pea

pinkish, grey, dull

as speakerR H

H*0.50

0.830.720.86

as speakerR H

H*0.50

0.770.570.75

8 Results

magenta, hot, rose

olive, puke, pea

pinkish, grey, dull

Experiment: image references

English → English*

Statistical MT

Semantic MT

alization and ablation techniques used in previouswork on understanding complex models (Strobeltet al., 2016; Ribeiro et al., 2016).

While structurally quite similar to the task ofmachine translation between pairs of human lan-guages, interpretation of neuralese poses a numberof novel challenges. First, there is no natural sourceof parallel data: there are no bilingual “speakers”of both neuralese and natural language. Second,there may not be a direct correspondence betweenthe strategy employed by humans and CDP agents:even if it were constrained to communicate usingnatural language, an automated agent might chooseto produce a different message from humans in agiven state. We tackle both of these challenges byappealing to the grounding of messages in game-play. Our approach is based on one of the coreinsights in natural language semantics: messages(whether in neuralese or natural language) havesimilar meanings when they induce similar beliefsabout the state of the world. We explore severalrelated questions:

• What makes a good translation, and underwhat conditions is translation possible at all(Section 4)?

• How can we build a model to translatebetween neuralese and natural language(Section 5)?

• How does this model trade off the compet-ing demands of translatability and accuracy inlearned policies (Section 6)?

We conclude by applying our approach to agentstrained on a number of different tasks, finding thatit outperforms a more conventional machine trans-lation criterion both when attempting to interoper-ate with a neuralese speaker and when predictingits hidden state.

2 Related work

A variety of approaches for learning deep policieswith communication were proposed essentially si-multaneously in the past year. We have broadlylabeled these as “communicating deep policies”;concrete examples include Lazaridou et al. (2016b),Foerster et al. (2016), and Sukhbaatar et al. (2016).The policy representation we employ in this paperis similar to the latter two of these, although thegeneral framework is agnostic to low-level model-ing details and could be straightforwardly applied

to other architectures. Analysis of communicationstrategies in all these papers has been largely ad-hoc, obtained by clustering states from which simi-lar messages are emitted and attempting to manu-ally assign semantics to these clusters. The presentwork aims at developing tools for performing thisanalysis automatically.

Most closely related to our approach is that ofLazaridou et al. (2016a), who also develop a modelfor assigning natural language interpretations tolearned messages; however, this approach relieson a combination of supervised cluster labels andis targeted specifically towards referring expres-sion games. Here we attempt to develop an ap-proach that can handle general multiagent interac-tions without assuming a prior discrete structure inspace of observations.

The literature on learning decentralized multi-agent policies in general is considerably larger(Bernstein et al., 2002; Dibangoye et al., 2016).This includes work focused on communication inmultiagent settings (Roth et al., 2005) and evencommunication using natural language messages(Vogel et al., 2013b). All of these approaches em-ploy structured communication schemes with man-ually engineered messaging protocols; these are, insome sense, automatically interpretable, but at thecost of introducing considerable complexity intoboth training and inference.

Our evaluation in this paper investigates com-munication strategies that arise in a number of dif-ferent games, including reference games and anextended-horizon driving game. Communicationstrategies for reference games were previosly ex-plored by Vogel et al. (2013a), Andreas and Klein(2016) and Kazemzadeh et al. (2014), and refer-

large bird, black wings, black crown

small brown, light brown, dark brown

Figure 2: Preview of our approach—best-scoring translationsgenerated for a reference game involving images of birds.The speaking agent’s goal is to send a message that uniquelyidentifies the bird on the left. From these translations it can beseen that the learned model appears to discriminate based oncoarse attributes like size and color.

alization and ablation techniques used in previouswork on understanding complex models (Strobeltet al., 2016; Ribeiro et al., 2016).

While structurally quite similar to the task ofmachine translation between pairs of human lan-guages, interpretation of neuralese poses a numberof novel challenges. First, there is no natural sourceof parallel data: there are no bilingual “speakers”of both neuralese and natural language. Second,there may not be a direct correspondence betweenthe strategy employed by humans and CDP agents:even if it were constrained to communicate usingnatural language, an automated agent might chooseto produce a different message from humans in agiven state. We tackle both of these challenges byappealing to the grounding of messages in game-play. Our approach is based on one of the coreinsights in natural language semantics: messages(whether in neuralese or natural language) havesimilar meanings when they induce similar beliefsabout the state of the world. We explore severalrelated questions:

• What makes a good translation, and underwhat conditions is translation possible at all(Section 4)?

• How can we build a model to translatebetween neuralese and natural language(Section 5)?

• How does this model trade off the compet-ing demands of translatability and accuracy inlearned policies (Section 6)?

We conclude by applying our approach to agentstrained on a number of different tasks, finding thatit outperforms a more conventional machine trans-lation criterion both when attempting to interoper-ate with a neuralese speaker and when predictingits hidden state.

2 Related work

A variety of approaches for learning deep policieswith communication were proposed essentially si-multaneously in the past year. We have broadlylabeled these as “communicating deep policies”;concrete examples include Lazaridou et al. (2016b),Foerster et al. (2016), and Sukhbaatar et al. (2016).The policy representation we employ in this paperis similar to the latter two of these, although thegeneral framework is agnostic to low-level model-ing details and could be straightforwardly applied

to other architectures. Analysis of communicationstrategies in all these papers has been largely ad-hoc, obtained by clustering states from which simi-lar messages are emitted and attempting to manu-ally assign semantics to these clusters. The presentwork aims at developing tools for performing thisanalysis automatically.

Most closely related to our approach is that ofLazaridou et al. (2016a), who also develop a modelfor assigning natural language interpretations tolearned messages; however, this approach relieson a combination of supervised cluster labels andis targeted specifically towards referring expres-sion games. Here we attempt to develop an ap-proach that can handle general multiagent interac-tions without assuming a prior discrete structure inspace of observations.

The literature on learning decentralized multi-agent policies in general is considerably larger(Bernstein et al., 2002; Dibangoye et al., 2016).This includes work focused on communication inmultiagent settings (Roth et al., 2005) and evencommunication using natural language messages(Vogel et al., 2013b). All of these approaches em-ploy structured communication schemes with man-ually engineered messaging protocols; these are, insome sense, automatically interpretable, but at thecost of introducing considerable complexity intoboth training and inference.

Our evaluation in this paper investigates com-munication strategies that arise in a number of dif-ferent games, including reference games and anextended-horizon driving game. Communicationstrategies for reference games were previosly ex-plored by Vogel et al. (2013a), Andreas and Klein(2016) and Kazemzadeh et al. (2014), and refer-

small brown, light brown, dark brown

Figure 2: Preview of our approach—best-scoring translationsgenerated for a reference game involving images of birds.The speaking agent’s goal is to send a message that uniquelyidentifies the bird on the left. From these translations it can beseen that the learned model appears to discriminate based oncoarse attributes like size and color.

small brown, light brown, dark brown71

Experiment: image references

Experiment: driving game

Neuralese ↔ English*

Statistical MT

Semantic MT1.49

How to translate

as speakerR H

H*0.50

0.830.720.86

as speakerR H

0.710.570.75

R / R H / H R / H

1.93 / 0.71 — / 0.771.35 / 0.641.49 / 0.67

1.54 / 0.67

Table 2: Behavior evaluation results for the driving game.Scores are presented in the form “reward / completion rate”.While less accurate than either humans or CDPs with a sharedlanguage, the models that employ a translation layer obtainhigher reward and a greater overall success rate than baselines.

Reference games Results for the two referencegames are shown in Table 1. The end-to-end trainedmodel achieves nearly perfect accuracy in bothcases, while a model trained to communicate innatural language achieves somewhat lower perfor-mance. Regardless of whether the speaker is a CDPand the listener a model human or vice-versa, trans-lation based on the belief-matching criterion in Sec-tion 5 achieves the best performance; indeed, whentranslating from neuralese to natural language, thelistener is able to achieve a higher score than it isnatively. This suggests that the automated agenthas discovered a more effective strategy than theone demonstrated by humans in the dataset, andthat the effectiveness of this strategy is preservedby translation. Example translations from the refer-ence games are depicted in Figure 2 and Figure 7.

as speakerR H

0.770.450.57

Table 3: Belief evaluation results for the driving game. Drivingstates are challenging to identify based on messages alone (asevidenced by the comparatively low scores obtained by single-language pairs) . Translation based on belief achieves the bestoverall performance in both directions.

Driving game Behavior evaluation of the drivinggame is shown in Table 2, and belief evaluation isshown in Table 3. Translation of messages in thedriving game is considerably more challenging thanin the reference games, and scores are uniformlylower; however, a clear benefit from the belief-matching model is still visible. Belief matchingleads to higher scores on the belief evaluation inboth directions, and allows agents to obtain a higherreward on average (though task completion ratesremain roughly the same across all agents).

Some example translations of driving game mes-sages are shown in Figure 8.

9 Conclusion

We have investigated the problem of interpretingmessage vectors from communicating deep policiesby translating them. After introducing a translationcriterion based on matching listener beliefs aboutspeaker states, we presented both theoretical andempirical evidence that this criterion outperformsa more conventional machine translation approachat both recovering the content of message vectorsand facilitating collaboration between neuraleseand natural language speakers.

at goal, done, left to top

going in intersection, proceed, going

you first, following, going down

Figure 8: Best-scoring translations generated for driving task.

as speakerR H

H*0.50

0.830.720.86

as speakerR H

0.710.570.75

R / R H / H R / H

1.93 / 0.71 — / 0.771.35 / 0.641.49 / 0.67

1.54 / 0.67

as speakerR H

0.770.450.57

9 Conclusion

as speakerR H

H*0.50

0.830.720.86

as speakerR H

0.710.570.75

R / R H / H R / H

1.93 / 0.71 — / 0.771.35 / 0.641.49 / 0.67

1.54 / 0.67

as speakerR H

0.770.450.57

9 Conclusion

at goaldoneleft to top

going in intersectionproceedgoing

you firstfollowinggoing down

• Classical notions of “meaning” apply even toun-language-like things (e.g. RNN states)

• These meanings can be compactly represented without logical forms if we have access to world states

• Communicating policies “say” interpretable things!

Conclusions so far

• Classical notions of “meaning” apply even tonon-language-like things (e.g. RNN states)

Conclusions so far

• Classical notions of “meaning” apply even tonon-language-like things (e.g. RNN states)

Conclusions so far

Limitations

argmina

KL( || )β( ) β( )a0

KL(p || q) = Σ p( ) logp( )

but what about compositionality?

Analogs of linguistic structure in deep representations

Jacob Andreas and Dan Klein

“Flat” semantics

as speakerR H

H*0.50

0.830.720.86

as speakerR H

0.710.570.75

R / R H / H R / H

1.93 / 0.71 — / 0.771.35 / 0.641.49 / 0.67

1.54 / 0.67

as speakerR H

0.770.450.57

9 Conclusion

as speakerR H

H*0.50

0.830.720.86

as speakerR H

0.710.570.75

R / R H / H R / H

1.93 / 0.71 — / 0.771.35 / 0.641.49 / 0.67

1.54 / 0.67

as speakerR H

0.770.450.57

9 Conclusion

as speakerR H

H*0.50

0.830.720.86

as speakerR H

0.710.570.75

R / R H / H R / H

1.93 / 0.71 — / 0.771.35 / 0.641.49 / 0.67

1.54 / 0.67

as speakerR H

0.770.450.57

9 Conclusion

at goaldone

going in intersectionproceedgoing

you firstfollowing

Compositional semantics

✔ ✔ ✔

message

✔ ✔ ✔

everything but the blue shapes

orange square and non-squares

84[FitzGerald et al. 2013]

✔ ✔ ✔

lambdax:not(blue(x))

lambdax:or(orange(x),not(square(x))

85[FitzGerald et al. 2013]

✔ ✔ ✔

Model architecture

✔ ✔ ✔

Model architecture

✔ ✔ ✔

1.02.3-0.30.4-1.21.1

Computing meaning representations

1.02.3-0.30.4-1.21.1

on the left

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

everything but squares

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

lambdax:not(square(x))

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

lambdax:not(square(x))

q( , ) =a KL( || )β( ) β( )a0

0.2299

q( , ) =a β( ) E[ ]β( )a0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

Experiments

“High-level” communicative behavior

“Low-level” message structure

Experiments

Comparing strategies

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔ ✔✔

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔ ✔

✔ ✔✔

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

Theories of model behavior: random

-0.11.30.5-0.40.21.0

✔ ✔ ✔

✔ ✔ ✔✔

✔ ✔

Theories of model behavior: literal

0 Literal Human

Evaluation: high-level scene agreement

Random Literal Human

Evaluation: high-level object agreement

Experiments

Collecting translation data

all the red shapes

blue objects

everything but red

green squares

not green squares

λx.red(x)

λx.blu(x)

λx.¬red(x)

λx.grn(x)∧sqr(x)

λx.¬(grn(x)∧sqr(x))

λx.red(x)

λx.blu(x)

λx.¬red(x)

λx.grn(x)∧sqr(x)

λx.¬(grn(x)∧sqr(x))

0.1-0.30.51.1

-0.30.20.10.1

1.4-0.3-0.50.8

0.2-0.20.5-0.1

0.3-1.3-1.50.1

Extracting related pairs

λx.red(x) λx.¬red(x)

λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x))

0.1-0.30.51.1 1.4-0.3-0.50.8

0.2-0.20.5-0.1 0.3-1.3-1.50.1

Extracting related pairs

0.1-0.30.51.1 1.4-0.3-0.50.8

0.2-0.20.5-0.1 0.3-1.3-1.50.1

Learning compositional operators

argmin

Evaluating learned operators

0.1-0.30.51.1 1.4-0.3-0.50.8

0.2-0.20.5-0.1 0.3-1.3-1.50.1

λx.f(x)

0.2-0.20.5-0.1

0.1-0.30.51.1 1.4-0.3-0.50.8

0.2-0.20.5-0.1 0.3-1.3-1.50.1

λx.f(x)

0.2-0.20.5-0.1 -0.20.4-0.30.0

0.1-0.30.51.1 1.4-0.3-0.50.8

0.2-0.20.5-0.1 0.3-1.3-1.50.1

λx.f(x)

0.2-0.20.5-0.1

-0.20.4-0.30.0

Evaluation: scene agreement for negation

122�4 �3 �2 �1 0 1 2 3 4 5�5

3all the toys that

are not red

every thing that is red

only the blue andgreen objects

all items that arenot blue or green

Predicted

Visualizing negation

Evaluation: scene agreement for disjunction

Predicted

�4 �3 �2 �1 0 1 2 3 4�3

3all of the red objects

the blue andred items

the blue objects

the blue andyellow items

all the yellow toys

all yellow orred items

Visualizing disjunction

• We can translate between neuralese and natural lang. by grounding in distributions over world states

• Under the right conditions, neuralese exhibits interpretable pragmatics & compositional structure

• Not just communication games—language might be a good general-purpose tool for interpreting deep reprs.

Conclusions

not sure

1.02.3-0.30.4-1.21.1 Thank you!

http://github.com/jacobandreas/{neuralese,rnn-syn}

Structure and Interpretation of Neural Codes · 2017. 9. 7. · Jacob Andreas. Translating...

Transcript of Structure and Interpretation of Neural Codes · 2017. 9. 7. · Jacob Andreas. Translating...

Structure and Interpretation of Neural Codes · 2017. 9. 7. · Jacob Andreas. Translating...

Documents

Transcript of Structure and Interpretation of Neural Codes · 2017. 9. 7. · Jacob Andreas. Translating...

Development in Turbulent Times - SpringerPaul Dobrescu National University of Political Studies and Public Administration, Bucharest, Romania Gabriela Drăgan Bucharest University

Jacob, Georges. Jacob - Impressions Dominicales - Helas

Learning to Compose Neural Networks for Question … to Compose Neural Networks for Question Answering Jacob Andreas and Marcus Rohrbach and Trevor Darrell and Dan Klein Department

Hochleistungs-SCR-Katalysatorsystem: Garant für … · 27. Internationales Wiener Motorensymposium 2006 1 Dr. Eberhard Jacob, Dr. Raimund Müller, Dr. Andreas Scheeder, Emitec GmbH,

A commercially available digitization system Fotiou Andreas Andreas Fotiou.

A Detector Upgrade for LDSS3 Mike Gladders Jacob Bean (on the phone) with Andreas Seifart, Josh Frieman, John Carlstrom.

Alex Drăgan - PPC compatibil cu publishing (2015.05.27, Impact Hub Bucharest)

Jacob Andreas - socialmedia-class.orgsocialmedia-class.org/slides/students_2017/Social... · Learning to Compose Neural Networks for Question Answering Question: Can a question-answering

Prof. Dr. Jacob H. Jacob

Emerson: American Jacob American Jacob. Jacob becomes Israel: He who strove with God—and triumphed.

Manuel Muñoz-Herrera, Jacob Dijkstra, Andreas Flache ......Manuel Munoz-Herrera~ Jacob Dijkstray Andreas Flachez Rafael Wittekx September 4, 2020 Abstract We develop a model of strategic

Vol. 10 No. 4 December 2010 - rjea.ier.gov.rorjea.ier.gov.ro/wp-content/uploads/revista/RJEA_2009_vol._10_no_.4... · Gabriela Drăgan – Director General of the European Institute

Tribute to «Jacob de HAAN» Jacob de HAAN

Translating Neuralese - arXiv · Translating Neuralese Jacob Andreas Anca Dragan Dan Klein ... human–human translation problems grounded in gameplay. In this paper, we focus our

Andreas Brett Andreas Leicher - Dr. Andreas U. Schmidt

Jacob “Jaques” Therien Je m'appelle Jacob “Jaques” Therien.

UNIVERSITY INDUSTRY COOPERATION IN CENTRAL AND …ceswp.uaic.ro/articles/CESWP2012_IV4_SER.pdf · 2013. 3. 31. · Gabriela Drăgan Academy of Economic Studies in Bucharest, Romania

Learning from Language...1 Abstract Learning from Language by Jacob Daniel Andreas Doctor of Philosophy in Computer Science University of California, Berkeley Prof. Dan Klein, Chair

San Andreas, showing stream offset. San Andreas – Carrizo plain.

Borggrafe, Andreas and Ohndorf, Andreas and Dachwald ...