Expert-augmented actor-critic for ViZDoom and Montezuma’s Revenge

Michał GarmulewiczUniversity of Warsaw

Henryk Michalewski, Piotr MiłosUniversity of Warsaw

deepsense.ai

AbstractWe propose an expert-augmented actor-critic algorithm,which we evaluate on two environments with sparse re-wards: Montezuma’s Revenge and a demanding maze fromthe ViZDoom suite. In the case of Montezuma’s Revenge, anagent trained with our method achieves very good results con-sistently scoring above 27,000 points (in many experimentsbeating the first world). With an appropriate choice of hyper-parameters, our algorithm surpasses the performance of theexpert data. In a number of experiments, we have observed anunreported bug in Montezuma’s Revenge which allowed theagent to score more than 800, 000 points.

1 IntroductionDeep reinforcement learning has shown impressive resultsin simulated environments (MKS+15; MBM+). However,current approaches often fail when rewards are sparse asthe cost of random exploration increases rapidly with thedistance to rewards. This substantially restricts the scope ofpossible real-world applications, e.g. rewards in robotics areoften assigned only once a task is completed and thus arebinary and sparse (AWR+17; VHS+). Additionally, randomexploration potentially leads to safety issues.

One way to improve the efficiency of exploration is to uti-lize expert data. The simplest technique, behavioral cloning,often suffers from compounding errors when drifting awayfrom supervisor’s demonstrations, see also Figure 7 in Sec-tion 6. This can be mitigated by iterative methods like DAg-ger (RGB) at the cost of a cumbersome data collection pro-cess. Recent work (HVP+) has analyzed performance ofbehavioral cloning on Atari 2600 games. In the case of Mon-tezuma’s Revenge, cloning of near-perfect human demon-strations (a score above 30,000 points) led to a policy whichscores merely 575 points.

We aim to address the shortcomings of cloning meth-ods by combining the ACKTR algorithm (WML+17) withsupervised learning from the expert data. Our method iseasy to understand and implement. The core of our con-tribution can be expressed by a single formula, see Equa-tion (1). The expert data is intended to guide the agent’sexploration. We evaluate our method on two environmentswith sparse rewards: Montezuma’s Revenge and a demand-ing MyWayHome maze from the ViZDoom suite 1. Interest-

1Please see videos of Montezuma’s Revenge gameplay

ingly, our method is able to achieve stronger performancethan the expert data, see Figure 3.

We describe our method in Section 3. In Section 4 wedescribe environments, training procedures, expert dataand neural architectures. In Section 5 we summarize ourexperiments. Section 6 contains conclusions and sugges-tions of further work. Code for experiments presented inthis paper is publicly available: https://github.com/ghostFaceKillah/expert.

2 Related workDeep reinforcement learning The celebrated paper(MKS+15) has sparked rapid development of deep rein-forcement learning. The DQN algorithm presented therefor the first time achieved (super)human performance inmany Atari games. Soon after, other algorithms were de-veloped e.g. A3C (MBM+) and TRPO (SLA+15). In thispaper we employ the Actor-Critic using Kronecker-FactoredTrust Region (ACKTR) algorithm (WML+17). ACKTR isa policy-gradient method that utilizes natural gradient tech-niques (Ama98) which have proved to be successful in in-creasing speed and stability of learning.

Expert data in reinforcement learning. A vast body ofliterature has been devoted to imitation learning. As thelearned policy influences the future states upon which it willbe evaluated, errors compound and make this method imprac-tical for tasks with long time horizon (RB) (e.g. behavioralcloning in Montezuma’s Revenge (HVP+) yielded lacklus-ter performance). The seminal work (RGB) describes theDAgger algorithm that alleviates this issue, however at thecost of the iterative expert data collection process. In ourmethod, we collect a fixed set of expert data, which is reusedin the whole experiment.

Value-based deep reinforcement learning methods havebeen combined with learning from demonstrations (HVP+;PPH+18). In these works, the learning algorithm is altered toensure that the expert data added to the replay buffer is usedefficiently. For example, no action is allowed to have a higher

(https://bit.ly/2wG8Zh5), exploiting an unreportedbug in Montezuma’s Revenge (https://bit.ly/2PC7cRi)and ViZDoom MyWayHome gameplay (https://bit.ly/2wMlxDL)

arX

iv:1

809.

0344

7v1

[cs

.LG

] 1

0 Se

p 20

18

Figure 1: Montezuma’s Revenge - four screenshots from various stages of a gameplay. The room on the right marks the beginningof the second world. The agent reaches this room after passing through all the rooms in the first world. For the map of the firstworld see https://atariage.com/2600/archives/strategy_MontezumasRevenge_Level1.html, forthe map of the second world see https://atariage.com/2600/archives/strategy_MontezumasRevenge_Level2.html.

Approach Meanscore

Maxscore

Trans.×106

Methods used

ExpAugAC (our method) 27,052 804,900 200 Expert loss term based on batches of prerecorded data

DQfD (HVP+) 4,740 - 200 750k batches of expert pretraining, 3 additional loss terms, prioritizing expert data replay

Behavioral cloning (HVP+) 575 - 24 –

Ape-X DQfD (PPH+18) 29,384 - 2,500 Methods from DQfD, temporal consistency loss, transformed Bellman operator

Playing hard YT (APB+18) 41,098 - 1,000 Auxiliary reward encouraging imitation of videos of expert gameplays

Learning MR from a SingleDemonstration (SC18)

- 74,500 - Decomposing task into a curriculum of shorter subtasks, assumes ability to set env. state

Unifying Count-BasedExploration (BSO+16)

3,439 6,600 100 Exploration-based auxiliary reward, mixing Double Q-learning target with MC return

Table 1: The state of the art for Montezuma’s Revenge.

Q-value than expert’s action, the n-step return loss is usedto propagate expert information and expert demonstrationsare sampled more often. A different approach, combiningDQN with expert data classification loss has been presentedin (LOB). However, authors do not test their method onMontezuma’s Revenge or ViZDoom. In our work, we onlyuse one expert imitation loss. Formulation of this loss ismotivated by the policy gradient theorem, see Formula (1).

Methods combining policy gradient algorithm with expertdemonstrations have been successfully applied to problemssuch as complex robotic manipulation (RKG+17) or dialoguemanagement (SBU+17). In the case of Atari environments, in(ZM18) expert data is used to pre-train policies that are laterpassed to a policy gradient algorithm. A substantial efforthas been put into integrating replay buffers into actor-criticalgorithms including (Waw09; WT13; WBH+).

Montezuma’s Revenge Significant research attention hasbeen dedicated to Montezuma’s Revenge as it is regarded

as a testing ground that challenges exploration abilitiesof a given learning algorithm. Table 1 summarizes the cur-

rent state of the art, including methods using expert data.Efforts have been made to tackle this environment by addingnatural language instructions (KSS17) (avg. 3,500 pts us-ing 60M transitions), extending model with intrinsic curios-ity rewards in (OBvdOM17) (avg. 3,700 pts using 150Mtransitions) or introducing hierarchy into the model (VOS+)(avg. 2600 pts using 1000M frames). Numbers in the paren-thesis represent scores and amount of training samples re-ported in these papers. Combining hierarchical learning andimitation learning has been demonstrated to be effective(LJA+18), although the work focuses only on the first screenof Montezuma’s Revenge.

Doom ViZDoom (KWR+) is a popular suite of reinforce-ment learning environments based on the first-person shootergame Doom. Impressive results have been obtained for vari-ous tasks in these environments. These tasks include in par-ticular navigation (SDK18; LPCS; PS17) and multi-playercombat (DK16; CL17; WT; KSGG16). In (ADKB18; JLK17;DSC+16; CLSS16) the ViZDoom suite was used as a bench-mark in other reinforcement learning experiments.

Curiosity-based intrinsic rewards have been used in(PAED17) to increase efficiency of exploration in ViZDoom.Authors of (PAED17) report that in the navigation taskMyWayHome (see Section 4 for a description) the standardactor-critic approach completely fails, while their methodsolves the task in 70% of the cases after using 10M transi-tions for training. Our algorithm always solves the task aftertraining on 5M transitions. To the best of our knowledge, ourmethod is the first one to use expert data in ViZDoom.

3 Expert-augmented ACKTRReinforcement learning is formalized in the framework ofMarkov decision processes (MDP). In this setting at eachtimestep t, the agent observes a state st ∈ S and chooses anaction at ∈ A according to policy π, which is a (potentiallystochastic) mapping from a given state to actions.

The environment then returns reward r(st, at) and evolvesto the next state st+1 according to a transition probabilityP (st+1|st, at). The goal of reinforcement learning is to finda policy within some class which maximizes

J (π) = Eπ[Rt],

where

Rt =

∞∑i=0

γir(st+i, at+i), γ ∈ (0, 1),

is the future discounted reward. In our applications the set ofavailable neural network policies is parametrized by θ. Policy-based methods (SML+15; SMSM99; Wil92) are inspired bythe Policy Gradient Theorem which states that

∇θJ (πθ) = Eπθ

[∑t

advt∇θ (log πθ(at|st))

],

whereadvt = Rt − b(st)

and b is an arbitrary function of the state, called a baselinefunction. In the context of A2C and ACKTR algorithms wetake the approximate state-value function Vθ as a baseline.Thus we aim to optimize the following loss

LA2C(πθ) =∑t

LA2Ct (πθ),

where

LA2Ct (πθ) = Eπθ

[−advt log πθ(at|st) +

1

2(Rt − Vθ(st))2

].

The main algorithmic contribution of our paper consists inadding an extra term to the loss function to accommodate theexpert data:

L(πθ) = LA2C(πθ)−λexpert

k

k∑i=1

advexperti log πθ(a

experti |sexpert

i ),

(1)where aexpert

i and sexperti are sampled from a fixed dataset of

expert rollouts.

We consider three variants of the expert advantage:

reward: advexpertt =

∑s≥0

γsrexpertt+s ,

critic: advexpertt =

∑s≥0

γsrexpertt+s − Vθ(st)

+

,

simple: advexpertt = 1.

where [x]+ = max(x, 0). We follow the ACKTR algorithmwhich estimates the natural gradient direction associated withthe current policy, see Algorithm 1 for additional details.

4 Environments, training and architectures

Doom MyWayHome is a navigation task defined in theViZDoom suite (KWR+). In MyWayHome the maze consistsof 9 rooms, see Figure 5.

The agent receives reward 1 if it reaches the goal locationin less than 2100 time steps. There are not other rewards inthe game. The goal is always located in the same place andthe agent always starts in a fixed location far from the goal.The action space consists of three actions: forward, turn left,turn right.

Figure 5: The map of the MyWayHome maze. The startingposition and the goal are always located in the same place.See Figure 6 for a screenshot from the environment.

conv conv conv fc

s

Vθ(s)

πθ(s)

Figure 2: Network architecture: stacked input frames are passed through a convolution layer with 32 filters size of 8 and stride 4,followed by 64 convolution filters of size 4 and stride 2, followed by 64 convolution filters of size 3 and stride 1. The output is fedinto a fully connected layer with 512 units. Next, two heads are attached. The first one with a single neuron for the value functionapproximation and the second one with the number of neurons matching the dimension of the action space. After applying thesoftmax function the second head outputs the policy. In all layers, except the top heads, we use the ReLU non-linearity.

Figure 6: An observation from the MyWayHome environment.The goal location is visible in the background in the nextroom.

Montezuma’s Revenge Montezuma’s Revenge belongsto the suite of Atari 2600 games (BNVB13). The taskin the game is to navigate in a labyrinth (see Figure1), collect valuable items and avoid deadly obstacles.Montezuma’s Revenge is known for its sparse rewards. Forexample, obtaining the only reward available in the first room(see the leftmost image in Figure 1) requires five “macroactions” such as “jump across a gap” or “go down a ladder”,each spanning over dozens of environment steps.

Training details Training details are shared acrossMontezuma’s Revenge and ViZDoom’s MyWayHome envi-ronments. The input to the policy consists of the stacked lastfour frames which are downscaled to 84 × 84 pixels andcast to grayscale. We use Algorithm 1 with the learning rate0.125, 32 actors and the time horizon T = 20.

For the expert data we use k = 256 (see Equation 1).Following (MBM+), given entropy H we introduce anadditional entropy regularization term: βentropy defined as∇θH(πθ(·|st)). We set βentropy = 0.001.

Our implementation of Algorithm 1 is based on the imple-mentation of the ACKTR algorithm provided in (DHK+17).

Data: Dataset of expert transitions(sexpertt , aexpert

t , sexpertt+1 , rexpert

t )Initialize neural net policy πθfor iteration← 1 to max steps do

for t← 1 to T doPerform action at according to πθ(a|st)Receive reward rt and new state st+1

endfor t← 1 to T do

Compute discounted future rewardestimator: Rt = rt + γrt+1 + . . .+γT−t+1rT−1 + γT−tVθ(sT )

Compute advantage estimator:advt = Rt − Vθ(st)

endCompute∇ of the A2C loss gA2C =

∇θ1T

∑Tt=1

[−advt log πθ(at|st) + 1

2(Rt − Vθ(st))2

]Sample mini-batch of k expert state-actionpairs

Compute expert advantage estimatoradvexpert

t for each state-action pairCompute∇ of the expert loss gexpert =

∇θ

∑ki=1

[−advexpert

i log πθ(aexperti |sexpert

i )]

Update ACKTR inverse Fisher estimateUse ACKTR Kronecker optimizer withgradient g = gA2C + λexpertgexpert.

endAlgorithm 1: Expert-augmented ACTKR. Pseudocode forone actor.

(a) (b)

(c) (d)

(e) (f)

Figure 3: Montezuma’s Revenge experimental results. See the top left plot for a comparison between various advantage estimatorsdefined in Section 3. In experiments depicted in the top right plot we vary the discount rate γ. The sweet spot is around γ = 0.995.We conjecture that even higher values could be beneficial, but would require better handling of the variance of empirical rewards.Plot in the center-left shows that smaller amount of expert data yields higher variance of results. In the bottom-left plot we testvarious choices of λexpert to assess utilization of the expert data. The performance peaks for the intermediate values. For smallvalues expert’s learning signal is too weak, while high values impair the final performance because the agent overfits to theexpert data, as seen in the train set accuracy plot (center-right). In the experiments presented in the bottom-right plot we providescores for optimized parameters, that is we apply γ = 0.995, the critic advantage estimator and selected values of λexpert.

Figure 4: Left: In ViZDoom behavior is similar for all expert advantage estimators. Right: performance of a curriculum learningin MyWayHome compared to our expert-augmented algorithm. The curriculum consists in re-spawning the agent in randomlocations. We experimentally verified that the ACKTR algorithm without the curriculum was unable to solve the MyWayHome task.This echoes observations made for another actor-critic model-free algorithm in (PAED17). Our behavioral cloning experimentsalso failed to solve this task. The curiosity-based method described in (PAED17) achieves an average score 0.7 after 10M offrames.

5 ExperimentsIn this Section we empirically verify a number of propertiesof our algorithm. In particular (see Figure 3 for a summary):

1. we check which of proposed advantage estimators offersthe best performance,

2. we check how the discount factor γ influences the rewardand what are consequences of decreasing the number ofexpert trajectories,

3. we verify what is the potential for generalization: in thecase of Montezuma’s Revenge we provided expert trajec-tories only from the first world of Montezuma’s Revenge,but our agents are capable of continuing the gameplayin the second world. For the map of the first worldsee https://atariage.com/2600/archives/strategy_MontezumasRevenge_Level1.html,for the map of the second world see https://atariage.com/2600/archives/strategy_MontezumasRevenge_Level2.html

4. we also grid the λexpert coefficient with respect to the totalreward and verify how λexpert is related to the accuracy onexpert data.

Montezuma’s RevengeFor a summary of the state of the art compared with ourresults see the Table on page 2, for evaluation videossee https://bit.ly/2wG8Zh5. Interestingly, our al-gorithm discovered a bug, see video https://bit.ly/2PC7cRi, which manifests through scores exceeding800, 000 in some evaluation rollouts (these rollouts are ex-cluded when we calculate the mean score). In all experimentsreported in this Section we performed at least 5 experimentswith different seeds. Shades represent min-max ranges. In allexperiments except for Figure 3 top-right and bottom-right,we set γ = 0.99; in the top-left, top-right and center-leftexperiments we set λexpert = 1; in all experiments exceptfor Figure 3 top-left we use the critic advantage estimatorand in all experiments except for Figure 3 center-left we use

all available expert trajectories (14 trajectories). In the ex-periment presented in the bottom-right we selected the bestperforming parameters from previous experiments, that is thecritic advantage estimator and γ = 0.995. We also set the ad-vantage estimator for expert-data λexpert equal to 0.125, 0.25or 0.5.

In order to address the question of generalizationfrom the first Montezuma’s Revenge world to the secondMontezuma’s Revenge world (see the third point at the be-ginning of this Section), we considered 6 runs for the bestparameters (λexpert = 0.125, γ = 0.995). Please note that allprovided expert trajectories were limited to the first world.Let us choose a training run which achieves the median scoreat the end of training. The stochastic policy associated withthis run receives average reward 27, 052 in 100 evaluations. Itcompletes the first world in 60% of the cases and outperformsthe best expert trajectory in 16% of the cases. In contrast, ifwe choose the policy not from the median, but from thebest run, it resolves the first world in 88% of the cases andoutperforms the best expert trajectory in 86% of the cases.Interestingly, in some rollouts (not taken into the average) ourpolicies exploit an unreported bug in Montezuma’s Revengeoccasionally scoring over 100, 000 points with a maximumof 804, 900 points.

ViZDoom MyWayHome

In Figure 4 we summarize an experiment which comparesthe efficiency of the model-free ACKTR algorithm with ourexpert-augmented method. The model-free approach withoutthe expert data does not work. However, the agent learns howto navigate the maze after a relatively mild simplification ofthe environment consisting in allowing the agent to re-spawnin random locations in the labyrinth, which can be seen as acurriculum learning. The agent manages to learn a consistentbehavior in 100M actions. However, if we allow expert data,the agent reaches even a better performance already after 5Mactions without any curriculum, see Figure 4.

6 Conclusions and future workBased on experimental results we claim that the algorithmpresented in this work is a practical method of getting goodperformance in cases when multiple interactions with theenvironment are possible and good quality expert data isavailable. It could be particularly useful in settings such asMontezuma’s Revenge, where neither supervised learningfrom expert data nor random exploration yield good results.

Using model-free reinforcement learning algorithms withexpert data brings a number of issues which are worthaddressing. In particular, our expert-augmented algorithmwould benefit from an adaptive λexpert coefficient and accu-racy rate targeting on expert trajectories (from our experi-ments follows that for overall performance of the algorithm itwould be most beneficial to keep accuracy between 0.6 and0.9, see Figure 3). In our view, another interesting researchdirection would be to design a proxy metric which wouldidentify situations requiring additional expert data. That issituations where we are far away from the expert trajectoryand simultaneously the model-free algorithm does not have acapacity to find any rewards on its own, see Figure 7. One canregard this as an automation of the DAgger algorithm (RGB).

Figure 7: Left: our agent in Pitfall! trained on an ex-pert trajectory which achieves the perfect score (see arun https://youtu.be/MX_e2UZLk1E of our agent).The expert trajectory and our agent run only above theground. Right: the agent falls into the lower level and isunable to return to the expert trajectory, see video https://youtu.be/07327Eh6zIM .

We also leave as future work the following other extensionsof the experiments presented in this article:

1. Measure performance of the algorithm when the quality ofexpert data varies.

2. Rewards in robotics are often assigned only once a taskis completed and thus are binary and sparse (AWR+17;VHS+). In our view it would be interesting to test methodspresented in this paper on robotic object manipulationtasks or in a simulated car environment.

3. Authors of (PPH+18, Table 3) report Bank Heist, Gravitar,Ms. Pacman, Pitfall! and Solaris as games where humanexperts are still stronger than AI even when expert trajecto-ries are available in the process of training. In an on-going

experiment, we assess the performance of our method inthese environments.

4. Using human knowledge in tasks routine for humans butdifficult for artificial intelligence, such as driving, seemsto be a particularly important application of methods pre-sented in this paper. We test these ideas in an on-goingexperiment with the Carla simulator (DRC+17).

5. In our experiments, the discount factor γ = 0.995 per-formed optimally, see Figure 3. Can we get further tohigher values of γ?

References[ADKB18] Artemij Amiranashvili, Alexey Dosovitskiy,Vladlen Koltun, and Thomas Brox. TD or not TD: Analyz-ing the role of temporal differencing in deep reinforcementlearning. In ICLR, 2018.

[Ama98] Shun-Ichi Amari. Natural gradient works efficientlyin learning. Neural computation, 10(2):251–276, 1998.

[APB+18] Y. Aytar, T. Pfaff, D. Budden, T. Le Paine,Z. Wang, and N. de Freitas. Playing hard exploration gamesby watching YouTube. CoRR, abs/1805.11592, 2018.

[AWR+17] Marcin Andrychowicz, Filip Wolski, Alex Ray,Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew,Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba.Hindsight experience replay. In NIPS, 2017.

[BNVB13] M. G. Bellemare, Y. Naddaf, J. Veness, andM. Bowling. The Arcade Learning Environment: An evalua-tion platform for general agents. JAIR, 47, 2013.

[BSO+16] Marc G. Bellemare, Sriram Srinivasan, Georg Os-trovski, Tom Schaul, David Saxton, and Rémi Munos. Uni-fying count-based exploration and intrinsic motivation. InNIPS 2016, pages 1471–1479, 2016.

[CL17] Devendra Singh Chaplot and Guillaume Lample.Arnold: An autonomous agent to play FPS games. In AAAI2017, pages 5085–5086, 2017.

[CLSS16] Devendra Singh Chaplot, Guillaume Lample, Kan-thashree Mysore Sathyendra, and Ruslan Salakhutdinov.Transfer deep reinforcement learning in 3D environments:An empirical study. In NIPS Deep RL Workshop, 2016.

[DHK+17] Prafulla Dhariwal, Christopher Hesse, OlegKlimov, Alex Nichol, Matthias Plappert, Alec Radford, JohnSchulman, Szymon Sidor, and Yuhuai Wu. OpenAI baselines.https://github.com/openai/baselines, 2017.

[DK16] Alexey Dosovitskiy and Vladlen Koltun. Learning toact by predicting the future. CoRR, abs/1611.01779, 2016.

[DRC+17] Alexey Dosovitskiy, Germán Ros, Felipe Codev-illa, Antonio López, and Vladlen Koltun. CARLA: an openurban driving simulator. In CoRL 2017, pages 1–16, 2017.

[DSC+16] Yan Duan, John Schulman, Xi Chen, Peter L.Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast re-inforcement learning via slow reinforcement learning. CoRR,abs/1611.02779, 2016.

[HVP+] Todd Hester, Matej Vecerik, Olivier Pietquin, MarcLanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan,Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John

Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI 2018.

[JLK17] Peter H. Jin, Sergey Levine, and Kurt Keutzer. Re-gret minimization for partially observable deep reinforcementlearning. CoRR, abs/1710.11424, 2017.

[KSGG16] Tejas D. Kulkarni, Ardavan Saeedi, Simanta Gau-tam, and Samuel J. Gershman. Deep successor reinforcementlearning. CoRR, abs/1606.02396, 2016.

[KSS17] Russell Kaplan, Christopher Sauer, and AlexanderSosa. Beating Atari with natural language guided reinforce-ment learning. CoRR, abs/1704.05539, 2017.

[KWR+] M. Kempka, M. Wydmuch, G. Runc, J. Toczek, andW. Jaskowski. Vizdoom: A doom-based ai research platformfor visual reinforcement learning. In CIG 2016.

[LJA+18] Hoang Minh Le, Nan Jiang, Alekh Agarwal,Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hierar-chical imitation and reinforcement learning. In ICML 2018,pages 2923–2932, 2018.

[LOB] Aravind S. Lakshminarayanan, Sherjil Ozair, andYoshua Bengio. Reinforcement learning with few expertdemonstrations. In NIPS 2016 Workshops: Deep Learningfor Action and Interaction.

[LPCS] Lisa Lee, Emilio Parisotto, Devendra Singh Chaplot,and Ruslan Salakhutdinov. LSTM iteration networks: Anexploration of differentiable path finding. In ICLR 2018Workshop.

[MBM+] Volodymyr Mnih, Adria Puigdomenech Badia,Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley,David Silver, and Koray Kavukcuoglu. Asynchronous meth-ods for deep reinforcement learning. In ICML 2016.

[MKS+15] Volodymyr Mnih, Koray Kavukcuoglu, David Sil-ver, Andrei A Rusu, Joel Veness, Marc G Bellemare, AlexGraves, Martin Riedmiller, Andreas K Fidjeland, Georg Os-trovski, et al. Human-level control through deep reinforce-ment learning. Nature, 518(7540):529, 2015.

[OBvdOM17] Georg Ostrovski, Marc G. Bellemare, Aäronvan den Oord, and Rémi Munos. Count-based explorationwith neural density models. CoRR, abs/1703.01310, 2017.

[PAED17] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros,and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. CoRR, abs/1705.05363, 2017.

[PPH+18] T. Pohlen, B. Piot, T. Hester, M. Gheshlaghi Azar,D. Horgan, D. Budden, G. Barth-Maron, H. van Hasselt,J. Quan, M. Vecerík, M. Hessel, R. Munos, and O. Pietquin.Observe and look further: Achieving consistent performanceon Atari. CoRR, abs/1805.11593, 2018.

[PS17] Emilio Parisotto and Ruslan Salakhutdinov. Neuralmap: Structured memory for deep reinforcement learning.CoRR, abs/1702.08360, 2017.

[RB] Stéphane Ross and Drew Bagnell. Efficient reductionsfor imitation learning. In AISTATS 2010.

[RGB] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell.A reduction of imitation learning and structured prediction tono-regret online learning. In AISTATS 2011.

[RKG+17] A. Rajeswaran, V. Kumar, A. Gupta, J. Schulman,E. Todorov, and S. Levine. Learning complex dexterousmanipulation with deep reinforcement learning and demon-strations. arXiv:1709.10087, 2017.

[SBU+17] Pei-Hao Su, Pawel Budzianowski, Stefan Ultes,Milica Gasic, and Steve J. Young. Sample-efficient actor-critic reinforcement learning with supervised data for dia-logue management. CoRR, abs/1707.00130, 2017.

[SC18] T. Salimans and R. Chen. Learning montezuma’srevenge from a single demonstration, OpenAI Blog, 2018.

[SDK18] Nikolay Savinov, Alexey Dosovitskiy, and VladlenKoltun. Semi-parametric topological memory for navigation.CoRR, abs/1803.00653, 2018.

[SLA+15] John Schulman, Sergey Levine, Pieter Abbeel,Michael I. Jordan, and Philipp Moritz. Trust region policyoptimization. In ICML, 2015, pages 1889–1897, 2015.

[SML+15] John Schulman, Philipp Moritz, Sergey Levine,Michael I. Jordan, and Pieter Abbeel. High-dimensionalcontinuous control using generalized advantage estimation.CoRR, abs/1506.02438, 2015.

[SMSM99] Richard S. Sutton, David A. McAllester, Satin-der P. Singh, and Yishay Mansour. Policy gradient methodsfor reinforcement learning with function approximation. InNIPS, pages 1057–1063, 1999.

[VHS+] M. Vecerík, T. Hester, J. Scholz, F. Wang,O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, andM. Riedmiller. Leveraging demonstrations for deep rein-forcement learning on robotics problems with sparse rewards.arXiv:1707.08817.

[VOS+] A. Vezhnevets, S. Osindero, T. Schaul, N. Heess,M. Jaderberg, D. Silver, and K. Kavukcuoglu. Feu-dal networks for hierarchical reinforcement learning.arXiv:1703.01161.

[Waw09] Pawel Wawrzynski. Real-time reinforcement learn-ing by sequential actor-critics and experience replay. NeuralNetworks, 22(10):1484–1497, 2009.

[WBH+] Ziyu Wang, Victor Bapst, Nicolas Heess,Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu,and Nando de Freitas. Sample efficient actor-critic withexperience replay. In ICLR 2017.

[Wil92] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learn-ing. Machine learning, 8(3-4):229–256, 1992.

[WML+17] Yuhuai Wu, Elman Mansimov, Shun Liao,Roger B. Grosse, and Jimmy Ba. Scalable trust-regionmethod for deep reinforcement learning using kronecker-factored approximation. CoRR, abs/1708.05144, 2017.

[WT] Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic curriculum learning. InICLR 2017.

[WT13] Paweł Wawrzynski and Ajay Kumar Tanwani. Au-tonomous reinforcement learning with experience replay.Neural Networks, 41:156 – 167, 2013.

[ZM18] Xiaoqin Zhang and Huimin Ma. Pretraining deepactor-critic reinforcement learning algorithms with expertdemonstrations. CoRR, abs/1801.10459, 2018.