Bridging the Gap: Providing Post-Hoc Symbolic Explanations for SequentialDecision-Making Problems with Inscrutable Representations

Sarath Sreedharan 1 Utkarsh Soni 1 Mudit Verma 1 Siddharth Srivastava 1 Subbarao Kambhampati 1

AbstractAs increasingly complex AI systems are intro-duced into our daily lives, it becomes importantfor such systems to be capable of explaining therationale for their decisions and allowing usersto contest these decisions. A significant hurdleto allowing for such explanatory dialogue couldbe the vocabulary mismatch between the user andthe AI system. This paper introduces methodsfor providing contrastive explanations in termsof user-specified concepts for sequential decision-making settings where the system’s model of thetask may be best represented as a blackbox sim-ulator. We do this by building partial symbolicmodels of the task that can be leveraged to an-swer the user queries. We empirically test thesemethods on a popular Atari game (Montezuma’sRevenge) and modified versions of Sokoban (awell known planning benchmark) and report theresults of user studies to evaluate whether peoplefind explanations generated in this form useful.

1. IntroductionRecent successes in AI have brought the field a lot of at-tention, and there is a lot of excitement towards deployingAI-based tools for solving various challenges faced in ourdaily lives. For these systems to be truly effective in the realworld, they need to be capable of working with a lay enduser. This means not just inferring optimal decisions, butalso being able to allow users to raise explanatory querieswherein they can contest the system’s decisions. An obsta-cle to providing explanations to such questions is the factthat the systems may not have a shared vocabulary with itsend users or have an explicit interpretable model of the task.More often than not, the system may be reasoning about thetask in a high-dimensional space that is opaque to even the

1School of Computing, Informatics, and Decision SystemsEngineering, Arizona State University, Tempe, AZ 85281 USA.Correspondence to: Sarath Sreedharan <ssreedh3@asu.edu>.

2020 ICML Workshop on Human in the Loop Learning (HILL2020). Copyright 2020 by the author(s).

developers of the system, let alone a lay user.

While there is a growing consensus within the explainableAI community that end-user explanations need to be framedin terms of user understandable concepts, the focus generallyhas been on introducing such methods for explaining one-shot decisions such as in the case of classifiers (c.f. (Kimet al., 2018; Ribeiro et al., 2016)). This is unfortunate asexplaining sequential decision-making problems presentchallenges that are mostly absent from the one-shot decision-making scenarios. In these problems, we not only have todeal with possible interrelationship between the actions inthe sequence, but may also need to explain conditions forthe executability of actions and the cost of executing certainaction sequences. Effectively, this means that explaining aplan or policy to a user would require the system to explainthe details of the domain (or at least the agent’s belief of it)in terms they can understand.

Barring a few exceptions in summarizing policies like(Hayes & Shah, 2017), most work in explaining sequentialdecision-making problems have thus used a model specifiedin a shared vocabulary as a starting point for explanation(Chakraborti et al., 2020). Our work aims to correct thisby developing methods that are able to field some of themost fundamental explanatory queries identified in the lit-erature, namely contrastive queries, i.e., questions of theform ‘why P (the decision proposed by the system) andnot Q (the alternative proposed by the user or the foil)?(Miller, 2018), in user-understandable terms. Our methodsachieve this by building partial and abstract symbolic mod-els (Section 2) expressed in terms of the user’s vocabularythat approximate the task details relevant to the specificquery raised by the user. Specifically, we will focus ondeterministic tasks where the system has access to a tasksimulator and we will identify (a) missing preconditions toexplain scenarios where the foil raised by the user resultsin an execution failure of an action and (b) cost functionapproximations to explain cases where the foil is executablebut suboptimal (Section 3). We learn such models by in-teracting with the simulator (on randomly sampled states)while using learned classifiers that detect the presence ofuser-specified concepts in the simulator states. Figure 1presents the overall flow of this process with illustrative

Bridging the Gap

Figure 1. The explanatory dialogue starts when the user presentsthe system with a specific alternate plan (foil). Here we considertwo foils, one that is invalid and another that is costlier than theplan. The system explains the invalid plan by pointing out an actionprecondition that was not met in the plan, while it explains the foilsuboptimality by informing the user about cost function. Each ofthese model information is expressed in terms of concepts specifiedby the user which we operationalize by learning a classifier foreach concept.

explanations in the context of a slightly updated version ofMontezuma’s Revenge (Wikipedia contributors, 2019). Ourmethods also allow for the calculation of confidence overthe explanations and explicitly take into account the factthat learned classifiers for user-specified concepts may benoisy. This ability to quantify its belief about the correctnessof explanation is an important capability for any post-hocexplanation system that may influence the user’s decisions.We evaluate the system on two popular sequential decisionmaking domains, Montezuma’s Revenge and a modifiedversion of Sokoban (Botea et al., 2002) (a game involvingplayers pushing boxes to specified targets). We present userstudy results that show the effectiveness of explanationsstudied in this paper (Section 5).

2. BackgroundIn this work, we focus on cases where a human observer istrying to make sense of plans proposed by an autonomoussystem. When the plan differs from the users’ expecta-tions, they try to make sense of this disparity by askingwhy some alternate expected plan was not proposed. Thegoal then becomes addressing such counterfactual queriesin terms of the dynamics of the domain in question, ex-pressed using user-specified concepts. We assume that thedecision-making algorithm computes the optimal solution,and thus our focus isn’t on how the algorithm came upwith the specific decisions, but only on why this action se-quence was chosen instead of an alternative that the userexpected. We assume access to a deterministic simulator

of the formMsim = 〈S,A, T, C〉, where S represents theset of possible world states, A the set of actions and T thetransition function that specifies the problem dynamics. Thetransition function is defined as T : S × A → S ∪ {⊥},where ⊥ corresponds to an invalid absorber-state gener-ated by the execution of an infeasible action. Invalid statecould be used to capture failure states that could occur whenthe agent violates hard constraints like safety constraints.Finally, C : S × A → R captures the cost function ofexecuting an action at a particular state (with cost of aninfeasible action taken to be infinite). We will overloadthe transition function T to also work on action sequence,i.e., T (s, 〈a1, ..., ak〉) = T (...T (T (s, a1), a2), ..., ak). Wewill look at goal-directed problems in the sense that thedecision-making system needs to come up with the plan,i.e., a sequence of actions, π = 〈a1, .., ak〉, that will drivethe state of the world to a goal state. In general we willuse the tuple Πsim = 〈I,G,Msim〉 to represent the decisionmaking problem, where I is the initial state and G the setof goal states. Moreover a plan is optimal if it achieves thegoal and there exists no cheaper plan that can achieve thegoal (where C(I, π) is the total cost of executing π).

We will use symbolic action models with preconditionsand cost functions (similar to STRIPS models (Geffner &Bonet, 2013)) as a way to approximate the problem for ex-planations. Such a model can be represented by the tupleΠS = 〈FS , AS , IS , GS , CS〉, where FS is a set of propo-sitional fluents defining the state space, AS is the set ofactions, IS is the initial state, GS is the goal specification.Each valid problem state in the problem is uniquely iden-tified by the subset of fluents that are true in that state (sofor any state s, s ⊆ FS). We will use the notation SMS todenote the space of all possible states for the modelMS .Each action a ∈ AS is further described in terms of thepreconditions preca (specification of states in which a isexecutable) and the effects of executing the action. We willdenote the state formed by executing action a in state s asa(s). We will focus on models where the preconditions arerepresented as a conjunction of state factors. If the actionis executed in a state with missing preconditions, then theexecution results in the invalid state (⊥). Unlike standardSTRIPS models, where the cost of executing action is inde-pendent of states, we will be using a state dependent costfunction of the form CS : 2F ×AS → R to capture the costof valid action executions. Internally, such state models maybe represented using conditional cost models of the typediscussed in (Geißer et al., 2016). In this paper, we won’ttry to reconstruct the exact cost function but will rather tryto estimate an abstract version of the cost function.

Bridging the Gap

3. Contrastive ExplanationsThe specific explanatory setting, illustrated in Figure 1, thatwe are interested in studying involves a decision-makingproblem specified by the tuple Πsim = 〈I,G,Msim〉 forwhich the system identifies a plan π. When presented withthe plan, the user of the system may either accept it orresponds by raising an alternative plan πf (the foil) thatthey believe should be followed instead. Now the systemwould need to provide an explanation as to why the plan πmay be preferred over the foil πf in question. The only twopossibilities here are that either the foil is inexecutable andhence can not be followed or it is costlier than the plan inquestion.1 More formally,

Definition 1 The plan π is said to be preferred over a foilπf for a problem Πsim = 〈I,G,Msim〉, if either of thefollowing conditions are met, i.e.,

1. πf is inexecutable, which means, either (a) T (I, πf ) 6∈G, i.e the action sequence doesn’t lead to a possiblegoal state, or (b) the execution of the plan leads to aninvalid state, i.e., T (I, πf ) = ⊥.

2. Or πf is costlier than π, i.e., C(I, π) < C(I, πf )

To concretize this interaction, consider an instance from amodified version of Montezuma’s revenge (Figure 1). Let’sassume the agent starts from the highest platform, and thegoal is to get to the key. The specified plan π may require theagent to make its way to the lowest level, jump over the skull,and then go to the key with a total cost of 20. Let us considera case where the user raises two possible foils that are quitesimilar to π, but, (a) in the first foil, instead of jumping theagent just moves left (as in it tries to move through the skull)and (b) in the second, instead of jumping over the skull, theagent performs the attack action (not part of the originalgame, but added here for illustrative purposes) and thenmoves on to the key. Now using the simulator, the systemcould tell that in the first case, moving left would lead toan invalid state and in the second case, the foil is moreexpensive. It may however struggle to explain to the userwhat particular aspects of the state or state sequence lead tothe invalidity or suboptimality. Even efforts to localize partsof its own internal state representation for possible reasonsby comparing the foil with similar states where actions areexecutable or cheaper may be futile, as even what constitutessimilar states as per the simulator may be conceptually quiteconfusing for the user. This scenario thus necessitates theuse of methods that are able to express possible explanationsin terms that the user may understand.

Representational Assumptions A quick note on some of

1If the foil is as good as the original plan, then the system couldswitch to foil without loss of optimality.

the representational assumptions we are making. The cen-tral one we are making is of course that it is possible toapproximate the applicability of actions and cost functionin terms of high-level concepts. Apart from the intuitiveappeal of such models (many of these models have their ori-gin in models from folk psychology), these representationschemes have been widely used to model real-world sequen-tial decision-making problems from a variety of domainsand have a clear real-world utility (Benton et al., 2019).We agree that there may be problems where it may not bedirectly applicable, but we believe this is a sound initialstep and applicable to many domains where currently Rein-forcement Learning (RL) based decision-making systemsare being successfully used, including robotics and games.

Apart from this basic assumption, we make one additionalrepresentational assumption, namely, that the preconditioncan be expressed as a conjunction of positive concepts. Notethat the assumption doesn’t restricts the applicability of themethods discussed here. Our framework can still cover caseswhere the action may require non-conjunctive preconditions.To see why, consider a case where the precondition of actiona is expressed as an arbitrary propositional formula, φ(C).In this case, we can express it in its conjunctive normalform φ′(C). Now each clause in φ′(C) can be treated as anew compound positive concept. Thus we can cover sucharbitrary propositional formulas by expanding our conceptlist with compound concepts (including negations and dis-juncts) whose value is determined from the classifiers forthe corresponding atomic concepts.

Concept maps: To describe explanations in this setting, thesystem needs to have access to a mapping from it’s inter-nal state representation to a set of high-level concepts thatare known to the user (for Montezuma this could involveconcepts like agent being on ladder, holding onto key, beingnext to a skull etc.). We will assume each concept corre-sponds to a propositional fact that the user associates withthe task’s states and believes that the dynamics of the taskare determined by these concepts. This means that as perthe user, for each given state, a subset of these concepts maybe either present or absent. Our method assumes that wehave access to binary classifiers for each concept that maybe of interest to the user. The classifiers provide us with away to convert simulator states to a factored representation.Such techniques have not only been used in explanation(c.f. (Kim et al., 2018; Hayes & Shah, 2017)) but also inworks that have looked at learning high-level representa-tions for continuous state-space problems (c.f. (Konidariset al., 2018)). Let C be the set of classifiers correspondingto the high-level concepts. For state s ∈ S, we will over-load the notation C and specify the concepts that are trueas C(s), i.e., C(s) = {ci|ci ∈ C ∧ ci(s) = 1} (where ciis the classifier corresponding to the ith concept, we willoverload this notation and also use it to stand for the label of

Bridging the Gap

the ith concept). The user could specify the set of conceptsby identifying positive and negative example states for eachconcept. These examples could then be used to learn therequired classifiers by using algorithms best suited for theinternal simulator state representation. This means that theexplanatory system should have some method of exposingsimulator states to the user. A common way to satisfy thisrequirement would be by having access to visual representa-tions for the states. The simulator state itself doesn’t needto be an image as long as we have a way to visualize it (forexample in Atari games where the states can be representedby the RAM state of the game controller but we can stillvisualize it). The concept list can also be mined from quali-tative descriptions of the domain and we can crowd sourcethe collection of example states for each concept.

Explanation using concepts: To explain why a given foilis not preferred over the specified plan, we will present infor-mation about the symbolic model expressed in user’s vocab-ulary, MC

S = 〈C, ACS ,C(I),C(G), CCS 〉. Where C(G) =⋂

sg∈G C(sg) and ACS contains a definition for each action

a ∈ A. The model is a sound abstraction of the simulatorMsim = 〈S, T,A, C〉 for regions of interest S ⊆ S , in sofar as, ∀s ∈ S and ∀a ∈ A, we have an equivalent actionaC ∈ AC

S , such that aC(C(s)) = C(T (s, a)) (assumingC(⊥) = ⊥) and CCS (C(s), a) = C(s, a). Note that estab-lishing the preference of plan does not require informingthe users about the entire model, but rather only the relevantparts. For conciseness, we will use ai for both the simulatoraction and the corresponding abstract action in the symbolicmodel as long as the context allows it to be distinguished.

For establishing the invalidity of πf , we just need to focuson explaining the failure of the first failing action ai, i.e.,the last action in the shortest prefix that would lead to aninvalid state (which in our running example is the move-leftaction in the state presented in Figure 1 for the first foil).We can do so by informing the user that the failing actionhas an unmet precondition, as per the symbolic model, inthe state it was executed in. Formally

Definition 2 For a failing action ai for the foil πf =〈a1, .., ai, .., an〉, ci ∈ C is considered an explanation forfailure if ci ∈ precai \C(si), where si is the state where aiis meant to be executed (i.e si = T (I, 〈a1, .., ai−1〉)).

In our example for the invalid foil, a possible explanationwould be to inform the user that move-left can only beexecuted in states for which the concept skull-not-on-leftis true; and the concept is false in the given state. Thisformulation is enough to capture both conditions for foilinexecutability by appending an additional goal action atthe end of each sequence. The goal action causes the stateto transition to an end state and it fails for all states exceptthe ones in G. Our approach to identifying the minimalinformation needed to explain specific query follows from

studies in social sciences that have shown that selectivity orminimality is an essential property of effective explanations(Miller, 2018).

For explaining the suboptimality of the foil, we have toinform the user about CCS . To ensure minimality of explana-tions, rather than generating the entire cost function or eventrying to figure out individual conditional components ofthe function, we will instead try to learn an abstraction ofthe cost function Cabss , defined as follows

Definition 3 For the symbolic model MCS =

〈C, ACS ,C(I),C(G), CCS 〉, an abstract cost func-

tion CabsS : 2C × ACS → R is specified as follows

CabsS ({c1, .., ck}, a) = min{CCS(s, a)|s ∈ SMCS∧ {c1, .., ck} ⊆

s}].

Intuitively, CabsS ({c1, .., ck}, a) = k can be understood asstating that executing the action a, in the presence of con-cepts {c1, .., ck} costs at least k. We can use CabsS in anexplanation of the form

Definition 4 For a valid foil πf = 〈a1, .., ak〉, a plan π anda problem Πsim = 〈I,G,Msim〉, the sequence of conceptsets of the form Cπf

= 〈C1, ..., Ck〉 along with Cabss is con-sidered a valid explanation for relative suboptimality of thefoil (denoted as CabsS (Cπf

, πf ) > C(I, π)), if ∀Ci ∈ Cπf,

Ci is a subset of concepts presents in the correspondingstate (where state is I for i = 1 and T (I, 〈a1, ..., ai−1〉) fori > 1) and Σi={1..k}CabsS (Ci, ai) > C(I, π)

In the earlier example, the explanation would include thefact that executing the action attack in the presence of theconcept skull-on-left, will cost at least 500 (as opposed tooriginal plan cost of 20).

4. Identifying Explanations throughSample-Based Trials

For identifying the model parts for explanatory query, wewill rely on the agent’s ability to interact with the simulatorto build estimates. Given the fact that we can separate thetwo cases at the simulator level, we will keep the discussionof identifying each explanation type separate and only focuson identifying the model parts once we know the failuretype.

Identifying failing precondition: To identify the missingpreconditions, we will rely on the simple intuition thatwhile successful execution of an action a in the state sjwith a concept Ci doesn’t necessarily establish that Ci isa precondition, we can guarantee that any concept false inthat state can not be a precondition of that action. Thisis a common line of reasoning exploited by many of themodel learning methods (c.f (Carbonell & Gil, 1990; Stern

Bridging the Gap

& Juba, 2017)). So we start with the set of concepts thatare absent in the the state (sfail) where the failing action(afail) was executed, i.e., poss prec set = C \ C(sfail). Wethen randomly sample for states where afail is executable.Each new sampled state si where the action is executablecan then be used to update the possible precondition set asposs prec set = poss prec set ∩ C(si). That is, if a stateis identified where the action is executable but a conceptis absent then it can’t be part of the precondition. We willkeep repeating this sampling step until the sampling budgetis exhausted or if one of the following exit conditions ismet. (a) In cases where we are guaranteed that the con-cept list is exhaustive, we can quit as soon as the set ofpossibilities reduce to one (since there has to be a missingprecondition at the failure state). (b) The search results in anempty list. The list of concepts left at the end of exhaustingthe sampling budget represents the most likely candidatesfor preconditions. An empty list here signifies the fact thatwhatever concept is required to differentiate the failure statefrom the executable one is not present in the initial con-cept list (C). This can be taken as evidence to query theuser for more task-related concepts. Any locality considera-tions for sampled states, like focusing on states close to theplan/foil, can be baked into the sampler. The full specifica-tion of the algorithm is provided in the supplementary file(https://bit.ly/38TmDPA).

Identifying cost function: Now we will employ a similarsampling based method to identify the right cost functionabstraction. Unlike the precondition failure case, there is nosingle action we can choose but rather we need to choose alevel of abstraction for each action in the foil (though it maybe possible in many cases to explain the suboptimality offoil by only referrring to a subset of actions in the foil). Ourapproach here would be to find the most abstract representa-tion of the cost function at each step such that of the totalcost of the foil becomes greater than that of the specifiedplan. Thus for a foil πf = 〈a1, ..., ak〉 our objective become

minC1,...,CkΣi=1..k‖Ci‖ subject to Cabss (Cπf , πf ) > C(I, π)

For any given Ci, Cabss (Ci, ai) can be approximated bysampling states randomly and finding the minimum costof executing the action ai in states containing the conceptsCi. We can again rely on a sampling budget to decide howmany samples to check and enforce required locality withinsampler. Similar to the previous case, we can identify theinsufficiency of the concept set by the fact that we aren’tbe able to identify a valid explanation. The algorithm canbe found in the supplementary file (https://bit.ly/38TmDPA).

Confidence over explanations: Though both the methodsdiscussed above are guaranteed to identify the exact modelin the limit, the accuracy of the methods is still limited bypractical sampling budgets we can employ. So this meansit is important that we are able to establish some level ofconfidence in the solutions identified. To assess confidence,

Figure 2. A simplified probabilistic graphical models for explana-tion inference, Subfigure (A) and (B) assumes classifiers to becompletely correct, while (C) and (D) presents cases with noisyclassifier.

we will follow the probabilistic relationship between therandom variables as captured by Figure 2 (A) for precon-dition identification and Figure 2 (B) for cost calculation.Where the various random variables captures the followingfacts: Osa - indicates that action a can be executed in states, ci ∈ pa - concept ci is a precondition of a, Osci - the con-cept ci is present in state s, Cabss ({ci}, a) ≥ k - the abstractcost function is guaranteed to be higher than or equal to kand finally OC(s,a)≥k - stands for the fact that the actionexecution in the state resulted in cost higher than or equalto k. We will allow for inference over these models, byrelying on the following simplifying assumptions - (1) thedistribution of concepts over the state space is independentof each other, (2) the distribution of all non-preconditionconcepts in states where the action is executable is the sameas their overall distribution across the problem states (whichcan be empirically estimated), (3) cost distribution of anaction over states corresponding to a concept that does notaffect the cost function is identical to the overall distribu-tion of cost for the action (which can again be empiricallyestimated). The second assumption implies that you are aslikely to see a non-precondition concept in a sampled statewhere the action is executable as the concept was likely toappear at any sampled state (this distribution is denoted aspci ). While the third one implies that for a concept that hasno bearing on the cost function for an action, the likelihoodthat executing the action in a state where the concept ispresent will result in a cost greater than k will be the sameas that of the action execution resulting in cost greater thank for a randomly sampled state (pC(.,a)≥k).

For a single sample, the posterior probability of ex-planations for each case can be expressed as follows:For precondition estimation, updated posterior proba-bility for a positive observation can be computed asP (ci ∈ pa|Osci ∧O

sa) = (1− P (ci 6∈ pa|Osci ∧O

sa)), where

P (ci 6∈ pa|Osci ∧Osa) =

pci ∗ P (ci 6∈ pa)

P (Osci |Osa)

Bridging the Gap

and for the case of cost function approximation

P (Cabss ({ci}, a) ≥ k|Osci ∧OC(s,a)≥k) =

P (Cabss ({ci}, a) ≥ k)

P (Cabss ({ci}, a) ≥ k)) + pC(.,a)≥k ∗ P (¬Cabss ({ci}, a) ≥ k))

Full derivation of above formulas can be found in the sup-plementary file (https://bit.ly/38TmDPA). The dis-tribution used in the cost explanation, can either be limitedto distribution over states where action ai is executable orallow for the cost of executing an action in a state where itis not executable to be infinite.

Using noisy concept classifiers: Given how unlikely it isto have access to a perfect classifier for any concept, a morepractical assumption to adopt could be that we have accessto a noisy classifier. However, we assume that we also haveaccess to a probabilistic model for its prediction. That is, wehave access to a function PC : C→ [0, 1] that gives the prob-ability that the concept predicted by the classifier is actuallyassociated with the state. Such probability functions couldbe learned from the test set used for learning the classifier.Allowing for the possibility of noisy observation generallyhas a more significant impact on the precondition calcula-tion than the cost function approximation. Since we arerelying on just generating a lower bound for the cost func-tion, we can be on the safer side by under-approximating thecost observations received (though this could lead to largerthan required explanation). In the case of precondition esti-mation, we can no longer use a single failure (execution ofan action in a state where the concept is absent) as evidencefor discarding the concept. Though we can still use it asan evidence to update the probability of the given conceptbeing a precondition. We can remove a particular possibleprecondition from consideration once the probability of itnot being a precondition crosses a specified threshold.

To see how we can incorporate these probabilistic observa-tions into our confidence calculation, consider the updatedrelationships presented in Figure 2 (C) and (D) for precondi-tion and cost function approximation. Note that in previoussections, we made no distinction between the concept beingpart of the state and actually observing the concept. Now wewill differentiate between the classifier saying that a conceptis present (Osci) from the fact that the concept is part of thestate (ci ∈ C(S)). Now we can use this updated modelfor calculating the confidence. We can update the poste-rior of a concept not being a precondition given a negativeobservation (Os¬ci ) using the formula

P (ci 6∈ pa|Os¬ci ∧Osa) =

P (Os¬ci |ci 6∈ pa ∧Osa) ∗ P (ci 6∈ pa|Osa)

P (O¬ci |Osa)

Similarly we can modify the update for a positive obser-vation to include the observation model and also do thesame for the cost explanation. For calculation of cost con-fidence, we will now need to calculate P (OC(s,a)≥k|ci 6∈C(s), Cabss ({ci}, a) ≥ k). This can either be empirically cal-culated from samples with true label or we can assume that

Figure 3. Montezuma Foils: Left Image shows foils for screen 1,(A) Move right instead of Jump Right (B) Go left over the edgeinstead of using ladder (C) Go left instead of jumping over theskull. Right Image shows foil for screen 4, (D) Move Down insteadof waiting.

Figure 4. Sokoban Foils: Left Image shows foils for Sokoban-switch, note that the green cell will turn pink once the agent passesit. Right Image shows foil for screen Sokoban-cell.

this value is going to be approximately equal to the overalldistribution of the cost for the action.

The derivations for all of the above expressions and formulasfor the other cases can be found in the supplementary file.

5. EvaluationFor validating the soundness of the methods discussed be-fore, we tested the approach on the open-AI gym’s determin-istic implementation of Montezuma’s Revenge (Brockmanet al., 2016) for precondition identification and two modifiedversions of the gym implementation of Sokoban (Schrader,2018) for cost based foil evaluation. The first version ofSokoban included a switch the player could turn on to re-duce the cost of pushing the box (we will refer to this versionas Sokoban-switch) and the second version (Sokoban-cells)included particular cells from which it is costlier to pushthe box. We used RAM-based state representation for Mon-tezuma and images of game state for the Sokoban variations.To add richer preconditions to the settings, we added a wrap-per over the original simulators for all the games to render

Bridging the Gap

any action that fails to change the current agent state as anaction failure. For Montezuma, we selected four invalidfoils for the game, three from screen 1 and one from screen4 and for Sokoban, we selected one valid but suboptimal foilfor each variation. The specifics of the games, foils, plansand state representation are provided in the supplementaryfile (https://bit.ly/38TmDPA). The plan and foilsused can be found in Figure 3 (for Montezuma) and Figure4 (for Sokoban variants).

Concept learning: For Montezuma, we specified ten con-cepts for each screen and for the Sokoban variations weused a survey to collect the set of concepts. The surveyallowed participants to interact with the game through aweb interface, and at the end, they were asked to specifygame concepts that they thought were relevant for particularactions. For Sokoban-switch, we collected data from six par-ticipants and received 25 unique concepts and for Sokoban-cell we collected data from seven participants and received38 unique concepts. In both domains, we wrote scripts toidentify positive and negative examples for each conceptfrom randomly sampled states of the game. For Sokobanvariants, we rejected any concept that resulted in less thanten samples after 1000 episodes of sampling and conse-quently we focused on just 20 concepts for Sokoban-switchand 32 concepts for Sokoban-cell respectively. We usedAdaBoost Classifier (Freund et al., 1999) on Montezuma,and Convolutional Neural Networks (CNNs) for Sokobanvariants. The CNN architecture involved four convolutionallayers followed by three fully connected layers that gave abinary classification output. Montezuma classifiers had anaverage accuracy of 99.72%, while that for Sokoban-switchwas 99.46% and for Sokoban-cell was 99.34%.

Explanation identification: As mentioned previously,we ran the search for identifying preconditions for Mon-tezuma’s foils and cost function identification on Sokoban.From the collected list of concepts, we doubled the final con-cept list used by including negations of each concept. So forMontezuma we used 20 concepts per screen, and 40 and 64concepts were used for Sokoban-switch and Sokoban-cell.The probabilistic models for each classifier were calculatedfrom the corresponding test sets. For precondition identifica-tion, the search was run with a sampling budget of 500 anda cutoff probability of 0.01 for each concept. The searchwas able to identify the expected explanation for each foiland had a mean confidence of 0.5044 for foils in screen 1and a confidence value of 0.8604 for the foil in screen 4.The ones in screen 1 had lower probabilities since they werebased on more common concepts and thus their presencein the executable states was not a strong evidence for thembeing a precondition. For cost function identification, thesearch was run with a sampling budget of 750 and all thecalculations, including both computing the concept distri-bution and updating the probability of explanation, were

limited to states where the action was executable. Again thesearch was able to find the expected explanation. We had anaverage confidence of 0.9996 for the Sokoban-switch and0.998 for the Sokoban-cell.

User study: With the basic explanation generation methodin place, we were interested in evaluating if users would findsuch an explanation helpful. Specifically, the hypothesis wetested were

Hypothesis 1: Missing precondition information is a usefulexplanation for action failures.

Hypothesis 2: Abstract cost functions are a useful explana-tion for foil suboptimality.

To evaluate this, we performed a user study with all thefoils used along with the generated explanation and a sim-ple baseline explanation. For precondition case (H1), thebaseline involved pointing out just the failing action and thestate it was executed. For the cost case (H2), it involvedpointing out the exact cost of executing each action in thefoil. The users were asked to choose the one they believedwas more useful (the choice ordering was randomized to en-sure the results were counterbalanced) and were also askedto report on a Likert scale the completeness of the chosenexplanation. For each foil, we took the explanation gener-ated by the search and converted it into text by hand. Thesubjects were also given the option to provide suggestionson what they think would help improve the completenessof the explanation in a free text field. For H1, we collected20 replies in total (five per foil) and 19 out of the 20 partici-pants selected precondition based explanation as the choice.On the question of whether the explanation was complete,we had an average score of 3.35 out of 5 on the Likert scale(1 being not at all complete and 5 being complete). ForH2, we again collected 20 replies in total (ten per foil) andfound 14 out of 20 participants selected the concept-basedexplanation over the simple one. The concept explanationshad on average a completeness score of 3.36 out of 5. Theresults seems to suggest that in both case people did preferthe concept-based explanation over the simple alternative.The completeness results suggest that people may like, atleast in some cases, to receive more information about themodel.

6. Related WorkThere is an increasing number of works investigating theuse of high-level concepts to provide meaningful post-hocexplanations to the end-users. The representative worksin this direction include TCAV (Kim et al., 2018) and itsvarious offshoots like (Luss et al., 2019) that have focusedon one-shot decisions. Authors of (Hayes & Shah, 2017)have looked at the use of high-level concepts for policy sum-maries. They use logical formulas to concisely characterize

Bridging the Gap

various policy choices, including states where a specific ac-tion may be selected (or not). Unlike our work, they are nottrying to answer why the policy ends up choosing a specificaction (or not). (Waa et al., 2018) looks at addressing sub-optimality of foils while supporting interpretable features,but it requires the domain developer to specifically encodepositive and negative outcomes to each action. Anotherrelated work is the approach studied in (Madumal et al.,2020). Here, they are also trying to characterize dynamicsin terms of high-level concepts. Though in their example,they assume that the full structural relationship betweenthe various variables is provided upfront. The explanationsdiscussed in this paper can also be seen as a special case ofModel Reconciliation explanation (c.f (Chakraborti et al.,2017)), where the human model is considered to be emptyand our use of abstractions is also connected to the HELMexplanatory framework introduced in (Sreedharan et al.,2018). Usefulness of preconditions as explanation has alsobeen considered in other explanatory works like (Winikoff,2017; Broekens et al., 2010). Our effort to associate ac-tion cost to concepts could also be contrasted to efforts in(Juozapaitis et al., 2019) and (Anderson et al., 2019) whichexplain in terms of interpretable reward components. Un-fortunately, their method relies on having reward functionbeing represented using interpretable components.

7. ConclusionWe view the approaches introduced in the paper as the firststep towards designing more general symbolic explanatorymethods for sequential decision-making problems that op-erate on inscrutable representations. The current methodsfacilitate generation of explanations in user-specified termsfor sequential decisions by allowing users to query the sys-tem about alternative plans. We implemented these methodin multiple domains and evaluated the effectiveness of theexplanation using user studies. While contrastive explana-tions are answers to questions of the form “Why P and notQ?”, we have mostly focused on refuting the foil (the “notQ?” part). This is because, in the presence of a simulator, itis easier to show why the plan is valid by simulating the planand presenting the state trace. We can further augment suchtraces with the various concepts that are valid at each stepof the trace. Also, note that the methods discussed in thispaper can still be used if the user’s questions are specified interm of temporal abstraction over the agent’s action space.As long as the system can simulate the foils raised by theuser, we can keep the rest of the methods discussed in thepaper the same. In future, we would like to investigate morecomplex tasks, such as those with stochastic dynamics orpartial observability. Also, we could look at extending themethods to support partial foils (where the user only speci-fies some part of the plan) and develop methods that allowfor efficient acquisitions of new concepts from the user.

As mentioned, the readers can find the supplementary filecontaining the algorithm pseudocodes, derivation of all theformulas (along with formulas for noisy classifier cases) andadditional experiment details at the link https://bit.ly/38TmDPA.

AcknowledgmentsKambhampati’s research is supported in part by ONRgrants N00014-16-1-2892, N00014-18-1-2442, N00014-18-1-2840, N00014-9-1-2119, AFOSR grant FA9550-18-1-0067, DARPA SAIL-ON grant W911NF-19-2-0006,NSF grants 1936997 (C-ACCEL), 1844325, NASA grantNNX17AD06G, and a JP Morgan AI Faculty Researchgrant.

ReferencesAnderson, A., Dodge, J., Sadarangani, A., Juozapaitis, Z.,

Newman, E., Irvine, J., Chattopadhyay, S., Fern, A., andBurnett, M. Explaining Reinforcement Learning to MereMortals: An Empirical Study. In IJCAI, 2019.

Benton, J., Lipovetzky, N., Onaindia, E., Smith, D. E., andSrivastava, S. (eds.). Proceedings of the Twenty-NinthInternational Conference on Automated Planning andScheduling, ICAPS 2018, Berkeley, CA, USA, July 11-15,2019, 2019. AAAI Press. URL https://aaai.org/ojs/index.php/ICAPS/issue/view/239.

Botea, A., Muller, M., and Schaeffer, J. Using abstractionfor planning in sokoban. In International Conference onComputers and Games, pp. 360–375. Springer, 2002.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J.,Schulman, J., Tang, J., and Zaremba, W. Openai gym.CoRR, abs/1606.01540, 2016. URL http://arxiv.org/abs/1606.01540.

Broekens, J., Harbers, M., Hindriks, K., Van Den Bosch,K., Jonker, C., and Meyer, J.-J. Do you get it? user-evaluated explainable bdi agents. In German Conferenceon Multiagent System Technologies, pp. 28–39. Springer,2010.

Carbonell, J. G. and Gil, Y. Learning by experimentation:The operator refinement method. In Machine learning,pp. 191–213. Elsevier, 1990.

Chakraborti, T., Sreedharan, S., Zhang, Y., and Kambham-pati, S. Plan explanations as model reconciliation: Mov-ing beyond explanation as soliloquy. In IJCAI, pp. 156–163, 2017.

Chakraborti, T., Sreedharan, S., and Kambhampati, S. Theemerging landscape of explainable ai planning and deci-sion making. arXiv preprint arXiv:2002.11697, 2020.

Bridging the Gap

Freund, Y., Schapire, R., and Abe, N. A short introduc-tion to boosting. Journal-Japanese Society For ArtificialIntelligence, 14(771-780):1612, 1999.

Geffner, H. and Bonet, B. A concise introduction to modelsand methods for automated planning. Synthesis Lectureson Artificial Intelligence and Machine Learning, 8(1):1–141, 2013.

Geißer, F., Keller, T., and Mattmuller, R. Abstractions forplanning with state-dependent action costs. ICAPS, 2016.

Hayes, B. and Shah, J. A. Improving robot controller trans-parency through autonomous policy explanation. In 201712th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 303–312. IEEE, 2017.

Juozapaitis, Z., Koul, A., Fern, A., Erwig, M., and Doshi-Velez, F. Explainable Reinforcement Learning via Re-ward Decomposition. In IJCAI Workshop on explainableAI (XAI), 2019.

Kim, B., M., W., Gilmer, J., C., C., J., W., , Viegas, F., andSayres, R. Interpretability Beyond Feature Attribution:Quantitative Testing with Concept Activation Vectors(TCAV) . ICML, 2018.

Konidaris, G., Kaelbling, L. P., and Lozano-Perez, T. Fromskills to symbols: Learning symbolic representations forabstract high-level planning. Journal of Artificial Intelli-gence Research, 61:215–289, 2018.

Luss, R., Chen, P.-Y., Dhurandhar, A., Sattigeri, P., Zhang,Y., Shanmugam, K., and Tu, C.-C. Generating contrastiveexplanations with monotonic attribute functions. arXivpreprint arXiv:1905.12698, 2019.

Madumal, P., Miller, T., Sonenberg, L., and Vetere, F. Ex-plainable reinforcement learning through a causal lens.In AAAI, 2020.

Miller, T. Explanation in artificial intelligence: Insightsfrom the social sciences. Artificial Intelligence, 2018.

Ribeiro, M. T., Singh, S., and Guestrin, C. Why should itrust you?: Explaining the predictions of any classifier.In Proceedings of the 22nd ACM SIGKDD internationalconference on knowledge discovery and data mining, pp.1135–1144. ACM, 2016.

Schrader, M.-P. B. gym-sokoban. https://github.com/mpSchrader/gym-sokoban, 2018.

Sreedharan, S., Srivastava, S., and Kambhampati, S. Hier-archical expertise level modeling for user specific con-trastive explanations. In IJCAI, pp. 4829–4836, 2018.

Stern, R. and Juba, B. Efficient, safe, and probably ap-proximately complete learning of action models. arXivpreprint arXiv:1705.08961, 2017.

Waa, J., Diggelen, J. v., Bosch, K., and Neerincx, M.Contrastive Explanations for Reinforcement Learningin Terms of Expected Consequences. In IJCAI Workshopon explainable AI (XAI), 2018.

Wikipedia contributors. Montezuma’s revenge (videogame) — Wikipedia, the free encyclopedia, 2019. URLhttps://en.wikipedia.org/w/index.php?title=Montezuma%27s_Revenge_(video_game)&oldid=922152932. [Online; accessed14-January-2020].

Winikoff, M. Debugging agent programs with why? ques-tions. In Proceedings of the 16th Conference on Au-tonomous Agents and MultiAgent Systems, pp. 251–259,2017.