ciia-50741

8/12/2019 ciia-50741

1/25

Current Issues in Auditing American Accounting AssociationVolume 8, Issue 1 DOI: 10.2308/ciia-507412014Pages C1C25

COMMENTARY

On the Use of Checklists in Auditing: A Commentary

J. Efrim Boritz and Lev M. Timoshenko

SUMMARY: Experimental studies concerning fraud (or red flag) checklists often areinterpreted as providing evidence that checklists are dysfunctional because their useyields results inferior to unaided judgments ( Hogan et al. 2008 ). However, some of thecriticisms leveled against checklists are directed at generic checklists applied byindividual auditors who combine the cues using their own judgment. Based on a reviewand synthesis of the literature on the use of checklists in auditing and other fields, we offer a framework for effective use of checklists that incorporates the nature of the audit task,checklist design, checklist application, and contextual factors. Our analysis of checklistresearch in auditing suggests that improvements to checklist design and to checklistapplication methods can make checklists more effective. In particular, with regard to fraudrisk assessments, customizing checklists to fit both client circumstances and thecharacteristics of the fraud risk assessment task, along with auditor reliance on formalcue-combination models rather than on judgmental cue combinations, could make fraudchecklists more effective than extant research implies.

Keywords: checklists; red flag checklists; decision aids; fraud risk assessment.

INTRODUCTIONChecklists are widely used in auditing to ensure compliance with various requirements, such as

completing audit procedures in the appropriate sequence and without omission, collecting or compiling all relevant materials required to complete an audit file, and supporting judgment tasks bylisting the key questions to be asked and answered in order to reach appropriate judgments or conclusions. However, there is a widely held view that the use of diagnostic checklists for fraud risk

J. Efrim Boritz is a Professor and Lev M. Timoshenko is a Ph.D. candidate , both at the University of Waterloo .

We acknowledge the helpful comments on an earlier version of this paper provided by Karen Pincus, and the researchassistance of Chris Wong. We gratefully acknowledge the funding provided by the University of Waterloos Center for Information Integrity and Information Systems Assurance.

Submitted: March 2013 Accepted: February 2014

Published Online: February 2014

8/12/2019 ciia-50741

2/25

assessment in financial audits yields dysfunctional outcomes (e.g., Hogan, Rezaee, Riley, and Velury2008 ; Jamal 2008 ), and this view is spreading to other uses of checklists (e.g., Wheeler andArunachalam 2008 ; Seow 2011 ). We agree that, when checklists are incomplete, are too long,contain inappropriate content, or are applied inappropriately, they can result in checklist failure.However, some interpretations of audit-checklist failures take a relatively narrow perspective andblame the failure on simply the use of a checklist rather than on weaknesses in the design or application of the checklist. Abandoning the use of checklists (e.g., red flag checklists) in favor of unaided judgment on the basis of such a limited perspective may not be the best option for large firmswith multi-location audits that require a degree of standardization and coordination in the performanceof audit tasks. Our review of literature on theuse of checklists in auditing and other fields suggests thatimprovements in checklist design and application can make checklists more effective.

This study is motivated by the contrast between some of the negative views of checklistsexpressed in the extant audit literature; in particular, research on the use of red flag checklists tosupport fraud risk assessment and a considerable body of evidence on the successful use of checklists in many other fields including psychology (Fischhoff, Slovic, and Lichtenstein 1978),aviation ( FAA 2007 ; Turner 2001 ), nuclear power generation ( Carvalho, dos Santos, and Vidal

2005 ; Carvalho, Vidal, and de Carvalho 2007 ), and medicine ( Pronovost et al. 2006 ; Haynes et al.2009 ; Ely, Graber, and Croskerry 2011 ). Gawandes (2009) book, The Checklist Manifesto,summarizes the benefits achieved by using checklists in the medical field. Kahneman, Lovallo, andSibony (2011) support Gawandes (2009) endorsement of checklists, suggesting that checklistscan be helpful in many areas, most notably in corporate decision making. In fact, Kahneman et al.(2011 , 54) provide a 12-question checklist designed to protect corporate decision makers againstvarious defects in thinking, although Kahneman (2011 , 249) recognizes that some practitionersvoice concerns about the impersonality of procedures that are guided by statistics and checklists.We believe that criticisms of checklists in auditing should be reassessed in light of theseobservations.

In this article, we review and synthesize the literature on checklists in auditing and other fields

and consider a comprehensive array of factors that affect the use and effectiveness of checklists inauditing, with a focus on diagnostic judgment-oriented checklists. Raiffa (1968) defines a judgmental checklist as an approach to aiding decisions that decomposes a judgment intocomponents that decision makers can assess, and subsequently combine. We provide aframework for thinking about checklists that recognizes that checklist-based outcomes are a jointproduct of the task and task environment, checklist design, checklist application, and contextualfactors. We conclude that appropriately designed and adequately customized checklists, appliedstrategically using appropriate cue-combination methods, can be effective.

CHECKLIST USE IN AUDITING AND OTHER FIELDSIn auditing, checklists have long been utilized to assist auditors in completing a variety of tasks

including client acceptance and retention decisions ( Bell, Be dard, Johnstone, and Smith 2002 ),inherent risk assessments ( Boritz, Albuquerque, and Kielstra 1991 ; Be dard and Graham 2002 ),internal control evaluations ( Ashton 1974 ; Trotman, Yetton, and Zimmer 1983 ; Boritz 1985 ),substantive test planning ( Blocher, Esposito, and Willingham 1983 ; Bonner, Libby, and Nelson1996 ), and fraud risk assessments ( Pincus 1989 ). In fact, checklists are a part of GAAS, as theAICPA (2002 , 193197) provides lists of red flags related to misstatements arising from fraudulentfinancial reporting, as well as misappropriation of assets. Ironically, just as the virtues of checklistsare being extolled in other fields (e.g., Gawande 2009 ; Kahneman 2011 ), the use of checklists in

Boritz and Timoshenko C2

Current Issues in Auditing Volume 8, Issue 1, 2014

8/12/2019 ciia-50741

3/25

auditing is being challenged, especially for tasks such as fraud risk assessment. Results of experimental studies examining the use of fraud red flag checklists ( Pincus 1989 ; Asare andWright 2004 ; Seow 2009 ) are interpreted as providing evidence that checklists are harmful rather than helpful in detecting fraud ( Jamal 2008 , 103), and that they restrict the auditors generation of ideas for fraud detection ( Hogan et al. 2008 , 239240). Also, recent papers question theeffectiveness of checklists for audit and tax professionals, such as Seow (2011) for an internalcontrol evaluation task, and Wheeler and Arunachalam (2008) for a tax research task. Theseauthors argue that checklists increase confirmation bias and focus the users attention solely oncues included in the checklist. Bamber, Carpenter, and Hammersley (2007) suggest that theimportance of red flag checklists is diminished with the introduction of SAS No. 99s ( AICPA 2002 )fraud brainstorming requirement. The authors argue that a SAS No. 99 brainstorming sessioncannot be readily reduced to a checklist or form ( Bamber et al. 2007 , 45).

We reviewed all of the audit research that we could identify that involved the use of checklists,as well as relevant research in medicine and other fields. Overall, our literature review suggeststhat the question of checklists effectiveness for various tasks is far from settled. While somestudies find mixed or weak evidence of checklist usefulness ( Blocher et al. 1983 ; Bonner et al.

1996 ), or report dysfunctional consequences of their application ( Pincus 1989 ; Asare and Wright2004 ; Bamber et al. 2007 ; Wood 2012 ), others conclude that checklists are useful for either performing an audit task ( Butler 1985 ; Alon and Dwyer 2010 ) or for learning ( Seow 2009 ; J. Rose,McKay, Norman, and A. Rose 2012 ). Ashton (1974) , R. Libby and P. Libby (1989 ), Bonner et al.(1996) , Eining, Jones, and Loebbecke (1997) , Bell and Carcello (2000) , A. Rose and J. Rose(2003 ), and Marley (2011) all document improved effectiveness related to using checklist-basedaids. Effectiveness gains appear to be stronger for checklist-based audit decision aids thatsupplement a checklists knowledge retrieval function with a model for knowledge aggregation.Nelson and Tan (2005 , 54) suggest that decision aids can be used to enhance independence andfocus the expertise of large firms in the key areas for which they are most vulnerable to auditfailure. However, potential improvements may be attenuated by issues associated with auditor

nonreliance on checklists ( Boatsman, Moeckel, and Pei 1997 ). That is, even when the reliability of a checklist is supported by evidence, auditors may nevertheless overrule it because of apreference for relying on their own judgments or a wish to avoid particular consequences, such asexceeding the budget.

We now examine checklist use in other fields to determine whether auditing practice canbenefit from insights provided by research and practice in those fields. In aviationone of the firstindustries to adopt checklistschecklists are intended to assist with operational procedures sothat pilots can confirm that omission of any critical procedural step is detected and remediatedbefore it becomes consequential. Compliance with standards is achieved via error trapping ( FAA2007 ; Turner 2001 ), whereby a procedure continues only if all critical errors are intercepted andaddressed. Psychology research identifies checklist applications in many areas, including nuclear

power generation, in which diagnostic judgmental checklists called fault trees can assistpersonnel in determining the causes of system breakdowns ( Fischhoff et al. 1978 ).1 In addition,

1 A fault tree is a judgmental checklist organized as a decision tree. In more precise terms, in a fault tree, asystem failure state is postulated and sequences of more basic faults contributing to this state are laid out in asystematic manner ( U.S. NRC 1981 , I-8). Checklists with similar properties to fault trees have been used inauditing research and practice in connection with expert systems developed in the 1980s and 1990s for evaluating loan loss provisions, tax provisions, inherent risk assessments, going concern evaluations, andother diagnostic tasks with embedded decision trees.



8/12/2019 ciia-50741

4/25

Fischhoff et al. (1978) explain that fault trees can help people who deal with complex falliblesystems to better describe and comprehend these systems. Carvalho et al. (2005) and Carvalho etal. (2007) report that, at nuclear power plants, personnel use both procedural and judgmentalchecklists. In plant procedures, the actions of operators are described by flowcharts to illustrate thesequence of the systems actions, as well as checklists in which operators document the specific

manual actions they executed. According to Carvalho et al. (2005) and Carvalho et al. (2007) ,checklists help operators adapt to a high degree of plant automation. Specifically, when they workwith checklists, operators become more aware of plant conditions, because checklists promptthem to look for information that is not readily available without a stimulus.

In healthcare, checklists initially were adopted to ensure that doctors and nurses in hospitalscompleted critical procedures. For example, checklists were implemented in intensive care units(ICUs) to help prevent post-surgery complications ( Pronovost et al. 2006 ). Further, the introductionof a 19-item operating room checklist resulted in an impressive 50 percent reduction in surgicaldeaths ( Haynes et al. 2009 ). For more information on the use of such checklists in a healthcaresetting, please refer to Gawande (2007) or, for a more comprehensive discussion, Gawande(2009) . Recently, academics and practitioners have raised questions about whether the successful

implementation of procedural checklists can be expanded to judgmental checklists that could beused for medical diagnoses ( Ely et al. 2011 ; Winters et al. 2009 ; Winters, Aswani, and Pronovost2011 ). Ely et al. (2011) suggest that a combination of checklists possessing different strengthscould help to minimize a number of individual cognitive biases (such as anchoring, availability,base rate neglect, premature closure, and others; for details refer to Ely et al. [2011] ), as well as toavoid groupthink when a team of doctors makes a diagnosis.

Based on our review and synthesis of the literature on checklists in auditing and other fields,we developed Table 1, which classifies checklists along two dimensions: (1) generic versuscustomized, and (2) procedural versus judgmental. A generic checklist is a standardized list of common items to be considered, without being tailored to specific circumstances, whereas acustomized checklist is tailored to fit a specific situation ( Cowperthwaite 2012 ). A proceduralchecklist provides a basic outline of the work to be performed, including all or most of the importantsteps to be carried out, along with their sequence, which the user can modify as required toaccommodate the details of a particular application. In addition to ensuring standardization of procedures across, for example, audits, geographical locations, and audit team members, and thecompleteness and appropriate sequencing of procedures, checklists provide documentation of theprocedures actually performed, which can facilitate review ( Anderson 1977 , 413). Recent findingsin PCAOB inspection and peer review reports ( Gramling and Watson 2009 ; PCAOB 2013 ) suggestthat audit documentation remains a major area of concern for regulators, particularly for audits of accounting estimates (e.g., fair values) and for fraud risk assessment. The use of checklists,therefore, can provide the additional benefit of fulfilling documentation requirements imposed byregulatory and standard-setting bodies, such as the PCAOB.

Table 1 serves as a reminder that, when assessing the strengths and weaknesses of checklists, it is important to distinguish among the various types of checklists represented in thefour cells in Table 1. As stated earlier, the focus of this paper is on judgment-oriented checklistssuch as those used in client acceptance and continuance, internal control evaluation, and fraudrisk assessment, summarized in the cells in the bottom half, and especially the lower right handcorner of Table 1.

Our review of the literature identified factors that influence the effectiveness of judgmentchecklists that we have summarized in Figure 1 under four main headings: nature of the task,



8/12/2019 ciia-50741

5/25

T A B L E 1

C l a s s i c a t i o n o f C h e c k l i s t s

C h e c k l i s t T y p e

G e n e r i c :

C h e c k l i s t s T h a t A r e A p p l i c a b l e t o

M u l t i p l e S u b j e c t s

i n a V a r i e t y o f S i t u a t i o n s

C u s t o m i z e d :

C h e c k l i s t s A d o p t e d t o

a P a r t i c u l a r S u b j e c t

a n d / o r t o S p e c i c C i r c u m s t a n c e s

P r o c e d u r a l ( a c t i o n a b l e p r o c e d u r e )

c h e c k l i s t s d e s i g n e d t o a s s i s t w i t h

c o m p l i a n c e w i t h a s p e c i e d

s e q u e n c e o f a c t i o n s s u c h a s

c o m p l e t i o n o f a u d i t d o c u m e n t a t i o n .

F A A c h e c k l i s t s ( F A A 2 0 0 7 ; T u r n e r

2 0 0 1 )

N u c l e a r p o w e r p l a n t p r o c e d u r a l

c h e c k l i s t s ( C a r v a l h o e t a l . 2 0 0 5 a n d

C a r v a l h o e t a l . 2

0 0 7 )

P r o c e d u r a l g e n e r i c a u d i t c h e c k l i s t s

( B l o c h e r e t a l . 1

9 8 3 ; M c D a n i e l 1 9 9 0 )

C h e c k l i s t o f c o m m o n p i t f a l l s a n d

c o g n i t i v e f o r c i n g f u n c t i o n s f o r a

s p e c i c d i s e a s e ( E l y e t a l . 2

0 1 1 ) a

P r o c e d u r a l c u s t o m i z e d a u d i t c h e c k l i s t s

( C o w p e r t h w a i t e 2 0 1 2 )

J u d g m e n t a l ( c o g n i t i v e p r o c e d u r e )

c h e c k l i s t s d e s i g n e d t o a s s i s t w i t h

m a k i n g a j u d g m e n t s u c h a s

a c c e p t a n c e o f a n e x i s t i n g o r

p o t e n t i a l c l i e n t , i

n t e r n a l c o n t r o l

e v a l u a t i o n , o r a s s e s s m e n t o f f r a u d

r i s k .

R e p r o d u c t i v e a p p r o a c h t o d i a g n o s i s

c h e c k l i s t ( E l y e t a l . 2

0 1 1 )

F a u l t t r e e ( F i s c h h o f f e t a l . 1

9 7 8 )

D i f f e r e n t i a l d i a g n o s i s c h e c k l i s t ( E l y e t

a l . 2

0 1 1 )

b

J u d g m e n t a l c u s t o m i z e d a u d i t

c h e c k l i s t s ( C o w p e r t h w a i t e 2 0 1 2 )

G e n e r i c r e d a g c h e c k l i s t ( P i n c u s

1 9 8 9 ; A I C P A 2 0 0 2 )

C u s t o m i z e d r e d a g c h e c k l i s t

( B o a t s m a n e t a l . 1

9 9 7 ; B e l l a n d

C a r c e l l o 2 0 0 0 )

a

C o g n i t i v e f o r c i n g f u n c t i o n c h e c k l i s t s e n s u r e t h a t a c o r r e c t p r o c e d u r e i s f o l l o w e d , a

n d / o r p r e v e n t a n u n t o w a r d e v e n t ( E l y e t a l . 2

0 1 1 , 3 1 0 )

.

b

D i f f e r e n t i a l d i a g n o s i s c h e c k l i s t s b r i n g t o a t t e n t i o n d i a g n o s e s t h a t s h o u l d n o t b e m i s s e d a n d t h o s e t h a t a r e , i n p r a c t i c e , f r e q u e n t l y m i s s e d ( E l y e t a l . 2

0 1 1 , 3 1 0 )

.



8/12/2019 ciia-50741

6/25

F I G U R E 1

F a c t o r s A f f e c t i n g t h e E f f e c t i v e n e s s o f a C h e c k l i s t



8/12/2019 ciia-50741

7/25

checklist design, checklist application, and contextual factors. Below, we discuss these in turn,using fraud risk assessment as an illustrative example of a diagnostic audit task.

NATURE OF THE TASK The nature of the task determines the need forand should influence the design ofdecision

aids, such as checklists, to enhance task performance. Although we discuss a variety of auditchecklists throughout this paper, in this section we focus on fraud checklists because of theprominence of their coverage in the literature and the attention paid by regulators and practitionersto the auditors assessment of and response to fraud risk ( McKee 2010 ; PCAOB 2012 ). Fraud riskgenerally is classified into two main types: financial statement fraud and misappropriation of assets. 2 Most of the fraud-related auditing literature addresses financial statement fraud, and wealso focus on this type of fraud. One key difference between fraud detection in an audit setting andtasks in other fields in which checklists are used (e.g., nuclear power generation, aviation, medicalpractice) is in the strategic nature of financial reporting fraud. The fraud perpetrator uses deceptionto mislead auditors and counter the auditors attempts to detect fraud through the use of red flagchecklists, whereas no such intentional deception is at play in other fields in which checklists are

used.Economic game theory suggests that the most effective course of action for a

fraud-perpetrating manager is to follow a randomized strategy when perpetrating the fraud (i.e.,one that makes it difficult to predict where and when the fraud was perpetrated), and that theauditors optimal response is to randomize the audit strategy to make it more difficult for fraud-perpetrating managers to guess where and when the auditor might look for fraud ( Jamal2008 , 102). However, current risk-based audit practice represents a type of focused (non-randomized) strategy, which game theory suggests is suboptimal for countering a randomizedfraud strategy, because managers can use the auditors red flag checklists to guess the auditorsstrategy and perpetrate fraud in areas that do not appear to be risky. Therefore, proponents of thisview argue that a risk-based audit is more suitable for detecting errors, because they are

unintentional, than fraud, which is deliberate and strategic ( Jamal 2008 , 102103). Research byHoffman and Zimbelman (2009) concludes that the strategic nature of fraud requires strategicthinking on the part of the auditor, including whether and how red flag checklists are deployed.

We believe that it is difficult for management to implement a randomized fraud strategybecause of the constraining effects of the double entry accounting system and the need to recruitcollaborators. These factors can reduce the unpredictability of the methods used by fraudperpetrators and can permit the successful application of checklist-based decision aids, asdocumented by Eining et al. (1997) and Bell and Carcello (2000) . However, we agree that strategicthinking on the part of the auditor is required.

2 Financial statement fraud is the intentional misstatement or omission of amounts or disclosures in financialstatements, typically by top management, to deceive users. Misappropriation of assets (i.e., employee fraud)is the second main category of fraud ( AICPA 2002 ). Financial statement fraud is, on average, much more likelyto be material than employee fraud, even though it occurs much less frequently than does employee fraud(Wells 1997 ), because top management has the ability to influence larger amounts than do employees, andthe financial misrepresentations designed to mislead users, if successful, are by definition material.



8/12/2019 ciia-50741

8/25

CHECKLIST DESIGNA checklists design greatly influences its effectiveness (e.g., Glover, Prawitt, and Spilker

1997 ). The framework in Figure 1 identifies several key checklist design issues that influence theeffectiveness of checklists: customization (related to checklist type), comprehensiveness versusdiagnosticity, and structure of the checklist content.

Customization

Recent discussions in the medical and auditing fields suggest that customizing diagnostic judgment checklists is necessary to increase their effectiveness ( Ely et al. 2011 ; Cowperthwaite2012 ). According to Ely et al. (2011) , diagnostic medical checklists are tailored to specificdiseases. In fraud risk assessment and, more generally, in audit settings, customization isaccomplished by adapting the checklist to the characteristics particular to the client and/or theindustry in which the client operates, the staff mix specific to the engagement, and to the areas of the audit in which professional judgment is important ( Cowperthwaite 2012 , 39). 3 Cowperthwaite(2012) suggests that auditors can start with a generic checklist and then tailor it based on factors

such as their knowledge of the client and understanding the requirements of GAAS. Thecustomization helps to omit irrelevant items in the checklist and emphasize items important to theengagement ( Cowperthwaite 2012 ).

In the fraud risk assessment task, besides enhancing an auditors risk assessments, anadditional and important benefit of customization is its positive effect on the checklist users abilityto translate a diagnostic judgment into an appropriate audit program. Hammersley (2011) providesevidence that a red flag fraud checklist can produce appropriate fraud risk assessments, butshould be supplemented with client-specific contextual cues (i.e., the clients circumstance in theyear under audit) in order to effectively translate the fraud risk assessments into appropriatechanges to the audit plan. These benefits of customization are in line with the critique that usingstandardized checklists may not reflect an allocation of audit work weighted toward high-risk

areas ( PCAOB 2006 ; emphasis added). In other words, customization can eliminate a mismatchbetween the decision aid and the task situation, as well as between the decision aid and thedecision maker, which has been shown to cause passive application of the aid (i.e., itsmechanical use without engaging in active thinking about the task) and inappropriate reliance(Glover et al. 1997 ). Some examples of checklists that would be useful are client acceptance/retention checklists, fraud risk checklists, and internal control evaluation checklists.

Comprehensiveness versus Diagnosticity

Two potentially conflicting design-related issues associated with checklist effectiveness arecomprehensiveness of cues and the relative diagnosticity of positive and negative cues. Pincus

(1989) states that the omission of apparently useful red flags in her checklist highlights the need tobetter investigate financial statement fraud indicators, and that developing a comprehensive set of such indicators might increase the effectiveness of fraud checklists. 4 As the search for useful red

3 The staff mix includes the levels of experience possessed by the audit team members, as well as theinvolvement of specialists (e.g., IT specialists, forensic specialists).

4 In Pincus (1989) , the participants who did not use the red flag checklist relied on a number of red flags notincluded in the checklist, such as public company status, the auditors opinion about certain components of theclients governance, and the clients cash management skills.



8/12/2019 ciia-50741

9/25

flags continues, auditors should be aware of new types of red flags identified and evaluated byresearchers and regulators, such as disparities between financial and nonfinancial indicators(Brazel, Jones, and Zimbelman 2009 ; ASC 2011 ); e.g., growth in revenue outpacing that inwarehouse space, number of retail outlets, or employee headcount.

However, as desirable as the comprehensiveness of cues in a checklist may be, too manycues can cause diagnosticity problems because a large proportion of red flags identified in theliterature and in auditing standards are ineffective in distinguishing between fraud and nonfraudengagements ( Bell and Carcello 2000 ), which raises the issue of red flags predictive ability.Pincus (1989) notes that her results are consistent with users of checklists under emphasizingnegative indicators suggesting potential fraud. This, in turn, can result in judgment errors at thecue-combination stage (i.e., the stage at which the auditor synthesizes the cues obtained from allof the items included in the checklist into a single overall judgment of fraud risk). While there couldbe many reasons for such an under emphasis, Pincus (1989) believes that an excessive number of red flags with weak predictive ability likely draws attention away from those with strong predictiveability, suggesting that checklist design can be improved by identifying and utilizing a small number of highly predictive items, and dropping the large number of red flags with limited predictive ability,as in Bell and Carcello (2000) .5 Pincuss (1989) suggestions are confirmed in more recent studiesby Wood (2012) and Be dard and Graham (2002) . Wood (2012) produces evidence that, for achecklist used in a fraud risk assessment task, nonpredictive cues dilute the predictive value of diagnostic cues. Be dard and Graham (2002) find that a negatively oriented checklist causesauditors to identify more risk factors compared to auditors who use a positively oriented checklist, if the client has high engagement risk. 6

Woods (2012) and Be dard and Grahams (2002) findings suggest that audit firms couldimprove their risk assessment procedures through relatively simple modifications to checklistdesign. In practice, at least some audit firms appear to be using fraud risk checklists for which redflag selection is based on statistical procedures that utilize historical data from a variety of their clients. For example, in Boatsman et al. (1997) an audit firm provided the researchers with an

actual decision aid that used 24 red flag indicators to aid in fraud risk assessment. These indicatorswere extracted statistically from a wider set of red flags documented in prior research andprofessional literature, resulting in an effective checklist ( Boatsman et al. 1997 , 219). 7 In addition,auditors should not disclose details of their audit plans, including their final checklists composition,to the client. Moreover, if the auditor feels that there is a need to be strategic, then she/he cancollect the information related to red flags with limited predictive ability, but not use them in her/hisfinal risk assessments (however, this can increase the cost of using checklists).

5 Pincuss (1989) questionnaire is based on the questionnaire in Romney, Albrecht, and Cherrington (1980) ,which contains items with a wide variance in predictive ability ( Albrecht and Romney 1986 ). Bell and Carcellos(2000) checklist relies on a small number of significant red flags ( i.e., weak internal control environment, rapidcompany growth, inadequate or inconsistent profitability, undue emphasis on meeting earnings projections,lying to or evasiveness with auditor, aggressive financial reporting, and ownership status).

6 In Be dard and Graham (2002) , a checklist has a negative focus if client risk and its consequences areemphasized and a positive focus if such factors are not emphasized.

7 The aid validation demonstrated a classification accuracy of 81 percent for both fraud and nonfraud audits.



8/12/2019 ciia-50741

10/25

Structure

Boritz (1985) highlights three benefits of applying hierarchical structures to audit checklists:structures clarify problems through decomposition and modularization of information sets;structures limit the use of heuristics and biases of auditors; structures incorporate knowledgeand expertise that highlight relationships and information cues. Wilks and Zimbelman (2004)provide evidence that structural improvements can make fraud risk assessment checklists moreeffective. In their experiment, audit managers who used fraud checklists in conjunction with thedecomposition of fraud risk assessments according to the fraud triangle (which postulates thatthree conditions must be present for a fraud to occur: incentive, opportunity, and attitude)assessed fraud risk better than did managers who relied on checklists without such decomposition(holistic approach), but only when opportunity and incentive cues indicated low risk. This resultdemonstrates that the decomposition aid is more effective than a traditional checklist in certaincontexts. 8 In extant auditing research, this study often is cited as evidence that checklists are notuseful ( Hogan et al. 2008 ). However, we believe that Wilks and Zimbelmans (2004) resultsindicate that improvements in cue organization can increase a checklists effectiveness.

CHECKLIST APPLICATIONThe framework in Figure 1 identifies two key checklist application issues that influence the

effectiveness of checklists: the method of combining cues and individual versus group use of checklists.

Method of Combining Cues

Kahneman (2011) describes two reasoning systems that govern cue processing: an intuitivesystem (System 1) that rapidly combines cues into a composite based on prior knowledge,heuristics, stereotypes, expectancies, scripts, and schemas ( Bargh and Chartrand 1999 ), and a

deliberative system (System 2) that combines cues in accordance with a rational decision model.Because of its speed and efficiency, the intuitive reasoning system often preempts the deliberativesystem and leads people to rely on heuristics and biases. Sloman (1996) points out that, evenwhen a person tries to be rule governed, responses prompted by unconscious associations withinformation cues encroach on judgment quality. This conclusion leads to the rationale for usingdecision aids to compensate for the risks associated with the application of intuitive judgment tocombine the cues identified with the help of checklists.

Recent behavioral studies suggest that deliberative thinking can be induced to improve fraudrisk assessments. For example, if opportunistic managers anticipate and take advantage of atraditional risk-based audit strategy by concealing misstatements within low-risk accounts moreoften than within high-risk accounts, auditors can successfully predict and counter such managersexpectations if they are prompted to think strategically about managers expected responses totheir audit approach ( Hoffman and Zimbelman 2009 , 2012 ). This supposition suggests thatchecklists usefulness for fraud risk assessments can be increased if the checklists are designed toencourage deliberative, strategic reasoning about the appropriate audit response to identifiedrisks. One possible way to accomplish this is to incorporate items that help auditors identify and

8 The decomposition is distinct from grouping checklist items, which has been incorporated in the standards(AICPA 2002 ; Wilks and Zimbelman 2004 , 741).



8/12/2019 ciia-50741

11/25

direct their attention to accounts and classes of transactions for which the traditional auditapproach is expected to produce a low-risk estimate.

Beyond inducing deliberative instead of intuitive cue weighting and combination, is thepossibility of using computational models. Extant psychology research suggests that individualsare not skilled at identifying complex, nonlinear relationships within data ( Hammond and Summers

1972 ). Therefore, simple, additive models utilizing limited information, such as those based onlinear regression, frequently perform better at identifying relationships than do experts, even whena large amount of data is available (e.g., Dawes 1971 ). Pincus (1989) mentions that she did nothave a reliable empirical cue-processing model at her disposal when she conducted her study;therefore, cues were evaluated using only the auditors judgment. In contrast, Eining et al. (1997)and Bell and Carcello (2000) had two different but reliable cue-processing models and were able tocompare the usefulness of three types of fraud cue-processing aids: a checklist, a statisticalmodel, and an expert system. Eining et al. (1997) and Bell and Carcello (2000) confirm that auditor judgment based on using the checklist alone is inferior to unaided judgment; however, the fraudrisk assessments obtained with the statistical model and the expert system are far superior tothose obtained through the use of unaided judgment. 9 The key point that must be emphasized

here is that the checklist-based identification of cues is necessary (although not sufficient) for theeffective functioning of both the statistical model and the expert system. In other words, theproblem with the red flag checklist is not the use of a checklist per se but the weak design of thechecklist and the ineffective combination of the cues identified using the checklist.

Using cue-combination models also can benefit other types of audit checklists including thoseintended for client acceptance and continuation judgments, control risk evaluation, and planning of substantive tests. For client acceptance and continuation, Bell et al. (2002) discuss a sophisticateddecision aid developed by KPMG that decomposes the decision into several categories andsubcategories. This process involves an extensive set of checklists that include informationgathered by the audit staff, which is then combined into an overall judgment of acceptance/continuation risk by a set of mathematical algorithms. For planning checklists, Bonner et al. (1996)find that a checklist combined with a model for aggregating judgment components performssignificantly better than unaided judgment, while a checklist alone gives only a slight improvement,because of the ability of the combined aid to facilitate both knowledge retrieval and aggregation.For control checklists, Ashton (1974) states that a likely reason for inconsistencies in internalcontrol risk judgments among auditors is differential weightings of the internal control indicatorsthey use. He suggests that a possible solution is to rely on a model that approximates truestatistical weights for the combination of indicators into an overall judgment ( Ashton 1974 , 153).Libby and Libby (1989) demonstrate that making component judgments of the strength of theindividual controls, and then aggregating the component judgments using a mechanical model issuperior to audit seniors global control risk assessments.

These results provide two key insights into the usefulness of audit checklists. First, it appearsthat auditors judgment processes, when using checklist-gathered evidence, breaks down whenthey combine the cues obtained from the checklist into a final judgment. Second, the cue-combination models and expert systems are, in essence, a combination of two decision aidsachecklist for finding the cues (knowledge retrieval) and a model for aggregating the set of cues

9 Interestingly, a contemporaneous study by Shelton, Whittington, and Landsittel (2001) demonstrates that, inpractice, Big N and second-tier audit firms did not rely widely on expert systems and other tools for processingcues obtained from their red flag checklists.



8/12/2019 ciia-50741

12/25

(knowledge aggregation). For the model to be effective, an appropriately small set of diagnosticcues to be processed by the model is required. Thus, a suitably designed audit checklist applied jointly with an effective cue-combination model can be a useful tool in conducting audits. Inaddition, it can reduce liability, as will be discussed later.

Individual versus Group Judgment In the medical field, both procedural operating room and judgmental diagnostic checklists are

used in group settings. Such settings involve doctors specializing in different clinical disciplines,nurses, and patients, who tend to use distinct terminologies to describe the same phenomena,thus causing miscommunication because of translation errors ( Winters et al. 2009 ). Winters et al.(2009) suggest that checklists help to democratize knowledge by standardizing and facilitatingthe reliable translation of information, so that the same knowledge awareness of the importance of a given set of cues is imparted to persons with different perspectives, professional cultures, andeducational backgrounds. There also are suggestions that general diagnostic checklists thatprompt doctors to optimize their cognitive approach can reduce groupthink, whereby a diagnosisby one member of a group of doctors is hastily adopted by the others ( Croskerry 2007 ; Ely et al.2011 ).

In auditing, as in medicine, many tasks involve multiperson teams. Prior research has shownthat mathematical composites of checklist-based individual judgments can outperform individualsin judgment tasks, such as internal control evaluation ( Trotman et al. 1983 ). Fraud riskassessments in financial audits are, at least in part, an interactive group process, as SAS No. 99(AICPA 2002 ) stipulates that auditors must discuss the potential for material misstatement causedby fraudulent activities, including an exchange of ideas or brainstorming among the members of the audit team. These brainstorming sessions have been shown to be effective ( Hoffman andZimbelman 2009 , 2012 ), and a positive role of checklists in such group judgment and decisionmaking is their ability to reduce miscommunication. Alon and Dwyer (2010) investigate how the

brainstorming component of SAS No. 99 ( AICPA 2002 ) influences checklist use and reliance, andthe effectiveness of fraud risk assessments. Using an approach similar to that employed by Pincus(1989) , Alon and Dwyer (2010) find that two-person groups of students with a checklist decision aidoutperform individuals with and without the decision aid, because of information processing gainsarising from the group interaction. Further, groups with the decision aid identified more qualityfraud ideas than those without the aid, suggesting that checklists add to decision quality for brainstorming groups ( Alon and Dwyer 2010 , 248). These results counter Bamber et al.s (2007)argument that the SAS No. 99 brainstorming requirement reduces the importance of fraudchecklists.

IMPACT OF CONTEXTUAL FACTORSPsychology research recognizes the importance of contextual factors for both individual and

group judgment and decision making ( Kerr and Tindale 2004 ; Bonner 2008 ), implying that thesefactors also influence the use of checklists. The framework in Figure 1 identifies the following threekey contextual factors that affect checklist use in auditing: performance pressures, legal liabilityconsiderations, and auditor characteristics (e.g., Ng and Tan 2003 ; Hammersley 2011 ).10

10 Performance pressures are those arising from accountability structures, time pressures, and financial andother incentives.



8/12/2019 ciia-50741

13/25

Performance Pressures

Gawande (2009) reports that procedural checklists are very effective in high-pressure workenvironments (e.g., operating rooms, airplane cockpits). These checklists improve outcomesbecause they draw users attention to tasks that have to be completed, but may often beoverlooked if there is no checklist. This creates an expectation that such checklists would performwell in other high-pressure workplaces, such as those encountered by auditors. Surprisingly,research studying how procedural checklists are and should be used in audits is almostnonexistent. McDaniel (1990) investigates how the joint imposition of program structure (consistingof detailed procedures and/or checklists) and time constraints affects auditors performance.McDaniel finds that, when time pressure is low, structure significantly increases auditeffectiveness, efficiency, and consistency, and when time pressure increases, audit efficiencyincreases and effectiveness decreases. Structure is associated with smaller decreases in auditeffectiveness at higher levels of time pressure, but the difference is not significant ( McDaniel 1990 ,268). This result is somewhat inconsistent with the effective performance of procedural checklistsin the high-pressure tasks described in Gawande (2009) , so the issue requires further investigation. Blocher et al. (1983) find that when auditors use procedural checklists for planninganalytical review, they plan more analytical and more tests of details procedures, compared towhen they do not use such checklists.

Legal Liability Considerations

Sutton, Young, and McKenzie (1994) investigate liability issues arising from an audit firmsuse of an expert system (as discussed, such systems may be based on structured checklists) andidentify two sources of risk. One is that the joint judgment of the auditor and the expert system maynot supply the required level of expertise, and the auditor will be held liable under GAAS fieldworkstandards ( AICPA 1993 ; PCAOB 2001 ) for not properly planning and supervising the use of thesystem. The other is that not relying on an expert system (when one is available) may be turnedagainst the auditor because the existence of such a system can increase the level of expertise aprudent practitioner should exercise. Jennings, Kneer, and Reckers (1993) find that, when GAASstandards are not readily at hand for a particular area, jurists use audit firms internal decision aidsas surrogate standards against which to gauge auditors performance. Results in Lowe, Reckers,and Whitecotton (2002) indicate that, if a decision aid is reliable, and the auditor followed itsrecommendations, the jurors attributed lower responsibility to the auditor. This indicates that jurorsplace a high value on decision aids and expect auditors to use them when effective ones areavailable.

Auditor Characteristics

Some auditors are very hesitant to rely on the judgment output of mechanical decision aids,particularly when faced with performance pressures such as financial incentives, justification, andoutcome feedback. 11 More experienced auditors appear to rely on decision aids less because of their confidence, perhaps even overconfidence, in their own expertise. Boatsman et al. (1997)found that auditors were more convinced of the credibility of the checklist-based decision aid when

11 Nelson and Tan (2005) provide a literature review that includes a section on decision aids; however, a recentreview by Knechel, Krishnan, Pevzner, Shefchik, and Velury (2013) barely mentions them.



8/12/2019 ciia-50741

14/25

it predicted no fraud rather than fraud and their reliance on a decision aid was a function of their concern about the severity of anticipated penalties for incorrect decisions. 12 From a practical pointof view, an interesting finding of the study is that a simultaneous increase in the penalties for auditfailure and for over auditing generates a higher reliance on the aid. Kaplan, Reneau, andWhitecotton (2001) find that auditors with an external locus of control rely more on a mechanicalcue-combination decision aid than those with an internal locus of control, and that involvingdecision makers in the aids development enhances reliance (the effect is more pronounced for internal locus of control auditors). Sieck and Arkes (2005) discover that peoples overconfidence inthe quality of their intuitive judgment adds to their reluctance to use effective cue-combining aids,and that only enhanced calibration feedback (involving answering several questions from memory)reduced overconfidence and increased reliance on the aid. Gomaa, Hunton, and Rose (2008) findthat practitioners rely more on decision aids when either litigation risk or control risk is high; whenboth risks are high, litigation risk amplifies the auditors awareness of legal defensibility and, thus,increases decision aid reliance, even though the auditors confidence in the quality of their judgment deteriorates. In other words, there is a tension between the overconfidence that couldlead auditors to ignore or override decision aids and the pressure to reduce legal liability that could

lead auditors to subordinate their judgment to that of the checklist.

RECOMMENDATIONS AND CONCLUSIONOur review of existing audit research indicates that procedural checklists are generally

thought to perform satisfactorily; whereas, there is a widely held view that the use of diagnosticchecklists for fraud risk assessment in financial audits yields dysfunctional outcomes (e.g., Hoganet al. 2008 ; Jamal 2008 ), and that this view is spreading to other uses of checklists (e.g., Wheeler and Arunachalam 2008 ; Seow 2011 ). However, we believe that abandoning the use of checklists,such as red flag checklists, in favor of unaided judgment may not be the best option for large firmswith multilocation audits that require a degree of standardization and coordination. Our analysis of the literature along dimensions of the suggested framework in Figure 1 identified various strategiesto improve checklist effectiveness. These strategies are presented in Table 2, using the fraud riskassessment checklist as an illustrative example. Below, we highlight some of the keyrecommendations in this table.

Our analysis indicates that cue processing appears to be the weak link in the use of judgmental checklists, implying that they need to be integrated with a cue-combining model, analgorithm based on a statistical model or expert system, to achieve better performance. If anappropriate cue-combining model is not available, then checklists need to be designed to trigger deliberative strategic thinking on the part of the auditor to better counter the strategic nature of management behavior. For example, in a fraud risk assessment context, this could beaccomplished by encouraging auditors to critically address accounts that the traditional auditapproach labels as low risk, or to consider managements anticipation of their audit programchanges in response to identified risks.

Another option is to use groups to mitigate the weaknesses associated with unaided individual judgment (such as in a fraud risk brainstorming session), even though groups are themselvessubject to a number of factors that can reduce judgmental group performance ( Kerr and Tindale2004 ). Checklists can exert positive effects on group judgment by mitigating miscommunication,

12 Boatsman et al.s (1997) decision aid consists of a list of 24 red flags with a subsequent mechanicalcombination of cues to estimate fraud potential. It was provided to the authors by an audit firm.



8/12/2019 ciia-50741

15/25

T A B L E 2

O b j e c t i v e s o f a n d

S t r a t e g i e s f o r I m p r o v i n g E f f e c t i v e n e s s o f D i a g n o s t i c C h e c k l i s t s

, U s i n g a R e d F l a g C h e c k l i s t a s a n

I l l u s t r a t i v e E x a m p l e

F a c t o r s A f f e c t i n g

C h e c k l i s t U s e

F i n d i n g s i n P r i o r L i t e r a t u r e

I m p r o v e m e n t O b j e c t i v e s

S t r a t e g i e s f o r I m p r o v e m e n t

C h a r a c t e r i s t i c s o f T a s k

( e . g . ,

F r a u d R i s k

A s s e s s m e n t )

E c o n o m i c g a m e t h e o r y m o d e l s

f r a u d c r e a t i o n a s a s t r a t e g i c

a c t i v i t y ( J a m a l 2 0 0 8 ) , b u t a u d i t o r s

c a n s u c c e s s f u l l y c o u n t e r

m a n a g e r s a c t i o n s w h e n

p r o m p t e d t o t h i n k s t r a t e g i c a l l y

a b o u t m a n a g e r s r e s p o n s e s t o t h e

a u d i t a p p r o a c h ( H o f f m a n

a n d

Z i m b e l m a n 2 0 0 9

, 2 0 1 2 ) .

I n c r e a s e c h e c k l i s t

e f f e c t i v e n e s s a t c o u n t e r i n g

m a n a g e m e n t s t r a t e g i c

f r a u d - c r e a t i n g b e h a v i o r .

D e s i g n c h e c k l i s t s t o e n c o u r a g e

s t r a t e g i c r e a s o n i n g , f

o r

e x a m p l e b y i n c o r p o r a t i n g i t e m s

f o c u s i n g t h e a u d i t o r s a t t e n t i o n

o n a c c o u n t s a n d c l a s s e s o f

t r a n s a c t i o n s t h a t a r e l o w r i s k

u n d e r t h e t r a d i t i o n a l a u d i t

a p p r o a c h .

C h e c k l i s t D e s i g n : T y p e

D i a g n o s t i c c h e c k l i s t s

, s u c h

a s f r a u d

c h e c k l i s t s

, m a y r e q u i r e

c u s t o m i z a t i o n i n o r d e r t o

b e

e f f e c t i v e ( E l y e t a l . 2

0 1 1 ;

C o w p e r t h w a i t e 2 0 1 2 ) .

I m p r o v e t h e t b e t w e e n t h e

f r a u d c h e c k l i s t a n d t h e

f r a u d r i s k a s s e s s m e n t

t a s k , a

s w e l l a s b e t w e e n

t h e f r a u d c h e c k l i s t a n d t h e

u s e r - a u d i t o r ( G l o v e r e t a l .

1 9 9 7 ) .

C u s t o m i z e t h e c h e c k l i s t t o t h e

c l i e n t , c

l i e n t s i n d u s t r y , a

n d

e n g a g e m e n t / a u d i t r m s t a f f m i x

( C o w p e r t h w a i t e

2 0 1 2 ) .

C h e c k l i s t D e s i g n :

C o n t e n t / S t r u c t u r e

T r a d i t i o n a l f r a u d c h e c k l i s t s m a y o m i t

a n u m b e r o f h i g h p r e d i c t i v e a b i l i t y

r e d a g s ( P i n c u s 1 9 8 9 ) ; h o w e v e r ,

a n e x c e s s i v e n u m b e r o f

n o n d i a g n o s t i c r e d a g s m a y

s u p p r e s s t h e d i a g n o s t i c i t y o f

d i a g n o s t i c r e d a g s . C

a t e g o r i z i n g

c u e s i n t o f r a u d t r i a n g l e c a t e g o r i e s

c a n i m p r o v e j u d g m e n t s ( W i l k s

a n d Z i m b e l m a n 2 0 0 4 )

C r e a t e a m o r e

c o m p r e h e n s i v e l i s t o f h i g h

p r e d i c t i v e a b i l i t y r e d a g s ,

b u t a v o i d n o n p r e d i c t i v e

r e d a g s .

I n c l u d e a n d t e s t n e w l y d i s c o v e r e d

r e d a g s s u c h a s t h e d i s p a r i t y

i n g r o w t h b e t w e e n n a n c i a l a n d

n o n n a n c i a l i n d i c a t o r s ( B r a z e l e t

a l . 2

0 0 9 ; A S C 2 0 1 1 ) ; e l i m i n a t e

n o n p r e d i c t i v e r e d a g s .

S t r u c t u r e c h e c k l i s t t o

e n h a n c e c u e d i a g n o s t i c i t y

.

S t r u c t u r e / c a t e g o r i z e c u e s u s i n g

a p p r o p r i a t e c a t e g o r i e s .

( c o n t i n u e d o n n e x t p a g e )



8/12/2019 ciia-50741

16/25

8/12/2019 ciia-50741

17/25

groupthink, or other factors known to reduce judgmental group performance. In particular, theymay mitigate miscommunication between various specialists (e.g., forensic specialists, taxspecialists, valuation specialists) and regular audit team members during the risk assessmentplanning phase of the audit.

Contextual factors, such as performance pressures and the legal environment, need to be

considered by audit team managers, as they can affect the degree of reliance placed by audit teammembers on checklists and impact the translation of assessed risk into changes in the auditprogram. In audit settings, the use of checklists renders an additional benefit of fulfillingdocumentation requirements imposed by regulatory and standard-setting bodies, such as thePCAOB. The application of judgmental checklists can produce necessary evidence supportingauditors conclusions in the areas in which appropriate documentation is currently lacking. For example, red flag checklists can be useful for documenting the estimate of risk of materialmisstatement because of fraud, while checklists designed for fair value auditing (e.g., PwC 2011 ;Deloitte 2012 ) can improve the documentation in that area.

Many important questions related to the use of audit checklists remain unexplored and openfor future research. Cowperthwaite (2012) asserts that customization may mitigate some of the

weaknesses of generic red flag checklists while retaining some of their benefits; however, suchcustomization has not been researched in audit settings and raises several questions. 13 If theengagement team is qualified to modify the checklist (and is allowed by their firm to do so), thenwhy would they not be able to make good use of the standard checklist? Will audit teams take thetime needed to modify checklists appropriately? If the engagement team modifies the checklist,might they make it worse, not better? These questions suggest that much useful research could beconducted in this area. Gawande (2009 , 120) asserts that a good checklist must possess twoqualitiesit should be short and precise. However, among the many studies discussed in thiscommentary, only Bell and Carcello (2000) and Boatsman et al. (1997) address the issue to someextent, reflecting on the theoretical and practical aspects of cues diagnosticity. Clear criteria for the precision and inclusion/exclusion of questions in an audit checklist are yet to be developed.

Another area for research is the communication-enhancing role of checklistsboth thecommunication between different types of specialists and between supervisors and subordinates(Gawande 2009 , 67, 103; KPMG 2011 ). Evidence needs to be obtained on what role checklistscan play in integrating the diverse personnel involved in the audit process.

REFERENCESAlbrecht, W. S., and M. B. Romney. 1986. Red-flagging management fraud: A validation. Advances in Accounting 3:

323333.Alon, A., and P. Dwyer. 2010. The impact of groups and decision aid reliance on fraud risk assessment. Management

Research Review 33 (3): 240256.American Institute of Certified Public Accountants (AICPA). 1993. Codification of Statements on Auditing Standards .

New York, NY: AICPA.American Institute of Certified Public Accountants (AICPA). 2002. Consideration of Fraud in a Financial Statement

Audit . Statement on Auditing Standards No. 99. New York, NY: AICPA.Anderson, R. J. 1977. The External Audit 1: Concepts and Techniques . Toronto, ON: Pitman.Asare, S., and A. Wright. 2004. The effectiveness of alternative risk assessment and program planning tools in a fraud

setting. Contemporary Accounting Research 21 (2): 325352.

13 We are grateful to the editor for suggesting these questions.


http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1108/01409171011030390http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1108/01409171011030390http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1506/L20L-7FUM-FPCB-7BE2http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1506/L20L-7FUM-FPCB-7BE2http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1108/01409171011030390http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1108/01409171011030390

8/12/2019 ciia-50741

18/25

Ashton, R. H. 1974. An Experimental study of internal control judgments. Journal of Accounting Research 12 (1): 143157.

Auditing Standards Committee (ASC). 2011. RE: Concept Release on Possible Revisions to PCAOB StandardsRelated to Reports on Audited Financial . PCAOB Rulemaking Docket Matter No. 34; PCAOB Release No. 2011-003. Available at: http://pcaobus.org/Rules/Rulemaking/Docket034/Concept_Release.pdf

Bamber, E. M., T. D. Carpenter, and J. S. Hammersley. 2007. The Influence of Documentation Specificity and Fraud Risk Priming on Auditor Fraud Judgments and Evidence Evaluation Decisions . Working paper, J. M. Tull Schoolof Accounting and The University of Georgia Terry College of Business.

Bargh, J. A., and T. L. Chartrand. 1999. The unbearable automaticity of being. American Psychologist 54 (7): 462479.

Be dard, J. C., and L. E. Graham. 2002. The effects of decision aid orientation on risk factor identification and audit testplanning. Auditing: A Journal of Practice & Theory 21 (2): 3956.

Bell, T. B., and J. V. Carcello. 2000. A decision aid for assessing the likelihood of fraudulent financial reporting. Auditing: A Journal of Practice & Theory 19 (1): 169184.

Bell, T. B., J. C. Be dard, K. M. Johnstone, and E. F. Smith. 2002. KRisk(SM): A computerized decision aid for clientacceptance and continuance. Auditing: A Journal of Practice & Theory 21 (2): 97113.

Blocher, E., R. S. Esposito, and J. J. Willingham. 1983. Auditors analytical review judgments for payroll expense. Auditing: A Journal of Practice & Theory 3 (1): 7591.

Boatsman, J., C. Moeckel, and B. Pei. 1997. The effect of decision consequences on auditors reliance on decisionaids in audit planning. Organizational Behavior and Human Decision Processes 71 (2).

Bonner, S. E. 2008. Judgment and Decision Making in Accounting . Upper Saddle River, NJ: Pearson Education, Inc.Bonner, S. E., R. Libby, and M. W. Nelson. 1996. Using decision aids to improve auditors conditional probability

judgments. The Accounting Review 71 (2): 221240.Boritz, J. E. 1985. The effect of information presentation structures on audit planning and review judgments.

Contemporary Accounting Research l (2): 193218.Boritz, J. E., A. Albuquerque, and R. G. Kielstra. 1991. A prototype expert system for the assessment of inherent risk

and prior probability of error. In Decision Support and Expert Systems for Management Accountants , edited byArmitage, H. M., and J. E. Boritz. Toronto, ON: Society of Management Accountants of Canada.

Boritz, J. E., N. Kochetova-Kozloski, and L. A. Robinson. 2013. Are Fraud Specialists Relatively More Effective at Modifying Audit Programs in the Presence of Fraud Risk? Working paper, University of Waterloo School of Accounting and Finance and Saint Marys University Sobey School of Business.

Brazel, J. F., K. L. Jones, and M. F. Zimbelman. 2009. Using nonfinancial measures to assess fraud risk. Journal of Accounting Research 47: 11351166.

Butler, S. A. 1985. Application of a decision aid in the judgmental evaluation of substantive test of details samples.Journal of Accounting Research 23 (Autumn): 513526.

Carvalho, P. V. R., I. L. dos Santos, and M. C.R. Vidal. 2005. Nuclear power plant shift supervisors decision makingduring microincidents. International Journal of Industrial Ergonomics 35: 619644.

Carvalho, P. V. R., M. C.R. Vidal, and E. F. de Carvalho. 2007. Nuclear power plant communications in normative andactual practice: A field study of control room operators communications. Human Factors and Ergonomics inManufacturing 17 (1): 4378.

Cowperthwaite, P. 2012. Are checklists killing our profession? CA Magazine (September): 3839.Croskerry, P. 2007. Timely recognition and diagnosis of illness. In Safe and Effective: The Eight Essential Elements of

an Optimal Medication-Use System , edited by N. MacKinnon, 7993. Ottawa, ON: Canadian PharmacistsAssociation.

Dawes, R. 1971. A case study of graduate admissions: Application of three principles of decision making. AmericanPsychologist 26: 180188.

Deloitte LLP (Deloitte). 2012. Qualitative Goodwill Impairment Assessment . Available at: http://www.iasplus.com/en/publications/us/roadmap-series/qualitative-goodwill-impairment-assessment

Eining, M. M., D. R. Jones, and J. K. Loebbecke. 1997. Reliance on decision aids: An examination of auditorsassessment of management fraud. Auditing: A Journal of Practice & Theory 16 (2): 119.

Ely, J. W., M. L. Graber, and P. Croskerry. 2011. Checklists to reduce diagnostic errors. Academic Medicine 86 (3):307313.

Federal Aviation Administration (FAA). 2007. Section 12: Aircraft Checklists for 14 CFR parts 121/135 iFOFSIMSFAvailable at: http://fsims.faa.gov/PICDetail.aspx?docId 8900.1 ,Vol.3,Ch32,Sec12

Fischhoff, B., P. Slovic, and S. Lichtenstein. 1978. Fault trees: Sensitivity of estimated failure probabilities to problemrepresentation. Journal of Experimental Psychology: Human Perception and Performance 4 (2): 330344.


http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2490532http://pcaobus.org/Rules/Rulemaking/Docket034/Concept_Release.pdfhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/0003-066X.54.7.462http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2002.21.2.39http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2000.19.1.169http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2002.21.2.97http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1006/obhd.1997.2720http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1911-3846.1985.tb00376.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1475-679X.2009.00349.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1475-679X.2009.00349.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2490824http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1016/j.ergon.2005.01.010http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1002/hfm.20062http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1002/hfm.20062http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/h0030868http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/h0030868http://www.iasplus.com/en/publications/us/roadmap-series/qualitative-goodwill-impairment-assessmenthttp://www.iasplus.com/en/publications/us/roadmap-series/qualitative-goodwill-impairment-assessmenthttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1097/ACM.0b013e31820824cdhttp://fsims.faa.gov/PICDetail.aspx?docId=8900.1http://fsims.faa.gov/PICDetail.aspx?docId=8900.1http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/0096-1523.4.2.330http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/0096-1523.4.2.330http://fsims.faa.gov/PICDetail.aspx?docId=8900.1http://fsims.faa.gov/PICDetail.aspx?docId=8900.1http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1097/ACM.0b013e31820824cdhttp://www.iasplus.com/en/publications/us/roadmap-series/qualitative-goodwill-impairment-assessmenthttp://www.iasplus.com/en/publications/us/roadmap-series/qualitative-goodwill-impairment-assessmenthttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/h0030868http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/h0030868http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1002/hfm.20062http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1002/hfm.20062http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1016/j.ergon.2005.01.010http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2490824http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1475-679X.2009.00349.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1475-679X.2009.00349.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1911-3846.1985.tb00376.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1006/obhd.1997.2720http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2002.21.2.97http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2000.19.1.169http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2002.21.2.39http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/0003-066X.54.7.462http://pcaobus.org/Rules/Rulemaking/Docket034/Concept_Release.pdfhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2490532

8/12/2019 ciia-50741

19/25

Gawande, A. 2007. The checklist. New Yorker Annals of Medicine . (December 10).Gawande, A. 2009. The Checklist Manifesto: How to Get Things Right . New York, NY: Metropolitan Books.Glover, S. M., D. F. Prawitt, and B. C. Spilker. 1997. The influence of decision aids on user behavior: Implications for

knowledge acquisition and inappropriate reliance. Organizational Behavior and Human Decision Processes 72(2): 232255.

Gomaa, M. I., J. E. Hunton, and J. M. Rose. 2008. The effects of control risk and litigation risk on decision aid reliance.International Journal of Applied Decision Sciences 1 (1): 80106.

Gramling, A. A., and M. G. Watson. 2009. Analysis of peer review reports: A focus on deficiencies of the top-20triennially inspected firms. Current Issues in Auditing 3 (2): A1A14.

Hammersley, J. S. 2011. A review and model of auditor judgments in fraud-related planning tasks. Auditing: A Journal of Practice & Theory 30 (4): 101128.

Hammond, K., and D. Summers. 1972. Cognitive control. Psychological Review 79: 5867.Haynes, A. B., T. G. Weiser, W. R. Berry, S. R. Lipsitz, A-H. S. Breizat, E. P. Dellinger, T. Herbosa, S. Joseph, P. L.

Kibatala, M. C. M. Lapitan, A. F. Merry, K. Moorthy, R. K. Reznick, B. Taylor, and A. A. Gawande. 2009. Asurgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine 360: 491499.

Hoffman, V. B., and M. F. Zimbelman. 2009. Do strategic reasoning and brainstorming help auditors change their standard audit procedures in response to fraud risk? The Accounting Review 84 (3): 811837.

Hoffman, V. B., and M. F. Zimbelman. 2012. How strategic reasoning and brainstorming can help auditors detectfraud. Current Issues in Auditing 6 (2): P25P33.

Hogan, C. E., Z. Rezaee, R. A. Riley, Jr., and U. K. Velury. 2008. Financial statement fraud: Insights from theAcademic Literature. Auditing: A Journal of Practice &Theory 27 (2): 231252.

Jamal, K. 2008. Mandatory audit of financial reporting: A failed strategy for dealing with fraud. Accounting Perspectives 7 (2): 97110.

Jennings, M., D. C. Kneer, and P. M. J. Reckers. 1993. The significance of audit decision aids and pre-case juristsattitudes on perceptions of audit firm culpability and liability. Contemporary Accounting Research 9 (2): 489507.

Kahneman, D. 2011. Thinking, Fast and Slow . New York, NY: Doubleday.Kahneman, D., D. Lovallo, and O. Sibony. 2011. Before you make that big decision. Harvard Business Review (June):

5060.Kaplan, S. E., J. H. Reneau, and S. Whitecotton. 2001. The effects of predictive ability information, locus of control,

and decision maker involvement on decision aid reliance. Journal of Behavioral Decision Making 14: 3550.Kerr, N. L., and R. S. Tindale. 2004. Group performance and decision making. Annual Review of Psychology 55: 623

655.Knechel, W. R., G. V. Krishnan, M. Pevzner, L. B. Shefchik, and U. K. Velury. 2013. Audit quality: Insights from the

academic literature. Auditing: A Journal of Practice & Theory 32 (S1): 385421.KPMG. 2011. Elevating Professional Judgment in Auditing . Montvale, NJ: KPMG.Libby, R., and P. A. Libby. 1989. Expert measurement and mechanical combination in control reliance decisions. The

Accounting Review 64 (4): 729747.Lowe, J., P. Reckers, and S. Whitecotton. 2002. The effect of decision aid use and reliability on jurors evaluations of

auditor liability. The Accounting Review 77: 185202.Marley, R. N. 2011. An Empirical Investigation of Decision Aids to Improve Auditor Effectiveness in Analytical Review .

Theses and Dissertations Paper 3232. Available at: http://scholarcommons.usf.edu/etd/3232McDaniel, L. S. 1990. The effects of time pressure and audit program structure on audit performance. Journal of

Accounting Research 28 (2): 267285.McKee, T. 2010. The cry wolf problem in current fraud auditing standards. The CPA Journal (January): 6062.Nelson, M., and H.-T. Tan. 2005. Judgment and DM research in auditing: A task, person, and interpersonal interaction

perspective. Auditing: A Journal of Practice & Theory 24 (S): 4172.Ng, T. B.-P., and H.-T. Tan. 2003. Effects of authoritative guidance availability and audit committee effectiveness on

auditors judgments in an auditor-client negotiation context. The Accounting Review 78 (3): 801818.Pincus, K. V. 1989. The efficacy of a red flags questionnaire for assessing the possibility of fraud. Accounting,

Organizations and Society 14 (1/2): 153163.PricewaterhouseCoopers LLP (PwC). 2011. Qualitative Goodwill Impairment Assessment . Available at: http://www.

pwc.com/us/en/audit-assurance-services/valuation/publications/goodwill-impairment-testing.jhtmlPronovost P., D. Needham, S. Berenholtz, D. Sinopoli, H. Chu, S. Cosgrove, B. Sexton, R. Hyzy, R. Welsh, G. Roth, J.

Bander, J. Kepros, and C. Goeschel. 2006. An intervention to decrease catheter-related bloodstream infectionsin the ICU. New England Journal of Medicine 355: 27252732.


http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1006/obhd.1997.2735http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1504/IJADS.2008.017953http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ciia.2009.3.2.A1http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ajpt-10145http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ajpt-10145http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/h0031851http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1056/NEJMsa0810119http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1056/NEJMsa0810119http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/accr.2009.84.3.811http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ciia-50283http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2008.27.2.231http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1506/ap.7.2.2http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1506/ap.7.2.2http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1911-3846.1993.tb00894.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1002/1099-0771(200101)14:13.0.CO;2-Dhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1146/annurev.psych.55.090902.142009http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ajpt-50350http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/accr.2002.77.1.185http://scholarcommons.usf.edu/etd/3232http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2491150http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2491150http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2005.24.s-1.41http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/accr.2003.78.3.801http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1016/0361-3682(89)90039-1http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1016/0361-3682(89)90039-1http://www.pwc.com/us/en/audit-assurance-services/valuation/publications/goodwill-impairment-testing.jhtmlhttp://www.pwc.com/us/en/audit-assurance-services/valuation/publications/goodwill-impairment-testing.jhtmlhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1056/NEJMoa061115http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1056/NEJMoa061115http://www.pwc.com/us/en/audit-assurance-services/valuation/publications/goodwill-impairment-testing.jhtmlhttp://www.pwc.com/us/en/audit-assurance-services/valuation/publications/goodwill-impairment-testing.jhtmlhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1016/0361-3682(89)90039-1http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1016/0361-3682(89)90039-1http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/accr.2003.78.3.801http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2005.24.s-1.41http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2491150http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2307/2491150http://scholarcommons.usf.edu/etd/3232http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/accr.2002.77.1.185http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ajpt-50350http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1146/annurev.psych.55.090902.142009http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1002/1099-0771(200101)14:13.0.CO;2-Dhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1111/j.1911-3846.1993.tb00894.xhttp://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1506/ap.7.2.2http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1506/ap.7.2.2http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/aud.2008.27.2.231http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ciia-50283http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/accr.2009.84.3.811http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1056/NEJMsa0810119http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1056/NEJMsa0810119http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1037/h0031851http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ajpt-10145http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ajpt-10145http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.2308/ciia.2009.3.2.A1http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1504/IJADS.2008.017953http://localhost/var/www/apps/conversion/tmp/scratch_5/dx.doi.org/10.1006/obhd.1997.2735

8/12/2019 ciia-50741

20/25

Public Company Accounting Oversight Board (PCAOB). 2001. AT Section 101 Attest Engagements . Available at:http://pcaobus.org/Standards/Attestation/Pages/AT101.aspx#ps-pcaob_eb7d73cb-d5be-41e4-9242-5f204591ad07

Public Company Accounting Oversight Board (PCAOB). 2006. Testimony Concerning Sarbanes-Oxley at Four:Protecting Investors and Strengthening the Markets . Available at: http://pcaobus.org/News/Speech/Pages/09192006_OlsonHouseFinancialServicesCommittee.aspx

Public Company Accounting Oversight Board (PCAOB). 2012. Current Trends and Issues in Public Company Auditing . Available at: http://pcaobus.org/News/Speech/Pages/10262012_Franzel.aspx

Public Company Accounting Oversight Board (PCAOB). 2013. In the Matter of Ernst and Young LLPs Quality Control Remediation Submission . PCAOB Release No. 104-2013-087. Available at: http://pcaobus.org/Inspections/Documents/05232013_EYReportStatement.pdf

Raiffa, H. 1968. Decision Analysis . Reading, MA: Addison-Wesley.Romney, M. B., W. S. Albrecht, and D. J. Cherrington. 1980. Auditors and the detection of fraud. Journal of

Accountancy 149 (5): 6369.Rose, A. M., and J. M. Rose. 2003. The effects of fraud risk assessments and a risk analysis decision aid on auditors

evaluation of evidence and judgment. Accounting Forum 27 (3): 312338.Rose, J. M., B. A. McKay, C. S. Norman, and A. M. Rose. 2012. Designing decision aids to promote the development

of expertise. Journal of Information Systems 26 (1): 734.Seow, J.-L. 2009. Cue usage in financial statement fraud risk assessments: Effects of technical knowledge and

decision aid use. Accounting and Finance 49: 183205.

Seow, P.-S. 2011. The effects of decision aid structural effectiveness on decision making outcomes. International Journal of Accounting Information Systems 12: 4056.Shelton, S. W., O. R. Whittington, and D. Landsittel. 2001. Auditing firms fraud risk assessment practices. Accounting

Horizons 15 (1): 1933.Sieck, W. R., and H. R. Arkes. 2005. The recalcitrance of overconfidence and its contribution to decision aid neglect.

Journal of Behavioral Decision Making 18: 2953.Sloman, S. A. 1996. The empirical case for two systems of reasoning. Psychological Bulletin 119 (

ciia-50741

Documents

Transcript of ciia-50741