Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab:...
Transcript of Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab:...
![Page 2: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/2.jpg)
Page 2
European Court of Auditors
• The ECA is an EU institution foundedin 1977 and based in Luxembourg.
• It has 28 Members, one from each EU country.
• Our 900 staff are nationals of all EUcountries. Our auditors include experts in accountancy, economics, public administration and law.
![Page 3: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/3.jpg)
Page 3
Our role among EU institutions
![Page 4: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/4.jpg)
Page 4
Checking where the EU money goes
• We audit whether policies funded from the EU budget are implemented correctly and achieve value for money.
• Financial audits • checking whether the EU’s annual accounts are correct.
• Compliance audits • checking whether EU revenue and expenditure comply with the rules.
• Performance audits • checking whether EU money is well spent.
![Page 5: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/5.jpg)
Page 5
Challenges in EU audit
• Creating more added value in our audit work, better assurance for the European taxpayer
• Different national legal frameworks in Member States
• Different grant management systems, different languages
• Demand for auditing entire populations as external auditor
• We are not data producers, we have to work with what we get from auditees
![Page 6: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/6.jpg)
Page 6
Challenges in EU audit
• „We should not await some external magical solution to show us the way forward. There are ‘no magic recipes’ ready for use, but rather several technological options that we should start testing, experimenting with and embedding in our audit practice.”
Tytti Yli-Viikari, Auditor General of Finland
• „If you can’t describe what you are doing as a process, you don’t know what you’re doing.”
W. Edwards Deming
![Page 7: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/7.jpg)
Page 7
ECALab: bringing new technologies to auditors since 2017
![Page 8: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/8.jpg)
Page 8
Process Mining in public audit
Inspirations for using process mining in EU audit (both for compliance and performance).
Process data is everywhere!
Inspirations for using process mining in EU audit (both for compliance and performance).
Process data is everywhere!
![Page 9: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/9.jpg)
Page 9
How to get started
• Find a specific and well-structured use case• Obtain data and consult your Data Protection Officer• Use visualisation to communicate the results• Translate the results into business objectives / performance indicators• Work together with domain experts• Anticipate an iterative process between data analysts and auditors
![Page 10: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/10.jpg)
Page 10
4 examples from our process mining journey
1. Performance audit of European Public Consultations
2. Audit Automation Pilot for EU Executive Agencies
3. Analysing the core business process of an EU agency
4. Analysing the grant management system of a Member State authority
From own data to standardised external data and non-standard external data
![Page 11: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/11.jpg)
Page 11
1. Performance auditing of European Public Consultations
• Public consultations allow European citizens to express their views on EU action (e.g. summertime consultation, with 4,6 million respondents)
• Analysing the sequence of events in a consultation was part of the overall audit• Well-structured, small dataset, compiled by our auditors• Process Mining was used as proof of concept, for internal presentations
Special report no 14/2019: ‘Have your say!’: Commission’s public consultations engage citizens, but fall short of outreach activitieshttps://www.eca.europa.eu/en/Pages/DocItem.aspx?did=50895
![Page 12: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/12.jpg)
Page 12
First pilot, European Public Consultations (2019)
![Page 13: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/13.jpg)
Page 13
Process conformance checking on automatically inferred model
3 public consultations didn’t have a synopsis report
2 public consultations had additional steps after the legislative proposal
![Page 14: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/14.jpg)
Page 14
Performance auditing of European Public Consultations
• Outcome:
• Process Mining was widely accepted and understood at all levels of the organisation
![Page 15: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/15.jpg)
Page 15
2. Audit automation pilot for EU Executive Agencies
• Executive agencies are set up for a limited period of time to manage specific tasks related to EU programmes (e.g. Research Executive Agency)
• Extract workflow data from the Commission’s financial information system• Supplementary information to auditors for risk analysis• Pilot for developing new audit methodology (using automated checks, RPA,
process mining and visual analytics)
ECA Journal 1/2020– BIG DATA & digital audithttps://www.eca.europa.eu/Lists/ECADocuments/JOURNAL20_01/JOURNAL20_01.pdf
![Page 16: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/16.jpg)
Full population audit – process flowchart
![Page 17: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/17.jpg)
Full population audit – process flowchart
![Page 18: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/18.jpg)
Full population audit - exceptions
![Page 19: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/19.jpg)
Full population audit - performance
![Page 20: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/20.jpg)
ABAC workflow data
Page 20
Violation of segregations of duties occurs twice in a specific sub-population, and for cases having a total duration over a year
Violation of segregations of duties occurs twice in a specific sub-population, and for cases having a total duration over a year
Full population audit – segregation of duties
Agent 5 acted as verifying AND authorising officer. (Later checks proved that he had the necessary authorisations to do so.)
Agent 5 acted as verifying AND authorising officer. (Later checks proved that he had the necessary authorisations to do so.)
![Page 21: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/21.jpg)
Page 21
Creating a social network based on the event log
These are authorizing officers, who act as gateways between normal employees and the accounting system (the transactions go through them for processing, but they don’t otherwise interact with other agents)
These are authorizing officers, who act as gateways between normal employees and the accounting system (the transactions go through them for processing, but they don’t otherwise interact with other agents)
![Page 22: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/22.jpg)
Page 22
2. Audit automation pilot for EU Executive Agencies
• Outcome:
• Successfully proved that Process Mining can be directly used for audit purposes on structured data extracts from the European Commission’s accounting system
• We can easily compare the processes of different organisations that use the same data structure
• Effortless solution for detecting trends/patterns and zooming in on areas of interest (great for risk assessment!)
![Page 23: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/23.jpg)
Page 23
3. Analysing the core business process of a specific European Agency
• This Agency registers certain types of claims from clients, but clients can also contest other clients’ claims if they suspect infringement
• In the framework of our annual audit, we wanted to better understand their core processes and how the claims are managed
• Their system is fairly complex with extensive documentation• We received a well-structured database extract from the Agency’s workflow
management system
2018 audit of EU agencies in briefhttps://www.eca.europa.eu/en/Pages/DocItem.aspx?did=51300
![Page 24: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/24.jpg)
Process mining in practice – raw data
Page 24
![Page 25: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/25.jpg)
Page 25
Process mining in practice – raw data
• Raw data often times have to be pre-processed (and/or simplified)
• Check event histograms for event distribution• Identify the real case ID• Check with domain experts• Merge/consolidate or split/enrich
• As an external auditor, we have to work with database extracts / flat files• Customs scripts (Python with Pandas or Dask) for data preparation
![Page 26: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/26.jpg)
Process mining in practice – after simplification
Page 26
![Page 27: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/27.jpg)
These are in fact two different processes merged into one
Page 27
![Page 28: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/28.jpg)
And we finally arrive at the process that is in the documentation!
Page 28
![Page 29: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/29.jpg)
Page 29
3. Analysing the core business process of a specific European Agency
• Outcome:
• We proved that Process Mining can quickly analyse complex systems and processes using external data from an auditee
• Process Mining can result in significant time savings compared to a lengthy analysis of documents and flowcharts
• Process Mining can provide certain insights that would otherwise be hidden from an external auditor using traditional methods
![Page 30: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/30.jpg)
Page 30
4. Process analysis of a paying agency in an EU Member State
• Pilot for auditing the grant application process in a Member State• We received a database extract from their system, that reflects their own
business logic, not our data needs for process mining• The structure of the data was not directly suitable for process mining:
• Transposition was required• Repetitions were missing• Instead of „activities” the data contained process „checkpoints”
• Extensive pre-processing needed
ECA Journal 1/2020– BIG DATA & digital audithttps://www.eca.europa.eu/Lists/ECADocuments/JOURNAL20_01/JOURNAL20_01.pdf
![Page 31: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/31.jpg)
Process mining as a tool for visual analytics
Page 31
![Page 32: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/32.jpg)
Page 32
4. Process analysis of a paying agency in an EU Member State
• Outcome:
• Legacy systems used by public administrations may not be able to provide the full spectrum of data needed for Process Mining
• Nevertheless, process mining could still provide meaningful insights on:• Segregation of duties• Observation of deadlines• Proportion of beneficiaries selected for on-the-spot checks• Indirect loop observations based on timestamp anomalies
• This pilot allowed us to plan ahead and anticipate future challenges
![Page 33: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/33.jpg)
Page 33
Lessons learned
• We don’t need a hypothesis, we need to understand the data we are using
![Page 34: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/34.jpg)
Page 34
Lessons learned
• We don’t need a hypothesis, we need to understand the data we are using
• Data pre-processing can make or break the analysis
![Page 35: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/35.jpg)
Page 35
Lessons learned
• We don’t need a hypothesis, we need to understand the data we are using
• Data pre-processing can make or break the analysis
• The investment in understanding and pre-processing the data will pay off in subsequent audit tasks
![Page 36: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/36.jpg)
Page 36
Lessons learned
• We don’t need a hypothesis, we need to understand the data we are using
• Data pre-processing can make or break the analysis
• The investment in understanding and pre-processing the data will pay off in subsequent audit tasks
• Arriving at audit findings is an iterative process by a multi-disciplinary team, involving the data analyst, domain experts and the auditee
![Page 37: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/37.jpg)
Page 37
Conclusion
• „Process mining is one of the most mature technological options for auditors to extract evidential value from data and data flows, especially (but not only) for financial and compliance audit purposes.”
(Jans et al., 2014; Werner, 2017)
![Page 38: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/38.jpg)
Conclusion
Page 38
https://www.eca.europa.eu/Lists/ECADocuments/JOURNAL20_01/JOURNAL20_01.pdf
![Page 39: Process Mining in EU audit - Fluxicon · 2020. 10. 20. · W. Edwards Deming. Page 7. ECALab: bringing new technologies to auditors since 2017. Page 8. Process Mining in public audit.](https://reader036.fdocuments.in/reader036/viewer/2022071414/610e540027e1b7086344cde6/html5/thumbnails/39.jpg)
QUESTIONS?
Page 39
Email: [email protected]: www.linkedin.com/in/zsvarga/GitHub: github.com/zseebrzTableau Public: public.tableau.com/profile/zsolt.varga#!/