wissen.nutzen.

WebScraping Notices of Insolvency Proceedings with R

Using publicly available data to enhance survey response quality

uRos2019, 20./21. May 2019, București

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

Insolvency Proceedings and Official Statistics

2

© Federal Statistical Office of Germany (Destatis)

Debtor Insolvency Court

Insolvency Administrator

Official Statistics

Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

Insolvency Proceedings and Official Statistics

3

© Federal Statistical Office of Germany (Destatis)

Opening of Insolvency Proceedings

Closing of Insolvency Proceedings

Residual Debt Discharge Granted or not Granted

Residual Debt Discharge Revoked

Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

Problem: Two Respondents on One Case

4

© Federal Statistical Office of Germany (Destatis)

≠

Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

A Further Source: Publication Obligations of Insolvency Courts

5

© Federal Statistical Office of Germany (Destatis) Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

www.insolvenzbekanntmachungen.de

6

© Federal Statistical Office of Germany (Destatis) Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

Unrestricted Search

7

An unrestricted search, "- All insolvency courts -", by announcements is possible according to § 2 of the regulation to public announcements in insolvency proceedings in the Internet only within two weeks after the first day of publication. After expiry of this period, only a detailed search is permitted.

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

Unrestricted Search

8

An unrestricted search, "- All insolvency courts -", by announcements is possible according to § 2 of the regulation to public announcements in insolvency proceedings in the Internet only within two weeks after the first day of publication. After expiry of this period, only a detailed search is permitted.

To achieve a complete picture of the opened proceedings, one needs to search every two weeks

Cron Job

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

Data Collection

9

Objective: Complete list of identifiers (case number, court) of opened proceedings, as well as of proceedings with statistically relevant decisions.

Path: R-script that, within the two-week period, searches the texts of the notices in the unrestricted search for certain keywords and notes the corresponding case/court identifiers.

© Federal Statistical Office of Germany (Destatis) Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

Data Collection

10

First collection in March 2016. Since then, full picture of openings and monthly lists of relevant procedures for which notifications should be received.

Currently about 3.7 million decisions have been analysed …

© Federal Statistical Office of Germany (Destatis) Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

11

© Federal Statistical Office of Germany (Destatis)

0

20.000

40.000

60.000

80.000

100.000

120.000

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

2017 2018 2019

Number of Court Publications

wissen.nutzen.

12

© Federal Statistical Office of Germany (Destatis)

0

2.000

4.000

6.000

8.000

10.000

12.000A

pril

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

May

June July

Aug

ust

Sep

tem

ber

Oct

ober

Nov

embe

r

Dec

embe

r

Janu

ary

Febr

uary

Mar

ch

Apr

il

2016 2017 2018 2019

Num

ber o

f Ope

ned

Inso

lven

cy P

roce

edin

gs

Number of Opened Insolvency Proceedings ...

Difference … Court Publications … Official Statistics

wissen.nutzen.

13

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

What to do with the Data?

14

Main Goal:

Use data to get information about court decisions that should trigger a report by the insolvency administrators …

… so we can remind them that a report is due.

© Federal Statistical Office of Germany (Destatis)

?

Icons made by Eucalyp, Kiranshastry and Freepik from www.flaticon.com are licensed by CC 3.0 BY

wissen.nutzen.

What to do with the Data?

15

Identification of relevant court decisions is done by applying a set of regular expressions to the full text of the decisions.

Resulting lists are reasonably short and can be matched with incoming reports.

Staff can use the information to contact insolvency administrators efficiently.

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

What we cannot do?

16

Privacy matters.

Court decisions contain personal data of debtors and insolvency administrators. Therefore full texts of court decisions are not stored.

Only Case-IDs as well as type and time of decision are stored.

Emojis by Noto Color Emoji

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

Pit Falls

17

Q: What if one day the service is relaunched?

A: A legal basis for direct data access needs to be formulated.

Picture by Albert Uderzo

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

Possible Improvements

18

Keyword search currently via regular expressions. Are there any smarter ways to analyze text?

Since we have collected a training set, could it be done with supervised learning approaches?

Emojis by Noto Color Emoji

© Federal Statistical Office of Germany (Destatis)

wissen.nutzen.

Contact

19

Joerg Feuerhake

joerg.feuerhake@destatis.de

Tel.: 0049 611 75 4116

© Federal Statistical Office of Germany (Destatis)