Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of...
-
Upload
julissa-miner -
Category
Documents
-
view
213 -
download
0
Transcript of Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of...
![Page 1: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/1.jpg)
Universität Hamburg Institut für Wirtschaftsinformatik
Prof. Dr. D.B. Preßmar
Final Results of the NN3 Neural Network Forecasting Competition
Sven F. Crone, Konstantinos Nikolopoulos and Michele Hibon
![Page 2: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/2.jpg)
Can NN modelling be automated for business forecasting? Evaluate progress in NN modelling since M3 Disseminate Explicit knowledge on “best practices”
2005 SAS & IIF Grant2005 SAS & IIF Grant
RATIONAL
OBJECTIVES
RESULTS
DISCUSSION
FURTHER RESEARCH
![Page 3: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/3.jpg)
2005 SAS & IIF Grant2005 SAS & IIF Grant
RATIONAL
![Page 4: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/4.jpg)
Only 1 evaluation of NN within Forecasting Competitions Distinct fields of research and participation
NN: breakthrough or passing fad?NN: breakthrough or passing fad?
Reid1969
Santa Fe1991
BUSINESS FORECASTING COMPETITIONS
NN COMPETITIONS
Suykens1998
Reid1972
Newbold & Granger1974
Makridakis & Hibon1979
M-Competition 1982
M2-Competition 1988
M3-Competition 2000
H-Competition,Hibon 2006
EUNITE2001
ANNEXG2001
BI Cup2003
CATS2005ISF052005
ISF06 ANNEX 2006
WCCI2006
Only 1 NN entryBalkin & Ord
![Page 5: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/5.jpg)
Most NN competitions = classification (EUNITE’02, WCCI06 etc.) Limited evidence on Regression evaluations
Visit http://www.neural-forecasting.com/competition_data.htm
CI Time Series CompetitionsCI Time Series Competitions
Time Series Data Format Length Submis.
SANTA FE 1991Gershenfeld & Weigend
2 univariate4 multivariate
UV: Laser, UV: Artificial, Sleep, Exchange rate, Astrophysics, Music
1000, 34000, 300000, 100000,
27704, 380830+
Black Box 1998 Suykens & Vandewalle
1 univariate Physics2000
(1000)17
EUNITE 2001 1 multivariate Electrical Load35040(31)
56
ANNEXG 2001Dawson et al.
1 multivariate Hydrology1460 pointsHydrology
12
BI Cup 2003Weber
1 multivariate Sugar sales365 days
(14)10+
CATS 2005, IEEELendasse,
1 univariatein 5 parts
Artificial4905pointas
(95 points, 5*19)25
ISF2005Crone
2 univariate Airline, M3-Competition 144, 85 16
ANNEXG / ISF2006Dawson et al., Crone
3 multivariate Hydrology1460 pointsHydrology
12
WCCI 2006 Predictive Uncertainty, Gawley
1 univariate3 multivariate
UV: Synthetic, Precipitation, Temperature, SO2
380, 10000, 10000, 21000
9
![Page 6: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/6.jpg)
Conduct competition on industry data Evaluate different NN methodologies Can NN forecasting be AUTOMATED on many time series?
Reasons? Modelling DecisionsReasons? Modelling Decisions
Gap between forecasting & NN domains NN evaluations on different data types No positive evidence on M-type data
• Short time series• Noisy time series
Discouraging research findings NN cannot forecast seasonal time series No valid & reliable methodology to model NN No automation of NN modelling possible
![Page 7: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/7.jpg)
Can NN modelling be automated for business forecasting? Evaluate progress in NN modelling since M3 Disseminate Explicit knowledge on “best practices”
2005 SAS & IIF Grant2005 SAS & IIF Grant
OBJECTIVESa) What is the performance (accuracy, robustness
& resources) of NN in comparison to established forecasting methods?
b) What are the current “best practice” methodologies utilised by researchers to model NN for time series forecasting
![Page 8: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/8.jpg)
Multiple Hypothesis Testing similar to M3-competition
Competition DesignCompetition Design
Multiple empirical Time Series Complete set of 111 time series Reduced set of 11 time series Representative structures monthly industry data
• long & short time series• Seasonal and non-seasonal series
Scaled observations for anonymity No domain knowledge 18 steps ahead forecasts
Simulated ex ante (out of sample) evaluation
Multiple error measures & computational time
Testing of conditions under which NN perform well/bad
NN3 COMPETITION
![Page 9: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/9.jpg)
Competition DesignCompetition Design
46 Submissions for the reduced dataset
9 benchmarks
22 submissions for the complete dataset
8 benchmarks
SubmissionsSubmissions
![Page 10: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/10.jpg)
2005 SAS & IIF Grant2005 SAS & IIF Grant
Automated AI/CI approaches can very well do the job! (batch forecasting)
Balkin’s and Ord approach was not very ‘bad’ after all..
Performance was verified across many metrics (including MASE), parametric + non-parametric
Performance was verified with multiple hypothesis: long/short, seasonal/non seasonal, difficult/easy
So… WHAT do we know NOW that we did not knew before NN3?
![Page 11: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/11.jpg)
2005 SAS & IIF Grant2005 SAS & IIF Grant
Time Series Benchmarks are very hard to beat! Forecast Pro, Theta model and Marc Wildi’s Stat benchmark outperform overall all CI/AI approachesFor the ‘harder’ part of the NN3 dataset – 25 short+non-seasonal series – CI approaches managed to outperform all other approaches!! Full automation seems to be possible in large scale forecasting tasks
+ Side results… New Stat benchmarks that perform outstandinglyImprovement of established forecasting engines in the last 10
years
So… WHAT do we know NOW that we did not knew before NN3?
![Page 12: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/12.jpg)
+
![Page 13: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/13.jpg)
+
+
![Page 14: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/14.jpg)
+
![Page 15: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/15.jpg)
![Page 16: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/16.jpg)
![Page 17: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/17.jpg)
![Page 18: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/18.jpg)
![Page 19: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/19.jpg)
![Page 20: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/20.jpg)
2005 SAS & IIF Grant2005 SAS & IIF Grant
Computational times….
Leaders of the field (Academia + Commercial)
Time series features that would necessitate the use of AI/CI approaches
Replication in a competition of the M3 volume (NN5…111, tourism competition…1000+)
Best practices?
Full automation??
and… WHAT we still DO NOT…
![Page 21: Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone,](https://reader035.fdocuments.in/reader035/viewer/2022070411/56649c7d5503460f94932205/html5/thumbnails/21.jpg)
?Sven, Kostas & Michele