Post on 08-Apr-2018
8/7/2019 Evaluating Trading Model Results With Statistical Process Control Methods
http://slidepdf.com/reader/full/evaluating-trading-model-results-with-statistical-process-control-methods 1/5
EvaluatingTradingModelResults
withStatisticalProcessControlMethods
MSF 585
Alex Bolotov3/2/2011
8/7/2019 Evaluating Trading Model Results With Statistical Process Control Methods
http://slidepdf.com/reader/full/evaluating-trading-model-results-with-statistical-process-control-methods 2/5
Statistical process control (SPC) is the application of statistical methods
to the monitoring and control of a process to ensure it operates at its full potential
to produce expected results. In October 1984 Journal of Quality Technology
published an article by Lloyd Nelson where methods for this control were defined.
Nelson rules are a method in process control of determining if some measured
variable is out of control (unpredictable versus consistent). We are interested in
applying those rules to our trading model to determine the power of the model to
produce repeatable results over the time. Our goal is to detect when our model falls
into “out-of-control” condition. “Out-of-control” condition is best described as
unpredictable and non-random event. More “out-of-control” conditions in our
system’s performance we observe, less confident we are a favorable future
behavior.
Nelson identified 8 rules for detecting “out-of-control” condition. The rules arebased around the mean value and standard deviation of the samples. In our
example we explore 3 situations.
8/7/2019 Evaluating Trading Model Results With Statistical Process Control Methods
http://slidepdf.com/reader/full/evaluating-trading-model-results-with-statistical-process-control-methods 3/5
First check will give us a warning about a sample that’s grossly out of control.
Second rule will warn us about a potential regime shift and indicate an existence of
some prolonged bias in the data. Third rule alarms us about potential lead up to the
first event (that is large spike).
Trading Model. Simple (that is no rebalancing) carry trade model was picked for
the experiment. During first two years of data, 01/01/2000 to 01/01/2002, data on
8/7/2019 Evaluating Trading Model Results With Statistical Process Control Methods
http://slidepdf.com/reader/full/evaluating-trading-model-results-with-statistical-process-control-methods 4/5
exchange rates of several major currency crosses and interest rates was collected.
Holding position was optimized based on that data.
Currency Trading Positions
Currency
Short Yield Currency
Long Yield
CHF -8.30% 0.50
%
AUD 47.00
%
3.80
%CAD -2.00% 1.00
%
CZK 11.00
%
1.20
%DKK -
53.00%
1.00
%
HUF 95.00
%
5.15
%GBP -7.00% 0.03
%
JPY 20.00
%
0.50
%NOK -
13.00%
1.10
%
EUR 1.40% 1.40
%NZD -5.00% 3.00
%
MXN 72.00
%
4.00
%SEK -
33.00%
1.40
%
PLN 3.00% 4.00
%SGD -
93.00%
0.20
%
ZAR 26.00
%
5.00
%USD -
61.10%
0.50
%
Total -
275.40%
-
1.84%
Total 275.4
0%
11.2
3%
The strategy expects to earn about 9.4% annually with standard deviation of about
7%.
During test period, 01/01/2000 to 01/01/2002, we calculate average 5 day returns
of the model and use them to define Lower Control Limit (LCL) and Upper Control
Limit (UCL) as values 3 standard deviations away from the mean return. Graph
below charts our 5 day returns from 2000 to the beginning of 2011 with values
approximately +1% and -1% being LCL and UCL. We can confirm that model
performed as we expected until 10/11/2008 when series of exceptionally large
returns (positive and negative) signaled of a rapid regime shift in a model behavior.
Another measure of models return is average range during consecutive 5 day
periods. Just as before, control limits for range are three standard deviations above
8/7/2019 Evaluating Trading Model Results With Statistical Process Control Methods
http://slidepdf.com/reader/full/evaluating-trading-model-results-with-statistical-process-control-methods 5/5
and below the expected return. Chart below graphs 5 day ranges versus an Upper
Limit Control (just below 5%). Several violations of that threshold prior to
10/11/2008 were positive for the model and did not require adjustments.
Two other tests were also performed in an attempt to detect early regime shifts.
Using MATLAB I tried to find out an occurrence where 9 or more points were on one
side of the mean. Observing such a cluster of results one could suspect a presence
of a bias in trading model. None such occurrence was detected. Third test tried to
detect multiple observations above 2 standard deviations in a row. Yet again, prior
to 10/11/2008 there were no such occurrences. After 10/11/2008, however, there
were multiple situations when model’s return fall above 2 standard deviations for
several periods in a row.
Just as in the original paper, we confirmed that this type of model, if properly
constructed, tends to behave in the expected way. Low volatility with consistent
results makes this type of strategy an attractive investment opportunity. On the flip
side, we were not able to detect any warning signs that would alarm us about
upcoming volatility swings.
Next chart demonstrates growth of the fully invested portfolio. Between 01/01/2002
and 10/10/2008 we realized approximately 10% annual rate of return with expected
volatility. Once the system generated a 3 Sigma return, we were essentiallystopped-out.
We demonstrated that SPC methods can be successfully applied for evaluation of
trading model’s performance results. Not only can we get a clear answer whether or
not our system have “repeatability guarantee”, but they can also pinpoint a definite
location when we need to re-evaluate our model.