Evaluating Trading Model Results With Statistical Process Control Methods

8/7/2019 Evaluating Trading Model Results With Statistical Process Control Methods

http://slidepdf.com/reader/full/evaluating-trading-model-results-with-statistical-process-control-methods 1/5

EvaluatingTradingModelResults

withStatisticalProcessControlMethods

MSF 585

Alex Bolotov3/2/2011



Statistical process control (SPC) is the application of statistical methods

to the monitoring and control of a process to ensure it operates at its full potential

to produce expected results. In October 1984 Journal of Quality Technology

published an article by Lloyd Nelson where methods for this control were defined.

Nelson rules are a method in process control of determining if some measured

variable is out of control (unpredictable versus consistent). We are interested in

applying those rules to our trading model to determine the power of the model to

produce repeatable results over the time. Our goal is to detect when our model falls

into “out-of-control” condition. “Out-of-control” condition is best described as

unpredictable and non-random event. More “out-of-control” conditions in our

system’s performance we observe, less confident we are a favorable future

behavior.

Nelson identified 8 rules for detecting “out-of-control” condition. The rules arebased around the mean value and standard deviation of the samples. In our

example we explore 3 situations.



First check will give us a warning about a sample that’s grossly out of control.

Second rule will warn us about a potential regime shift and indicate an existence of

some prolonged bias in the data. Third rule alarms us about potential lead up to the

first event (that is large spike).

Trading Model. Simple (that is no rebalancing) carry trade model was picked for

the experiment. During first two years of data, 01/01/2000 to 01/01/2002, data on



exchange rates of several major currency crosses and interest rates was collected.

Holding position was optimized based on that data.

Currency Trading Positions

Currency

Short Yield Currency

Long Yield

CHF -8.30% 0.50

%

AUD 47.00

%

3.80

%CAD -2.00% 1.00

%

CZK 11.00

%

1.20

%DKK -

53.00%

1.00

%

HUF 95.00

%

5.15

%GBP -7.00% 0.03

%

JPY 20.00

%

0.50

%NOK -

13.00%

1.10

%

EUR 1.40% 1.40

%NZD -5.00% 3.00

%

MXN 72.00

%

4.00

%SEK -

33.00%

1.40

%

PLN 3.00% 4.00

%SGD -

93.00%

0.20

%

ZAR 26.00

%

5.00

%USD -

61.10%

0.50

%

Total -

275.40%

-

1.84%

Total 275.4

0%

11.2

3%

The strategy expects to earn about 9.4% annually with standard deviation of about

7%.

During test period, 01/01/2000 to 01/01/2002, we calculate average 5 day returns

of the model and use them to define Lower Control Limit (LCL) and Upper Control

Limit (UCL) as values 3 standard deviations away from the mean return. Graph

below charts our 5 day returns from 2000 to the beginning of 2011 with values

approximately +1% and -1% being LCL and UCL. We can confirm that model

performed as we expected until 10/11/2008 when series of exceptionally large

returns (positive and negative) signaled of a rapid regime shift in a model behavior.

Another measure of models return is average range during consecutive 5 day

periods. Just as before, control limits for range are three standard deviations above



and below the expected return. Chart below graphs 5 day ranges versus an Upper

Limit Control (just below 5%). Several violations of that threshold prior to

10/11/2008 were positive for the model and did not require adjustments.

Two other tests were also performed in an attempt to detect early regime shifts.

Using MATLAB I tried to find out an occurrence where 9 or more points were on one

side of the mean. Observing such a cluster of results one could suspect a presence

of a bias in trading model. None such occurrence was detected. Third test tried to

detect multiple observations above 2 standard deviations in a row. Yet again, prior

to 10/11/2008 there were no such occurrences. After 10/11/2008, however, there

were multiple situations when model’s return fall above 2 standard deviations for

several periods in a row.

Just as in the original paper, we confirmed that this type of model, if properly

constructed, tends to behave in the expected way. Low volatility with consistent

results makes this type of strategy an attractive investment opportunity. On the flip

side, we were not able to detect any warning signs that would alarm us about

upcoming volatility swings.

Next chart demonstrates growth of the fully invested portfolio. Between 01/01/2002

and 10/10/2008 we realized approximately 10% annual rate of return with expected

volatility. Once the system generated a 3 Sigma return, we were essentiallystopped-out.

We demonstrated that SPC methods can be successfully applied for evaluation of

trading model’s performance results. Not only can we get a clear answer whether or

not our system have “repeatability guarantee”, but they can also pinpoint a definite

location when we need to re-evaluate our model.

Evaluating Trading Model Results With Statistical Process Control Methods

Documents

Transcript of Evaluating Trading Model Results With Statistical Process Control Methods