Everything You Always Wanted to Know About Ultra High...

MotivationData Structure and Exchange Market Rules

Data ManagementEconometric Analysis

Conclusions

Everything You Always Wanted to KnowAbout Ultra High-Frequency Data

but Were Afraid to Ask

Christian T. Brownlees1 Giampiero M. Gallo1

1Dipartimento di Statistica “G. Parenti”Università degli Studi di Firenze

22nd September 2005

Brownlees & Gallo (2005) UHFD

Conclusions

Outline

1 Motivation

2 Data Structure and Exchange Market Rules & Procedures

3 Ultra High-Frequency Data ManagementData CleaningData HandlingTime Series Construction

4 Econometric Analysis of Ultra High-Frequency DataStylised FactsA MEM Approach to High Frequency Dynamics

5 Conclusions

Conclusions

Ultra High-Frequency Analysis

The availability of financial data at a very high frequency hasbeen one of the most relevant innovations in the field ofquantitative analysis of the financial markets over the past fewyears.

Conclusions

The practitioner’s world is also slowly moving towards an ultrahigh-frequency approach to financial analysis.

Conclusions

Unfortunately the analysis of this data is not particularly simple.The main causes of this difficulty are

the need to understand the market mechanisms whichhave an impact in the series dynamics,

the operational complications which arise whenmanipulating these data,

models based on daily observations do not seem to besuccessful in capturing data patterns at higher frequencies.

Conclusions

This contribution tries to directly address these issues:

it explains the structure of the high frequency datasets andhighlights the relevant aspects of the market mechanisms;

it surveys simple and general methods for the datamanagement and manipulation which have beenimplemented in a freely available package for MATLAB;

it presents some basic stylised facts and models for highfrequency modelling.

Conclusions

The Focus of the Analysis

There are many financial markets which produce tick data.

The focus of this research is the the New York Stock Exchange(NYSE).

A review of the ultra high-frequency datasets from this source isparticularly interesting since the type of information available forequity markets is in fact qualitatively different and much moredetailed than the one available for the OTC markets.

The empirical illustrations of this presentation will be made inreference to the time series of General Electric (GE) in 2002.

Conclusions

Ultra High-Frequency Data Description

Ultra high-frequency databases are complex to analyse

huge number of ticks;

the time interval between ticks is random;the sequence of ticks

could contain some wrong ticks,might not be time ordered andmight have to be reconstructed from other tick sequences;

a tick contains additional information which may not be ofinterest for the specific purposes of the analysis (like NYSEsequence number, trading conditions, etc.);

the structure of the ticks depends on the rules of thefinancial institution which produces/collects the data.

Conclusions

The New York Stock Exchange

The NYSE is a hybrid market, in that it is both an agency andan auction market. It is an agency market since the buy andsell orders are executed by an agent, the market maker, who inthe NYSE is called the specialist. The NYSE is also an auctionmarket, since on the exchanges’ floor the brokers participateactively in the negotiation and thus they contribute to thedetermination of the transaction price.

Despite the global trend towards computer automated tradingsystems, the “human” element has a very crucial role in thetrading mechanisms, and this fact has a deep impact on thedata.

Conclusions

The New York Stock Exchange

The NYSE is a hybrid market, in that it is both an agency andan auction market. It is an agency market since the buy andsell orders are executed by an agent, the market maker, who inthe NYSE is called the specialist. The NYSE is also an auctionmarket, since on the exchanges’ floor the brokers participateactively in the negotiation and thus they contribute to thedetermination of the transaction price.

Despite the global trend towards computer automated tradingsystems, the “human” element has a very crucial role in thetrading mechanisms, and this fact has a deep impact on thedata.

Conclusions

Types of Data

Book The book contains detailed information on limitorders.

Quotes Quote data contains information regarding the besttrading conditions available on the exchange.Information of interest: bid/ask time stamps, prices,volumes.

Trades Trade data contains information regarding theorders which have been executed on the exchange.Information of interest: transaction time stamp, pricevolume.

Conclusions

Types of Data

Conclusions

Types of Data

Conclusions

Data CleaningData HandlingTime Series Construction

Ultra High-Frequency Data Management

The preliminary steps that have to be undertaken beforestarting the analysis of UHFD are:

detecting and removing wrong observations from the rawUHFD: Data Cleaning,

constructing the time series of interest for the objectives ofthe analysis: Data Handling.

Unfortunately these operations are far less trivial than onecould expect.The literature is not particularly clear on how these operationsare performed.

Conclusions

Data Cleaning

Raw UHFD is well known for being prone to errors.

We need methods for eliminating wrong obsevations. This kindof preliminary data manipulation is often referred as filtering ordata cleaning.

Data cleaning aims to eliminate from the ultra high-frequencytime series observations which does not reflect the marketactivity, in practise however these methods cannot go beyonddetecting outliers.

Conclusions

Data Cleaning

Conclusions

Data Cleaning

Conclusions

Where Do Errors Come From?

It is not really clear where errors come from and whichtheir causes are.

Falkenberry (2002) reports that errors are present both infully automated and partly automated trading systemssuch as the NYSE.

According to the author, trading intensity is the maindeterminant of errors: the higher the velocity in trading, thehigher the probability that some error will be committed inreporting trading information.

Conclusions

What Does “Raw” Data Mean?

Trade and quote data for the GE stock on the 11st of November 2002from (a) 9:30 to 16:05 and (b) 10:30 to 10:31.

Conclusions

Data Cleaning

Easy procedure on tick–by–tick price time series (detect a priceout of line with the current market momentum).

Volume data: no special data cleaning recipes other thanassessing the plausibility of a certain volume (either judgingupon the corresponding price or resorting to classical statisticaloutlier detection methods).

Algorithms proposed in the literature for washing away wrongobservations (e.g. algorithm used at Olsen’s -Dacorogna et al,2001). NYSE data require simpler steps.

Conclusions

Data Cleaning

Conclusions

Data Cleaning

Conclusions

Data Cleaner

Ultra High-Frequency Cleaning Algorithm

For each observation remove the i th tick if∣∣xi − x̃[i−1:i−k−1]

∣∣ > 3√

s2[i−1:i−k−1] + γ2

k window size

γ granularity tolerance level

x̃[i−1:i−k−1] 5% trimmed sample mean of the last k ticks

s2[i−1:i−k−1] sample variance of the last k ticks

Conclusions

Data Cleaner

∣∣ > 3√

s2[i−1:i−k−1] + γ2

k window size

Conclusions

Data Cleaner

∣∣ > 3√

s2[i−1:i−k−1] + γ2

k window size

Conclusions

Data Cleaner

∣∣ > 3√

s2[i−1:i−k−1] + γ2

k window size

Conclusions

Data Cleaner

∣∣ > 3√

s2[i−1:i−k−1] + γ2

k window size

Conclusions

Some Comments

The procedure is heuristic but it has several appealing features:

it is simple,

it can be used for on-line data cleaning as it only uses pastinformation,

the impact of the use of such method is negligible if thecleaned data are further aggregated at lower frequencies.

Conclusions

Some Comments

it is simple,

Conclusions

Some Comments

it is simple,

Conclusions

1-minute of Trading Activity on the NYSE

Conclusions

Spot the Differences with Daily Financial Series

Simultaneous Observations There are more transactionsreported at the same time at very different pricelevels.

Irregularly Spaced Observations The timing betweensubsequent events is random.

Bid–Ask Bounce The price changes are discrete and the pricehas a tendency to bounce around the currentmid–price.

Conclusions

Opening and Closing

Conclusions

What’s a Bit Weird?

Opening The first transaction/quotation of the day is notnecessarily the opening transactions/quotation

Closing the closing does not occur at 16:00 (the officialclosing time) but some time later. The lasttransaction/quotation of the day is not necessarilythe closing transactions/quotation.

Non Synchronised trades and quotes are not synchronised.

Conclusions

What to Do Then?

Simultaneous Observations → Aggregate Them

Irregularly Spaced Observations → Construct a RegularFrequency Series (if needed)

Random Closing of the Markets → “Extend” the Trading Dayuntill 16:05:00

The MATLAB toolbox for UHFD analysis contains a number ofprocedures for performing these task.

Conclusions

What to Do Then?

Conclusions

What to Do Then?

Conclusions

What to Do Then?

Conclusions

More on the Construction of Regular Frequency Series

Origin: irregularly spaced time series {(ti , xi)}Ni=1

Destination: regular frequency time series {(t∗j , x∗j )}N∗j=1

We propose using mapping functions like:

x∗j = f ( {(ti , xi) | ti ∈ (t∗j−1, t∗j ]} )

the value x∗j is constructed using the information available t∗j−1until t∗j

Conclusions

Origin: irregularly spaced time series {(ti , xi)}Ni=1

Destination: regular frequency time series {(t∗j , x∗j )}N∗j=1

We propose using mapping functions like:

x∗j = f ( {(ti , xi) | ti ∈ (t∗j−1, t∗j ]} )

the value x∗j is constructed using the information available t∗j−1until t∗j

Conclusions

Some simple but useful methods are:

First: x∗j = xf where tf = min{ti |ti ∈ (t∗j−1, t∗j ]}Minimum: x∗j = min{xi |ti ∈ (t∗j−1, t∗j ]}

Maximum: x∗j = max{xi |ti ∈ (t∗j−1, t∗j ]}Last: x∗j = xl where tl = max{ti |ti ∈ (t∗j−1, t∗j ]}Sum: x∗j =

∑ti∈(t∗j−1,t∗j ] xi

Count: x∗j = #{(xi , ti)|ti ∈ (t∗j−1, t∗j ]}

Conclusions

This approach is very different from what has been proposedby authors such as Dacorogna et al (2001).

We believe that this approach is better than other methodswhich mix past and future information in a potentiallydangerous manner.

Conclusions

This approach is very different from what has been proposedby authors such as Dacorogna et al (2001).

We believe that this approach is better than other methodswhich mix past and future information in a potentiallydangerous manner.

Conclusions

Time Series Construction

Intra-daily transaction data of the GE ticker in 2002.

Time series of interest: intra-daily returns rd ,t , volumes vd ,t

and number of transactions per interval td ,t

Frequencies: 14 different intra-daily frequencies from 1minute to 3 hours

Conclusions

Raw Data & Preliminary Data Treatment

Summary of the preliminary operations performed on the data.

number of raw observations 3907055 100%ticks out of time scale -14353 -0.37%filtered ticks -12262 -0.31%simultaneous ticks -1471227 -37.66%number of clean observations 2409213 61.66 %

Conclusions

Number of Observations and Missing Observations

Frequency Num. of Observations % of Missing1 minute 97074 4.27%2 minutes 48562 4.22%

...5 minutes 19448 4.10%6 minutes 16211 4.08%10 minutes 9735 3.99%

...30 minutes 3249 3.88%

...2 hours 1002 3.65%3 hours 752 3.59%

Conclusions

Stylised FactsA MEM Approach to High Frequency Dynamics

Econometric Analysis of Transaction Data

Hourly returns rd ,t , volumes vd ,t and number of transactions td ,t .

Conclusions

Seasonal Moments

Conclusions

Distributional Shapes Across Frequencies

Conclusions

Autocorrelation Across Frequencies

Conclusions

A MEM Approach to High Frequency Dynamics

The Multiplicative Error Model (MEM)(Engle (2002), Engleand Gallo (2003)) is a generalisation of GARCH modelswhich can be useful to model non–negative time seriesprocesses such as volatility, durations, volumes, and soforththe canonical MEM specification is

xt = µt · εt εt ∼ D(1, φ)

wherethe conditional mean is given by

µt = ω +

p∑j=1

αjxt−j +

q∑j=1

βjµt−j

εt is an i.i.d disturbance with non–negative support and unitmean.

Conclusions

µt = ω +

p∑j=1

αjxt−j +

q∑j=1

βjµt−j

Conclusions

µt = ω +

p∑j=1

αjxt−j +

q∑j=1

βjµt−j

Conclusions

µt = ω +

p∑j=1

αjxt−j +

q∑j=1

βjµt−j

Conclusions

High Frequency Multiplicative Component MEM

As in Chanda et al (2005) we adopt a Multiplicative Componentmodel for intra-daily data.We are the first ones however to employ a MultiplicativeComponent MEM for the modelling of intra-daily volumes andnumber of transactions.