Thursday, April 07, 2016

Mean reversion, momentum, and volatility term structure

Everybody know that volatility depends on the measurement frequency: the standard deviation of 5-minute returns is different from that of daily returns. To be precise, if z is the log price, then volatility, sampled at intervals of τ, is 

volatility(τ)=√(Var(z(t)-z(t-τ)))

where Var means taking the variance over many sample times. If the prices really follow a geometric random walk, then Var(τ)≡Var((z(t)-z(t-τ)) ∝ τ, and the volatility simply scales with the square root of the sampling interval. This is why if we measure daily returns, we need to multiply the daily volatility by √252 to obtain the annualized volatility.

Traders also know that prices do not really follow a geometric random walk. If prices are mean reverting, we will find that they do not wander away from their initial value as fast as a random walk. If prices are trending, they wander away faster. In general, we can write

Var(τ)  ∝ τ^(2H)

where H is called the "Hurst exponent", and it is equal to 0.5 for a true geometric random walk, but will be less than 0.5 for mean reverting prices, and greater than 0.5 for trending prices.

If we annualize the volatility of a mean-reverting price series, it will end up having a lower annualized volatility than that of a geometric random walk, even if both have exactly the same volatility measured at, say, 5-min bars. The opposite is true for a trending price series.  For example, if we try this on AUDCAD, an obviously mean-reverting time series, we will get H=0.43.

All of the above are well-known to many traders, and are in fact discussed in my book. But what is more interesting is that the Hurst exponent itself can change at some time scale, and this change sometimes signals a shift from a mean reversion to a momentum regime, or vice versa. To see this, let's plot volatility (or more conveniently, variance) as a function of τ. This is often called the term structure of (realized) volatility. 

Start with the familiar SPY. we can compute the intraday returns using midprices from 1 minutes to 2^10 minutes (~17 hrs), and plot the log(Var(τ)) against log(τ). The fit, shown below,  is excellent. (Click figure to enlarge). The slope, divided by 2, is the Hurst exponent, which turns out to be 0.494±0.003, which is very slightly mean-reverting.




But if we do the same for daily returns of SPY, for intervals of 1 day up to 2^8 (=256) days, we find that H is now 0.469±0.007, which is significantly mean reverting. 




Conclusion: mean reversion strategies on SPY should work better interday than intraday.

We can do the same analysis for USO (the WTI crude oil futures ETF). The intraday H is 0.515±0.001, indicating significant trending behavior. The daily H is 0.56±0.02, even more significantly trending. So momentum strategies should work for crude oil futures at any reasonable time scales.


Let's turn now to GLD, the gold ETF. Intraday H=0.505±0.002, which is slightly trending. But daily H=0.469±0.007: significantly mean reverting! Momentum strategies on gold may work intraday, but mean reversion strategies certainly work better over multiple days. Where does the transition occur? We can examine the term structure closely:




We can see that at around 16-32 days, the volatilities depart from straight line extrapolated from intraday frequencies. That's where we should switch from momentum to mean reversion strategies.

One side note of interest: when we compute the variance of returns over periods that straddle two trading days and plot them as function of log(τ), should τ include the hours when the market was closed? It turns out that the answer is yes, but not completely.  In order to produce the chart above where the daily variances initially fall on the same straight line as the intraday variances, we have to count 1 trading day as equivalent to 10 trading hours. Not 6.5 (for the US equities/ETF markets), and not 24. The precise number of equivalent trading hours, of course, varies across different instruments.

===


Industry Update
Upcoming Workshops

There are a lot more to mean reversion strategies than just pairs trading. Find out how to thrive in the current low volatility environment favorable to this type of strategies.

93 comments:

one_fell_swoop said...

How did you calculate the 10 hours = 1 day, simply by fitting the 1 day point onto your linear fit, solving for it to put it on the line? Isn't there a risk one will wash out significant overnight trending/mean-reversion that way? And, I suppose more generally, are there any references for how to deal with overnight gaps when trying to do trading on the 3-5 day timeframe using intraday data?

Ernie Chan said...

Hi one_fell_swoop,

You are right that 10 hours= 1 day is determined by trying different numbers, and finding the one where intraday and daily variances coincide at the same T where they overlap.

Indeed, this does not capture seasonal mean reversion such as what may occur only overnight. This study is only concerned with "time-invariant" mean reversion, or "homogeneous" mean reversion.

The general way to deal with overnight gap is that you can to treat the overnight bar differently from the intraday bars, by multiplying its variances by some adjustment factor. This factor is determined by a procedure outlined above. (Instead of adjusting the equivalent time, you can adjust the equivalent variance.)

Ernie

Anonymous said...

Hi Ernie,

how long is the window length over which you calculate the variance?
You only state "variance over many sample times"... Or do you want to keep it a secret? :)

I figured, if the slope is near 1.0, varying the window length may push the slope above or below 1 randomly. How do you handle this? (But SPY seems always to be mean reverting on an interday basis?!)

BR!

Ernie Chan said...

Hi BR,

I computed the variances over 20130116-20160115.

Obviously, a different period might give a different answer. But within this period, you can see that I have included the standard errors (SE) in the slope. For many of the results (though not SPY intraday), a slope of 1 is not within 2x SE.

Ernie

Michael Harris said...

Hello Ernie,

Interesting article. Is there a problem with the x-axis of the GLD chart? 16 and 32 days cannot have a base 2 log between 13 and 15, if I get this right of course.

Ernie Chan said...

Thanks, Michael.

The units of x-axis (t) is actually in minutes. Apologies for not making that explicit.

So at log2(t)=14, we have t=2*14/10/16 days=27 days. (The factor of 10 due to the equivalence of 10 trading hours and 1 trading day.)

Ernie

Michael Harris said...

Thank, now it makes sense.

Ernie Chan said...

Actually, I meant t=2*14/10/60 days=27 days above!

ERnie

Danil K said...
This comment has been removed by the author.
Danil K said...

Hi Ernie!
Nice artical again, but I have one remark. Hurst exponent is different for brownian motions with different distributions. You can simulate it with Monte Carlo and test out.

Ernie Chan said...

Hi Danil,
Are you referring to fractional Brownian motion?
Ernie

Danil K said...

I refer to the article of this guy http://mechanicalforex.com/2016/03/the-hurst-exponent-and-forex-trading-instruments.html

Eduardo Gonzatti said...

Hi there, Dr Chan,

Have you considered using this measurement of "momentum or MR indicator" as possible half life for cointegrated residues?
Or, have you/do you use the Hurst exponent in cointegrated pairs or residues in order to further filter them as "more" mean reversal if they show an H<.5?

Thanks

Ernie Chan said...

Hi Eduardo,
No, the transition time scale does not indicate how long it would take for a price series to mean-revert. For that, you have to use Ornstein-Uhlenbeck equation.

As we approach the transition time from the mean reverting region, we should expect the half life to approach infinity.

But yes, the lower the Hurst exponent, the shorter the half life of mean reversion.

Ernie

HK said...

Hi Ernie,

No matter intraday or swing trades, is that pair or more than pair kind of mean reverse strategies would need much smaller chance to cut lose than momentum? Seem like even the best momentum traders would cut lose 4 out of 10 times which is mentally hard to cut lose that often.

-HK

Ernie Chan said...

Hi HK,
I do not recommend stop loss for mean reversion strategy in general, unless it is never expected to be triggered. See last chapter of my second book Algorithmic Trading.
Ernie

Eduardo Gonzatti said...

Thanks for the answer, Dr.!
I too use the OU equation for HL of Mr pairs trading.

What I tried to imply was that, maybe the transition time scale is an additional filter for the filtering of possible pairs in the half life part. Like, if a pair has a HL (OU Calculated) of 20 days, but the transition time scale shows that in approximately 10 days there is a shift from MR to Momentum, maybe one should not hold the MR trade in the pair/etf/etc for more than the latter, in this case, 10 days.

I may be wrong, just thinking out loud here

Thanks

Ernie Chan said...

Hi Eduardo,
That's a good point. Yes, you can use the transition time as an upper limit to your holding period.
Ernie

Gustavowoltmann said...


Hi there, Great blog you have there, really. I learned so much from your posts already so please juse keep up the good work! :)

Ernie Chan said...

Thanks, Gustavo!

Ernie

dawei luo said...
This comment has been removed by the author.
dawei luo said...

Great article! Very helpful information. I was trying to reproduce the SPY interday plot. I used log daily price of SPY. But seems I get a different result. For example, the first data point, 1 day lag, the log variance I got is -9.61, but your plot shows should be around -8.9. Am I missing anything? Thanks

Anonymous said...

Hi Ernie,

Have you tried Digital Signal Processing (DSP) trading strategies?

Are they profitable?

Thanks.

Eduardo Gonzatti said...

Also interested in hearing Ernie's opinion on this..

(the financial hacker blog / Zorro platform lead programmer's blog talks extensively about this subject, very interesting)

Best

Eduardo Gonzatti said...
This comment has been removed by the author.
Ernie Chan said...

Hi Dawei,
Thanks for your kind words.

Did you use consolidated close price for your daily plot? If so, they always exaggerate mean reversion, due to noise (see my article on Beware of Low Frequency Data, April 2015). I use midprice for my intraday plots.

Also, I don't know if you use the same data period as my 20130116-20160115 for SPY?

In any case, one data point isn't that important. What's important is the Hurst exponent, which averages out the noise.

Ernie

Ernie Chan said...

Hi Eduardo,
Many techniques in trading can be considered Digital Signal Processing. E.g. Kalman filter, wavelets, etc. Some of them are useful, others are not. So one needs to be specific about the technique.
Ernie

Anonymous said...

Hi Ernie,

Have you read books (Digital Signal Processing) written by John Ehlers?

http://www.mesasoftware.com/ehlers_books.htm

He mentioned Sinewave indicator in his books.

Thanks.

Ernie Chan said...

No, I haven't read John Ehlers.
Do these indicators work for you?
Ernie

Anonymous said...

Hi Ernie,

His books provide codes for Tradestation and Multicharts.

He also sells his codes on his website.

I just begin to read his books and test his codes.

Basically, he said his indicators can tell Trend mode and Cycle mode.

Eduardo Gonzatti said...

Sorry, I was referring specifically to high/low bypass filters and the very well documented FFT.

Here's the link from another blog http://www.financial-hacker.com/build-better-strategies-part-2-model-based-systems/

Ernie Chan said...

Hi Eduardo,
Thanks for the link.

No, I haven't tried hi/lo pass filters or FFT in trading. I have tried wavelets, but with no major gains.

Ernie

dawei luo said...

Dr. Chan
Thank you very much for your reply. I was using the same date period SPY data. Since I was trying to do the interday plot, so I used SPY daily price data. As you suggested, I changed to midprice. (I used 0.5*(High+Low)). But the hurst exponent I got is only about 0.387.

Ernie Chan said...

Hi Dawei,
Midprice does not mean the mid of high and low. It means the mid of bid and ask at the market close.
Ernie

Anonymous said...

Hi Ernie,

May I ask what is your intraday trends strategy of E-mini S&P 500 futures?

Thanks.

Ernie Chan said...

Please feel free to email me to discuss that.

Ernie

Steven Gross said...

Hi Ernie,
I plotted the autorcorrelation of daily returns of USO and found a statistically significant peak at a lag of 1 day. However the value is negative indicating mean reversion. So I tested two simple strategies to confirm this daily MR behavior and the difference in equity curves confirms this small daily MR behavior. Not sure how to connect autocorrelation of returns with the Hurst parameter and your results.
Thanks,

Steve

Ernie Chan said...

Hi Steven,
Your result is indeed contrary to that from Hurst exponent.
However, as an additional test, I would suggest you test using the midprice of the bid-ask at market close, not the consolidated close.

As I wrote before on this blog, consolidated closes have a tendency to reveal false mean reversion that nobody can trade on.

Ernie

Anonymous said...

Hi Ernie,

Do you trade long/short equity strategies?


Thanks.


Ernie Chan said...

Yes, I do.
Ernie

Anonymous said...

Hi Ernie,

What kinds of factors do you use in Long/Short equity strategies?

Thanks.


Ernie Chan said...

Mainly lagged returns.

Ernie

Anonymous said...

Hi Ernie,

Why don't you use fundamental factors, such as ROE?

For lagged returns, do you use PCA?

Thanks.

Ernie Chan said...

We prefer short term trading, hence fundamental factors are not of much help.

We don't use PCA currently, but it is under active research.

Ernie

Anonymous said...

Hi Ernie,

May I ask how long is "short-term" trading in your definition and strategies?

Thanks.

Ernie Chan said...

Short term to us is intraday.
Ernie

Anonymous said...

Hi Ernie,

Do we usually include intercept when we run linear regression?

Thanks.

Ernie Chan said...

Yes, including intercept is usually recommended, unless you have a fundamental reason not to.
Ernie

Anonymous said...

Hi Ernie,

For stocks pairs trading, do we need to include intercept when we run rolling linear regression?

Thanks.

Ernie Chan said...

It depends on the exact pair. But generally I won't.
Ernie

Anonymous said...

Hi Ernie,

Many thanks for a great blog!

I'm looking for an FX broker to trade a G10 strategy. Do you think IB is the best broker for an account of around $100k? What other alternatives should I look at? Do you have a good set up to recommend for low latency?

Many thanks

Ernie Chan said...

Yes, IB is as good as any in terms of FX for accounts <= $100K.

For low latency connection, ask Sam at speedytradingservers.com.

Ernie

Anonymous said...

Just curious, what options would you look at for accounts >$100K?

Ernie Chan said...

If you are an Eligible Contract Participant, as defined by CFTC, then you can open any prime broker account, and access any FX ECN directly such as HotspotFX, LMAX, EBS, etc.

Ernie

Anonymous said...

Hi Ernie,

What is the reasonable transaction cost assumption (including commissions, spread, and slippage) for S&P 500 stocks?

IB is the broker. Thanks.

Ernie Chan said...

We typically assume 5bps one way transaction cost for SP500 stocks.

Ernie

Anonymous said...

Hi Ernie,

It seems intraday long-short mean reverting strategy does notwork for SP500 stocks recently.

Is that right? Thanks.

Ernie Chan said...

I agree it is hard to make long-short stock strategies work this year (see hedge fund reports also on this category).

Ernie

Eduardo Gonzatti said...

Hi Dr,

Barging in this subject about SP500 and plain vanilla LS..

Have you ever tried to pair/triplet trade them in a very short term? Not HFT sub ms, us, but minutes wise, instead of EOD.

I had some success in doing so in 2014 and 2015, although not in the US Market (didn't try it). Looking for really small distortions from the average mean, using real time bid and ask can get you in and out very fast, may me worthwhile if your costs are not high. I stopped doing so because as a retail trader my discounts were not big enough to make it worthwhile, gross profit was real, nevertheless.

Ernie Chan said...

Hi Eduardo,
No, we haven't tried day trading stocks pairs like you suggested. I agree there may still be opportunities there.
Ernie

Anonymous said...

Hi Ernie,

Do you know why it is hard to make long-short stock strategies work this year?

How do we deal with it? Thanks.

Ernie Chan said...

Typically, long-short strategies depend on volatility to earn returns. Volatility in the stock market has been very low in the last few months.

You can always run a short volatility strategy in this market condition.

Ernie

Anonymous said...

Hi Ernie,

Thank you for quick response!

May I ask what are short volatility strategies you would recommend?

Thanks.

Ernie Chan said...

See the VX strategy discussed around Figure 5.12 in my book Algorithmic Trading.

ERnie

Anonymous said...

Hi Ernie,

I find that intraday long-short mean reverting strategy does not work for SP500 stocks since May 2010.

Thanks.

Ernie Chan said...

Thanks for sharing your experience!

Ernie

GustavoWoltmann said...

You have amazing explanations of price actions. anyone who wants to learn forex trading should look at your stuff which has helped me tremendously with my trading.

Anonymous said...

wow good

Anonymous said...

Hi Ernie,

What is the reasonable assumption of transaction costs for Russell 2000 stocks?

Thanks.

Ernie Chan said...

10 bps one way.

Ernie

Anonymous said...

Hi Ernie,

Do you trade only intraday strategies?
Are the capacity of intraday strategies limited?

Thanks.

Ernie Chan said...

We trade mostly intraday strategies, because of their higher statistical significance and lower risk. Yes, they do have lower capacity, but then we don't have billions to manage at this point. We are, however, working on strategies with longer holding period and higher capacity, and will be able to launch soon.

Ernie

Anonymous said...

Hi Ernie,

Do you trade Russell 2000?

Thanks.

Ernie Chan said...

Not at the moment.

Ernie

Anonymous said...

Hi Ernie,

Thank you for quick response.

Is it because of ill-liquid for small cap stocks?

Thanks.

Ernie Chan said...

You are correct.

Ernie

Sqrt Alpha said...

Great article!

Chris Jakobsen said...

Hi Ernie,
If I understand it correctly from your second book p. 45, the Hurst Exponent can be between +1/-1. When 0.50 is random walk, it strikes be as bold to claim that H = 0.56 is STRONGLY trending as you do in the above article. I would have guessed that strongly trending would be H = 0.8 or thereabouts??

Chis.

Ernie Chan said...

Hi Chris,
Actually H is between 0 and 1. It isn't realistic to have negative H, because that would imply prices remain constant over the long term.

Whether a price series is trending or not depends on whether it is statistically significantly greater than 0.5. Some of the price series I noted in the article passed this significance test by a good margin. However, I failed to find the adjective "strongly" mentioned in my article above. Can you please point out the sentence?

Ernie

Chris Jakobsen said...

Sorry, between 0 and 1, my bad.

I was referring to the following you wrote: "We can do the same analysis for USO (the WTI crude oil futures ETF). The intraday H is 0.515±0.001, indicating significant trending behavior. The daily H is 0.56±0.02, even more significantly trending."

Chris

Ernie Chan said...

Hi Chris,
Yes, I use the word "significant" in a specific sense. It means that it is more than 2 standard deviations away from the mean.
Ernie

Chris Jakobsen said...

The mean of what exactly?

Ernie Chan said...

Chris,
The significance testing in this specific context is to see if the Hurst exponent for random data of the same size will have the same value as what we obtained. The conclusion is that if we assume Gaussian distribution of such values, the chance that this happens is less than 2.5%. Hence with better than 97.5% probability this is a trending price series.

See also p. 16 of my second book, section on Statistical Significance of Backtesting: Hypothesis Testing.

Ernie

PavelB said...

Hi Ernie,

I was trying to replicate your results and noticed that because of you are taking logarithm from time based on 2, i.e. log2(Time), you should also take log2 from variance, i.e. log2(Var). Please correct me if I'm wrong. You didn't make this explicit, so I was straggling a bit. Thank you for the great topic.

Pavel

Ernie Chan said...

Hi Pavel,
The log2(t) on the x-axis is for display purposes only. In my actual linear regression, I have taken the natural log of both variance and timescale.
Hope this helps.
Ernie

Andrew Bannerman said...

Hi Ernie

I have a question regarding this formula. I am aiming to replicate your study:

volatility(τ)=√(Var(z(t)-z(t-τ)))

z = log price
τ = time interval?
t = what plugs into t ?

if i understand the process is below:
Take variance over many sample times, in this case... mid price of 1 minute to 2^10 (1024 minutes or 17 hours) for intraday

Once we do the above calculation, we need to multiply the daily volatility by √252? to obtain annualized volatility? or is that not needed?

for the plot:

plot the log(Var(τ)) against log(τ)

is this a rolling window across a start date to end date in your example:
2013-01-06 to 2016-01-15

for τ

would i take the width of the window, lets say for intraday.. 1 minute to 2^10. And i slide that width of window up to the end date. So a rolling variance you could say?


Or do i just take to and from and work out plot the log(Var(τ)) against log(τ)?

If that is the case, then how would i plug in the time periods into τ??

It would be log variance of mid price from 1 minute to 2^10 2013-01-06 to 2016-01-15 to enter in log(Var(τ))... then against log(τ).. what would I enter in here?

Let me know if you can perhaps make it in simple terms for someone like me :)

Thanks a lot!
Andrew



Ernie Chan said...

Hi Andrew,
No, you do not annualize the volatility in this study. The whole point of the exercise is that we should not assume a Gaussian diffusion process for the log prices. I.e. Hurst exponent is not necessarily 0.5.

Yes, the window for computation is entire data set.

For any given time t taken from the data set, the time bar for the computation of log return is tau. I don't want to call it a "window", since it is just a bar (1 minute? 1 hour?) So it is just log(price(t+tau))-log(price(t)). You will have as many data points as the number of t in your data set. You will compute the variance of these data points (log returns). For different tau, you get different variances. These different variances vs tau form the plot.

Ernie

Andrew Bannerman said...

Ok so for each day I would , lets say mid prices of 1 minutes to 2^10 minutes (1024 minutes)
Lets say start date of: 2013-01-06

1. Compute log returns from 1 minute to 1024 minutes ( so 1 minute bar for each incremental step to 1024 minutes)
2. Compute the variance of those returns
3. Subtract the variance of those returns - the log returns?

How does one form the plot especially over date range: 2013-01-06 to 2016-01-15

We are plotting the variance of log returns against the log returns right?

Just struggling with how to structure it and how 1 minute to 1024 minutes relate over the 'n' range period.


OR

for the first 1 minute bar to the last 1024 1 minute bar...
we do the log returns and the variance of the log returns...

From date range: 2013-01-06 to 2016-01-15

log(Var(1min bar)) - log(1min bar)
We do that for every bar... first 1 minute bar up to the last 1024 minute bar....

and the result of that... is what we plot?

I can get this, just need a lil more 'dumbing down' again!

I do see the value in it, if i could view the markets nature in this way, it means could 'fit' a model suited to exploit the characteristics of the that market. At least that's the initial thinking


Ernie Chan said...

Andrew,
Your first scheme is closer to the way.

You don't just compute the returns on one day. You should, for e.g., compute the 1-min log returns on all days, and then compute the variance of them. There is also no need for 3): no need to subtract anything from the variance.

You have now a set of tau (1-min, 1-hour, 1-day, etc.), and a corresponding set of variances. Plot the log of those variances against the log of tau.

Ernie

Cherkassky said...

Hi Ernie,

Great blog - I'm a fan of your books as well. I was wondering for some clarification on this part of your post:

If we annualize the volatility of a mean-reverting price series, it will end up having a lower annualized volatility than that of a geometric random walk, even if both have exactly the same volatility measured at, say, 5-min bars. The opposite is true for a trending price series.

Are you saying that when returns are negatively autocorrelated (mean reverting time series), if you take the standard deviation as a measure of volatility then you will underestimate the "true" standard deviation of the process? And vice versa for positively autocorrelated (trending) returns?

This makes sense intuitively, but I ran some simulations in R and am not finding results that match with my intuition. We can simulate an AR(1) model in r with positive and negative autocorrelation:

> sd(arima.sim(model=list(ar=.5),sd=1,n=1000))
> sd(arima.sim(model=list(ar=-.5),sd=1,n=1000))

and it turns out both of these actually overestimate the true standard deviation (which is 1). Does this match with your intuition? Or am I examining a different phenomenon

Also, if you look at the wikipedia article here

https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation#Effect_of_autocorrelation_.28serial_correlation.29

it seems like from the equation stated at the top - having a negative autocorrelation parameter would lead to sample variance overestimating the true variance and vice versa - which I think is opposite to your statement about mean reverting/trend following variance.

Ernie Chan said...

Hi Cherkassky,
Thank you for your kind words.

There is no "true" volatility. Any volatility measurement is a function of the time scale it is measured. You should use the volatility for the time scale of your trading strategy.

In particular, I was not comparing sample (unconditional) variance vs the conditional variance (1 in your example) of an ARIMA process. I was measuring the unconditional variance at various time scales.

Hope this clarifies.

Ernie

Andrew Bannerman said...

Hi Ernie

I feel I was able to create the hurst exponent by your procedure:

https://flare9xblog.wordpress.com/2017/09/24/modelling-the-hurst-exponent/

Ernie Chan said...

Hi Andrew,
Thanks for letting us know!
Ernie