Customer Reviews

Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signals

5 star:		(38)
4 star:		(9)
3 star:		(2)
2 star:		(1)
1 star:		(10)

 

 

In this thought-provoking work, David Aronson tests more than 6,400 technical analysis rules and finds that none of them offer statistically significant returns when applied to trading the S&P 500. This result, presented at the end of his work, is not disappointing to dedicated students of technical analysis who draw from the book not a new trading technique but instead take away a new, and more effective, approach to system development and trading. Those seeking the single best indicator or day trading pattern will be disappointed after reading Evidence-Based Technical Analysis, just as they will be disappointed in their trading until they advance beyond seeking the Holy Grail of Trading.

Most books and articles about technical analysis focus on applying a specific technique in pursuit of success in the markets. This one is different in that it outlines an entirely new process of thinking, and through the application of this new thought process, success can be attained. Part I of Evidence-Based Technical Analysis is called, "Methodological, Psychological, Philosophical, and Statistical Foundations" and Aronson uses this title as an outline to define the processes which should underlie system development.

The scientific method changed the world, and made the advances of modern society possible. Until now, technical analysis has been considered more of an art than a science to many practitioners and escaped the scrutiny of the scientific method. With recent advances in computing power and analysis software, it is now possible for virtually anyone to search through years of data and identify seemingly profitable trading rules. Aronson presents the scientific method, combined with the philosophy of science as explained by Karl Popper, as an antidote to this very real danger.

Well designed experiments in any scientific inquiry are based upon a verifiable hypothesis grounded in detailed observations. Popper contributed the concept of falsification to this framework, which readily lends itself to mechanical trading system design. As Aronson writes, "Popper's central contention was that a scientific inquiry was unable to prove a hypothesis to be true. Rather, science was limited to identifying which hypotheses were false."

In technical analysis, we can never prove that if the NYSE Advance-Decline Line reaches a new high, the Dow Jones Industrial Average will always be higher thirty days later. But, we can test this hypothesis to see if it is not true. This simple example illustrates the beginning of Aronson's scientific approach to the markets.

Many of the dangers of data mining and curve fitting are grounded in psychology, and Aronson thoroughly explains many of the common problems that can contribute to inaccurate observations. Carefully studying his sections on logic and psychology should lead to better market observations, which should lead to profitable systems.

The chapters on statistical analysis are worth more than the price of the book in itself. Aronson presents a clear primer on statistics, and leaves the reader with all they need to understand how to design a statistically valid experiment. In what may very well be a publishing first, he presents clear, detailed and understandable descriptions of bootstrap and Monte Carlo randomization methods.

This book is well-researched and presents actionable ideas to advance the study of technical analysis. Although none of the rules Aronson tested proved to be statistically significant, he helpfully devotes a section to explaining the limitations of his test results. Armed with this information, and the knowledge provided in the rest of the book, the thoughtful analyst can develop better insights into the market and perform better backtests to identify profitable strategies.

Professor Aronson's book is a fascinating read for anyone frustrated with the current state of technical research and a must-read for those new to the field. I believe the Market Technicians Association should include it in its Chartered Market Technician curriculum.

After a few years of studying and using technical analysis, I was left with the distinct feeling that there was an elephant in the room: most of the methods used by market technicians haven't been rigorously examined for risk-adjusted performance. Elaborate and often contradictory theories and strategies have been presented by saying "my personal experience has been..." or something similar. Eventually, TA began to seem more a religious choice rather than a science of observing and predicting the markets (let alone successful investing).

Aronson's book follows a structure that is designed to break through generations of instruction from pontificating gurus. He discusses the reason TA's rules are suspect, provides a brief history of empiricism ("the scientific method") and then delves into descriptive and inductive statistics to move the field forward. Those readers fortunate enough to have an undergraduate background in philosophy and statistics will find the reading somewhat basic but the application of these fields to a critical appraisal of TA refreshing. Finally, he applies his rigorous testing to a large set of TA rules.

Key takeaway: The way to develop and test strategies going forward.

This book doesn't test 6400 binary rules,unless you see a Price/Moving average cross with a period of 1 to 200 as 200 rules. I see it as 1 rule with 1 optimizing parameter. A lot of stuff get repeated and the book shouldn't be longer than 200 pages. There is a lot to learn if you are a novice trader and if you've never tried to develop a trading system. I think this book isn't worth half the selling price. Rather buy "The Encyclopedia of Trading Strategies (Hardcover)" by Jeffrey Owen Katz. You will learn something about GA and NN. Don't waste your time on this book. I'm selling mine.

David Aronson's Evidence Based Technical Analysis ("EBTA") is a fantastic book, and one which our industry has sorely needed. It is a "How to Do Research" book that details the scientific method with regard to the markets. Everyone in the field should both read the book and practice what it preaches. But that won't happen, which is both bad news and good news. The bad news is that the vast majority of market traders who do not practice what the book preaches will lose money. The good news is that those who do will most certainly prosper. As the numbers of the former outnumber those of the latter, the few will earn a lot from the many.

The long (over 100 pages) psychology "preface" is extremely important to Aronson's body of work. I found it hugely interesting, but fear that others may not, or worse. In fact, the psychology preface itself indicates that this work will be reviled (my words) by the multitude. People do not like their sacred cows criticized.

The problem is that most market practitioners use methods with little or no scientific basis. Even if shown evidence of faulty logic, people continue to believe its validity. This is also true in the medical profession as Aronson illustrated and which scared the daylights out of me.

For anything to be scientifically testable, it must be possible to prove it wrong. However, many of the technical analysis disciplines cannot be defined. Thus they cannot be disproved. Consequently they have no scientific validity. They may have some anecdotal importance, but true science is lacking.

Let us say that one of the market gurus espouses that when the chart of XYZ resembles "Pattern A", the stock is destined to rally. To test that we have to define Pattern A and we have to define "rally", and we should provide some time parameters in which to work or fail. The trouble is that the guru cannot define any of that. But the guru still believes in his work, and all of the investors who pay monthly fees for his expertise believe it also. Anyone who criticizes the guru or the validity of Pattern A is looking to get flamed.

EBTA preaches that technical analysis research should be conducted like quantitative analysis research. Those who treat TA as a casual discipline will get casual results. The book is not an easy read, but it is an easier and much more interesting read than the "bible" of the CFA community, Quantitative Methods for Investment Analysis (DeFusco, et al.). I own both books and certainly consider EBTA more valuable than the CFA manual, worshipped by thousands. Don't expect to download all of Aronson's knowledge the first time - read it again. I did and learned more the second time through.

Aronson is meticulous and provides "service after the sale". I had recently traded emails with him about an article he had cited. He was prompt to respond and discus the implications of our expanded research. I have the feeling that he is like this with everyone.

In conclusion I have to say, that if you cannot do what EBTA preaches, at least get yourself a money manager who does.

Bill Rafter
President
Mathematical Investment Decisions, Inc.

The author mentions: "[while] the set of rules I tested did not give significant results many footnotes point to studies that do" and "I certainly don't claim that there are no TA rules that work. In fact I cite references to numerous peer reviewed studies that discuss TA rules that have proven out in rigorous testing. However, for the rule set that I specified, which I felt at the outset would contain some good rules, none had sufficient performance to reject the null hypothesis".

In other words, the chosen rule set did not contain some good rules as the author expected. The question is then:
why not to include in the book (= to submit to test and select those that in fact worked) some of these "numerous peer reviewed studies that have proven out in rigorous testing"? Any fear that getting them published would make them no longer useful? Well, but they have already been published as cited in the book.

The author does mention that "the purpose of the book was to present a method of testing rules". But instead of just using this method to disprove every single study presented in the book, it would make a lot more sense to use the very same method to confirm at least some successful studies too. This would make the book a lot more useful, well-balanced and positive.

My experience with "technical" trading books has been that they are
either pseudo science of the quasi-astrological variety (eg, wave
theory or fibonacci hocus-pocus) or entertaining and vague
reminiscences of some successful trader(s). The "serious" literature,
instead, is written in statistical/mathematical formalisms which you
may be able to understand given an adequate background, but even then
serious effort and time is required and the sheer distance from the
formalisms to the pits is fraught.

Professor Aronson has done an admirable and unique job of dispensing
with the hocus-pocus and applying the requisite rigor to the many
difficulties associated with analyzing the results of historical
backtesting and datamining. His bibliography seemingly covers all
serious (and some not so) references of interest to the algorithmic
trader. Months after having consumed the book, I still refer to the
bibliography to glean interesting sources.

As an earlier reviewer noted, the early portion of the book in which
he debunks TA and establishes basic statistical literacy for later
chapters may or may not be of interest to many readers, but his
chapters five and six are genuine contributions to the field and
should be read, studied and implemented by anyone looking to derive
tradable strategies based on historical testing.

Anyone who has spent time attempting to optimize parameters or
otherwise evince trading rules from historical data will have quickly
learned that if you look at enough cases, you will always find
something that looks good but (alas!) really isn't. Fool's gold,
indeed! This is the first trading book which directly addresses this
issue in a rigorous yet accessible fashion.

This is not the book for someone who is looking for the magical key to
a kingdom of riches (I recommend Voltaire's "Candide"). But if you
are an algorithmic trader looking to get a statistical grounding and
some concrete methods for assessing your backtested results, there's
simply no better book available.

I had the opportunity to hear the author discuss his book at a Market Technician event at the Bloomberg building in NYC just over 2 years ago.
Even though I have some disputes regarding his methodologies, reading his work was time well spent.

The first half of the book was heavy on philosophy and psychology, driving home the idea that the "scientific method" (whatever that is), is our only guard against self delusion.

As I am familiar with both philosophy of science, as well as psychology, it seemed to me as if he was stating the obvious, but I suspect for most traders, it is something that needs to be repeated.

The next half of the book elaborates on how he feels a trading system should be tested. This part of the book has some very good ideas, as well as some unfortunate errors.

The very good parts involve his discussion of the permutation test. The Classical hypothesis testing we are all taught in intro stats, has problems when multiple comparisons are made, as researchers in genetics and biology are finding out.

In most cases, some correction (ie. the Bonferroni) is made for "false positives" ie. significant results that are due to chance because so many tests were made. These corrections have a drawback in that they make the problem of false negatives--ie. accepting the null when it is false, more likely. In other words, corrections reduce the power of the test. The permutation test the author describes does not have that problem.

Yet, this is where the author makes some significant errors (forgive the pun). He seems to believe that significance testing alone tells him all he needs to know: ie. if a rule fails to attain "statistical significance" it is worthless. That is completely false.

The "significance" test, or P value, gives the probability of getting results equal to or greater than the observed result ASSUMING THAT THE NULL IS TRUE P(data|null).

What we are interested in is: what is the probability that some rule has a positive (negative) return, ASSSUMING OUR DATA IS TRUE P(alternate|data). These are two different questions, and it is the latter one that we are interested in.

In order to determine how informative a statistical experiment is, we need to define a few things: 1. the EFFECT SIZE (abbreviated as d)--the size of the difference between the null and the alternate hypothesis.

We also need the POWER (1-B), where B is the probability of a false negative--accepting the null when it is false. For a complete analysis, would also need a prior probability--ie. a subjective probability that some hypothesis is worth investigating.

Once we have some idea how the alternate hypothesis might be distributed, we can then compare it to the null, and see how much these distributions overlap. if the null and the alternate hypothsis have distributions that share a large area, the experiment does not provide much information, either way, to decide. Other, subjective factors, will lead us to choose one hypothesis over another.

More formally, it is logarithm of [(data|null) / (data|alternate)]. This is called the likelihood ratio, and measures the evidence of the experiment for or against a particular hypothesis.

The author is completely unaware of the drawbacks of his testing procedures, nor is he aware of the need for prior (subjective) probabilities to determine the utility of his results.

In short, he seems completely unaware of the Bayesian approach to this problem.

In spite of these drawbacks, the book is worthwhile. Some of the indicators that aren't significant, look interesting when you closely examine their confidence intervals.

Some of the rules (available on the website), have a confidence interval where the area on one side of zero is 4-7 times larger than the area on the other side.

Take the last rule on his website:
(http://www.evidencebasedta.com/SignificanceTestResultsof6402Rules.txt)

The mean daily return of this rule is -0.102501. The confidence interval is -0.23433 to 0.03027. Even though it isn't "significant" most of the area of the confidence interval is negative, up to 7.6 times greater than the area above zero. Suffice it to say, the probability that the rule has a negative return is very high, and might be a worthy candidate to act on from a contrarian POV.

The next half of the book elaborates on how he feels a trading system should be tested. This part of the book has some very good ideas, as well as some unfortunate errors.

The very good parts involve his discussion of the permutation test. The Classical hypothesis testing we are all taught in intro stats, has problems when multiple comparisons are made, as researchers in genetics and biology are finding out.

In most cases, some correction (ie. the Bonferroni) is made for "false positives" ie. significant results that are due to chance because so many tests were made. These corrections have a drawback in that they make the problem of false negatives--ie. accepting the null when it is false, more likely. In other words, corrections reduce the power of the test. The permutation test the author describes does not have that problem.

Yet, this is where the author makes some significant errors (forgive the pun). He seems to believe that significance testing alone tells him all he needs to know: ie. if a rule fails to attain "statistical significance" it is worthless. That is false.

The "significance" test, or P value, gives the probability of getting results equal to or greater than the observed result ASSUMING THAT THE NULL IS TRUE.

In order to determine how informative a statistical experiment is, we need to define a few things: 1. the EFFECT SIZE (abbreviated as d)--the size of the difference between the null and the alternate hypothesis.

We also need the POWER (1-B), where B is the probability of a false negative--accepting the null when it is false. We would also need a
prior probability--ie. a subjective probability that some hypothesis is worth investigating.

Once we have some idea how the alternate hypothesis might be distributed, we can then compare it to the null, and see how much they overlap. if the null and the alternate hypothsis have distributions that overlap (this area can be calculated), the statistical experiment has low power, and does not provide much information to decide what is preferable to believe.

The author is completely unaware of the drawbacks of his testing procedures, nor is he aware of the need for prior (subjective) probabilities to determine the utility of his results. In short, he seems completely unaware of the Bayesian approach to this problem.

In spite of these drawbacks, the book is worthwhile. Some of the indicators that aren't significant, look interesting when you closely examine their confidence intervals.

Some of the rules (available on the website), have a confidence interval where the area above zero is 4-7 times larger than the area below.

Take the last rule on his website:
(http://www.evidencebasedta.com/SignificanceTestResultsof6402Rules.txt)

The mean daily return of this rule is -0.102501. The confidence interval is -0.23433 to 0.03027. Even though it isn't "significant" most of the area of the confidence interval is negative, up to 7.6 times greater than the area above zero. Suffice it to say, the probability that the rule has a negative return is very high, and might be a worthy candidate to act on from a contrarian POV.

The reviews for this text encompass the full gamut of responses. At one end reviewers complain that the text does not offer a system or even ideas about how to make money; in fact, elements of the text identify and characterize the challenges inherent in profittable trading. At the other end are reviewers who praise the book for enlightening them and changing how they approach trading.

A medical analogy may serve to decribe these disparate responses. Some patients with terminal cancer don't want to be told; instead, they wish to live out their remaining time in ignorance so they may enjoy themselves. An entirely understandable and human response. Others want to understand what modern medicine has to say about their condition. They want this information in order to make informed decisions. This text places the practitioners of TA in the position of our unlucky patient.

Essentially, the text has taken what can be described as a scientific approach towards evaluating TA. The results are both enlightenting and sobering for those with an open mind and the willingness to think. In no way does the text state that TA cannot be used to trade profittably. Quite the opposite; the author provides an unqualified yes in this regard. On the other hand, he also provides real data about the pitfalls and problems inherent in the development of an automated trading system.

Personally, I prefer to have as much information as possible when I'm betting and I found the book increased the quality of my thought process. Five stars and money well spent.

David Aronson persuasively presents his case - the scientific method and proper statistical evaluations must be applied to technical analysis in order to assure its relevance. Most technical indicators and concepts are subjective and untrustworthy as they lack predictive power. Objective technical methods are better because they can be tested to see if they contain the legitimate knowledge and predictive power we seek. But evidence-based technical analysis (EBTA) goes further. Reports of profitable back tested results are not enough because testing which has not taken into account data mining biases or that shades statistical methods can produce deceptive or mediocre results.

The value of this book will be reinforced by the future research of supporters of EBTA. Critics of EBTA will declare that technical analysis is more art than science. Aronson exposes the weakness of that claim by showing how statistical methods can be applied to commonly used technical indicators, taking into account the data-mining biases he so well describes, to seek trading signals with statistically significant returns. He shares his results in Part II of the book.

This book is well-written and organized. Aronson's lifetime pursuit of intellectual honesty is evident, and his Wall Street experience is solid. Numerous footnotes stimulate further study.

This is a tough book to review. The material covered is simply not covered anywhere else I've found, and it is absolutely crucial in building a scientific approach to building trading systems. As such, you pretty much have to read this book if you want to trade and not lose your shirt. On the other hand, it's got some fairly serious flaws.

The author seems to be a "seat of his pants" proprietary trader who eventually got science-religion, and became a scientific trader. As such, it is probably more or less oriented towards people like him; people who may not have been exposed to ideas like "standard deviation" or "statistical distribution" before they read this book. I'm not sure it succeeds in explaining this issues. I found the explanations to be excellent and extremely clear; but I have a Ph.D. in physics, and have been thinking in these terms since I was a teenager. Will some 40 year old knuckle-dragger who has never heard of the Student-T distribution get anything from this? I don't know. I kind of suspect he won't. Can I think of a better way to explain these concepts to an older student coming to the ideas for the first time? Nope; certainly not. I'd probably just give them this book and hope for the best. The other flaw is also kind of a strength: the author "talks" too much. This book is over 500 pages long. The crucial material in it; the explanation of White's reality check and the Monte-Carlo analogue by Tim Masters is really only a couple of pages. Most of the other text is interesting and well written as the author is a learned and experienced man, but, well, Aronson could use an editor. I believe Ambrose Bierce once reviewed a book with, "The covers of this book are too far apart." This is unfortunately sort of true here.

I'll say it again: this book is the only one I know of which deals seriously with the issue of data mining bias. This is what separates the men from the boys. It's easy to build signal processing techniques which find real signal in financial time series (and yes, they work a lot better than the lame TA signals the author uses), but more difficult to find out when these techniques are lying. I'm planning on giving away a piece of software you can use to find some kinds of signal fairly painlessly: I probably won't give away the "reality check" stuff, because that's the hard part.

What would I have liked in the ideal world? Maybe a little less Popper and bad history of science, drop the specific test he did and add more technical stuff on the various forms of reality check. For example, the reality checks described here deal exclusively with simple entry points: how do you deal with more complex entries and exits and money management? There are ways of doing this for certain, but this book is only the beginning in figuring them out. What do you do about signals which have the Markov property, or, for example, what do you do with signal-finding algorithms which have the bootstrap property baked into them already? What about a data mining reality check for Sharpe ratio? What do you do when you have a signal with varying probability of being true? By this, I mean, you may have a signal you have determined has a 51% chance of being correct, and in some cases, you may have a signal which you know has a 54% chance of being correct (you probably will never have a 99% correct signal; not in finance anyway); what you do with such signals is different. Sometimes you have a signal where you have no idea what the probability of success is: these need to also be handled differently. There are also issues with correlations between trading systems, bet sizing ... I supposed there are lots of issues like this which I would have liked to see addressed in a book like this, but until someone writes such a book, we have to make due with this book, Grinold and Kahn and SSRN. Speaking of Grinold and Kahn, while this is probably outside the author's field of expertise, application of these ideas to classical macro/microeconomic models used in the "alpha plus" investment funds would have been incredibly awesome. Those fields use plain old regression to build their accounting based models. G & K's book doesn't mention much beyond Student-T tests for backtesting (Elton and Gruber does mention the bootstrap without telling much on how to use one). Applying the machinery of White's reality check to this "arbitrage pricing model" sort of thing would have been a huge win: far more interesting than using it on various technical analysis methods as he does in the second to last chapter. Anyway, that's how I would have rolled.