The (Mis)Behavior of Markets

The grand aim of all science is to cover the greatest number of empirical facts by logical deduction from the smallest number of hypotheses or axioms.

Thus, reading this volume will not make you rich. But it will make you wiser— and may thereby save you from getting poorer.

The aim of science is parsimony. The goal of this book is simplicity.

The seemingly improbable happens all the time in financial markets.

I agree with the orthodox economists that stock prices are probably not predictable in any useful sense of the term. But the risk certainly does follow patterns that can be expressed mathematically and can be modeled on a computer. Thus, my research could help people avoid losing as much money as they do, through foolhardy underestimation of the risk of ruin. Thinking about markets as a scientific system, we may eventually craft a stronger financial industry and a better system of regulation.

The precise market mechanism that links news to price, cause to effect, is mysterious and seems inconsistent. Threat of war: Dollar falls. Threat of war: Dollar rises. Which of the two will actually happen? After the fact, it seems obvious; in hindsight, fundamental analysis can be reconstituted and is always brilliant. But before the fact, both outcomes may seem equally likely. So how can one base an investment strategy and a risk profile entirely on this one dubious principle: I can know more than anybody else?

The fundamental concept: Prices are not predictable, but their fluctuations can be described by the mathematical laws of chance. Therefore, their risk is measurable, and manageable. This is now orthodoxy to which I subscribe—up to a point.

The average height of the U.S. adult male population is about 70 inches, with a standard deviation around two inches. That means 68 percent of all American men are between 68 and 72 inches tall; 95 percent between 66 and 74 inches; 98 percent between 64 and 76 inches. The mathematics of the bell curve do not entirely exclude the possibility of a 12-foot giant or even someone of negative height, if you can imagine such monsters. But the probability of either is so minute that you would never expect to see one in real life.

My heresy is a different, fractal kind of statistical relationship, a “long memory.

a pattern emerges, and the eye is smarter than we normally give it credit for— especially at perceiving how things change. The worst fake stands out from the rest, like a criminal in a police line-up.

Extreme price swings are the norm in financial markets—not aberrations that can be ignored.

Prices are not driven solely by real-world events, news, and people.

Patterns are the fool’s gold of financial markets. The power of chance suffices to create spurious patterns and pseudo-cycles that, for all the world, appear predictable and bankable.

We have been mis-measuring risk. Greater knowledge of a danger permits greater safety. For centuries, shipbuilders have put care into the design of their hulls and sails. They know that, in most cases, the sea is moderate. But they also know that typhoons arise and hurricanes happen. They design not just for the 95 percent of sailing days when the weather is clement, but also for the other 5 percent, when storms blow and their skill is tested. The financiers and investors of the world are, at the moment, like mariners who heed no weather warnings.

As will be seen, I am not a Luther fomenting schism in the Church. I am an Erasmus who, through study, reason, and good humor, tries to talk some sense. My aim: To change the way people think, so that reform may go forward.

We cannot know everything. Physicists abandoned that piped ream during the twentieth century after quantum theory and, in a different way, after chaos theory. Instead, they learned to think of the world in the second way, as a black box. We can see what goes into the box and what comes out of it, but not what happens inside; we can only draw inferences about the odds of input A producing output Z. Seeing nature through the lens of probability theory is what mathematicians call the stochastic view. The word comes from the Greek stochastes, a diviner, which in turn comes from stokhos, a pointed stake used as a target by archers. We cannot follow the path of every molecule in a gas; but we can work out its average energy and probable behavior, and thereby design a very useful pipeline to transport natural gas across a continent to fuel a city of millions. If the physical world is so uncertain, so difficult to know precisely, then how much more uncertain and unknowable must be the world of money? Finance is a black box covered by a veil. Not only are the inner workings hidden, but the inputs are also obscured, by bad economic data, conflicting news reports, or outright deception. What coefficient of correction should I apply to a broker’s self-serving stock tip? And then there is the most confounding factor of all, anticipation. A stock price rises not because of good news from the company, but because the brightening outlook for the stock means investors anticipate it will rise further, and so they buy. Anticipation is a feature unique to economics. It is psychology, individual and mass—even harder to fathom than the paradoxes of quantum mechanics. Anticipation is the stuff of dreams and vapors. Yet in economics, there must be scores of academic journals in which scholars struggle to follow Laplace, trying to model the inner workings of the economy in all its splendid detail. They work from vast databases of prices and production. They make assumptions about human behavior, and so hypothesize intricate relations among the rate of savings, the rate of interest, and other economic variables. They try to seize in a moment a very complicated thing.

Still, the idea of chance in markets is difficult to grasp, perhaps because, unlike the anonymous particles in a magnet or molecules in a gas, the millions of people who buy and sell securities are real individuals, complex and familiar. But to say the record of their transactions, the price chart, can be described by random processes is not to say the chart is irrational or haphazard; rather, it is to say it is unpredictable. Again, word derivations are helpful. The English phrase “at random

One of the founders of modern probability theory, the late Russian mathematician Andrei Nikolaievitch Kolmogorov, wrote, “the epistemological value of probability theory is based on the fact that chance phenomena, considered collectively and on a grand scale, create a non-random regularity.

A complex pattern can appear to emerge from even the simplest random process. A key point in my work: Randomness has more than one “state,

In 1806, Legendre published a treatise on the calculation of orbits that included a supplement entitled, “On the method of least squares.

least-squares

Some bells may be squatter, and some narrower. But each has the same mathematical formula to describe it, and requires just two numbers to differentiate it from any other: the mean, or average, error, and the variance or standard deviation, an arbitrary yardstick that expresses how widely the bell spreads.

An interest in the history of ideas is good for the scientist’s soul. Hence, my books often devote a portion to the contemplation of individual scientists, physical or social, and their vicissitudes. To understand why the orthodox theory of financial markets and investment is so flawed, it first helps to review it —and there is no better way than by portraying a few men of the twentieth century who stand out as especially influential, regardless of whether one agrees with them or not. They are Louis Bachelier, Harry Markowitz, William Sharpe, and the duo of Fischer Black and Myron Scholes. The first, hero of this chapter, was a maverick, a lone visionary who overcame the general apathy and occasional opprobrium of his contemporaries and doggedly pursued his unique view of the financial world. The others, appearing in the next chapter, were secure in their professions and honored by their peers; their importance was to have made the boldest strokes that completed the canvas begun by Bachelier. There were many other hands, some of which historians might argue were equally significant. But every story must start somewhere, and this one must begin with Bachelier.

Lévy later apologized to Bachelier that “an impression, produced by a single initial error, should have kept me from going on with my reading of a work in which there were so many interesting ideas.

one can never know everything. Instead, Bachelier tried to estimate the odds that prices will move then, a novel approach. And he did so brilliantly by observing "a strange and unexpected" analogy between the "diffusion" of heat through a substance and how a bond price wanders up and down. Both, he saw, are processes that you cannot precisely forecast. At the level of particles in matter or of individuals in markets, the details are just too complicated; you can never discriminate and describe every relevant factor or analyze exactly how they all interrelate to spread energy or energize spreads. But in both fields, you can back away from the messy details of how or who and see the broad pattern of probability that describes the whole sys-tem. So, in the most specific of his models, Bachelier adapted the equations of one field to the problems of another. In this model, he started by looking at the bond market as what he called “a fair game.

In 1905, Albert Einstein developed for it equations very similar to Bachelier’s own equations of bond-price probability—though Einstein never knew that. Regardless, one cannot help but marvel that the movement of security prices, the motion of molecules, and the diffusion of heat could all be of the same mathematical species. As will be seen, it is one of many such strange liaisons in nature. Bachelier did not stop at the-ory: He also tested his equations against real prices for options and futures contracts. The theories worked. For instance, he calculated that the buyer of a forty-five day option at half a franc has 40 percent odds of earning a profit. He was uncannily close: Looking back at real trading data, he found 39 percent of such options had in fact yielded a profit to their buyers. "The market, unwittingly, obeys a law which governs it, the law of probability," he concluded.

At its heart: In an ideal market, security prices fully reflect all relevant information. A financial market is a fair game in which buyer balances seller. Given that, the price at any particular moment must be the “right

They also serve who only sit and hold.

Buy or sell too much or at the wrong time, and you lose money.

All models by necessity distort reality in one way or another. A sculptor, when modeling in stone or clay, does not try to clone Nature; he highlights some things, ignores others, idealizes or abstracts some more, to achieve an effect. Different sculptors will seek different effects. Likewise, a scientist must necessarily pick and choose among various aspects of reality to incorporate into a model. An economist makes assumptions about how markets work, how businesses operate, how people make financial decisions. Any one of these assumptions, considered alone, is absurd.

When a statistician finds a result he had been expecting, he tends not to put his tests under as critical a microscope as he should—especially when he is also assuming a Gaussian world.

Stock prices are not independent. Today's action can, at least slightly, affect tomorrow's action. The standard model is, again, wrong.

why does the old order continue? Habit and convenience. The math is, at bottom, easy and can be made to look impressive, inscrutable to all but the rocket scientist. Business schools around the world keep teaching it.

History is replete with ironies.

The methods of fractal geometry have become part of the toolkit of fluid dynamics, hydrology, and meteorology. Its power comes from its unique ability to express a great deal of complicated, irregular data in a few simple formulae.

A fractal, again, is a pattern or shape whose parts echo the whole.

As the great mathematician David Hilbert put it a century ago: “The first and oldest problems in every branch of mathematics spring from experience and are suggested by the world of external phenomena.

The fastest way to simplify things is to spot the symmetries, or invariances—the fundamental properties that do not change from one object under study to another. A fractal has a special kind of invariance or symmetry that relates a whole to its parts: The whole can be broken into smaller parts, each an echo of the whole.

For a complex natural shape, dimension is relative. It varies with the observer. The same object can have more than one dimension, depending on how you measure it and what you want to do with it. And dimension need not be a whole number; it can be fractional. Now an ancient concept, dimension, becomes thoroughly modern.

Such is the power of fractals and chance working together: Simple rules build complex structures, and complex structures deconstruct into simple rules.

The gulf between rich and poor has always been part of the human condition, but Pareto resolved to measure it.

Given a starting condition, what is the probability that some event will happen? The absolute odds of being a billionaire are very low; but according to Pareto’s formula, the conditional probability of making a billion dollars once you have made half a billion is the same as that of making a million once you have made half a million. Money begets money, power makes power. Unfair, but true—both socially and mathematically.

A great many small price movements are found in the same cotton market with a few enormous jumps; a great many rare words are in the dictionary with a small number of common words; vast legions of poor people coexist in the world with a privileged few plutocrats. Uneven. Unfair, perhaps. But still indisputable.

The market is very risky—far more risky than if you blithely assume that prices meander around a polite Gaussian average.

Anyone studying the cotton price records could easily imagine he was seeing “corrections,

resistance levels,

Let us mull the promises that science makes to society to win its support. The grand promise is to endeavor solving the great mysteries—to the list of which I have added one. But there is also a more practical promise. It consists in helping society to improve, to prevent it from acting on the basis of theories that sound nice but are not true to the facts, and to help it act on the basis of facts—even if those facts have yet to find a theory that fully explains them.

How much does the past shape the future? A moral philosopher would phrase it this way: Is it fate that determines our course, or do we choose our paths afresh with each new decision? A mathematician trades in another terminology: Is one event dependent on another, or independent from it? If Event B is dependent on Event A, then A’s occurrence changes the odds of B happening.

Most often, the strongest correlations are the short-term ones between periods close together; the weakest are those between periods far apart. If you plot all the correlations, from short-term to long-range, you get a rapidly falling curve. How fast it falls varies from one economic quantity to another. Inflation is “persistent

Hurst’s work suggested something more radical to me: correlations that decrease, but so slowly that they seem never to vanish completely, no matter how far back in time you go.

No one is alone in this world. No act is without consequences for others. It is a tenet of chaos theory that, in dynamical systems, the outcome of any process is sensitive to its starting point—or, in the famous cliché, the flap of a butterfly’s wings in the Amazon can cause a tornado in Texas. I do not assert markets are chaotic, though my fractal geometry is one of the primary mathematical tools of “chaology.

Pictures can deceive as well as instruct. The brain highlights what it imagines as patterns; it disregards contradictory information. Human nature yearns to see order and hierarchy in the world. It will invent it where it cannot find it.

Never hurry and never publish any result based on a single tool.

Recall the definition of a fractal: a pattern or object whose parts echo the whole, only scaled down. By contrast, a multifractal has more than one scaling ratio in the same object—some parts of the object shrink quickly, others slowly.

And it is also the way price-changes in a financial market can cluster into zones of high drama and slow evolution.

Keep it simple is the catchphrase of good models.

The goal is to simplify but not oversimplify.

Remember: fractals are not about the “things

the multifractal model successfully predicts what the data show: that at short time-frames prices vary wildly, and at longer timeframes they start to settle down.

That is exactly what we have done, repeatedly, using a common computer technique called a Monte Carlo simulation. The result was excellent forgeries of the market—not identical, but statistically similar to the genuine article. What good is a forgery, you may ask? An explanation is in order. Whenever you compress data—whether a computer file or a price series—you reduce it to fewer pieces of information, to a small number of parameters. Then when you decompress it again, you do not get the full set of data back again; instead, you get something that is close enough to the original for whatever purpose you have.

FOR A REAL GRASP of economics, skip the books and lectures. Get into the garment trade.

My understanding of economics comes not from abstract theory, but from observation.

To truly understand something, you must experience it—get it under your fingertips.

Weather affects harvests, and harvests affect prices.

Imagine, finally, the world economy: a chamber of mirrors. Each company relays, distorts, and attenuates the economic signals as they flash around the globe. The signals fade in time. But it can take months, years, or decades for a signal to become so weak and remote as to be unremarkable. Such is long-term dependence in an economy: Every event, no matter how remote or long ago, echoes across all other events.

Turbulence is dangerous. Its output—the pressure or velocity of water, the average or change in price—can swing wildly, suddenly. It is hard to predict, harder to protect against, hardest of all to engineer and profit from. Conventional finance ignores this, of course. It assumes the financial system is a linear, continuous, rational machine. That kind of thinking ties conventional economists into logical knots.

Common sense and folk wisdom are often wrong, of course, but must never be ignored.

According to the standard model of finance, in which prices vary according to the bell curve, the odds of ruin are about 10-20. Translation: One chance in a hundred billion billion. With odds like that, you are more likely to get vaporized by a meteorite landing on your house than you are to go bankrupt in a financial market. But if prices vary wildly, as I showed in the cotton market, the odds of ruin soar: They are on the order of one in ten or one in thirty.

In a financial market, volatility is concentrated, too; and it is no mystery why. News events—corporate earnings releases, inflation reports, central bank pronouncements—help drive prices.

Big news causes big market action. And that action concentrates in small slices of time.

Some of the most successful investors are those who did, in fact, get the timing right.

Continuity is common human assumption. If we see a man running at one moment here and a half-hour later there, we assume he has run a line covering all the ground in between. It does not occur to us that he may have stopped to rest and then hitched a ride.

Financial prices certainly jump, skip, and leap—up and down. In fact, I contend the capacity for jumps, or discontinuity, is the principal conceptual difference between economics and classical physics. In a perfect gas, as molecules collide and exchange heat, their billions of individually infinitesimal transactions collectively produce a genuine “average

Statistically speaking, the risks of a day are much like those of a week, a month, or a year. But the price variations scale with time. Again, all charts look the same. In the case of cotton, I found all the price variations followed the same statistical properties for days over a few decades and for months over eighty years. All the lines were equally wiggly. Why would this be? First, I surmise, economics differs from physics in having no intrinsic time scales. The chart of a day’s activity looks like that of a month because, from the narrow viewpoint of the probability of losses or gains, a day really is like a month.

In fractal analysis, time is flexible. The multifractal model describes markets as deforming time expanding it here, contracting it there. The more dramatic the price changes, the more the trading time-scale expands. The duller the price chart, the slower runs the market clock. Some researchers have tried linking this concept to trading volume: High volume equals fast trading time. That is a connection not yet estab-lished, and it need not be. Time deformation is a mathematical convenience, handy for analyzing the market; and it also happens to fit our subjective experience. Time does not run in a straight line, like the markings on a wooden ruler. It stretches and shrinks, as if the ruler were made of balloon rub-ber. This is true in daily life: We perk up during high drama, nod off when bored. Markets do the same.

I am an optimist. I would rather not dismiss the existence of invariances but continually look for them hiding in non-obvious places. Invariances make life easier. If you can find some market properties that remain constant over time or place, you can build better, more useful models and make sounder financial decisions. My multi-fractal model works with just such set of consistent parameters.

A pattern in an index or price chart looks like one that has happened before, and so you bet the chart will keep moving in the same way.

People want to see patterns in the world. It is how we evolved. We descended from those primates who were best at spotting the telltale pattern of a predator in the forest, or of food in the savannah. So important is this skill that we apply it everywhere, warranted or not. We see patterns where there are none.

Chance alone can produce deceptively convincing patterns. It takes no great leap of the imagination to see how such spurious patterns could also appear in otherwise random financial data. This is not to say that price charts are meaningless, or that prices all vary by the whim of luck. But it does say that, when examining price charts, we should guard against jumping to conclusions that the invisible hand of Adam Smith is somehow guiding them. It is a bold investor who would try to forecast a specific price level based solely on a pattern in the charts.

There is something in the human condition that abhors uncertainty, unevenness, unpredictability. People like an average to hold onto, a target to aim at—even if it is a moving target.

To be sure, I do not argue there is no such thing as intrinsic value. It remains a popular notion, and one that I myself have used in some of my economic models. But the turbulent markets of the past few decades should have taught us, at the least, that value is a slippery concept, and one whose usefulness is vastly over-rated. So how, you ask, does one survive in such an existentialist world, a world without absolutes? People do it rather well all the time. The prime mover in a financial market is not value or price, but price differences; not averaging, but arbitraging. People arbitrage between places or times. Between places: I had a friend who made his life as graduate student less tough by buying a convertible cheaply in his snowy home state, Minnesota, repairing it with his own hands, and then driving it to sunny California to sell dear. And arbitrage between times: A scalper buys a block of tickets today, and hopes to profit next month by reselling them dearly once the show is sold out. These arbitrage tactics assume no “intrinsic

The individual bets are small; but it is, for them, a game of large numbers. Many small profits can mount.

Under conventional portfolio theory, based on all the old assumptions of Brownian motion in prices, you build a portfolio by laboriously calculating how all the assets in a portfolio vary against each other; good diversification would mean some stocks zig when others zag. But Bouchaud’s method takes it as given that prices exhibit long-term dependence, have fat tails, and scale by a power law. He focuses, then, only on the odds for a crash—sharp, catastrophic price drops. After all, it is not small declines that wipe an investor out, it is the crashes.

fractals are back in vogue among many in finance, although, it must be said, as with any fashion there is often more show than substance.

modern portfolio theory bases everything on the conventional market assumptions that prices vary mildly, independently, and smoothly from one moment to the next. If those assumptions are wrong, everything falls apart: Rather than a carefully tuned profit engine, your portfolio may actually be a dangerous, careering rattletrap.

portfolio managers can more frequently resort to what is called stresstesting. It means letting a computer simulate everything that could possibly go wrong, and seeing if any of the possible outcomes seem so unbearable that you want to rethink the whole strategy. The technology is called a Monte Carlo simulation. You tell a computer how you think prices vary— specifically, what kind of random-number generator it should use. You feed it all the initial data: the particular stocks, their price histories, your strategy for buying them. Then you press the start button. Using the rules of randomness you gave it, the computer starts generating a series of hypothetical prices for each stock—in essence, it simulates one investor’s possible experience with the portfolio. Then it does it again, and again, thousands of times, like someone flipping a coin over and over to see if the odds for getting heads really are fiftyfifty. At the end, it totes up all the scores from all the runs: That tells you which simulated outcomes happened most often, and hence, which might be most likely in real life. It also tells you which outcomes are unlikely but, if they occurred, devastating. Finally, you use your own intelligence to decide whether you like the scenario the computer paints. If not, you decide the portfolio is too risky and you start again. It sounds like a computational nightmare. Indeed, when this technique first appeared some decades ago in physics, it was not for the mathematically faint of heart. But computers are faster and cheaper now; software to perform these calculations now comes shrink-wrapped. You can simulate the performance of an options contract, for instance, in less than a minute on a standard personal computer. And so the technique has already spread over the past decade into many corners of finance. I urge that it become a standard tool of portfolio construction.

A fundamental problem is the Black-Scholes assumption of constant volatility —in essence, that the world does not change.

I am a persistent man. Once I decide something, I hold to it with extraordinary tenacity. I pushed and pushed to develop my ideas of scaling, power laws, fractality, and multifractality. I pushed and pushed to get out into the scholarly world with my message of wild chance, fat tails, long-term dependence, concentration, and discontinuity. Now I am pushing and pushing again, to get these ideas out into a broader marketplace where they may finally do some concrete good for the world.

It is the Hippocratic Oath to “do no harm.