THE NEW TOOLS

The desire to explain the present and the quest to predict the future are closely linked. The key to both is finding those fundamental laws that shape the course of events. The new financial science is searching for laws it can use to make models of the markets that explain their present behaviour and forecast their future evolution.

Identifying the fundamental laws and building the mathematical models requires tools. The financial theory of the 1960s and 1970s relied heavily on traditional statistical techniques and early computer systems. These tools were limited, so the models that emerged were highly simplified and often at odds with market realities.

The models described efficient markets where prices followed a random path. They assumed that all market participants shared the same objectives and trading time frames. Prices moved because the participants reacted rationally and immediately to news. And because news is unpredictable, the markets themselves must be unpredictable.

The new financial science has since discovered pattern and predictability in the markets. The volatility of stock and currency prices follow regular daily patterns, for instance. Periods of high volatility tend to be more intense and last for longer than the old theory suggests. The impact of news is not necessarily immediate or obvious. And the new science recognises that there are significant differences in objectives and trading time frames of the participants and that these are critical to explaining the dynamics of the markets.

These new discoveries would not have been possible without a whole range of new tools. These include databases that gather high frequency price data, advanced statistical and mathematical techniques, and computer-based analytical and pattern-recognition systems.

Some of these tools have come from finance itself, while others have been borrowed from disciplines such as physics and computer science. Often researchers have combined techniques and tools from several sources to fit the unique characteristics of the financial markets. Because of the link between understanding and prediction, many of the tools used for long-term analysis are also used in short-term forecasting.

When scientists discovered that they could often explain the complex behaviour of natural systems with simple deterministic equations they thought to look for the same phenomenon in the markets. No-one has yet found a simple set of equations that explain market behaviour. However, some of the scientific techniques for studying dynamic natural systems are useful in finance. But researchers have had to abandon the deterministic element of the techniques and replace it with sophisticated ways of modelling probability drawn from finance and other fields. According to Andreas Weigend, assistant professor in the department of computer science and institute of cognitive science at the University of Colorado at Boulder, this combination of techniques has created a new set of mathematical tools more capable of describing the complex realities of the markets than any previously available.

Frank Diebold, professor in the department of economics at the University of Pennsylvania, agrees. The emergence over the past five to 10 years of 'modern sophisticated financial techniques, modern computer power and modern statistical tools', all interacting with one another, has 'let us study the properties of various sorts of asset returns and various sorts of dynamics in the financial markets in ways we never could before,' he says.

Meanwhile, computer scientists have developed a new breed of artificial intelligence (AI) systems that can learn from experience and adapt to change in their environment. Many of these systems, such as neural networks, fuzzy logic and genetic algorithms, are based on simplified models of how the human brain or other natural systems work. These systems have caught the attention of the financial community because they can apply them directly to market data to search for patterns or structure.

Traditional artificial intelligence requires humans to first analyse a problem and then translate this analysis into a program for a machine to carry out. In contrast, a neural network is given data and then 'learns' to find a solution. While humans need to guide the learning, they do not need to program the steps to the solution, or even know what the solution might be. Neural networks are adept at finding patterns in streams of data, such as market price movements, and a number of financial organisations have tried them in this role, including Citibank, Credit Suisse and Fidelity Management & Research.

Genetic algorithms mimic natural evolution to rapidly find optimal solutions to problems where there are many variables, such as selecting instruments for a portfolio. Fuzzy logic replaces the rigid logic of traditional computing with the more flexible human-like reasoning. All these now form part of a broad range of techniques generally called 'machine learning'. Together they provide a toolset that can be used to tackle almost any analytic or forecasting problem in finance, says Weigend.

According to Tim Bollerslev, professor in the department of finance at the JL Kellog Graduate School of Management at Northwestern University, Illinois, many financial organisations have made money using techniques such as genetic algorithms and neural networks. 'But they also realise the problems with them and the need to go further,' he says.

Early success in applying tools such as neural networks has usually been short-lived. Weigend says the problem lies in too simplistic an approach. The way forward is 'thoughtful combinations' of mathematical models and tools, applying the appropriate techniques after a careful evaluation of the problem. Even these thoughtful combinations still face a number of challenges. The first lies in the nature of the data with which the systems must work.

The availability of high frequency data has been key to the advancement of the new financial science. But as Charles Goodhart, Norman Sosnow Professor of Banking and Finance at the London School of Economics, has pointed out, there are serious limitations to the available market data despite the size of databases.

Foreign exchange data, for example, does not include volume or counterparty information. Data is liable to errors from many sources. The speed at which the markets now operate means mistakes are made during data input. Even where the data are entered correctly, they may be poor indicators for the underlying economic processes, such as industrial production or unemployment. Or the data may be difficult to assess, or subject to revision, as is the case with tax revenues. The result is data that has a high proportion of 'noise' - misleading content that obscures the genuine information.

Noise causes problems for machine learning systems because they are driven by data. Because of their inherent flexibility, machine learning systems can end up modelling the noise rather than the information if they are not carefully controlled.

Another objection to neural networks and genetic algorithms is that they appear to be 'black boxes' - closed systems whose operations are impenetrable. Data is put into the system at one end, and a solution appears mysteriously out the other. While this was to a large extent true of early neural networks, 'we are now way beyond that', insists Weigend. 'Lots of people are doing good work to make them transparent,' he says. Genetic algorithms, however, lag some way behind in their development.

Bollerslev warns that even where the tools are mature and transparent they are only part of the solution. Neural networks or genetic algorithms can be useful in trading situations, where they may be able to uncover patterns in market data. 'But in terms of understanding why it is that markets behave the way they do, the black box approach isn't going to help us,' he says.

'You might uncover a pattern and then it breaks down, because the pattern was just uncovered on the basis of a particular history of data and you don't really have any structural theory to guide you as to why it is that pattern exists. You're going to get something spurious that will not work for the infinite future.'

Weigend acknowledges the problem. 'Finding governing equations with proper long-term properties may not be the most reliable way to determine the parameters for good short-term forecasts, and a model that is useful for short- term forecasts may have incorrect long-term properties,' he says . Part of the thoughtfulness required to develop new financial tools is recognising the complexity of the problems and choosing the most appropriate approach to their solution.

There must also be due recognition of costs. A trading system that doesn't take into account transaction costs can appear phenomenally successful, says Weigend. But profits 'can get wiped out as soon as you put trading costs in'.

Mounting competitive pressures in the industry means that the benefits of any new systems must be clear and more or less immediate. According to Oswald Gruebel, member of the board at Credit Suisse, the period for measuring the return on cost has got shorter. 'I think the payback time of systems these days has to be measured in months rather than years,' he says.

Cost often determines how quickly practitioners transform a new science into new tools. 'Academics ignore costs in many cases, and quite appropriately, but costs are very relevant to people who want to develop practical implementations,' says Diebold. For example, some of the recently developed models for estimating volatility involve multiple complex calculations.

'For a corporation or trading entity monitoring thousands of asset returns it's really pretty much out of the question to do those sophisticated numerical optimisations in real-time given current technology,' says Diebold. 'So people have worked out ways to approximate them, or which are hopefully close approximations, but which are much cheaper and therefore much more practical.'

Dean LeBaron, founder and chairman of Boston-based Batterymarch Investment Management, says evaluating the benefits of research and tools is often tricky. The amount of research an organisation can do is almost infinite, with unpredictable benefits. With tools, the cost is often based not on intrinsic value but on what is being replaced or on a guess at what seems a reasonable fee, says LeBaron.

'It's a very irrational method of pricing. Probably the cost should be based entirely on the benefits. A sharing of the performance (of the system) or something of this nature would seem more just instead of a fixed fee,' he says.

A further problem in evaluating new tools is that the trading environment is already swamped with information technology. 'You can surround yourself with tools and with each one you use the others less,' say LeBaron. 'The average trader sits in front of an array of information that no human can possibly keep track of. They are better off slimming it back.'

Not everyone agrees that cost is a major issue. Kurt Kohler, risk manager for spot foreign exchange trading at the United Bank of Switzerland in Geneva, says an effective forecasting system can quickly pay for itself. But traders can compromise the success of a system by not using it properly.

'The problem is with the discipline,' says Kohler. Where a system is implementing a trading strategy there has to be commitment and consistency in its use. 'The problem is that although the system takes the decision, somebody must go and trade, and if the person decides not to trade what can the system do?' Ad hoc use can undermine the systems' analysis and calculations.

Grubel challenges this, and suggests that traders are likely to resist any system that imposes decisions upon them. 'Traders would argue that if they had to do what the system told them they wouldn't be creative.' Therefore, many traders prefer charting techniques or technical analysis, which may be less scientific, but do not demand any particular course of action, he says.

Kohler believes that a generation gap underlies the conflict of views. The managers who make the purchasing decisions tend to be of an older generation sceptical of the new technology. They don't understand it, says Kohler, but at the same time they aren't the ones who daily face the risks of operating in the markets. The traders, who tend to be younger and more familiar with the potential of computers, and who actually work the markets and take the risks, are more interested in what the new tools have to offer. Proving the value of a system to management can be a lengthy and difficult process, says Kohler.

Senior practitioners often base their reservations about new tools on their experience of the way the markets evolve and change. They know only too well how the markets have defied most previous attempts at forecasting. For them, even if a forecasting system was generally correct over a period of, say, five years it would still not convince.

'(They) would not believe it,' says Grubel. 'They would argue that it might have been right for the last five years but this doesn't give a guarantee that it would be right for the next five years.'

There is some justification in the claim, says LeBaron. In the past forecasting tools have not been dynamic. They have been calibrated on historical data and cannot adjust to the evolving markets. 'The new tools and the new mathematics almost always have a dynamic feedback aspect to them,' says LeBaron.

At present, this dynamic feedback is usually restricted to the short-term forecasting aspect of systems rather than their underlying models. The systems take in real-time market data and adjust their predictions according to market fluctuations, but the underlying models remain static. The aim in future, says LeBaron, is for machine learning to be applied to the models as well, so they adjust themselves when they sense that the markets have undergone some kind of evolution.

A fully adaptive system would still face a fundamental criticism: if many practitioners adopted the new tool its effectiveness would soon evaporate.

'If everyone knows that something works and if everybody piles in, the relatively profitability is likely to get driven to zero fairly quickly,' says Goodhart.

Weigend agrees. 'You see very significant changes in the dynamics of the markets on a time scale of one or two years because big banks are using similar techniques. For example, a simple moving average strategy used to make money until '91 or '92, but now it's useless.' However, if the tools are more sophisticated and exploit fundamental laws rather than transient market inefficiencies, then their effects will not evaporate as the technology spreads. As long as the markets remain heterogeneous, with a divers range of participants in terms of their objectives and trading time scales, trading opportunities will persist. Researchers may create more accurate models, and tools may arrive which evolve with the markets, but these will not diminish the need to trade or destroy the dynamic activity of the markets.

The idea that any new trading tool will only work if used by a few has, meanwhile, led to an atmosphere of secrecy. There is often an unwillingness to submit systems to objective testing, which makes it difficult to judge the validity of claims about many tools. Some believe that, anyway, it makes no sense to sell a system that can make profitable forecasts.

'If a (profitable forecasting) system should exist you probably would not offer it for sale,' says Grubel. 'The logic is not there. Why should you sell something that is printing money for you?'

The counter-argument is that those who have the expertise to create tools do not necessarily have the expertise to manage money or to operate successfully in the markets. Nor is it irrational to share research or to expose systems to empirical testing. It is through the open exchange of ideas that a science progresses. Applying the ideas of the new financial science is a complex business. It requires considerable intellectual and technological resources and is open to many approaches. There are no simple secrets to the markets and no short-cuts to building useful tools. And the problem does not stand still.

'The markets are continuously evolving,' says Goodhart. 'The evolution depends on what people are thinking. It is the interaction between people that is so important.'

The new science attempts to model the interaction between people participating in the markets. The new tools try to apply these models to forecast the future. This attempt to understand and to model the fundamental laws of the markets is not just about profit. It is about controlling risk too. As LeBaron says: 'Understanding that some of the same tools can be used for controlling risk and for controlling return is increasingly important.' The new tools have a key role to play across the whole spectrum of financial activity.