John Bates of Progress explains how complex event processing works and how it can simplify the use of algorithms for finding and capturing trading opportunities.
A brief summary of Complex Event Processing
Complex Event Processing (CEP) is about treating actions that happen all the time as specific events, which describe the action, and then being able to analyze those events as they are streaming through a system, while looking through them for patterns that create opportunities or threats. In the trading world, this means things like trading opportunities, such as monitoring a set of instruments across multiple trading venues and looking for particular patterns. Those patterns might be high frequency trading (HFT), statistical arbitrage, correlation relationship between two items, or even execution algorithms that are slicing orders based on some predefined metric.
The threats often focus around pre-trade risk. For example, will placing the trade exceed predefined risk levels, or run into potentially abusive trades, like a wash trade. CEP is about being able to monitor business in real-time to analyze what is happening now and, based on that, to try to predict what is about to happen and act on it immediately.
The value of Complex Event Processing
The world of trading is so fast moving. Research done by the AITE Group suggests that the average lifespan of a trading algorithm can be as short as three months. This is because new trading patterns are constantly coming to light and ones that might have been very successful might no longer be available as the markets become more efficient. In the old days, trading algorithms were like a cottage industry, in much the same way as the making of muskets used to be. Highly paid and highly skilled craftsmen would handcraft the algorithm. It was the domain of the very rich and not very many could be involved in the game.
With the advent of CEP technologies in the last ten years, now anyone can find patterns in fast-flowing data feeds, but more importantly, CEP provides the tools for business people to describe new algorithms quickly. This means that traders can keep up with a trading world that is moving ever faster, and which the handmade craftsmen struggle to keep up with. Suddenly, it has become easier for smaller firms to create algorithms to compete with the larger ones. There has been a revolution in software for the trading space, in that firms of all sizes now have access to the technology that was previously available only to Tier 1 banks.
Peeking under the bonnet
In a CEP platform, there is an engine which has the tools that allow you to model and visualize new strategies as they are running, as well as see any opportunities or threats. On top of this is an adaptive layer, with connectors to convey different formats of events in and out of the processing engine, taking in market data and sending out trades. CEP platforms can work off a simple consolidated feed, but organizations find that it is better to connect to trading venues directly because it reduces the latency and things can be seen as they happen.
MiFID has undoubtedly made its impact on the industry. FIXGlobal collates opinion from Nomura’s Andrew Bowley and BT Global Service’s Chris Pickles on the success of MiFID and its next manifestation.
Having digested the massive changes MIFID brought to the EU two years ago, what has the financial community learnt from the content of MIFID 1 and the process whereby it was developed and implemented?
Andrew Bowley (Nomura): First and foremost we must conclude that MiFID has worked. We now have genuine competition and higher transparency across Europe.
Costs are down. MTFs (Multilateral Trading Facilities) have brought in cheaper trading rates and simpler cost structures, and most exchanges have followed with substantial fee cuts of their own. Indeed this pattern is also clearly demonstrated by exception. The one country where MiFID has not been properly introduced is Spain and this is one country where fees have effectively been increased. This teaches us that complete implementation is the key and the European Commission needs to look hard at such exceptions.
We have also seen clearing rates reduced, though the fragmentation itself has caused clearing charges to increase as a proportion of trading fees as typically the clearers charge per execution. Interoperability should help address that, assuming a positive outcome of the current regulatory review.
In terms of lessons learnt from the process we must consider that we have experienced a dramatic change in a short period of time, and should allow more time for the market to adjust before fully concluding or looking to further wholesale change. We are certainly still in a period of transition - new MTFs are still launching; and the commercial models of all of these, mean that we are far from the final equilibrium. To have so many loss-making MTFs means that we cannot be considered to be operating in a stable sustainable environment.
Chris Pickles (BT Global Services): MiFID is a principles-based directive: it doesn’t aim to give detail, but to establish the principles that should be incorporated in national legislation and that should be followed by investment firms (both buy-side and sell-side). Some market participants may have felt that this approach allowed more flexibility, while others wanted to see specific rules for every possible occasion. The European Commission has perhaps taken the best approach by allowing investment firms and regulators to establish themselves what are the best ways of complying with the MiFID principles, and has perhaps “turned the tables” on the professionals. If the European Commission had tried to tell the professionals how to do their job, the industry would have been up in arms. Instead, MiFID says what has to be achieved – best execution. Leaving the details of how to achieve this to the industry means that the industry has to work out how to achieve that result. This takes time, effort and discussion. FIX Protocol Ltd. helped to drive that discussion by jointly creating the “MiFID Joint Working Group” in 2004. And the discussion is still continuing. A key thing that the industry has learned – and continues to learn – is to ask “why”. Huge assumptions existed before MiFID that are now being questioned or proven to be wrong. On-exchange trading doesn’t always produce the best price. Liquidity does not necessarily stick to existing 100% execution venues. Transparency is not sufficient by just looking at on-exchange prices. And the customer is not necessarily receiving “best execution” from today’s execution policy.
Getting to the bottom of naked sponsorship and high-frequency trades.
FIX: What does the buy-side want from Direct Market Access (DMA)?
David Polen: There are two distinct market segments that use DMA - the human trader and the blackbox. I like to call this “Human DMA” and “Highfrequency Trading (HFT) DMA”.
With Human DMA, the extreme is a buy-side that has traders manually executing trades and looking at market data over the Internet; with HFT DMA, the extreme is a blackbox co-located at the exchange. One market segment is sub-millisecond and the other is more than tens of milliseconds - sometimes hundreds of milliseconds.
The human trader manually clicking around on a front-end is more interested in the full range of services a broker can provide than he is in latency. Although speed is always important, he’s keen on being able to access all his applications via one front-end versus having to go to different windows. He’s looking for his broker to be a one-stop-shop, providing all the necessary services, such as algorithms and options and basket trading, in one easy and convenient bundle. He wants clean and compliant clearing and settlement.
The high-frequency trader is different. He has his own algorithms and smart order routers (SORs). He wants to get to the market as quickly as possible and needs credit and also memberships to the various execution venues.
FIX: What is the controversy around naked sponsorship for highfrequency traders?
DP: With naked sponsorship, the HFT is trading directly on the exchange, and the broker is only seeing the orders and trades afterwards. To help with this flow, exchanges have built in risk checks, so the broker can rely on the exchanges for pre-trade risk management.
To get a view across the exchanges, the broker consolidates the posttrade information through drops of the orders and trades. Although, the broker has a reasonably good view of the risk at all times, it can take as long as a minute to turn off a buyside that has exceeded their pre-set risk parameters. This is often exaggerated into a doomsday scenario where a buyside trades up to $2 billion of stock in those 60 seconds, but that ignores the exchange’s own controls which would not be set to $2 billion. It is a lot more likely for a buy-side to barely stay within its risk limits at each exchange, but exceed the overall allotted risk by multiples. Brokers need to have measurements in place to prevent that.
FIX: What are the key concerns with latency?
DP: The best way to lower latency is to get rid of as many message hops as possible. Co-locating at an exchange is obvious as it eliminates network hops. Although, co-location is important, it does come with infrastructure costs that not all high-frequency traders are willing to bear, for example, they may need to co-locate at each exchange.
Some buy-sides or brokers may co-locate at only one exchange and use that venue’s network to access others. Co-location also depends on the buy-side’s trading strategy. High-frequency traders need to understand where they want to trade. They can’t think of the market as a montage when they’re trying to achieve the lowest execution latency. There’s no time to sew together the fragmented marketplace if you’re also trying to be incredibly reactive to each and every exchange.
It’s also important to focus on latency within each exchange. Shaving another 100 microseconds off your DMA solution may not matter much if you are hitting an exchange port that is using old hardware or if you are overloading a port at the exchange and not load-balancing to another port. You also have to be aware of the protocol you are using: some exchanges have created legacy FIX sessions that are wrappers around internal technology and can be quite slow converters.
They are now creating “next generation” API’s that are native FIX and much faster, but these sessions may only offer a subset of available messages, so you have to consider routers that send the legit subset down the fast FIX pipe.
Has the industry found its latest villain in the form of dark pools? Not so, argued a group of traditional and alternative trading venue operators over dinner in Singapore last week. Dark pools are nothing new; they’re just finding their feet in Asia’s rugged exchange landscape. FIXGlobal’s Becky Merrett took a look at developments in the industry.
Caught in the middle of a seasonal Singapore early evening downpour, a group of regional specialists make the dash from their taxis to the warmth of an Italian restaurant. The roll-call reads like the Who’s Who of trade execution – Singapore Exchange (or SGX as it is most commonly known), BlocSec, Liquidnet, Chi-X, ITG, Bank of America Merrill Lynch and Credit Suisse. Have dark pools taken over from hedge funds as the baddie-du-jour in Asia? Should dark pools and exchanges compete, cooperate or co-exist?
Before the first cork was pulled opinions were flying, fuelled by the discussion’s volunteer ‘devil’s advocate’ in the form of Credit Suisse’s James Rae, (also Co-Chair of the FPL Singapore Working Group). “What is the purpose of a dark pool versus a traditional exchange? Why do we need alternative venues in Asia? And how do they interchange?” he asked.
“Dark pools have been around for a number of years,” argued Bank of America Merrill Lynch’s Mark Wheatley, fresh off a plane from Japan. “They’re not new in Asia. It’s mostly an extension of the internal broker systems. The alternative trading systems (ATS) we see now in Asia is the industry responding to the demands of their clients by creating a more formalised system.”
Discussing whether ATS should or not should not exist was pointless it seemed, as I toyed with my antipasti. “These are market and client-driven initiatives. All markets evolve, and financial markets evolve faster than most. To try and back track is both unwanted and unwarranted. Judging from the response from the markets in the US and Europe, ATS are here to stay,” Chi-X’s Rob Rooks stated emphatically.
Competitors or complementary? Before the starters were finished, we’d killed the notion that ATS were going to slip quietly away into the night. Instead the conversation turned to the respective roles of traditional exchanges versus off-exchange platforms.
Liquidnet’s Greg Henry weighed in. “An exchange is about price discovery, it’s about listing and taking companies to market. Our focus is on efficiency, latency, liquidity and best execution.”
Unsurprisingly, it was a common view among the alternative venues around the table. Trading activity on the NYSE, they argued, now accounted for less than 30 percent of revenue. The role of traditional exchanges was increasingly focused on listing and sourcing capital. “The stringent regulations on listing, the information required, it provides a comfort blanket for investors,” Henry argued.
“At the end of the day, we all have to create the structure that works for our clients,” Henry concluded.
The right structure for your client? It was a theme that emerged again and again over the evening. The overriding – although not unanimous – feeling was that dark pools catered for one kind of investor, while exchanges provided security and solace for others.
“We don’t want to list organisations. The compliance involved in the process doesn’t fit with our business model. We’re more interested in a symbiotic relationship between ATS and exchanges. We attract different investors with different strategies. The investors trading through our venue are more likely than not to only hold a position for 10 minutes or less,” explained Rooks.
It was time for our lone exchange operator to pitch in. “We’re comfortable with the competition. Although, if we see a proliferation of venues, such as we’ve seen in the US, this is not going to help the region,” said SGX’s Bob Caisley. “We feel that the best way to move forward is to understand what dark pools offer and to let our clients access this technology,” he added.
It was an understandable position, given the recent announcement of a joint venture between SGX and multilateral-trading facility, Chi-X. The deal, which was inked in August, is aiming to launch its Chi-East non-displayed liquidity pool by June 2010. Clearly the move has raised the stakes as it is the first time in Asia that a dark pool has the backing of a regional exchange.
While Madoff maybe a dirty word for many investors, the scandal at the end of 2008 certainly helped propel the often forgotten operational departments of an asset manager into the limelight.
Never before had so many institutional investors got caught up in such heavy losses caused directly by the lack of internal controls and oversight at an asset management firm. Historically, losses had been mainly driven by poor performance from asset classes, for example the equity markets after the tech bubble burst. The Madoff scandal highlighted investors’ need to understand much more than the team running money on their behalf.
Pre-Madoff most institutional investors focussed the vast majority of their efforts selecting an asset manager based on investment process and portfolio strategy. In the new world the selection process for identifying asset managers has become more rigorous, with many more investors focusing on the quality of the operations, control and support functions, in addition to the portfolio management team. An institutional investor study carried out by Mercer in 2009 found 41% of respondents carried out some sort of operational due diligence in 2009, up from 13% in 2008.
As investors start to embrace operational due diligence, it is having a material impact on the way asset managers position themselves during the selection process, and more importantly, the actual controls and processes applied behind the scenes. For the first time, asset managers are being asked questions about their audit process, their compliance department, valuation methodologies and pricing sources.
On top of a strong investment process, investors are looking for:
A strong organisational structure promoting good corporate governance
Formal segregation of duties between front and back office activities
Infrastructure and systems to match the size and complexity of assets under management
Appropriately experienced and qualified staff
Documented policies and procedures
Reputable service providers including lawyers, auditors, custodians and administrators
Robust back-up plans in case of disruptions due to power failures or other disruptive events
In order to find answers to the points listed above, many investors are attending meetings on-site or appointing a specialist operational due diligence provider to review an asset manager’s operations, control and support functions. Again this is a major change from existing behaviour where most of the selection process took place at the investor’s or a consultant’s office.
The informal feedback Mercer has gathered indicates that many investors are surprised by some of the findings at asset managers. There have been instances of asset managers running their whole technology infrastructure using spreadsheets, and fax machines heavily used for asset confirmation. Whilst these types of processes were acceptable in the 1990s they are not sufficiently robust for current day asset managers with many employees running complex products, managing millions or billions of investor’s money. Both of these examples show a lack of investment in infrastructure and the reliance on manual processes, which increase the probability of losses through mistakes or fraud.
Other issues uncovered reveal a more fundamental issue with the business model of a firm. A basic requirement of any asset manager is to demonstrate a clear segregation of duties between front and back office staff. Barings and Nick Leeson was a prime example of where a lack of segregation of duties can bring disastrous consequences. As the front office trader, Nick had influence over the back office, which allowed trades to be hidden in the now infamous 888 Account. All too often asset managers allow some individuals to both execute and confirm trades, breaking down the fundamental segregation of duties between doers and checkers vital to maintaining the integrity of the investment process and reducing the risk of mistakes or fraud.