Understand the Power of Data Convergence

Max Colas of CameronTec looks at smarter approaches to information overload and explains how improved management of data convergence can result in greater business insight and edge.

Every day Twitter delivers 300 million messages, 4% of which are actual news. Every 20 minutes, 3 million messages are published on Facebook and 10 million comments are added. Such mind-blowing numbers would be anecdotic if they did not highlight a trend – perhaps even a threat – that is also relevant in the trading world: information overload.

As usage of FIX grows globally and firms increasingly rely on their trading platform to contribute to their business edge, the risk for FIX users is that they focus on the wrong snippets of information, or miss the truly relevant trends. Addressing those challenges becomes a differentiator for FIX technology providers.

Previous generations of monitoring systems focused on displaying information, for instance by adding value in the shaping of data or user-friendliness of the interface like displaying logs with FIX tag/value expansion or showing “conversation views” that gathered together relevant messages. The mostly static log formats even allowed vendors to claim some degree of compatibility across FIX engines. Although useful, such systems are inherently flawed for two reasons:

1.              They assume that FIX operators should approach information linearly, and

2.              They expect all information that is relevant to a business to be contained in the logs.

Neither assumption proves true in today’s environment.

When algorithmic trading is involved, it is not unusual for FIX logs to grow by 10,000 lines per second for each session. When data flows converge from a number of FIX nodes across a pan-European topology, the dataset size can increase by multiple orders of magnitude. We are way past the display of logs on a screen. Gone is the linear approach to FIX data; gone is the time of perusing pages of logs one after another, of X-term windows scrolling slowly on a screen.

In fact, the only approach that remains at this point is to expect monitoring systems to deliver on two channels: “I tell you in advance what I am interested in and you notify me when it occurs” and “I tell you what I am interested in and you bring me the relevant results”. These approaches are not new: in the outside world, they are called Google Alerts and Google Queries. Technologies developed to implement this paradigm in the financial industry, such as California-based Splunk, have been in use for a few years. They all tend to gravitate around the convergence of data into one central repository to broaden the breath of searches. This, too, is an industry trend that is highly relevant to the FIX world, with a peculiar edge that is worth analyzing.