Concurrency a Real-Time Problem for Financials

Financial service providers face big challenges as they adapt to use real-time big data analytics to find the needles in the haystack, says Emil Werr, Head of Enterprise Data Architecture for NYSE Euronext.

NYSE Euronext, ground zero for a large portion of the world’s activity in futures and options trading with markets in the US and Europe, offers market access and trading technologies to their trader base.  The financial market operator says that it processes a rough aggregate of over 2TB of data a day across its markets, creating significant IT challenges, particularly due to the real-time nature of their business.

Concurrency, says Werr, is especially challenging in the financial services space, where users of the system are provisioning data on their own.  “It’s not just a data problem,” says Werr.  “It’s also how many different users are hitting the system with different types of workloads at the same exact time.”

 Organizations in financial services like NYSE Euronext face tough challenges says Werr, as the need to model and analyze data in real time, as close as possible to when a trade or particular event occurs is becoming increasingly important in a market where the fastest systems tend to dominate the field.

 “When you basically take a traditional data base technology, what you do is you explode that because in order to get the kind of speed that you want to do the analytics, you’re going to create a lot of additional overhead like indexing and all sorts of other stuff,” says Werr.  “That basically will end up increasing the amount of storage you want to use and store and also increases the time to market in terms of being able to produce the kind of results that you want to do from an analytics perspective.”

The company has turned to IBM’s PureData System to help them manage the big data challenges it faces, and are expecting a 20x boost in performance.  “Using PureData for analytics, we should be able to minimize and mitigate some of the latency – we should be closer to real time,” says Werr.

Related Items:

IBM Targets Virtualized Big Data 

IBM Puts Pure Spin on Project Sparta 

Watson: Coming to a Cloud Near You