Big Data – It’s About Real Time

Conflating business intelligence with big data is like confusing a photograph and a high definition video, says Gernot Fels, principle product manager with Fujitsu.

“In the early days, people were collecting gigabytes to terabytes of data from mainly internal sources,” says Fels. “It was mainly SQL databases, and this structured data was periodically analyzed in order to figure out what had happened in the past – it did not really matter whether these analytics jobs took hours, days, or weeks.”

Big data, on the other hand, says Fels, is about real time analysis of massive amounts of unstructured data generated at a very high velocity, needing the same high velocity to analyze the data in real time. The main characteristics are “large volumes, versatile data sources, various data types, high velocity in terms of generation and analysis, and all of this to create value for the organization.”

This, of course, requires a significant infrastructure, says Fels. “The biggest challenge with regards to infrastructure is you have to keep processing time constant, while data volumes increase, and that means that traditional approaches with SQL databases and server scale up and server scale out do not really work because sooner or later, you will reach their limits, so new approaches are needed.

For enterprises moving in the big data direction, Fels prescribes a distributed parallel processing architecture with distributed data and IO loads across the nodes of a server cluster, where processing is taking place where the data resides. “Due to the shared nothing architecture, the scalability is basically unlimited,” he says.

Storage is an important piece to the puzzle that must also be considered, says Fels, who says that an all flash array will accelerate access over disk storage systems, but for real IO bottleneck avoidance, an in-memory database is the ticket. 

Related Items:

High Performance Big Data Use Cases

Concurrency a Real-Time Problem for Financials

How Facebook Fed Big Data Continuuity