At the crux of many an enterprise-scale big data system stands either MapReduce or a parallel database management system. But which is more efficient?
Researchers from Dublin Institute of Technology, the University of Lisbon, and the National University of Ireland Galway sought to answer that question when they studied the direct comparisons between MapReduce and parallel DBMS.
The team argues that while plenty of research has been published on the topic, the results of those studies represent conflicting and inconsistent viewpoints. “Some presented MapReduce as more flexible and capable of producing better performance,” the team noted. “Others have presented the opposite opinion. Finally, some suggested that the two approaches can be used together.”
Further, none of the studies referenced in the article were published in the last two years, indicating that the time is ripe to re-examine the issue.
One of the biggest flaws with the parallel DBMS is that it is ill-suited to deal with a single node failure. As a result, a big query or job running on parallel DBMS will have to intermittently stop and store everything such that it can be re-started upon node failure. Meanwhile, MapReduce can simply shift jobs to other nodes if one fails. “As more nodes are added into the parallel DBMS environment, the chance of a node failure increases. Parallel DBMS do not handle node failure. MapReduce has been designed to run on thousands of nodes and is inherently fault tolerant.”
Parallel DBMS is also limited by its SQL-based roots. While the Structured Query Language serves as an efficient method for writing queries on relational data, it struggles with unstructured text or social media data. By assigning value to characters or words of text in the mapping stage, MapReduce is designed to overcome this difficulty. Also, since MapReduce is not limited by the bounds of the SQL language, more complex questions can be asked of the data.
As such, multiple-step queries that foster more complex analytics are better suited for MapReduce. “The aggregation in SQL is not able to process these multiple step data flows. Performing these tasks in many steps reduces the performance benefits gained from parallel DBMS.”
Finally, since MapReduce is an open source technology, it can be implemented with relatively low cost compared to parallel DBMS, which the researchers labeled an “expensive investment.”
Of course, there were some advantages to parallel DBMS, otherwise the discussion would not even take place. In fact, one of MapReduce’s edges—the lack of SQL limitation—is also a significant drawback, as the average skill level of someone using MapReduce must exceed that of one who uses SQL. Many organizations do not have the talent at their disposal to deal in complex MapReduce queries.
Also, parallel DBMS sports a better capability for repeated structured data processing. Once a large dataset makes it into the DBMS, it can be queried more repeatedly and efficiently than if one were using MapReduce. “Parallel DBMSs are excellent at efficient querying of large data sets. Conversely, MapReduce is much slower by comparison, but is significantly better at complex analytics and ETL tasks.”
There is opportunity, the researchers noted, for the two systems to work in tandem. Parallel DBMS can suffer from an I/O bottleneck when large amounts of data are input. If the data is structured and iterative queries need to be run on it, it would be advantageous to get that data into the DBMS. MapReduce could help compress the data before transferal. “MapReduce can be useful for one-time queries while parallel DBMS can be useful for repeated queries. This property of MapReduce makes it ideal for transforming data prior to load into the Parallel DBMS.”
MapReduce has parallel DBMS beat in I/O efficiency, cost, surviving node failure, ability to handle unstructured data, and complex analytics capabilities. Meanwhile, parallel DBMS takes the cake when it comes to ease of use and repeated queries of structured data.
As it stands now, whether to rely on one or the other is dependent on the organization’s resources and application requirements. However, the two should necessarily be considered to be at odds with each other, as likely some form of a combination of the two is more likely to produce a comprehensive, all-data management system.
Related Articles
eScience and a Tale of Two Cities
.gif)
Discussion
There are 0 discussion items posted.
Join the Discussion
To join the discussion, become a registered user today!