SQL Databases Integrating NoSQL-like Features
Relational databases (SQL) have been used for decades by nearly every type of business around the world. The technology is reliable, based on stable standards, and has been mature for more than 20 years.
While the concept of non-relational databases (NoSQL) actually pre-dates relational databases, they only became viable about 10 years ago. However, when they did, they were quickly adopted, primarily by two different types of businesses. The first were large digital economy companies who needed to scale to levels never achieved before, and the second were tech startups that were looking for cheap and easy database solutions that could scale when needed.
Given the rise in popularity of NoSQL functionality, SQL database vendors have taken notice and are implementing NoSQL-like functionality within their database structure.
But before we dive deeper into this movement, it’s essential to have a better understanding of why NoSQL has gained prominence.
The Allure of NoSQL
NoSQL databases shine in two main areas: scale and flexibility.
On the infrastructure front and regarding scalability, for decades if your business grew, your amount of data grew as well, traditionally meaning that your only option was to scale up to a bigger database server. However, with the cloud and commodity servers, companies can now scale out by deploying NoSQL databases to distribute and replicate data across data centers (and around the world if necessary) solving both performance and high-availability concerns.
Administration of relational databases remains a tricky and costly operation, even for open-source offerings, creating barriers to adoption by startups. The need for DBA (database administrators) roles is drastically reduced with NoSQL databases, and even disappears when they are fully managed in the cloud.
On the data structure front, flexibility has been the key. The tradition has been to create data models following the rules of normalization, a method anchored in the times when storage hardware was expensive and people tried to avoid data duplication. But the rules of normalization create huge constraints on evolution. Any change in the database schema to support application evolutions means heavy handling and costly coordination that limit a company’s agility. Aside from the infrastructure innovation mentioned above, the real revolution of NoSQL is the flexibility to rapidly adapt to the changes required by the business. This aspect of NoSQL databases has been mistakenly called “schema-less.”
While the term had the advantage of striking people’s attention, it gives a false impression that the developer does not need to think about the structure of data to be stored. In fact, data modeling is even more important in NoSQL databases than previously required. A better term is probably “dynamic schema” to represent the ongoing flexibility.
NoSQL Lowers Total Cost of Ownership
Thanks to these two main features, the total cost of ownership (TCO) of applications goes down dramatically because enterprise development becomes nimbler, and IT departments can respond faster to rapid changes in a competitive environment.
NoSQL provides ease of getting started, scalability, reliability, flexibility, and agility, leading to a less costly program to react to business needs. It’s probably not enough to justify rewriting an application for this reason only, but for any new project, it just makes sense to adopt NoSQL to prevent increasing technical debt and creating a costly legacy.
One argument often proposed as an advantage of relational database is their consistency, referential integrity, and ability to handle transactions via two-phase commits. And it is true that these functions are built in the technology, which is still useful today in the case of big, local, monolithic applications. However, this advantage quickly disappears in a connected world of integrated asynchronous and stateless web services, which requires that applications (and not databases) again handle the tolerance for failures during transaction processing. With NoSQL databases, came the realization that most businesses could afford to have “eventual consistency,” particularly if it helped keep impatient website visitors happy.
The Impact of JSON
While there are many different ways to design a relational database for a given application, the rules of normalization and referential integrity provide a sort of guardrail against several design issues. With NoSQL document databases, the nature of JSON is so flexible and powerful that it may give new users a false sense of security. Data modeling becomes even more important for NoSQL than with relational databases and enable users to create different what-if scenarios that can be discussed among team members around a picture of the schema design.
As soon as NoSQL databases started to become mainstream, SQL database vendors began to feel some effects. The easiest thing for traditional database vendors has been to add support for storage of JSON documents. This approach is a response to the flexibility advantage of NoSQL document databases. But one needs to be careful about each vendor’s ability to natively index deeply nested objects, without which the capability is not comparable. At the same time, it appears that each vendor’s scalability strategy remains pretty much untouched. This means that you still cannot really scale out as naturally as with NoSQL technology.
Despite this, it’s very hard to find the real proportion of users actually storing JSON data in SQL databases. The vendors themselves might not know that proportion, but it is a “no regret” defensive move in an attempt to keep NoSQL databases out of enterprise IT departments. In any case, it seems that the SQL DB’s addition of JSON storage is an after-thought whereas NoSQL document DBs have obviously been purpose-built. At the same time, we can observe the acquisitions by large SQL database vendors of some NoSQL database shops as acknowledgement of the growing presence of the NoSQL market. One could argue that product managers are going to be busy over the next couple of years reconciling the different products into a coherent offering.
It does seem that a new NoSQL database is announced almost every month. It is hard to keep up! But as with every emerging market segment, we’re likely heading towards shakeups, consolidation, and convergence. Established players are going to defend their turf based on their track record, while challengers will play the innovation card. Some challengers will emerge, while others will get acquired or fold. Let’s just hope that the best technology wins, rather than the best marketing.
NoSQL Database Vendors Respond
Of course, NoSQL vendors have not been sitting idle. First, they’re moving from pure-plays to a multi-model approach. Also, after years of calling NoSQL databases “non-relational,” people realize that relationships exist within the data semantics, whether the database supports foreign key constraints, or not. As a result, users are asking NoSQL database vendors to add relational features, for example, MongoDB introducing a lookup function and read-only views across multiple collections to fulfill user requests, or others adding ACID capabilities to counter the argument of lack of transactional support. In addition, some are adding SQL-like query syntax to help developer transition.
One final thought to consider. Paradoxically, it is so much more natural to do data modeling of JSON if you’ve never had to normalize before, but habits die hard if you’re a SQL veteran. As a result, if you’re coming from a relational databases background, data modeling of JSON requires a mind shift, whether inside JSON-aware traditional databases, or NoSQL document databases. One needs to challenge principles thought to be immutable, namely normalization and transactional atomicity.
To properly leverage the advantages of NoSQL, you need to force yourself to forget about Third Normal Form, and instead think in terms of how data is going to be accessed. You want to join the data at the time of writing to the database, so everything you need is stored together when you read it. This mind shift is much harder if, within the same database, you may be tempted to use the ability to store data in a traditionally normalized way.
NoSQL players are continuing to add features to contribute to the maturity of the technology, and with that, it is possible that NoSQL databases will stand on their own to finally become mainstream and viable in the eyes of enterprise IT. Meanwhile, SQL vendors will likely try to convince their installed base to stick with what they know. While it may feel safe to standardize on just one older technology, IT departments should not deprive themselves of the benefits of a newer technology. They should experiment and judge for themselves in order to use the appropriate tool to fulfill each specific need.
About the Author: Pascal Desmarets is the founder and CEO of Hackolade. He leads all corporate efforts of producing visual tools to smooth the onboarding of NoSQL technology in corporate IT landscapes. Hackolade’s software combines the comfort and simplicity of graphic data modeling with the power of NoSQL document databases. He can be reached at [email protected]