DDN Updates Storage Appliance for Really Big Data
There is big data. And then there is really, really big data. With the capability to store up to 983 petabytes and 32 trillion unique objects in one namespace, the new Web Object Scaler (WOS) offering unveiled by DataDirect Networks today is designed for truly massive collections of unstructured data.
DDN’s WOS is an object-based data store designed to get around the technical and administrative bottlenecks presented by traditional file systems and provide extremely fast and scalable storage for massive collections of unstructured data, such as movies and media content; MRI and CT scans; satellite and telecommunications data; archives of social media data; genomics and seismic data. Governments, spy agencies, media companies, healthcare organizations, and scientific research organizations are all adherents to the WOS way of doing things.
The WOS data store presents a REST-based API that allows developers to build WOS support directly into their applications, no matter what language they’re written in. It’s a different take on the data storage dilemma than, say Hadoop has taken, and it works very well for really big data environments, where the application (not the file system) takes ownership of the data.
With the release of DDN’s WOS 3.0 software and the new WOS 7000 storage appliances, the number and size of objects that can be stored in a single object data store, and the speed at which they can be accessed, has gone up tremendously.
A single WOS7000 is 4U rack-mountable server with slots for 60 spinning SAS or SATA disks (or even SSDs, although they are less commonly used), representing about 40TB of capacity. With the WOS 3.0 software powering the object data store, up to 128 appliances can be hooked together (each WOS7000 or WOS6000 appliance represents two nodes), providing just under 31PB of total storage capacity available for a single cluster, which could include appliances distributed around the world and connected via 10GbE pipes. What’s more, the data can be accessed at the rate of 256 million reads per second, and written at the rate of 64 million writes per second using SAS HDDs.
That’s pretty big, but DDN makes it bigger. The Chatsworth, California company enables users to create federations of up to 8,192 WOS nodes, which drives the total storage up to 32 trillion unique objects and 983 petabytes of capacity. (Yes, that is just 17PB short of an exabyte of storage. Why? Because you might need it someday.)
The hardware certainly sounds compelling, but DDN has delivered new capabilities on the software front, too. The biggest new feature in WOS 3.0 is the addition of a new parallel search capability that enables users to search petabytes worth of data in a matter of minutes.
The capability for users to define their own metadata and to store it in a tiny key value store database that accompanies each object is key to delivering this speedy new search capability, say Tom Leyden, director of product marketing for the WOS product line.
“If you were to run a large WOS storage infrastructure and you want to store satellite images, you could store image specific metadata so you could add information with each satellite image, with each object,” he says. “If you had a multi petabytes worth of satellite images, each one of those images has its own tiny database. If you’re doing a search, all those databases are going to be searched in parallel at the same time. If we stored the data in large database, would take a lot longer, because it couldn’t be done in parallel. It would need to go through all your database records.”
DDN’s HPC customers will be intrigued with another new feature in WOS 3.0, which enables users to connect their WOS appliances to DDN’s GRIDScaler file systems. The GRIDScaler file system provides very fast access to data in HPC environments, but it doesn’t work well with geographically dispersed data sources.
With the new GRIDScaler Bridge, the WOS systems can be used to make data available to HPC resources located around the world. WOS was designed to be deployed in a geographically distributed manner,” Leyden says. “You can set up a WOS object storage environment, spread it over three or more sites worldwide, and still it will look like one platform.”
The final piece of the WOS 3.0 puzzle concerns upgrades. According to Leyden, enhancements in the software enable users to perform upgrades in a non-disruptive manner. “A customer who has WOS running can add nodes or do maintenance updates while the system is online, and the users won’t notice,” Leyden says.
The new WOS 7000 appliances and WOS 3.0 software are available now.