How to Move 80PB Without Downtime

When the online photo company Shutterfly decided to move its entire data center recently, the possibility of downtime was a big issue. After all, the company had 80 petabytes of customer data spread across tens of thousands of spinning disks, and those disks wouldn’t be spinning while being physically moved.

Months later, after the last deliver made its way to Shutterfly‘s new data center, not one piece of data was lost or even temporarily unavailable from the company’s website. How did it do it? The answer, in short, is erasure encoding, the “RAID on steroids” technology that’s one of the big benefits of adopting the object-based file system from Cleversafe.

Shutterfly realized early on that it needed something more resilient than traditional file-based storage. Since it was founded in 1999, the company has grown tremendously, and today the $783-million company houses about 19 billion images for customers, which are converted into photo-books and calendars and etched into coffee mugs. If the pics are not available, the company doesn’t record any revenue, and if the company doesn’t record any revenue, stockholders get angry.

The problem with traditional file-based storage systems is that, even with the strongest RAID protection, there’s the possibility of losing data when data volumes creep into the single-digit PB range. The bigger the dataset, the longer the RAID rebuild times, and the higher the probability that enough additional drives will fail before the RAID rebuild completes the task of regaining parity across the disks. It’s the black hole of PB-scale storage on RAID, and it’s not a very scary place to be.

Shutterfly had already grown to the point where it made more sense to build its own white-box X86 server infrastructure, but it needed some sophisticated software on top to manage all that data. So it turned to Cleversafe, which is one of the most respected members of a new crop of object-based storage software companies.

When Shutterfly outgrew its original data center (the cost of electricity was becoming burdensome), the decision to standardize on Cleversafe’s object store, called dsNet, ensured a relatively drama-free move. The company’s use of erasure encoding ensured that no piece of data was ever unavailable, even if hundreds of Shutterfly disks were offline while being physically moved.

Russ Kennedy, senior vice president of product strategy and customer solutions at Cleversafe, describes erasure encoding. “It takes object data and transforms it into a set of mathematical equations, and those equations what you need to solve for,” he tells Datanami. “You need to solve the variables in the equation in order to get enough information to retrieve the data and return it back.”

The equation’s core components are the concept of width, which is the number of slices that are associated with an individual objected, and a threshold, which is the minim number necessary to read and write that object, Kennedy says. “As long as you maintain good deltas between width and threshold, you can actually take parts of the system offline,” he says.

“Given that Shutterfly was able to maintain their values of width and threshold, they were able to maintain the integrity of the data by taking some of the systems offline, physically shipping them over to the other data center, then bringing them back online,” Kennedy continues. “Their service never went offline. It was 100 percent available. It took them several months to do this, but they were able to physically relocate everything without ever taking the system offline.”

Another image-oriented company that’s getting good dividends with Cleversafe is Zenfolio, which offers Web-based photo presentation and archive services to professional photographers. Like many small online startups, Zenfolio (now owned by Art.com) started life on Amazon Web Services. But as the company grew, it found AWS storage insufficient for its needs.

“Before switching to Cleversafe, we had roughly 2.5 petabytes of data,” says Alex Fedotov, the CTO and co-founder of Zenfolio. “While the public cloud was a cost-efficient storage option for us in the beginning, it started to become very expensive as our data grew.”

The company looked at several “do-it-yourself” solutions but instead opted for a more shrink-wrapped solution that would require less handholding by its IT staff. The company opted for Cleversafe’s dsNet object-store technology running on Cleversafe’s Slicestor appliance. The company is adding more than 100TB of data per month, and has 10PB of capacity to store it.

“Nearly a year into the relationship, we are pleased with the decision to utilize Cleversafe,” Fedotov tells Datanami. “Since the solution is easy to manage, we can keep our IT staff small and IT employees can go about their work duties without worrying about storage issues. We don’t have to think about the technology, and it doesn’t take a lot of bodies to manage. With storage in good hands, the team can focus on innovations that improve the customer experience.

Of course, spinning disks will inevitably fail and need to be replaced. There’s no way around this, unless you move to SSDs, which last a lot longer, but are very expensive and are more geared toward serving fast data than big data. According to Kennedy, the combination of Cleversafe’s erasure encoding and dispersal mechanism gives customers more time to replace those broken drives.

“Drives can fail, and they do fail, all the time,” Kennedy says. “With Cleversafe’s approach, we minimize the amount of rebuild activity necessary because when we detect a drive as failed, we move the data that can be read off to other neighboring drives in the single enclosure, if you will. So the new drives come in, the balance process puts data back on the new drives and they’re off and running. The ability to tolerate failure minimizes the staff necessary to manage the operation. Certainly that’s key when you get to hundreds of petabytes in overall capacity.”

Any operation that’s managing more than 1PB of data should look into the use of an object-based file store, Kennedy says. Object stores are not cheap, and they may require some coding work to integrate with your existing application, especially if they don’t already support a REST-based interface. It took Shutterfly six weeks to validate the Cleversafe implementation. But considering the benefits it received in data availability, it was well worth it.

Related Items:

Software-Defined Storage Takes Off As Big Data Gets Bigger

Demand for Software-Defined Storage Is Up: Survey