Build or Buy? That’s the Big (Data) Question
“You can learn a lot from my failures, maybe,” says Ron Van Holst, an HPC director at Ontario Centres of Excellence. With decades of experience designing and building high-end telecommunications gear and HPC systems, Van Holst has a better grasp on the tradeoffs you’ll make when you choose to build or buy your next big data system.
Van Holst was into a little bit of everything when was building telecommunications gear in the 1990s. From the programming languages and the compilers to the operating system and communication protocols, everything was built from scratch to meet strict industry requirements. “Even the bolts that bolted the chassis to the ground were specially designed to withstand earthquakes,” Von Holst told the audience during his keynote at the recent Leverage Big Data event held in Ponte Vedra, Florida.
After the dot-com bust, Van Holst landed at an HPC hardware startup where, once again, he put his engineering skills to use designing cutting-edge systems. While the company leveraged some prebuilt systems—including AMD Opteron processors, Mellanox interconnects, and the “fastest and most expensive FPGAs we could find”–the rest of the stack was created from scratch by Van Holst and his colleagues.
While his HPC system outperformed a similarly equipped Cray supercomputer, the startup went belly-up after the venture capitalists pulled out. After that experience, Van Holst and his colleagues wondered whether they took the right approach. “In hindsight, we wondered if maybe we should have bought this or partnered for that,” Van Holst said. “I tell a lot of people I have a $40M education in HPC…I’ve probably made as many mistakes as I have successes in build versus buy.”
Startups dot the big data and HPCs landscapes today, and Van Holst sees the pattern repeating itself, where a lot of “brilliant codes” are eager to build everything themselves. Not so fast. “You can learn from my failures, maybe and just think through these things,” he says. “In build versus buy, there’s no right answer. But are you asking yourselves the right questions?”
As Van Holst sees it, there are 14 key questions that users should ask themselves as they go through the pros and cons of build versus buy. Each customer will weigh the costs and benefits differently, because no two scenarios will ever be the same. “It depends on your circumstances whether you build or buy,” he says. “The answer isn’t always what you might expect.”
For example, standards may play a huge role in the build or buy decision. In the telecommunications industry, where companies expect systems to be functioning in excess of 99.999 percent of the time, you may have no choice but to build. “We had an unplanned outage of less than 2 minutes per year,” in the telecom industry, Van Holst says. “When you have that kind of specification, you tend to build everything from scratch.”
Lock-in is another topic that often comes up. Nobody wants to be locked in to a particular vendor or solution provider, but it may not be a big deal to certain organizations. “Sometimes you get locked into that solution, and it can be a real problem,” Van Holst says. “But another person may not care about lock in. [They may say] ‘Do what you want, just give me more of what I want.'”
Buying a solution today often means subscribing to a cloud service, but some organizations are hesitant to have their data go offsite. In academic settings, research organizations may have built-in incentives to keep their processing on-site, since they can get discounts on hardware and get software essentially for free. “The build versus buy scenario in a university setting can be very, very different,” he says.
There are similar dynamics at play when it comes to hidden costs and security in the buy versus build scenarios. Many organizations that have purchased cloud services have been surprised by unexpected costs at the end of the month, Van Holst says. And it can be very difficult for organizations in regulated industries to move any of their data offsite, due to strict security procedures.
There are a lot of factors that play into the build versus buy decision. But at the end of the day, the human factor trumps everything else. Everything may be pointing towards a “build” scenario, but if you can’t find the right people to build it—that means system architects, not IT support personnel–then you’re back to square one.
“You may not have a choice. You may not have the right staff with the right training and the background,” Van Holst says. “With the phenomenal growth of data, there aren’t enough people out there to do this type of work, and that gap is growing.”
Van Holst’s full title, by the way, is director of research development for HPC at the Ontario Centres of Excellence. You can check out the full video of his keynote address at LBD’15 below.