Software Development Strategies for the Age of Data
As someone who has been in the software industry since the ‘80s, I have seen many trends come and go. The “Age of Data” that we are entering is the most profound – and most exciting – time of transformation I have yet experienced. If you look at businesses experiencing explosive growth right now, nearly all of them have data as their core value driver. The company may be a dating site, a search engine, an e-tailer or even a traditional telecomms or BPO, but in all cases powerful new revenue streams come from the ability to rapidly connect to new data sources, and then to enrich, analyze and act on that data – at scale.
But as we march toward the Internet of Things, as all objects become sensor-rich smart objects, that data will be generated at volumes that dwarf historical levels and swamp existing data infrastructures. In the near future, maintaining competitive edge will come from exploring new ways to ingest growing torrents of data from a myriad of sources, use advanced analytics to isolate meaningful correlations and patterns in near real-time, and prompt business actions that deliver value based those insights.
One of the hallmarks of this “Age of Data” is that data is born digitally and flows constantly. It’s transformative to think about data in terms of flows coursing through “data pipelines” rather than as a centralized, monolithic asset encased in a static environment and updated only at carefully determined intervals with rigid processes. We should expect to see a break from traditional high-friction and brittle ETL and static BI architectures to more ‘organic’ flow-based designs that can adapt to an ever-changing big data landscape. Dizzying breakthroughs in database technologies – NoSQL, Analytic, Graph, etc.; big data platforms like Hadoop; cloud-based integration and advanced predictive analytics all hold promise to assist in the transition.
Software architects and developers must lead the way by aligning with line-of-business stakeholders to understand this radical shift, and innovate with new software platforms that cascade thru extreme high-performance streaming and persistence tiers, and allow for a new class of highly-iterative data and computationally intensive analytic workloads.
Being able to unlock the value in big data with the almost unlimited and economical processing power of modern commodity hardware is enough to make both technologists and business leaders drool – if the software pieces of the puzzle can catch up. The bad news is that many (yes, even very famous) software stacks are creaking on decades-old heritage, and frankly unprepared to economically scale in the Age of Data. The good news is that, while some of the long-entrenched stacks will stumble under the weight of big data, we’ve entered a “Wild West” phase where a rush of new software technologies geared for the Age of Data will continue to appear. Not all of the new technologies will prove viable in the long run, and not all of the established players will falter, but now is the time to experiment and embrace these future-friendly players.
The potential for business outcomes is breathtaking, but will only accrue to innovators who recognize that this is not your father’s data landscape, that we are on the cusp of a whole new generation of super-scaling software to help you survive and thrive in the Age of Data. The race is on to turn these raw mountains of data into meaningful business action – to “out-predict” the competition. We are moving irrevocably from the human-powered “art” of decisioning to the data-powered “science” of decisioning, spurred by the promise of advanced analytics to help us make ever more timely and accurate decisions.
If you’re with a new organization that’s just formulating its data infrastructure, you have the luxury to adopt modern software architectures from the ground up. If you’re with an established organization, there are many opportunities to leverage existing infrastructure, and evolve towards more scalable and economic software infrastructure as you go.
How to get started? If you haven’t already, I suggest that you look for a part of your business where a project could deliver business impact through augmented, data-driven actions, or maybe even fully automated actions. You can leverage the data in your existing infrastructure (or from outside sources) with an agile architecture to capture, analyze and act on data to drive better decisions.
Here are a few tips to get you started:
- Look for a part of your business where adding decision science will make a difference. Where you can mine mountains of internet- or machine-generated data using advanced analytic techniques to generate predictive models to augment the decision process.
- Look for software that is hardware-agnostic, that gives you easy access to the latent power that today sits largely untapped in commodity multicore servers or clusters.
- Harness the power of parallelism. Think in terms of technology combinations that fully and elastically scale up and scale out (SUSO), exploiting both fine-grained thread-level and coarse-grained process level parallelism, so you have future-proofing built in as your data volumes and varieties expand over time.
- Think in terms of both design-time and run-time agility. You want technology that is designed for and natively operates in the modern era, but that has a mature, robust management layer and ecosystem to enable rapid coding and easy updating as the rapid pace of technology evolution continues.
- Choose architectures that give you game-changing price-performance capability. Avoid traditional architectures that can only scale through expensive “big wallet” approaches that consume alarming amounts of pricy hardware and software.
The Age of Data demands modern software architectures that can help organizations transform big data into business value. For a creative technologist eager to bring unprecedented value to the business side of the organization, it’s an exciting time.
Actian Corporation’s CTO Michael Hoskins directs the company’s technology innovation strategies and evangelizes accelerating trends in big data, and cloud-based and on-premises data management and integration. Mike, a Distinguished and Centennial Alumnus of Ohio’s Bowling Green State University, is a respected technology thought leader who has been featured in TechCrunch, Forbes.com, Datanami, The Register and Scobleizer. He speaks at events worldwide on trends and projections across the industry. Follow him on Twitter: @MikeHSays