Cloud Migration Is a Boon to Operations. But It Creates a Perfect Storm of Data Challenges

(solarseven/Shutterstock)

As businesses prepare for the post-pandemic world, now is the best time to understand and modernize their data architectures. The new reality may not exactly go back to business as usual– remote working, online presence, and optimized operations may continue to be essential business drivers for the foreseeable future.

To do that successfully means companies must ensure their data systems can accommodate the agility, security, volume, and velocity necessary for effective integration that supports real-time operational analytics and business decision making. The heightened awareness of the value that data systems can bring to enterprises amidst post-pandemic transformation and modernization, for example through cloud adoption, will help businesses reinvent economically-critical business sectors, such as retail, where there are increasing premiums put on customer engagement.

Surviving the Consumer Apocalypse

The pandemic drastically changed the way consumer-centric businesses operate. It shut down brick and mortar stores, prompted employees to work remotely, and put added pressure on IT and security organizations, who, virtually overnight, had to ensure that remote work capabilities, digital customer experiences, and operational systems were firing on all cylinders.

When physical stores closed quickly, successful retail businesses recognized that digital operations were the new face of their companies. Case in point, Macy’s, the longstanding American department store chain founded in 1858, knew even before the pandemic that technology modernization would be key to meeting the growing expectations of shoppers.

Its pivot toward modernization, centered on cloud adoption, was instrumental to improving digital and mobile experiences, site stability, store technology, fulfillment and logistics, and integrating its front line and back office. Ultimately, these digital capabilities helped Macy’s weather the pandemic storm, turning a profit from Q4 2020 into Q1 2021, preventing the eventuality of bankruptcy that many of its competitors faced, and leading to a strong sense of optimism for the future hinged on strong online growth.

Macy’s pivoted to cloud during COVID-19 and came out ahead (andersphoto/Shutterstock)

With excellent customer service at the heart of Macy’s modernization initiative, the company focused on creating a technology stack that would enable it to tap into the power of real-time data to solve critical pieces of the retail puzzle. From optimizing inventory management, to increasing collaboration between employees across locations and roles, to helping build omnichannel experiences for customers, Macy’s modernization efforts resulted in the ability to create seamless and personalized shopping experiences that have become vital to its competitive advantage.

The Cloud to the Rescue

Prior to the pandemic, cloud programs were a nice-to-have option. The pandemic greatly accelerated cloud adoption as companies came to realize that the agility, cost, performance and security that the cloud offers could be their critical lifeline. Cloud adoption is now considered a necessity by many in IT organizations across industries. Those that were especially affected by the pandemic (airlines, entertainment, retail, lodging, etc.) have turned to the cloud to streamline and reduce costs, such as expenditures on data centers and the operations staff, cooling energy, and other factors needed to support more traditional operations.

While cloud adoption also lays the groundwork for transformation, it can also create a perfect storm that presents a number of data challenges related to how information is stored, accessed and analyzed. Notably, there’s a time value to data, particularly for companies that cater to fickle consumers in quickly evolving industries like retail. There is limited utility to customer data if enough time passes between when it is collected and when it is analyzed. The useful lifespan of data is painfully short – only 30 minutes for tactical data and eight hours for operational data, according to a Nucleus Research study.

As companies rush to migrate systems to the cloud, they quickly realize that data integration is crucial to digital transformation. While cloud adoption is the engine that drives digital transformation, data is the fuel. But legacy data integration capabilities that rely on batch technology can never deliver the benefits companies need for full cloud enablement.

Real-Time Business Benefit

The value of data depends on an organization’s ability to integrate it with key elements of the modern technology stack – both cloud sources and cloud targets as well as the databases, data warehouses, applications, storage, and messaging systems associated with them. By doing so, real-time data can feed real-time decision making, delivering actual business value. In contrast, batch integration delivers data in one lump group, creating inefficiencies while constricting network bandwidth and the ability to process and analyze this data.

(Source: Nucleus Research)

Within many enterprises, legacy data processing systems have been in place for decades with potentially thousands of applications built on top of them. Modernizing these systems are necessary alongside cloud adoption, but often requires the utilization of many cloud-enabled technologies and clouds themselves for optimal business value. Data integration capabilities can unify these disparate systems to drive added value, including cost savings, data redundancy, and improved backup capabilities, among others, that help meet the demand of modern, data-centric businesses.

Scalability is another core concern. Cloud footprints can grow easily and data capabilities need to keep pace. This is not just a volume and velocity concern – the ability to change and manipulate only essential, individual data values instead of entire databases can alleviate hindrances to scalability that limit an organizations’ ability to glean value from data. In this regard, change data capture (CDC) is a crucial technology. CDC helps to ensure that data in IT systems is quickly ingested, transformed, and usable for analytics. In turn, this accelerates business intelligence and improves data management systems.

Data replication is also critical from generating business value from cloud migration. Reliability is a common concern with data integration where low latency replication across a distributed architecture is necessary to support operations in the cloud. Bringing it back to our earlier example, robust data replication capabilities are part of Macy’s special sauce. Its architecture enables the company to address rapidly changing customer needs by delivering accurately replicated data in real time across its varying enterprise systems even during peak holiday times when transactional workloads are at their highest. This is another way that the time value of data is essential in aiding businesses in their abilities to make critical decisions that improve customer experience, and, ultimately, their bottom lines.

Taking the cloud out of the equation, upgrading and migrating data systems while keeping a business running is a major challenge. But ensuring you have the right real-time capabilities that will complement cloud migration and modernization will alleviate some of the headaches and ensure that a business is able to glean actionable intelligence and other benefits from its investment. With this will come data integrity and added efficiency that will limit downtime associated with cloud adoption while ensuring access to real-time analytics and improved operational decision making.

About the author: Steve Wilkes is the founder and CTO of Striim, a provider of streaming data solutions, and a 2019 Datanami Person to Watch. Prior to founding Striim, Steve was the senior director of the advanced technology group at GoldenGate Software, which was acquired by Oracle, where he took the lead for Oracle’s cloud data integration strategy. His earlier career included Senior Enterprise Architect at The Middleware Company, Principal Technologist at AltoWeb and a number of product development and consulting roles including Cap Gemini’s advanced technology group.

Related Items:

Fueled by Kafka, Stream Processing Poised for Growth

Embracing the Future – The Push for Data Modernization Today

Filling Your Data Lake One Stream at a Time