Follow Datanami:
April 30, 2019

In 2019, Disorderly Is the New Orderly

Chris Stewart

(GiroScience/Shutterstock)

With spring right around the corner, what is more satisfying than a good ‘spring cleaning’? The launch of Marie Kondo’s series, Tidying Up with Marie Kondo, on Netflix this past January has fueled a mass-movement towards organization in 2019, already prompting many to scour their homes and beeline to their nearest charity shops with bag loads of items that are no longer ‘sparking joy.’ Kondo is encouraging us to clear the clutter out of our lives.

Similarly, data administrators have been mildly OCD about ‘cleaning up’ their data for decades – long before the KonMari method ever hit our screens. To the echoes of the battle cry, ‘Garbage in, garbage out,’ we’ve been drilled for years to meticulously get our data houses in order. Extracting value from data is now the holy grail for most companies.

Curiously enough, the developments that have been transforming data into the new gold are also prompting us to say farewell to long‑held notions about perfectly ordered datasets. Disorderly has become the new orderly.

Fresh Perspectives

We used to save data primarily to look back at what had happened, but now we’re increasingly using data to predict what’s going to happen.

Is the KonMari method the right approach for data? (Image courtesy: KonMari)

Data is becoming more and more important for steering company operations – fueled by the growing number of data points and external online data sources available, as well as ever more powerful computers and closer-knit global markets.

Companies are actively seeking new ways to disrupt others or even themselves. With progressively greater understanding, we’re constantly stringing together new data sources within existing data infrastructures to gain yet more new insights, so that we can in turn become even more efficient, enter new markets, perform better, improve quality, or whatever it is we’re looking to achieve. The primary factor in selecting data sources is the value of the insights we expect to obtain, whereas the ease with which we can access this data has become secondary.

In effect, it’s boiled down to those of us responsible for data infrastructures ensuring that any data deemed valuable be made accessible, reorganizing the infrastructure accordingly and then throwing this back into disarray again as the next data source hits our radar.

Making Room for Change

Managing a data warehouse is anything but boring in this day and age. Having to add a whole new room to your data house at a moment’s notice has become the new norm, whereas implementing a rigid data infrastructure could be tantamount to a nail in your company’s coffin.

It’s complex work, too. Dealing with virtually infinite data formats is no mean feat, as well as juggling off-the-shelf and/or proprietary legacy systems, ultramodern sensors and cutting-edge business tools installed throughout the organization or pulled in from the cloud.

Structuring data is a natural tendency (Olga Salt/Shutterstock)

In the knowledge that tomorrow your data landscape will probably look a whole lot different than it did today, it’s advisable to automate any processes that don’t rely on human input. Luckily, tools for just this purpose are already available. Programming SQL code, writing scripts and managing metadata are all examples of systematic processes that we can automate. Doing so creates space for contemplating the future and whichever data infrastructure tomorrow may bring.

Sparking Value

Nowadays, the race for success and using available data more intelligently than your competitors demands creativity, rather than orderliness.  As Oscar Wilde said: ‘Without order, nothing can exist. Without chaos, nothing can evolve.’

Author of Undercover Economist, Tim Harford, also recently urged for a touch more chaos in his book Messy, claiming it can yield something unexpected and surprisingly beneficial. We often deem chaos an obstacle – one that we instinctively attempt to avoid – yet it can also force us to be more creative and solve problems more intelligently. Hence, we should use the ‘mindspace’ freed up by automating our data warehouse processes to investigate which data is actually ‘sparking value.’

Here are a couple of things to keep in mind that can help even the most organized IT leaders embraces the chaos.

1. Chaos Can Move Your Company Forward

Data from your end customers helps fuel what direction or decision your business takes next. While disorderliness might seem overwhelming, it can pave the path to the future. Even what seems to be the most insignificant bit of data can spark an idea for the next big service, product or campaign.

2. Disorderly Data Can Inspire Creative Thinking

When your data is strung together and not ordered in separate places, you have the ability to look at the broad view of all of your data and piece together things you may not have seen before. Looking at data as a whole, rather than as little boxes paints a bigger picture, which can lead to more creative thinking.

So, accept the messy reality of your data and relish the magic that a touch of chaos mixed with a dash of smart automation can bring. It’s the fuel that fires the true data-driven organization.

Biography: Chris Stewart is VP and General Manager of WhereScape USA. Previously serving as WhereScape USA’s Director of Services, Chris oversaw the professional services and support for WhereScape’s North American customer base. A strong data warehousing team advocate, Chris has personally designed and implemented data warehouses for more than 20 years. Before joining WhereScape, Chris led data warehouse architecture for several large clinical quality improvement and supply chain companies. Chris is a graduate of the University of North Carolina at Charlotte, with a B.S. in Mathematics and Computer Science.

Related Items:

Chaotic Nihilists and Semantic Idealists

How to Survive Tech’s Chaotic Transformation

Tech’s Hottest New Trend: Data Governance

Datanami