Enterprise Search in 2020 and Beyond: 5 Trends to Watch
Enterprise search has been stuck in the 1990s for two decades now.
It’s hard to believe search hasn’t kept pace with the explosion of data. The two must go hand in hand. Yet many of us go to work every day at big companies with complex knowledge management systems and fat IT budgets. We sit down, make a coffee, and before we know it, we’re transported back to 1997 whenever we need to find a document.
Knowledge has continued to grow at an exponential rate, with new data sources being created and captured haphazardly all the time. Enterprise search providers have tried to evolve at the same pace, but they’ve failed. Knowledge workers spend almost 30% of their time searching for information. Most of that time is wasted as workers don’t find what they’re looking for and give up, making decisions based on anecdotal information or intuition.
Information is spread across unstructured data sources like emails, vendor PDFs, MS Office files, and images, all of which are isolated and hard to search without the “magic” keywords or wizard-level metadata filtering techniques. At the same time as terms like “data glut” are coming in vogue, speed to incorporate fast-moving insights has never been more important. Agility has always been an almost impossible goal for entrenched organizations, but it’s become critical just to maintain status quo in our hyperactive information age.
For enterprises operating in this highly competitive world, knowledge management is a top priority. Assets and market position are no longer the solid competitive differentiators they once were. Most businesses have access to the same assets, and barriers to entry aren’t as high as they were before the information age.
Now, an organization’s knowledge, it’s hard-won lessons, collective experience, and hidden insight are of major value. Capturing and recalling the right information can give a huge productivity boost for companies (try finding another common function in global business in which 30% “loss” is inbuilt) and when KM is deployed strategically, it holds the keys to the future kingdom. For the first time in many years, enterprise search tools are catching up with the data they’re meant to manage.
Enterprise Search has been Stretched to the Breaking Point: What’s Next?
Every year, companies spend billions trying to wrangle data into submission, but it’s always been just a little bit outside their grasp. The global enterprise search market is expected to reach $5 billion by 2020, as enterprises reach for solutions that save time while processing tricky data sources like email at scale. IDC predicts that by 2025, unstructured data will account for 80% of enterprise knowledge.
At the same time, a study by Coseer found that more than 70% of enterprise data lives in documents 30 pages or long, and only niche enterprise search tools with less than 1% market share meaningfully search tables or images. As businesses struggle to source enterprise search tools which can accommodate the explosion of unstructured data and search through inaccessible formats, there are several key trends on the horizon. For enterprise search to move from keywords and metadata filtering to industry 4.0-level sophistication, five key aspects of search will change.
1. The Very Definition of Knowledge Will Change
In the past, “knowledge” was contained in heavily structured, organized documents and databases. Enterprises employed a team of file clerks to manage entire rooms full of carefully curated binders. Now, data sources are all but endless, and knowledge workers act as their own haphazard, untrained file clerks, hunting down misplaced files and searching through pages of emails for tiny nuggets of information.
Modern business’ desperate attempt to extract insight from outdated file storage systems and cumbersome search is like squeezing blood from a stone. The prospect of finding and collecting valuable insights hidden in an engineer’s email, a forgotten internal database, or a disused Sharepoint are tantalizing to any exec looking to help his/her organization get ahead. And that’s only part of the story; these same execs must contend not just with the huge explosion of data within their own company, but the brave new world of internet data sources as well. Most companies are finally getting the hang of social feeds, but before they get a chance to recover, they’ll have to tackle incoming streams from IoT devices.
It’s imperative that the enterprise search tools of 2020 and beyond can capture and manage all these data sources in real time. From proprietary data, carefully created and organized by file clerks to automated capture, extraction, and tracking of the thousands of insights, the right tools will be able to accommodate this massive shift in the very nature of knowledge.
2. Focus Will Shift from Managing Documents and Databases to Managing Knowledge
Knowledge management is critical because data gets out of hand quickly, and in many ways. As we’ve discussed, the definition of knowledge is expanding. It’s already spread across multiple formats. Different file formats have their virtues; most big companies operate several databases with differing security, geographic and functional access. Maybe a few databases are external and run on a completely different system. We can practically hear knowledge workers’ frustrated groans as we run down this list.
Even if our knowledge workers find the file they’re looking for on any given day, many of these documents are too large to search meaningfully. We accept this mess because there has never been a better way. Knowledge must be freed from the shackles of formatting for enterprise search to really become powerful. As long as knowledge workers are expected to be their own file clerks, managing several completely different systems, all using clunky keyword search, valuable data will be hidden and your team will waste time searching. Will they read through an 80-page document to find one statistic, even if the document is full of useful information? Would you? We’d venture to guess no. All that hard-won organizational wisdom may as well not exist at all.
Enterprise search in 2020 will overcome the formatting problem by extracting information from its “structure.” Freed from formatting, knowledge can be recalled and used precisely, and data integrity and version control become easier to manage because the document itself is no longer center stage. Search will be knowledge-based rather than keyword-based. Collaboration and data integrity will work in conjunction. Strong collaborative drafting capabilities, robust version control, and flexible yet strict privacy policies will all safeguard enterprise knowledge while making it more accessible than ever.
3. Knowledge Will Be Pushed to the Exact Point of Consumption Vs. Pulled by the User
Once search tools begin to accommodate the changing nature of enterprise knowledge, amazing things start to happen. Knowledge workers may finally ask simple questions in natural language and get concrete, data-driven answers in return. They won’t believe that in the past they were simply shoved in the direction of a few dubious options and asked to read through a few hundred pages.
How is this possible? Through AI-driven Natural Language Search, or NLS. NLS is a specialized application of AI uniquely designed to unlock insight from unstructured, free-flowing text. It works differently than other AI techniques like deep learning which identify patterns after analyzing vast quantities of training data. NLS takes unstructured data and creates a “word mesh” like we’d create a mindmap to connect concepts related to a big idea.
Because of this, NLS can understand context – which means that it will return the same answer regardless of user phrasing. Even more amazing, AI-driven NLS can take the user to the exact spot on the page where their answer is found. With keyword-based search, this was never possible. In the enterprise search of 2020 though, it’s non-negotiable.
4. AI and Natural Language Understanding Will Grow, But Deep Learning Loses Some Sparkle
Deep learning is an amazing technology that can colorize black and white images, classify thousands of objects in photos, add sounds to silent movies – even draw. But it’s not great for enterprise data. Why? Because deep learning doesn’t work for unstructured data.
Even if all data in question is structured, there are logistical hurdles as well. Most enterprises don’t have enough data to train neural networks. It’s difficult to pin down how much data is needed for any given project, but consider this: Google, one of the top holders of data worldwide, needed to crowdsource over 210,000 articles in Kazakh before it could complete it’s Google Translate project.
Even if a firm is lucky enough to have enough data to train a deep learning engine, each data point needs to be painstakingly annotated, tagged, and labelled. And this isn’t a project to pass to an intern – expensive SME’s with domain expertise typically shoulder this burden. Because it takes so long to collect, clean and prepare all the necessary data, deep learning projects can take months or years to start delivering results. The most infamous example of this is the IBM Watson/M.D. Anderson debacle. In 2013, the announcement that M.D. Anderson Cancer Center and IBM had teamed up to expedite clinical trials rocked the AI and healthcare industries. $62 million dollars and several years later, the project was cancelled without ever producing expected results.
Contrast all of this with NLS; a technology built to handle unstructured data like texts in natural, everyday language. It can train on much smaller datasets, there is no data annotation or tagging required, and these two efficiencies mean huge time savings – leaving space and budget for multiple iterations, which bring accurate results in weeks or months rather than years.
5. 2020 Will Be the Year of Enterprise Search
No longer a humble tool in the toolbox, enterprise search in the 2020s is poised to become a key driver of productivity, strategic decision-making, and competitive advantage. At first glance, a technology solution may not seem like a sustainable competitive advantage. But think again and it’s easy to see that enterprise search is more than just an accelerant.
It’s a lever which makes accessible decades of enterprise experimentation and wisdom, never before utilized. As knowledge grows and takes new forms, as knowledge workers realize what’s possible when armed with search which can understand their queries in a contextual sense, and as AI takes its place among indispensable enterprise tools, companies will finally get the most out of their data. And they’ll wonder how they went so long without it.
Looking Beyond the 2020s
When we think about knowledge as potential – the critical link that connects human wisdom and creativity, technology, and organizational know-how – it’s hard not to be excited for the future of enterprise search. By moving from siloed, inaccessible data warehouses too far out of the way to ever be useful, your organization can position itself at the forefront of Industry 4.0.
All of this potential already exists, hiding in your much-maligned enterprise search tools.
About the author: Praful Krisha helps teams in Fortune 500 and Government to automate complex cognitive workflows based on next generation enterprise search. He is a cognitive computing and AI automation expert and an avid coder, who loves to experiment. Praful founded Coseer after a career with a hedge fund and a turnaround CEO role. Praful is an alumnus of McKinsey, Harvard, and IIT. He is also a hapless dad and lives in San Francisco.