New Data.World Report Finds a Technique For Making LLMs 3x More Accurate in Answering Business Questions

(Ole.CNX/Shutterstock)

The coming year is expected to be a breakout year for generative AI, especially for organizational AI readiness. While the race to drive business value from Large Language Models (LLMs) is at an all-time high, there are some looming concerns. 

A pair of new studies from Stanford and UC Berkeley have shown that LLMs can provide a false output based on fabricated citations – also referred to as “hallucinations”. McKinsey has also cited concerns about the accuracy of LLMs. These studies highlight the need for more research on the accuracy of GenAI technology. 

So far we have not been able to fully understand to what extent LLMs can accurately answer complex business questions over SQL databases and what impact can Knowledge Graphs have on improving the accuracy and explainability of LLMs.  A new report by data.world can help us address these questions. 

Data.world, an AI-ready data catalog platform, released a landmark benchmark report on LLM response accuracy on real business queries. The report shows that using Knowledge Graphs with LLMs helps boost the accuracy of the responses by 3 times over SQL databases. 

“The main conclusion from our research is that investing in the Knowledge Graph provides much higher accuracy for LLM-powered question-answering systems on SQL databases. And ultimately, to succeed in this AI world, enterprises must treat the business context and semantics as a first-class citizen,” says Dr. Juan Sequeda, Head of the data.world AI Lab.

The results of the data.world report has been validated by members of dbt Labs’ developer experience team through a series of tests. The test results showed an 83 percent accuracy rate for natural language questions being answered through AI, with several questions that delivered correct answers in 100 percent of attempts. 

(CREATIVE WONDER/Shutterstock)

The testing by dbt Labs showed that layering structured Semantic Knowledge on top of data leads to a much stronger ability to correctly answer ad-hoc questions about organizational data with LLMs. 

According to data.world CEO Brett Hurt, there is no doubt about the productivity lift offered by LLMs but the challenge is how to use LLMs in a way that the results are explainable, accurate, and governed. Hurt believes that the bench report by data.world shows the power of combining LLMs with Knowledge Graphs. 

Organizations that have been reluctant to use LLM due to fear of inaccuracies in the business context, however, the data.world report indicates that now organizations can be more optimistic. They can incorporate Knowledge Graphs as part of their technical strategy to boost the LLM accuracy rate. 

Whether organizations are looking to use AI to improve on operational inefficiencies, increase revenue, or gather customer feedback, the accuracy of the AI is of critical importance. The data.world report has offered vital insight into the direction that can lead to more accurate AI results. 

Related Items 

How to Accelerate Time-to-Insight with Knowledge Graph

data.world Expands Snowflake Collaboration with New Exclusive Offering to Accelerate Data Trust

Data Fabrics: The Killer Use Case for Knowledge Graphs