Companies Look Past Chatbots for AI Payoff

The disconnect reflects the fact that discussion of AI, and the public’s understanding of it, largely focuses on the public-facing chatbots. Behind the scenes, a lot of companies are deploying more specialized, internal AI tools that tap their data—and that is where they are looking for the real payoff from AI.

Such companies are connecting large language models to their data with a technique known as retrieval-augmented generation, or RAG. It is an obscure term, but at heart the idea is fairly straightforward: Retrieve a company’s data and use it to augment the work of generative AI. 

“It’s massive. Most of what we do is RAG-based,” said Sylvain Duranton, global leader of BCG X, a tech build-and-design unit of Boston Consulting Group that has worked with large corporations.

AI’s infamous hallucinations reflect the fact that large language models are trained on the internet. It was possible to prompt an AI chatbot to expound on the differences between eggs from a hen and eggs from a cow because it was trained on internet data that included a videogame with cow eggs, Duranton said. 

If a company creates an AI-based call-center agent, on the other hand, it will use RAG technology to import the processes, procedures and data used by human agents or operators. The agent’s large language model extracts its answers from that data.

“It’s a way to protect yourself,” Duranton said.

Once in place, however, these systems can open up a range of uses that go beyond call-center agents and other now-familiar enterprise AI applications such as writing software code. 

Shorenstein Properties, a real-estate investment company based in San Francisco, is in a pilot program that is designed to lead to the automated tagging of all of its files using a RAG-based AI system. The goal is to eliminate many of the drawbacks in a time-consuming manual system, in which people might make errors or simply skip the process altogether. The company plans to put the tagging system into production in the next few months.

Files can also be organized quickly into “knowledge bases” and interrogated with AI, according to Egnyte, a cloud-based platform that companies use to access, share and manage business content.

Shorenstein in the past few weeks has started a proof of concept project using Egnyte to extract data from prospectuses on properties for sale, documents that can often run 60 pages, and organize it into reports that could help the company make efficient business decisions and improve processes.

The creation of knowledge bases also might allow Shorenstein to analyze lease data for a city and almost instantly assess which ones are about to expire, which it says helps with market and data analysis, according to Sam Ghnaim, senior vice president, information technology, at Shorenstein.

Developing these new capabilities requires more than just tech.

Technology previously allowed for a decentralized approach to IT architecture, according to Duranton, the BCG X leader. But today’s AI requires constant access to all of an enterprise’s data, which means it needs to be deeply and broadly interconnected to the entire architecture, he said.

Deeper applications of AI often will require equally deep changes in workflow and process as well, according to Heidi Messer, chairperson and co-founder of Collective[i], an enterprise AI company that produces intelligence, applications and agents designed to generate revenue insights and optimization across business functions.

A corporation’s revenue team might forecast demand each quarter, for example, by polling business leaders throughout the company. It is hardly a scientific process, which explains why surprises are relatively common when companies report their actual results.

Over time, it might turn out that AI makes more reliable revenue predictions. But that would require changing the way the revenue team works. The older practice of data collection would come to an end, and the team might work more closely with other units in the company on functions such as supply chain management, pricing and hiring, Messer said.

“There are much deeper applications of AI that go beyond LLMs, and they will have a massive impact,” she said. “But to realize those gains, companies will need to disrupt or replace traditional workflows with ways of working that align with these technological advancements.”

Write to Steven Rosenbush at steven.rosenbush@wsj.com