Black box AI isn’t enough: Why enterprise consulting is moving to grounded models
Presented by SAP
In an age where anyone can set up an LLM, the real differentiator is not the AI technology itself, but the institutional knowledge on which it is based. Internal and partner consultants leading operational transformation cannot risk hallucinating if their recommendations impact integrated processes across supply chain, manufacturing, finance and other core functions.
“Grounded AI is non-negotiable because accuracy is not optional when, for example, we execute multi-million dollar transformation projects within the SAP ecosystem,” said Natalie Han, VP and Chief Product Officer, Gen AI at SAP Business AI. “Retrieval-augmented generation technology and the ability to anchor answers in trusted business knowledge provide accurate code interpretation, best practice guidance, and clear decision-making support, bringing true trust to AI-powered advice.”
A fully-fledged AI assistant like SAP Joule for Consultants has tremendous value in production use cases, she adds. SAP Joule has terabytes of institutional data that is continuously managed and updated, so a consultant can be assured that he or she has the latest SAP best practices and methodologies when relying on Joule, while accelerating project delivery.
“We save 14% rework time and save consultants 1.5 hours per day per user, which is huge considering how expensive consultants are now,” says Han. “Early adopters like Wipro have estimated that they have saved 7 million manual hours for their consultants.”
The basis of SAP Joule
SAP Joule is as certified as any consultant, says Sachin Kaura, chief architect of SAP Business AI. The tool was born in 2023, when GPTs famously passed a simulated bar exam and created buzz about LLMs’ ability to handle large amounts of context. It is widely recognized that the SAP ecosystem, along with its associated domain ontology and taxonomy, is incredibly large and very complex to navigate. The question became: how could an AI co-pilot be used to navigate that complexity when it was actually anchored in the SAP ecosystem itself?
Sachin Kaura started experimenting with groundbreaking LLM models by putting them through the same certification exams as SAP consultants. Initial results were poor, but after extensive context tuning and a focus on delivering value to the partner ecosystem, Joule now consistently scores 95% or higher.
“Not only were we testing from a data perspective, but we were also able to work with all of our consultants to arrive at what we call the gold data set,” Han added. “It is non-deterministic, language-based, and deeply based on the expertise of human consultants. We worked with the entire consulting organization to manually label the gold data set for all products. That has become the foundation for everything we do even now.”
A state-of-the-art indexing pipeline
Joule for Consultants remains up to date in real time. A state-of-the-art indexing pipeline pushes new SAP documentation and release content into the model as it is published, giving consultants confidence that every response reflects the most up-to-date guidance.
“This is purely technical work done by our data scientists and engineers, using a lot of underlying SAP technology,” Kaura explains. “We leverage SAP’s business base layer, document base services, and a host of purpose-built systems to stay on top of current events in the system.”
SAP Business AI also has alignment at the board level, making this not just a one-team effort, but a company-wide priority. They have built strong internal partnerships with content owners across SAP, including SAP Learning, SAP Community, SAP Help, product teams, and consultant teams. Together they continuously update proprietary content such as SAP Notes, Knowledge Base Articles (KBAs), and other domain-specific guidelines that reflect SAP’s evolving best practices.
All this means that Joule for Consultants can use that constantly updated data and provide answers in near real time. It’s the kind of research that would otherwise take a consultant hours to complete. But information straight from the source gives consultants the most current and authoritative guidance available, eliminating the early missteps that can derail a project months later if the scope isn’t aligned with the latest capabilities.
Ensuring enterprise-level security
SAP builds a product that is relevant, reliable and responsible, says Han. As a company incorporated in Europe, it takes data privacy seriously and adheres to GDPR and other EU business regulations. At the heart of SAP Business AI is the AI Foundation, the AI operating system that powers AI with built-in security, ethics and orchestration, using automation and intelligence to manage lifecycles, optimize resources and increase resilience.
All LLMs used by SAP and its customers operate within the AI foundation, which protects private and proprietary data from leaks. In addition to data protection, SAP also addresses bias, ethics and security at the enterprise level, involving humans in performing checks and balances.
“We have an enterprise-level security framework in place, as well as rapid injection and guardrail testing,” says Kaura. “The orchestration layer, built within the AI Foundation, anonymizes the input and moderates it to prevent malicious content. That ensures that the output we give to our customers is relevant to the SAP ecosystem, relevant to the domain they are asking for, and not just generic LLM excess. This set of tools, from the framework layer to the application layer to the product standards, and also the very thorough testing is crucial to secure our product. Then and only then can our customers and partners achieve.”
Pushing the boundaries of Joule for consultants
“We’ve barely scratched the surface of what LLMs and agentic AI have to offer,” says Han. “Gaining access to knowledge is just the beginning. We will gain a much deeper understanding of customers’ SAP systems and help them implement and transform their journey. The product team and our engineers are working to make the tool more transformative, capable of gaining more insights, connecting to customers’ systems, and understanding and optimizing their processes, including code generation and handling customer code migration.”
The next step is to add a second layer of grounding. SAP’s customer base is vast and its partner ecosystem has implemented numerous business scenarios. Anchoring Joule in SAP’s institutional knowledge was the first milestone; the next is layering each customer’s own context: historical system data, process designs, implementation blueprints, and internal documentation. As a result, Joule changes from SAP-conscious to customer-conscious and offers guidance that matches the way a company actually operates.
“Think of it as anchoring your knowledge on top of SAP knowledge, giving you more accurate and relevant guidance,” says Kaura. “Information that might otherwise be lost can be stored on top of Joule for Consultants. Our system processes this information and ensures it gets to you in the right way and at the right time.”
Thanks to this extensive basis, Joule can adapt his guidance to the role of the consultant – whether he or she works as an architect, functional consultant or technical consultant.
“We provide the information they need for a particular customer configuration,” Han explains. “Then we can answer not only generic questions, but also their specific configuration. From there it is a step further to generate more insights and take more actions.”
Sponsored articles are content produced by a company that pays for the post or has a business relationship with VentureBeat, and is always clearly marked. For more information please contact sales@venturebeat.com.
