Ground AI in company data
Give assistants and agents permission-aware access to company documents and data through retrieval (Azure AI Search and Foundry IQ) or Microsoft 365 grounding (Work IQ), instead of retraining models on private content.
The problem
A model only knows what it was trained on. It has a knowledge cutoff and cannot see your policies, contracts, product data or tickets. Company content is spread across SharePoint, databases, blob storage and the web, and models accept a limited amount of input per request.
Security is the hard part: an assistant must return only what the person asking is allowed to see, even when a senior user asks about another department's data.12
The approach
Pick the grounding layer by audience.
- Staff working in Microsoft 365: licensed Microsoft Copilot already grounds answers in Microsoft Graph, Work IQ and semantic indexing, within each user's permissions. Copilot Chat grounds in the web plus content users upload or open, and in limited mailbox content in Outlook.
- Custom apps and agents: use Azure AI Search. Microsoft recommends agentic retrieval for new RAG work: a model plans subqueries, runs them in parallel across knowledge sources and returns results with citations. Query planning is in preview, so Microsoft points to classic RAG (hybrid keyword and vector search with semantic ranking) when only generally available features are acceptable.
- Shared knowledge for many agents: Foundry IQ knowledge bases, built on Azure AI Search, connect sources such as Azure Blob Storage, SharePoint, OneLake and the web. They can sync access control lists for supported sources, honor Purview sensitivity labels and run queries under the caller's Entra identity. Agents in Foundry Agent Service, Microsoft Agent Framework or any custom app can call them.
Prepare content properly: chunking, embeddings, OCR for scanned files and scheduled refresh. Measure groundedness and relevance with evaluators before launch.
When a simpler route is enough: if users only need answers from a fixed set of documents, classic RAG without an agent is enough. Fine-tuning is not a substitute: Microsoft says it does not replace retrieval for current information.12345
Conceptual architecture
Diagram as a list
Agents & automation
- Agent or applicationCalls the knowledge base and presents answers
AI models & platform
- Agentic retrieval (query planning)Splits questions into subqueries and reranks resultsConnects to Foundry Models
- Foundry ModelsWrites the grounded answer with citationsConnects to Agent or application
Data & analytics
- Sources: SharePoint, Azure Blob Storage, OneLake, Azure SQL, webHold the content to ground onConnects to Indexing pipeline (chunking, embeddings, OCR)
- Indexing pipeline (chunking, embeddings, OCR)Turns documents into searchable chunks with metadataConnects to Azure AI Search index
- Azure AI Search indexKeyword, vector and hybrid retrievalConnects to Foundry IQ knowledge base
- Foundry IQ knowledge baseCombines sources and retrieval settings, shared by agentsConnects to Agentic retrieval (query planning)
Security, identity & governance
- Microsoft Entra identity and Purview labelsEnforce who may see which content at query time
Technologies and their roles
Azure AI Search2
Retrieval engine
Provides indexing, hybrid search, semantic ranking and agentic retrieval for RAG.
Microsoft Foundry1
Knowledge layer (Foundry IQ)
Foundry IQ knowledge bases give agents permission-aware answers with citations.
Microsoft Foundry Agent Service6
Agent runtime
Hosted and prompt agents can connect to Foundry IQ knowledge bases and tools such as SharePoint.
Microsoft 365 Copilot3
Grounding for Microsoft 365 users
Licensed Copilot grounds in Graph, Work IQ and semantic indexing across Microsoft 365.
Microsoft Purview1
Sensitivity labels
Foundry IQ honors Purview sensitivity labels when it enforces permissions.
What you need first1
- An inventory of sources with owners, refresh needs and permission models
- Permissions cleaned up at the source, because retrieval respects what exists
- An Azure AI Search service that supports agentic retrieval, and a model deployment for query planning
- Microsoft Entra identities for users and agents so queries can run under the caller's identity
- A test set of real questions with expected sources
Risks and how to reduce them
- Answers include content the user should not see2
- Use sources that support permission sync, run queries under the caller's identity and apply filter-based security for other sources.
- Preview features end up in production1
- Check which agentic retrieval features are generally available for your Search REST API version; use classic RAG where GA is required.
- Stale answers from outdated content1
- Schedule indexer runs and retire obsolete documents at the source.
- Weak relevance or unsupported claims7
- Use hybrid queries and semantic ranking, and track groundedness and relevance evaluators.
Documented examples
Premier League
In productionPremier League: an AI fan companion built on Microsoft Foundry
up approximately 20% (measured)
Media and entertainment, United Kingdom (global audience)
Related
Sources
- What is Foundry IQ? (opens in a new tab)
- RAG and generative AI: Azure AI Search (opens in a new tab)
- Overview of Microsoft Copilot Chat (opens in a new tab)
- Fine-tuning in Microsoft Foundry (opens in a new tab)
- Business plan for AI agents (Cloud Adoption Framework) (opens in a new tab)
- What is Microsoft Foundry Agent Service? (opens in a new tab)
- Observability in generative AI (opens in a new tab)
Thank you. Your correction was sent.
The editors check it against the sources. If you left an email address, they may reply about it.