Microsoft AI-103: Building Agents That Stay Grounded
An AI agent that answers accurately in a demonstration may behave very differently once it encounters conflicting documents, permission boundaries, unavailable tools, or users who ask it to do something unexpected. Microsoft AI-103, Developing AI Apps and Agents on Azure, takes that practical problem seriously. Its scope combines Microsoft Foundry, generative and agentic applications, evaluation, and multimodal AI capabilities.
Microsoft’s AI-103 skills guide defines the current assessed engineering responsibilities. Preparation should begin with that issuer outline and then test not only whether a feature can be assembled but why specific models, grounding methods, deployment choices and safeguards fit the workload. No third-party question collection, including an unverified product listing, establishes the official content or provenance of the exam.
Start with a support application that must answer questions about warranty terms and guide customers through troubleshooting. A large language model may offer flexible dialogue, but that does not automatically make it the best component for every subtask. A small model may be adequate for constrained classification; a multimodal model may be needed when customers submit images; a retrieval system may be essential when the answer depends on approved policy documents.
AI-103 expects candidates to compare model capabilities, use Microsoft Foundry, choose deployment methods, and plan for quota, latency, cost, and scaling. A technically impressive design can still fail if it exceeds rate limits during peak traffic or produces unpredictable costs. Include those operational constraints from the first architecture sketch. Earlier Azure AI engineering work is placed in context by Microsoft AI-102's historical AI-engineering path, which is useful background but not an active examination route. The AI-103 Foundry deployment architecture guide is useful when model selection starts to affect networking, quotas and deployment choices.
Retrieval-augmented generation can make an application more useful by placing relevant reference material in context. The hard part is proving that the retrieved information is right for the question. An index containing duplicate policy versions or badly segmented documents may return plausible but wrong evidence. A citation pointing to a document is not proof that the generated interpretation is faithful.
Design test cases where two policies disagree, a document is missing, or the question refers to an exception buried in an appendix. Inspect retrieval relevance separately from answer quality. If the right passage never reaches the model, prompt polishing alone may not solve the problem. Microsoft includes search-index health, grounding quality, evaluation, and monitoring in the exam scope for good reason. Grounding alone does not define safe actions; Microsoft AB-620 Copilot Studio agents explores what changes when an assistant gains tools and cross-system permissions.
A customer asks whether a product purchased six weeks ago can be returned. The search index contains an old 30-day policy, a newer 45-day policy for selected products, and an internal exception intended only for authorized support staff. All three documents look relevant; only one applies to the customer, region and purchase date. Define document authority and access scope before tuning search. Without effective dates, product metadata and audience permissions, even a beautifully phrased answer can cite the wrong rule.
Build an evaluation set with three scenarios: a straightforward return under the current policy, a purchase outside the applicable period, and a request involving an internal exception the customer must not see. Record the expected evidence passage for each. Examine separately whether retrieval returns the correct source and whether the model represents it faithfully. Lexical retrieval, vector search, hybrid ranking and semantic reranking can change the candidate passages, but the right setting is the one shown to improve these cases. Fluent language is not an evaluation metric for correctness.
Now remove the current document from the test index. The assistant should identify the evidence gap instead of inventing a refund rule based on a superseded document. A useful response can explain what is known, mark what needs confirmation and route the request to support. Diagnose retrieval misses, stale ingestion and unsupported model inferences separately; each requires a different fix.
Test the warranty assistant with two customers who have different permissions and policy eligibility. Give each search request the actual caller identity and check that document-level permissions were ingested and enforced; a prompt asking the model not to reveal a staff-only rule is not sufficient. The assistant should cite the correct dated policy without surfacing another account or an internal exception. Repeat the test after policy changes and document deletion, because stale indexed access metadata can undermine otherwise accurate grounded answers.
Be precise about how the test user’s identity reaches search. The Azure AI Search identity-string filter compares stored user or group identifiers, but does not authenticate customers or confirm their memberships. Derive the caller and authorized groups from the application’s verified session, not a user-supplied question or tool argument. Remove the staff role from a test account, repeat the exact warranty query, and prove that the internal exception is absent from retrieved passages. If the agent switches from vector to lexical retrieval, the same caller-derived restriction must apply to that fallback rather than trusting a prompt to conceal the text.
A warranty assistant might be allowed to retrieve an order’s status and create a service ticket. It should not silently authorize a refund beyond policy or disclose another customer’s order. Define tool schemas that express permitted inputs, validate them before execution, and separate read access from business-changing actions. Approval workflows make sense where the consequence of an incorrect action is meaningful.
AI-103 covers agent roles, knowledge, tools, function calling, memory, and orchestrated multi-agent solutions. Multi-agent design is not a shortcut around access controls. If one agent asks another to act, the system still needs to know which user initiated the request, what authority applies, and how to trace the outcome.
A read-only order lookup and ticket creation call require different safeguards. Check the initiating customer’s identity and the lookup’s data scope. For ticket creation, validate the tool inputs, associate the action with an authorized user and save an auditable result. If the network times out after creating the ticket, a blind retry can create duplicates. Use an idempotency key or a comparable deduplication mechanism in the application design, and explicitly test the timeout and retry sequence rather than assuming function calling is exactly-once.
The agent can propose a refund or escalation, but a policy service and the caller’s permissions decide what may execute. For a ticket or refund tool, validate order ID, eligibility and approved amount before acting. Supply an operation identifier so a timed-out call can be reconciled without a duplicate write. The trace should distinguish proposed action, human approval, persisted result and any unknown outcome. A confident model response after a failed confirmation is not proof that the customer received a refund; stop or escalate if state is uncertain.
The assessment is not solely a generative chat exam. Microsoft’s objectives include computer vision, image and video understanding, text analysis, and information extraction. Think again about the warranty scenario. A customer may upload a photo of a damaged product and a scanned receipt. The solution needs to interpret visual evidence and extract relevant fields, not merely discuss the case in text.
A well-designed pipeline might classify an attachment, extract specific information, attach confidence or verification signals, and present an uncertain result for review. The task determines whether to use a specialized service, multimodal model, or a combined approach. Do not assume that the newest general model automatically replaces every focused tool.
Production AI needs telemetry beyond successful API responses. Monitor response times, token use, tool failures, retrieval quality, safety issues, and the proportion of tasks that require human intervention. A sudden drop in usefulness may be caused by data ingestion problems, a changed source document, or an external service issue rather than by the base model.
The official outline also covers managed identity, private networking, role policies, keyless authentication, guardrails, trace logging, and evaluation. A trustworthy solution must be secure and explainable to its operators. Logging should support investigation without turning into uncontrolled storage of sensitive user content.
For focused revision, prototype a warranty-support assistant in a test environment. Give it a small document collection with conflicting versions, one read-only order lookup, one ticket-creation action, a customer-uploaded image, and a set of adversarial or ambiguous questions. Write down how each part is verified and how the app behaves when it cannot answer safely.
This exercise covers the central AI-103 distinction: a useful AI application is not just generated language. It is a monitored system that can retrieve evidence, perform bounded actions, process different input types, and respond predictably when its knowledge or authority runs out.
