Testing AI agents is often slowed down by the need to collect, clean, and maintain evaluation datasets. In a new Microsoft Mechanics short, Azure AI Foundry is shown generating synthetic test data for an agent evaluation workflow, then pairing that data with built-in criteria for quality, agent behavior, and safety checks.
What Microsoft demonstrated
The video focuses on a compact but important part of the AI application lifecycle: preparing data for evaluation. Instead of uploading an existing dataset, the workflow creates a synthetic dataset directly in Azure AI Foundry. The example prompt asks the tool to create data for evaluating a coding agent, and the demo generates 90 rows that can be used as test inputs.
From there, the evaluation configuration moves to criteria selection. Azure AI Foundry includes built-in evaluators for agents, quality dimensions, and safety. The demo also shows that some evaluation criteria can be edited or removed when they are not relevant to the specific agent being tested.
Why synthetic test data matters
For IT and cloud teams, synthetic evaluation data can shorten the path from prototype to validated deployment. Real production data may be sensitive, incomplete, hard to label, or simply unavailable early in a project. Synthetic data gives teams a way to start testing behavior before they have a mature dataset or a fully instrumented production environment.
This is especially useful for agentic systems. A coding agent, support assistant, or operations copilot can behave differently depending on task phrasing, missing context, policy constraints, or unsafe requests. Generating many test cases quickly helps teams explore those edge cases earlier and more consistently.
Operational impact for AI teams
The practical value is not just speed. A repeatable evaluation workflow helps teams standardize how they decide whether an AI agent is ready for broader use. Built-in evaluators reduce the amount of custom scoring infrastructure teams need to build, while editable criteria let them align the test plan with the agent's actual purpose.
The safety evaluators are also important. As organizations move from experimentation to production AI, they need evidence that agents are being checked for risky outputs and policy violations. Keeping safety criteria in the evaluation pipeline makes responsible AI review part of the normal deployment process rather than a separate last-minute activity.
Key takeaways
- Azure AI Foundry can generate synthetic datasets when an existing evaluation dataset is not available or would take too long to prepare.
- Teams can guide dataset creation with a prompt that describes the agent or scenario being tested.
- Built-in evaluators help assess agent behavior, output quality, and safety risks.
- Evaluation criteria should be tailored to the agent's actual workload; unnecessary checks can create noise, while missing safety checks can create blind spots.
- Synthetic data is best used as part of a broader testing strategy, not as a complete replacement for real-world validation.
Bottom line
This Microsoft Mechanics demo highlights a practical pattern for enterprise AI delivery: use synthetic test data to move faster, then apply structured evaluators to make the results measurable. For cloud teams building agents on Azure, the combination can reduce setup friction, improve consistency, and make safety evaluation easier to include from the start.
Source: Microsoft Mechanics video