Evaluation

Dans .NET, le framework d’évaluation s’appuie sur Microsoft. Extensions.AI.Evaluation. Les évaluateurs implémentent l’interface IAgentEvaluator, et l’orchestration est fournie via des méthodes d’extension sur AIAgent et Run.

Les types principaux résident dans l’espace de noms Microsoft.Agents.AI :

using Microsoft.Agents.AI;

Dans Python, l’infrastructure d’évaluation fait partie du package principal agent_framework. Les évaluateurs implémentent le protocole Evaluator, et l'orchestration est fournie par les fonctions evaluate_agent() et evaluate_workflow().

from agent_framework import (
    evaluate_agent,
    evaluate_workflow,
    EvalItem,
    EvalResults,
    LocalEvaluator,
)

Vérifications intégrées

Agent Framework est fourni avec des vérifications intégrées pour les scénarios courants :

using Microsoft.Agents.AI;

var local = new LocalEvaluator(
    EvalChecks.KeywordCheck("weather", "temperature"),  // Response must contain these keywords
    EvalChecks.ToolCalledCheck("get_weather")            // Agent must have called this tool
);

Évaluateurs de fonction personnalisés

Permet FunctionEvaluator.Create() d’encapsuler n’importe quelle fonction en tant que vérification de l’évaluateur. Plusieurs surcharges de fonction sont disponibles selon les données requises :

using Microsoft.Agents.AI;

var local = new LocalEvaluator(
    // Simple: check only the response text
    FunctionEvaluator.Create("is_concise",
        (string response) => response.Split(' ').Length < 500),

    // With expected output: compare against ground truth
    FunctionEvaluator.Create("mentions_city",
        (string response, string? expectedOutput) =>
            expectedOutput != null && response.Contains(expectedOutput, StringComparison.OrdinalIgnoreCase)),

    // Full context: access the complete EvalItem
    FunctionEvaluator.Create("used_search",
        (EvalItem item) => item.Conversation.Any(m =>
            m.Text?.Contains("search", StringComparison.OrdinalIgnoreCase) == true))
);

Vérifications intégrées

Agent Framework est fourni avec des vérifications intégrées pour les scénarios courants :

from agent_framework import (
    LocalEvaluator,
    keyword_check,
    tool_called_check,
    tool_calls_present,
    tool_call_args_match,
)

local = LocalEvaluator(
    keyword_check("weather", "temperature"),  # Response must contain these keywords
    tool_called_check("get_weather"),          # Agent must have called this tool
    tool_calls_present,                        # All expected tool call names were made
    tool_call_args_match,                      # Expected tool calls match on name + args
)

Évaluateurs de fonction personnalisés

Utilisez le décorateur @evaluator pour envelopper n'importe quelle fonction en tant que vérification d'évaluation. Les noms de paramètres de la fonction déterminent les données qu’elle reçoit à partir des EvalIteméléments suivants :

from agent_framework import evaluator, LocalEvaluator

@evaluator
def is_concise(response: str) -> bool:
    """Check response is under 500 words."""
    return len(response.split()) < 500

@evaluator
def mentions_city(response: str, expected_output: str) -> bool:
    """Check response contains the expected city name."""
    return expected_output.lower() in response.lower()

@evaluator
def used_tools(conversation: list, tools: list) -> float:
    """Score based on tool usage. Returns 0.0–1.0 (>= 0.5 passes)."""
    tool_calls = [c for m in conversation for c in (m.contents or []) if c.type == "function_call"]
    return min(len(tool_calls) / max(len(tools), 1), 1.0)

local = LocalEvaluator(is_concise, mentions_city, used_tools)

Noms de paramètres pris en charge : query, , response, expected_outputexpected_tool_callsconversationtools, , . context

Types de retour : bool, float (≥ 0,5 = pass), dict avec score ou passed clé, ou CheckResult. Les fonctions asynchrones sont gérées automatiquement.

using Microsoft.Agents.AI.AzureAI;

var foundry = new FoundryEvals(chatConfiguration, FoundryEvals.Relevance, FoundryEvals.Coherence);

from agent_framework.foundry import FoundryEvals

evals = FoundryEvals(
    project_client=project_client,
    model="gpt-4o",
    evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
)

using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry;

var foundry = new FoundryEvals(chatConfiguration, FoundryEvals.Relevance, FoundryEvals.Coherence);

AgentEvaluationResults results = await agent.EvaluateAsync(
    new[]
    {
        "What's the weather in Seattle?",
        "Plan a weekend trip to Portland",
        "What restaurants are near Pike Place?",
    },
    foundry);

results.AssertAllPassed();  // Throws if any item failed

EvaluateAsync est une méthode d’extension sur AIAgent. Il exécute l’agent une fois par requête, convertit chaque interaction en un EvalItem, et transmet le lot à l’évaluateur.

from agent_framework import evaluate_agent
from agent_framework.foundry import FoundryEvals

evals = FoundryEvals(
    project_client=project_client,
    model="gpt-4o",
    evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
)

results = await evaluate_agent(
    agent=my_agent,
    queries=[
        "What's the weather in Seattle?",
        "Plan a weekend trip to Portland",
        "What restaurants are near Pike Place?",
    ],
    evaluators=evals,
)

for r in results:
    print(f"{r.provider}: {r.passed}/{r.total}")
    r.raise_for_status()  # Raises EvalNotPassedError if any item failed

evaluate_agent exécute l’agent une fois par requête, convertit chaque interaction en un EvalItem, et transmet le lot à l’évaluateur. Il retourne un EvalResults par fournisseur évaluateur.

AgentEvaluationResults results = await agent.EvaluateAsync(
    new[] { "What's the weather in Seattle?" },
    foundry,
    numRepetitions: 3);  // Each query runs 3 times independently
// Results contain 3 items (1 query × 3 repetitions)

results = await evaluate_agent(
    agent=my_agent,
    queries=["What's the weather in Seattle?"],
    evaluators=evals,
    num_repetitions=3,  # Each query runs 3 times independently
)
# Results contain 3 items (1 query × 3 repetitions)

AgentEvaluationResults results = await agent.EvaluateAsync(
    new[] { "What's 2+2?", "Capital of France?" },
    foundry,
    expectedOutput: new[] { "4", "Paris" });

Vous pouvez également spécifier les appels d’outils attendus :

AgentEvaluationResults results = await agent.EvaluateAsync(
    new[] { "What's the weather in NYC?" },
    new LocalEvaluator(EvalChecks.ToolCalledCheck("get_weather")),
    expectedToolCalls: new[]
    {
        new[] { new ExpectedToolCall("get_weather") },
    });

from agent_framework import evaluate_agent, ExpectedToolCall

results = await evaluate_agent(
    agent=my_agent,
    queries=["What's 2+2?", "Capital of France?"],
    expected_output=["4", "Paris"],
    evaluators=evals,
)

Vous pouvez également spécifier les appels d’outils attendus :

results = await evaluate_agent(
    agent=my_agent,
    queries=["What's the weather in NYC?"],
    expected_tool_calls=[ExpectedToolCall("get_weather", {"location": "NYC"})],
    evaluators=local,
)

var response = await agent.RunAsync(new[] { new ChatMessage(ChatRole.User, "What's the weather?") });

AgentEvaluationResults results = await agent.EvaluateAsync(
    new[] { response },
    new[] { "What's the weather?" },
    foundry);

from agent_framework import Message, evaluate_agent

response = await agent.run([Message("user", ["What's the weather?"])])

results = await evaluate_agent(
    agent=agent,
    responses=response,
    queries="What's the weather?",
    evaluators=evals,
)

// Full conversation as context
AgentEvaluationResults results = await agent.EvaluateAsync(
    new[] { "Plan a 3-day trip to Paris" },
    foundry,
    splitter: ConversationSplitters.Full);

// Per-turn: each exchange scored independently
var items = EvalItem.PerTurnItems(conversation);
var perTurnResults = await evaluator.EvaluateAsync(items);

Vous pouvez également implémenter un séparateur personnalisé en implémentant IConversationSplitter:

public class SplitBeforeToolCall : IConversationSplitter
{
    public (IReadOnlyList<ChatMessage> QueryMessages, IReadOnlyList<ChatMessage> ResponseMessages) Split(
        IReadOnlyList<ChatMessage> conversation)
    {
        // Custom split logic
        for (int i = 0; i < conversation.Count; i++)
        {
            if (conversation[i].Text?.Contains("tool_call") == true)
                return (conversation.Take(i).ToList(), conversation.Skip(i).ToList());
        }
        return ConversationSplitters.LastTurn.Split(conversation);
    }
}

from agent_framework import evaluate_agent, ConversationSplit

# Full conversation as context
results = await evaluate_agent(
    agent=agent,
    queries=["Plan a 3-day trip to Paris"],
    evaluators=evals,
    conversation_split=ConversationSplit.FULL,
)

# Per-turn: each exchange scored independently
from agent_framework import EvalItem

items = EvalItem.per_turn_items(conversation)
# Pass items directly to an evaluator
per_turn_results = await evaluator.evaluate(items)

Vous pouvez également fournir un séparateur personnalisé : toute fonction appelable qui prend une conversation et retourne (query_messages, response_messages).

def split_before_memory(conversation):
    """Split just before a memory-retrieval tool call."""
    for i, msg in enumerate(conversation):
        for c in msg.contents or []:
            if c.type == "function_call" and c.name == "retrieve_memory":
                return conversation[:i], conversation[i:]
    # Fallback to default
    return EvalItem._split_last_turn_static(conversation)

results = await evaluate_agent(
    agent=agent,
    queries=queries,
    evaluators=evals,
    conversation_split=split_before_memory,
)

using Microsoft.Agents.AI;
using Microsoft.Agents.AI.AzureAI;

Run run = await workflowRunner.RunAsync(workflow, "Plan a trip to Paris");

AgentEvaluationResults results = await run.EvaluateAsync(
    new FoundryEvals(chatConfiguration, FoundryEvals.Relevance));

Console.WriteLine($"Overall: {results.Passed}/{results.Total}");

// Per-agent breakdown
if (results.SubResults != null)
{
    foreach (var (name, sub) in results.SubResults)
    {
        Console.WriteLine($"  {name}: {sub.Passed}/{sub.Total}");
    }
}

results.AssertAllPassed();

from agent_framework import evaluate_workflow
from agent_framework.foundry import FoundryEvals

evals = FoundryEvals(project_client=project_client, model="gpt-4o")
result = await workflow.run("Plan a trip to Paris")

eval_results = await evaluate_workflow(
    workflow=workflow,
    workflow_result=result,
    evaluators=evals,
)

for r in eval_results:
    print(f"{r.provider}: {r.passed}/{r.total}")
    for name, sub in r.sub_results.items():
        print(f"  {name}: {sub.passed}/{sub.total}")

Vous pouvez également passer queries directement et l’infrastructure exécute le flux de travail pour vous :

eval_results = await evaluate_workflow(
    workflow=workflow,
    queries=["Plan a trip to Paris", "Book a flight to London"],
    evaluators=evals,
)

using Microsoft.Agents.AI;
using Microsoft.Agents.AI.AzureAI;

IReadOnlyList<AgentEvaluationResults> results = await agent.EvaluateAsync(
    new[] { "What's the weather in Seattle?" },
    evaluators: new IAgentEvaluator[]
    {
        new LocalEvaluator(
            EvalChecks.KeywordCheck("weather"),
            FunctionEvaluator.Create("is_helpful", (string r) => r.Split(' ').Length > 10)),
        new FoundryEvals(chatConfiguration, FoundryEvals.Relevance, FoundryEvals.Coherence),
    });

// results[0] = local evaluator results
// results[1] = Foundry evaluator results
foreach (var r in results)
{
    Console.WriteLine($"{r.Provider}: {r.Passed}/{r.Total}");
}

from agent_framework import evaluate_agent, evaluator, LocalEvaluator, keyword_check
from agent_framework.foundry import FoundryEvals

@evaluator
def is_helpful(response: str) -> bool:
    return len(response.split()) > 10

foundry = FoundryEvals(
    project_client=project_client,
    model="gpt-4o",
    evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.COHERENCE],
)

results = await evaluate_agent(
    agent=agent,
    queries=["What's the weather in Seattle?"],
    evaluators=[
        LocalEvaluator(is_helpful, keyword_check("weather")),
        foundry,
    ],
)

# results[0] = local evaluator results
# results[1] = Foundry evaluator results
for r in results:
    print(f"{r.provider}: {r.passed}/{r.total}")

Évaluateurs MEAI

Le framework d’évaluation .NET s’intègre directement à Microsoft. Extensions.AI.Evaluation évaluateurs. Les évaluateurs de qualité et de sécurité de MEAI travaillent sans adaptateur :

using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Microsoft.Extensions.AI.Evaluation.Safety;

// Quality evaluators
AgentEvaluationResults results = await agent.EvaluateAsync(
    new[] { "What's the weather?" },
    new CompositeEvaluator(
        new RelevanceEvaluator(),
        new CoherenceEvaluator(),
        new GroundednessEvaluator()),
    chatConfiguration: new ChatConfiguration(evalClient));

// Safety evaluators
AgentEvaluationResults safetyResults = await agent.EvaluateAsync(
    new[] { "What's the weather?" },
    new ContentHarmEvaluator(),
    chatConfiguration: new ChatConfiguration(evalClient));