By AI Agent Metric

The By AI Agent metric lets supervisors configure AI-driven evaluations that assess multiple aspects of a conversation using agents hosted on the Agent Platform. It uses a two-level structure — a parent metric containing sub-metrics, each with its own evaluation question, weight, and adherence logic. A single agentic evaluation call can analyze several aspects of a conversation, returning structured responses and justifications for each sub-metric.

When to Use This Metric

Use By AI Agent for evaluation scenarios that require:

Scenario	Description
Multi-dimensional assessments	Evaluate several facets under one parent metric
Autonomous AI analysis	Leverage AI Agents to interpret and assess interactions contextually
Weighted evaluations	Assign different weights to sub-metrics to prioritize specific aspects
Efficient execution	Reduce API calls by evaluating multiple sub-metrics in one request
Seamless configuration	Select agentic apps from the same workspace without entering endpoint URLs

Prerequisites

Before creating a By AI Agent metric, confirm:

You have access to both Quality AI and Agent Platform.
The same workspace is available across both platforms.
You have permissions to view and deploy agentic apps.
The By AI Agent Metric feature is enabled for your workspace.
At least one agentic app is configured on the Agent Platform with the required response structure.

If no agentic app is configured in your workspace, the Agent App dropdown shows no options. If the agentic app’s response structure doesn’t match the required contract, the Test Connection fails and blocks metric configuration.

Configure By AI Agent Metric

Step 1: Navigate to Metric Configuration

Navigate to Quality AI > Configure > Evaluation Forms > Evaluation Metrics.
Select + New Evaluation Metric.
From the Evaluation Metrics Measurement Type dropdown, select By AI Agent.

Step 2: Create the Parent Metric

Enter a descriptive Name (for example, Compliance Disclosure).
Select the Language for the AI Agent’s evaluation.

Step 3: Select the Agentic App

In the Agent App dropdown, choose from available apps in your workspace.
Select the Environment (for example, Draft, Version 1, Version 2).

Step 4: Test Connection and Fetch Sub-Metrics

Select Test Connection.
The system sends a test call to the selected app and retrieves available sub-metrics.
Retrieved sub-metrics appear under the parent metric with editable fields.

Step 5: Configure Sub-Metrics

After a successful connection, all sub-metrics from the agentic app appear with their reference names.

Select Edit next to the Weightage column to configure each sub-metric:

Field	Description
Display Name	Label for the sub-metric
Question	Evaluation question for this sub-metric
Positive Weightage	Weight assigned when the criterion is met
Negative Weightage	Weight deducted when the criterion is not met
Fatal Error	If enabled, failing this sub-metric marks the entire interaction as a critical failure

Select Create to save the sub-metric.

Use Case: UDAP Compliance

For financial services compliance, a single parent metric can evaluate multiple aspects in one API call:

Sub-Metric	Weight	What It Evaluates
Fee Disclosure	25%	All applicable fees are clearly explained
Interest Rate Accuracy	30%	Interest rate information is accurate
Benefit Explanation	20%	Benefits are clearly described
Exclusion Details	15%	All exclusions are clearly listed
Terms Clarity	10%	Overall clarity of terms

Evaluation Flow

At runtime:

Each parent metric triggers a single agentic call.
The agent analyzes the conversation and returns structured results for all sub-metrics.
Each sub-metric’s adherence, justification, and metadata are automatically extracted and displayed.

Required Response Format

The Agent Platform must return responses in this JSON format for Quality AI to process sub-metric results:

{
  "botId": "string",
  "accountId": "string",
  "conversationId": "string",
  "agentEvaluation": [
    {
      "PARENTMETRIC_ID_VALUE": {
        "subMetrics": [
          {
            "subMetricId": "string",
            "subMetricName": "string",
            "justification": "string",
            "messageIds": ["array"],
            "timestamps": ["array"],
            "source": "agent | customer",
            "isQualified": "YES | NO | NA",
            "failureReason": "string"
          }
        ]
      }
    }
  ]
}

Sample Response

{
  "botId": "bot_001",
  "accountId": "account_001",
  "conversationId": "conv_001",
  "agentEvaluation": [
    {
      "eval_001": {
        "subMetrics": [
          {
            "subMetricId": "sm_001",
            "subMetricName": "Loan Inquiry Identification",
            "justification": "Agent correctly identified the customer's loan-related query.",
            "messageIds": ["msg_001"],
            "timestamps": ["2025-10-17T10:00:00Z"],
            "source": "agent",
            "isQualified": "YES",
            "failureReason": ""
          },
          {
            "subMetricId": "sm_002",
            "subMetricName": "Loan Eligibility Explanation",
            "justification": "Agent provided loan eligibility information.",
            "messageIds": ["msg_002", "msg_004"],
            "timestamps": ["2025-10-17T10:00:10Z", "2025-10-17T10:00:35Z"],
            "source": "agent",
            "isQualified": "YES",
            "failureReason": ""
          }
        ]
      }
    }
  ]
}

Modules

Platform Services

References

When to Use This Metric

Prerequisites