Skip to main content
Back to NLP Topics Utterance Testing lets you enter user inputs and see how the NLP engine processes and matches them. Use it to validate intent identification and train the app directly from test results.

​
How to Test

  1. Open the app and go to Testing > Utterance Testing.
  2. (Optional) For a multiple-intent model, select the Intent Model to scope ML detection.
  3. In the Type a user utterance field, enter an utterance. Example: Book a flight.
  4. Review the result: single match, multiple matches, or no match.

​
Match Types

Match TypeDescriptionBehavior in Production
Definitive MatchHigh-confidence exact match.Single: task executes directly. Multiple: user is prompted to choose.
Probable MatchReasonably scored but not definitive.Sent as “Did you mean?” suggestions to the user.
Possible outcomes:
  • Single match: Correct — proceed or train further. Incorrect — mark as incorrect and select the right intent.
  • Multiple matches: Select the correct radio button and train.
  • Unidentified intent: Select an intent from the dropdown and add training.

​
Entity Match

Matched entities are displayed after intent detection. Processing order: NER and pattern entities first, then remaining entities. Entity match details (v8.0+):
FieldDescription
Identification EngineML, FM, or KG
Training TypeNER, pattern, entity name, system concept, etc. Click for pattern match details.
Confidence ScoreML NER score (only when Conditional Random Field is the NER model).

​
NLP Analysis

Each test result includes an NLP Analysis box showing shortlisted intents, which model found them, scores, and the winning intent.
NLP scores are absolute values. They can only be compared against other tasks for the same input, not across different utterances.

​
ML Model

Matches the input against task labels and training utterances. Multi-sentence inputs are tested sentence by sentence. More training utterances increase discovery chances.

​
FM Model

Scores every task using a custom NLP algorithm based on task names, synonyms, and patterns. Click Processed Utterance to see how the input was analyzed. FM scoring factors:
FactorDescription
Words MatchedScore for the number of words matching the task name or utterances.
Word CoverageRatio of matched words to total task words (name, fields, utterances, synonyms).
Exact WordsWords matched without synonyms.
Sentence StructureBonus for sentence structure match.
Word PositionScore for words near the start of the sentence.
Order BonusBonus for words in the same order as the task label.
Role BonusBonus for matched primary/secondary roles (subject/verb/object).
Spread BonusBonus based on distance between first and last matched words.
PenaltyPenalty for phrases before the task name or conjunctions in the task label.
FM scoring varies by language:
  • German/French: Uses word roles, Universal Parts of Speech, and dependency relations.
  • Other languages: Uses original word, role in sentence, and processed word (spell-corrected).

​
Knowledge Graph

Extracts terms from the utterance and maps them to the KG to fetch relevant paths. Paths covering more than a threshold number of terms are shortlisted. A path with 100% term coverage and a matching FAQ is a perfect match. See Knowledge Graph Training.

​
Ranking and Resolver

Determines the final winning intent across all engines.
  • If ML or KG finds a perfect match, R&R presents it without rescoring (even multiple perfect matches are shown as options).
  • All other good/unsure matches are rescored using the FM model. If the rescored intent exceeds the threshold, it is also considered a match.
R&R V1 intent elimination reasons:
  • Intents matched only by entity values (date, number) from ML are eliminated.
  • Possible matches are eliminated if a definitive match exists.
  • A definitive match is eliminated if another definitive match appeared earlier in the utterance (multi-intent case).
  • Intent patterns following a definitive match are eliminated.
  • Intents below the minimum threshold are eliminated.
  • Definitive matches against a negative pattern are eliminated.
  • Intents with unmet pre-conditions are eliminated.
  • KG “Search In Answer” definitive matches are eliminated if another match exists.
R&R V2 ranks only definitive matches from ML and KG (not FM) without rescoring. The R&R window shows the winning intent, ranking score, eliminated intents and their reasons, and the final result. See Ranking and Resolver.

​
Training from Utterance Testing

​
Utterance-Based Training

  1. After testing, open training options:
    • Unmatched intent: Select an intent from the Select an Intent dropdown.
    • Multiple matches: Select the radio button for the correct intent.
    • Single match: Click the matched intent name.
  2. The utterance appears in the ML Utterances field. Click Save to add it to the intent.
  3. Under Intent Synonyms, add synonyms for each word in the task name.
  4. Under Intent Patterns, add task patterns.
  5. Click Re-Run Utterance to verify improved scores.

​
Matched Training Data

The Ranking and Resolver NLP Analysis window shows the training data (utterances, synonyms, questions, patterns, traits) that led to intent qualification.
NLP EngineSupported ModelsMatched Training Data Shown
Machine LearningStandard, Few-shotSample utterance for every qualified intent.
Knowledge GraphOntology, Few-shotQuestions/alternate questions of qualified FAQs.
Fundamental MeaningAll modelsPatterns for qualified intents.
For qualified and eliminated intents, the system also shows the elimination reason alongside the matched utterance, processed utterance, and winning intent.

​
Train with FAQ

To match utterances to FAQs:
  • Set terms, term configuration, or classes from the FAQ page, train the KG, and retest.
  • Add the utterance as an alternate question to the FAQ from the Knowledge Graph page, train, and retest.
See Knowledge Graph Training.

​
Mark an Incorrect Match

When a user input matches the wrong task, mark it as incorrect and select the right intent to retrain.