What is Evaluator?

Evaluator has its own item in the NavBar.

Main topics:

  • Admins can open it when the conversation evaluator is enabled for the organization.
  • ‍Exclusively evaluates bot conversations with AI Agents. So, human agent interactions are excluded from scoring.
  • Evaluator Plans:
    • Free Plan evaluates a limited sample of conversations.
    • Premium Plan evaluates up to the dataset size configured for the organization.‍
  • Daily results:
    • You can check the latest update or see "Running" while a run is in progress.

How scores work

Results use a scale from 1 to 5. Achieving an average of 3 or more constitutes a passing grade.

The KPI's Overview includes:

  • Conversations Evaluated: total number of conversations analyzed during the last 24 hours.
  • Avg. Knowledge Score: how accurate and complete the factual answers are.
  • Avg. Service Score: how effectively the bot fulfilled the user request.
  • Avg. QA Score: overall average between Service Score and Knowledge Score.
  • Pass Rate: percentage of conversations that passed quality evaluation.

Sections

  • Overview: the KPIs for the period you select with Filter overview by date. You can also use All time, All bots, and Filter by date.
  • Contact Reasons Overview: the leading contact reasons for that view.
  • Last 30 Days: Contact Reason Analysis and Scores Over Time.
  • Conversations: every evaluated conversation in the current filters.

Review a conversation

Filters: General score (Passed or Not Passed), Has handoff (With handoff or Without handoff), Knowledge score, Service score, Contact reason, Channel, Provider Account, Bot, Country, and Language.

Table columns: Evaluated at, Conversation ID, Has handoff, User mood, Knowledge score, Service score, AI agent issues, and Contact reason.

  • User mood is the evolution of the user mood from the beginning to the end of the conversation.
  • AI agent issues exclusively shows guardrails info.

Open a row to read the conversation. The chat labels a passing result Passed (score/5) and a failing result Failed score/5.

Expand a row for the reasoning behind the mood and the scores.

Download CSV exports the conversations that match the current filters.

If nothing matches, the page says There is no data to display.

Was this resource helpful?
Yes 👍
No 👎
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.