How to measure the quality of conversations with AI beyond simple sentiment?

Most of us know this scenario. Contact with a helpline or a bot, a seemingly simple issue, and a conversation that drags on endlessly. Questions unrelated to the problem, suggestions already tried, until the sole purpose of the conversation becomes „connect me to a human”.

I recently experienced this first-hand with the bot of one of the Arab airlines. A simple matter, a long conversation, a series of loops and unhelpful answers, ultimately resolved by a consultant in a few minutes. Notably: sentiment analysis of my messages would likely have shown a neutral tone. The communication remained polite. The frustration, however, grew with every turn.

This is the core measurement problem. Standard metrics for evaluating AI conversations, response time, deflection rate, per-message sentiment, measure individual frames. Frustration, however, plays out across the entire movie: in repeating the same intent for the third time, in suggestions of steps already taken, in a conversation that has stopped making progress, even though no single message contains anything „negative”. The customer service literature has a name for the most damaging variant of this phenomenon: double deviation – a failed repair attempt causes more harm than the original error itself.

For our conversational tool, we built a working system whose task is to monitor precisely these phenomena. Instead of asking „is this message negative?”, the system analyzes the trajectory of the entire session: is the conversation making progress, has it entered a loop, did an error lead to repair or deterioration, is the final state better than the initial one and at which point the trajectory broke. The evaluation is performed by a language model acting as a judge (LLM-as-a-Judge), operating on fixed rubrics, with a hard requirement to point to the transcript fragment justifying each verdict. The system does not interpret the user’s „true emotions” – it registers only what is observable in the text.

The architecture runs in production and has processed 1,146 sessions across five industries to date. One observation illustrates the scale of the problem well: of the 119 sessions flagged by the system as priority for intervention, not a single one ended with the user’s issue fully resolved, even though a significant portion looked like „correct” conversations that a classical report would pass without objection.

The solution has prototype status: the presented results are preliminary observations from a pilot, and formal validation independent annotation, inter-rater agreement, correlation with business indicators is in progress.

A full description of the solution with architecture, risk taxonomy, output data schema, and validation plan you can find in the whitepaper below.