Interaction Log on Every Turn
Intent, confidence, sentiment, mode, sources, buttons and timings for every conversation — rated or not — stored on the platform's own database.
Chatbot Analytics for Employees: What They Ask Athena, What Gets Resolved, Where the Gaps Are
Every Athena conversation writes an interaction log — the intent, the confidence, the sources cited, the thumbs and the reason. Chatbot Analytics turns that log into the list of intents, documents and field help to fix next.
Most chatbot dashboards stop at volume and a thumbs-up percentage. Athena Insights reports helpful rate alongside how many answers were rated at all, classifies every thumbs-down into a root cause with an owning team, and opens a quality review in BR Resolve when a high-risk benefits answer is disputed — for employees signed in from SAP SuccessFactors or from Workday, UKG, ADP, Dayforce and Paycom through the same SSO.
Resolved in chatThumbs with reason codes5 root-cause bucketsQuality Events → BR ResolveEmployee-scoped · cohort-safe
A Bad Answer Becomes a Configuration Fix, Not a Rumor
Athena answers thousands of benefits questions a month without a person in the loop. The only way that stays safe is if every answer leaves a trail — which intent matched, how confident the match was, which document was cited, whether the employee pressed the button, and what they said when it was wrong. Chatbot Analytics reads that trail and turns it into three lists: the intents that need examples, the documents that are missing, and the answers a person must review before they are given again.
CaptureEvery turn writes an interaction logRateOne-click thumbs, reasons, recoveryClassifyFive root causes, five ownersRemediateFix at source, close the loopAny HRISSame analytics for every sign-in
5root-cause buckets for negative feedback — content, retrieval, reasoning, benefits data, user intent — each with an owning team
2numbers that always travel together: helpful rate and rated coverage, because voluntary ratings are self-selected
1Athena Quality Event created automatically when a high-risk benefits intent — eligibility, COBRA, HSA, ACA, a life-event window — gets a thumbs-down
3measured steps for every action button: shown, clicked, completed — so navigation is counted, not assumed
Working Flow — Eight Views and One Ask, in Order
The shipped chatbot analytics workspace inside BR Analytics, as it runs: eight views on aggregates, one period, one Ask box on the chatbot catalog. The last column is the rule the platform holds at that step.
| # | Who · where | What happens | The rule at this step |
|---|---|---|---|
| 1 | Admin opens BR Analytics Home tile → Athena chatbot analytics | The chatbot analytics workspace opens on A-1 Overview with last month as the period; the employee chatbot widget is hidden on these screens | The analytics bot and the employee bot are never on screen together |
| 2 | Sets the period and filters Workspace header | Period and filters apply to every view; aggregates are recomputed for that window | Every figure is an aggregate — no view lists a person |
| 3 | Reads A-1 Overview and A-2 Usage Views | Conversations, unique users, channels and trend for the period | Unique users are a distinct count of people on conversations; slices by domain, intent or channel are never added together |
| 4 | Reads A-3 Intents View | Top intents and how they resolved, by domain | The same conversation can carry more than one intent; counts are per slice and are not summed across slices |
| 5 | Reads A-4 Quality View | Satisfaction, thumbs-down rate, escalations | Satisfaction uses only the current feedback row per interaction; superseded ratings are history |
| 6 | Reads A-5 Knowledge and A-6 Performance Views | Knowledge gaps and unanswered topics; latency, tool health and failure rates | Message text, feedback text and tool payloads never appear — they are outside the analytics contract |
| 7 | Reads A-7 Predictions View | Forecast volume, intents and quality for the coming period | A forecast is an estimate stored as analytics output; it is labelled as such and never posts anything |
| 8 | Reads A-8 Recommendations View | Suggested intent, prompt, knowledge and tool changes with an estimated impact | A recommendation changes nothing by itself — no intent, prompt or tool is written from this view; the fix is made in BRConfig by a person |
| 9 | Asks the Ask box Chatbot analytics Ask | A question about the chatbot's health, answered in a fixed order — session, intent, playbook getters, allow-listed read — or refused | Aggregates only: the Ask must not return a person, an employee id, message text, feedback text or embeddings |
| 10 | Runs maintenance Backfill · recalculate | Rebuilds the analytics aggregates for a period | Maintenance touches analytics tables only — never the live chatbot, its configuration or a conversation |
One Thumbs-Down, Five Stages
Follow a single disputed answer — “when does my coverage end after I leave?” — from the interaction log through the reason code, the root cause, the remediation and the re-test, and see why an employee on a non-SAP entity gets the same loop.
Every turn writes a record — before anyone rates it.
When an employee asks Athena a question, the platform logs the intent it matched, the confidence of the match, which mode answered — an employee-scoped query, a policy document, an approved field-help text — the sources it cited, the buttons it showed and the timings. Clicks on those buttons are tracked silently through to completion. None of this depends on the employee pressing a thumb; the log exists for every conversation, and it is stored on the platform's own database alongside the benefit records it explains.
What the interaction log holds
- Intent, confidence and sentiment per message; the mode that answered
- Documents and FAQ entries cited — and questions that found no document at all
- Button shown → clicked → completed for every next-step action
- Model, provider, version and prompt template, for traceability
Why it matters
You can measure the chatbot on every answer, not only on the few employees choose to rate.
Identifiers never appear in an answer, sessions expire, and the employee can download or end their own conversation at any time.
Thumbs that sit under the answer — with a reason and a way out.
The controls belong to the specific answer — Sources, Copy, 👍, 👎 — never floating at the bottom of the chat. A thumbs-up is frictionless: a selected state and a short acknowledgment. A thumbs-down opens a small panel in place, not a modal: a reason — wrong plan, wrong benefit, incorrect, incomplete — then a recovery: try again, show the source, ask differently, contact Benefits, create a case. Ratings are reversible because accidental taps happen on a phone. A separate flag, “report potentially incorrect benefits information”, is not a dislike; it opens a quality review straight away.
What the employee sees
- Controls under the answer they are rating — never a floating bar
- 👍 selected state and a brief thank-you; “what worked?” sampled, not asked every time
- 👎 inline panel: reason code → recovery choice → optional comment
- A do-not-include-sensitive-information notice on every comment box
Why it matters
Feedback separated from its answer measures the conversation; feedback under the answer measures the answer.
Employee and administrator Athena use different reason taxonomies; admin reasons link to the configuration node, plan or rule being worked on.
Five root causes, five owners — sorted automatically.
A thumbs-down with a reason is an analytics row; a thumbs-down on a high-risk intent is an Athena Quality Event carrying the full context — conversation and message, role, tenant, intent, query, response, answer and retrieval confidence, model and prompt version, the documents and versions cited, the tables read, latency, the reason, the comment and the outcome. Each event is auto-classified into one of five buckets so the right team sees it: content that is wrong or missing, retrieval that picked the wrong source, reasoning that misread a right source, benefits data that was itself wrong, or a question the intent catalog does not yet understand.
The five buckets and who owns them
- A · Content — a document is missing, outdated or ambiguous → content owner
- B · Retrieval — the right document exists but was not chosen → configuration
- C · Reasoning — the source was right, the answer misread it → prompt and evaluation
- D · Benefits data — the record Athena read was wrong → HRIS Sync or the owning module
- E · User intent — the question is not in the catalog yet → intent examples
Why it matters
A cluster of complaints arrives as a cause with a confidence, not as forty transcripts to read.
Comments pass the platform's data-protection controls before an administrator can read them; a leaked SSN or claim number never reaches the console.
Fix at source, re-test, and watch the answer change.
Chatbot Analytics does not ask administrators to read conversations. It summarises a cluster with a likely root cause and a confidence, proposes the remediation — retire a document version, add three example phrasings to an intent, correct a field-help text, open a data case — and a named owner commits it in BRConfig or in the owning module. The event moves through BR Resolve as an Athena Quality Review; when the fix is live, the original question is re-asked and the new answer, sources and ranking are stored linked to the old one. Time from the first thumbs-down to that re-test is the KPI leadership sees.
What closes the loop
- AI proposes the fix; a named content or configuration owner commits it
- Athena Quality Review in BR Resolve with the event context attached
- Automatic re-test of the original question, linked to the disputed answer
- Time-to-remediate, root-cause mix and traceability as the governed KPIs
Why it matters
Raw thumbs-up percentage is not the quality KPI; how fast a wrong answer stops being given is.
Low-confidence clusters flag missing intent examples before anyone complains; unanswered policy topics flag the documents to upload next.
The same loop for every employee, whatever HRIS signed them in.
Athena serves an employee signed in from SAP SuccessFactors inside the SuccessFactors experience, and an employee signed in from Workday, UKG, ADP, Dayforce or Paycom through the same SSO and HRIS Sync. The interaction log, the feedback layers, the root-cause classification and the remediation queue are identical — and so is the privacy model: answers are employee-scoped, analytics are cohort-suppressed, and an entity or department under the minimum never appears as a row. Chatbot Analytics reports by entity, language and channel, so a group can see whether the Spanish-language mobile experience resolves as well as the English desktop one.
How the analytics are cut
- By entity and HRIS source — SuccessFactors, Workday, UKG, ADP, Dayforce, Paycom
- By language and channel — desktop, BR mobile app, embedded widget
- By department and location, with cohorts under the minimum suppressed
- By intent risk class — high-risk benefits intents reported separately
Why it matters
A quality problem that only affects one entity or one language is visible the week it starts.
Bucket D events — a wrong benefits record — route to HRIS Sync when the source is the HRIS, whichever HRIS it is.
Ground Rules of Chatbot Analytics — What the Platform Holds
Measuring an employee-facing assistant means reading a lot of sensitive conversation data. These are the rules that make the measurement safe.
| Rule | How it works | Why it is there |
|---|---|---|
| Aggregates only on the Ask path | The analytics Ask answers from aggregated chatbot KPIs; it must not return a person | An analyst can ask “what went wrong last week?” without ever seeing whose conversation it was |
| Columns outside the contract cannot be queried | The allow-listed read may not select an employee id, message text, feedback text, embeddings or tool-call payloads | Privacy is enforced in the query contract, not by asking the model nicely |
| Ask order, then fail | Session → intent routing → playbook getters (optional rewrite of the wording from those facts) → allow-listed text-to-SQL → fail; no KPI is invented | The model rewords computed facts; it does not compute them |
| Satisfaction from current feedback | Only the current feedback row per interaction counts; older rows are history | A changed rating replaces the earlier one instead of adding to it |
| Unique users are distinct people | Getters count distinct people on conversations; the allow-listed read uses the daily unique-users metric and never recomputes from raw chats | Two ways to count would give two numbers |
| No summing across slices | Total conversations and unique users are never added across domain, intent or channel slices | The same conversation sits in more than one slice; the sum double-counts |
| Forecasts and impact are estimates | Predictions and recommendation impact are labelled estimates and stored as analytics output | Nobody reads a projection as a promise |
| Recommendations do not tune the bot | A-8 writes no prompt, intent or tool change; a person makes the change in BRConfig and the loop re-measures it | Every change to what employees hear is a decision someone made |
| Three catalogs stay apart | This Ask box shares no prompt, table list or session with Enrollment Athena Admin or the employee Athena | A chatbot-health question cannot reach the enrollment population or an employee's private chat |
| Maintenance is not change | Backfill and recalculate rebuild aggregates only | Fixing a dashboard never changes the assistant |
Core Capabilities of Chatbot Analytics for Employees
Built for a benefits chatbot whose wrong answer has consequences — not a generic bot dashboard counting sessions.
Three-Layer Feedback
One-tap thumbs under the answer, an inline reason-and-recovery panel, and a quality layer that turns clusters into cases — plus a separate report-incorrect-information flag.
Root-Cause Classification
Every negative event sorted into content, retrieval, reasoning, benefits data or user intent, with a confidence and an owning team.
Athena Quality Events
High-risk benefits intents — eligibility, coverage termination, COBRA, HSA, ACA, life-event windows — open a review in BR Resolve with the full event context.
Gap Finders
Low-confidence clusters flag missing intent examples; unanswered policy topics flag missing documents; button funnels flag actions nobody completes.
Governed KPIs & Employee View
Helpful rate always with rated coverage; traceability, root-cause mix and time-to-remediate as the KPIs; employees see their own Insights and can download their transcript.
The Eight Views — A-1 to A-8
One workspace, eight views, one default period of last month. Each view reads aggregates the platform maintains; none of them lists a conversation.
A-1 · Overview
The period at a glance — conversations, unique users, resolution and satisfaction, each with its trend.
A-2 · Usage
Volume by day, channel and domain, with unique users as a distinct count on conversations.
A-3 · Intents
Top intents and how they resolved — answered in chat, handed to a case, abandoned.
A-4 · Quality
Satisfaction from the current feedback row per interaction, thumbs-down rate and reason codes, escalations.
A-5 · Knowledge
Knowledge gaps: topics asked and not answered, sources missing or stale, by domain.
A-6 · Performance
Latency, tool health and failure rates for the period — the plumbing behind the answers.
A-7 · Predictions
Forecast volume, intents and quality for the coming period, labelled as estimates.
A-8 · Recommendations
Suggested intent, prompt, knowledge and tool changes with an estimated impact — made by a person in BRConfig, never by this view.
A Generic Bot Dashboard vs. Athena Insights
Chatbot vendors publish deflection rates and thumbs-up percentages. A benefits assistant needs to know which answer was wrong, why, who owns it and how long it took to stop.
| Question | Generic chatbot dashboard | Athena Insights |
|---|---|---|
| How good is the bot? | A thumbs-up percentage from the few who rated. | Helpful rate with rated coverage — and resolved-in-chat for every conversation. |
| What went wrong? | A list of transcripts to read. | A cluster with a root cause and a confidence — content, retrieval, reasoning, data or intent. |
| Who fixes it? | Whoever reads the transcripts. | A named owner per bucket, with the fix proposed and committed in BRConfig or the owning module. |
| A wrong COBRA answer | One more row in the export. | A Quality Event in BR Resolve with the full event context, before the answer is given again. |
| Did the fix work? | Wait for the next survey. | An automatic re-test stored linked to the disputed answer; time-to-remediate reported. |
| The employee who complained | Never hears back. | Sees the corrected answer in their own Insights view, with their transcript downloadable. |
| Employees on another HRIS | A second bot, a second dashboard. | The same Athena, the same log, cut by entity, language and channel. |
Athena reporting on Athena
Ask the Quality Console What to Fix This Week
Administrators do not read conversations to find problems. They ask the Athena admin assistant on the Quality Console, and it answers from the interaction log, the event queue and the intent catalog — then drafts the remediation for a content owner to commit.
- “Which intents lost confidence this month?” — the clusters, the phrasings that missed, and the examples to add
- “What are employees asking that we have no document for?” — unanswered policy topics ranked by volume, with the upload drafted
- “Why did coverage_termination get fourteen thumbs-down?” — the root cause, the two document versions, the fix
- “Does the Spanish mobile experience resolve as well as English desktop?” — resolved-in-chat and helpful rate by language and channel, cohort-safe
- “Show me open Quality Events for the Workday entity” — the BR Resolve queue filtered to that entity
Why did coverage_termination pick up 14 thumbs-down this week?
Athena · from the event queueAll 14 are one cluster, classified as bucket B — retrieval — with confidence 0.87. The index holds two versions of the Termination of Coverage document; 11 of the 14 answers cited the 2024 version, which says coverage ends on the termination date. Version 3, effective January, says end of month. The employees' recovery choice was “show source” in 9 cases and “contact Benefits” in 5. Because this is a high-risk intent, a Quality Event is already open in BR Resolve.
Sources: Athena Quality Event AQE-0412 · Document index · Termination of Coverage v2, v3 · Intent catalog · coverage_termination
Open the eventCompare v2 and v3
Propose the fix and re-test it once it's committed.
AthenaProposed, not applied: retire Termination of Coverage v2 from the index and pin v3 as the citation for coverage_termination. The Benefits content owner is assigned in BR Resolve. When it is committed, the original 14 questions are re-asked automatically and the new answers are stored linked to the disputed ones; the employees will see the corrected answer in their own Insights.
Assign ownerSchedule re-test
Ask about any intent, document or event…
Why HR and Service Desk Leaders Choose Athena Insights
Every Answer Counts
The log exists for 100% of conversations; ratings add to it rather than being the only signal.
Causes, Not Transcripts
Negative feedback arrives as a cluster with a root cause, a confidence and an owning team.
High-Risk Means Reviewed
A disputed eligibility, COBRA, HSA or ACA answer opens a quality review before it is given again.
Measured by the Fix
Time-to-remediate and traceability are the KPIs, with helpful rate always shown beside rated coverage.
Private by Design
Employee-scoped answers, cohort-suppressed analytics, screened comments and downloadable transcripts.
One Assistant, Any HRIS
The same Athena, log and loop for employees signed in from SuccessFactors or any other HRIS.
Where the Signals Go — BRConfig, BR Resolve, Communication HUB
Chatbot Analytics is the read side of the Athena Quality & Learning Loop. BRConfig is where intents, examples, documents, buttons and field help are changed; BR Resolve is where Quality Events live as cases; Communication HUB is where a missing explanation becomes a job aid; HRIS Sync is where a wrong record gets fixed at source. The console points at each, and Athena drafts the change.
- BRConfig — intents, example phrasings, table hints, documents, actions and field help — the remediation lands here without a code deploy
- BR Resolve — Athena Quality Reviews for high-risk events, with the event context attached and the transcript carried into the case
- Communication HUB — a job aid or notice drafted when employees keep asking the same unanswered question
- HRIS Sync — bucket D cases where the benefits record Athena read came from the HRIS and was wrong
- Employee Insights — the employee's own view of their questions, ratings and corrected answers, with a downloadable transcript
Bring Your Employees' Top Ten Questions. We'll Show You the Loop.
Give us the ten questions your service desk answers most, and one you know Athena would get wrong today. We'll load your documents, ask the questions, press thumbs-down on the bad one, and show you the root cause, the proposed fix, the re-test and the employee's corrected view — on your data, in your tenant.
Helpful rate with rated coverage5 root-cause bucketsQuality Events → BR ResolveEmployee-scoped answersCohorts under 5 suppressedAny HRIS via SSO
Chatbot Analytics for Employees FAQs
The measurement layer of the Athena AI benefits assistant: what employees ask, how confidently each question matched an intent, which document or record answered it, whether the next-step button was used, how the answer was rated and why, and — for negative feedback — the root cause, the owner, the fix and the time it took. In Benefits Reimagined it is the read side of the Athena Quality & Learning Loop.
No. Voluntary ratings are self-selected and skew positive, so the helpful rate is always reported beside rated coverage — the share of answers that were rated at all — and resolved-in-chat is measured on every conversation. The governed KPIs are traceability, the root-cause mix and time-to-remediate.
A small panel opens under that answer asking for a reason — wrong plan, wrong benefit, incorrect, incomplete — and offering a recovery: try again, show the source, ask differently, contact Benefits or create a case. The rating is reversible. If the intent is high-risk — eligibility, coverage termination, COBRA, HSA, ACA or a life-event window — an Athena Quality Event is created automatically and routed to BR Resolve.
Automatically, into five root-cause buckets with different owning teams: A content (a document is missing or outdated), B retrieval (the right document was not chosen), C reasoning (a right source was misread), D benefits data (the record itself was wrong) and E user intent (the question is not yet in the catalog). Clusters are summarised with a likely cause and a confidence rather than as transcripts to read.
AI proposes the remediation; a named owner commits it — intents, examples, documents, buttons and field help in BRConfig, a wrong record through HRIS Sync or the owning module, a missing explanation as a Communication HUB job aid. The event is tracked as an Athena Quality Review in BR Resolve, and the original question is re-tested automatically once the fix is live.
Their own Insights view — the questions they asked, how they rated the answers and any corrected answer — and they can download their transcript or end a conversation at any time. Answers are drawn only from the employee's own records; no employee can see another's.
Answers never print identifiers, comment boxes carry a do-not-include-sensitive-information notice and pass data-protection screening before an administrator can read them, analytics are cut by entity, department, language and channel with any cohort under a configurable minimum suppressed, and the logs live on the platform's own database on SAP BTP or AWS.
Yes. Employees signed in from Workday, UKG, ADP, Dayforce or Paycom reach the same Athena through SSO and HRIS Sync, write the same interaction log and go through the same feedback and remediation loop. Analytics can be filtered by HRIS source and entity, and data-bucket cases route to HRIS Sync whichever system the record came from.
Eight, inside BR Analytics: A-1 Overview, A-2 Usage, A-3 Intents, A-4 Quality, A-5 Knowledge, A-6 Performance, A-7 Predictions and A-8 Recommendations, with last month as the default period. All of them read aggregates; none lists a conversation or a person.
No. The Ask answers from aggregated chatbot KPIs only and its allow-listed read may not select an employee id, message text, feedback text, embeddings or tool-call payloads. It shares no prompt, table list or session with the employee-facing Athena or with the enrollment Ask box. If a question cannot be answered from the contract it fails rather than guesses.
No. A-8 shows suggested intent, prompt, knowledge and tool changes with an estimated impact; the change itself is made by a person in BRConfig, and the workspace measures the result. Backfill and recalculate rebuild the analytics aggregates only — they never touch the live assistant.
Unique users are a distinct count of people on conversations, never a sum of daily rows or of domain, intent or channel slices — the same conversation sits in more than one slice, so a sum double-counts. Satisfaction uses only the current feedback row per interaction; a changed rating replaces the earlier one.
Measure the Answer. Fix the Cause. Tell the Employee.
Every conversation logged, every thumbs-down classified, every high-risk dispute reviewed, every fix re-tested — for employees signed in from SAP SuccessFactors and from every other HRIS.