Chatbot improvements for BMO Assist
Background: introducing BMO Assist
In 2023, BMO's DMC team launched BMO Assist, a chatbot designed to support existing customers with quick answers to their queries.
​
Challenge: finding a matching response
Our fallback message was the default response (i.e., "I didn't understand that") when the chatbot didn't understand what the customer was saying.
​
It became a source of frustration for users because it occurred too frequently—about two-thirds of the time!
This made the chatbot seem less helpful and likely made customers avoid using it.
We had set a rule that only responses with a 60% or higher chance of being correct would appear.
But this often meant we missed chances to help users get the correct answer.
A poor user experience. The chatbot couldn't match a response even if it were close.
Solution: present "close enough" responses
Initially, the product owner wanted to introduce slots to train our chatbot better. However, implementing it was lengthy and required a steep learning curve for the team. So, we went with the lowest-hanging fruit: improve the fallback experience.
As an alternative to the repetitive "Sorry, I don't get it" response, we introduced a new approach called 'fall-forward.' The chatbot now presents customers with a few potential responses that are close enough matches (even below the 60% confidence score).
.png)
​
So, instead of just saying, "Huh?" we suggest some answers. This could save time because maybe one of those suggestions is right.​
My role: designing UI-friendly CTAs for each response
-
Establish a UI-friendly content pattern for the fall-forward responses.
-
Developed 300+ fall-forward CTAs following the content pattern for consistency.
-
Create messaging for the fall-forward response.
-
Educate devs and QA on changes to the chatbot content with design documentation.
Team
-
Developers​ (x2)
-
QAs (x3)
-
Business architect
-
Product owner
-
Agile delivery manager
-
Conversation designer (me)
"Not sure if I got it right. Do you need information on any of these?"
Locking a card
Cancelling a payment
None of the above
Content pattern for UI-friendly CTAs
​
Each one of these CTAs map to a response (i.e., intent). Use one of these formats:
​
-
[Verb ending in 'ing'] + [Noun]
-
[Noun only] *only when the inquiry isn't about an action the user can perform (e.g., FAQ about bills)
-
[Question] *only if the other 2 patterns don’t work (e.g., Why am I receiving alerts?)
​
Max. character count: 48
Real examples
Challenges
-
Keeping the character count under 48 for French.
-
Manually having to write and add responses to the chatbot spreadsheet and Amazon Lex.
-
Despite best efforts to make the CTAs all parallel with a verb-first content pattern, it was only sometimes possible.
Outcome
Prior to the fall-forward implementation, the bot was able to find a response to a typed message about 45% of the time. After the fall-forward, we have almost doubled it at 86.5%!
13,150
Fall-forward responses invoked in the 1st week
90%
Increase in correct responses
43%
Responses contributed to total thumbs up