COVID-19 Chatbot dashboard

Web analytics and feedback

Search…

Web analytics and feedback

Mass.gov analytics

Web analytics

Mass.gov events reporting

Installing Google Tag Manager

Cross-domain analytics

Best practices

Digital analytics products

Feedback

How to implement a new feedback form using Formstack

Archive

Click tracking short URL's for SMS text messages

Digital analytics team release process

Feedback API & UI: A collection of thoughts

COVID-19 Chatbot dashboard

Last updated: May 10, 2021

Overview
‌‌This dashboard shows data about how people interact with the COVID-19 vaccine chatbot on Mass.gov, including the open and usage rate, whether users find the chatbot useful, the number of questions or messages users have, and link clicks. It aims to help us answer questions about if people who use the chatbot find what they need more easily, who the users of the chatbot are, and how they're interacting with it.
‌The chatbot launched 03/24/2021 12:00 pm (EDT), and the final version of the chatbot dashboard released on 5/4/2021.
The data on this dashboard comes from Google Analytics and a PostgreSQL database hosted in xFact's AWS account. Some data are derived from blends of Google Analytics data.
Dashboard filters
‌On the top of this dashboard, you will see filters that let you change:
The time range that graphs and tables are set to (default time range is set to 03/22/2021 to today)
The page(s) that graphs and tables are set to (default page is all pages with the chatbot)
A filter is used to include pages with the chatbot label:include "label" contains "covid19vaccinechatbot"
Page‌ 1: Vaccine Chat Clicks 
Did visitors click on the chat icon?
These scorecards show the number of times people click to open the chatbot and the click rate. The filter for the number of total opens is event action = chatbot: Open , and this filter is used in all other metrics that measure or slice opens. The formula for the Open rate is Total Opens/Total Unique Pageviews. 
Note that Click Rate is calculated using a blend of Google Analytics data.​
What is the age group of users who clicked on the chat icon?
This bar chart shows the number of users who clicked on the chat icon in each age group. The chart’s filter isevent action = chatbot: Open.
What devices did users use?
The pie chart shows the percentage of users who clicked on the chat icon in each device category. The chart’s filter includes event action = chatbot: Open.
Did users click on pre-populated questions?
‌These scorecards show the number of times people click on the pre-populated questions that appear when you open the chatbot, and the rate at which they interact with those questions. The filter for the scorecard Total Clicks isevent action = Element visibility AND event label = Ask another question. A formula calculates interaction rate: Asked a question/Total Users. Again, the interaction rate is based on a data blend.
What links did users click?
This table shows what links people click and how many times they click on them. The table’s filter includes event action contains link clicks AND event target contains chatbot.
Did users find the chatbot useful?
The dashboard’s last section shows the nos per 1,000 opens and yeses per 1,000 opens. The metrics measure yes/no feedback per 1,000 people who open the chatbot. 
The formula to calculate them is the total number of nos (or yeses)  / the total number of opens * 1,000.
The filter for yes answers is to include event action=Element visibility AND event label=Interaction was helpful and for no answer is to include event action=Element visibility AND event label=Interaction was not helpful .

Note about data blending
Data blending is needed to calculate the open rate, the interaction rate, and the yes/nos per 1,000 opens. There is “+ blend data” link underneath the data source on the rightmost of the data studio in the Edit mode (on the “Data” tab). 
Click on “Add another data source”, and choose the corresponding dimension, metric, and filter, and then hit SAVE. After blending the data, enter the formula in the metric field, and we should be able to calculate the rate.

Page‌ 2: Traffic by time and referrals
Users and Pageviews for Vaccine Chat pages by time
This table at the top left shows the trend of users and pageviews to Vaccine Chat pages by time. The 2 scorecards right next to it show the total number of site users and the total number of pageviews that pages with the chatbot receive. 
A filter of label contains covid19vaccinechatbot is applied on the table and the scorecards.
Users and Pageviews for Vaccine Chat pages by date
This table at the top right shows the trend of users and pageviews to Vaccine Chat pages by date.
Previous Mass.gov pages| “(entrance)” means they began on Vaccine Chat pages 
This page and the previous page path table at the bottom left shows user paths through pages and the number of pageviews.  
A filter of label contains covid19vaccinechatbot is applied.
Top referral traffic to pages with Vaccine Chat
This table shows what websites bring people to the vaccine chat pages and the number of unique pageviews.

A filter of label contains covid19vaccinechatbot is applied.
Page‌ 3: Vaccine Chat Interactions
All the tables and scorecards on this page are connected to a Google Sheet that contains data from the xFact database. This Google Sheet is updated automatically. There are 2 scorecards above each table and they show the total number and the daily number on the current date which will be updated at 5:01 pm every day.
Number of times a user had at least 2 interactions
This table shows the trend of the number of times a user had at least 2 interactions with the chatbot, which include the greeting and 1 or more interactions. 
Average number of user interactions with the chatbot
This chart shows the trend of the average number of user interactions with the chatbot. 
Number of times a user completed an interaction
This chart shows the trend of the number of times a user interacted with the chatbot by clicking on feedback or the “bye” responses. 
Number of times a visitor asked 2 or more vaccine questions
This table shows the trend of the number of times a visitor asked at least 2 or more distinct questions. 
Page‌ 4: Vaccine Chat Messages
The data source on this page is a Google Sheet which contains data pulled from the xFact's database. There are 2 scorecards above each table and they either show the total number and the daily number on the current date or show the weighted average percentage and the percentage on the current which will be updated at 5:01 pm every day.
Total number of messages
This table shows the trend of the total number of interactions with the chatbot including greetings, button clicks, questions. 
Clicks on chat icon or person typed in a greeting like “hello”
This table shows the trend of the total number of clicks on the chat icon or a user typed in a greeting message such as “hello.” Clicks on the icon will only be counted when the greeting message is populated. If a user minimizes the chat icon and reopens it, the click will not be counted. 
Percentage of questions out of scope
This table shows the trend of the weighted average percentage of people asking questions that are not in the library of chatbot questions, i.e. that it isn't prepared to answer.
Percentage of messages in scope
This table shows the rolling weighted average percent of people asking in-scope questions, i.e. what percent of people ask a question that the chatbot "knows" how to answer.
Page 5: Top Vaccine Chat Intents
The data source for this page is a Google Sheet that contains data pulled from xFact's database. This page shows all the pre-populated response categories that the vaccine chatbot gives to its users. The pie chart on the left shows the top 10 responses by percentage. The table on the right shows all the responses by count and percentage of the total count. 
Page 6: Top Vaccine Chat Visitor Input
The page is directly connected to xFact's PostgreSQL database. It shows all the messages that the vaccine chat visitors have for the chatbot, including the pre-populated buttons or typing manually. /greet corresponds to a visitor opening the chat window, or re-opening after closing with the "X" icon. It's triggered as soon as users open the window, and no manual typing is involved. /negative, /affirmative, /vaccine_location, /eligibility, /vaccine_documents, and /pre-registration are pre-populated buttons. The remaining messages are typed by the visitors manually.
The pie chart on the left shows the top 10 vaccine chat input by percentage. The table on the right shows all the input by count and percentage of the total count. 
A filter of type_name = user is applied to the pie chart and the table. This filter is to include the messages or questions from the vaccine chat users only, which means the responses from the chatbot are excluded. 

Note on how to connect to PostgreSQL on data studio:
1. Choose PostgreSQL as the data source
2. Enter the required information for Database Authentication 
3. Choose "chatbot_events" table
4. Open access to certain IP addresses so that Data Studio can access the database. (one-time) For details: https://support.google.com/datastudio/answer/7288010?hl=en
   

Page 7: Chatbot User Intent Clustering
The data source for this page is a Google Sheet using all the data from the tab "Cluster", which is pulled from xFact's database. This page shows the cluster analysis of chatbot user intents. The analysis is done weekly and so it only makes sense in a weekly range.
On the top of this page, you can filter the data by a weekly range selector. On the right-hand side, a scatter chart is used to show the clustering. Each point on the chart represents a question or message that users enter into the chatbot. The more similar messages/words/phrases, the closer they're plotted together. Cluster and Message are the dimensions, and "x" &"y" are the metrics on the chart, and the chart is sort by "y". Next to the chart, there's a fixed-sized list for us to select the user intent and only one intent is allowed each time. Underneath the chart and the list, there's a table showing all the messages and which cluster the message belongs to.
Page 8: Chatbot Intents Trigram
The data source for this page is a Google Sheet using all the data from the tab "Trigrams", which is pulled from xFact's database. This page shows the trigram of user intents. The trigram shows three-word combos that represent the most frequent word between the messages. The word combination is created based on algorithms which searches all the messages under a single topic within a date range and groups the top three combination of words and then represent them as a count. This helps us look at the messages without looking into each message individually.
​
Page 9: Vaccine Chat Visitor Input and Predicted Intents
The page is directly connected to xFact's PostgreSQL database. The pie chart on the top shows the top 10 predicted intents that the bot returns for the visitor input by percentage. The visitor input filter underneath the date range selector allows us to search for specific visit input and see what predicted intents the bot returns.
The table under the pie chart shows all the exact messages under each predicted intent with the total number of count and the percentage of count.
A filter of type_name = user is applied to the pie chart and the table. This filter is to include the messages or questions from the vaccine chat users only, which means the responses from the chatbot are excluded. 

Note on how to connect to PostgreSQL on data studio:
1. Choose PostgreSQL as the data source
2. Enter the required information for Database Authentication 
3. Choose "chatbot_events" table
4. Open access to certain IP addresses so that Data Studio can access the database. (one-time) For details: https://support.google.com/datastudio/answer/7288010?hl=en​
​


 



​
​
​
​
​