Paidleave.mass.gov User Feedback

Overview

Page 2 of the Paidleave.mass.gov web analytics & user surveys dashboard contains user survey results from Formstack. The goal of this page is to show the feedback claimants and employers give after interacting with the paidleave.mass.gov website.

The Formstack surveys can be found at:

• www.mass.gov/forms/employer-feedback-about-paidleavemassgov

• www.mass.gov/forms/applicant-feedback-about-paidleavemassgov

And the feedback itself, in addition to Formstack, ends up in this Google Sheet.

Easy/hard responses per 1000 users

These visualizations are meant to give PFML stakeholders an at-a-glance sense of user satisfaction trends across paidleave.mass.gov. Unfortunately, calculating them in Data Studio is a little complicated. Let's unpack. Here's what we need to do.

• Create a blended data set in data studio

• Create calculated fields that sum up the number of "easy" and "hard" responses

• Separate claimants from employers (we don't want to divide by all users, just the ones who might've responded to the form for each segment).

• Create chart level calculated fields

User segments

To set up the claimants segment, import this template into your paidleave.mass.gov google analytics. It should look like this:

The full filter (obscured in the image) is paidleave.mass.gov(.)*(\/|%2F)employers(\/|%2F) .

The employers segment (here is the template) looks like this:

The regex is the same as the previous template. (This one includes paths with /employers/ ).

Calculated fields

Since the paidleave data can have any value of a 5-item Likert scale from "Very easy" to "Very hard," we need to do a little work to count "easy" and "hard" responses. We can add calculated fields to both the Employer and Claimant data sources that contain the following formula:

SUM(CASE WHEN 
Overall, how easy or hard did you find it to complete the steps you needed on paidleave.mass.gov?
in ('Easy', 'Very easy') THEN 1 ELSE 0 END) 

Note:

• Data Studio should offer to autofill "Overall, how easy..." for you

• We're collapsing "Easy" and "Very easy" into a single category. We do the same thing with "Hard" and "Very hard." When we're done, we have formulas for adding up the total number of "easy" and "hard" responses. You can see these formulae when editing each data source:

Resources -> Manage added data sources

View calculated fields

Create a data blend

We then blend the claimants and employers data with paidleave.mass.gov web analytics data. To see how this is configured: Resource -> Manage blended data -> choose "edit" for vw_employ_ga or vw_claim_ga (employers and claimants respectively).

• More about blending data

Create chart-specific calculated fields

The charts bring all of this together. They have claimant and employer GA segments re

• 1000 * (sum__easy / sum(users))

• 1000 * (sum__hard) / sum(users))

...where sum__easy and sum__hard are the calculated fields we added 2 steps above

User survey page filters

You can filter by:

• Date (default is the last 14 days excluding the current day)

• Browser. Note that the "browser" filter only applies to the Google Analytics data, not to the feedback data. We do have user_agent in the Formstack data, but we haven't transformed it to appear here. Even if we did, it'd be pretty tough to get it to match up with the GA data.

Metrics and context

Employer & Claimant feedback responses

The doughnut and stacked bar charts show the distributions of answers to the survey question, "Overall, how easy or hard did you find it to complete the steps you needed on paidleave.mass.gov?" While the Formstack forms ask additional questions, this one most clearly summarizes the user's reaction.

There are separate visualizations for the Employer and Claimant surveys, as paidleave.mass.gov directs different segments to different forms. Users can respond Very easy, easy, neither easy nor hard, hard, very hard.

The pie chart visualizes the total amount and type of feedback, while the bar chart breaks it out by day.

You can also use the "user response" filter to limit what data appears in both charts.

Sessions sources for feedback

The table shows the source and medium of traffic visiting the 2 surveys, who provided the feedback (employer or claimant), and the number of unique pageviews these pages have received per traffic source.

Previous pages for feedback pages

This table shows the previous page path of the users who give feedback, the type of feedback collected (employer or claimant), and the number of unique pageviews for each previous page, broken out by the previous page path.

Employer and Claimant raw feedback

These tables show the raw answers to the survey question, "Tell us more about what made this easy or hard. What worked well for you? What could be better or clearer?" The tables are sorted by the most recent date the feedback was received.

Employer and Claimant feedback score

These line charts show scores that summarize response types for both Employer and Claimant feedback.

• Responses of "easy" or "very easy" receive a score of 1

• A response of "neither easy nor hard" receives a score of 0

• Responses of "hard" or "very hard" receive a score of -1

One of the line charts shows an average daily score, and the other a sum. The difference is that the latter is additive, while the former will always be between 0 and 1. The purpose of these scores is to provide a quick overview of whether feedback is generally positive or negative.