2021 analytics release notes

July 1, 2021
Release Google Search Console data to CMS dashboards. (DP-18333)
Add internal documentation to ETL repo. (DP-22092, DP-22110)
May platform activity report. (DP-22222, DP-22388)
Implement cross-domain tracking between Mass.gov & vaxmillions.com. (DP-22360, DP-22364, DP-22365, DP-22366)
Feedback manager export now includes an organization column. (DP-20386)
June 24, 2021
Clean up repo post bigquery refactor. (DP-21998)
Add internal search data (from search.mass.gov) to CMS dashboards. (DP-22161, DP-22162)
Offboard Geoff. (DP-22304)
Report to Joe on analytics question authors have had during the past few months. (DP-22306)
June 17, 2021
Remove SSNs from Formstack. (DP-22238)
Hand off cross-domain data to DCR/EEA. (DP-22219)
Fixed bug that prevented you from exporting CSV data in the feedback manager (DP-22225)
Vaccine preregistration cleanup work/ (DP-22244, DP-22236)
June 10, 2021
Begin filtering additional social security formats out of formstack data. (DP-22157)
Add additional summary tables for archiving web analytics data. (DP-12389, 	DP-21902, DP-21992)
Combine all BigQuery API calls into one job. (DP-21726)
Add partitioning to analytics.pageviews. (DP-21718)
Release platform activity report. (DP-21985)
Documenting DCR xdomain analytics work. (DP-22073)
Update other tag manager containers for DCR xdomain pageview. (DP-22136)
Drop GA tables. (DP-22138)
Add validation errors tags for DCR xdomain tracking. (DP-22156)
Chatbot discovery. (DP-22185)
DCR xdomain QA. (DP-22205, DP-22207)
Document how to upgrade Superset. (DP-22091)
Add documentation for using Airflow. (DP-22093)
June 3, 2021
Make it possible to query BQ and XD data using Athena. (DP-21629)
Investigate if we filter out phone numbers in Formstack. (DP-21795)
Review massdcrcamping website. (DP-22069)
Create tags and triggers for DCR camping site. (DP-22071)
Create task path variable. (DP-22072)
Set up GA view for DCR. (DP-22074)
Set up Glue definitions in Terraform. (DP-22154)
Add link click coverage that doesn’t collect personal data to DCR camping tags. (DP-22178)
Merge dependabot update for Flask-Appbuilder. (DP-22202)
Plan data structure for search.mass.gov data. (DP-22106)
Explore search.mass.gov data. (DP-22108)
May 27, 2021
Filter query parameters out of Google Analytics “All Public Traffic” view. (DP-21224)
Add partitioning to analytics.entrance_groups. (DP-21743)
Fix bug that caused the “author” and “organization” fields to clear when you submitted a Feedback Manager query. (DP-21788)
Create basic variables for reserveamerica cross-domain tracking. (DP-22070)
Conduct informal interviews to better understand use case for search.mass.gov data. (DP-22107)
May 21, 2021
Filter query parameters out of Google Analytics “All Public Traffic” view. (DP-21224)
Add partitioning to analytics.entrance_groups. (DP-21743)
Fix bug that caused the “author” and “organization” fields to clear when you submitted a Feedback Manager query. (DP-21788)
Create basic variables for reserveamerica cross-domain tracking. (DP-22070)
Conduct informal interviews to better understand use case for search.mass.gov data. (DP-22107)
May 20, 2021
Modify API options to allow for search terms (DP-21656)
Run analytics schema backfills (DP-21381)
Add partitioning to previous page summary (DP-21741)
Add partitioning to next page summary (DP-21742)
Onboarding for Geoff St. Pierre (DP-21781)
Update ETL repo changelog (DP-21997)
Add documentation for Formstack DAG (DP-21413)
May 13, 2021
Work toward simplifying ETL processing. (DP-21701, DP-21747)
Lock down Superset permissions. (DP-22003)
Make sure develop environment database has all necessary data. (DP-21593)
Release March platform activity report. (DP-21605)
Update chatbot dashboard. (DP-21715)
Add partitioning to analytics.page_events. (DP-21717, DP-21971)
Automate creating a JIRA ticket when a job fails. (DP-21776, DP-21987)
Create eligibility form data set. (DP-21860)
Add vaccine options to vaccine locations map. (DP-21893)
Investigate and fix Drupal ETL failure. (DP-21959)
May 6, 2021
Work toward making it possible to search for labels in the feedback manager (DP-21649, DP-21650, DP-21805)
 Adjust structure of data in S3 (DP-21714, DP-21787)
Add documentation for partitioning (DP-21738)
Fixed a bug in the Feedback Manager where the organization and author filters didn’t work if you filtered for more than 1 organization or author. (DP-21762)
Understand how alerts work in Airflow. (DP-21775)
Back Formstack cleanup date up to 5 weeks. (DP-21794)
April 29, 2021
Handoff PFML analytics materials. (DP-21412)
Adjust formatting of how labels come over from Drupal API. (DP-21662)
Eliminate redundant scripts from ETL processing. (DP-21699, DP-21700, DP-21708)
April 22, 2021
Propose new, slimmed-down architecture for ETL processing. (DP-21424)
Eliminate bq_ and xd_ page_trace. (DP-21668)
Update chatbot documentation. (DP-21693)
April 15, 2021
Report on traffic to children metric. (DP-20964)
Create February platform activity report. (DP-21128)
Explore partitioning bq_events. (DP-21429)
Test regex performance to see if we can offer a regex search for feedback manager. (DP-21654)
April 8, 2021
Fix bug where wrong organizations were being associated with some pages in the reporting schema. (DP-21240)
Various items for tagging vaccinesignup.mass.gov. (DP-21405)
Investigate and improve indexes throughout databases. (DP-21448)
Convert pageviews and scores view into materialized view so that Drupal API calls won’t take down reporting db. (DP-21563)
Drop bq_ and xd_ node. (DP-21579)
Automate Color and Curative codes dropoff. (DP-21590)
Cleanup after warehouse refactor. (DP-21601)
Update backpublish.py. (DP-21612)
April 1, 2021
Automate Twilio logs ingestion. (DP-21581)
Automate sending codes to Color and Curative. (DP-21469)
Add a date column to bq_node. (DP-21428)
Create documentation for chatbot dashboard. (DP-21399)
Create chatbot analytics dashboard. (DP-21296)
Use BETWEEN instead of DATE_TRUNC. (DP-20996)
Optimize ETL processing. (DP-20787, DP-20782, DP-20781, DP-20780)
March 25, 2021
 Interview MassHealth authors about service family dashboard. (DP-21012)
Onboard Noah P. (DP-21426)
Fix backpublish script bug. (DP-21536)
March 18, 2021
Optimize several warehouse queries. (DP-20784, DP-20786)
Upgrade Postgres to latest stable version. (DP-20982, DP-21427)
Address RDS instances under heavy load. (DP-21280)
Add vaccinesignup.mass.gov to warehouse. (DP-21397)
Update vaccine dashboard. (DP-21430)
Add vaccine feedback form to ETL. (DP-21437)
Generate CSVs for contact lists. (/DP-21441)
March 11, 2021
Optimize several warehouse queries. (DP-20778, DP-20788)
Fix bug that caused nos per 1000 contributions on the service family dashboard to add up to more than 100% (DP-20931)
Document Formstack data flow (DP-21189)
Add items to the vaxfinder & vaccine content dashboard (DP-21211)
Investigate mysterious link clicks in vaccinesignup.mass.gov analytics. (DP-21270)
Update nos per 1000 on DUA dashboard. (DP-21311)
Add vaxfinder.mass.gov to cross-domain ETL. (DP-21312)
Update GTM and ETL for paidleave.mass.gov path changes. (DP-21323)
Use new terraform version for ETL databases. (DP-21329)
March 4, 2021
Move paidleave.mass.gov query params to a separate dimension. (DP-21049)
Factor fact_site_improve out of existence. (DP-20777)
Change date formatting on Superset charts for clarity. (DP-20925)
Create paidleave feedback scoring metric. (DP-20987)
Add Drupal tables to data dictionary. (DP-21217)
Formstack cleanup failing (429 error). (DP-21235)
Get travel form automated job running for yesterday. (DP-21267)
Get UI data for Karthik. (DP-21287)
Duplicate chatbot tags for cross-domain property. (DP-21297)
Webserver container hasn’t upgraded. (DP-21313)
To-dos for attestation data privacy. (DP-21314)
Make bq_node performant again. (DP-21316) 
Add tracking for new promo page buttons. (DP-21340)
Dev reporting DB out of space. (DP-21351)
Feb. 25, 2021
 Make development environment useful again. (DP-20770)
Prep for ESC meeting on Feb. 17. (DP-21117)
Add vaxfinder formstack form to ETL. (DP-21177)
Stop running fact_exits in warehouse schema. (DP-20774)
Stop running dim_source in warehouse schema. (DP-20775)
Upgrade Airflow to newest stable version. (DP-20981)
Refine COVID-19 vaccine promo page dashboard. (DP-21146)
Add chatbot tracking. (DP-21170)
QA data coming out of refactored warehouse. (DP-21181)
Siteimprove data is not accurate on service family dashboards. (DP-21195)
Update monthly_dev_cleanup script for new tables. (DP-21268)
Feb. 18, 2021
Warehouse.dim_source load failing because of junk data in ETL. (DP-21206)
Verify that query parameters are correctly parsed when aggregating paidleave.mass.gov sessions. (DP-21051)
Add data from attestation form to postgres and s3. (DP-21081)
Onboard Jane Lee. (DP-21110)
Make vaccine feedback available for xFact. (DP-21150)
Tag vaxfinder.mass.gov. (DP-21176)
Division by zero error in reporting ETL. (DP-21180)
Various improvements for production ETL environment. (DP-21182)
Remove misleading columns in analytics.daily_feedback_count. (DP-17283)
Feb. 11, 2021
Take fact_page and fact_daily_feedback offline. (DP-20767, DP-20776) 
Report on our options for using AWS more efficiently. (DP-20863)
Create new creds for HED licensing personnel. (DP-20907)
Create January Platform Activity Report. (DP-20898, DP-20963)
Optimize use of dev environment Superset databases. (DP-20985)
Research and talk to DTA about service family data. (DP-21011)
Create reports for vaccine landing page. (DP-21065)
Document COVID-19 vaccine dashboard. (DP-21092)
Update vaccine query to include more pages (DP-21102)
Feb. 4, 2021
Analysis for ESC meeting presentation. (DP-21014)
Catchup DAG runs for January. (DP-21062)
Write script to generate First Responders map. (DP-20797)
Platform comms (DP-20921, DP-20920)
Generate accessible Excel file as alternative to map. (DP-21015)
Shutdown unused RDS instance. (DP-21019)
Investigate QA check error. (DP-21052)
“Refresh materialized views”  task failed on 1/27 (DP-21053)
Tag Caspio interactions. (DP-21066)
Monthly_kpi_components table doesn’t have Feb. data prior to SiteImprove DAG running. (DP-21105)
Jan. 28, 2021
Refactor temp_analytics schema to skip fact_page_events. (DP-20766)
Top-trafficked service family research. (DP-20864)
Analytics ETL is failing dev environment. (DP-20999)
Add sensors for cross-domain data and siteimprove data. (DP-20452, DP-20806)
Update DUA feedback for DUA dashboard. (DP-20883)
QA rich text link tags post Drupal release. (DP-20897)
Upgrade Terraform to latest version. (DP-20901)
Shutdown DTA churn EC2 instance. (DP-20905)
Re-imagine how we backup data in AWS. (DP-20912)
Add page-by-page breakouts for broken links and grade level to pilot service family dashboard. (DP-20967)
Update data dictionary. (DP-20980)
Adjust colors on mass vaccine map. (DP-21016)
Fix vaccine report file name. (DP-20950)
Jan. 21, 2021
 You can now see “Nos per 1000 unique pageviews” for your entire organization on the Organization Web Analytics dashboard. (DP-16202)
Formstack data missing for a few days this fall. (DP-20517)
Delete surplus data in FormstackUI Fraud form. (DP-20541)
Add an additional question to Formstack UI Fraud form. (DP-20800)
Optimize the data for Promotional page configured metrics. These should now load much faster. (DP-20838)
Provide PFML users with GA access & write up instructions for Any on cross-domain property access. (DP-20858)
Fix issue with Formstack API not adjusting for daylight savings. (DP-20865)
Superset legends cut off most labels. Unfortunately, there’s no easy fix, and we settled for rewriting many of the labels so that they were shorter. (DP-20903)
Modify “hide chart” script to work on Service Family dashboard. Users should now see only the KPIs that are relevant. (DP-20904)
Jan. 14, 2021
QA all paidleave.mass.gov dashboard flow scorecards. (DP-20789)
Add new paths to paidleave taskPath variable. (DP-20728)
Add child page path to service families lookup table. (DP-20807)
Rerun cross-domain ETL for Jan. 3. (DP-20857)
QA KPI charts on service family dashboard. (DP-20866)
Convert hours to milliseconds in ETL or in Superset for duration KPIs. (DP-20902)
Generate a data dictionary for ETL repo. (DP-14189)
Release prototype service family dashboard. (DP-19806)
Investigate remaining (not set) event labels in Google Analytics. (DP-20391)
Document DUA dashboard. (DP-20711)
Cleanup tableau EC2 instance on AWS. (DP-20762)
Automate vaccine user feedback query. (DP-20765)
Create report on plastform activity. (DP-20813)
Add page-by-page nos per 1000 contribution chart to service family dashboard. (DP-20827)
Figure out and fix whatever’s causing ETL health check errors. (DP-20872)
Jan. 6, 2021
Add KPI metrics reporting views to DB. (DP-20645)
Fix Formstack ETL failure for 12-29. (DP-20799)
Spotcheck GTM tags in different browsers. (DP-20314)
Courts/Bentley project time tracking ticket. (DP-20522)
Document paidleave.mass.gov dashboard. (DP-20700)
Update UI benefits calculator with new variable. (DP-20811)
Fix travel metrics DAG failure. (DP-20822)
Update PFML contribution calculator variable. (DP-20826)
​
​
Digital analytics products - Previous
Release notes
2020 analytics release notes
Last modified 6mo ago