2021 analytics release notes

Releases made in 2021.

July 1, 2021

• Release Google Search Console data to CMS dashboards. (DP-18333)

• Add internal documentation to ETL repo. (DP-22092, DP-22110)

• May platform activity report. (DP-22222, DP-22388)

• Implement cross-domain tracking between Mass.gov & vaxmillions.com. (DP-22360, DP-22364, DP-22365, DP-22366)

• Feedback manager export now includes an organization column. (DP-20386)

June 24, 2021

• Clean up repo post bigquery refactor. (DP-21998)

• Add internal search data (from search.mass.gov) to CMS dashboards. (DP-22161, DP-22162)

• Offboard Geoff. (DP-22304)

• Report to Joe on analytics question authors have had during the past few months. (DP-22306)

June 17, 2021

• Remove SSNs from Formstack. (DP-22238)

• Hand off cross-domain data to DCR/EEA. (DP-22219)

• Fixed bug that prevented you from exporting CSV data in the feedback manager (DP-22225)

• Vaccine preregistration cleanup work/ (DP-22244, DP-22236)

June 10, 2021

• Begin filtering additional social security formats out of formstack data. (DP-22157)

• Add additional summary tables for archiving web analytics data. (DP-12389, DP-21902, DP-21992)

• Combine all BigQuery API calls into one job. (DP-21726)

• Add partitioning to analytics.pageviews. (DP-21718)

• Release platform activity report. (DP-21985)

• Documenting DCR xdomain analytics work. (DP-22073)

• Update other tag manager containers for DCR xdomain pageview. (DP-22136)

• Drop GA tables. (DP-22138)

• Add validation errors tags for DCR xdomain tracking. (DP-22156)

• Chatbot discovery. (DP-22185)

• DCR xdomain QA. (DP-22205, DP-22207)

• Document how to upgrade Superset. (DP-22091)

• Add documentation for using Airflow. (DP-22093)

June 3, 2021

• Make it possible to query BQ and XD data using Athena. (DP-21629)

• Investigate if we filter out phone numbers in Formstack. (DP-21795)

• Review massdcrcamping website. (DP-22069)

• Create tags and triggers for DCR camping site. (DP-22071)

• Create task path variable. (DP-22072)

• Set up GA view for DCR. (DP-22074)

• Set up Glue definitions in Terraform. (DP-22154)

• Add link click coverage that doesn’t collect personal data to DCR camping tags. (DP-22178)

• Merge dependabot update for Flask-Appbuilder. (DP-22202)

• Plan data structure for search.mass.gov data. (DP-22106)

• Explore search.mass.gov data. (DP-22108)

May 27, 2021

• Filter query parameters out of Google Analytics “All Public Traffic” view. (DP-21224)

• Add partitioning to analytics.entrance_groups. (DP-21743)

• Fix bug that caused the “author” and “organization” fields to clear when you submitted a Feedback Manager query. (DP-21788)

• Create basic variables for reserveamerica cross-domain tracking. (DP-22070)

• Conduct informal interviews to better understand use case for search.mass.gov data. (DP-22107)

May 21, 2021

• Filter query parameters out of Google Analytics “All Public Traffic” view. (DP-21224)

• Add partitioning to analytics.entrance_groups. (DP-21743)

• Fix bug that caused the “author” and “organization” fields to clear when you submitted a Feedback Manager query. (DP-21788)

• Create basic variables for reserveamerica cross-domain tracking. (DP-22070)

• Conduct informal interviews to better understand use case for search.mass.gov data. (DP-22107)

May 20, 2021

• Modify API options to allow for search terms (DP-21656)

• Run analytics schema backfills (DP-21381)

• Add partitioning to previous page summary (DP-21741)

• Add partitioning to next page summary (DP-21742)

• Onboarding for Geoff St. Pierre (DP-21781)

• Update ETL repo changelog (DP-21997)

• Add documentation for Formstack DAG (DP-21413)

May 13, 2021

• Work toward simplifying ETL processing. (DP-21701, DP-21747)

• Lock down Superset permissions. (DP-22003)

• Make sure develop environment database has all necessary data. (DP-21593)

• Release March platform activity report. (DP-21605)

• Update chatbot dashboard. (DP-21715)

• Add partitioning to analytics.page_events. (DP-21717, DP-21971)

• Automate creating a JIRA ticket when a job fails. (DP-21776, DP-21987)

• Create eligibility form data set. (DP-21860)

• Add vaccine options to vaccine locations map. (DP-21893)

• Investigate and fix Drupal ETL failure. (DP-21959)

May 6, 2021

• Work toward making it possible to search for labels in the feedback manager (DP-21649, DP-21650, DP-21805)

• Adjust structure of data in S3 (DP-21714, DP-21787)

• Add documentation for partitioning (DP-21738)

• Fixed a bug in the Feedback Manager where the organization and author filters didn’t work if you filtered for more than 1 organization or author. (DP-21762)

• Understand how alerts work in Airflow. (DP-21775)

• Back Formstack cleanup date up to 5 weeks. (DP-21794)

April 29, 2021

• Handoff PFML analytics materials. (DP-21412)

• Adjust formatting of how labels come over from Drupal API. (DP-21662)

• Eliminate redundant scripts from ETL processing. (DP-21699, DP-21700, DP-21708)

April 22, 2021

• Propose new, slimmed-down architecture for ETL processing. (DP-21424)

• Eliminate bq_ and xd_ page_trace. (DP-21668)

• Update chatbot documentation. (DP-21693)

April 15, 2021

• Report on traffic to children metric. (DP-20964)

• Create February platform activity report. (DP-21128)

• Explore partitioning bq_events. (DP-21429)

• Test regex performance to see if we can offer a regex search for feedback manager. (DP-21654)

April 8, 2021

• Fix bug where wrong organizations were being associated with some pages in the reporting schema. (DP-21240)

• Various items for tagging vaccinesignup.mass.gov. (DP-21405)

• Investigate and improve indexes throughout databases. (DP-21448)

• Convert pageviews and scores view into materialized view so that Drupal API calls won’t take down reporting db. (DP-21563)

• Drop bq_ and xd_ node. (DP-21579)

• Automate Color and Curative codes dropoff. (DP-21590)

• Cleanup after warehouse refactor. (DP-21601)

• Update backpublish.py. (DP-21612)

April 1, 2021

• Automate Twilio logs ingestion. (DP-21581)

• Automate sending codes to Color and Curative. (DP-21469)

• Add a date column to bq_node. (DP-21428)

• Create documentation for chatbot dashboard. (DP-21399)

• Create chatbot analytics dashboard. (DP-21296)

• Use BETWEEN instead of DATE_TRUNC. (DP-20996)

• Optimize ETL processing. (DP-20787, DP-20782, DP-20781, DP-20780)

March 25, 2021

• Interview MassHealth authors about service family dashboard. (DP-21012)

• Onboard Noah P. (DP-21426)

• Fix backpublish script bug. (DP-21536)

March 18, 2021

• Optimize several warehouse queries. (DP-20784, DP-20786)

• Upgrade Postgres to latest stable version. (DP-20982, DP-21427)

• Address RDS instances under heavy load. (DP-21280)

• Add vaccinesignup.mass.gov to warehouse. (DP-21397)

• Update vaccine dashboard. (DP-21430)

• Add vaccine feedback form to ETL. (DP-21437)

• Generate CSVs for contact lists. (/DP-21441)

March 11, 2021

• Optimize several warehouse queries. (DP-20778, DP-20788)

• Fix bug that caused nos per 1000 contributions on the service family dashboard to add up to more than 100% (DP-20931)

• Document Formstack data flow (DP-21189)

• Add items to the vaxfinder & vaccine content dashboard (DP-21211)

• Investigate mysterious link clicks in vaccinesignup.mass.gov analytics. (DP-21270)

• Update nos per 1000 on DUA dashboard. (DP-21311)

• Add vaxfinder.mass.gov to cross-domain ETL. (DP-21312)

• Update GTM and ETL for paidleave.mass.gov path changes. (DP-21323)

• Use new terraform version for ETL databases. (DP-21329)

March 4, 2021

• Move paidleave.mass.gov query params to a separate dimension. (DP-21049)

• Factor fact_site_improve out of existence. (DP-20777)

• Change date formatting on Superset charts for clarity. (DP-20925)

• Create paidleave feedback scoring metric. (DP-20987)

• Add Drupal tables to data dictionary. (DP-21217)

• Formstack cleanup failing (429 error). (DP-21235)

• Get travel form automated job running for yesterday. (DP-21267)

• Get UI data for Karthik. (DP-21287)

• Duplicate chatbot tags for cross-domain property. (DP-21297)

• Webserver container hasn’t upgraded. (DP-21313)

• To-dos for attestation data privacy. (DP-21314)

• Make bq_node performant again. (DP-21316)

• Add tracking for new promo page buttons. (DP-21340)

• Dev reporting DB out of space. (DP-21351)

Feb. 25, 2021

• Make development environment useful again. (DP-20770)

• Prep for ESC meeting on Feb. 17. (DP-21117)

• Add vaxfinder formstack form to ETL. (DP-21177)

• Stop running fact_exits in warehouse schema. (DP-20774)

• Stop running dim_source in warehouse schema. (DP-20775)

• Upgrade Airflow to newest stable version. (DP-20981)

• Refine COVID-19 vaccine promo page dashboard. (DP-21146)

• Add chatbot tracking. (DP-21170)

• QA data coming out of refactored warehouse. (DP-21181)

• Siteimprove data is not accurate on service family dashboards. (DP-21195)

• Update monthly_dev_cleanup script for new tables. (DP-21268)

Feb. 18, 2021

• Warehouse.dim_source load failing because of junk data in ETL. (DP-21206)

• Verify that query parameters are correctly parsed when aggregating paidleave.mass.gov sessions. (DP-21051)

• Add data from attestation form to postgres and s3. (DP-21081)

• Onboard Jane Lee. (DP-21110)

• Make vaccine feedback available for xFact. (DP-21150)

• Tag vaxfinder.mass.gov. (DP-21176)

• Division by zero error in reporting ETL. (DP-21180)

• Various improvements for production ETL environment. (DP-21182)

• Remove misleading columns in analytics.daily_feedback_count. (DP-17283)

Feb. 11, 2021

• Take fact_page and fact_daily_feedback offline. (DP-20767, DP-20776)

• Report on our options for using AWS more efficiently. (DP-20863)

• Create new creds for HED licensing personnel. (DP-20907)

• Create January Platform Activity Report. (DP-20898, DP-20963)

• Optimize use of dev environment Superset databases. (DP-20985)

• Research and talk to DTA about service family data. (DP-21011)

• Create reports for vaccine landing page. (DP-21065)

• Document COVID-19 vaccine dashboard. (DP-21092)

• Update vaccine query to include more pages (DP-21102)

Feb. 4, 2021

• Analysis for ESC meeting presentation. (DP-21014)

• Catchup DAG runs for January. (DP-21062)

• Write script to generate First Responders map. (DP-20797)

• Platform comms (DP-20921, DP-20920)

• Generate accessible Excel file as alternative to map. (DP-21015)

• Shutdown unused RDS instance. (DP-21019)

• Investigate QA check error. (DP-21052)

• “Refresh materialized views” task failed on 1/27 (DP-21053)

• Tag Caspio interactions. (DP-21066)

• Monthly_kpi_components table doesn’t have Feb. data prior to SiteImprove DAG running. (DP-21105)

Jan. 28, 2021

• Refactor temp_analytics schema to skip fact_page_events. (DP-20766)

• Top-trafficked service family research. (DP-20864)

• Analytics ETL is failing dev environment. (DP-20999)

• Add sensors for cross-domain data and siteimprove data. (DP-20452, DP-20806)

• Update DUA feedback for DUA dashboard. (DP-20883)

• QA rich text link tags post Drupal release. (DP-20897)

• Upgrade Terraform to latest version. (DP-20901)

• Shutdown DTA churn EC2 instance. (DP-20905)

• Re-imagine how we backup data in AWS. (DP-20912)

• Add page-by-page breakouts for broken links and grade level to pilot service family dashboard. (DP-20967)

• Update data dictionary. (DP-20980)

• Adjust colors on mass vaccine map. (DP-21016)

• Fix vaccine report file name. (DP-20950)

Jan. 21, 2021

• You can now see “Nos per 1000 unique pageviews” for your entire organization on the Organization Web Analytics dashboard. (DP-16202)

• Formstack data missing for a few days this fall. (DP-20517)

• Delete surplus data in FormstackUI Fraud form. (DP-20541)

• Add an additional question to Formstack UI Fraud form. (DP-20800)

• Optimize the data for Promotional page configured metrics. These should now load much faster. (DP-20838)

• Provide PFML users with GA access & write up instructions for Any on cross-domain property access. (DP-20858)

• Fix issue with Formstack API not adjusting for daylight savings. (DP-20865)

• Superset legends cut off most labels. Unfortunately, there’s no easy fix, and we settled for rewriting many of the labels so that they were shorter. (DP-20903)

• Modify “hide chart” script to work on Service Family dashboard. Users should now see only the KPIs that are relevant. (DP-20904)

Jan. 14, 2021

• QA all paidleave.mass.gov dashboard flow scorecards. (DP-20789)

• Add new paths to paidleave taskPath variable. (DP-20728)

• Add child page path to service families lookup table. (DP-20807)

• Rerun cross-domain ETL for Jan. 3. (DP-20857)

• QA KPI charts on service family dashboard. (DP-20866)

• Convert hours to milliseconds in ETL or in Superset for duration KPIs. (DP-20902)

• Generate a data dictionary for ETL repo. (DP-14189)

• Release prototype service family dashboard. (DP-19806)

• Investigate remaining (not set) event labels in Google Analytics. (DP-20391)

• Document DUA dashboard. (DP-20711)

• Cleanup tableau EC2 instance on AWS. (DP-20762)

• Automate vaccine user feedback query. (DP-20765)

• Create report on plastform activity. (DP-20813)

• Add page-by-page nos per 1000 contribution chart to service family dashboard. (DP-20827)

• Figure out and fix whatever’s causing ETL health check errors. (DP-20872)

Jan. 6, 2021

• Add KPI metrics reporting views to DB. (DP-20645)

• Fix Formstack ETL failure for 12-29. (DP-20799)

• Spotcheck GTM tags in different browsers. (DP-20314)

• Courts/Bentley project time tracking ticket. (DP-20522)

• Document paidleave.mass.gov dashboard. (DP-20700)

• Update UI benefits calculator with new variable. (DP-20811)

• Fix travel metrics DAG failure. (DP-20822)

• Update PFML contribution calculator variable. (DP-20826)