2021 analytics release notes
Releases made in 2021.

July 1, 2021

    Release Google Search Console data to CMS dashboards. (DP-18333)
    Add internal documentation to ETL repo. (DP-22092, DP-22110)
    May platform activity report. (DP-22222, DP-22388)
    Implement cross-domain tracking between Mass.gov & vaxmillions.com. (DP-22360, DP-22364, DP-22365, DP-22366)
    Feedback manager export now includes an organization column. (DP-20386)

June 24, 2021

    Clean up repo post bigquery refactor. (DP-21998)
    Add internal search data (from search.mass.gov) to CMS dashboards. (DP-22161, DP-22162)
    Offboard Geoff. (DP-22304)
    Report to Joe on analytics question authors have had during the past few months. (DP-22306)

June 17, 2021

    Remove SSNs from Formstack. (DP-22238)
    Hand off cross-domain data to DCR/EEA. (DP-22219)
    Fixed bug that prevented you from exporting CSV data in the feedback manager (DP-22225)
    Vaccine preregistration cleanup work/ (DP-22244, DP-22236)

June 10, 2021

    Begin filtering additional social security formats out of formstack data. (DP-22157)
    Add additional summary tables for archiving web analytics data. (DP-12389, DP-21902, DP-21992)
    Combine all BigQuery API calls into one job. (DP-21726)
    Add partitioning to analytics.pageviews. (DP-21718)
    Release platform activity report. (DP-21985)
    Documenting DCR xdomain analytics work. (DP-22073)
    Update other tag manager containers for DCR xdomain pageview. (DP-22136)
    Drop GA tables. (DP-22138)
    Add validation errors tags for DCR xdomain tracking. (DP-22156)
    Chatbot discovery. (DP-22185)
    DCR xdomain QA. (DP-22205, DP-22207)
    Document how to upgrade Superset. (DP-22091)
    Add documentation for using Airflow. (DP-22093)

June 3, 2021

    Make it possible to query BQ and XD data using Athena. (DP-21629)
    Investigate if we filter out phone numbers in Formstack. (DP-21795)
    Review massdcrcamping website. (DP-22069)
    Create tags and triggers for DCR camping site. (DP-22071)
    Create task path variable. (DP-22072)
    Set up GA view for DCR. (DP-22074)
    Set up Glue definitions in Terraform. (DP-22154)
    Add link click coverage that doesn’t collect personal data to DCR camping tags. (DP-22178)
    Merge dependabot update for Flask-Appbuilder. (DP-22202)
    Plan data structure for search.mass.gov data. (DP-22106)
    Explore search.mass.gov data. (DP-22108)

May 27, 2021

    Filter query parameters out of Google Analytics “All Public Traffic” view. (DP-21224)
    Add partitioning to analytics.entrance_groups. (DP-21743)
    Fix bug that caused the “author” and “organization” fields to clear when you submitted a Feedback Manager query. (DP-21788)
    Create basic variables for reserveamerica cross-domain tracking. (DP-22070)
    Conduct informal interviews to better understand use case for search.mass.gov data. (DP-22107)

May 21, 2021

    Filter query parameters out of Google Analytics “All Public Traffic” view. (DP-21224)
    Add partitioning to analytics.entrance_groups. (DP-21743)
    Fix bug that caused the “author” and “organization” fields to clear when you submitted a Feedback Manager query. (DP-21788)
    Create basic variables for reserveamerica cross-domain tracking. (DP-22070)
    Conduct informal interviews to better understand use case for search.mass.gov data. (DP-22107)

May 20, 2021

    Modify API options to allow for search terms (DP-21656)
    Run analytics schema backfills (DP-21381)
    Add partitioning to previous page summary (DP-21741)
    Add partitioning to next page summary (DP-21742)
    Onboarding for Geoff St. Pierre (DP-21781)
    Update ETL repo changelog (DP-21997)
    Add documentation for Formstack DAG (DP-21413)

May 13, 2021

    Work toward simplifying ETL processing. (DP-21701, DP-21747)
    Lock down Superset permissions. (DP-22003)
    Make sure develop environment database has all necessary data. (DP-21593)
    Release March platform activity report. (DP-21605)
    Update chatbot dashboard. (DP-21715)
    Add partitioning to analytics.page_events. (DP-21717, DP-21971)
    Automate creating a JIRA ticket when a job fails. (DP-21776, DP-21987)
    Create eligibility form data set. (DP-21860)
    Add vaccine options to vaccine locations map. (DP-21893)
    Investigate and fix Drupal ETL failure. (DP-21959)

May 6, 2021

    Work toward making it possible to search for labels in the feedback manager (DP-21649, DP-21650, DP-21805)
    Adjust structure of data in S3 (DP-21714, DP-21787)
    Add documentation for partitioning (DP-21738)
    Fixed a bug in the Feedback Manager where the organization and author filters didn’t work if you filtered for more than 1 organization or author. (DP-21762)
    Understand how alerts work in Airflow. (DP-21775)
    Back Formstack cleanup date up to 5 weeks. (DP-21794)

April 29, 2021

    Handoff PFML analytics materials. (DP-21412)
    Adjust formatting of how labels come over from Drupal API. (DP-21662)
    Eliminate redundant scripts from ETL processing. (DP-21699, DP-21700, DP-21708)

April 22, 2021

    Propose new, slimmed-down architecture for ETL processing. (DP-21424)
    Eliminate bq_ and xd_ page_trace. (DP-21668)
    Update chatbot documentation. (DP-21693)

April 15, 2021

    Report on traffic to children metric. (DP-20964)
    Create February platform activity report. (DP-21128)
    Explore partitioning bq_events. (DP-21429)
    Test regex performance to see if we can offer a regex search for feedback manager. (DP-21654)

April 8, 2021

    Fix bug where wrong organizations were being associated with some pages in the reporting schema. (DP-21240)
    Various items for tagging vaccinesignup.mass.gov. (DP-21405)
    Investigate and improve indexes throughout databases. (DP-21448)
    Convert pageviews and scores view into materialized view so that Drupal API calls won’t take down reporting db. (DP-21563)
    Drop bq_ and xd_ node. (DP-21579)
    Automate Color and Curative codes dropoff. (DP-21590)
    Cleanup after warehouse refactor. (DP-21601)
    Update backpublish.py. (DP-21612)

April 1, 2021

    Automate Twilio logs ingestion. (DP-21581)
    Automate sending codes to Color and Curative. (DP-21469)
    Add a date column to bq_node. (DP-21428)
    Create documentation for chatbot dashboard. (DP-21399)
    Create chatbot analytics dashboard. (DP-21296)
    Use BETWEEN instead of DATE_TRUNC. (DP-20996)
    Optimize ETL processing. (DP-20787, DP-20782, DP-20781, DP-20780)

March 25, 2021

    Interview MassHealth authors about service family dashboard. (DP-21012)
    Onboard Noah P. (DP-21426)
    Fix backpublish script bug. (DP-21536)

March 18, 2021

    Optimize several warehouse queries. (DP-20784, DP-20786)
    Upgrade Postgres to latest stable version. (DP-20982, DP-21427)
    Address RDS instances under heavy load. (DP-21280)
    Add vaccinesignup.mass.gov to warehouse. (DP-21397)
    Update vaccine dashboard. (DP-21430)
    Add vaccine feedback form to ETL. (DP-21437)
    Generate CSVs for contact lists. (/DP-21441)

March 11, 2021

    Optimize several warehouse queries. (DP-20778, DP-20788)
    Fix bug that caused nos per 1000 contributions on the service family dashboard to add up to more than 100% (DP-20931)
    Document Formstack data flow (DP-21189)
    Add items to the vaxfinder & vaccine content dashboard (DP-21211)
    Investigate mysterious link clicks in vaccinesignup.mass.gov analytics. (DP-21270)
    Update nos per 1000 on DUA dashboard. (DP-21311)
    Add vaxfinder.mass.gov to cross-domain ETL. (DP-21312)
    Update GTM and ETL for paidleave.mass.gov path changes. (DP-21323)
    Use new terraform version for ETL databases. (DP-21329)

March 4, 2021

    Move paidleave.mass.gov query params to a separate dimension. (DP-21049)
    Factor fact_site_improve out of existence. (DP-20777)
    Change date formatting on Superset charts for clarity. (DP-20925)
    Create paidleave feedback scoring metric. (DP-20987)
    Add Drupal tables to data dictionary. (DP-21217)
    Formstack cleanup failing (429 error). (DP-21235)
    Get travel form automated job running for yesterday. (DP-21267)
    Get UI data for Karthik. (DP-21287)
    Duplicate chatbot tags for cross-domain property. (DP-21297)
    Webserver container hasn’t upgraded. (DP-21313)
    To-dos for attestation data privacy. (DP-21314)
    Make bq_node performant again. (DP-21316)
    Add tracking for new promo page buttons. (DP-21340)
    Dev reporting DB out of space. (DP-21351)

Feb. 25, 2021

    Make development environment useful again. (DP-20770)
    Prep for ESC meeting on Feb. 17. (DP-21117)
    Add vaxfinder formstack form to ETL. (DP-21177)
    Stop running fact_exits in warehouse schema. (DP-20774)
    Stop running dim_source in warehouse schema. (DP-20775)
    Upgrade Airflow to newest stable version. (DP-20981)
    Refine COVID-19 vaccine promo page dashboard. (DP-21146)
    Add chatbot tracking. (DP-21170)
    QA data coming out of refactored warehouse. (DP-21181)
    Siteimprove data is not accurate on service family dashboards. (DP-21195)
    Update monthly_dev_cleanup script for new tables. (DP-21268)

Feb. 18, 2021

    Warehouse.dim_source load failing because of junk data in ETL. (DP-21206)
    Verify that query parameters are correctly parsed when aggregating paidleave.mass.gov sessions. (DP-21051)
    Add data from attestation form to postgres and s3. (DP-21081)
    Onboard Jane Lee. (DP-21110)
    Make vaccine feedback available for xFact. (DP-21150)
    Tag vaxfinder.mass.gov. (DP-21176)
    Division by zero error in reporting ETL. (DP-21180)
    Various improvements for production ETL environment. (DP-21182)
    Remove misleading columns in analytics.daily_feedback_count. (DP-17283)

Feb. 11, 2021

    Take fact_page and fact_daily_feedback offline. (DP-20767, DP-20776)
    Report on our options for using AWS more efficiently. (DP-20863)
    Create new creds for HED licensing personnel. (DP-20907)
    Create January Platform Activity Report. (DP-20898, DP-20963)
    Optimize use of dev environment Superset databases. (DP-20985)
    Research and talk to DTA about service family data. (DP-21011)
    Create reports for vaccine landing page. (DP-21065)
    Document COVID-19 vaccine dashboard. (DP-21092)
    Update vaccine query to include more pages (DP-21102)

Feb. 4, 2021

    Analysis for ESC meeting presentation. (DP-21014)
    Catchup DAG runs for January. (DP-21062)
    Write script to generate First Responders map. (DP-20797)
    Platform comms (DP-20921, DP-20920)
    Generate accessible Excel file as alternative to map. (DP-21015)
    Shutdown unused RDS instance. (DP-21019)
    Investigate QA check error. (DP-21052)
    “Refresh materialized views” task failed on 1/27 (DP-21053)
    Tag Caspio interactions. (DP-21066)
    Monthly_kpi_components table doesn’t have Feb. data prior to SiteImprove DAG running. (DP-21105)

Jan. 28, 2021

    Refactor temp_analytics schema to skip fact_page_events. (DP-20766)
    Top-trafficked service family research. (DP-20864)
    Analytics ETL is failing dev environment. (DP-20999)
    Add sensors for cross-domain data and siteimprove data. (DP-20452, DP-20806)
    Update DUA feedback for DUA dashboard. (DP-20883)
    QA rich text link tags post Drupal release. (DP-20897)
    Upgrade Terraform to latest version. (DP-20901)
    Shutdown DTA churn EC2 instance. (DP-20905)
    Re-imagine how we backup data in AWS. (DP-20912)
    Add page-by-page breakouts for broken links and grade level to pilot service family dashboard. (DP-20967)
    Update data dictionary. (DP-20980)
    Adjust colors on mass vaccine map. (DP-21016)
    Fix vaccine report file name. (DP-20950)

Jan. 21, 2021

    You can now see “Nos per 1000 unique pageviews” for your entire organization on the Organization Web Analytics dashboard. (DP-16202)
    Formstack data missing for a few days this fall. (DP-20517)
    Delete surplus data in FormstackUI Fraud form. (DP-20541)
    Add an additional question to Formstack UI Fraud form. (DP-20800)
    Optimize the data for Promotional page configured metrics. These should now load much faster. (DP-20838)
    Provide PFML users with GA access & write up instructions for Any on cross-domain property access. (DP-20858)
    Fix issue with Formstack API not adjusting for daylight savings. (DP-20865)
    Superset legends cut off most labels. Unfortunately, there’s no easy fix, and we settled for rewriting many of the labels so that they were shorter. (DP-20903)
    Modify “hide chart” script to work on Service Family dashboard. Users should now see only the KPIs that are relevant. (DP-20904)

Jan. 14, 2021

    QA all paidleave.mass.gov dashboard flow scorecards. (DP-20789)
    Add new paths to paidleave taskPath variable. (DP-20728)
    Add child page path to service families lookup table. (DP-20807)
    Rerun cross-domain ETL for Jan. 3. (DP-20857)
    QA KPI charts on service family dashboard. (DP-20866)
    Convert hours to milliseconds in ETL or in Superset for duration KPIs. (DP-20902)
    Generate a data dictionary for ETL repo. (DP-14189)
    Release prototype service family dashboard. (DP-19806)
    Investigate remaining (not set) event labels in Google Analytics. (DP-20391)
    Document DUA dashboard. (DP-20711)
    Cleanup tableau EC2 instance on AWS. (DP-20762)
    Automate vaccine user feedback query. (DP-20765)
    Create report on plastform activity. (DP-20813)
    Add page-by-page nos per 1000 contribution chart to service family dashboard. (DP-20827)
    Figure out and fix whatever’s causing ETL health check errors. (DP-20872)

Jan. 6, 2021

    Add KPI metrics reporting views to DB. (DP-20645)
    Fix Formstack ETL failure for 12-29. (DP-20799)
    Spotcheck GTM tags in different browsers. (DP-20314)
    Courts/Bentley project time tracking ticket. (DP-20522)
    Document paidleave.mass.gov dashboard. (DP-20700)
    Update UI benefits calculator with new variable. (DP-20811)
    Fix travel metrics DAG failure. (DP-20822)
    Update PFML contribution calculator variable. (DP-20826)
Last modified 3mo ago