Skip to main content

Changelog

Unreleased​

💎 Premium​

  • BigQuery utilities module
  • Google Search Console module

🧰 Core​

  • nothing planned

🤝 Community​

  • nothing planned

Released​

💎 [Premium v2.2.11]​

2026-08-18

🚀 Added​

  • Single-touch splits (ga4_attribution_channel_assist_report) in the GA4 Attribution module - single_touch_journeys, closed_single_touch_journeys and single_touch_revenue separate journeys where a channel was the only touch, and new additive sums (sum_traffic_touches, sum_journey_length_days, sum_days_to_conversion) let dashboards derive average path length and average days to conversion per channel. See Output Tables
  • With-vs-without comparison (ga4_attribution_micro_conversion_report) in the GA4 Attribution module - new journeys_without, closed_journeys_without, closing_conversion_count_without and closing_revenue_without columns quantify a micro-conversion's lift on final outcomes from a single table; with + without ties back to the total journey count, and every configured micro-conversion now appears on every journey date (zero-filled when it did not fire)
  • Column descriptions from config in the GA4 Attribution module - config-driven columns (extra dimensions, metrics, custom channel groupings) now get BigQuery column descriptions automatically, taken from the upstream GA4 column docs or an optional description key on the config entry
  • Compile-time config validation in the GA4 Attribution module - invalid dimension paths (e.g. a ga4_sessions-only field in source_change mode) now fail at compile time with an actionable message instead of erroring mid-run

Changed​

  • Sharper agent skills - the side-effecting skills (reprocessing, custom params, custom models, managing runs, upgrade) end with checkable "Done when" criteria so a half-finished procedure is not reported as done; the onboarding skill was restructured for reliability; the agent guides (AGENTS.md/CLAUDE.md/GEMINI.md) gained a Language section disambiguating the product terms that get mixed up (reprocess vs rebuild, managed vs custom, release vs workflow, ...), and MY-COMPANY.md gained an "Our vocabulary" section for your own terms
  • rebuild_schedule semantics documented in the GA4 Attribution module - the cron gate is evaluated when the Dataform release compiles, not when the workflow executes, so the release must recompile at least as often as the schedule needs to take effect. The config comment and docs now spell this out

🐛 Fixed​

All in the GA4 Attribution module, from a full correctness review:

  • Dates near midnight - journey and model dates were derived in UTC while the rest of the module uses LOCAL_TIMEZONE; touches near midnight landed in the wrong partition
  • Micro-conversion "without" denominators - the *_without columns only covered dates a micro-conversion fired on, understating them for anything that does not fire daily; they now cover every journey date
  • Journey boundaries - inactivity_gap_days is now measured in fractional days (a 47-hour gap trips a 1-day boundary), pre_conversion_lookback_days no longer shatters older touches into single-touch journeys, and an abandoned journey now lapses via the inactivity gap instead of staying open forever
  • Credit edge cases - position_based with both weights 0 dropped the conversion's credit on 2-touch journeys (now splits 50/50, and 2-touch renormalization follows the configured weights); credit is summed at full float precision and rounded once
  • Touch ordering and source propagation - a conversion on a session's first event no longer ties ahead of its own traffic touch, and a conversion event's traffic source now reaches the rest of its session
  • Smaller fixes - zero-touch journeys report path_length 0 (was 1), a duplicate labs.perplexity.ai source-category row fanned out touches on the LIKE join, the conversion-lag <= 30 bucket boundary, and NULL dimensions normalize to '(not set)' / ongoing journeys to '(no conversion)'

Schema Changes​

  • New columns on ga4_attribution_channel_assist_report and ga4_attribution_micro_conversion_report. Attribution tables are rebuilt on the module's rebuild_schedule (default: every run), so the columns appear after the next scheduled rebuild - no manual action needed

💎 [Premium v2.2.10]​

2026-08-04

🚀 Added​

  • Agent skills - every installation now ships ten task-specific skills your AI assistant loads on demand, in two identical copies: .claude/skills/ (Claude Code, Cursor) and .agents/skills/ (Codex CLI, Gemini CLI). They cover setup (ga4df-onboarding), querying, pipeline debugging, custom parameters, custom models, reprocessing and backfills, running the pipeline, the Core→Premium upgrade, cost monitoring, and writing your own skills. Ask in plain language - "backfill last March", "why is today missing", "add a custom event parameter" - and the assistant follows the product's own procedure instead of improvising. See Agent Skills and Onboarding with an AI Agent
  • Bundled scripts the assistant (or you) can run directly: patch_struct_schema.sh adjusts a BigQuery STRUCT column so a new custom parameter can be added without a full rebuild, recompile_release.py recompiles the release configuration from the CLI, and push_dataform_workspace.py pushes a local copy of the workspace back through the Dataform API
  • Setup reference guides the assistant consults before changing configuration: a full ga4 module option reference (every option, its default, whether it needs a schema patch, and what changing it does to history), an attribution setup guide, and a Cloud Storage export setup guide

Changed​

  • Core config arrays now ship empty in the GA4 Attribution module (conversions, attribution_models, touchpoint.source_change_unique_fields) and the Google Ads module (conversion_action_names). Array entries in a core config merge into every installation and cannot be removed from your own config file, so the starter entries now live in your editable copy under includes/custom/modules/<module>/ where you can trim or replace them
  • batch-reprocess.sh is safer - it validates its inputs, prints the batch plan and asks for confirmation before running anything (--yes skips the prompt for non-interactive use), stops on the first failed batch and prints the exact command to resume from there. Default tags now include transactions

🐛 Fixed​

  • Parameter Detection first build - ga4_parameter_detection_report_daily failed to compile on a first (non-incremental) build when compiling with the Dataform CLI; compilations in the Dataform interface were unaffected. The module now builds from scratch either way

🛠️ [Installer v2.2.11]​

2026-08-04

🚀 Added​

  • Agent skill installation - new installations and package updates now deliver the agent skills that ship with Premium v2.2.10+

💎 [Premium v2.2.9]​

2026-07-10

🚀 Added​

  • AI-ready workspace - every installation now ships context files for AI agents: AGENTS.md / CLAUDE.md / GEMINI.md (one agent guide, three standard filenames), LINEAGE.md (every table's grain, upstream sources and downstream consumers) and MY-COMPANY.md (a business-context template you fill in once; preserved across updates). Point Gemini, Claude, Cursor or any coding agent at your Dataform workspace and it knows the boundaries, the data model and your business. See AI-Ready Workspace
  • Source group column (source_group) in the GA4 module - opt-in consolidation of the many source spellings of common platforms (Facebook, Instagram, TikTok, Bing, AI assistants like ChatGPT and Perplexity, ...) into one clean label, added inside fixed_traffic_source (ga4_events) and session_source / last_non_direct_traffic_source (ga4_sessions). Enable with ADD_SOURCE_GROUP_COLUMN, customize via SOURCE_GROUP_ARRAY. See GA4 Configuration
  • Attribution rebuild schedule (rebuild_schedule) in the GA4 Attribution module - a 3-field cron expression that gates the rebuild layer (journeys, models and all reports) to cut cost, e.g. "* * 1" rebuilds Mondays only. Touchpoint tables keep running incrementally every execution. Default "* * *" (rebuild on every run). See Attribution Configuration
  • event_original_occurrence_timestamp on ga4_events - new GA4 export field passed through: for late-arriving events processed after the 72h window (e.g. SDK events), it preserves when the event originally occurred on the device, while event_date/event_timestamp reflect processing time

Changed​

  • Self-describing tables - every output table's BigQuery description now states its grain ("1 row per ..."), and every transformation CTE carries a purpose comment; column descriptions across the documentation set were rewritten for clarity. Helps humans and AI agents alike

Schema Changes​

  • event_original_occurrence_timestamp (top-level, ga4_events) is added automatically on the next incremental run; historical dates stay NULL until reprocessed
  • source_group is added inside existing STRUCT columns when ADD_SOURCE_GROUP_COLUMN is enabled - on an existing installation this requires a schema adjustment or table rebuild, plus a reprocess to backfill history

🛠️ [Installer v2.2.10]​

2026-07-10

🚀 Added​

  • MY-COMPANY.md preserved on updates - the business-context template ships with Premium v2.2.9 and is created when missing, but once you've filled it in, package updates never overwrite it (same guarantee as definitions/custom/ and includes/custom/)

💎 [Premium v2.2.8]​

2026-06-22

🚀 Added​

  • GA4 Attribution Modeling module - multi-touch attribution from raw GA4 events: a touchpoint stream (session or source_change mode) is grouped into user journeys, conversion credit is apportioned across touches by fractional models (first-touch, last-touch, linear, position-based, time-decay) plus an opt-in participation model, and six additive, date-partitioned reports cover channel chains, journey summaries, channel assists, channel transitions (Sankey), conversion lag, and micro-conversion lift. Opt-in (enabled: false); define your conversions and journey.closing_conversions to populate it. See GA4 Attribution Modeling
  • GA4_DATASET_PROJECT - read the GA4 export from a different GCP project than the one your outputs are written to (e.g. a dev/test project reading the production export). Set vars.GA4_DATASET_PROJECT in workflow_settings.yaml; defaults to the output project when unset. The Dataform service account needs BigQuery Data Viewer on the source dataset. See Cross-Project Setup

🛠️ [Installer v2.2.8]​

2026-06-16

🐛 Fixed​

  • Preserve customer GitHub connection on Dataform repo updates - repo updates no longer reconcile away a customer's out-of-band GitHub connection. The Dataform repository's externally-managed git_remote_settings (and workspace compilation overrides) are now left untouched on refresh, so the GitHub remote stays connected and workspaces no longer end up in a conflicted git state

💎 [Premium v2.2.7]​

2026-06-15

🐛 Fixed​

  • Custom lineage views skipped when the step is disabled - the custom GA4 lineage actions (ga4_events_custom, int_ga4_sessions_custom, ga4_sessions_custom) are no longer materialized as orphan pass-through views when CUSTOM_LINEAGE.<table> is false; downstream tables already read the core tables directly in that case
  • is_final timezone fix - the data-finality flag now computes CURRENT_DATE() in the install's LOCAL_TIMEZONE instead of timezone-naive UTC, so the 72-hour late-data window flips on the correct day boundary for the data

💎 [Premium v2.2.6]​

2026-06-11

🚀 Added​

  • Export to Google Cloud Storage - opt-in cross-cloud export that writes your output tables to a GCS bucket as partitioned Parquet, ready to load into Snowflake, Databricks, BigLake, or any Parquet-aware engine. Off by default; enable it globally with the GCS_EXPORT_* settings in workflow_settings.yaml or per-module with a GCS_EXPORT override. Partitioned tables are written as hive-partitioned folders (one file set per date), incremental tables export only the current load window, and the export runs as a best-effort step that never fails the table build. See Export to Google Cloud Storage
  • Open-text SQL filter (sql) for CUSTOM_SESSION_PARAMS_ARRAY and CUSTOM_SESSION_TOTALS in the GA4 module - a raw SQL boolean predicate as an escape hatch when eventFilter/paramFilter can't express the condition (e.g. matching on page.path). Mutually exclusive with eventFilter/paramFilter; for eventsToCount a sql entry must also set renameTo
  • fallbackField for custom channel grouping in the GA4 module - choose which default field unmatched traffic falls back to when useDefaultChannelGroupingAsFallback: true (defaults to default_channel_grouping)

Changed​

  • Empty default conversion_action_names in the Google Ads module - the default config now ships an empty list instead of the Purchase/Add to Cart/30s on page placeholders; define your own conversion actions in the custom config. Existing installs keep their configured values
  • Empty default listAttribution dimensions in the Item Funnel module - dimsToStore and pickDimsFromTheseEvents now ship empty by default; populate them in the custom config to enable item list attribution

🛠️ [Installer v2.2.7]​

2026-06-07

🚀 Added​

  • VPC Service Controls warning - the installer now detects when VPC Service Controls are active on the project and surfaces a warning during setup, so you can resolve access restrictions before they cause silent failures
  • Billing warning - the installer now surfaces a warning when billing configuration may affect the deployment, reducing ambiguity during first-time setups

💎 [Premium v2.2.5]​

2026-06-07

🚀 Added​

  • Brave classified as a search engine in source categories — sessions originating from Brave Search are now correctly attributed to Organic Search in the default channel grouping
  • Improved AI referrer classification — ChatGPT and Perplexity are now classified more accurately in the default channel grouping, reducing misattribution to direct or referral

🐛 Fixed​

  • fixed_traffic_source attribution fix — the GA4 BigQuery export can deliver hits out of order: a hit with an earlier timestamp may arrive in a later batch and therefore lack collected_traffic_source. The previous logic selected the earliest hit, which could be one of those under-informed hits, causing sessions to show incorrect or missing traffic source data. Source derivation now selects the hit with the most information, and cases where timestamp and batch ordering disagree are handled correctly. Multi-source sessions now derive the last source only from hits that have collected_traffic_source set. This resolves incorrect Referral classification for some Organic sources, including gclid traffic without a campaign appearing as Google Referral.

💎 [Premium v2.2.4]​

2026-04-28

🚀 Added​

  • all_conversions and all_conversions_value in the Google Ads module - aggregate of metrics_conversions and metrics_conversions_value across all configured conversion_action_names, added to google_ads_campaign_report, google_ads_ad_group_report, google_ads_ad_report, and google_ads_keyword_report. Provides a stable column name for downstream dashboards regardless of which individual conversion actions are configured
  • addDefault flag for CUSTOM_SESSION_TOTALS in the GA4 module - when true, merges a curated baseline of eventsToCount, uniqueFields, and sumFields (page_view, session_start, ecommerce events, transaction_id, user_id, device.category, page hostname/path, engagement_time_msec, purchase_revenue) with your custom entries (your entries win on renameTo/name collision). Defaults to true for new installs; existing installs without the flag continue to see only their explicitly configured entries
  • onSchemaChange: EXTEND on google_ads_dim_ad_groups, google_ads_dim_ads, google_ads_dim_keywords, and google_ads_gclids - column additions in future releases no longer require a rebuild

🐛 Fixed​

  • Apostrophes and other special characters in Google Ads conversion_action_names (e.g. Client's Call) now work - conversion action names are wrapped in BigQuery triple-quoted string literals so escaping no longer breaks Dataform's compile pipeline
  • Clearer error messages for Google Ads misconfigurations - separate, actionable errors when enabled: true but gads_sources is missing/empty/malformed, when conversion_action_names is missing/empty/malformed, and when enabled: false but gads_sources contains entries
  • Generic config merge now concatenates and dedupes arrays (by renameTo/name/value, custom wins) instead of replacing whole arrays - lets core ship baseline arrays users can extend without redeclaring

Schema Changes​

This release introduces new columns on Google Ads report tables (all_conversions, all_conversions_value). Thanks to onSchemaChange: EXTEND no rebuild is required, but historical values for these columns will only populate from this version forward. Run a reprocessing job over the desired date range if you need them backfilled.

For CUSTOM_SESSION_TOTALS, existing installs see no schema change unless they explicitly opt in by setting addDefault: true in includes/custom/modules/ga4/config.js.

🛠️ [Installer v2.2.2]​

2026-04-08

🚀 Added​

  • Automated GA4 Export Trigger - automatically update your GA4 output tables the moment the GA4 daily events export lands in BigQuery (Premium 💎)
  • Email Alerts for Dataform Errors - get notified by email when a Dataform workflow invocation fails (Premium 💎)

💎 [Premium v2.2.3]​

2026-04-08

🚀 Added​

  • Custom Lineage (ga4_events_custom, int_ga4_sessions_custom, ga4_sessions_custom) in the GA4 module - inject custom transformation layers between core pipeline tables as views or incremental tables, with automatic routing for all downstream modules
  • Reprocessing & Batching - reprocess specific date ranges without rebuilding entire history, with a batch script (scripts/batch-reprocess.sh) for large backfills that avoids BigQuery's 6-hour timeout
  • PII Detection in the Parameter Detection module - automatically detects personally identifiable information in parameter names and values, with 19 configurable value patterns and a pattern tester view
  • Multi-property support in the Parameter Detection module - property_id and property_name columns in all output tables for per-property parameter auditing
  • paramFilter for Session Totals - eventsToCount, uniqueFields, and sumFields in CUSTOM_SESSION_TOTALS now support the same paramFilter syntax as CUSTOM_SESSION_PARAMS_ARRAY for conditional aggregation

🐛 Fixed​

  • ignore_referrer case sensitivity - now correctly matches GA4's lowercase "true" value
  • Medium attribution CASE reordering to prevent misclassification
  • Module dependency resolution when a module config is disabled
  • ref() resolution for disabled modules

Schema Changes​

Updating to this version adds property_id and property_name columns to ga4_parameter_detection_report_daily via onSchemaChange: EXTEND. No full rebuild is required, but if you need historical values for these fields you will need to rebuild the table.

💎 [Premium v2.1.2]​

2026-03-09

🚀 Added​

  • Item list attribution report (ga4_item_list_attribution_report) in the Item Funnel module - attributes e-commerce funnel metrics to the item list where an item was first or last seen in a session (e.g. "Related Products", "Homepage Carousel"), with first-touch (metrics_first) and last-touch (metrics_last) attribution models

🛠️ [Installer v2.2.0]​

2026-03-07

Changed​

Strict Act As mode update

We have updated the installer to align with the Strict Act-As design pattern for Dataform execution, improving security, isolation of responsibilities, and auditability. Previously, all permissions were assigned to the Dataform Default Service Account, combining orchestration and BigQuery data access. With this update, permissions are now explicitly split between two service accounts.

  • Dataform Default Service Account
    • Dataform Service Agent
    • Secret Manager Secret Accessor
    • Service Account Token Creator
    • Service Account User

This service account is now limited to orchestration, scheduling, metadata access, and impersonation only.
It no longer has direct access to BigQuery data.

  • Custom Service Account
    • BigQuery Data Editor
    • BigQuery Data Viewer
    • BigQuery Job User

This service account is used exclusively for query execution in BigQuery and is accessed only via impersonation during Dataform runs.

This change brings the installer in line with Google Cloud security best practices and Dataform Strict Act-As mode requirements.

💎 [Premium v2.1.1]​

2026-02-17

Schema Changes​

Updating to this version will add property_name to all output tables. No full rebuild required thanks to onSchemaChange: EXTEND.

🚀 Added​

  • Cross-project multi-property: EXTRA_GA4_DATASETS_ARRAY now supports datasets in different GCP projects and custom property names (string, dot-notation, or object format)
  • property_name column across ga4_events, ga4_sessions, and ga4_transactions — companion to property_id, shows the custom name if configured
  • CLUSTER_BY config (Premium): prepend custom clustering columns before fixed trailing columns on ga4_events and ga4_sessions
  • onSchemaChange: EXTEND on all intermediate and output tables for automatic schema evolution

Changed​

  • Removed "same GCP project" requirement for multi-property setups

💎 [Premium v2.0.5]​

2025-12-18

Schema Changes​

Updating to this version will require some tables to be fully rebuild because of a addition of new columns

  • int_ga4_sessions and ga4_sessions with new first_user_id column
  • int_ga4_transactions and ga4_transactions with new property_id column
  • ga4_events with new records in event_params

🚀 Added​

  • unique_search term, gad_campaingid, gad_source as standard event params
  • gclsrc as param to check to classify as Google
  • first_user_id in int_ga4_sessions and ga4_sessions
  • property_id as column in int_ga4_transactions and ga4_transactions
  • is_multiday_session column : Flag indicating if the session has events across more than one day

🐛 Fixed​

  • Add missing column documentation fields

Changed​

  • Standardized SQL style
  • Renaming modules and tables
  • Changed directory structure and main config file moved

🧰 🤝 [Core v1.19.0]​

2025-12-18

Schema Changes​

Updating to this version will require some tables to be fully rebuild because of a addition of new columns

  • int_ga4_sessions and ga4_sessions with new first_user_id column
  • int_ga4_transactions and ga4_transactions with new property_id column
  • ga4_events with new records in event_params

🚀 Added​

  • unique_search term, gad_campaingid, gad_source as standard event params
  • gclsrc as param to check to classify as Google
  • first_user_id in int_ga4_sessions and ga4_sessions
  • property_id as column in int_ga4_transactions and ga4_transactions
  • is_multiday_session column : Flag indicating if the session has events across more than one day

🐛 Fixed​

  • Add missing column documentation fields

Changed​

  • Standardized SQL style
  • Renaming modules and tables
  • Changed directory structure and main config file moved

💎 [Premium v2.0.4]​

2025-11-11

🚀 Added​

  • Custom Channel Groupings

💎 [Premium v2.0.3]​

2025-11-06

🚀 Added​

  • Multi-property support
  • Google Ads module

💎 [Premium v2.0.2]​

2025-10-23

🚀 Added​

  • Extend custom session-level parameter filtering options

💎 [Premium v2.0.1]​

2025-10-10

🐛 Fixed​

  • Superform Labs internal bugfix iteration

💎 [Premium v2.0.0]​

2025-09-22

🚀 Added​

  • User stitching
  • Support for using 'intraday' and 'fresh' tables
  • BigQuery cost monitoring + dashboard
  • Session totals in ga4_sessions
  • Custom parameter aggregation at session level
  • Custom parameter detection + dashboard
  • Items reporting table
  • Conversion rate reporting table

🧰 🤝 [Core v1.18.0]​

2025-08-27

🚀 Added​

  • AI referrers added to source_categories.json for improved default channel grouping logic.
  • Default channel grouping in int_ga4_sessionsnow matches the source categories with LIKE operator for enhanced pattern matching.

🐛 Fixed​

  • Added default_config.js warning text + reordered file contents for better readability.
  • Updated is_measurement_protocol_hit since MP supports device parameters since 2025-06-11.
  • Added app_info.id check to fix assertions_sessions_validity resulting in false positive for app events.
  • Added events_fresh_* filter to assertions_tables_timeliness to prevent false positives.

🧰 🤝 [Core v1.17.0]​

2025-05-13

🚀 Added​

  • Labeling Tables and Jobs. 5 labels categories are availble: Operation, Brand, Licence type, Package version, Tool.

Changed​

  • Assertions set to false by default.

🛠️ [Installer]​

2025-05-13

🚀 Added​

  • Installer is now displaying Licence type, installer and models version.
  • Backfilling job run from installer is now tracked with specific label operation: backfilling.

🧰 🤝 [Core v1.16.0]​

2025-04-09

🚀 Added​

  • Bluesky referrals are now classified as Organic Social in the default channel group.
  • “Organic AI” added as new default channel group. Most common LLM referrers are added to this group. This feature can be enabled/disabled using the config option EXTRA_CHANNEL_GROUPS.

🧰 🤝 [Core v1.15.0]​

2025-03-11

🐛 Fixed​

  • Replaced --- with ; while ensuring no duplicate semicolons appear, as the Dataform CLI was unable to handle them correctly.
  • Proper handling of sessions spanning midnight in incremental runs. Previously, sessions crossing midnight could result in duplicate session_ids. Multi-day sessions are now correctly appended to the intermediate table and properly deduplicated in the output.

🧰 🤝 [Core v1.14.0]​

2025-02-05

🐛 Fixed​

  • Refactored LOOKBACK_MILLIS to be defined inline instead of a separate variable.
  • Removed LAST_NON_DIRECT_LOOKBACK_MILLIS from default_config.js, ensuring calculations use a single inline formula.

🛠️ [Installer]​

2025-02-05

🚀 Added​

  • Added support for more than 50 datasets

🧰 🤝 [Core v1.13.0]​

2025-01-28

Changed​

  • Updated generateFilterTypeFromListSQL function to properly handle NULL values.
  • Now uses coalesce(column, "") to avoid NULL-related filtering issues.

🧰 🤝 [Core v1.12.0]​

2025-01-23

Changed​
  • Excluded "fresh" tables from exports in ga4_events.sqlx.
  • Ensures that temporary fresh tables do not interfere with data processing.

🛠️ [Installer]​

2025-01-23

🚀 Added​

  • Added user menu with logout

🧰 🤝 [Core v1.10.0]​

2024-12-19

🚀 Added​

  • Initial commit introducing ga4_transactions model.
  • Documentation formatting and structure for ga4_transactions columns.
  • Supports incremental processing with nested items.
  • Expanded ga4_sessions with new columns for traffic source tracking.
  • Improved demo_daily_sessions_report schema with platform and stream ID tracking.
  • Introduced _run_timestamp column in ga4_events for better traceability.
  • Updated documentation to include newly added columns in ga4_events.
  • Additional columns to ga4_sessions and ga4_events tables.
  • Introduced platform in session struct for better tracking.

🐛 Fixed​

  • Improved logic for properties without "final" days, ensuring they now increment correctly in sessions, transactions, and events models.

Changed​

  • Adjusted schema definitions for ga4_transactions to improve refund tracking.
  • Updated documentation to reflect the schema modifications.
  • Refined handling of refunds in transaction roll-ups.
  • Refactored function names to improve readability.
  • Added JSDoc comments for key helper functions.
  • Default channel grouping logic to ensure "Direct" fallback for NULL values.
  • Moved all event transformation logic to outputs/ga4_events.
  • Simplified upstream intermediate models to reduce redundancy.

🧰 🤝 [Core v1.6.0]​

2024-10-24

Changed​

  • Streamlined data partitioning to improve performance.
  • Introduced detailed column documentation for ga4_events and ga4_sessions.

Removed​

  • Removed stream_id from clusterBy fields.

Changed​

  • Removed non-intraday columns from session_traffic_source_last_click struct.

🛠️ [Installer]​

2024-10-24

🚀 Added​

  • Installer version

🧰 🤝 [Core v1.5.0]​

2024-10-16

🚀 Added​

  • Introduced exit_content_group to the exit_page struct.
  • Added all click_ids to last non-direct session attribution logic.
  • Two new columns in the events table.
  • Expanded session_traffic_source_last_click struct to include additional ad tracking fields.
  • Added publisher struct to store publisher-level ad attributes.

🐛 Fixed​

  • Updated default channel grouping logic to replace NULL values with "Direct".

Changed​

  • Improves content categorization for session exits.
  • Refactored configuration settings to be more user-friendly.
  • Updated upstream tables to be compatible with newly introduced columns.
  • Reclassified newsletter traffic under "Email" in default channel grouping.
  • Ensured cpc values labeled as "Other Advertising" rather than "Other".
  • Updated GA4 start date to "2020-01-01" for improved backfilling capabilities.
  • Adjusted transformation dataset names to align with a more structured workflow.

🛠️ [Installer]​

2024-10-07

🚀 Added​

  • Added popup for checking package

🧰 🤝 [Core v1]​

2024-09-01

🐛 Fixed​

  • Resolved an issue with session attribution logic by adjusting lookback window calculations.
  • Corrected logic for determining the last non-direct traffic source.
  • Fixed assertions referencing outdated schema names.
  • Prevented session breakage due to incrementality by ensuring each session ID retains only the first occurrence within a day.
  • Resolved an issue where time.event_date did not exist, replacing references with event_date.
  • Ensured incremental deletion logic uses the correct column reference.
  • Updated dataset references in SQL configurations to use workflow_settings.yaml.
  • Ensured transformation and output datasets are correctly referenced.
  • Resolved compilation issues across multiple files.
  • Ensured validation checks pass successfully.
  • Ensured transaction_id is correctly included in the demo table.
  • Adjusted filtering logic to capture relevant purchase events.
  • Addressed minor technical issues in diagnostic queries.
  • Adjusted assertions schema references.
  • Fixed issues with source and medium classification to improve accuracy.
  • Ensured backward compatibility with older data structures.
  • Resolved duplicate column issue in session data processing.
  • Corrected first_click_ids reference to last_click_ids to align with intended logic.
  • Updated generateClickIdTrafficSourceSQL and generateTrafficSourceSQL functions to properly handle NULL values in traffic source structs.
  • Ensured has_source and is_direct_session fields are always TRUE or FALSE, never NULL.
  • Resolved NULL values appearing in last_non_direct_default_channel_grouping.
  • Resolved inconsistencies in how session source and medium were assigned.
  • Resolved duplicate column issue related to batch_page_id and batch_ordering_id.
  • Corrected classification of organic shopping in channel grouping logic.
  • Ensured column name handling is consistent across custom parameters.
  • Corrected page_number calculation to use batch_page_id ordering.
  • Corrected incremental processing logic by ensuring table_suffix is treated as a date where necessary.
  • Corrected typo in variable name (EVENTS_TO_EXLUDE → EVENTS_TO_EXCLUDE).

🚀 Added​

  • Implemented detection of Measurement Protocol (MP) hits.
  • Introduced session_duration_s to track session lengths.
  • Added descriptions and comments for better maintainability.
  • Introduced ga4_int_events.sqlx to process and transform GA4 event data.
  • Introduced ga4_int_sessions.sqlx for session processing and modeling.
  • Included engagement time in event ID calculations to improve deduplication.
  • Introduced logic to remove duplicate event IDs.
  • Created daily_sessions_report.sqlx for aggregated session insights.
  • Enabled better clustering by adding stream_id as a clustering key.
  • Assertion checks for event ID uniqueness, session duration validity, and table timeliness.
  • Added logging table to store assertion results.
  • Introduced ga4_events.sqlx, a new events table for better tracking without intermediate dependencies.
  • Updated session processing to reference ga4_events instead of ga4_int_events.
  • Implemented a new diagnostics table to track GA4 data quality and anomalies.
  • Added monitoring for self-referrals, duplicate transactions, and empty ecommerce item arrays.
  • Introduced reporting on unique page and traffic source cardinality over time.
  • Added timestamp conversion for local time zone handling.
  • Ensured proper assignment of cpc to paid traffic sources when necessary.
  • Added click_id extraction to improve attribution tracking.
  • Added assertion checks for session duration validity and event ID uniqueness.
  • Introduced new staging files for cleaner data ingestion.
  • Added tagging for assertions to improve tracking and filtering.
  • Introduced new columns: is_active_user, batch_event_index, batch_page_id, batch_ordering_id.
  • Explicitly named all struct fields in collected_traffic_source and items to avoid schema changes breaking incremental builds.
  • Introduced session_traffic_source_last_click in both events and sessions tables.
  • Added last_non_direct_default_channel_grouping for more comprehensive reporting.
  • Added shopping_free as a recognized medium in Organic Shopping classification.
  • Introduced user_properties support for event tracking.
  • Moved source_categories configuration to a JSON file for easier maintenance.
  • Added a package.json file to manage dependencies and versioning.
  • Introduced hostname filters in custom_config.js.
  • Added exclusion and inclusion lists for hostname filtering in event processing.
  • Implemented is_final logic for incremental processing.

Changed​

  • Modified incremental logic to correctly process events without session identifiers.
  • Refactored session attribution logic to ensure correct traffic source tracking.
  • Updated incremental logic to improve event tracking accuracy.
  • Adjusted incremental table configuration for performance improvements.
  • Corrected logic for handling gclid, wbraid, and gbraid in traffic source identification.
  • Replaced hardcoded values with variables in custom_config.js for better flexibility.
  • Improved lookback window calculations for session attribution.
  • Updated misattribution handling logic.
  • Schema changes for int_ga4_events to improve structure.
  • Updated event and session processing structure for better maintainability.
  • Renamed source_categories.source_category to sc.source_category for clarity and consistency.
  • Renamed models to follow consistent naming conventions.
  • Updated reports to reference correct field names.
  • Changed references from stg_ga4_events to ga4_events in session processing queries.
  • Updated function calls to correctly extract event parameters.
  • Improved click_id extraction logic.
  • Updated dataset suffix naming conventions to improve consistency.
  • Adjusted assertion logic to use new dataset variables for better clarity.
  • Improved first and last session attribution logic for traffic sources.
  • Refactored logic to improve handling of click-based attribution.
  • Defaulted missing values to 'Direct' for better reporting.
  • Ensured source_categories are consistently applied using first/last logic for accurate session attribution.
  • Adjusted event processing logic to ensure proper handling of UTM parameters across event batches.
  • Replaced event_date and session_date references with table_suffix for incremental processing.
  • Removed unnecessary lower() function from page_path processing.
  • Renamed variables in configuration to better reflect their actual purpose.
  • Cleaned up unused code blocks and added clarifying comments.
  • Adjusted last non-direct logic to ensure accurate attribution.
  • Updated classification for mobile push notifications to Mobile Push Notifications.
  • Ensures older events are only updated when necessary, reducing unnecessary recomputation.
  • Ensured shopping_free medium and Shopping Free Listings campaigns are classified properly.
  • Moved session logic into int_ga4_sessions for better modularity.
  • Refactored last_non_direct_traffic_source fields and added session_traffic_source_last_click fields.
  • Updated delete statement to use parse_date('%Y%m%d', table_suffix) instead of date(table_suffix).
  • Updated generateParamsSQL to properly handle user_properties in event processing.
  • Replaced temporary deduplication column with direct row_number() approach.
  • Standardized date_checkpoint declaration across multiple queries.
  • Renamed session columns for clarity and consistency.
  • Moved page parameters into a structured page field.
  • Applied LOWER() function to standardize traffic source data.
  • Converted date(table_suffix) to parse_date('%Y%m%d', table_suffix) for consistency and improved performance.

Deprecated​

  • Deprecated older logic for event and session processing, replacing it with a streamlined approach.
  • Deprecated package.json in favor of workflow_settings.yaml
  • Deleted stg_ga4_sessions table.

🛠️ [Installer]​

2024-09-01

🚀 Added​

  • Added link to doc

🛠️ [Installer]​

2024-08-01

🚀 Added​

  • Added check repository state during install process

🧰 🤝 [Core v0]​

2024-03-10

🌟 Init by Artem Korneev​