Product Architecture
We organize files and directories into two distinct buckets: core and custom. When we release new versions of GA4Dataform, the installer only updates the contents of the core directories, leaving custom untouched. We designed this approach to prevent accidental overwrites of any customizations you've made after the initial installation.
File and Directory Management
Do not add new files to the core directory or modify existing core files directly. Any changes made to core may be lost during updates. Place your custom files outside the core directory.
You can safely add new directories anywhere in the repository outside of core. We recommend placing new directories within the custom directory structure.
For example:
To add a new reporting directory, create it as: custom/reporting
Dataform Directories
| Directory | Description |
|---|---|
| definitions | Contains all directories and files related to building models |
| core/01_sources | Contains declarations.js and (future) staging models |
| core/02_intermediate | Contains intermediate models |
| core/03_outputs | Contains output models that should be used for downstream queries |
| core/assertions | Contains all the assertions that check the data quality of our model |
| core/extra | Contains any extra files that fulfill an individual purpose |
| core/modules 💎 | Contains modules (currently only Premium) |
| core/modules/bq_cost_monitoring | Contains BigQuery cost analysis models with intermediate processing and comprehensive reporting |
| core/modules/cvr_per_event | Contains conversion rate analysis models for event-level performance metrics |
| core/modules/ga4_parameter_detection | Contains automated parameter discovery models with configuration suggestions |
| core/modules/items_funnel_report | Contains e-commerce funnel analysis models with session-level item tracking |
| core/utility | Contains utility operations for maintenance tasks like label management |
| definitions/custom | Contains all the custom models that are not part of the core package |
| includes | Contains all JS files with reusable variables and functions that help manage the repository |
| includes/core/documentation | Contains the JSON files of table fields and descriptions of the output tables |
| includes/core/extra | Contains all extra files that fulfill an individual purpose |
| includes/core/modules 💎 | Contains configuration files and helpers for premium module functionality |
| includes/custom | Contains all JS files that can be used to customize your setup (config.js) |
| includes/custom/modules | Contains customizable configuration files for premium modules (JSON/YAML formats) |
Model Descriptions
| Model | Description |
|---|---|
| int_ga4_sessions | GA4 intermediate sessions table that incrementally queries ga4_events table and creates session-level dimensions and metrics |
| int_ga4_transactions | GA4 intermediate transactions table that incrementally queries ga4_events table and prepares the needed dimensions and metrics |
| int_info_schema_jobs 💎 | Intermediate table processing BigQuery INFORMATION_SCHEMA.JOBS for cost monitoring and query analysis |
| int_info_schema_table_options 💎 | Intermediate table processing table metadata and options for cost tracking |
| int_info_schema_table_storage_usage_timeline 💎 | Intermediate table tracking storage usage over time for cost optimization |
| int_tables_labels 💎 | Intermediate table managing table labels for cost allocation and organization |
| ga4_events | GA4 output events table that incrementally queries the raw GA4 export and applies partitioning, clustering, cleaning, and several fixes |
| ga4_sessions | GA4 output sessions table that adds last non-direct click attribution and can be used for further transformations or aggregations |
| ga4_transactions | GA4 output transactions table that holds nested items, transaction totals and running totals from purchase and refund events |
| bq_cost_overview 💎 | BigQuery cost overview table providing comprehensive cost analysis across projects and datasets |
| bq_cost_processing 💎 | BigQuery processing cost table tracking query execution costs and resource usage |
| bq_cost_reporting 💎 | BigQuery cost reporting table with aggregated cost metrics for dashboards |
| bq_cost_storage 💎 | BigQuery storage cost table monitoring data storage costs and trends |
| info_schema_jobs_queries 💎 | Query-level job information table for detailed query performance and cost analysis |
| info_schema_jobs_scripts 💎 | Script-level job information table tracking batch operations and scheduled queries |
| report_cvr_per_event 💎 | Conversion rate analysis table providing event-level performance metrics and optimization insights |
| ga4_parameter_detection_agg 💎 | Aggregated parameter detection table that aggregates information from the ga4_parameter_detection_daily table |
| ga4_parameter_detection_config_suggestions 💎 | Configuration suggestions view that returns a config.js compatible JavaScript object with discovered parameters |
| ga4_parameter_detection_daily 💎 | Daily parameter detection table that queries from the raw GA4 export to monitor parameter usage patterns and trends |
| ga4_items_sessions 💎 | Items-level session table aggregating product data per session for e-commerce analysis |
| report_items_funnel 💎 | E-commerce funnel report table tracking product metrics from view to purchase |
| update_labels | Utility table managing label updates and maintenance operations across datasets |
| demo_daily_sessions_report | Demo daily session aggregate table that can be connected to Looker Studio for reporting |
| demo_diagnostics | Demo diagnostics table that checks for several issues in the past 64 days |
| source_categories | Materializes the core/extra/source_categories.json file into a table. Used for Default Channel Grouping |
JavaScript Files
| File | Description |
|---|---|
| core/default_config.js | Contains default configuration options that are used as a fallback if custom/config.js is not populated |
| core/helpers.js | Contains all helper functions that are used to produce SQL code for different use cases |
| core/documentation/helpers.js | Contains helper functions for generating and maintaining table documentation |
| custom/config.js | Contains all configuration options that can be used to customize how and what data gets queried. It will always take precedence over core/default_config.js |
| core/extra/source_categories.json | Contains which source category a domain should be treated as. Used for Default Channel Grouping |
Dataform Repository Structure
definitions/
├── core/
│ ├── 01_sources/
│ │ └── declarations.js
│ ├── 02_intermediate/
│ │ ├── int_ga4_sessions.sqlx
│ │ └── int_ga4_transactions.sqlx
│ ├── 03_outputs/
│ │ ├── ga4_events.sqlx
│ │ ├── ga4_sessions.sqlx
│ │ └── ga4_transactions.sqlx
│ ├── assertions/
│ │ ├── assertion_logs.sqlx
│ │ ├── assertions_event_id_uniqueness.sqlx
│ │ ├── assertions_session_duration_validity.sqlx
│ │ ├── assertions_session_id_uniqueness.sqlx
│ │ ├── assertions_sessions_validity.sqlx
│ │ ├── assertions_tables_timeliness.sqlx
│ │ ├── assertions_transaction_id_completeness.sqlx
│ │ └── assertions_user_pseudo_id_completeness.sqlx
│ ├── extra/
│ │ └── ga4/
│ │ └── source_categories.js
│ ├── modules/ 💎
│ │ ├── bq_cost_monitoring/
│ │ │ ├── 02_intermediate/
│ │ │ │ ├── int_info_schema_jobs.sqlx
│ │ │ │ ├── int_info_schema_table_options.sqlx
│ │ │ │ ├── int_info_schema_table_storage_usage_timeline.sqlx
│ │ │ │ └── int_tables_labels.sqlx
│ │ │ └── 03_outputs/
│ │ │ ├── bq_cost_overview.sqlx
│ │ │ ├── bq_cost_processing.sqlx
│ │ │ ├── bq_cost_reporting.sqlx
│ │ │ ├── bq_cost_storage.sqlx
│ │ │ ├── info_schema_jobs_queries.sqlx
│ │ │ └── info_schema_jobs_scripts.sqlx
│ │ ├── cvr_per_event/
│ │ │ └── report_cvr_per_event.sqlx
│ │ ├── ga4_parameter_detection/
│ │ │ ├── ga4_parameter_detection_agg.sqlx
│ │ │ ├── ga4_parameter_detection_config_suggestions.sqlx
│ │ │ └── ga4_parameter_detection_daily.sqlx
│ │ └── items_funnel_report/
│ │ ├── ga4_items_sessions.sqlx
│ │ └── report_items_funnel.sqlx
│ └──utility/
│ └── update_labels.sqlx
└── custom/
├── demo_daily_sessions_report.sqlx
└── demo_diagnostics.sqlx
includes/
├── core/
│ ├── documentation/
│ │ ├── ga4_events.json
│ │ ├── ga4_sessions.json
│ │ ├── ga4_transactions.json
│ │ ├── helpers.js
│ │ └── modules/ 💎
│ │ ├── bq_cost_monitoring/
│ │ │ ├── bq_cost_overview.json
│ │ │ ├── bq_cost_processing.json
│ │ │ ├── bq_cost_reporting.json
│ │ │ ├── bq_cost_storage.json
│ │ │ ├── info_schema_jobs_queries.json
│ │ │ └── info_schema_jobs_scripts.json
│ │ ├── cvr_per_event/
│ │ │ └── report_cvr_per_event.json
│ │ ├── ga4_parameter_detection/
│ │ │ ├── ga4_parameter_detection_agg.json
│ │ │ └── ga4_parameter_detection_daily.json
│ │ └── items_funnel_report/
│ │ ├── ga4_items_sessions.json
│ │ └── report_items_funnel.json
│ ├── extra/
│ │ └── source_categories.json
│ ├── default_config.js
│ └── helpers.js
├── custom/
│ ├── modules/ 💎
│ │ ├── bq_cost_monitoring/
│ │ │ └── config.json
│ │ ├── cvr_per_event/
│ │ │ └── config.json
│ │ ├── ga4_parameter_detection/
│ │ │ └── config.yaml
│ │ └── items_funnel_report/
│ │ └── config.json
│ └── config.js
├── .gitignore
├── package-lock.json
├── package.json
└── workflow_settings.yaml
BigQuery Output
GA4Dataform produces tables to 3 datasets in BigQuery.
superform_outputs_123456: used for storing the output tables that should be used for downstream queries superform_quality_123456: used for storing the quality control results (assertions) superform_transformations_123456: used for storing the intermediate and staging tables that are used during the build process
If you leave the default dataset names untouched and enable all modules, you will see the following structure:
GCP project
├── superform_outputs_123456 (dataset)
│ ├── demo_daily_sessions_report (tables)
│ ├── demo_diagnostics
│ ├── ga4_events
│ ├── ga4_sessions
│ ├── ga4_transactions
│ ├── bq_cost_overview 💎
│ ├── bq_cost_processing 💎
│ ├── bq_cost_reporting 💎
│ ├── bq_cost_storage 💎
│ ├── info_schema_jobs_queries 💎
│ ├── info_schema_jobs_scripts 💎
│ ├── report_cvr_per_event 💎
│ ├── ga4_parameter_detection_agg 💎
│ ├── ga4_parameter_detection_config_suggestions 💎
│ ├── ga4_parameter_detection_daily 💎
│ ├── ga4_items_sessions 💎
│ └── report_items_funnel 💎
│
├── superform_quality_123456
│ ├── assertion_logs
│ ├── assertions_event_id_uniqueness
│ ├── assertions_session_duration_validity
│ ├── assertions_session_id_uniqueness
│ ├── assertions_sessions_validity
│ ├── assertions_tables_timeliness
│ ├── assertions_transaction_id_completeness
│ └── assertions_user_pseudo_id_completeness
│
└── superform_transformations_123456
├── int_ga4_sessions
├── int_ga4_transactions
├── source_categories
├── int_info_schema_jobs 💎
├── int_info_schema_table_options 💎
├── int_info_schema_table_storage_usage_timeline 💎
└── int_tables_labels 💎