GCP analytics pipeline with BigQuery, dbt, and Cloud Run

End-to-end GCP analytics pipeline: GCS landing zone -> BigQuery raw dataset -> dbt star schema (stg/ + marts/) -> Cloud Run service that runs dbt build on demand -> Airflow that calls Cloud Run on schedule -> Cloud Build CI/CD -> Terraform-managed infrastructure.

This is the GCP analytics track of the learnwithparam.com data engineering catalogue. The companion AWS workshops are aws-glue-spark-etl (a focused PySpark + Glue job) and aws-lakehouse-iceberg-snowflake (a full lakehouse with Iceberg + Snowflake).

Start the course: learnwithparam.com/courses/gcp-bigquery-dbt-pipeline Continue into the full program: learnwithparam.com/data-engineering-bootcamp

What you'll build

By the end of the workshop you will have:

• A GCS landing layer for raw CSVs, with the bundled Olist Brazilian e-commerce sample as the working dataset
• A BigQuery raw dataset loaded from GCS via an external table or scheduled load
• A dbt project with stg/ (cleaned, typed, deduped) and marts/ (star schema: dim_customers, dim_orders, dim_products, dim_sellers, fact_order_items, fact_order_payments) plus four BI-ready aggregated views
• A Cloud Run service (cloud_run_dbt/app.py) — Flask app that exposes POST /run_transformation to trigger dbt source freshness + dbt build
• An Airflow DAG that calls the Cloud Run endpoint on schedule and waits for completion
• A Cloud Build pipeline (cloud_run_dbt/cloudbuild.yml) that rebuilds and redeploys the Cloud Run image whenever you push to main
• A Terraform configuration that provisions GCS, IAM, service accounts, and Cloud Build triggers

Architecture

                      ┌─────────────────────┐
                      │  Source CSVs (Olist) │
                      └──────────┬──────────┘
                                 │ gsutil cp
                                 ▼
                      ┌─────────────────────┐
                      │   GCS raw bucket    │
                      └──────────┬──────────┘
                                 │
                                 ▼
                      ┌─────────────────────┐
                      │  BigQuery raw set   │   ecommerce_raw.olist_*
                      └──────────┬──────────┘
                                 │
                Airflow ──▶ POST /run_transformation
                                 │
                                 ▼
              ┌────────────────────────────────────┐
              │  Cloud Run dbt service             │
              │  cloud_run_dbt/app.py (Flask)      │
              │  └─ dbt source freshness + build   │
              └──────────┬─────────────────────────┘
                         │
                         ▼
              ┌────────────────────────────────────┐
              │  BigQuery analytics datasets       │
              │   ecommerce_stg, ecommerce_marts   │
              │   star schema + agg views          │
              └────────────┬───────────────────────┘
                           │
                           ▼
                ┌────────────────────────┐
                │  BI: Looker / Sheets   │
                └────────────────────────┘

Tech

• Python 3.11, uv for env management
• dbt 1.10 with the BigQuery adapter
• dbt-duckdb for CI / smoke validation (no GCP creds required)
• Cloud Run + Flask for the on-demand dbt runner
• Apache Airflow 2.10 locally via Docker Compose for orchestration
• Cloud Build for CI/CD on main
• Terraform for GCP infrastructure (GCS, IAM, service accounts, Cloud Build triggers)

Quick start

# 1. Setup env
make setup

# 2. Run the smoke test (no GCP creds required)
make smoke

# 3. Run the Cloud Run dbt service locally on http://localhost:8080
make cloud-run-local

# 4. Spin up local Airflow on http://localhost:8081
make airflow-up

# 5. Statically validate Terraform
make terraform-validate

When you are ready to deploy to GCP, set GCP_PROJECT_ID, BQ_DATASET, BQ_RAW_DATASET, GCS_RAW_BUCKET in .env, then cd terraform && terraform init && terraform apply -var "gcp_project_id=$GCP_PROJECT_ID".

Smoke test

make smoke

The smoke test never touches GCP. It validates that:

• uv sync succeeds and every Python file parses
• All dbt + Cloud Build YAML files are syntactically valid
• dbt parse --target ci succeeds (validates the project graph + macros + var resolution against a duckdb shim profile)
• cloud_run_dbt/app.py boots and /health returns {"status": "ok"}
• terraform validate passes (skipped if terraform is not on PATH)
• The repo contains zero hard-coded references to upstream GCP project ids

Repository structure

gcp-bigquery-dbt-pipeline/
├── cloud_run_dbt/                 ← Cloud Run dbt runner
│   ├── app.py                     ← Flask app: /, /health, /run_transformation
│   ├── Dockerfile
│   ├── cloudbuild.yml             ← Cloud Build → Cloud Run deploy
│   ├── requirements.txt
│   └── dbt/
│       ├── dbt_project.yml
│       ├── profiles.yml           ← dev (BigQuery) + ci (DuckDB) targets
│       └── models/
│           ├── stg/               ← cleaned + deduped staging models
│           └── marts/             ← star schema + BI aggregates
├── airflow/                       ← Local Airflow stack (docker-compose)
│   ├── docker-compose.yml
│   ├── Dockerfile
│   ├── requirements.txt
│   └── dags/
├── terraform/                     ← GCS / IAM / service accounts / Cloud Build triggers
│   ├── provider.tf
│   ├── variables.tf
│   ├── gcs.tf
│   ├── iam_service_account.tf
│   ├── cloud_build.tf
│   └── outputs.tf
├── datasets/
│   └── olist_ecommerce_dataset/   ← Subsampled Olist CSVs (~500 rows each)
├── pyproject.toml                 ← uv-managed dependencies
├── Dockerfile                     ← python:3.11-slim + uv (workshop dev container)
├── docker-compose.yml             ← workshop dev container running Cloud Run app
├── Makefile                       ← setup / smoke / dbt-parse / airflow-up / terraform-validate
├── smoke_test.sh
├── CLAUDE.md
└── README.md

Deploy to GCP

Once make smoke is green and gcloud auth login && gcloud auth application-default login are done:

# 1. Apply infrastructure
cd terraform
terraform init
terraform apply -var "gcp_project_id=$GCP_PROJECT_ID"

# 2. Upload the raw CSVs to GCS
gsutil -m cp -r datasets/olist_ecommerce_dataset/*.csv gs://${GCS_RAW_BUCKET}/raw/

# 3. Load the CSVs into the raw BigQuery dataset (one-time bootstrap)
for f in datasets/olist_ecommerce_dataset/*.csv; do
  table=$(basename "$f" .csv)
  bq load --autodetect --replace \
    --source_format=CSV \
    "${GCP_PROJECT_ID}:${BQ_RAW_DATASET}.${table}" \
    "gs://${GCS_RAW_BUCKET}/raw/${table}.csv"
done

# 4. Push to main — Cloud Build rebuilds and redeploys the Cloud Run service
git push origin main

# 5. Trigger the dbt run
curl -X POST -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  "${CLOUD_RUN_DBT_URL}/run_transformation" -d '{}'

Airflow then takes over the daily refresh.

Progression

After this workshop, look at:

• aws-lakehouse-iceberg-snowflake — the same end-to-end shape on AWS with Iceberg + Snowflake
• aws-glue-spark-etl — focused PySpark + Glue ETL pattern
• end-to-end-data-pipeline — a 20-service production data platform that includes streaming + ML observability

Learn more

• Start the course: learnwithparam.com/courses/gcp-bigquery-dbt-pipeline
• Data Engineering Bootcamp: learnwithparam.com/data-engineering-bootcamp
• All courses: learnwithparam.com/courses

License

MIT. See LICENSE.