Web Signals Crawler

A proof-of-concept domain crawler that measures advertising-related signals for a list of domains and produces one aggregated row of metrics per domain. For each domain it visits the homepage and up to four additional internal pages. On each page it simulates lightweight user behaviour (page load, initial viewport inspection, one or two downward scrolls) and then dwells on the page for approximately 45 seconds total, taking a snapshot every 5 seconds. This dwell time gives display ads sufficient time to refresh, making the refresh-rate estimate more reliable. Ad-related signals are extracted from the DOM at each snapshot using injected JavaScript.

---

Metrics computed

A2CR (Ad-to-Content Ratio)

A2CR = visible_ad_area / visible_content_area

Measured at each snapshot (initial viewport, after scroll 1, after scroll 2 if the page is long enough).

What counts as an ad element:

• All <iframe> elements (the vast majority of display ads are served in iframes).
• Any element whose id, class, name, or data-* attributes contain substrings like advertisement, advert, adsense, prebid, doubleclick, dfp, adunit, adslot, ad-slot, ad-unit, ad-container, ad-wrapper, ad-banner.
• Word-boundary matches for short tokens: \bad\b, \bads\b, \bbanner\b, \bsponsor\b, \bgpt\b.
• Elements smaller than 100 px² are excluded (tracking pixels, invisible markers).
• Ancestor deduplication: only outermost matching elements count, so nested <div class="ad-wrapper"><iframe ...> contributes once, not twice.

What counts as content area: Sum of visible areas of p, h1–h6, article, main, section elements within the current viewport. Falls back to window.innerWidth × window.innerHeight when no content elements are visible.

Domain-level aggregation:

• avg_a2cr: mean of per-page average A2CRs.
• max_a2cr: highest single-snapshot A2CR observed across all pages.

ads_in_view

At each snapshot, the count of outermost ad elements (see above) that intersect the current viewport. Values are averaged and maxed at the page level, then averaged and maxed at the domain level.

domain_avg_refresh_rate

Heuristic: The crawler observes each page in two phases:

1. Movement phase — initial viewport + up to two scrolls (3 snapshots, ~5 s total).
2. Dwell phase — the page is held open for the remainder of the 45-second budget; a snapshot is taken every 5 seconds (~8 additional snapshots).

At each snapshot the JavaScript captures the .src of every iframe that qualifies as an ad. Between consecutive snapshots, any iframe slot whose src changed (and was non-empty in both snapshots) is counted as one refresh event. This is a positional comparison — the same DOM-order index implies the same ad slot.

page_avg_refresh_rate = total_refresh_events / max(num_snapshots - 1, 1)

domain_avg_refresh_rate = mean(page_avg_refresh_rate across all pages)

The 45-second window is configurable (total_time_per_page_seconds). The snapshot interval is also configurable (refresh_snapshot_interval_seconds, default 5 s). The heuristic is intentionally conservative: it only counts src changes on already-loaded iframes, not initial loads.

---

Project structure

web-signals-crawler-v2/
├── main.py                    # CLI entry point
├── requirements.txt
├── README.md
├── domains_example.txt        # Sample input
├── page_observation_example.json
├── IMPLEMENTATION_NOTES.md
├── crawler/
│   ├── __init__.py
│   ├── config.py              # Config dataclass + env var overrides
│   ├── models.py              # Pydantic models (Snapshot, PageObservation, DomainRow, …)
│   ├── discover.py            # URL normalization, link extraction, page scoring
│   ├── extract.py             # JS extraction string + Python metric helpers
│   ├── browser.py             # Playwright lifecycle, snapshot orchestration
│   ├── aggregate.py           # Domain-level aggregation
│   ├── sinks.py               # LocalSink, BigQuerySink, factory function
│   └── utils.py               # run_id generator, logging setup
└── tests/
    ├── __init__.py
    ├── test_url_normalization.py
    └── test_aggregate.py

---

Setup

Requirements: Python 3.11+

# 1. Install Python dependencies
pip install -r requirements.txt

# 2. Install the Chromium browser for Playwright
playwright install chromium

---

How to run

Basic run (uses defaults from domains_example.txt):

python main.py

With options:

python main.py \
  --input domains_example.txt \
  --output-dir output \
  --max-domains 3 \
  --domain-concurrency 2 \
  --page-concurrency 2

Run without headless mode (shows the browser window):

python main.py --no-headless --max-domains 1

Disable BigQuery entirely:

python main.py --disable-bigquery

Available flags:

Flag	Default	Description
--input FILE	domains_example.txt	Newline-delimited domain list
--output-dir DIR	output	Where to write output files
--max-domains N	(all)	Limit domains processed (useful for testing)
--no-headless	headless=True	Show the browser window
--disable-bigquery	BQ enabled	Skip BigQuery; write only to local files
--domain-concurrency N	3	Parallel domains
--page-concurrency N	2	Parallel pages per domain

---

BigQuery behaviour

On startup (unless --disable-bigquery is passed), the crawler:

1. Instantiates a google.cloud.bigquery.Client for the configured project (scope3-prod by default, overridable via BQ_PROJECT env var).
2. Calls client.get_dataset(dataset_ref) to verify the dataset (niki_sandbox) is accessible.
3. Attempts to get the target table (domain_signals_poc); if it does not exist, tries to create it with the expected schema.
4. If all steps succeed, final domain rows are streamed to BigQuery.
5. If any step fails (permission denied, network error, missing package, etc.), the crawler logs a warning and automatically falls back to writing domain rows to output/domain_signals.ndjson.

Raw page observations are always written locally regardless of the BigQuery setting.

Authentication:

gcloud auth application-default login

Or set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of a service account key file.

---

Output files

All files are written to the output/ directory (configurable via --output-dir).

File	Format	Description
page_observations.ndjson	NDJSON (append)	One record per crawled page with all snapshots and metrics
domain_signals.ndjson	NDJSON (append)	One aggregated row per domain
manifest.json	Pretty JSON	Run summary (timing, counts, output mode)

NDJSON files are opened in append mode so partial results accumulate even if the run is interrupted.

---

Limitations of this PoC

• Ad detection is heuristic-only. The crawler detects elements based on class/id name patterns and iframes. It will miss ads served via shadow DOM or non-standard markup, and may over-count promotional content that uses the word "banner" in its CSS class.
• Refresh detection is approximate. Only iframe src changes are tracked; ads that refresh via JavaScript DOM replacement without changing an iframe src are not detected.
• No JavaScript rendering fallback. Pages that require significant JS interaction beyond basic scrolling (SPAs with lazy navigation, paywalls, cookie consent modals) may produce lower-quality observations.
• No rate limiting or politeness delays between pages. This PoC is not suitable for large-scale crawling without adding per-domain delays and robots.txt checking.
• Single-machine only. The concurrency model uses asyncio + Playwright within a single Python process. There is no distributed task queue.
• Viewport fingerprinting. Using a fixed desktop UA and viewport may trigger bot-detection on some sites, causing zero-ad observations even on ad-heavy pages.

---

Running tests

pytest tests/

Tests cover URL normalization, internal link filtering, page scoring/selection, snapshot metric computation, and domain-level aggregation. No Playwright is required to run the tests.