Skip to content

SKILL.md

Latest commit

 

History

History
124 lines (90 loc) · 4.88 KB

SKILL.md

File metadata and controls

124 lines (90 loc) · 4.88 KB
Raw
name web-to-pdf
description Use when the user wants to scrape a website, tutorial, or documentation and convert it into a clean, bookmarked A4 PDF. Triggers on requests like "save this site as PDF", "download tutorial as PDF", "convert web pages to PDF", "export docs to PDF". Handles multi-page tutorials with navigation sidebar, SVG diagrams with CJK fonts, ad removal, and bookmark generation.

Web to PDF

Convert multi-page web tutorials/docs into a single, clean, bookmarked A4 PDF.

Overview

Scrapes all pages from a web tutorial site, strips ads/navigation, fixes SVG font issues, combines into one styled HTML, then renders to a bookmarked A4 PDF using headless Chromium.

When to Use

  • User provides a URL and wants it saved as PDF
  • Multi-page tutorial/documentation sites (e.g., runoob.com, w3schools, MDN)
  • User wants ads removed, only article content kept
  • User needs bookmarks/outline in the PDF
  • SVG diagrams contain CJK text that may lose fonts in conversion

Core Workflow

1. Discover pages  →  Fetch index page, extract sidebar/nav links
2. Fetch & clean   →  For each page: download HTML, extract article, remove ads
3. Fix SVGs        →  Download SVGs, replace fonts, inline into HTML
4. Combine HTML    →  Cover + TOC + all chapters, A4 CSS, page breaks
5. Render PDF      →  Playwright (headless Chromium) → page.pdf()
6. Add bookmarks   →  Calculate chapter positions → pypdf adds outline

Implementation

Use the bundled web_to_pdf.py script in this skill directory.

Dependencies (auto-installed into a venv)

requests beautifulsoup4 playwright pypdf

System requirement: brew install pango (for fallback; Playwright/Chromium is primary)

Chromium install: playwright install chromium

Usage

# The script is at:
~/.claude/skills/web-to-pdf/web_to_pdf.py

# Basic usage:
python3 web_to_pdf.py <start_url> [--output path.pdf]

# Example:
python3 web_to_pdf.py https://www.runoob.com/claude-code/claude-code-tutorial.html \
  --output ~/Claude_Code_Tutorial.pdf

Key Decisions

Decision Choice Why
PDF renderer Playwright (Chromium) Only engine that renders SVG foreignObject + CJK correctly
NOT WeasyPrint WeasyPrint loses SVG foreignObject text Mermaid diagrams use foreignObject for labels
SVG handling Download → replace fonts → inline <img src=".svg"> can't access system fonts; inlining fixes this
CJK font fallback PingFang SC, Heiti SC, Hiragino Sans GB macOS built-in; covers zh-CN/zh-TW
Bookmark generation Playwright renders → pypdf adds outlines Chromium PDF export has no bookmark API; post-process with pypdf
Page estimation element.top / (totalHeight / totalPages) Proportional mapping; accurate within 1 page

Content Extraction Strategy

  1. Find main content: #content > .article-body > article > div with "content"/"article" class
  2. Remove by selector: script, iframe, .adsbygoogle, #sidebar, footer, nav, .share-bar, #comments
  3. Remove by pattern: class/id matching ad[s_-], sponsor, promo, google
  4. Remove hidden: style="display:none"
  5. Remove HTML comments

SVG Font Fix

SVGs from sites like runoob.com use Mermaid.js which specifies:

font-family: "Comic Neue", "Comic Sans MS", "Noto Sans SC", cursive, sans-serif;

These fonts are usually missing on developer machines. Fix:

svg_text = re.sub(
    r'"Comic Neue".*?sans-serif',
    '"PingFang SC", "Heiti SC", "Hiragino Sans GB", sans-serif',
    svg_text
)

Then replace <img src="x.svg"> with inline <svg>...</svg>.

Bookmark Calculation

Chromium's page.pdf() doesn't support outlines. Workaround:

  1. Before PDF generation, run JS to get each chapter heading's Y offset
  2. After PDF generation, calculate: page_number = int(y_offset / (total_height / total_pages))
  3. Use pypdf.PdfWriter.add_outline_item(title, page_number) to inject bookmarks

Common Mistakes

Mistake Fix
Using WeasyPrint for SVG-heavy pages Switch to Playwright; WeasyPrint can't render foreignObject
Not waiting for images to load Add wait_until="networkidle" + extra timeout
SVG text disappearing in PDF Inline SVGs and replace font-family with system fonts
Bookmarks all pointing to page 1 Don't trust Chromium named destinations; use proportional calculation
Rate limiting / 403 errors Add User-Agent header, 0.5s delay between requests
Encoding issues Force resp.encoding = 'utf-8' before parsing

Adapting for Other Sites

The script's content extraction is generic but may need tuning:

  1. Different content selectors: Modify CONTENT_SELECTORS list in the script
  2. Different nav structure: Override discover_pages() for sites without sidebar nav
  3. Different ad patterns: Extend AD_SELECTORS list
  4. Non-CJK sites: Font replacement can be skipped (set --no-font-fix)