compression-side-information-priors

Extending the Wyner-Ziv / Sgarro framework to learned generative priors as decoder side information.

Overview

This research package extends the classical Wyner-Ziv distributed lossy compression framework to the case where the decoder's side information is not a correlated signal but a learned generative prior — a model that approximates the true source distribution to within a Kullback-Leibler divergence δ.

The central question: How much does prior approximation error cost in compression rate?

Key Results

Theorem 1 — Mutual-Information Degradation

A learned prior with KL divergence δ (nats) from the true source provides side information whose mutual information is reduced by at most δ/ln(2) bits:

I(X; S_learned) ≤ I(X; S_true) − δ/ln(2)

Each nat of approximation error costs ~1.443 bits of side-information quality.

Theorem 2 — Gaussian Rate-Distortion with Learned Prior

For a Gaussian source X ~ N(0, σ²) with Gaussian side information S = X + N(0, σ_S²), the learned prior inflates the effective side-information noise:

σ_S_eff² = σ_S² + σ² × (1 − e^(−2δ))

The achievable rate becomes:

R(D | S_learned) = max{0, ½·log₂(σ²/D) − ½·log₂(1 + σ²/σ_S_eff²)}

and the bitrate savings are:

Savings(δ) = ½·log₂(1 + σ²/σ_S_eff²)

Limiting cases:

• δ → 0: recovers the classical Wyner-Ziv rate (perfect prior)
• δ → ∞: recovers the no-side-information rate R(D) (useless prior)

Theorem 3 — Optimal Prior-Investment Tradeoff

If improving the prior from δ to δ' < δ costs B(δ) = B₀ + b·log₂(1/δ) bits, the total rate R_total(δ) = B(δ) + R(D|S_learned(δ)) has a unique minimizer δ* satisfying:

b / (δ* · ln 2) = d/dδ R(D | S_learned(δ)) |_{δ=δ*}

The optimal investment balances the marginal cost of a better prior against the marginal savings in compressed bitrate.

Installation

git clone git@github.com:drwjkirkpatrick-web/compression-side-information-priors.git
cd compression-side-information-priors
pip install -e .

Usage

from compression_side_information_priors import (
    GaussianWynerZiv, LearnedPriorModel, PriorOptimizer,
    rate_savings_learned_prior, simulate_wyner_ziv_gaussian,
)

# Classical Wyner-Ziv: perfect side information
wz = GaussianWynerZiv(sigma_x2=1.0, sigma_s2=0.5, D=0.1)
print(f"Classical rate: {wz.R_wz_classical():.4f} bits")

# With a learned prior (delta = 0.5 nats)
wz_learned = GaussianWynerZiv(
    sigma_x2=1.0, sigma_s2=0.5, D=0.1,
    prior=LearnedPriorModel(delta=0.5),
)
print(f"Learned rate:   {wz_learned.R_wz_learned():.4f} bits")
print(f"Savings:        {wz_learned.savings():.4f} bits")

# Optimal prior investment
opt = PriorOptimizer(sigma_x2=1.0, sigma_s2=0.5, D=0.1, B0=0.0, b=1.0)
delta_star, R_star = opt.optimize()
print(f"Optimal delta:  {delta_star:.4f}")
print(f"Optimal rate:   {R_star:.4f} bits")

# Numerical verification
results = simulate_wyner_ziv_gaussian(n_samples=200_000)

Project Structure

compression-side-information-priors/
├── compression_side_information_priors/
│   ├── __init__.py          # Public API exports
│   └── core.py              # Core theory & implementation
├── tests/
│   └── test_core.py         # 30+ tests (all pass)
├── proofs/
│   ├── main_theorem.tex     # LaTeX proof (3 theorems)
│   └── main_theorem.pdf     # Compiled PDF
├── pyproject.toml
├── requirements.txt
├── .gitignore
└── README.md

Testing

pytest tests/ -v

All tests pass, covering:

• Classical Wyner-Ziv formulas (Gaussian MI, conditional variance, rate bounds)
• KL divergence for Gaussians
• Learned-prior effective noise and mutual information
• Rate savings and monotonicity in δ
• Joint source-channel coding feasibility
• Successive refinement with generative base layer
• Optimal prior-investment convexity and optimum
• Monte-Carlo simulation vs closed-form (MMSE, MI, quantizer)

Mathematical Framework

Classical Wyner-Ziv

R(D|S) = inf_{p(Y|X): E[d]≤D} I(X;Y) − I(X;S)

The decoder exploits side information S to save I(X;S) bits per sample.

Extension to Learned Priors

When S is replaced by a learned prior with KL error δ:

• Side information quality degrades by δ/ln(2) bits
• For Gaussian sources, this maps to an effective noise inflation
• Rate savings become a function of δ, monotonically decreasing
• There is an optimal δ* trading prior quality against compressed rate

License

MIT

Author

Walker Kirkpatrick