feat(config): move from hydra to pydantic models

CLAUDE.md

@@ -0,0 +1,109 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+**gentropy** is Open Targets' Python framework for post-GWAS analysis. It harmonizes, statistically analyzes, and prioritizes genetic signals to assist drug discovery. The codebase is built on **PySpark** with **Hail** for genomic data processing, using **Hydra** for configuration management.
+
+## Key Architecture
+
+### Core Abstractions
+
+- **`Dataset`** (`src/gentropy/dataset/dataset.py`) - Abstract base class wrapping a PySpark DataFrame with schema validation. All dataset types inherit from this. Provides `read()`, `from_parquet()`, `filter()`, `validate_schema()`, `valid_rows()` (QC filtering), and partitioning utilities.
+
+- **`Session`** (`src/gentropy/common/session.py`) - SparkSession wrapper with custom config (write mode, output partitions, Hail setup, BGZIP codec, dynamic allocation). Use `Session.find()` to get the active session.
+
+- **`config.py`** (`src/gentropy/config.py`) - Hydra config store with dataclass-based step configurations. All steps are registered via `register_config()`. Run `gentropy.cli:main` via Hydra to execute steps.
+
+- **CLI** (`src/gentropy/cli.py`) - Entry point `gentropy` command. Uses `@hydra.main` decorator with `instantiate(cfg.step)` to run the configured pipeline step.
+
+### Directory Structure
+
+```
+src/gentropy/
+  cli.py                    # CLI entry point (Hydra)
+  config.py                 # All step configurations (dataclasses)
+  common/
+    session.py              # SparkSession wrapper (core runtime)
+    schemas.py              # Schema validation utilities
+    spark.py, udf.py, stats.py, genomic_region.py
+  dataset/                  # Dataset types (DataFrame wrappers)
+    dataset.py              # Base Dataset ABC
+    study_locus.py, summary_statistics.py, variant_index.py
+    study_index.py, colocalisation.py, l2g_*.py, molecular_complex.py
+    l2g_features/           # Locus-to-gene feature engineering
+  datasource/               # External data sources (FINNGEN, gnomAD, GWAS Catalog, etc.)
+  method/                   # Algorithms (finemapping, clumping, QC, colocalisation)
+    colocalisation/, l2g/   # Subdirectories for complex methods
+  external/                 # Cloud storage (GCS, S3)
+  assets/                   # Schemas, data files, log4j config
+```
+
+Top-level Python files (`pics.py`, `finngen_studies.py`, `susie_finemapper.py`, etc.) are **Step classes** - each orchestrates a pipeline stage by reading datasets, applying methods, and writing results.
+
+### Locus-to-Gene (L2G)
+
+The L2G system (`src/gentropy/l2g.py`, `src/gentropy/method/l2g/`) predicts gene-causal relationships using XGBoost models. Key files:
+- `LocusToGeneFeatureMatrixStep` - builds feature matrices from credible sets
+- `LocusToGeneStep` - trains/evaluates models
+- `LocusToGeneEvidenceStep` / `LocusToGeneAssociationsStep` - generates evidence/associations output
+
+## Development Commands
+
+### Setup
+```bash
+uv sync                    # Install dependencies
+uv run pre-commit install  # Install pre-commit hooks
+```
+
+### Linting & Formatting
+```bash
+make check                 # Run ruff + pydoclint
+uv run ruff check src/gentropy .   # Lint only
+uv run ruff format src/gentropy .  # Format only
+uv run pydoclint --config=pyproject.toml src  # Docstring lint
+```
+
+### Type Checking
+```bash
+uv run mypy src/gentropy   # Semistrict mypy (see pyproject.toml [tool.mypy])
+```
+
+### Tests
+```bash
+make test                  # Full test suite (combined coverage)
+uv run pytest -m 'not download_jars_from_web and not no_shared_spark'  # Default run
+uv run pytest tests/gentropy/dataset/ -v --no-cov  # Single module, no coverage
+uv run pytest tests/gentropy/dataset/test_study_locus.py -v --no-cov -k test_valid_rows  # Single test
+make test-no-shared-spark-session   # Tests isolated from shared SparkSession
+```
+
+Test markers: `step_test`, `download_jars_from_web`, `no_shared_spark`
+
+### Build & Documentation
+```bash
+make build                 # Build Python package (uv build)
+make build-documentation   # Start local mkdocs server (uv run mkdocs serve)
+make build-docker          # Build Docker image
+```
+
+### Pre-commit Hooks
+The repo uses pre-commit with: ruff, ruff-format, mypy, interrogate, pydocstyle, pydoclint, yamllint, commitlint (conventional commits), uv-lock check. Run `pre-commit run --all-files` to validate.
+
+### Post-Change Verification
+After any changes that touch multiple files or cross-references, always run both:
+```bash
+make test                  # Full test suite
+uv run pre-commit run --all-files  # All pre-commit hooks
+```
+This ensures no import breakage, linting issues, or type errors are introduced.
+
+## Coding Conventions
+
+- **Python 3.11-3.13**, Google-style docstrings (pydocstyle convention)
+- **Type annotations** required everywhere (mypy semistrict: no implicit optional, no re-export, disallow generics, etc.)
+- **Schema validation** at Dataset construction time via `validate_schema()` and `compare_struct_schemas()`
+- **QC flags** use `@qc_test` decorator on methods; datasets expose `get_QC_mappings()` and `get_QC_column_name()`
+- **Hydra configs** use `_target_` field for instantiation; MISSING fields are required at runtime
+- **All external data sources** live under `datasource/` with subpackages per source (finngen, gnomad, gwas_catalog, etc.)

src/gentropy/biosample_index.py

@@ -2,10 +2,26 @@
 
 from __future__ import annotations
 
+from typing import Annotated
+
+from pydantic import BaseModel, Field
+
 from gentropy.common.session import Session
 from gentropy.datasource.biosample_ontologies.utils import extract_ontology_from_json
 
 
+class BiosampleIndexDefaults(BaseModel, frozen=True):
+    """Defaults for BiosampleIndexStep.
+
+    All fields are mandatory input/output paths - no defaults.
+    """
+
+    cell_ontology_input_path: Annotated[str, Field(description="Path to cell ontology input file.")]
+    uberon_input_path: Annotated[str, Field(description="Path to Uberon ontology input file.")]
+    efo_input_path: Annotated[str, Field(description="Path to EFO ontology input file.")]
+    biosample_index_path: Annotated[str, Field(description="Output path for biosample index dataset.")]
+
+
 class BiosampleIndexStep:
     """Biosample index step.
 
@@ -14,31 +30,25 @@ class BiosampleIndexStep:
 
     def __init__(
         self,
+        config: BiosampleIndexDefaults,
         session: Session,
-        cell_ontology_input_path: str,
-        uberon_input_path: str,
-        efo_input_path: str,
-        biosample_index_path: str,
     ) -> None:
         """Run Biosample index generation step.
 
         Args:
-            session (Session): Session object.
-            cell_ontology_input_path (str): Input cell ontology dataset path.
-            uberon_input_path (str): Input uberon dataset path.
-            efo_input_path (str): Input efo dataset path.
-            biosample_index_path (str): Output biosample index dataset path.
+            config: Step configuration defaults.
+            session: Active gentropy session.
         """
         cell_ontology_index = extract_ontology_from_json(
-            cell_ontology_input_path, session.spark
+            config.cell_ontology_input_path, session.spark
         )
-        uberon_index = extract_ontology_from_json(uberon_input_path, session.spark)
+        uberon_index = extract_ontology_from_json(config.uberon_input_path, session.spark)
         efo_index = extract_ontology_from_json(
-            efo_input_path, session.spark
+            config.efo_input_path, session.spark
         ).retain_rows_with_ancestor_id(["CL_0000000"])
 
         biosample_index = cell_ontology_index.merge_indices([uberon_index, efo_index])
 
         biosample_index.df.coalesce(session.output_partitions).write.mode(
             session.write_mode
-        ).parquet(biosample_index_path)
+        ).parquet(config.biosample_index_path)

src/gentropy/colocalisation.py

@@ -2,30 +2,66 @@
 
 from __future__ import annotations
 
-from typing import Literal, NotRequired, TypedDict
+from typing import Annotated, Any
 
 import pyspark.sql.functions as f
+from pydantic import BaseModel, Field, field_validator
 
 from gentropy.common.session import Session
 from gentropy.dataset.study_locus import FinemappingMethod, StudyLocus
 from gentropy.method.colocalisation import ColocalisationMethod
 
+_VALID_COLOC_METHODS = frozenset(m.name for m in ColocalisationMethod)
 
-class ColocalisationMethodParams(TypedDict):
-    """Colocalisation method parameters."""
 
-    priorc1: NotRequired[float]
-    """Prior on variant being causal for trait 1. Defaults to 1e-4. For coloc, coloc_pip, coloc_pip_ecaviar method only."""
-    priorc2: NotRequired[float]
-    """Prior on variant being causal for trait 2. Defaults to 1e-4. For coloc, coloc_pip, coloc_pip_ecaviar method only."""
-    priorc12: NotRequired[float]
-    """Prior on variant being causal for both traits. Defaults to 1e-5. For coloc, coloc_pip, coloc_pip_ecaviar method only."""
-    overlap_size_cutoff: NotRequired[int]
-    """Minimum number of overlapping variants before filtering. Defaults to 0. For coloc method only."""
-    posterior_cutoff: NotRequired[float]
-    """Minimum overlapping Posterior probability cutoff for small overlaps. Defaults to 0.0. For coloc method only."""
-    pseudocutoff: NotRequired[float]
-    """Pseudocount to avoid log(0). Defaults to 1e-10. For coloc method only."""
+class ColocalisationDefaults(BaseModel, frozen=True):
+    """Defaults for ColocalisationStep.
+
+    All values are frozen - create a new instance to override.
+    """
+
+    credible_set_path: Annotated[str, Field(description="Input credible sets path.")]
+    coloc_path: Annotated[str, Field(description="Output colocalisation path.")]
+    colocalisation_method: Annotated[
+        str,
+        Field(
+            description=(
+                "Colocalisation method. One of: "
+                + ", ".join(sorted(_VALID_COLOC_METHODS))
+                + " (case-insensitive)."
+            )
+        ),
+    ]
+    restrict_right_studies: Annotated[
+        list[str] | None, Field(description="Restrict right side studies.")
+    ] = None
+    gwas_v_qtl_overlap_only: Annotated[
+        bool, Field(description="Only GWAS vs molQTL overlaps.")
+    ] = False
+    colocalisation_method_params: Annotated[
+        dict[str, Any] | None, Field(description="Method parameters.")
+    ] = None
+
+    @field_validator("colocalisation_method", mode="before")
+    @classmethod
+    def validate_colocalisation_method(cls, v: object) -> object:
+        """Validate colocalisation method name.
+
+        Args:
+            v: Raw field value.
+
+        Returns:
+            object: The original value if valid.
+
+        Raises:
+            ValueError: If value is not a recognised colocalisation method.
+        """
+        if isinstance(v, str) and v.upper() not in _VALID_COLOC_METHODS:
+            raise ValueError(
+                f"colocalisation_method must be one of "
+                f"{sorted(_VALID_COLOC_METHODS)} (case-insensitive), got {v!r}"
+            )
+        return v
 
 
 class ColocalisationStep:
@@ -36,54 +72,32 @@ class ColocalisationStep:
 
     def __init__(
         self,
+        config: ColocalisationDefaults,
         session: Session,
-        credible_set_path: str,
-        coloc_path: str,
-        colocalisation_method: Literal[
-            "coloc", "ecaviar", "coloc_pip", "coloc_pip_ecaviar"
-        ],
-        restrict_right_studies: list[str] | None = None,
-        gwas_v_qtl_overlap_only: bool = False,
-        colocalisation_method_params: ColocalisationMethodParams | None = None,
     ) -> None:
         """Run Colocalisation step.
 
-        This step allows for running two colocalisation methods: ecaviar and coloc. The default behaviour is all gwas vs all gwas plus all gwas vs all molecular-QTLs.
-
         Args:
-            session (Session): Session object.
-            credible_set_path (str): Input credible sets path.
-            coloc_path (str): Output path.
-            colocalisation_method (Literal["coloc", "ecaviar", "coloc_pip", "coloc_pip_ecaviar"]): Colocalisation method. Use 'coloc_pip_ecaviar' to run both ColocPIP and eCAVIAR and merge results.
-            restrict_right_studies (list[str] | None): List of study IDs to restrict the right side of the colocalisation overlaps to, e.g. all gwas vs a single studyId. Defaults to None.
-            gwas_v_qtl_overlap_only (bool): If True, restricts the right side of colocalisation overlaps to only molecular-QTL studies, e.g. all gwas vs all molQTLs. Defaults to False.
-            colocalisation_method_params (ColocalisationMethodParams | None): Keyword arguments passed to the colocalise method of Colocalisation class. Defaults to None
-
-        Keyword Args:
-            priorc1 (float): Prior on variant being causal for trait 1. Defaults to 1e-4. For coloc method only.
-            priorc2 (float): Prior on variant being causal for trait 2. Defaults to 1e-4. For coloc method only.
-            priorc12 (float): Prior on variant being causal for both traits. Defaults to 1e-5. For coloc method only.
-            overlap_size_cutoff (int): Minimum number of overlapping variants before filtering. Defaults to 0.
-            posterior_cutoff (float): Minimum overlapping Posterior probability cutoff for small overlaps. Defaults to 0.0.
-            pseudocutoff (float): Pseudocount to avoid log(0). Defaults to 1e-10. For coloc method only.
+            config: Step configuration defaults.
+            session: Active gentropy session.
         """
-        cm = ColocalisationMethod.get_method_class(colocalisation_method)
-        cs = StudyLocus.from_parquet(session, credible_set_path)
+        cm = ColocalisationMethod.get_method_class(config.colocalisation_method)
+        cs = StudyLocus.from_parquet(session, config.credible_set_path)
 
-        if colocalisation_method.upper() == ColocalisationMethod.COLOC.name:
+        if config.colocalisation_method.upper() == ColocalisationMethod.COLOC.name:
             cs = cs.filter(
                 f.col("finemappingMethod").isin(FinemappingMethod.methods_with_lbf())
             )
 
         overlaps = cs.find_overlaps(
-            restrict_right_studies=restrict_right_studies,
-            gwas_v_qtl_overlap_only=gwas_v_qtl_overlap_only,
+            restrict_right_studies=config.restrict_right_studies,
+            gwas_v_qtl_overlap_only=config.gwas_v_qtl_overlap_only,
         )
-        params = colocalisation_method_params or {}
+        params = config.colocalisation_method_params or {}
         result = cm.colocalise(overlapping_signals=overlaps, **params)
 
         (
             result.df.coalesce(session.output_partitions)
             .write.mode(session.write_mode)
-            .parquet(coloc_path)
+            .parquet(config.coloc_path)
         )