Reproduce corrupt lockfiles by maresb · Pull Request #862 · conda/conda-lock

maresb · 2025-10-28T19:24:27Z

Description

Introduces a pipeline that reproduces a very complex chain of events leading to corrupted lockfiles due to the upstream issue mamba-org/mamba#4052.

This pipeline serves as a test case for producing a robust fix.

netlify · 2025-10-28T19:24:34Z

✅ Deploy Preview for conda-lock ready!

Name	Link
🔨 Latest commit	`3c8ac6e`
🔍 Latest deploy log	https://app.netlify.com/projects/conda-lock/deploys/690e4ca57fe4fe0008bf2ed0
😎 Deploy Preview	https://deploy-preview-862--conda-lock.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

…ocumenting scripts, prerequisites, and outputs

Builds on the mamba 2.6.0 cache-layout fix to also defend against the corrupt ``repodata_record.json`` entries that mamba/micromamba 2.1.1-2.3.2 wrote to the package cache (mamba-org/mamba#4052, fixed upstream in mamba-org/mamba#4110 / mamba 2.6.0). Even after upgrading to mamba 2.6.0, existing corrupt cache entries persist and can produce a silently-broken lockfile (zero-deps records propagate, ``apply_categories`` doesn't reach those packages, then ``LockedDependency.to_v1()`` emits zero entries per empty-category set -- the package vanishes from disk). Implements all five mitigations from conda#896: 1. *Corruption detection + heal in disk fallback.* ``_is_mamba_2x_corrupt_record`` matches the ``timestamp == 0`` + empty ``license`` signature (corroborated by empty ``depends`` or missing ``sha256``). When detected, ``_heal_corrupt_record`` overlays the record with ``info/index.json`` (extracted from the package tarball, never affected by the bug) so canonical ``depends`` / ``constrains`` / ``timestamp`` / ``license`` / ``build_number`` are recovered. Cache-only fields (``url`` / ``channel`` / ``md5`` / ``sha256``) are preserved when non-empty. A WARNING surfaces each heal so users can ``mamba clean -a`` to remove the bad cache. 2. *pkgs_dirs leak detection.* conda-lock sets ``CONDA_PKGS_DIRS`` to an isolated temp dir in ``conda_env_override``, but several mamba/conda releases have merged the env var with condarc rather than replacing it -- letting corrupt entries from the user's global cache leak into the solver's view. ``_warn_on_pkgs_dirs_leak`` reports any extra paths. 3. *Degraded-path warning.* The whole disk-fallback branch in ``_reconstruct_fetch_actions`` now emits a single WARNING per platform listing the affected packages. Mamba 2.6.0+ keeps full repodata in LINK and avoids the disk path entirely; reaching it after upgrading is a sign of older cache entries or older solver versions. 4. *Orphan rescue.* After ``apply_categories``, any planned package with an empty category set would silently disappear from the v1 lockfile output. We rescue such packages into ``"main"`` (so the lockfile stays consistent with what the solver planned) and WARN with the package list. Two common causes: solver auto-installs (e.g. pip, wheel) that aren't transitive deps of anything requested -- benign -- and the corrupt-cache failure mode -- the actionable case the warning surfaces. 5. *Lockfile-side heal of carry-forward corruption.* A lockfile generated against a corrupt mamba 2.1.1-2.3.2 cache inherits empty ``dependencies: {}`` for affected entries. Without intervention the lockfile is then a *carrier* for the bug: every subsequent ``conda-lock lock --update`` reads it back via ``to_fetch_action()`` and emits a fresh lockfile that's still corrupt -- forever, even on mamba 2.6.0+. ``_heal_locked_dependencies_from_cache`` runs in ``update_specs_for_arch`` *before* ``fake_conda_environment`` and ``to_fetch_action()``, scanning for empty-deps entries and recovering the canonical ``depends`` from ``info/index.json`` in the package cache. Tests: - ``tests/test_conda_solver.py`` covers each mitigation at unit level (signature detection, heal-from-index, un-healable case, pkgs_dirs leak warning, degraded-path warning, orphan rescue, lockfile-side heal helpers). - ``tests/test_conda_solver.py::test_pr862_corrupt_2_1_1_cache_does_not_drop_packages`` is an end-to-end test driven by the reproduction harness from ``tests/test-corrupt-repodata/`` (originally PR conda#862). The harness pins the *exact* metadata that micromamba 2.1.1 wrote when installing from an explicit lockfile; the test: a. warms a cache by ``micromamba create -f 01-explicit.lock`` (real package downloads, current correct metadata); b. extracts ``2.1.1-pkgs.tar.gz`` and copies its corrupt ``info/{index,repodata_record}.json`` over the warmed cache, exactly mirroring stage 04 of the PR conda#862 pipeline; c. runs ``conda-lock lock`` against the dev environment with ``CONDA_PKGS_DIRS`` pointing at the hybrid cache; d. compares the resulting package set to a fresh clean-cache baseline (built in the same test run, so no version drift). Without the fix, conda-lock against this cache produced 110 packages (78 with empty ``dependencies``) -- the committed ``lockfile-2.1.1-pkgs-lock-with-mamba.yml`` reference. With the fix applied the package set matches the clean baseline exactly, with at most a handful of legitimate empty-deps entries (packages like ``nlohmann_json-abi``). The PR conda#862 stage scripts are excluded from mypy/ty (they're Docker-based one-shot data generators, not runtime code).

Builds on the mamba 2.6.0 cache-layout fix to also defend against the corrupt ``repodata_record.json`` entries that mamba/micromamba 2.1.1-2.3.2 wrote to the package cache (mamba-org/mamba#4052, fixed upstream in mamba-org/mamba#4110 / mamba 2.6.0). Even after upgrading to mamba 2.6.0, existing corrupt cache entries persist and can produce a silently-broken lockfile (zero-deps records propagate, ``apply_categories`` doesn't reach those packages, then ``LockedDependency.to_v1()`` emits zero entries per empty-category set -- the package vanishes from disk). Mitigations: 1. *Corruption detection + heal in disk fallback.* ``_is_mamba_2x_corrupt_record`` matches the ``timestamp == 0`` + empty ``license`` signature (corroborated by empty ``depends`` or missing ``sha256``). When detected, ``_heal_corrupt_record`` overlays the record with ``info/index.json`` (extracted from the package tarball, never affected by the bug). Cache-only fields (``url``/``channel``/``md5``/``sha256``) are preserved when non-empty. Each heal logs a WARNING. 2. *pkgs_dirs leak detection.* ``_warn_on_pkgs_dirs_leak`` reports any extra paths that mamba/conda merged in from condarc despite the env-var override. 3. *Degraded-path warning.* The disk-fallback branch in ``_reconstruct_fetch_actions`` emits a single WARNING per platform listing the affected packages. 4. *Reverse-propagation of categories from .dependencies.* The conda solver routinely auto-installs packages that aren't transitive deps of anything requested -- ``pip``/``wheel`` being canonical. ``apply_categories`` walks *forward* from each requested root through ``.dependencies`` and never reaches them. Without intervention they end up with empty categories and ``to_v1()`` silently drops them. We now reverse-propagate: an orphan inherits the union of categories of the packages it depends on (e.g. ``pip`` depends on ``python``, so ``pip`` lands in whatever category ``python`` was requested in -- *not* a default of ``main``, which would change install semantics). 5. *Hard-fail on residual orphans.* After reverse-propagation, any package still without a category is the corruption-induced failure mode: empty ``.dependencies`` means nothing for reverse propagation to inherit from. We raise ``OrphanLockedDependencyError`` rather than silently drop or stuff the package into ``main``. 6. *Lockfile-side heal of carry-forward corruption.* A lockfile generated against a corrupt mamba 2.1.1-2.3.2 cache inherits empty ``dependencies: {}`` for affected entries. ``_heal_locked_dependencies_from_cache`` runs in ``update_specs_for_arch`` *before* ``fake_conda_environment`` and ``to_fetch_action()``, scanning empty-deps entries and recovering ``depends`` from ``info/index.json`` in the package cache. When the cache cannot confirm one way or the other (legit-empty vs corrupt-empty), the entry is left untouched and the orphan check in ``solve_conda`` is the responsible escalation point: legit-empty packages are reachable forward via ``apply_categories``; corruption- induced ones aren't reachable forward *or* backward and hard-fail. End-to-end test backed by the PR conda#862 reproduction harness (``tests/test-corrupt-repodata/``): ``test_pr862_corrupt_2_1_1_cache_does_not_drop_packages`` walks the *full* committed pipeline minus Docker: - Stage 04: ``conda-lock lock`` against a hybrid cache with real package files (warmed by ``micromamba create -f 01-explicit.lock``) and corrupt ``info/{index,repodata_record}.json`` overlaid from ``2.1.1-pkgs.tar.gz``. - Stage 05: ``conda-lock render --kind=explicit`` to the explicit lockfile that ``conda-lock install`` consumes verbatim. Both stages run in two parallel scenarios -- corrupt cache and fresh clean cache -- and the test asserts: - *Structural lockfile equivalence:* parsing both unified lockfiles via ``parse_conda_lock_file`` produces the same set of ``(name, version, frozenset(categories))`` tuples. This catches category-mutation bugs (the previous round's "rescue into main" mistake would have shown up here) as well as silent drops. - *Explicit-URL equivalence:* the rendered explicit lockfiles -- the user-visible install surface, per conda#862's "packages are missing if and only if they are missing from the explicit lockfile" -- have identical URL sets. - *Empty-deps headroom:* the corrupt-cache lockfile has no more empty ``.dependencies`` entries than the clean control (a handful of legitimately empty packages like ``nlohmann_json-abi``). Parametrized over both ``mamba`` and ``conda-standalone`` -- stage 04 itself runs both, since different conda-flavoured binaries handle leaked ``pkgs_dirs`` differently and the bug surfaces in both. Plus unit tests for each helper (signature detection, heal-from-index, deferred-when-cache-missing, legit-empty-deps acceptance, pkgs_dirs leak warning, degraded-path warning, reverse-propagation handles auto-install orphans, hard-fail on unrecoverable orphans). The PR conda#862 stage scripts are excluded from mypy/ty/pre-commit -- they are Docker-based one-shot data generators, not runtime code.

Builds on the mamba 2.6.0 cache-layout fix to also defend against the corrupt ``repodata_record.json`` entries that mamba/micromamba 2.1.1-2.3.3 wrote to the package cache (mamba-org/mamba#4052, fixed upstream in mamba-org/mamba#4110 / mamba 2.6.0). The 2.3.3 partial fix recovered ``depends`` and ``constrains`` from ``info/index.json`` but still wrote ``timestamp: 0``, ``license: ""``, ``build_number: 0``, ``track_features: ""`` -- enough to trip our detector. Even after upgrading to mamba 2.6.0, existing corrupt cache entries persist and can produce a silently-broken lockfile (zero-deps records propagate, ``apply_categories`` doesn't reach those packages, then ``LockedDependency.to_v1()`` emits zero entries per empty-category set -- the package vanishes). Mitigations: 1. *Corruption detection + heal in disk fallback.* ``_is_mamba_2x_corrupt_record`` matches the ``timestamp == 0`` + empty ``license`` signature (covers the full 2.1.1-2.3.3 range). ``_heal_corrupt_record`` overlays the record with ``info/index.json`` -- never affected by the bug. Cache-only fields (url/channel/md5/sha256) are preserved when non-empty. WARNING per heal. 2. *pkgs_dirs leak detection.* Reports paths that mamba/conda merged in from condarc despite the env-var override. 3. *Degraded-path warning.* Single WARNING per platform when ``_reconstruct_fetch_actions`` takes the disk-fallback branch. 4. *Reverse-propagation of categories from .dependencies.* Solver auto-installs (pip, wheel) aren't transitive deps of anything requested, so ``apply_categories`` walking forward from roots never reaches them. Reverse-prop: an orphan inherits the union of categories of the packages it depends on. ``_truncate_main_category`` runs after reverse-prop so a package inheriting both ``main`` and ``dev`` collapses to ``{main}`` (the same rule ``apply_categories`` applies to forward-walked packages); this prevents v1 serialization from emitting the package twice. 5. *Hard-fail on residual orphans.* Anything still without a category after reverse-prop is the corruption-induced failure mode (empty ``.dependencies`` so nothing for reverse-prop to inherit from). Raises ``OrphanLockedDependencyError``. 6. *Lockfile-side heal of carry-forward corruption.* ``_heal_locked_dependencies_from_cache`` runs in ``update_specs_for_arch`` *before* ``fake_conda_environment`` and ``to_fetch_action()``, scanning empty-deps lockfile entries and recovering ``depends`` from ``info/index.json``. The cache is the discriminator between legit-empty (``tzdata``, ``python_abi``, ``nlohmann_json-abi``, ``_libgcc_mutex``, ...) and corrupt-empty: - Cache confirms canonical depends are non-empty -> heal. - Cache confirms canonical depends are empty -> legit-empty, leave alone. - Cache has no entry: ambiguous on its own, decide via the whole platform's heal outcome: - *Some entries healed and others ambiguous* -> mixed-confirm pattern. Cache had enough metadata for some packages but not others; the unhealed ones are overwhelmingly likely to be corruption-induced. Raise ``LockfileHealError`` rather than carry the bug forward on every subsequent ``conda-lock lock --update``. - *All ambiguous* -> cache is just empty (e.g. fresh CI env, user hasn't run ``conda-lock install``). WARN with prescriptive text and let the orphan check downstream catch the silent-vanishing variant. End-to-end test backed by the PR conda#862 reproduction harness: ``test_pr862_corrupt_2_1_1_cache_does_not_drop_packages`` walks the *full* committed reproduction pipeline minus Docker, parametrized over both ``[mamba]`` and ``[conda-standalone]``: - Stage 04: ``conda-lock lock`` against a hybrid cache (real package files warmed by ``micromamba create -f 01-explicit.lock``, with the corrupt ``info/{index,repodata_record}.json`` from ``2.1.1-pkgs.tar.gz`` overlaid). - Stage 05: ``conda-lock render --kind=explicit`` to the explicit lockfile that ``conda-lock install`` consumes verbatim. Both stages run in two parallel scenarios -- corrupt cache and fresh clean cache -- and the test asserts: - *Structural lockfile equivalence:* parsing both unified lockfiles via ``parse_conda_lock_file`` yields the same set of ``(name, version, frozenset(categories))`` tuples. This catches category-mutation bugs (the previous round's "rescue into main" mistake would have shown up here) as well as silent drops. - *Explicit-URL equivalence:* the rendered explicit lockfiles -- the user-visible install surface, per conda#862's "packages are missing if and only if they are missing from the explicit lockfile" -- have identical URL sets. - *Empty-deps headroom:* corrupt-cache lockfile has no more empty ``.dependencies`` entries than the clean control. Plus unit tests for: signature detection, heal-from-index, mixed-cache hard-fail, all-cache-miss WARNING, legit-empty-deps acceptance, pkgs_dirs leak warning, degraded-path warning, mixed-root reverse propagation with main truncation, hard-fail on unrecoverable orphans, auto-install orphan inheritance. To make the conda-standalone parametrization actually exercise CI, ``conda-standalone`` is added to ``environments/dev-environment.yaml`` and the dev environments are relocked. The PR conda#862 stage scripts are excluded from mypy/ty/pre-commit -- they are Docker-based one-shot data generators, not runtime code.

Builds on the mamba 2.6.0 cache-layout fix to also defend against the corrupt ``repodata_record.json`` entries that mamba/micromamba 2.1.1-2.3.3 wrote to the package cache (mamba-org/mamba#4052, fixed upstream in mamba-org/mamba#4110 / mamba 2.6.0). The 2.3.3 partial fix recovered ``depends`` and ``constrains`` but still wrote ``timestamp: 0`` and ``license: ""`` -- enough to trip our detector. Even after upgrading to mamba 2.6.0, existing corrupt cache entries persist and a corrupt lockfile (one *generated* against such a cache) is itself a carrier for the bug -- every subsequent ``conda-lock lock --update`` reads it back via ``to_fetch_action()`` and emits a fresh lockfile that's still corrupt. Mitigations: 1. *Corruption detection + heal in disk fallback.* ``_is_mamba_2x_corrupt_record`` matches the ``timestamp == 0`` + empty ``license`` signature (covers the full 2.1.1-2.3.3 range). ``_heal_corrupt_record`` overlays the record with ``info/index.json`` (extracted at install time, never affected by the bug). Cache-only fields (url/channel/md5/sha256) preserved when non-empty. WARNING per heal. 2. *pkgs_dirs leak detection.* Reports paths that mamba/conda merged in from condarc despite the env-var override. 3. *Degraded-path warning.* Single WARNING per platform when ``_reconstruct_fetch_actions`` takes the disk-fallback branch. 4. *Reverse-propagation of categories from .dependencies.* Solver auto-installs (pip, wheel) aren't transitive deps of anything requested, so ``apply_categories`` walking forward never reaches them. Reverse-prop: an orphan inherits the union of categories of the packages it depends on. Followed by ``_truncate_main_category`` so a package inheriting both ``main`` and ``dev`` collapses to ``{main}`` -- same rule ``apply_categories`` applies to forward-walked packages -- and v1 serialization doesn't emit the package twice. 5. *Hard-fail on residual orphans.* Anything still without a category after reverse-prop is the corruption-induced failure mode. Raises ``OrphanLockedDependencyError``. 6. *Lockfile-side heal of carry-forward corruption.* ``_heal_locked_dependencies_from_cache`` runs in ``update_specs_for_arch`` *before* ``fake_conda_environment`` and ``to_fetch_action()``, scanning empty-deps lockfile entries and recovering ``depends`` from ``info/index.json``. The cache-vs- lockfile *contradiction* (``healed > 0`` -- cache says non-empty, lockfile says empty) is the only valid evidence of corruption: - ``healed > 0`` and any entries are ambiguous (no cache info) -> raise ``LockfileHealError``. The contradiction is the fingerprint of a corrupt-cache-derived lockfile, and silently carrying the ambiguous entries forward would let the bug recur. - All-ambiguous (cache is just empty, e.g. fresh CI env) -> WARN with prescriptive text and let the orphan check downstream decide. - ``confirmed_legit_empty > 0`` (cache *agreed* depends are empty for some package) is *not* evidence of corruption: a partial cache containing ``info/index.json`` for ``tzdata``, ``_libgcc_mutex``, etc. but not other packages is normal. An earlier predicate (``healed + confirmed_legit_empty > 0``) gave false-positive hard-fails on clean lockfiles that just happened to have one of those leaf packages cached. Repair advice leads with regenerate-from-sources on a clean cache; ``conda-lock install`` is demoted to a diagnostic-only note for investigating carry-forward metadata, with an explicit warning not to install a corrupt lockfile into a real environment (any packages already vanished during v1 serialization are unrecoverable). End-to-end test backed by the PR conda#862 reproduction harness: ``test_pr862_corrupt_2_1_1_cache_does_not_drop_packages`` walks the full committed reproduction pipeline minus Docker, parametrized over both ``[mamba]`` and ``[conda-standalone]``. It warms a real cache, overlays the corrupt ``info/{index,repodata_record}.json`` from ``2.1.1-pkgs.tar.gz``, runs ``conda-lock lock`` against the hybrid cache, renders the explicit lockfile (stage 05), and asserts: - *Structural lockfile equivalence* via ``parse_conda_lock_file`` -- the set of ``(name, version, frozenset(categories))`` tuples must match a clean control. Catches category-mutation bugs as well as silent drops. - *Explicit-URL equivalence* of the rendered explicit lockfiles -- the user-visible install surface, per conda#862's "packages are missing if and only if they are missing from the explicit lockfile". - *Empty-deps headroom*. Plus unit tests for: signature detection, heal-from-index, mixed-cache hard-fail (contradiction), all-cache-miss WARNING, partial-cache-of- legit-empty does NOT hard-fail (regression test for the predicate fix), legit-empty-deps acceptance, pkgs_dirs leak warning, degraded-path warning, mixed-root reverse propagation with main truncation, hard-fail on unrecoverable orphans, auto-install orphan inheritance. To make the conda-standalone parametrization exercise both CI and local ``pixi run pytest``, ``conda-standalone`` is added to: - ``environments/dev-environment.yaml`` (CI source of truth) - ``environments/conda-lock-python-3.10.yaml``, ``environments/conda-lock-python-3.14.yaml`` (relocked) - ``pixi.toml`` (local dev source of truth) ``pixi.lock`` will need a one-time ``pixi lock`` to refresh the solved package set; this could not be done in the sandbox where the change was developed (the conda-pypi mapping endpoint was blocked). CI is unaffected -- it consumes the conda-lock environment files above, not pixi. The PR conda#862 stage scripts are excluded from mypy/ty/pre-commit -- they are Docker-based one-shot data generators, not runtime code.

Builds on the mamba 2.6.0 cache-layout fix to also defend against the corrupt ``repodata_record.json`` entries that mamba/micromamba 2.1.1-2.3.3 wrote to the package cache (mamba-org/mamba#4052, fixed upstream in mamba-org/mamba#4110 / mamba 2.6.0). The 2.3.3 partial fix recovered ``depends`` and ``constrains`` but still wrote ``timestamp: 0`` and ``license: ""`` -- enough to trip our detector. Mitigations: 1. *Corruption detection + heal in disk fallback.* ``_is_mamba_2x_corrupt_record`` matches the ``timestamp == 0`` + empty ``license`` signature. ``_heal_corrupt_record`` overlays the record with ``info/index.json`` (extracted at install time, never affected by the bug). WARNING per heal. 2. *Rich-LINK corruption guard.* The ``_link_action_as_fetch`` fast path bypasses ``_get_repodata_record`` and therefore the corruption detector. We re-check the corruption signature in the fast path so a LINK whose metadata carries the bug routes to disk fallback (where ``_heal_corrupt_record`` can recover from ``info/index.json``) rather than synthesizing a corrupt FETCH directly. Mamba 2.6.0+ is supposed to heal cache records before emitting them in LINK; we no longer depend on that external invariant. 3. *pkgs_dirs leak detection.* Reports paths that mamba/conda merged in from condarc despite the env-var override. 4. *Degraded-path warning.* Single WARNING per platform when ``_reconstruct_fetch_actions`` takes the disk-fallback branch. 5. *Reverse-propagation of categories from .dependencies.* Solver auto-installs (pip, wheel) aren't transitive deps of anything requested. Reverse-prop: an orphan inherits the union of categories of the packages it depends on. ``_truncate_main_category`` runs after reverse-prop so a package inheriting both ``main`` and ``dev`` collapses to ``{main}`` -- v1 serialization doesn't emit the package twice. 6. *Hard-fail on residual orphans.* Anything still without a category after reverse-prop is the corruption-induced silent-vanishing failure mode. Raises ``OrphanLockedDependencyError`` rather than silently drop. 7. *Lockfile-side heal of carry-forward corruption (per-entry).* ``_heal_locked_dependencies_from_cache`` runs in ``update_specs_for_arch`` *before* ``fake_conda_environment`` and ``to_fetch_action()``. The cache is a *per-entry* discriminator: - Cache says non-empty depends -> heal in place. The contradiction (cache vs lockfile) is per-entry proof of corruption for that entry. - Cache says empty depends -> legit-empty, leave alone. - Cache has no entry -> ambiguous. Healable-elsewhere is NOT evidence about ambiguous-here: partial caches are normal (a user may have ``libzlib`` cached but not ``tzdata``). We WARN with prescriptive text and let the orphan check downstream cover the silent-vanishing variant. We do NOT hard-fail on the ``healed > 0 + ambiguous`` mixed case -- that predicate false-positived on the ordinary scenario where one corrupt entry is healable while a legit-empty leaf is simply not yet in the cache. The warning is honest about a residual limitation: a corrupt entry that is itself a requested or otherwise-reachable root may carry forward unflagged when its missing transitive deps happen to be reachable via other paths in the graph. The repair guidance is to regenerate the lockfile from sources on a known-clean cache; ``conda-lock install`` is demoted to a diagnostic-only note for investigating carry-forward metadata, with an explicit warning not to install a corrupt lockfile into a real environment (any packages already vanished during v1 serialization are unrecoverable). End-to-end test backed by the PR conda#862 reproduction harness: ``test_pr862_corrupt_2_1_1_cache_does_not_drop_packages`` walks the full committed reproduction pipeline minus Docker, parametrized over both ``[mamba]`` and ``[conda-standalone]``. It warms a real cache, overlays the corrupt ``info/{index,repodata_record}.json`` from ``2.1.1-pkgs.tar.gz``, runs ``conda-lock lock`` against the hybrid cache, renders the explicit lockfile (stage 05), and asserts structural lockfile equivalence + explicit-URL equivalence + empty- deps headroom against a clean control. Plus unit tests for: corruption signature detection, heal-from-index, rich-LINK corruption guard, heal-in-place + WARN on ambiguous (includes the explicit healed-libzlib + legit-empty-tzdata-leaf scenario the predicate now correctly handles), legit-empty-deps acceptance, partial-cache-of-legit-empty does NOT raise, pkgs_dirs leak warning, degraded-path warning, mixed-root reverse propagation with main truncation, hard-fail on unrecoverable orphans, auto-install orphan inheritance. To make the conda-standalone parametrization exercise both CI and local ``pixi run pytest``, ``conda-standalone`` is added to: - ``environments/dev-environment.yaml`` (CI source of truth) - ``environments/conda-lock-python-3.10.yaml``, ``environments/conda-lock-python-3.14.yaml`` (relocked) - ``pixi.toml`` (local dev source of truth) ``pixi.lock`` will need a one-time ``pixi lock`` to refresh the solved package set; this could not be done in the sandbox where the change was developed. CI is unaffected -- it consumes the conda-lock environment files above, not pixi. The PR conda#862 stage scripts are excluded from mypy/ty/pre-commit -- they are Docker-based one-shot data generators, not runtime code.

Builds on the mamba 2.6.0 cache-layout fix to also defend against the corrupt ``repodata_record.json`` entries that mamba/micromamba 2.1.1-2.3.3 wrote to the package cache (mamba-org/mamba#4052, fixed upstream in mamba-org/mamba#4110 / mamba 2.6.0). The 2.3.3 partial fix recovered ``depends`` and ``constrains`` but still wrote ``timestamp: 0`` and ``license: ""`` -- enough to trip our detector. Scope is best-effort, not comprehensive. See "Limitations" below. Mitigations: 1. *Corruption detection + heal in disk fallback.* ``_is_mamba_2x_corrupt_record`` matches ``timestamp == 0`` + empty ``license``. ``_heal_corrupt_record`` overlays the record with ``info/index.json`` (extracted at install time, never affected by the bug). WARNING per heal. 2. *Rich-LINK corruption guard.* The ``_link_action_as_fetch`` fast path now re-checks the corruption signature on the LINK before synthesizing a FETCH. A LINK that carries the bug's metadata is rejected and routed to disk fallback for healing, instead of riding straight into a corrupt FETCH. We no longer rely on the external invariant that the solver heals before emitting LINK. 3. *pkgs_dirs leak detection.* Reports paths that mamba/conda merged in from condarc despite the env-var override. 4. *Degraded-path warning.* Single WARNING per platform when ``_reconstruct_fetch_actions`` takes the disk-fallback branch. 5. *Reverse-propagation of categories from .dependencies.* Solver auto-installs (pip, wheel) aren't transitive deps of anything requested. Reverse-prop: an orphan inherits the union of categories of the packages it depends on. ``_truncate_main_category`` runs after reverse-prop so a package inheriting both ``main`` and ``dev`` collapses to ``{main}`` -- v1 serialization doesn't emit the package twice. 6. *Hard-fail on residual orphans.* Anything still without a category after reverse-prop is the corruption-induced silent-vanishing failure mode. Raises ``OrphanLockedDependencyError`` rather than silently drop. 7. *Lockfile-side heal of carry-forward corruption (per-entry).* ``_heal_locked_dependencies_from_cache`` scans empty-deps lockfile entries before ``fake_conda_environment``/``to_fetch_action()``. The cache is a *per-entry* discriminator: - Cache says non-empty depends -> heal in place. The contradiction (cache vs lockfile) is per-entry proof of corruption for that entry. - Cache says empty depends -> legit-empty, leave alone. - Cache has no entry -> ambiguous. Healable-elsewhere is NOT evidence about ambiguous-here: partial caches are normal (a user may have ``libzlib`` cached but not ``tzdata``). We WARN with prescriptive text. The repair guidance leads with regenerate-from-sources on a known-clean cache; ``conda-lock install`` is demoted to a diagnostic-only note for investigating carry-forward metadata, with an explicit warning not to install a corrupt lockfile into a real environment. Limitations (best-effort, not comprehensive): - *Categorized-corrupt-carrier* case is uncovered by the orphan check. When a corrupt entry is itself a requested or otherwise-reachable root and its missing transitive deps happen to be reachable via other paths in the graph, the lockfile remains internally inconsistent (its dependency edges don't match the channel's repodata) but its package set is intact, so neither the orphan check nor a per-entry cache check can flag it. The all-ambiguous WARNING is the visibility layer for this case; the operator-side repair is to regenerate from sources. End-to-end test backed by the PR conda#862 reproduction harness: ``test_pr862_corrupt_2_1_1_cache_does_not_drop_packages`` walks the full committed reproduction pipeline minus Docker, parametrized over both ``[mamba]`` and ``[conda-standalone]``. It warms a real cache, overlays the corrupt ``info/{index,repodata_record}.json`` from ``2.1.1-pkgs.tar.gz``, runs ``conda-lock lock`` against the hybrid cache, renders the explicit lockfile (stage 05), and asserts structural lockfile equivalence + explicit-URL equivalence + empty- deps headroom against a clean control. The test is each-parametrization ~5 minutes and depends on live conda-forge state in two solves. It's now marked ``@pytest.mark.corrupt_cache_repro`` so it can be deselected from default unit-test runs via ``pytest -m "not corrupt_cache_repro"``; CI continues to include it. Plus unit tests for: corruption signature detection, heal-from-index, rich-LINK corruption guard (regression test for the bypass the reviewer flagged), heal-in-place + WARN on ambiguous (includes the healed-libzlib + legit-empty-tzdata-leaf scenario the predicate now correctly handles), legit-empty-deps acceptance, partial-cache-of- legit-empty does NOT raise, pkgs_dirs leak warning, degraded-path warning, mixed-root reverse propagation with main truncation, hard-fail on unrecoverable orphans, auto-install orphan inheritance. To make the conda-standalone parametrization exercise both CI and local ``pixi run pytest``, ``conda-standalone`` is added to: - ``environments/dev-environment.yaml`` (CI source of truth). - ``environments/conda-lock-python-3.10.yaml``, ``environments/conda-lock-python-3.14.yaml`` (relocked). - ``pixi.toml`` and ``pixi.lock`` (local dev). The PR conda#862 stage scripts are excluded from mypy/ty/pre-commit -- they are Docker-based one-shot data generators, not runtime code.

Earlier the ``corrupt_cache_repro`` marker was registered but ``addopts`` didn't filter it out, so plain ``pytest`` (and the CI unit-test job) still collected and ran the two heavy live-conda-forge parametrizations of ``test_pr862_corrupt_2_1_1_cache_does_not_drop_packages``. Move the e2e to opt-in: - ``addopts`` in ``pyproject.toml`` now defaults to ``-m "not corrupt_cache_repro"``. ``pytest`` collected count drops from 63 to 61. - ``.github/workflows/test.yml`` gains a dedicated ``run-corrupt-cache-repro`` step that overrides ``addopts`` and invokes ``-m corrupt_cache_repro``. Coverage is preserved without paying ~10 minutes on every shard: the step runs once per (os, python-version) on shard 1, ubuntu-only (the fixture is a linux-64 cache snapshot from the PR conda#862 harness). Local invocations: - ``pytest`` -> default unit collection only. - ``pytest -m corrupt_cache_repro`` -> just the e2e. - ``pytest -m ""`` -> bypass the filter, run everything.

…e signature `update_specs_for_arch`'s cache-backed heal only runs during a fresh solve. `conda-lock render`, `conda-lock install`, and a no-solve `conda-lock lock` (where the source content hash is still valid) all go through `parse_conda_lock_file` without ever entering the heal path -- so a previously-generated, corrupt carrier lockfile would pass through those entry points silently. Add a heuristic warning that fires unconditionally on every parse: if any platform has more than `_LEGIT_EMPTY_DEPS_PER_PLATFORM_THRESHOLD` (=10) conda entries with empty `dependencies`, log a single WARNING naming conda#896 / mamba-org/mamba#4110, listing affected platforms, and pointing at "regenerate from sources on a known-clean cache" as the safe repair. The threshold sits well above the legitimate baseline (~5: tzdata, python_abi, _libgcc_mutex, _openmp_mutex, nlohmann_json-abi) and well below the corruption fingerprint (tens-to-hundreds per platform). Scope: - Covered: every read-only entry point that calls `parse_conda_lock_file` -- render, install, lock-no-solve, conda-standalone render path. The warning surfaces in stderr/log. - Not covered: in-place rewrite of the on-disk lockfile. `parse_conda_lock_file` is read-only, and read-only callers cannot necessarily re-solve, so we warn rather than raise. - Not covered: distinguishing legit-empty (`tzdata`) from corrupt-empty per entry. Lockfile alone doesn't carry that signal; only the count gap does. False-positive risk is low given the threshold. Tests (`tests/test_conda_solver.py`, section 7): - `test_parse_conda_lock_file_warns_on_corrupt_lockfile`: parses the committed PR conda#862 2.1.1 corrupt fixture, asserts the warning with conda#896, "regenerate from sources", and "Do NOT install". - `test_parse_conda_lock_file_quiet_on_clean_lockfile`: parses the 2.1.0 clean fixture, asserts no conda#896 warning (proves the threshold doesn't false-positive on legit-empty leaves). - `test_render_corrupt_lockfile_emits_warning`: invokes `conda-lock render --kind=explicit` on the corrupt fixture in a subprocess, asserts the warning propagates to stderr. https://claude.ai/code/session_01F6kPbdaqoKb9XARYdbRVhd

The previous parse-time detector warned only when more than 10 empty-deps conda entries appeared on a single platform. The reviewer correctly pointed out two failure modes: 1. Small environments with fewer than 11 packages can be just as corrupt; the threshold silently passes them through. 2. In practice corruption affects only some fraction of entries, so any pure count-based heuristic is fundamentally fragile. Replace the threshold with a per-entry classifier that exploits a mathematical invariant: a conda package whose build string identifies it as built against a Python ABI MUST declare a python dependency. There is no legitimate ``pyhd*``, ``pypy*``, or ``py<digit>...`` build with empty ``depends``. So a single such entry with empty ``dependencies`` is proof of corruption -- threshold-free, works on lockfiles of any size. Verified against the committed PR conda#862 fixtures: 48 of 77 corrupt empty-deps entries match the python-built signal (zero false positives in the clean baseline -- ``_libgcc_mutex``, ``tzdata``, ``python_abi``, ``nlohmann_json-abi`` all have non-``py*`` builds). For the residual native-binary entries (e.g. ``openssl``, ``krb5``) where lockfile-only evidence cannot prove corruption per-entry, fall back to a small absolute count (>3) of unclassified empty-deps entries on a single platform after stripping the well-known legit-empty allowlist (``_*`` ABI mutexes, ``tzdata``, ``ca-certificates``, ``*-abi``, ``*_abi``). The threshold here is deliberately low so even a small carrier lockfile trips, with the allowlist absorbing the conservative side. Tests cover both signals and the negative cases: - ``test_detector_warns_on_single_pynoarch_empty_entry``: 1 ``babel-pyhd8ed1ab_0`` with empty deps, lockfile of size 1, triggers warning (the small-corrupt case the reviewer flagged). - ``test_detector_warns_on_single_arch_python_empty_entry``: same for ``mypy-py314h*``. - ``test_detector_quiet_on_legit_empty_leaves``: 5 well-known legit-empty packages, no warning. - ``test_detector_warns_on_small_native_lib_corruption``: 4 native libs (>3 unclassified threshold) trips the fallback. - ``test_detector_quiet_on_two_native_empties_below_threshold``: 2 unclassified empties stay quiet to avoid false positives in tiny envs. - The two existing fixture-based tests still pin the large-lockfile behavior end-to-end (corrupt and clean PR conda#862 fixtures). https://claude.ai/code/session_01F6kPbdaqoKb9XARYdbRVhd

Two precision cleanups, no policy change. 1. ``CONDA_LOCK_ALLOW_ORPHANED_LOCKFILE=1`` warning is now explicit about the category consequence. The previous wording said "band-aid" and "use it only if you understand the consequences" without naming the actual consequence: every orphan gets assigned to ``main``, which means it will be installed by every ``conda-lock install`` invocation -- including ones that did NOT pass ``--dev-dependencies`` or ``-e <category>``. If the orphans include dev-only solver artifacts (the canonical reason a user reaches for the escape hatch), this over-installs them into production environments. A user enabling the escape hatch should not have to read the source to discover this. Both the pre-raise message and the runtime WARNING line now spell it out directly. The corresponding test now asserts the warning contains ``main`` and the ``--dev-dependencies`` / ``-e <category>`` flag-name(s) so the language can't quietly regress. 2. Drop stale "orphan rescue" wording in the comment introducing the PR conda#862 end-to-end repro section. Rescue is gone; the pipeline now reads ``cache-side heal -> apply_categories forward walk -> orphan check -> v1 serialization``. The other surviving "rescue" references in this file are intentional historical notes (a regression assertion that guards against the previous round's "rescue into main" mistake, and an inline "without any reverse-prop rescue" disclaimer in the new positive test) and stay. https://claude.ai/code/session_01F6kPbdaqoKb9XARYdbRVhd

The reviewer's high-priority finding: ``conda_solver.py`` had become an incident transcript -- mamba 2.6 cache layout, libmamba URL parity, ``repodata_record.json`` healing, lockfile carry-forward healing, orphan policy, envvar escape semantics, and solver subprocess orchestration all glued together with long comments explaining why each new bug-fix bolt-on is safe. That structure buries hard-won production knowledge as private helpers inside a legacy mega-module. This refactor makes the architecture visible in module names instead of comments. Module layout ------------- ``conda_lock/solver/repodata_cache.py`` (459 LOC, was inline in conda_solver): - URL normalization with libmamba parity (``compare_cleaned_url``, ``remove_secrets_and_login_credentials``). - Hierarchical and legacy candidate-path derivation (``candidate_record_paths``). - Identity validation between cache records and LINK actions (``record_matches_link``). - mamba 2.1.1-2.3.3 stub-record detection (``is_mamba_2_1_to_2_3_stub_record`` -- renamed from ``_is_mamba_2x_corrupt_record``; the new name says what version range it matches and what shape it detects). - ``info/index.json``-based heal of those stubs (``heal_corrupt_record``). - ``get_repodata_record``, ``get_pkgs_dirs``, ``warn_on_pkgs_dirs_leak``. ``conda_lock/solver/dry_run.py`` (166 LOC): - ``link_action_as_fetch``: rich-LINK to FETCH synthesis (mamba 2.6.0 fast path), with the corruption-signature check at the threshold so a pass-through corrupt record can't ride into a synthesized FETCH. - ``reconstruct_fetch_actions``: orchestrates rich-LINK fast-path then disk-fallback via repodata_cache. ``conda_lock/solver/lockfile_heal.py`` (296 LOC): - ``LockfileHealReport`` dataclass with three explicit tuples (``healed``, ``confirmed_legit_empty``, ``ambiguous``). The reviewer flagged that the previous integer return threw away exactly the information a caller or test needs to reason about what happened. Reverse-impossible classification is now part of the contract: a name can be in at most one tuple, so orchestration code cannot accidentally collapse legit-empty with corrupt-but-unprovable evidence. - ``heal_locked_dependencies_from_cache`` (renamed from the underscored private helper) is now silent: it classifies, it doesn't log. Logging policy belongs to the orchestration layer. - ``find_cache_index_for_locked_dep`` (renamed from ``_find_index_json_for_dep``). ``conda_lock/solver/graph_integrity.py`` (163 LOC): - ``assert_no_orphaned_conda_packages(planned, platform)`` owns the entire orphan policy: definition, no-reverse-propagation rationale, ``python -> pip`` (``add_pip_as_python_dependency``) explanation, ``CONDA_LOCK_ALLOW_ORPHANED_LOCKFILE=1`` escape hatch, and the over-install warning. Promoted from inline inside ``solve_conda`` because, as the reviewer noted, this is one of the central invariants of the lockfile -- not a local cleanup check. ``conda_lock/conda_solver.py`` shrank from 1500 to 629 lines and is now orchestration glue: subprocess invocation for fresh solves and updates, dryrun-to-LockedDependency translation, ``apply_categories``, ``assert_no_orphaned_conda_packages``, and the fake-prefix machinery for ``--update``. The user-facing heal-summary WARNINGs (``healed`` and ``ambiguous`` reports) moved here -- this is the layer that owns user-facing log policy. Test layout ----------- The reviewer flagged the same disease in the 2k+ line ``tests/test_conda_solver.py`` (URL helpers, cache filesystem, solver graph, update path, live e2e all in one file). Split by architectural boundary: - ``tests/unit/test_repodata_cache.py`` (32 tests, 715 LOC): URL helpers, hierarchical-path derivation, record matching, ``get_repodata_record`` retry loop, mamba 2.1.1-2.3.3 stub detection and ``info/index.json``-based heal, ``pkgs_dirs`` leak warning. - ``tests/unit/test_dry_run_actions.py`` (16 tests, 310 LOC): ``link_action_as_fetch`` synthesis and rejection, ``reconstruct_fetch_actions`` rich-LINK and disk-fallback, degraded-path WARNING. - ``tests/component/test_lockfile_heal.py`` (12 tests, 621 LOC): ``heal_locked_dependencies_from_cache`` per-entry classification (with the report-asserting tests rewritten around the new ``LockfileHealReport`` contract), and the two ``update_specs_for_arch`` orchestration tests that drive the full ``healed``/``ambiguous`` log policy. - ``tests/component/test_graph_integrity.py`` (6 tests, 466 LOC): forward-reachability, ``python -> pip`` healthy shape, envvar escape hatch, hard-fail variants. - ``tests/e2e/test_corrupt_repodata_repro.py`` (live conda-forge driven, 300 LOC): drives the full pipeline against the committed PR conda#862 corrupt fixture. - ``tests/support/corrupt_repodata.py`` (145 LOC): shared helpers (cache-warm + corrupt-overlay, ``conda-lock lock`` driver, explicit-render diff). Promoted out of e2e so future component tests can reuse the harness. Naming ------ Promoted private helpers to public-API names where the test suite reaches into them, keeping a single underscore only on genuinely internal helpers (``_libmamba_strip_url_secrets``, ``_normalize_url_for_compare``, ``_hierarchical_cache_subpath``, ``_link_action_explicit_or_derived_url``, ``_locked_dep_as_link_action``, ``_index_matches_link``, ``_parse_match_spec_to_name_version``). The renames intentionally make the concept visible in the function name -- ``is_mamba_2_1_to_2_3_stub_record`` says what range and what shape; ``find_cache_index_for_locked_dep`` says what file and what input. CI / tooling ------------ - ``.github/workflows/test.yml``: ``run-corrupt-cache-repro`` step now points at ``tests/e2e/test_corrupt_repodata_repro.py`` (was ``tests/test_conda_solver.py``). - All callers of the renamed APIs updated; lint, format, and mypy clean across the new layout. - 66 tests pass (32 + 16 + 6 + 12) with no behavior changes; the ``corrupt_cache_repro`` parametrizations stay in the dedicated CI step on linux-64 / shard 1. Net diff: ``conda_solver.py`` 1500 -> 629 LOC. New layout isolates each domain into a module a future maintainer can read in one sitting, with the integration concerns left in the orchestration entry point where they belong. No changes to external API or behavior. https://claude.ai/code/session_01F6kPbdaqoKb9XARYdbRVhd

Capture the failure-mode data from PR conda#862 so subsequent behavioral commits can run against a known-bad cache without needing live conda-forge or Docker. Three pinned-metadata archives represent the three relevant mamba versions: - ``2.1.0-pkgs.tar.gz``: clean baseline. - ``2.1.1-pkgs.tar.gz``: corrupt (the bug's first appearance). - ``2.3.3-pkgs.tar.gz``: corrupt (the bug's last appearance before mamba 2.6.0's fix). Each archive contains ``info/index.json`` plus ``info/repodata_record.json`` exactly as that mamba version wrote them when installing the same explicit lockfile, so overlaying the metadata on a freshly warmed cache produces a "hybrid" cache: real package files + bug-faithful metadata. That is what later end-to-end tests will drive ``conda-lock lock`` against. The ``05-render-explicit-lockfiles.py`` harness plus the per-version ``lockfile-...yml`` outputs lock the reproduction inputs. Add ``tests/support/corrupt_repodata.py`` with the helpers a component test needs to drive the harness: warmed-cache fixture factory, corrupt-metadata overlay, ``conda-lock lock`` / ``conda-lock render --kind=explicit`` driver wrappers, and explicit-lockfile-URL extraction. None of this calls into new production code yet -- it is pure test scaffolding so the next commits can focus on behavior. Exclude ``tests/test-corrupt-repodata/`` from mypy / ty / pre-commit since the stage scripts are Docker-based one-shot data generators, not runtime code. The committed fixtures (.tar.gz / .lock / .yml) are what tests consume. References conda#896, mamba-org/mamba#4052, mamba-org/mamba#4110.

The unit + component tests exercise the cache-side heal, lockfile-side heal, and graph-integrity invariants in isolation with hand-crafted fixtures. They cannot reach the silent-vanishing path that conda#896 actually reports, because that path is only reachable through the full pipeline: ``LINK``-only dryrun -> ``reconstruct_fetch_actions`` -> ``get_repodata_record`` -> cache-side heal -> ``apply_categories`` forward walk -> ``assert_no_orphaned_conda_packages`` -> v1 serialization. Add ``tests/e2e/test_corrupt_repodata_repro.py`` to drive that whole pipeline against the corrupt-metadata fixtures committed in ``tests/test-corrupt-repodata/``. The test parametrizes over ``mamba`` and ``conda-standalone`` (both solvers were affected by conda#896 because both read leaked ``pkgs_dirs``) and asserts: 1. *Structural lockfile equivalence*: parsing both unified lockfiles via ``parse_conda_lock_file`` produces the same set of ``(name, version, category)`` tuples. Catches silent drops and any "rescue into main" miscategorization. 2. *Explicit-URL equivalence*: ``conda-lock render --kind=explicit`` against both unified lockfiles produces identical URL sets. PR conda#862 stage 05's whole point is that "packages are missing from the resulting environment if and only if they are missing from the explicit lockfile," so this is the user-visible bug surface. 3. *Empty-deps regression*: corrupt-cache run does not produce significantly more empty-``dependencies`` entries than the clean-cache run, proving the heal pipeline recovered them. The supporting fixture factory and harness wrappers live in ``tests.support.corrupt_repodata`` (added with the fixture data), so the test body stays focused on the assertions. The test is marked ``corrupt_cache_repro`` for CI isolation in the next commit; locally it runs in ~2-5 minutes per parametrization because it warms a real conda-forge cache and drives full ``conda-lock lock`` and ``conda-lock render`` invocations.

The ``corrupt_cache_repro`` end-to-end test drives live conda-forge in two independent solves (corrupt-cache vs. clean-control) and runs ~2-5 minutes per parametrization. Putting it in the default unit-test collection would dominate every contributor's local pytest run and inflate every PR's CI wall clock for negligible additional confidence on most changes. Three pieces of CI plumbing: 1. ``pyproject.toml`` declares the ``corrupt_cache_repro`` marker and adds ``-m "not corrupt_cache_repro"`` to the default ``addopts``. ``pytest`` invocations from contributors, editors, and the unit-test CI shards exclude the e2e by default. Local opt-in is ``pytest -m corrupt_cache_repro``; bypass-everything is ``pytest -m ""``. 2. ``.github/workflows/test.yml`` adds a dedicated ``run-corrupt-cache-repro`` step that runs once per ``(os, py-version)`` on shard 1 / linux-64 only -- the fixture is a linux-64 cache snapshot from the PR conda#862 harness, so other platforms have nothing to test against. The step uses ``-o "addopts=-rsx --verbose"`` to clear the project-default exclusion and ``-m corrupt_cache_repro`` to opt back into just the e2e parametrizations. 3. The CI step targets ``tests/e2e/test_corrupt_repodata_repro.py`` directly so a future test author who marks an unrelated test with ``corrupt_cache_repro`` doesn't accidentally end up in the heavy CI lane. Behavior of the unit + component tests is unchanged.