feat: add cross-platform model parameter effectiveness harness

[arumajirou]

Summary

Implements TAJ-58 as a reusable repository-owned parameter-effectiveness harness instead of one-off Windows/Linux scripts.

Core behavior

• Paired control/treatment execution with identical seed/repeat.
• Minimum two distinct seeds enforced by contract.
• Effect surfaces: acceptance, trial count/search budget, history, prediction, metric, runtime.
• Expected relations: change, increase, decrease, invariant.
• Verdicts distinguish effective, accepted-no-observable-effect, expectation violation, inconclusive, unsupported and failed.
• Numeric effect evidence stores count/mean/std/min/max/conservative worst.
• Actual adapter runs require success + accepted + finite output before a pair is eligible.
• Evidence bundle: suite/results/environment/summary/manifest/SHA256SUMS with run ID and platform/Python provenance.
• Explicitly Development/synthetic only: Holdout and Prospective are never consumed.

Built-in adapters

• MLForecast / AutoMLForecast: real Auto model fit/predict, Optuna trial count, prediction SHA, output shape, finite check, synthetic Hit@±1, runtime. Supports current Auto model family and known AutoMLForecast constructor/fit arguments.
• StatsForecast: real model-constructor argument probing through StatsForecast.forecast with deterministic synthetic data, prediction SHA, shape/finite, Hit@±1 and runtime.
• Adapter protocol/registry documented for NeuralForecast, Darts, sktime, Time-Series-Library, BasicTS and other providers without changing the core engine.

Cross-platform verification

Adds a dedicated GitHub Actions matrix for ubuntu-latest and windows-latest on Python 3.13. It runs focused Ruff, mypy, core tests, real MLForecast/StatsForecast adapter tests and the same committed JSON CLI suite on both OSes. It deliberately avoids full extras/Ray on Windows.

Initial real parameter probes

• MLForecast AutoLinearRegression: num_samples 1 -> 2 must increase real observed Optuna trial count on seeds 1 and 42.
• StatsForecast SeasonalNaive: season_length 2 -> 7 must change real predictions on seeds 1 and 42.

Governance

• Primary framework purpose is parameter-effectiveness evidence, not unbiased accuracy promotion.
• No Holdout scoring.
• No Prospective scoring.
• Formal runtime/device/GPU certification remains a separate gate.

Linear: TAJ-58

Summary by Sourcery

Introduce a reusable cross-platform parameter-effectiveness harness for forecasting model/library arguments, with built-in MLForecast and StatsForecast adapters, a CLI and evidence bundle, tests, docs, and dedicated CI.

New Features:

• Add parameter-effectiveness core engine with adapter registry, probe evaluation and suite execution producing structured verdicts and numeric aggregates.
• Provide built-in MLForecast and StatsForecast adapters for probing real model arguments against defined effect surfaces using synthetic Development data.
• Expose a CLI for running JSON-defined parameter-effectiveness suites and writing hashed evidence bundles for cross-platform comparison.

Enhancements:

• Document the parameter-effectiveness framework, supported effect surfaces, verdicts, adapters and extension protocol for additional forecasting libraries.

CI:

• Add a focused cross-platform GitHub Actions workflow that runs linting, type checks, parameter-effectiveness tests and a reusable JSON CLI smoke suite on Ubuntu and Windows.

Tests:

• Add unit tests for the core engine, evidence bundle generation and adapter behavior, plus cross-library probes validating real trial count and prediction changes.

[sourcery-ai]

Reviewer's Guide

Introduces a reusable, cross-platform parameter-effectiveness harness for forecasting libraries, with strict pydantic contracts, a core pairing/aggregation engine, built-in MLForecast and StatsForecast adapters, a JSON-driven CLI, focused tests, documentation, and a dedicated Windows/Linux CI workflow on Python 3.13.

Sequence diagram for running a parameter-effectiveness JSON suite via CLI

sequenceDiagram
  actor User
  participant cli_main as cli_main
  participant Suite as ParameterSuiteSpec
  participant Registry as AdapterRegistry
  participant Adapter as ParameterProbeAdapter
  participant Core as evaluate_probe

  User->>cli_main: main(["--spec","--output"])
  cli_main->>Suite: model_validate_json(spec)
  cli_main->>Registry: default_registry()
  cli_main->>Core: run_suite(suite,registry,output_dir)

  loop for each ParameterProbeSpec in suite.probes
    Core->>Registry: get(spec.library)
    Registry-->>Core: adapter
    Core->>Adapter: supports(spec)
    Adapter-->>Core: (supported,reason)
    alt supported
      loop for each seed,repeat
        Core->>Adapter: run(spec,control,seed,repeat)
        Adapter-->>Core: ProbeRunObservation
        Core->>Adapter: run(spec,treatment,seed,repeat)
        Adapter-->>Core: ProbeRunObservation
        Core->>Core: [compare surfaces, aggregate]
      end
    else unsupported
      Core->>Core: outcome=UNSUPPORTED
    end
  end

  Core-->>cli_main: list[ParameterProbeResult]
  cli_main->>cli_main: print summary, serialize JSON
  cli_main-->>User: exit code based on outcomes

Loading

File-Level Changes

Change	Details	Files
Add core parameter-effectiveness engine with adapter registry, probe evaluation, and evidence bundle writing.	Define ParameterProbeAdapter protocol, FunctionProbeAdapter helper, and AdapterRegistry with case-insensitive library lookup and alias support. Implement evaluate_probe to run paired control/treatment adapter calls across multiple seeds/repeats, enforce eligibility (success/accepted/finite), compare effect surfaces against ExpectedRelation, and compute EffectOutcome plus numeric aggregates. Implement run_suite to evaluate all probes in a suite and optional _write_evidence to persist suite, results, environment, CSV summary, manifest, and SHA256SUMS with run IDs and platform/Python provenance.	src/loto/parameter_effectiveness/core.py
Introduce strict pydantic contracts for probes, suites, observations, and results.	Define enums for parameter scope, effect surfaces, expected relations, and effect outcomes to normalize behavior and verdicts. Add ParameterProbeSpec and ParameterSuiteSpec with validation for distinct seeds, differing control/treatment, and unique probe IDs plus tunable tolerances and matching thresholds. Define ProbeRunObservation, PairedProbeObservation, NumericAggregate, and ParameterProbeResult models, including surface_value helpers and explicit holdout/prospective flags.	src/loto/parameter_effectiveness/contracts.py
Provide built-in MLForecast/AutoMLForecast and StatsForecast adapters and a default registry.	Implement MLForecastParameterAdapter to probe AutoMLForecast and Auto model arguments, route AUTO scope to constructor/fit scopes, run deterministic synthetic panel fits with Optuna, and expose trial_count, history, metric, best_value, prediction SHA, shape, and runtime. Implement StatsForecastParameterAdapter to probe model constructor arguments using deterministic synthetic univariate data, StatsForecast.forecast, and expose prediction SHA, shape, metric, and runtime while enforcing supported scopes/surfaces. Expose default_registry that registers MLForecast (with an automlforecast alias) and StatsForecast adapters into an AdapterRegistry without forcing imports at core construction time.	src/loto/parameter_effectiveness/adapters.py
Expose a public parameter_effectiveness package API and JSON-driven CLI for running suites cross-platform.	Add init to re-export core contracts, AdapterRegistry, FunctionProbeAdapter, evaluate_probe, and run_suite as the public API surface. Implement a CLI that parses a JSON ParameterSuiteSpec, executes the suite with default_registry, prints a human-readable summary plus JSON results, writes an evidence bundle via run_suite, and returns a non-zero exit code when outcomes are blocking (failed/unsupported/inconclusive/expectation_violated).	src/loto/parameter_effectiveness/__init__.py src/loto/parameter_effectiveness/cli.py
Add focused unit tests for the core engine, evidence writing, contracts, and real MLForecast/StatsForecast adapters.	Test core behavior around effective versus accepted-no-observable-effect outcomes, invariant violations, partial failures causing inconclusive results, evidence bundle contents and SHA256SUMS integrity, and contract rejection of single-seed probes. Add adapter-level tests that run real MLForecast AutoLinearRegression num_samples and StatsForecast SeasonalNaive season_length probes to assert expected EFFECTIVE outcomes, finite outputs, populated shapes, and prediction hashes.	tests/parameter_effectiveness/test_core.py tests/parameter_effectiveness/test_library_adapters.py
Document and wire CI for cross-platform parameter-effectiveness validation with a reusable example suite.	Add PARAMETER_EFFECTIVENESS.md describing the harness purpose, effect surfaces, verdict semantics, built-in adapter behavior, JSON suite usage, extension protocol for new libraries, CI shape, and its relationship to accuracy/runtime certification governance. Add a GitHub Actions workflow that runs focused Ruff and mypy, core and adapter pytest, and the JSON CLI smoke suite on both ubuntu-latest and windows-latest with Python 3.13 and constrained dependencies, then verifies the working tree is clean. Introduce an examples/parameter_effectiveness/cross_platform_smoke.json suite (partially shown) used by tests and CI to validate real MLForecast/StatsForecast parameter-effectiveness probes across platforms.	docs/PARAMETER_EFFECTIVENESS.md .github/workflows/parameter-effectiveness-ci.yml examples/parameter_effectiveness/cross_platform_smoke.json

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[arumajirou]

Focused verification update for head 33fd333db5bba7d283caab05b0de66e8128158c5:

• parameter-effectiveness-ci Ubuntu: PASS
• parameter-effectiveness-ci Windows: PASS
• Both OSes passed focused Ruff, focused mypy, real MLForecast/StatsForecast adapter tests, committed JSON CLI smoke, and clean-tree check.
• Windows checkout required a step-scoped core.protectNTFS=false compatibility override because legacy audit evidence contains : in tracked path names. The override is scoped only to the checkout action; it is not persisted globally.
• MLForecast num_samples probe was stabilized with a fixed safe feature configuration so the paired seeds test the requested search-budget effect rather than accidental invalid feature configurations.
• Holdout/Prospective remain unused.

The separate existing windows-portability-ci workflow can still fail on the repository's legacy NTFS-invalid audit paths and is not used as evidence for this focused parameter harness.