04_dependencies_ci.qmd

---
title: "Dependencies and Continuous Integration"
format: 
    html: default
    revealjs:
      output-file: 04_dependencies_ci_slides.html
      slide-number: true
footer: Python package development
---

## Dependencies

⏩ Dependencies are other pieces of software your project needs to work. 

🤔 They save time by reusing existing solutions, but can make your project harder to maintain if they change or are no longer available.

![](images/dependency_tree.png)



##

:::: {.columns}

::: {.column}

**Application**

*A program that is run by a user*

* command line tool
* script
* web application

Pin versions to ensure reproducibility, e.g. `numpy==1.11.0`
:::

::: {.column}

**Library**

*A program that is used by another program*

* *numpy*
* *scikit-learn*

Make the requirements as loose as possible, e.g. `numpy>=1.11.0`
:::

::::

::: {.notes}
Make the requirements loose, to avoid conflicts with other packages.

:::

## Dependency resolution

::: {.incremental}

- Python projects often depend on many packages, which may depend on others ("dependency tree").
- Different packages may require conflicting versions of the same dependency.
- Historically, Python tools did not always resolve these conflicts well, leading to "dependency hell." 😈
- Ensuring everyone gets the same working set of packages has been difficult, especially for larger projects. 🤪

:::

. . .

Modern tools like `uv` aim to solve these problems with faster and more reliable 😎

## uv {.smaller}

* 🚀 A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more.
* ⚡️ 10-100x faster than pip.
* 🗂️ Provides comprehensive project management, with a universal lockfile.
* ❇️ Runs scripts, with support for inline dependency metadata.
* 🐍 Installs and manages Python versions.
* 🛠️ Runs and installs tools published as Python packages.
* 🔩 Includes a pip-compatible interface for a performance boost with a familiar CLI.
* 💾 Disk-space efficient, with a global cache for dependency deduplication.
* ⏬ Installable without Rust or Python via curl or pip.
* 🖥️ Supports macOS, Linux, and Windows.

## Dependency management

`uv` is the recommended tool for managing a Python project including dependencies.

Example of pinning versions:

```{.toml filename="pyproject.toml"}
dependencies = [
  "numpy==1.11.0",
  "scipy==0.17.0",
  "matplotlib==1.5.1",
]
```
. . .

Or using a range of versions:

```{.toml filename="pyproject.toml"}
dependencies = [
  "numpy>=1.11.0",
  "scipy>=0.17.0",
  "matplotlib>=1.5.1,<=2.0.0"
]
```
. . .

Install dependencies:

```{.bash}
$ uv sync
```

## Development dependencies

I.e. dependencies needed for testing, building documentation, linting, etc. not needed to run the package. 

```{.toml filename="pyproject.toml"}
[dependency-groups]
dev = [
    "pytest>=8.4.1",
]
```

## Dependency management using uv

* Add a dependency:
```{.bash}
$ uv add matplotlib
```

* Remove a dependency:
```{.bash}
$ uv remove seaborn
```

* Add development dependency:
```{.bash}
$ uv add --dev pytest
```


## Creating an installable package

Create a new library project:

```{.bash}
$ uv init --lib
```

Start a Python session:
```{.bash}
$ uv run python
```

```{.python}
>>> import mini
>>> mini.foo()
42
```

. . .

Run tests:
```{.bash}
$ uv run pytest
...

tests/test_foo.py .                       [100%]

=============== 1 passed in 0.01s ===============
```


## Virtual environments

::: {.incremental}

* Creates a clean environment for each project
* Allows different versions of a package to coexist on your machine
* Can be used to create a reproducible environment for a project
* Virtual environments are managed by `uv`

:::


## Continuous Integration (CI)

* Running tests on every commit in a well defined environment ensures that the code is working as expected.

* It solves the "it works on my machine" problem.

* Executing code on a remote server is a good way to ensure that the code is working as expected.

* Example of CI services:
  - **GitHub Actions**
  - Azure Pipelines
  - Travis CI


## GitHub Actions {.smaller}

:::: {.columns}

::: {.column}

* Workflow are stored in the `.github/workflows` folder.
* Workflow is described in a YAML file.
* YAML is whitespace sensitive (like Python).
* YAML can contain lists, dictionaries and strings, and can be nested.

:::

::: {.column}

```bash
$ tree mikeio/.github/
mikeio/.github/
└── workflows
    ├── docs.yml
    ├── full_test.yml
    ├── notebooks_test.yml
    ├── perf_test.yml
    └── python-publish.yml
```

:::
::::

---

## Workflow example {.smaller}

```yaml
name: Quick test

on: # when to run the workflow
  push:
    branches: [ main]
  pull_request:
    branches: [ main ]

jobs: # which jobs to run
  build: # descriptive name 🙄
    runs-on: ubuntu-latest # on what operating system

    steps:
    - uses: actions/checkout@v3
    - name: Set up uv
      uses: astral-sh/setup-uv@v6
      with:
        python-version: "3.13"
    - name: Install dependencies
      run: |
        uv sync

    - name: Test with pytest
      run: |
        uv run pytest
```

---

🙂🚀

![](images/github_action_result.png)

---

☹️

![](images/github_action_result_fail.png)

## Benefits of CI

::: {.incremental}

* Run tests on every commit
* Test on different operating systems
* Test on different Python versions
* Create API documentation
* Publish package to PyPI or similar package repository

:::

## Triggers

* `push` and `pull_request` are the most common triggers
* `schedule` can be used to run the workflow on a schedule
* `workflow_dispatch` can be used to trigger the workflow manually

```{.yaml code-line-numbers="2-3|4-5|6-7|8"}
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 0 * * 0'
  workflow_dispatch:
```

## Jobs

* Operating system
* Python version
* ...

```{.yaml}
...
jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
        python-version: ["3.10","3.13"]
...
```

## GitHub Releases {.smaller}

:::: {.columns}

::: {.column}

* GitHub releases are a way to publish software releases.

* You can upload files, write release notes and tag the release.

* As a minimum, the release will contain the source code at the time of the release.

* Creating a release can trigger other workflows, e.g. publishing a package to PyPI.

:::

::: {.column}

![](images/github_release.png)

:::
::::


<https://github.com/pydata/xarray/releases/tag/v2022.12.0>




## Summary

::: {.incremental}

* Application vs library
* Use a separate virtual environment for each project
* Use GitHub Actions to run tests on every commit
* Use GitHub Releases to publish software releases

:::