feat(infra): Centralized MLflow server for team LabOps with DVC + audit dashboards

[petteriTeikari]

Context

Our current MLflow setup (/home/petteri/mlruns/) works well for single-researcher use, but the whole point of having an MLflow server is to share experiments across team members. A university lab (or any research group) needs:

1. A central experiment store — so the PI and all lab members can view, compare, and audit experiments from one place
2. Data versioning — so datasets are synchronized and version-controlled across local, lab server, and cloud environments
3. A simplified dashboard — because PIs and science managers don't need to see every MLflow artifact; they need high-level summaries

This issue describes the architecture for a LabOps stack: centralized MLflow + DVC for data versioning + a simplified audit dashboard.

Data sensitivity note: Many research institutes, hospitals, and government labs cannot send any data outside their premises (IRB constraints, HIPAA, GDPR, institutional policy). The on-prem options below are designed for exactly this scenario — everything stays on your network.

1. On-Prem MLflow Server (primary recommendation)

Why on-prem first

For clinical/biomedical research labs, on-prem is often not optional — it's mandatory. Patient data, IRB protocols, and institutional IT policies frequently prohibit any data leaving the facility network. Even metadata (experiment names, parameter values) can be sensitive if it encodes patient cohort information.

Self-hosting also means:

• Zero ongoing subscription cost — only electricity and hardware
• Full control over access — firewall rules, VPN-only access, no third-party data processing agreements needed
• Easier IRB/ethics board approval — you can demonstrate complete data sovereignty

Architecture

┌─────────────────────────────────────────────────────┐
│                 Lab Intranet                          │
│                                                      │
│  Researcher A ──┐                                    │
│  Researcher B ──┼──→ MLflow Server (http://lab:5000) │
│  Researcher C ──┘         │            │             │
│                    PostgreSQL     MinIO/NFS           │
│                    (metadata)    (artifacts)          │
└─────────────────────────────────────────────────────┘

Docker Compose (production-ready)

The MLflow repository itself now includes a Docker Compose setup for quick evaluation. For a lab deployment, use PostgreSQL + MinIO:

# docker-compose.yml
services:
  postgres:
    image: postgres:17
    environment:
      POSTGRES_DB: mlflow
      POSTGRES_USER: mlflow
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: unless-stopped

  minio:
    image: minio/minio
    ports:
      - "9000:9000"
      - "9001:9001"  # MinIO console
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    volumes:
      - mlartifacts:/data
    command: server /data --console-address ":9001"
    restart: unless-stopped

  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    ports:
      - "5000:5000"
    depends_on:
      - postgres
      - minio
    environment:
      MLFLOW_BACKEND_STORE_URI: postgresql://mlflow:${POSTGRES_PASSWORD}@postgres:5432/mlflow
      MLFLOW_S3_ENDPOINT_URL: http://minio:9000
      AWS_ACCESS_KEY_ID: ${MINIO_ROOT_USER}
      AWS_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD}
    command: >
      mlflow server
        --host 0.0.0.0
        --port 5000
        --backend-store-uri postgresql://mlflow:${POSTGRES_PASSWORD}@postgres:5432/mlflow
        --default-artifact-root s3://mlflow-artifacts/
    restart: unless-stopped

volumes:
  pgdata:
  mlartifacts:

Team members point their experiments here:

import mlflow
mlflow.set_tracking_uri("http://lab-server:5000")
mlflow.set_experiment("foundation-PLR/classification")

Detailed setup guides:

• Erik Dao: Production-Ready MLflow Setup — full Docker Compose walkthrough
• MinIO Blog: Setting Up MLflow with MinIO (2025)
• Official MLflow Remote Server Tutorial — uses Docker Compose with PostgreSQL + MinIO

Security for intranet deployment

For a server exposed only on the lab LAN, basic HTTP auth + Nginx with TLS is sufficient. Vladimir Rusakov's "MLflow for Poor Man" Part 2 walks through adding Nginx for traffic encryption and authentication. For labs behind NAT / dynamic IP, Part 3 covers OpenVPN as a reverse proxy.

2. Hardware Requirements

MLflow has no official hardware specification document — there's an open issue requesting one. The guidance below is synthesized from community reports, the neptune.ai cost analysis, and the component docs.

MLflow Tracking Server

The tracking server itself is lightweight. A fresh install with zero load consumes ~400 MB of RAM.

Tier	Users	CPU	RAM	Notes
Minimal (RPi 4)	1-3	4-core ARM	4 GB	Proven to work (Rusakov series)
Small lab	3-10	2-4 cores	8 GB	Intel NUC, mini PC, or repurposed desktop
Department	10-30	4-8 cores	16-32 GB	Rack server or beefy workstation

PostgreSQL (metadata backend)

PostgreSQL stores experiment metadata (run parameters, metrics, tags) — typically only a few hundred MB even for thousands of runs. It benefits enormously from SSD random I/O:

System RAM	shared_buffers	work_mem	Notes
4 GB	1 GB	256 MB	Minimum for responsive queries
8 GB	2 GB	1 GB	Comfortable for years of experiments

Use PGTune to calculate optimal settings for your specific hardware. See also: Why Your MLflow Server Needs PostgreSQL.

MinIO (artifact storage)

MinIO stores the actual artifacts (model pickles, plots, bootstrap results). Its hardware checklist recommends 32 GB RAM and NVMe SSDs for production, but for a lab-scale single-node deployment:

• 8 GB RAM is sufficient (set CI_CD=true env var to reduce pre-allocation to 256 MB for tiny machines)
• XFS filesystem recommended for best performance
• SSD strongly preferred — MinIO does not recommend HDD for production

Storage: SSD vs HDD (this matters!)

Factor	SSD	HDD	Impact on MLflow
Random read IOPS	10,000-1,000,000	75-200	PostgreSQL queries, MinIO metadata lookups
Sequential read	500-7,000 MB/s	80-160 MB/s	Artifact download speed (model files)
Latency	0.02-0.1 ms	3-12 ms	MLflow UI responsiveness

Bottom line: SSD makes the MLflow UI feel instant. HDD makes it feel sluggish, especially when browsing experiments with many runs. For a small lab, a single 500 GB-1 TB NVMe SSD is the single best investment.

Recommended hardware tiers

Tier 1: Raspberry Pi 4 (~$100)

Proven by Rusakov's "MLflow for Poor Man" series. A Pi 4 with 4 GB RAM running Docker Compose (MLflow + PostgreSQL + MinIO) served a team of 3 data scientists.

Critical: Use an external SSD via USB 3.0, not the microSD card. The SD card would be too slow for PostgreSQL random I/O and would wear out from write cycles. A USB-attached SSD transforms the Pi from "barely usable" to "surprisingly responsive."

Component	Spec	Cost
Raspberry Pi 4 (4 GB)	ARM Cortex-A72, 4 GB RAM	~$55
USB 3.0 SSD enclosure + 256 GB SSD	SATA SSD	~$35
Power supply + case	Official PSU	~$15
Total		~$105

Best for: Single researcher wanting remote access, or a very small lab (2-3 people) with modest experiment volume.

Tier 2: Intel NUC / Mini PC (~$400-700)

The pragmatic sweet spot for most small-to-medium labs. x86 architecture avoids ARM compatibility issues, and 16 GB RAM gives comfortable headroom.

Component	Spec	Cost
Intel NUC / Beelink / Minisforum	i5/Ryzen 5, 16 GB RAM	$350-600
1 TB NVMe SSD	Internal	$80-100 (or included)
Total		~$400-700

Best for: Labs of 3-10 researchers, years of experiment history, responsive UI.

Tier 3: NVIDIA Jetson (if you also need edge inference)

A Jetson Orin Nano (8 GB) or Orin NX (16 GB) can host the MLflow stack and serve as an inference device. This only makes sense if you have a dual use case (e.g., deploying a screening model to a clinic while also tracking experiments). Otherwise an Intel NUC gives you more RAM and storage per dollar.

Tier 4: Repurposed workstation / rack server

Most labs have old workstations gathering dust. A machine with 16-32 GB RAM and any SSD makes an excellent MLflow server for a department of 10-30 users. Install Ubuntu Server LTS, Docker, and the compose file above.

Network considerations

• Gigabit Ethernet is sufficient for artifact transfer within a building
• WiFi works but adds latency to artifact uploads (large pickle files)
• For multi-site access (e.g., hospital + university campus), use VPN rather than exposing the server to the internet

3. Dataset Versioning with DVC + S3 (or on-prem MinIO)

The MTI Lab blog post on DVC + S3 describes exactly the problem university labs face:

"Unlike companies, university labs are often unable to allocate their human resources to manage their infrastructure and data. In particular, small and medium-sized labs have no choice but to maintain their data in an ad-hoc manner."

Their proposed architecture synchronizes datasets across local machines, lab servers, and cloud compute using DVC backed by S3:

Local laptop  ←──DVC pull/push──→  S3 bucket  ←──DVC pull/push──→  Lab server
                                       ↕
                                  Cloud compute (ABCI, Lambda, etc.)

For data-sensitive labs, replace S3 with the same MinIO instance that stores MLflow artifacts. DVC supports S3-compatible backends:

dvc remote add -d lab_storage s3://dvc-datasets \
  --endpointurl http://lab-server:9000

Now both experiment artifacts and versioned datasets live on the same on-prem box. No data leaves the building.

Integration with our pipeline

configs/mlflow_registry/  ← Git-versioned (method registry, hyperparams)
data/public/              ← DVC-versioned (DuckDB databases, extracted metrics)
/home/petteri/mlruns/     ← MLflow-tracked (experiment runs, bootstrap results)

For our PLR pipeline specifically, DVC would version:

• The extracted DuckDB databases (foundation_plr_results.db)
• Raw PLR signal databases (large binary, currently Dropbox-synced)
• Bootstrap pickle files (542 files, ~1000 iterations each)

4. Cloud / Managed Alternatives

DagsHub — managed MLflow + DVC + Git (low-ops alternative)

For labs that can send data externally and want zero infrastructure maintenance, DagsHub provides a managed MLflow tracking server integrated with DVC and Git:

• Managed MLflow: Every DagsHub repo comes with a free MLflow tracking server (docs)
• DVC integration: DVC remotes are built in — no S3/MinIO setup needed
• Git-native: Standard Git workflow, familiar to researchers
• Free tier: Sufficient for small-to-medium labs

# Point experiments to DagsHub's managed MLflow
import mlflow, os
mlflow.set_tracking_uri("https://dagshub.com/<org>/<repo>.mlflow")
os.environ["MLFLOW_TRACKING_USERNAME"] = "<token>"
os.environ["MLFLOW_TRACKING_PASSWORD"] = "<token>"

This is the path of least resistance for labs where the PI wants experiment visibility without maintaining infrastructure, and where data sensitivity is not a constraint.

AWS / GCP hosted

For labs with cloud budgets, an AWS m5.large (~$70/month on-demand, ~$42/month reserved) with RDS PostgreSQL and S3 is a turnkey solution. See the neptune.ai cost analysis for detailed cost breakdowns — their key finding is that maintenance personnel costs often exceed hosting costs.

5. Simplified Audit Dashboard for PIs (LabOps)

The MLflow UI shows every artifact, parameter, and metric — useful for the researcher running experiments, overwhelming for the PI doing oversight. A science management dashboard would surface:

What PIs actually need to see

View	Content	Update frequency
Experiment health	How many runs completed vs failed this week?	Daily
Metric trends	Is AUROC/calibration improving across pipeline iterations?	Per experiment batch
Resource usage	GPU hours, storage growth, cost projection	Weekly
Reproducibility status	Which experiments have full provenance chains?	Continuous
Student activity	Who ran what, when (audit trail, not micromanagement)	Continuous

Implementation options

Option A: Grafana + MLflow PostgreSQL backend

• Grafana connects to the PostgreSQL backend directly (same on-prem box)
• Custom dashboards with time series of metrics, experiment counts, resource usage
• Alerting (e.g., "no experiments logged in 7 days", "calibration slope outside [0.8, 1.2]")
• The MLflow REST API can feed a lightweight ETL

Option B: Streamlit / Panel app

• Python-native, quick to build, runs on the same server
• Pull from MLflow API, render simplified views
• Lower maintenance than Grafana but less polished

Option C: DagsHub's built-in UI (if using managed option)

STRATOS compliance overlay

For clinical prediction model research, the dashboard should enforce STRATOS reporting (Van Calster 2024):

• Flag experiments that only report AUROC without calibration/DCA
• Highlight calibration slope drift across pipeline configurations
• Surface net benefit comparisons at clinical thresholds

This is continuous auditing — catching compliance issues as experiments are logged, not at paper submission time.

6. The LabOps Vision

Borrowing from MTI Lab's LabOps concept:

"We will define LabOps as a system of technologies that facilitates research in a laboratory. It is the university laboratory version of DevOps and MLOps."

The full stack for a research lab:

┌─────────────────────────────────────────────────────────┐
│                    PI / Lab Dashboard                     │
│  (Grafana / Streamlit / DagsHub UI)                      │
│  Experiment health · Metric trends · Reproducibility     │
├─────────────────────────────────────────────────────────┤
│                  Experiment Tracking                      │
│  MLflow (on-prem Docker Compose or DagsHub-managed)      │
│  Centralized for all lab members                         │
├─────────────────────────────────────────────────────────┤
│                  Data Versioning                          │
│  DVC + MinIO (on-prem) or DVC + S3 (cloud)               │
│  Dataset identity guaranteed across environments         │
├─────────────────────────────────────────────────────────┤
│                  Code Versioning                          │
│  Git + GitHub/GitLab                                     │
│  Reproducible configs (Hydra), registry-as-source        │
├─────────────────────────────────────────────────────────┤
│                  Compute                                  │
│  Local · Lab server · Cloud (Lambda, ABCI, etc.)         │
│  All pointing to same MLflow + DVC remote                │
└─────────────────────────────────────────────────────────┘

Scope & Priority

This is an aspirational / infrastructure issue. It documents the architecture for when this project (or the lab) scales beyond single-researcher use.

Immediate value even for one person: Moving from file-based mlruns/ to a proper MLflow server with PostgreSQL improves query performance, enables remote access from multiple machines, and provides a backup-friendly architecture (just backup the Docker volumes).

References

On-prem MLflow deployment

• MLflow Official Docker Compose — the simplest path to a reproducible on-prem deployment
• Erik Dao: Production-Ready MLflow Setup — full Docker Compose walkthrough with PostgreSQL + MinIO
• MinIO Blog: Setting Up MLflow with MinIO (2025)
• MLflow Official Remote Server Tutorial
• MLflow Tracking Server Architecture — --artifacts-only mode for scaling

Hardware and sizing

• MLflow GitHub Issue #10658 — Add hardware specs to docs (open, no official answer yet)
• MLflow GitHub Issue #5332 — ~400 MB baseline memory
• Neptune.ai: The Real Cost of Self-Hosting MLflow — infrastructure cost analysis
• Vladimir Rusakov: MLflow for Poor Man (Raspberry Pi 4) — proven 3-user deployment on RPi 4 + SSD
  • Part 2: Nginx + TLS
  • Part 3: OpenVPN reverse proxy
• MinIO Hardware Checklist
• PGTune Calculator — optimal PostgreSQL settings for your hardware
• Cast.ai: 7 Best Practices for Running MLflow Cost-Effectively

Data versioning and LabOps

• MTI Lab: Versioning a shared dataset using DVC and S3 — university lab data management, introduces the LabOps concept
• DVC Data Registries — pattern for shared dataset management

Managed alternatives

• DagsHub — managed platform integrating Git, DVC, and MLflow
• DagsHub MLflow integration docs — managed MLflow with DVC and Git version control

Methodology

• Van Calster B, Collins GS, Vickers AJ, et al. (2024) — STRATOS Initiative performance evaluation guidelines