pve-pool-watchdog

Daily fstrim and LVM thin pool monitor for Proxmox, with Discord alerts before the pool fills and pauses every guest on it.

Plain bash, cron, and curl. No LLM, no gateway, no API credits in the loop, so it keeps working when everything else is on fire.

Why this exists

On 2026-05-05, pve/data on a Proxmox host hit 100% during a scheduled update and paused a running VM with I/O errors. LVM thin pools overcommit space, and guests never hand freed blocks back to the pool without a trim. Once a thin pool's data or metadata fills, every guest on it freezes at once, not just the one that triggered it.

This watchdog runs every guest's fstrim first, so freed blocks actually get returned to the pool, then checks how full the pool is and posts to Discord if it crosses a threshold.

Demo

A normal run, from /var/log/pve-pool-watchdog.log:

2026-07-03 04:30:01 fstrim CT 101 ok
2026-07-03 04:30:02 fstrim CT 105 ok
2026-07-03 04:30:04 fstrim CT 113 ok
2026-07-03 04:30:07 fstrim VM 100 ok
2026-07-03 04:30:09 fstrim VM 104 skipped: guest agent is not running
2026-07-03 04:30:09 fstrim done: CTs ok=3 failed=0, VMs ok=1 skipped=1
2026-07-03 04:30:09 pool pve/data: data 62% | meta 41% | ~132G free of 348G [ok]
2026-07-03 04:30:09 all pools ok, no alert

A run that crosses the warning threshold sends a Discord message like:

:warning: **thin pool warning on `pve1`**
pve/data: data 83% | meta 52% | ~59G free of 348G [WARN]
Biggest thin volumes:
  vm-100-disk-0 210G of 250G (84%)
  vm-104-disk-0 95G of 150G (63%)
  ct-113-disk-0 40G of 64G (63%)
fstrim: 3 CTs + 1 VMs trimmed (0 CT fail, 1 VM no-agent)
Worth a look before it becomes a 3am problem.

And past the critical threshold:

:rotating_light: **thin pool CRITICAL on `pve1`** (data >= 90% or meta >= 80%)
pve/data: data 94% | meta 61% | ~21G free of 348G [CRIT]
Biggest thin volumes:
  vm-100-disk-0 235G of 250G (94%)
  ...
fstrim: 3 CTs + 1 VMs trimmed (0 CT fail, 1 VM no-agent)
If this fills, every guest on the pool pauses with I/O errors. Free space NOW: delete
old snapshots, unused disks, or move a disk off the pool.

Features

• Trims every running LXC container (pct fstrim) and every running VM with a QEMU guest agent (qm guest cmd fstrim) before checking pool usage, so freed blocks are actually returned before the check runs.
• Reads data% and metadata% for every LVM thin pool on the host from lvs. Metadata gets its own, tighter thresholds since a thin pool can break on full metadata even with data space to spare.
• Discord alert at WARN and CRIT severity, listing every pool's numbers, the three biggest thin volumes by allocated space, and the fstrim tally for the run.
• Silent when everything is healthy. Only writes to the log, no alert noise.
• Per-guest fstrim timeout so one stuck container or VM can't hang the whole run.
• --test forces a Discord post with current stats regardless of thresholds, for checking the webhook works end to end.
• --no-trim skips fstrim for a fast, report-only run.

Run it yourself

What you need

• A Proxmox VE host you already run. This tool only makes sense on a real Proxmox host, it isn't something you install standalone. If you don't run Proxmox yet, this isn't the project to start with.
• Root SSH access to that host. Either a saved SSH key or the root password, whichever you normally use to reach it. Check you can log in with ssh root@<your-host-ip> before going further.
• Bash 4 or newer, which every current Proxmox VE install already has, so there's nothing to install here, just something the script needs.
• (Optional but recommended) a Discord webhook URL, so the watchdog can actually alert you instead of only writing to a log file. In Discord: Server Settings > Integrations > Webhooks > New Webhook, pick or make a channel (an "announcements" or "alerts" channel works well), then Copy Webhook URL. It looks like https://discord.com/api/webhooks/123456789/AbCdEf.... Without this, the tool still runs and still logs, it just never posts anywhere.
• A machine to run the deploy command from. Any Mac, Linux box, or Windows machine with an ssh and scp client works (Windows 10/11 already ship both). This can be the same machine you're reading this on.

Step by step

Get the code first:

git clone https://github.com/KitsuneTech1/pve-pool-watchdog.git
cd pve-pool-watchdog

You should see pve-pool-watchdog.sh, deploy.sh, and an etc/ folder in the directory.

Option A: deploy.sh (recommended, works for one host or many)

deploy.sh copies the watchdog script, its cron schedule, and its log rotation config to any number of Proxmox hosts over SSH, in one command.

1. Run it, listing every host you want the watchdog on (replace with your own root@ip addresses):

   ./deploy.sh root@pve1 root@pve2

   For each host, you should see deployed ok printed. If a host doesn't have a config file yet, you'll also see a note reminding you alerts are disabled until you add one, that's expected on a brand new host, handled in the next step.
2. This step doesn't touch /etc/pve-pool-watchdog.conf on purpose, so re-running deploy.sh later to update the script is always safe and never wipes out a webhook URL you already set. On a host you're deploying to for the first time, though, that config file doesn't exist yet, so alerts stay silent until you create it:

   scp pve-pool-watchdog.conf.example root@<host-ip>:/etc/pve-pool-watchdog.conf
   ssh root@<host-ip> 'chmod 600 /etc/pve-pool-watchdog.conf'

   chmod 600 means only root can read or write the file, which matters here because it holds your webhook URL.
3. Edit that file on the host (ssh root@<host-ip> then nano /etc/pve-pool-watchdog.conf, or any editor you like) and fill in WEBHOOK_URL="https://discord.com/api/webhooks/..." with the URL from Discord. Leave every other line commented out unless you want to change the default thresholds, see the Config table below for what each one does.

It worked if: the command in "Verify end to end" below posts a message to your Discord channel.

Keep it running: nothing to set up, deploy.sh already installed a cron job (see Cron schedule below) that runs it automatically every day. There's no service to start or enable.

Option B: manual (no deploy.sh, one host, full control)

Use this if you'd rather see and run every step yourself, or deploy.sh doesn't work in your environment for some reason.

1. Copy the three files to the host:

   scp pve-pool-watchdog.sh root@HOST:/usr/local/sbin/pve-pool-watchdog.sh
   scp etc/cron.d/pve-pool-watchdog root@HOST:/etc/cron.d/pve-pool-watchdog
   scp etc/logrotate.d/pve-pool-watchdog root@HOST:/etc/logrotate.d/pve-pool-watchdog

2. Set correct permissions and create the log file:

   ssh root@HOST 'chmod 755 /usr/local/sbin/pve-pool-watchdog.sh
     chmod 644 /etc/cron.d/pve-pool-watchdog /etc/logrotate.d/pve-pool-watchdog
     touch /var/log/pve-pool-watchdog.log'

   chmod 755 makes the script executable. chmod 644 makes the cron and logrotate files readable by the system but only root-writable, which is what those files expect.
3. Set up the config file the same way as step 2 in Option A (copy pve-pool-watchdog.conf.example to /etc/pve-pool-watchdog.conf, chmod 600, fill in WEBHOOK_URL).

It worked if: the same "Verify end to end" command below posts to Discord.

Cron schedule

etc/cron.d/pve-pool-watchdog runs the script daily at 04:30, as root, with a 30 minute timeout, so it can never hang forever if something goes wrong. 04:30 is deliberately ahead of most nightly game-server update crons, so the pool gets trimmed and checked right before its riskiest hour.

There's no systemd unit for this project, and you don't need one. Cron is enough for a once-a-day job, and it's one less thing that has to stay running in the background.

Verify end to end

ssh root@HOST '/usr/local/sbin/pve-pool-watchdog.sh --test --no-trim'

--test forces a Discord post with the current pool numbers even when everything is healthy, so you don't have to wait for a real warning to know the webhook works. --no-trim skips the fstrim step so this check runs in a couple of seconds.

It worked if: you see a message show up in your Discord channel within a few seconds, formatted like the examples in the Demo section above.

Troubleshooting

• Nothing shows up in Discord, but the script runs without errors: WEBHOOK_URL is probably still empty in /etc/pve-pool-watchdog.conf. The script treats an empty webhook as "log only, don't alert" on purpose, so this can look like success when it isn't. Check the file has the real URL and is on the host you tested, not just your local repo copy.
• deploy.sh prints "NOTE: /etc/pve-pool-watchdog.conf missing, alerts disabled until created": this is expected on a host's first deploy, not an error. Follow step 2 in Option A to add the config file.
• A VM shows up as "skipped: guest agent is not running": this is normal and not a failure. Only VMs with the QEMU guest agent installed and running can be trimmed from the host side; containers (CTs) don't need this. Install the guest agent inside that VM if you want it trimmed too, otherwise it's safe to ignore.
• The cron job never seems to run: check it actually landed with cat /etc/cron.d/pve-pool-watchdog on the host, and that the file ends with a newline (cron silently ignores files that don't). Also check /var/log/pve-pool-watchdog.log for a line from around 04:30 to confirm it's firing.

Config

Copy pve-pool-watchdog.conf.example to /etc/pve-pool-watchdog.conf and chmod 600 it (it holds the webhook URL). This file is never committed to the repo, it's in .gitignore on purpose.

Key	Default	Meaning
WEBHOOK_URL	(empty)	Discord webhook to post alerts to. If empty, alerts are logged but not sent.
HOST_LABEL	$(hostname)	Name shown in alert messages.
DATA_WARN	80	Pool data% that triggers a WARN alert.
DATA_CRIT	90	Pool data% that triggers a CRIT alert.
META_WARN	70	Pool metadata% that triggers a WARN alert.
META_CRIT	80	Pool metadata% that triggers a CRIT alert.
CT_TRIM_TIMEOUT	180	Seconds to wait for a single container's fstrim before giving up on it.
VM_TRIM_TIMEOUT	180	Seconds to wait for a single VM's fstrim before giving up on it.
SKIP_FSTRIM	0	Set to 1 to never fstrim, report only. Same effect as --no-trim.

How it works

1. fstrim phase. Lists running containers with pct list and running VMs with qm list, then trims each one with a per-guest timeout. A VM without a guest agent fails cleanly and is counted as skipped, not fatal, that's the most common case.
2. Pool check. Reads lvs for every LVM thin pool (lv_attr =~ ^t) on the host, pulling data%, metadata%, and total size. Also pulls the three largest thin volumes by allocated space so an alert points at what's actually eating the pool.
3. Severity. Each pool is scored ok, WARN, or CRIT against the thresholds above. The worst severity across all pools decides whether an alert fires.
4. Alert. CRIT or WARN posts to the configured Discord webhook with per-pool numbers, the top volumes, and the fstrim tally. Healthy runs write to the log only. --test forces a post even when healthy, for checking the webhook end to end.

Uninstall

ssh root@HOST 'rm -f /usr/local/sbin/pve-pool-watchdog.sh /etc/cron.d/pve-pool-watchdog \
  /etc/logrotate.d/pve-pool-watchdog /etc/pve-pool-watchdog.conf'

License

MIT. Copyright (c) 2026 Kitsune Technologies LLC. Do what you like with it, commercially or otherwise, just keep the copyright notice. Full text in LICENSE.