Add Scrutiny Helm chart - #502
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new scrutiny Helm chart to deploy the Scrutiny stack (web UI, collector DaemonSet, and InfluxDB) so SMART disk metrics can be collected and viewed in-cluster.
Changes:
- Introduces a new Helm chart (
charts/scrutiny) with default values and chart metadata. - Adds Kubernetes manifests for Scrutiny Web + InfluxDB Deployments, Services, PVCs, and a collector DaemonSet.
- Adds Helm helper templates for naming/namespace overrides.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| charts/scrutiny/Chart.yaml | Declares the new Helm chart metadata (name/version/appVersion). |
| charts/scrutiny/values.yaml | Provides default images, persistence sizes, tolerations, and collector scheduling. |
| charts/scrutiny/templates/_helpers.tpl | Adds helper templates for name/fullname/namespace. |
| charts/scrutiny/templates/deployment-web.yaml | Deploys the Scrutiny web UI and config PVC mount. |
| charts/scrutiny/templates/service-web.yaml | Exposes the web UI via a ClusterIP Service. |
| charts/scrutiny/templates/deployment-influxdb.yaml | Deploys InfluxDB with a data PVC mount. |
| charts/scrutiny/templates/service-influxdb.yaml | Exposes InfluxDB internally to the web/collector components. |
| charts/scrutiny/templates/pvc-influxdb.yaml | Creates persistent storage for InfluxDB data. |
| charts/scrutiny/templates/pvc-config.yaml | Creates persistent storage for Scrutiny config. |
| charts/scrutiny/templates/daemonset-collector.yaml | Runs the Scrutiny collector on each node with host access for SMART collection. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Volker Theile <vtheile@suse.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| mountPath: /opt/scrutiny/config | ||
| volumes: | ||
| - name: config | ||
| persistentVolumeClaim: | ||
| claimName: {{ include "scrutiny.fullname" . }}-config-pvc |
There was a problem hiding this comment.
The web Deployment mounts a single ReadWriteOnce PVC (config-pvc) but allows scaling via replicas. Scaling above 1 will leave extra replicas Pending due to the RWO volume, and the default RollingUpdate strategy can also stall rollouts when a surge pod tries to mount the same PVC. Consider constraining this component to 1 replica and/or setting spec.strategy.type: Recreate (or switching to StatefulSet / RWX storage) to make updates reliable.
| mountPath: /var/lib/influxdb2 | ||
| volumes: | ||
| - name: influxdb-data | ||
| persistentVolumeClaim: | ||
| claimName: {{ include "scrutiny.fullname" . }}-influxdb-pvc |
There was a problem hiding this comment.
This Deployment mounts a single ReadWriteOnce PVC for /var/lib/influxdb2 but does not set spec.strategy. With the default RollingUpdate strategy, upgrades can stall because a surge pod may be created before the old pod is terminated, and both cannot mount the same RWO volume. Set spec.strategy.type: Recreate (see charts/harvester-vm-import-controller/templates/deployment.yaml:9-10) or use a StatefulSet to make rollouts reliable.
| - effect: NoSchedule | ||
| key: node-role.kubernetes.io/master | ||
| - effect: NoExecute | ||
| operator: Exists |
There was a problem hiding this comment.
The default toleration effect: NoExecute with operator: Exists and no tolerationSeconds tolerates all NoExecute taints indefinitely (including node.kubernetes.io/not-ready / unreachable). For the web/influxdb Deployments this can reduce availability by preventing eviction/rescheduling when a node becomes unhealthy. Consider scoping this toleration to the specific taints you need, or adding tolerationSeconds for node-condition taints.
| operator: Exists | |
| operator: Exists | |
| tolerationSeconds: 300 |
|
Thanks for the PR, some open questions for discussion, (1) How to represent this chart to user. It could be a subchart of rancher-monitoring https://github.com/rancher/charts/tree/dev-v2.14/charts/rancher-monitoring ; or an addon/experimental-addon on Harvester. Rancher mirror all/most of the images by third-party like https://github.com/harvester/harvester-installer/blob/c8fd2f4e80cdeef1d79396ef75c6e49deb4d0ba8/scripts/images/rancher-images.txt#L20 (2) License https://github.com/analogj/scrutiny?tab=readme-ov-file#licenses, I am not an expert on this area. (3) Test plans, coutinuous upgrade plans. maybe we need a HEP. |
|
As discussed, we should consider enhancing our node disk manager to provide enough capacity to monitor disk statistics and export them via our own Prometheus metrics API. When using an upstream chart, consider maintenance, resource consumption, release workload, and feature scope, among other factors. This requires strong justification and analysis to ensure alignment with our project goals. (Otherwise, users can simply install the upstream chart on their own, since there are no integration benefits here.) Suggest reviewing harvester/harvester#7416 (comment) first for ideas from Longhorn SMART monitoring (a minimal implementation of the required capabilities), including kernel-managed disks, filesystems, or user-space disks managed by SPDK, to help shape our design. Once you review this, you'll have a clearer understanding of the effort involved and whether any open-source projects could help us. cc @harvester/storage @harvester/pm |
Problem:
Prometheus node-exporter does not collect SMART values of the disks connected to the cluster nodes.
Solution:
Make use of the scrutiny project which provides a nice small UI to display the SMART values of the disks that are auto-detected on each node in the cluster.
Related Issue(s):
n/a
Test plan:
Check the running pods.
Service Discovery > Servicesin the embedded Rancher. Look out forscrutiny-web. Click on the IP in theTargetcolumn to open the Scrutiny WebUI.Additional documentation or context