Description
In a multi-tenant environment like Kubeflow, users typically operate with namespace-scoped ServiceAccounts (e.g., default-editor in a specific user namespace) and do not have cluster-wide read permissions.
When attempting to use the Botkube kubernetes source plugin scoped to a specific namespace (using namespaces.include: ["my-namespace"]) and assigning a namespace-scoped ServiceAccount via RBAC context, the plugin fails.
Based on the implementation details, it appears the plugin utilizes a cluster-scoped SharedInformerFactory during startup. This architecture demands cluster-wide list and watch permissions regardless of the namespaces.include configuration. The namespace filtering is executed in-memory only after attempting to fetch cluster-wide resources, making it impossible to use the plugin in strict namespace-scoped RBAC environments.
Interestingly, the botkube/kubectl executor plugin works perfectly fine in the exact same environment using the identical namespace-scoped ServiceAccount. Because the kubectl plugin makes on-demand API requests explicitly scoped to the target namespace (e.g., /api/v1/namespaces/my-namespace/pods), it does not require a ClusterRole. The issue is strictly isolated to how the kubernetes source plugin initializes its informers.
Expected behavior
When namespaces.include is configured with exact namespace values (and no .*), the kubernetes source plugin should initialize namespace-scoped informers (e.g., by utilizing SharedInformerFactoryWithOptions with WithNamespace in client configuration).
This would allow the plugin to request and monitor resources exclusively within the specified namespaces, enabling it to function correctly using only a namespace-scoped Role (get, list, watch within the namespace), just like the botkube/kubectl plugin does.
Actual behavior
The kubernetes plugin always attempts a cluster-wide list and watch API request on startup.
If the mapped ServiceAccount (e.g., Kubeflow's default-editor) only has namespace-level permissions, the API server rejects the request with a 403 Forbidden ... at the cluster scope error. This is in direct contrast to the botkube/kubectl plugin, which correctly respects the namespace scope and executes successfully without any 403 errors.
If we attempt to bypass Botkube's RBAC impersonation by setting context.rbac: {} and rely on a Pod ServiceAccount that only has namespace-scoped RoleBinding, the plugin prints "Start source streaming..." but silently fails in the background. The internal cache reflector falls into a CrashLoop because of the cluster-scope denial, resulting in no events being streamed.
(Additionally, debugging this is difficult because of the alphabetical sorting logic where the first source's log level overrides the others, hiding the background reflector errors even if log.level: debug is set on the target source.)
Steps to reproduce
Install Botkube in an environment where the target ServiceAccount only has namespace-level permissions (e.g., a Kubeflow environment with the default-editor SA).
Configure the kubernetes source plugin to monitor only a specific namespace, and bind the namespace-scoped SA:
Observe the Botkube logs. It throws a 403 Forbidden error because the internal informer attempts to list pods at the cluster scope (/api/v1/pods) instead of the namespace scope (/api/v1/namespaces/kbm-g-np-motif/pods).
Alternatively, change context.rbac: {} to use the base Pod SA with namespace-only permissions. Observe that it logs "Start source streaming..." but no events are ever captured due to silent background permission failures.
Trigger an action using the botkube/kubectl plugin with the same ServiceAccount, and observe that it works perfectly without permission issues.
Description
In a multi-tenant environment like Kubeflow, users typically operate with namespace-scoped ServiceAccounts (e.g., default-editor in a specific user namespace) and do not have cluster-wide read permissions.
When attempting to use the Botkube kubernetes source plugin scoped to a specific namespace (using namespaces.include: ["my-namespace"]) and assigning a namespace-scoped ServiceAccount via RBAC context, the plugin fails.
Based on the implementation details, it appears the plugin utilizes a cluster-scoped SharedInformerFactory during startup. This architecture demands cluster-wide list and watch permissions regardless of the namespaces.include configuration. The namespace filtering is executed in-memory only after attempting to fetch cluster-wide resources, making it impossible to use the plugin in strict namespace-scoped RBAC environments.
Interestingly, the botkube/kubectl executor plugin works perfectly fine in the exact same environment using the identical namespace-scoped ServiceAccount. Because the kubectl plugin makes on-demand API requests explicitly scoped to the target namespace (e.g., /api/v1/namespaces/my-namespace/pods), it does not require a ClusterRole. The issue is strictly isolated to how the kubernetes source plugin initializes its informers.
Expected behavior
When namespaces.include is configured with exact namespace values (and no .*), the kubernetes source plugin should initialize namespace-scoped informers (e.g., by utilizing SharedInformerFactoryWithOptions with WithNamespace in client configuration).
This would allow the plugin to request and monitor resources exclusively within the specified namespaces, enabling it to function correctly using only a namespace-scoped Role (get, list, watch within the namespace), just like the botkube/kubectl plugin does.
Actual behavior
The kubernetes plugin always attempts a cluster-wide list and watch API request on startup.
If the mapped ServiceAccount (e.g., Kubeflow's default-editor) only has namespace-level permissions, the API server rejects the request with a 403 Forbidden ... at the cluster scope error. This is in direct contrast to the botkube/kubectl plugin, which correctly respects the namespace scope and executes successfully without any 403 errors.
If we attempt to bypass Botkube's RBAC impersonation by setting context.rbac: {} and rely on a Pod ServiceAccount that only has namespace-scoped RoleBinding, the plugin prints "Start source streaming..." but silently fails in the background. The internal cache reflector falls into a CrashLoop because of the cluster-scope denial, resulting in no events being streamed.
(Additionally, debugging this is difficult because of the alphabetical sorting logic where the first source's log level overrides the others, hiding the background reflector errors even if log.level: debug is set on the target source.)
Steps to reproduce
Install Botkube in an environment where the target ServiceAccount only has namespace-level permissions (e.g., a Kubeflow environment with the default-editor SA).
Configure the kubernetes source plugin to monitor only a specific namespace, and bind the namespace-scoped SA:
Observe the Botkube logs. It throws a 403 Forbidden error because the internal informer attempts to list pods at the cluster scope (/api/v1/pods) instead of the namespace scope (/api/v1/namespaces/kbm-g-np-motif/pods).
Alternatively, change context.rbac: {} to use the base Pod SA with namespace-only permissions. Observe that it logs "Start source streaming..." but no events are ever captured due to silent background permission failures.
Trigger an action using the botkube/kubectl plugin with the same ServiceAccount, and observe that it works perfectly without permission issues.