AKS runs an internal mutating component called the admissionsenforcer (part of the AKS-managed "admissions enforcer" / addon-manager). After KEDA's ValidatingWebhookConfiguration (keda-admission) is created, AKS automatically injects a namespaceSelector into every webhook entry to exclude AKS control-plane / managed namespaces (so KEDA's webhooks never intercept system pods). When it does this, it takes Server-Side Apply (SSA) field ownership of those namespaceSelector fields under the field manager named admissionsenforcer.
So after the first deploy:
- keda-admission webhook exists
- AKS has mutated .webhooks[*].namespaceSelector and now owns those fields
In the second deploy, the KEDA chart tries to re-apply the same ValidatingWebhookConfiguration with the field manager helm. SSA detects that helm wants to set fields already owned by admissionsenforcer → field ownership conflict:
Below is the code trying to setup KEDA.
How to fix
We can use three options to fix the above issue.
- Tell Helm to take over the contested fields by forcing conflict resolution on apply, e.g. Helm's --force-conflicts (Helm 3.18+ exposes --server-side / --force-conflicts) — this lets helm win ownership of namespaceSelector.helm repo add kedacore https://kedacore.github.io/chartshelm repo updatehelm upgrade keda kedacore/keda --install `--namespace keda `--version 2.20.1 `--server-side=true `--force-conflicts `--set serviceAccount.operator.create=true `--set serviceAccount.operator.name=keda-operator `--set podIdentity.azureWorkload.enabled=true `--set podIdentity.azureWorkload.clientId=$aksUserAssignedIdentityClientId `--set podIdentity.azureWorkload.tenantId=$tenantId
- Disable AKS from managing the webhook namespaceSelector — KEDA documents an annotation that opts the webhook out of the AKS admissions enforcer mutation, so AKS stops claiming the field. KEDA's chart exposes a value to add this annotation to the webhook.helm repo add kedacore https://kedacore.github.io/chartshelm repo updatehelm upgrade keda kedacore/keda --install `--namespace keda `--version 2.20.1 `--set-string additionalAnnotations."admissions\.enforcer/disabled"=true ` `--set serviceAccount.operator.create=true `--set serviceAccount.operator.name=keda-operator `--set podIdentity.azureWorkload.enabled=true `--set podIdentity.azureWorkload.clientId=$aksUserAssignedIdentityClientId `--set podIdentity.azureWorkload.tenantId=$tenantId
- Disable/skip KEDA's validating webhook entirely (if you don't need it) so there's nothing for AKS to fight over.helm repo add kedacore https://kedacore.github.io/chartshelm repo updatehelm upgrade keda kedacore/keda --install `--namespace keda `--version 2.20.1 `--set webhooks.enabled=false `--set serviceAccount.operator.create=true `--set serviceAccount.operator.name=keda-operator `--set podIdentity.azureWorkload.enabled=true `--set podIdentity.azureWorkload.clientId=$aksUserAssignedIdentityClientId `--set podIdentity.azureWorkload.tenantId=$tenantId
Pros & Cons of each approach
Option 1: --server-side=true --force-conflicts
Pros:
- Keeps KEDA validating webhook enabled.
- Fixes the current upgrade conflict directly.
- AKS can still re-apply its namespaceSelector changes after the Helm upgrade.
- No need to delete KEDA or the webhook first.
Cons:
- Requires a Helm version that supports server-side upgrade and force conflicts.
- Changes Helm apply behavior for the whole KEDA release, not just the webhook.
- --force-conflicts is a strong override; it can hide future field-ownership conflicts.
- The same ownership fight may happen again on every deployment, so you keep needing the flags.
Option 2: Add AKS admissions enforcer opt-out annotation
Pros:
- Keeps KEDA validating webhook enabled.
- Fixes the root conflict by telling AKS not to modify that webhook.
- Does not require Helm server-side apply.
- No Helm version dependency.
- Cleaner long-term option if you want KEDA validation to stay on.
Cons:
- First deployment on an already-conflicted cluster may still fail once, because AKS already owns the field.
- To unstick the current cluster, you may need a one-time action like deleting keda-admission or doing one forced server-side upgrade.
- AKS will no longer inject its extra namespace selector into KEDA's webhook.
- You must trust KEDA's own webhook scope/selector, which is normally fine because it only validates KEDA CRDs.
- Removes the conflict completely because the webhook object is not created.
- No server-side apply needed.
- No Helm version dependency.
- Simple to understand: no webhook means nothing for AKS to fight over.
- You lose KEDA admission-time validation.
- Bad ScaledObject, ScaledJob, or TriggerAuthentication configs may not be rejected immediately.
- Problems may show up later through operator logs/events instead of a clear apply-time error.
No comments:
Post a Comment