Saturday, 23 May 2026

Fix Keda Upgrade on AKS with Helm Fail Due to Field Ownership Conflicts

 AKS runs an internal mutating component called the admissionsenforcer (part of the AKS-managed "admissions enforcer" / addon-manager). After KEDA's ValidatingWebhookConfiguration (keda-admission) is created, AKS automatically injects a namespaceSelector into every webhook entry to exclude AKS control-plane / managed namespaces (so KEDA's webhooks never intercept system pods). When it does this, it takes Server-Side Apply (SSA) field ownership of those namespaceSelector fields under the field manager named admissionsenforcer. 

So after the first deploy:

  • keda-admission webhook exists
  • AKS has mutated .webhooks[*].namespaceSelector and now owns those fields

In the second deploy, the KEDA chart  tries to re-apply the same ValidatingWebhookConfiguration with the field manager helm. SSA detects that helm wants to set fields already owned by admissionsenforcer → field ownership conflict:


Below is the code trying to setup KEDA.

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

helm upgrade keda kedacore/keda --install `
    --namespace keda `
    --version 2.20.1 `
    --set serviceAccount.operator.create=true `
    --set serviceAccount.operator.name=keda-operator `
    --set podIdentity.azureWorkload.enabled=true `
    --set podIdentity.azureWorkload.clientId=$aksUserAssignedIdentityClientId `
    --set podIdentity.azureWorkload.tenantId=$tenantId


How to fix
We can use three options to fix the above issue.

  1. Tell Helm to take over the contested fields by forcing conflict resolution on apply, e.g. Helm's --force-conflicts (Helm 3.18+ exposes --server-side / --force-conflicts) — this lets helm win ownership of namespaceSelector.

    helm repo add kedacore https://kedacore.github.io/charts
    helm repo update

    helm upgrade keda kedacore/keda --install `
        --namespace keda `
        --version 2.20.1 `
        --server-side=true `
        --force-conflicts `
        --set serviceAccount.operator.create=true `
        --set serviceAccount.operator.name=keda-operator `
        --set podIdentity.azureWorkload.enabled=true `
        --set podIdentity.azureWorkload.clientId=$aksUserAssignedIdentityClientId `
        --set podIdentity.azureWorkload.tenantId=$tenantId

  2. Disable AKS from managing the webhook namespaceSelector — KEDA documents an annotation that opts the webhook out of the AKS admissions enforcer mutation, so AKS stops claiming the field. KEDA's chart exposes a value to add this annotation to the webhook.

    helm repo add kedacore https://kedacore.github.io/charts
    helm repo update

    helm upgrade keda kedacore/keda --install `
        --namespace keda `
        --version 2.20.1 `
        --set-string additionalAnnotations."admissions\.enforcer/disabled"=true ` `
        --set serviceAccount.operator.create=true `
        --set serviceAccount.operator.name=keda-operator `
        --set podIdentity.azureWorkload.enabled=true `
        --set podIdentity.azureWorkload.clientId=$aksUserAssignedIdentityClientId `
        --set podIdentity.azureWorkload.tenantId=$tenantId

  3. Disable/skip KEDA's validating webhook entirely (if you don't need it) so there's nothing for AKS to fight over.

    helm repo add kedacore https://kedacore.github.io/charts
    helm repo update

    helm upgrade keda kedacore/keda --install `
        --namespace keda `
        --version 2.20.1 `
        --set webhooks.enabled=false `
        --set serviceAccount.operator.create=true `
        --set serviceAccount.operator.name=keda-operator `
        --set podIdentity.azureWorkload.enabled=true `
        --set podIdentity.azureWorkload.clientId=$aksUserAssignedIdentityClientId `
        --set podIdentity.azureWorkload.tenantId=$tenantId


Pros & Cons of each approach

Option 1: --server-side=true --force-conflicts

Pros:

  • Keeps KEDA validating webhook enabled.
  • Fixes the current upgrade conflict directly.
  • AKS can still re-apply its namespaceSelector changes after the Helm upgrade.
  • No need to delete KEDA or the webhook first.

Cons:

  • Requires a Helm version that supports server-side upgrade and force conflicts.
  • Changes Helm apply behavior for the whole KEDA release, not just the webhook.
  • --force-conflicts is a strong override; it can hide future field-ownership conflicts.
  • The same ownership fight may happen again on every deployment, so you keep needing the flags.


Option 2: Add AKS admissions enforcer opt-out annotation

Pros:

  • Keeps KEDA validating webhook enabled.
  • Fixes the root conflict by telling AKS not to modify that webhook.
  • Does not require Helm server-side apply.
  • No Helm version dependency.
  • Cleaner long-term option if you want KEDA validation to stay on.

Cons:

  • First deployment on an already-conflicted cluster may still fail once, because AKS already owns the field.
  • To unstick the current cluster, you may need a one-time action like deleting keda-admission or doing one forced server-side upgrade.
  • AKS will no longer inject its extra namespace selector into KEDA's webhook.
  • You must trust KEDA's own webhook scope/selector, which is normally fine because it only validates KEDA CRDs.

Option 3: Disable KEDA validating webhook

Pros:
  • Removes the conflict completely because the webhook object is not created.
  • No server-side apply needed.
  • No Helm version dependency.
  • Simple to understand: no webhook means nothing for AKS to fight over.
Cons:
  • You lose KEDA admission-time validation.
  • Bad ScaledObject, ScaledJob, or TriggerAuthentication configs may not be rejected immediately.
  • Problems may show up later through operator logs/events instead of a clear apply-time error.

No comments:

Popular Posts