Setting Up Kubernetes Event Drivern Autoscaling (KEDA) in AKS with Workload Identity

 We have discuss setting up workload identity in AKS to be used with application containers we deploy to AKS in the post "Setting Up Azure Workload Identity for Containers in Azure Kubernetes Services (AKS) Using Terra- Improved Security for Contianers in AKS". Kubernetes Event Drivern Autoscaling (KEDA) is the mechanism we need to use when we want to scale our deployments, or specially kubernetes jobs (A pod that runs for completion). In this post let's look at how to setup KEDA with workload identity, so that we can use KEDA in a later posts to run a Kubernets job, autoscaled based on the messages received to a storage queue or Azure service bus.

To get KEDA setup on AKS we can use helm. We have discussed setting up helm in WSL in the post here. As the first step we need to setup federated identity credential with the user assigned identity used for AKS, which we have discussed in detail in "Setting Up Azure Workload Identity for Containers in Azure Kubernetes Services (AKS) Using Terra- Improved Security for Contianers in AKS". Note that the namespace we are using for Keda is keda and the service account we are going to setup later is named as keda-operator.

# Federated identity credential for AKS user assigned id - used with workload identity service account for KEDA
resource "azurerm_federated_identity_credential" "keda" {
  name                = "${var.prefix}-${var.project}-${var.environment_name}-aks-keda-fic-${var.deployment_name}"
  resource_group_name = var.rg_name
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.aks_cluster.oidc_issuer_url # Open id connect issue url from AKS
  parent_id           = var.user_assigned_identity                             # user assigned identity id (Azure resource id)
  subject             = "system:serviceaccount:keda:keda-operator"             # system:serviceaccount:aksapplicationnamespace:workloadidentityserviceaccountname (to be created after AKS cluster is setup)

  depends_on = [
    azurerm_kubernetes_cluster.aks_cluster
  ]
  lifecycle {
    ignore_changes = []
  }
}

Terraform will setup federated identity credential as shown below.

Next we need to setup namespace keda once the cluster is up and running. We can use kubectl to apply below yaml to get the namespace keda created.

apiVersion: v1

kind: Namespace

metadata:

  name: keda

We need to setup the service account as well using below yaml. Replace ${sys_aks_uai_client_id}$ in the yaml to the client id of the user assigned identity used for AKS. Replace ${tenantid}$ with the Azure tenant id.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ${sys_aks_uai_client_id}$ # ${sys_aks_uai_client_id}$
    azure.workload.identity/tenant-id: ${tenantid}$ # "${tenantid}$"
  name: keda-operator # Referred by AKS user assigned identity federated credential
  namespace: keda

We can get the user assigned identity from terraform using output variables as shown below.

resource "azurerm_user_assigned_identity" "aks" {

  location            = azurerm_resource_group.instancerg.location

  name                = "${var.PREFIX}-${var.PROJECT}-${var.ENVNAME}-aks-uai"

  resource_group_name = azurerm_resource_group.instancerg.name

}

output "aks_uai_client_id" {

  value = azurerm_user_assigned_identity.aks.client_id

}

Then we can use below .sh file in WSL to run helm install for KEDA. Again replace ${sys_aks_uai_client_id}$ and ${tenantid}$ in the .sh file. 

helm repo add kedacore https://kedacore.github.io/charts

helm repo update

helm install keda kedacore/keda --namespace keda \

--set serviceAccount.create=false \

--set serviceAccount.name=keda-operator \

--set podIdentity.azureWorkload.enabled=true \

--set podIdentity.azureWorkload.clientId=${sys_aks_uai_client_id}$ \

--set podIdentity.azureWorkload.tenantId=${tenantid}$

As you can see in above we are setting up Keda in keda namespace. We are requesting to use the service account we have created previously, instead of creating a new one as the service account we have created correctly setup with the user assigned identity of AKS as workload identity.

With this our intial setup requirement for KEDA is completed and we can see below deployments running in keda namespace.

The keda-oprator and keda-oprator-metrics-apiserver containers will have folowing shown environment variables injected, so that the scaling with Aure service bus queue or Azure storage queue can be done with KEDA autorizing with Azure workload identity for AKS.

For storage queue based KEDA scaling we need user assigned identity assigned to storage queue as shown below.

resource "azurerm_storage_account" "queue" {

  name                             = "${var.PREFIX}${var.PROJECT}queuest"

  resource_group_name              = azurerm_resource_group.instancerg.name

  location                         = azurerm_resource_group.instancerg.location

  account_tier                     = "Standard"

  account_replication_type         = "LRS"

  account_kind                     = "StorageV2"

  access_tier                      = "Hot"

  allow_nested_items_to_be_public  = false

  min_tls_version                  = "TLS1_2"

  cross_tenant_replication_enabled = false

}

resource "azurerm_storage_queue" "video" {

  name                 = "demovideoqueue"

  storage_account_name = azurerm_storage_account.queue.name

}

resource "azurerm_storage_queue" "dotnet_video" {

  name                 = "dotnetvideoqueue"

  storage_account_name = azurerm_storage_account.queue.name

}

resource "azurerm_role_assignment" "storage_q_contributor" {

  principal_id                     = azurerm_user_assigned_identity.aks.principal_id

  role_definition_name             = "Storage Queue Data Contributor"

  scope                            = azurerm_storage_account.queue.id

  skip_service_principal_aad_check = true

}

For service bus it should be as below.

resource "azurerm_servicebus_namespace" "demo" {

  name                = "${var.PREFIX}-${var.PROJECT}-${var.ENVNAME}-sbus"

  location            = azurerm_resource_group.instancerg.location

  resource_group_name = azurerm_resource_group.instancerg.name

  sku                 = "Standard"

  tags = merge(tomap({

    Service = "service_bus"

  }), local.tags)

}

resource "azurerm_servicebus_queue" "dotnetvideo" {

  name                         = "dotnetvideoqueue"

  namespace_id                 = azurerm_servicebus_namespace.demo.id

  lock_duration                = "PT5M"

  requires_duplicate_detection = true

  enable_partitioning          = true

}

resource "azurerm_role_assignment" "sbq_contributor" {

  principal_id                     = azurerm_user_assigned_identity.aks.principal_id

  role_definition_name             = "Azure Service Bus Data Owner"

  scope                            = azurerm_servicebus_queue.dotnetvideo.id

  skip_service_principal_aad_check = true

}