Create AKS Cluster to Host Valkey Cluster (Open Source Redis)

 Since bitnami redis images have gone commercial now the best open source redis clsuter setup option is to use Valkey, which is guranteed to be open source forever. The source code repos for Valkey is here. We have discussed deploying redis cluster on AKS with bitnami in "Setup Redis Cluster with JSON and Search Modules on AKS with Binami Redis Using Custom Image" However, if we want to deploy Valkey/Redis in cluster in AKS and want to access the Valkey/Redis from outside the AKS cluster, we have to setup AKS networking in a way that each pod is allocated with an IP from the Azure virtual network subnet. In this post let's explore how to setup such an AKS cluster correctly.

Why we need pods in AKS to have IPs from virtual network subnet to deploy Valkey?

When we connect to a Valkey/Redis cluster it always provide diffrent IPs of master nodes to client. Therefore, clients need to acess to IPs of Valkey node pods.

We cannot use Azure CNI overlay for this AKS cluster as it would setup internal network CIDR for pod IPs. Such IPs are not visible outside of AKS. Therefore it is not possible use Azure CNI overlay for AKS clsuter, if we are to deploy Valkey clsuter that should be allowed to access outside of AKS.

We can Azure node subnet mode. But this is not ideal as we have to have a large subnet planned to support boith the node needs as well as pods IP requirement.

The best option is to use Azure CNI Pod Subnet, which would be really useful for this need.

Feature	Azure CNI Overlay	Azure CNI Pod Subnet	Azure CNI Node Subnet
Pod IP Source	Private overlay network	Dedicated Azure VNet subnet	Same Azure VNet subnet as nodes
Node IP Source	Azure VNet subnet	Azure VNet node subnet	Azure VNet subnet
Separate Pod Subnet Required	No	Yes	No
Pods Consume VNet IP Addresses	No	Yes	Yes
Pod IPs Visible in Azure VNet	No	Yes	Yes
Azure VM Can Connect Directly to Pod IP	No	Yes	Yes
On-premises Can Reach Pod IP Directly	No	Yes (through VPN/ExpressRoute)	Yes (through VPN/ExpressRoute)
Requires Kubernetes Service / Gateway / Load Balancer to Reach Pods	Yes	Usually No	Usually No
VNet IP Consumption	Very Low	Medium	High
Subnet Planning	Simple	Moderate	Requires careful planning
Risk of Running Out of VNet IPs	Very Low	Medium	High
Supports Large AKS Clusters	Excellent	Excellent	Limited by subnet size
Network Performance	Native Azure networking	Native Azure networking	Native Azure networking
Maximum Cluster Scalability	Very High	Very High	Depends on available subnet IPs
Can Use Different Subnets for Different Node Pools	Yes	Yes	Yes
Add New Node Pool in a New Subnet Without Changing Existing Node Pools	Yes	Yes	Yes
Easy to Expand by Creating New Node Pool Subnets	Yes	Yes	Yes (recommended when the original subnet is nearly full)
Need to Reserve Large Subnets Up Front	No	No	Usually Yes
Each Node Pool Can Have Its Own NSG	Yes	Yes	Yes
Pods Can Have Separate NSGs from Nodes	No	Yes	No
Separate Route Tables for Pods and Nodes	No	Yes	No
Simplest Deployment Model	Yes	No	Yes
Best IP Address Utilization	Excellent	Good	Poor
Can Grow Cluster Without Consuming Many VNet IPs	Yes	No	No
Good for Enterprises with Limited VNet Address Space	Excellent	Good	Not Ideal
Best Choice for New AKS Deployments	Yes (Microsoft recommendation for most new deployments)	When pods need direct VNet connectivity	Mainly existing deployments or smaller clusters
Main Advantage	Excellent scalability while conserving VNet IP addresses.	Pods are first-class Azure network citizens with full VNet visibility.	Simple networking with no dedicated pod subnet.
Main Limitation	Pods are not directly reachable using their overlay IP addresses.	Consumes VNet IP addresses for every pod.	Can quickly exhaust subnet IP addresses in growing clusters.

We can start by setting up required subnets as shown below.

The NSG

resource "azurerm_network_security_group" "nsg" {

  name                = "demo-nsg"

  location            = azurerm_resource_group.instancerg.location

  resource_group_name = azurerm_resource_group.instancerg.name

  tags = merge(tomap({

    Service              = "network security group"

    SystemClassification = local.systemclassification_default_tag

  }), local.tags)

}

Below is the node subnet setup

# AKS Node Subnet

resource "azurerm_subnet" "aks" {

  name                              = "demo-aks-snet"

  resource_group_name               = data.azurerm_virtual_network.env_vnet.resource_group_name

  virtual_network_name              = data.azurerm_virtual_network.env_vnet.name

  address_prefixes                  = ["${var.SUBNET_CIDR_AKS}"]

  private_endpoint_network_policies = "Enabled"

  default_outbound_access_enabled   = true

  service_endpoints = [

    "Microsoft.AzureActiveDirectory",

    "Microsoft.AzureCosmosDB",

    "Microsoft.EventHub",

    "Microsoft.KeyVault",

    "Microsoft.Storage",

    "Microsoft.Sql",

    "Microsoft.ServiceBus",

    "Microsoft.Web"

  ]

}

# Assign user assigned id of aks to aks subnet -

resource "azurerm_role_assignment" "aks_uai_snet" {

  principal_id         = azurerm_user_assigned_identity.aks.principal_id

  role_definition_name = "Network Contributor"

  scope                = azurerm_subnet.aks.id

}

# Associate AKS subnet with network security group

resource "azurerm_subnet_network_security_group_association" "aks_nsg" {

  subnet_id                 = azurerm_subnet.aks.id

  network_security_group_id = azurerm_network_security_group.nsg.id

}

Then we can setup a pod subnet. Notice that we have to setup the service delegation managed clusters.

# AKS pod subnet for Azure CNI Pod Subnet mode

resource "azurerm_subnet" "aks_pod" {

  name                              = "demo-aks-vnet-pod-snet"

  resource_group_name               = data.azurerm_virtual_network.env_vnet.resource_group_name

  virtual_network_name              = data.azurerm_virtual_network.env_vnet.name

  address_prefixes                  = [var.SUBNET_CIDR_AKS_POD]

  private_endpoint_network_policies = "Enabled"

  default_outbound_access_enabled   = true

  delegation {

    name = "aks-delegation"

    service_delegation {

      name    = "Microsoft.ContainerService/managedClusters"

      actions = ["Microsoft.Network/virtualNetworks/subnets/join/action"]

    }

  }

}

# Assign shared AKS user assigned identity to pod subnet for Azure CNI Pod Subnet mode

resource "azurerm_role_assignment" "aks_uai_pod_snet" {

  principal_id         = azurerm_user_assigned_identity.aks.principal_id

  role_definition_name = "Network Contributor"

  scope                = azurerm_subnet.aks_pod.id

}

# Associate AKS pod subnet with network security group

resource "azurerm_subnet_network_security_group_association" "aks_pod_nsg" {

  subnet_id                 = azurerm_subnet.aks_pod.id

  network_security_group_id = azurerm_network_security_group.nsg.id

}

Then when we setup AKS for default node pool we need to setu the subnets correctly as shown below.

We have to setup the networking for AKS for Azure CNI pod subnet.

If we are adding other node pools same node and pod subnets can be used. Or for pod subnet we can even have a diffrent subnet.

For setting up Valkey and allow blue green nodepool deployments we can setup below node pools in AKS.

• default system node pool - this is must have for aks
• blue system node pool
• blue self host node pool
• green system node pool
• green self host node pool

We will have only blue or green live. Once workloads shift to new node pool blue or green, old node pool blue or green destroyed to save costs. The apporach is useful to safely upgrade AKS cluster inplace without disrupting the running workloads. When upgrades only new node pools eet deployed with new version of kubernetes and workloads gradgually shifted from all node pool to new, which we will discuss in detail with elatic seach deployment on AKS as in GitHub repo . Note the GitHub repo is still work in progress and complete solution will be explained step by step in future blog posts and repo will evelove to blue green supported elastic deployment on AKS. Similar repo work will be there in future for Valkey as well.