Create AKS Cluster to Host Valkey Cluster (Open Source Redis)

 Since bitnami redis images have gone commercial now the best open source redis clsuter setup option is to use Valkey, which is guranteed to be open source forever. The source code repos for Valkey is here. We have discussed deploying redis cluster on AKS with bitnami in "Setup Redis Cluster with JSON and Search Modules on AKS with Binami Redis Using Custom Image" However, if we want to deploy Valkey/Redis in cluster in AKS and want to access the Valkey/Redis from outside the AKS cluster, we have to setup AKS networking in a way that each pod is allocated with an IP from the Azure virtual network subnet. In this post let's explore how to setup such an AKS cluster correctly.

Why we need pods in AKS to have IPs from virtual network subnet to deploy Valkey?

When we connect to a Valkey/Redis cluster it always provide diffrent IPs of master nodes to client. Therefore, clients need to acess to IPs of Valkey node pods.

We cannot use Azure CNI overlay for this AKS cluster as it would setup internal network CIDR for pod IPs. Such IPs are not visible outside of AKS. Therefore it is not possible use Azure CNI overlay for AKS clsuter, if we are to deploy Valkey clsuter that should be allowed to access outside of AKS.

We can Azure node subnet mode. But this is not ideal as we have to have a large subnet planned to support boith the node needs as well as pods IP requirement.

The best option is to use Azure CNI Pod Subnet, which would be really useful for this need.

Mount Per Pod Managed Disk to AKS Pod to Provide Isolated Disk Space for Processing Large Files

 In AKS processing files on AKS node disk by application pod is not a recommended approach. It could lead to issues in stability of the AKS node itself. Therefore, we have two options. Either mount an Azure fileshare to the pod as described in "Mount Azure Storage Fileshare Created with Terraform on AKS". The other option is to setup a per pod disk, specially when the need is to process larger files. This option is really useful when we have pods running a single job and when it lives for the lifetime of the job. Example case is use of KEDA scale object with per single message processing from rabbitmq as described in "RabbitMQ KEDA Trigger for AKS Deployed Apps to Scale Out the Apps Accurately".

Expentation is to get a per pod disk attached as shown below.

RabbitMQ KEDA Trigger for AKS Deployed Apps to Scale Out the Apps Accurately

 We have discussed "Setting Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator" in a previous post. Baed on rabbitmq messages we may want to setup a trigger to auto scale our apps to handle the workloads similar to what we have done with eventhubs, azure service bus etc, as described in below posts.

• Setting Up (KEDA) Authentication Trigger for Azure Storage Queue/Service Bus in AKS
• Scale Pods in AKS with Kubernetes Event Driven Autoscaling (KEDA) ScaledJob Based on Azure Service Bus Queue as a Trigger
• Multiple KEDA Triggers for a Scaled Job with Event Hubs in AKS

In this post, let's look at usage of keda scale objects and how we can setup to trigger scaling with rabbit mq messages accurately for AKS deployed apps. The expectation is to scale out apps based on number of messgaes in rabbitmq queues and effective process the the work, while scaling in when there is not much workload on the system.

Depending on above message loads the consumer apps scale out and in accrdingly.

Enable Updating GitHub Environment Variables via GitHub Actions Workflow

 To update environment  variables defined in GitHub environment, via a GitHub action workflow we cannot use the default GITHUB_TOKEN .

✅ GITHUB_TOKEN can interact with repository contents (depending on permissions), create releases, comment on PRs, update issues, etc.

❌ GITHUB_TOKEN cannot update repository environments, environment secrets, or environment variables via the GitHub REST API.

Environment Deployment Approvals for GitHub Action Workflow

 In GitHub actvaions workflows we can use environments to scope varibles and protection rules. Let's compare Azure DevOps pipeline environments to GitHub Actions workflow environments, differences regarding environment deployment approvals, in this post.

An environment can be configured for approvals as shown below.

Code for Creating a Multi Stage Workflow Structure with GitHub Actions - Mapping Azure DevOps Multi-Stage Pipelines to GitHub Actions - Part 3

 In the previous post "Create Multi Stage Pipeline Structure with GitHub Actions - The Layout - Mapping Azure DevOps Multi-Stage Pipelines to GitHub Actions - Part 2" we have dicussed the GitHub action workflow layout mapping to the Azure pipeline structure. The code for workflows can be structured as shown below for the pipeline requirement discussed in "Mapping Azure DevOps Multi-Stage Pipelines to GitHub Actions - Part 1". Note that we have seperated folder structure for job actions and step actions to organize the code properly in a manageable way. However, both are really composite actions in GitHub. Each action name is setup using a folder name as it must be action.yml or action.yaml for actions.

Create Multi Stage Workflow Structure with GitHub Actions - The Layout - Mapping Azure DevOps Multi-Stage Pipelines to GitHub Actions - Part 2

 In "Mapping Azure DevOps Multi-Stage Pipelines to GitHub Actions - Part 1" we have discussed the limitation of GitHub action workflows compared to Azure DevOps pipelines. Further, we have compared the features and possible mappings. In this post, let's try how we can create GitHub workflow structure to make it similar to Azure DevOps pipeline shown in "Mapping Azure DevOps Multi-Stage Pipelines to GitHub Actions - Part 1".

Lets now look at the a the outcome of GitHub workflow setup, and see how it maps to Azure pipeline structure above creating the below pipeline stages.

GitHub Workflow

Note that unlike below fully functional Azure pipeline here in GitHub workflow is only having the structure created only. No approval gates setup or actual steps of execution implemented. But structure is setup keeping the real working workflow in mind. The intialize stage in Azure pipeline publishes AKS manifest files as artifacts in same pipeline, to be used in pepeline jobs, without full repo checkout. However, this is intentionally skipped in GitHub workflow as it seems checkout in every job is required to get the actions and jobs to be executed in GitHub workflow.

Fix Keda Upgrade on AKS with Helm Fail Due to Field Ownership Conflicts

 AKS runs an internal mutating component called the admissionsenforcer (part of the AKS-managed "admissions enforcer" / addon-manager). After KEDA's ValidatingWebhookConfiguration (keda-admission) is created, AKS automatically injects a namespaceSelector into every webhook entry to exclude AKS control-plane / managed namespaces (so KEDA's webhooks never intercept system pods). When it does this, it takes Server-Side Apply (SSA) field ownership of those namespaceSelector fields under the field manager named admissionsenforcer. 

So after the first deploy:

• keda-admission webhook exists
• AKS has mutated .webhooks[*].namespaceSelector and now owns those fields

In the second deploy, the KEDA chart  tries to re-apply the same ValidatingWebhookConfiguration with the field manager helm. SSA detects that helm wants to set fields already owned by admissionsenforcer → field ownership conflict: