Using operators to setup RabbitMQ cluster in AKS (kubernetes) is a recomended approach. There are two RabbitMQ operators we need to deploy into AKS for setting up a RabbitMQ cluster and managing message setup in the deployed RabbitMQ cluster.
- Cluster Operator: Automate provisioning, management and oprations of a RabbitMQ cluster within AKS.
- Messaging Topology Operator: Managing message topologies within the deployed RabbitMQ cluster in AKS.
Once setup is completed the rabbitmq-system namespace should have two opertaors running as shown below.
The first step is to download below mentioned 3 yaml files from the GitHub releases.
- Cluster Operator: https://github.com/rabbitmq/cluster-operator
- Download the latest release cluster-operator.yml file.
- Cert Manager (there is option to skip this and use a generated certificate as described here): https://github.com/cert-manager/cert-manager
- Download the laterst release cert-manager.yaml file
- Cert manager is used to manage certificates required for the topology operator.
- Messaging Topology Operator: https://github.com/rabbitmq/messaging-topology-operator
- Download latest release messaging-topology-operator-with-certmanager.yaml file.
Then add them to the Azure devops repo path (here we use pipelines/aks_manifests/rabbitmq path). Note that the cluster-operator.yml is renamed to cluster-operator.yaml have the file name extension consistency. 
Then we can create a powershell script install-rabbitmq.ps1 in the path pipelines/scripts. This script will first install the cluster prtaor and will wait for it's pods to be ready before, start setting up the cert manager. Once the cert manager is setup it will wait for all cert manager pods to be ready. Cert manager pods need to be ready before starting the deployment of messaging topology operator. Once cert manager is ready the messaging topology operator is deployed and the script will validate whether the oprator pods are running as expected. The script below helps to fully automate the deployment of the RabbitMQ operators in AKS via Azure pipeline task.
param ( [string]$manifestPath ) function Invoke-AKS-App-Health-Check { param ( [string]$aksNamespace, [string[]]$apps, [int]$appHealthCheckMaxAttempts = 60, # Wait for maximum 5 minutes till apps are fully deployed and running in AKS [int]$appHealthCheckIntervalSeconds = 5, # Check cycle in each 5 seconds [int]$appReadyInitialWaitSeconds = 5 ) $appRestartCheckMaxAttempts = 2; # Max 10 minutes wait for restarts to stabilize $appRestartCheckIntervalSeconds = 300; # Wait time for restarts to stabilize (max cap 5 minutes in k8s restart policy https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy) $appRestartCheckMaxCycles = 1; # Only run maximum of two restart verification cycles setting max threshold to 10 minutes $appHealthCheckAttempt = 0; $appRestartCheckAttempt = 0; $appRestartCheckCycle = 0; Write-Host (-join('AKS app health check max attempts ',$appHealthCheckMaxAttempts)); Write-Host (-join('AKS app restart check max attempts ',$appRestartCheckMaxAttempts)); Write-Host (-join('AKS app restart check max cycles ',$appRestartCheckMaxCycles)); Write-Host ('--------------------------------------------------------'); Write-Host (-join('Waiting for namespace ',$aksNamespace,' apps to be ready...')); Start-Sleep -Seconds $appReadyInitialWaitSeconds; do { $allAppPodsReady = $true; $restaredPods = @{}; # Empty hash tabel $appHealthCheckAttempt++; foreach($app in $apps) { Write-Host ('--------------------------------------------------------'); Write-Host (-join('Inspecting app ',$app,' in namespace ',$aksNamespace,' ...')); kubectl get pods -l app.kubernetes.io/name=$app -n $aksNamespace # printing pods for pipeline logs Write-Host ('--------------------------------------------------------'); $pods = kubectl get pods -l app.kubernetes.io/name=$app -n $aksNamespace -o json | ConvertFrom-Json; if (($null -eq $pods) -or ($null -eq $pods.items) -or ($pods.items.Count -le 0)) { $allAppPodsReady = $false; Write-Error (-join('No pods found for ',$app,' in namespace ',$aksNamespace)); exit 1; } Write-Host (-join('Inspecting pod(s) for the app ',$app,' ...')); foreach($pod in $pods.items) { $podName = $pod.metadata.name; # Filter pod status - This is required as pod status phase in json, only has Pending and Running status which is not sufficient if pods terminate or crash $podInfoString = (kubectl get pods $podName -n $aksNamespace)[1] -replace '\s+',';'; $podInfo = $podInfoString.Split(';'); $podReady = $podInfo[1]; $podStatus = $podInfo[2]; if(($null -eq $pod.status) -or ($null -eq $pod.spec) -or ($null -eq $pod.status.containerStatuses) -or ($null -eq $pod.spec.containers) -or ($pod.status.containerStatuses.Count -le 0) -or ($pod.spec.containers.Count -le 0)) { Write-Host (-join($podName,' is not yet healthy. Marking app as not healthy...')) -ForegroundColor Yellow; $allAppPodsReady = $false; } else { $podRestarts = $pod.status.containerStatuses[0].restartCount $podContainerReady = $pod.status.containerStatuses[0].ready; $podContainerStarted = $pod.status.containerStatuses[0].started; Write-Host (-join($podName,' status is ',$podReady,' ',$podStatus,'. Restarts are ',$podRestarts,'. Container ready state is ',$podContainerReady,' started state is ',$podContainerStarted)); if (($podStatus -ne 'Running') -or ($podReady -ne '1/1') -or (-not $podContainerReady) -or (-not $podContainerStarted)) { Write-Host (-join($podName,' is not yet healthy. Marking app as not healthy...')) -ForegroundColor Yellow; $allAppPodsReady = $false; } elseif ($podRestarts -gt 0) { Write-Host (-join($podName,' has ',$podRestarts,' restart(s). collecting information for further verification...')); $restaredPods.Add($podName,$podRestarts); } } } Write-Host ('--------------------------------------------------------'); } if (($allAppPodsReady) -and ($restaredPods.Count -le 0)) { Write-Host (-join('All apps are ready in health check attempt ',$appHealthCheckAttempt,'.')) -ForegroundColor Green; } elseif (($allAppPodsReady) -and ($restaredPods.Count -gt 0)) { $appRestartCheckAttempt++; if($appRestartCheckAttempt -eq 1) { $appRestartCheckCycle++; if (($appHealthCheckAttempt + $appRestartCheckMaxAttempts) -gt $appHealthCheckMaxAttempts) { $appHealthCheckMaxAttempts = $appHealthCheckAttempt + $appRestartCheckMaxAttempts; } } if (($appRestartCheckAttempt -le $appRestartCheckMaxAttempts) -and ($appRestartCheckCycle -le $appRestartCheckMaxCycles)) { Write-Host (-join('All apps are ready in health check attempt ',$appHealthCheckAttempt,', but have restarts in ',$restaredPods.Count,' pod(s). Waiting for 5 minutes before reinspecting pod restarts...')) -ForegroundColor Yellow; Start-Sleep -Seconds $appRestartCheckIntervalSeconds foreach ($restartedPodName in $restaredPods.Keys) { $previousRestarts = $restaredPods[$restartedPodName]; $currentPod = kubectl get pods $restartedPodName -n $aksNamespace -o json | ConvertFrom-Json if (($null -eq $currentPod) -or ($null -eq $currentPod.status) -or ($null -eq $currentPod.status.containerStatuses) -or ($currentPod.status.containerStatuses.Count -le 0)) { Write-Host (-join($restartedPodName,' is not in a stable state.')) -ForegroundColor Yellow; $allAppPodsReady = $false; } else { $currentPodRestarts = $currentPod.status.containerStatuses[0].restartCount; if ($currentPodRestarts -gt $previousRestarts) { Write-Host (-join($restartedPodName,' restarts are not stabilized. Previous: ',$previousRestarts,' Current:',$currentPodRestarts)) -ForegroundColor Yellow; $allAppPodsReady = $false; } else { Write-Host (-join($restartedPodName,' restarts are stabilized. Previous: ',$previousRestarts,' Current:',$currentPodRestarts)); } } } if ($allAppPodsReady) { Write-Host (-join('Pod restarts are stabilized. All apps are ready in health check attempt ',$appHealthCheckAttempt,', in restart check cycle ',$appRestartCheckCycle,' and in restart check attempt ',$appRestartCheckAttempt,'.')) -ForegroundColor Green; } else { Write-Host (-join('Pod restarts are not stabilized. All apps are not ready in health check attempt ',$appHealthCheckAttempt,', in restart check cycle ',$appRestartCheckCycle,' and in restart check attempt ',$appRestartCheckAttempt,'. Waiting for ',$appHealthCheckIntervalSeconds,' seconds before next check...')) -ForegroundColor Yellow; Write-Host ('--------------------------------------------------------'); Write-Host ('--------------------------------------------------------'); Start-Sleep -Seconds $appHealthCheckIntervalSeconds } } else { Write-Error (-join('All apps are not ready in health check attempt ',$appHealthCheckAttempt,', in restart check cycle ',$appRestartCheckCycle,' and in restart check attempts ', $appRestartCheckAttempt,'. Deployment of RabbitMQ failed..')); exit 1; } } else { Write-Host (-join('All apps are not ready in health check attempt ',$appHealthCheckAttempt,' Waiting for ',$appHealthCheckIntervalSeconds,' seconds before next check...')) -ForegroundColor Yellow; Write-Host ('--------------------------------------------------------'); Write-Host ('--------------------------------------------------------'); $appRestartCheckAttempt = 0; Start-Sleep -Seconds $appHealthCheckIntervalSeconds } } until($allAppPodsReady -or ($appHealthCheckAttempt -ge $appHealthCheckMaxAttempts)) if ($allAppPodsReady) { Write-Host (-join('All apps ready in namespace ',$aksNamespace,'.')) -ForegroundColor Green; Write-Host ('--------------------------------------------------------'); Write-Host ('--------------------------------------------------------'); } else { Write-Error (-join('All apps are not ready in namespace ',$aksNamespace,'. Deployment of RabbitMQ failed.')); exit 1; } } $custerOperatorManifest = -join($manifestPath,'cluster-operator.yaml'); $certManagerManifest = -join($manifestPath,'cert-manager.yaml'); $topologyOperatorManifest = -join($manifestPath,'messaging-topology-operator-with-certmanager.yaml'); Write-Host (-join('Deploying RabbitMQ cluster operator with: ',$custerOperatorManifest, ' ...')); kubectl apply -f $custerOperatorManifest; Invoke-AKS-App-Health-Check -aksNamespace 'rabbitmq-system' -apps @('rabbitmq-cluster-operator'); Write-Host ('Successfully deployed RabbitMQ cluster operator.'); Write-Host ('========================================================='); Write-Host (-join('Deploying cert-manager with: ',$certManagerManifest, ' ...')); kubectl apply -f $certManagerManifest; Invoke-AKS-App-Health-Check -aksNamespace 'cert-manager' -apps @('cert-manager','cainjector','webhook'); Write-Host ('Successfully deployed cert-manager.'); Write-Host ('========================================================='); Write-Host ('Waiting 60 seconds for cert manager web hooks to be operational ...'); Start-Sleep -Seconds 60; # Need to wait for cert manager web hooks to be fully ready and operational, before deploying toology operator Write-Host (-join('Deploying RabbitMQ topology operator with: ',$topologyOperatorManifest, ' ...')); kubectl apply -f $topologyOperatorManifest; Invoke-AKS-App-Health-Check -aksNamespace 'rabbitmq-system' -apps @('messaging-topology-operator'); Write-Host ('Successfully deployed RabbitMQ topology operator.'); Write-Host ('=========================================================');
Then we can use a task in Azure pilines as below to execute the script to get the oprators deployed successfully. We need kubectl install task as a prerequisite to the below task Deploy RabbitMQ.
- task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
- task: AzureCLI@2 displayName: 'Deploy RabbitMQ' inputs: azureSubscription: '${{ parameters.serviceconnection }}' scriptType: pscore scriptLocation: inlineScript inlineScript: | $rgName = 'ch-mq-$(envname)-rg'; $aksName = 'ch-mq-$(envname)-aks-$(sys_app_deploy_instance_suffix)'; Write-Host $aksName az aks get-credentials -n $aksName -g $rgName --admin --overwrite-existing # Note that keda and nginx steps removed here to keep focus on the topic $(System.DefaultWorkingDirectory)/pipelines/scripts/install_rabbitmq.ps1 ` -manifestPath '$(System.DefaultWorkingDirectory)/pipelines/aks_manifests/rabbitmq/'; kubectl config delete-context (-join($aksName,'-admin'))
Pipeline run deploys the operators as shown below.
In the next post, let's look at how to deploy RabbitMQ cluster in AKS, by enhancing the script we used in this post, and using the deployed RabbitMQ cluster operator in AKS.
No comments:
Post a Comment