We have discussed "Setting Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator" in a previous post. Baed on rabbitmq messages we may want to setup a trigger to auto scale our apps to handle the workloads similar to what we have done with eventhubs, azure service bus etc, as described in below posts.
In this post, let's look at usage of keda scale objects and how we can setup to trigger scaling with rabbit mq messages accurately for AKS deployed apps. The expectation is to scale out apps based on number of messgaes in rabbitmq queues and effective process the the work, while scaling in when there is not much workload on the system.
Depending on above message loads the consumer apps scale out and in accrdingly.
As the first requirement we need to have a secret setup with rabbitmq credentials, in our AKS. Note that in this example, we have both rabbitmq and apps running on the same aks cluster. Therefore, we use cluster IP based urls. Refer "
Setting Up RabbitMQ Cluster in AKS Using RabbitMQ Cluster Operator". User name
${rabbitmq_user}$ and password
${rabbitmq_user_password}$ should be replaced with actual values.
---
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-service-credentials
namespace: demo
annotations:
rabbitmq.com/topology-allowed-namespaces: "demo"
type: Opaque
stringData:
username: "${rabbitmq_user}$"
password: "${rabbitmq_user_password}$"
uri: "http://rabbitmq-cluster.rabbitmq.svc.cluster.local:15672"
amqpuri: "amqp://rabbitmq-cluster.rabbitmq.svc.cluster.local:5672"
Then we can setup a triger authentication as shwon below.
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: demo-keda-trigger-auth-rabbitmq
namespace: demo
spec:
secretTargetRef:
- parameter: host
name: rabbitmq-service-credentials
key: uri
- parameter: username
name: rabbitmq-service-credentials
key: username
- parameter: password
name: rabbitmq-service-credentials
key: password
Then to scale a deployment we can setup keda scaled object as below. Here we run as single replica always as min replica count. The horizontal pod auto scaler is managed by keda scaled object. CPU and memery based scaling triggers are setup. Then we have message based triggers setup as well. If one instace/replica of app can handle 10 messges in parallel, we have setup trigger to half of it to ensure we scale out before the replica is at its max possible processing ability. This allows us to effectively process workloads, and scale in when there is no workloads. We can even setup multiple rabbitmq triggeres.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: video-messageprocessor-so
namespace: demo
annotations:
scaledobject.keda.sh/transfer-hpa-ownership: "true"
autoscaling.keda.sh/paused: "false"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: video-messageprocessor
pollingInterval: 5
initialCooldownPeriod: 300
minReplicaCount: 1
maxReplicaCount: 20
fallback:
failureThreshold: 3
replicas: 6
behavior: "currentReplicasIfHigher"
advanced:
restoreToOriginalReplicaCount: false
horizontalPodAutoscalerConfig:
name: video-messageprocessor-hpa
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 180
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 1
triggers:
- type: cpu
metricType: Utilization
metadata:
value: "50"
- type: memory
metricType: Utilization
metadata:
value: "30"
- type: rabbitmq
metadata:
protocol: http
excludeUnacknowledged: "false"
mode: QueueLength
value: "5" # half of queue concurrency — scale-out before pods are fully saturated.
activationValue: "0"
queueName: generatevideohandler
vhostName: /
unsafeSsl: "false"
useCachedMetrics: "false"
authenticationRef:
name: demo-keda-trigger-auth-rabbitmq
if we inspect the trigger here, we see its protocol is set to http. This allows us to enable the scaling to consider messages already delivered to consumer but yet to be processed as well into our scaling trigger consideration. See that we have set excludeUnacknowledged to false to ahcive this. excludeUnacknowledged only works when we use the http protocol for the triggers. Refer keda documentation here. The app does not have to work with http, only the trigger works this way. App consumer can connect to rabbitmq with AMQP 1.0 or previous version 0.9. We can define more than one rabbitmq trigger for a single scale job if required.
There can be processors that only process one message at a time (processing of single message require dedicated resources). In such cases, this accurate scaling with excludeUnacknowledged is more important. See below example, where we scale a statefulset with only one replica processing one message. When there is no messages we dont have any replica running to avoid unnecessary resource usage. The same can be applied for a deployemnt as well instead of a statefulset (statefulset used here to accomodate per pod managed disk allocation which we will discuss in a future post).
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: largevideo-messageprocessor-so
namespace: demo
annotations:
scaledobject.keda.sh/transfer-hpa-ownership: "true"
autoscaling.keda.sh/paused: "false"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: largevideo-messageprocessor
pollingInterval: 5
initialCooldownPeriod: 300
minReplicaCount: 0
maxReplicaCount: 20
fallback:
failureThreshold: 3
replicas: 6
behavior: "currentReplicasIfHigher"
advanced:
restoreToOriginalReplicaCount: false
horizontalPodAutoscalerConfig:
name: largevideo-messageprocessor-hpa
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 180
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 1
triggers:
- type: rabbitmq
metadata:
protocol: http
excludeUnacknowledged: "false"
mode: QueueLength
value: "1" # matches queue concurrency — one pod per message.
activationValue: "0"
queueName: generatelargevideohandler
vhostName: /
unsafeSsl: "false"
useCachedMetrics: "false"
authenticationRef:
name: demo-keda-trigger-auth-rabbitmq
No comments:
Post a Comment