Saturday 7 September 2024

Setup Application Ingress for AKS Using Nginx Ingress controller with Private IP

 We have dicussed "Deploy Nginx Ingress Conroller with Private IP (Limited Access to vNET) to AKS with Terraform, Helm and Azure DevOps Pipelines" and then "Automate Validation of Nginx Ingress Controller Setup in AKS with an Azure Pipeline Task" is discussed. As the next step, let's explore how to setup ingress for application deployed in AKS, using Nginx ingress controller deployed with private IP. The purpose is exposoing the application in AKS within the virtual network, so, that other applications in the same virtual network can access the application securely, without having to expose application in AKS publicly.

The expected outcome is to have ingress for apps setup as shown below.


The first step is to deploy the application (an API in this example) to AKS. we can use below yaml and aply it with kubectl apply using kubernetes task in Azure pipelines.

Exampe deploy tasks in Azure pipelines below.

      # Apply container scale setting and resource settings to k8s manifest
      - task: qetza.replacetokens.replacetokens-task.replacetokens@5
        displayName: 'Replace tokens in ${{ variables.deploymentyaml }}.yaml'
        inputs:
          rootDirectory: '$(System.ArtifactsDirectory)'
          targetFiles: ${{ variables.deploymentyaml }}
          actionOnMissing: fail
          tokenPattern: custom
          tokenPrefix: '${'
          tokenSuffix: '}$'

      - task: Kubernetes@1
        displayName: 'Deploy ${{ parameters.app }}'
        inputs:
          connectionType: 'Azure Resource Manager'
          azureSubscriptionEndpoint: '${{ parameters.serviceconnection }}'
          azureResourceGroup: 'ch-demo-$(envname)-rg'
          kubernetesCluster: 'ch-demo-$(envname)-aks-$(sys_app_deploy_instance_suffix)'
          useClusterAdmin: true
          namespace: 'demo'
          command: apply
          arguments: '-f ${{ variables.deploymentyaml }}'
          workingDirectory: '$(System.ArtifactsDirectory)'

The full application deployment kubernetes manifest is below. Note that variables defined in ${aksappname}$ will be replaced by Azure piplines with the correct values.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${aksappname}$
  namespace: demo
  labels:
    app: ${aksappname}$
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 25%
  minReadySeconds: 0
  selector:
    matchLabels:
      service: ${aksappname}$
  template:
    metadata:
      labels:
        app: ${aksappname}$
        service: ${aksappname}$        
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: demo-wi-sa
      nodeSelector:
        "kubernetes.io/os": ${buildos}$
      priorityClassName: ${aksapppriorityclassname}$
      #------------------------------------------------------
      # setting pod DNS policies to enable faster DNS resolution
      # https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
      dnsConfig:
        options:
          # use FQDN everywhere 
          # any cluster local access from pods need full CNAME to resolve 
          # short names will not resolve to internal cluster domains
          - name: ndots
            value: "2"
          # dns resolver timeout and attempts
          - name: timeout
            value: "15"
          - name: attempts
            value: "3"
          # use TCP to resolve DNS instad of using UDP (UDP is lossy and pods need to wait for timeout for lost packets)
          - name: use-vc
          # open new socket for retrying
          - name: single-request-reopen
      #------------------------------------------------------
      volumes:
        # `name` here must match the name
        # specified in the volume mount
        - name: demo-configmap-${aksappname}$-volume
          configMap:
            # `name` here must match the name
            # specified in the ConfigMap's YAML
            name: demo-configmap
      terminationGracePeriodSeconds: 300 # This must be set to a value that is greater than the preStop hook wait time.
      containers:
        - name: ${aksappname}$
          lifecycle:
            preStop:
              exec:
                command: [${containersleepcommand}$,"180"]
          image: ${aks_shared_container_registry}$.azurecr.io/demo/${aksappname}$:${Build.BuildId}$
          imagePullPolicy: Always
          # probe to determine the stratup success
          startupProbe:
            httpGet:
              path: /api/health
              port: container-port
            initialDelaySeconds: 30 # give 30 seconds to get container started before checking health
            failureThreshold: 30 # max 300 (30*10) seconds wait for start up to succeed
            periodSeconds: 10 # interval of probe (300 (30*10) start up to succeed)
            successThreshold: 1 # how many consecutive success probes to consider as success
            timeoutSeconds: 10 # probe timeout 
            terminationGracePeriodSeconds: 30 # restarts container (default restart policy is always)
          # readiness probe fail will not restart container but cut off traffic to container with one failure 
          # as specified below and keep readiness probes running to see if container works again
          readinessProbe: # probe to determine if the container is ready for traffic
            httpGet:
              path: /api/health
              port: container-port
            failureThreshold: 1 # one readiness fail should stop traffic to container
            periodSeconds: 20 # interval of probe
            successThreshold: 2 # how many consecutive success probes to consider as success after a failure probe
            timeoutSeconds: 10 # probe timeout
          # probe to determine the container is healthy and if not healthy container will restart
          livenessProbe: 
            httpGet:
              path: /api/health
              port: container-port
            failureThreshold: 3 # tolerates three consecutive faiures before restart trigger
            periodSeconds: 40 # interval of probe
            successThreshold: 1 # how many consecutive success probes to consider as success after a failure probe
            timeoutSeconds: 10 # probe timeout 
            terminationGracePeriodSeconds: 60 # restarts container (default restart policy is always)     
          volumeMounts:
          - mountPath: /etc/config
            name: demo-configmap-${aksappname}$-volume
          ports:
            - name: container-port
              containerPort: 80
              protocol: TCP
          env:
            - name: ASPNETCORE_URLS
              value: http://+:80
            - name: ASPNETCORE_ENVIRONMENT
              value: Production
            - name: CH_DEMO_CONFIG
              value: /etc/config/config_${envname}$.json
          resources:
                limits:
                  memory: ${aksappcontainermemorylimit}$ # the memory limit equals to the request!
                  # no cpu limit! this is excluded on purpose
                requests:
                  memory: ${aksappcontainermemorylimit}$
                  cpu: "${aksappcontainerrequestscpu}$"

Next, we have to define a cluster ip configuration as shown below to work with the application.

---
apiVersion: v1
kind: Service
metadata:
  name: ${aksappname}$-clusterip
  namespace: demo
  labels:
    app: ${aksappname}$
    service: ${aksappname}$
spec:
  type: ClusterIP
  ports:
    - port: 8091
      targetPort: 80
      protocol: TCP
  selector:
    service: ${aksappname}$

Then, we can define ingress with nginx as follows. Host name for application should be defined following the private dns zone setup  (refer "Deploy Nginx Ingress Conroller with Private IP (Limited Access to vNET) to AKS with Terraform, Helm and Azure DevOps Pipelines"). The private dns zone pointing to private IP used by Nginx (*.aksgreen.ch-demo-dev-eus-001.net points to private IP of Nginx). So, invoice-api app in AKS can have hsot name as 

invoice-api.aksgreen.ch-demo-dev-eus-001.net. The path setting below as / allows accessing any endpoint exposed by the API. The same thing with AGIC is /*.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${aksappname}$
  namespace: demo
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/proxy-body-size: 8m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    # kubernetes.io/ingress.class: nginx # for nginx deprecated
spec:
  ingressClassName: nginx
  rules:
  - host: ${aksapphostname}$
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ${aksappname}$-clusterip
            port:
              number: 8091

We can manually validate the deployed application in AKS, can be access using nginx ingress, by using an App gateway in the same virtual network as below using a backend setting pointing to invoice-api.aksgreen.ch-demo-dev-eus-001.net and do a health probe with http://invoice-api.aksgreen.ch-demo-dev-eus-001.net/api/health (health validation endpoint of the deployed API). Lets setup app gateway for testing the nginx as below.

App gateway backend pool

App gateway backend setting


App gateway listener


App gateway rule




App gateway health probe





When the probe is tested it shows connected and healthy, shich is confirming our ingress setup with nginx is correctly working within the virtual network.



In the next post, let's explore, how to automate health check of AKS deployed apps and validate ingress setup with nginx, using Azure pipelines.




No comments:

Popular Posts