Access Private Url within Azure vNet with Azure Pipeline Microsoft Hosted Pipeline Agent via AKS pod in the vNet

If we are using Microsoft hosted agents for Azure pipelines to deploy Azure infrastucture and need to access vNet protected urls of services deployed, we can use a pod in AKS cluster within same vNet, as a jump host. This gives us access endpoints in vNet and ability to resolve DNS defined in private DNS zones of the vNet. Let's look at staep by step how to achive this goal, while using a Microsoft hosted agent in Azure pipelines.

The expectation is to access url such as 
http://es-search.sh.aks.ch-demo-dev-euw-002.net/demoindex001/_count  so AKS hosted elastic seach is accessed via a AKS pod and get the results to the pipeline agent as shown below. Since microsoft hosted agent is outside the vNET it cannot directly reach this elastc search (deployed in AKS) url.

For this we can use below Azure CLI pipeline task and execute a script like shown in the task below.

steps:
- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: 'Avalanche Development - Azure DevOps'
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
     $rgName = 'ch-demo-dev-euw-002-rg';
     $aksName = 'ch-demo-dev-euw-002-aks';
     $elasticUrl = 'http://es-search.sh.aks.ch-demo-dev-euw-002.net';
     
     $tempCurlNamespace = 'tempcurl';
     $tempCurlApp = 'temp-curl';
     
     az aks get-credentials -n $aksName -g $rgName --admin --overwrite-existing;
     
     kubectl create ns $tempCurlNamespace;
     
     kubectl run $tempCurlApp `
       -n $tempCurlNamespace `
       --image=curlimages/curl:8.9.1 `
       --restart=Never `
       --dry-run=client -o yaml `
       -- sleep 3600 |
       yq eval '.spec.tolerations = [{"key": "nodepool", "operator": "Equal", "value": "shblue", "effect": "NoSchedule"}] |
                .spec.nodeSelector = {"kubernetes.io/os": "linux", "agentpool": "shblue"}' - |
       kubectl apply -f -;
     
     kubectl wait --for=condition=Ready pod/$tempCurlApp -n $tempCurlNamespace --timeout=60s;
     kubectl get pod -n $tempCurlNamespace;
     
     
     $elasticUser = 'elastic';
     $elasticPassword = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String((kubectl get secret es-search-es-elastic-user -n elastic-stack -o jsonpath="{.data.elastic}")));
     $AuthHeader = -join('Basic ',[Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(-join($elasticUser,':',$elasticPassword))));
     
     $countResponse = (kubectl exec temp-curl -n tempcurl -- curl -k -H "Authorization: $AuthHeader" `
          -H "Content-Type: application/json" `
          -X GET "http://es-search.sh.aks.ch-demo-dev-euw-002.net/demoindex001/_count") | ConvertFrom-Json;
     
     Write-Host '------------------------------------------------';
     Write-Host 'Count results';
     Write-Host $countResponse;
     Write-Host '------------------------------------------------';
     
     $nodeResponse = (kubectl exec temp-curl -n tempcurl -- curl -k -H "Authorization: $AuthHeader" `
          -H "Content-Type: application/json" `
          -X GET "http://es-search.sh.aks.ch-demo-dev-euw-002.net/_nodes/_all?filter_path=nodes.*.name,nodes.*.attributes") | ConvertFrom-Json;
     
     Write-Host '------------------------------------------------';
     Write-Host 'Node results';
     Write-Host $nodeResponse.nodes;
     Write-Host '------------------------------------------------';
     
     kubectl delete pod $tempCurlApp -n $tempCurlNamespace;
     kubectl delete ns $tempCurlNamespace;
     kubectl config delete-context (-join($aksName,'-admin'));
     

Now let's explore each part of the script above and what it does.

First we obtain kubernetes credentias for AKS cluster with below.

$rgName = 'ch-demo-dev-euw-002-rg';
$aksName = 'ch-demo-dev-euw-002-aks';
$elasticUrl = 'http://es-search.sh.aks.ch-demo-dev-euw-002.net';
     
$tempCurlNamespace = 'tempcurl';
$tempCurlApp = 'temp-curl';
     
az aks get-credentials -n $aksName -g $rgName --admin --overwrite-existing;

Then we are creating a temporary namespace and create a pod with curl  to use as jump host. We are waiting till the pod to be ready and pod is set to run for an hour.

kubectl create ns $tempCurlNamespace;
     
kubectl run $tempCurlApp `
       -n $tempCurlNamespace `
       --image=curlimages/curl:8.9.1 `
       --restart=Never `
       --dry-run=client -o yaml `
       -- sleep 3600 |
       yq eval '.spec.tolerations = [{"key": "nodepool", "operator": "Equal", "value": "shblue", "effect": "NoSchedule"}] |
                .spec.nodeSelector = {"kubernetes.io/os": "linux", "agentpool": "shblue"}' - |
       kubectl apply -f -;
     
     kubectl wait --for=condition=Ready pod/$tempCurlApp -n $tempCurlNamespace --timeout=60s;
     kubectl get pod -n $tempCurlNamespace;

We can see a pod starts to run once the above step is executed .

Then we are creating a request to url within the vNET using the jump host pod and get output to a variable and print it. We get  document count in a given elastic index of a elastic search cluster deplyed within AKS.

$elasticUser = 'elastic';
     $elasticPassword = [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String((kubectl get secret es-search-es-elastic-user -n elastic-stack -o jsonpath="{.data.elastic}")));
     $AuthHeader = -join('Basic ',[Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(-join($elasticUser,':',$elasticPassword))));
     
     $countResponse = (kubectl exec temp-curl -n tempcurl -- curl -k -H "Authorization: $AuthHeader" `
          -H "Content-Type: application/json" `
          -X GET "http://es-search.sh.aks.ch-demo-dev-euw-002.net/demoindex001/_count") | ConvertFrom-Json;
     
     Write-Host '------------------------------------------------';
     Write-Host 'Count results';
     Write-Host $countResponse;
     Write-Host '------------------------------------------------';

This prints details as shown below in pipline logs.

We excute another request and print reults as the next step. It is similar to above step so let's skip dicussing it in detail.

Finally we cleanup the created temp jump host pod and its namespace. Then remove the credentilas form the kubectl config.

kubectl delete pod $tempCurlApp -n $tempCurlNamespace;
kubectl delete ns $tempCurlNamespace;
kubectl config delete-context (-join($aksName,'-admin'));

This allows us to use a jump host and reach urls internal to the vNet in a Microsoft hosted agent job in Azure piplines.