Reducing Log Analytics Cost by Preventing Container Logs Ingestion from Azure Kubernetes Services (AKS)

 Monitoring is an essential part of a deployment of software on a platform such as AKS. However, once monitoring enabled there could be significant cost involved for monitoring data. When we enable log analytics workspace to ingest monitoring data from AKS, by default AKS will ingest all container logs, except for kube-system and gatekeeper-system namespaces. If our application are having large amout of container logs generated, then the cost will be lot higher for log analytics workspace. In case we are using app insights/ or an alternative system to monitor the application logs, for the applications deployed to AKS, we have enuough information to diagnose issues etc. Therefore, to reduce unnecessary cost log analytics can be done, by preventing AKS ingestion of the container logs to log analytics workspace. Let's explore the steps.

Expected outcome

As shown in the below figure, the log analytics workspace ingestion over time chart, is indicating the container logs data is no longer getting ingested after the change is applied.

 

Passing Environment Variables to Reusable Workflows in GitHub Actions

 We have discueed, that we have to use an environment variable to handle input parameter default values, if we are using trigger for workflow on push in the post "Setting Workflow Environment Variable Based on Input Parameter in GitHub Actions - on workflow_dispatch and Use a Default Value on push". If we have to pass on the input paramter value from a workflow to a reusable workflow, it does not work as expected and it is a limitation of reusable workflows  as explained in here. Let's try to understand with an example how to pass an env variable to a reusable workflow.

Setting Workflow Environment Variable Based on Input Parameter in GitHub Actions - on workflow_dispatch and Use a Default Value on push

 GitHub actions workflows support input only on manual trigger workflow_dispatch. What if we need to use default value in other triggers such as on push and use input parameter in case of manual trigger workflow_dispatch? Let's explore our options with an example.

Avoid GitHub Action Workflow Code Duplication - Using Composite Actions to Create Reusable Templates

 Azure DevOps yaml files for piplines can be created as templates to avoid duplicating same set of tasks in multiple areas of the pipelines. For GitHub actions workflow, we have seen here in "Build and Unit Tests for .NET Apps with GitHub Actions" the same steps are repeated in both Windows and Linux, build and unit test run. The solution to avoid duplicating same steps in multiple jobs in a GitHub actions workflow is the usage of composite actions. Let's modify our workflow implemented in "Build and Unit Tests for .NET Apps with GitHub Actions" to use composite action to build and unit test, and reuse that in both Windows and Linux job.

Running Pod Counts for All Services Summary - Grafana Chart with Azure Monitor for AKS

Monitoring the actual running pod count, against the desired pod count by each horizontal pod autoscaler, for your application is useful to get a summarize view, that will help to understand ow the pods are really scaling based on  scale out demand of each individual service. We can enable monitoring AKS cluster with Managed Grafana in Azure with Azure Monitor and Log Analytics Workspace. Let's write a query to and create Grafana chart for this purpose.

Expected Outcome

Each service actual scaling over time is shown as summary. This indicate based on load how each service in your application scale out based on the demanded by horizontal pod autoscaler.