Reducing Log Analytics Cost for App Insights by Removing Successful Dependency Logs Ingestion from .NET Applications

 We have discussed how to reduce log analytics cost by reducing container log ingestion from apps deployed to AKS in the post "Reducing Log Analytics Cost by Preventing Container Logs Ingession from Azure Kubernetes Services (AKS)". However, when we inspected the charts o f data ingestion after reducing container logs, the next major data volume is ingested by app dependecy logs, coming from .NET apps running in AKS cluster. App dependecy logs are mostly information logs coming from the .NET SDK and other dependecies such as Azure storage etc. These logs are useful when there is an issue or error in the dependecy calls. However, when the dependency has run with successful state, there is not much use of that information. Since, the highest cost for app insights log analytics cost is incurred by dependency logs after we have removed container logs, it is worth to reduce the cost of log analytics data ingestion for app insights, by removing the successful dependecy logs from the app insights.

The expected outcome is as shown below. After the removal of succesful dpenedecy logs data ingestion for dependecy logs has reduce to almost zero.

Note in below around 10:50 AM the fixed apps are deployed and dependency logs volume reduced significantly., There are few dependecy failures appearing as expected after the fix, but no more success dependecy logs.

In your .NET apps you need to define a successful app dependecy filter class as shown below. This will filter the succesful dependecies for app insights.

using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.ApplicationInsights.DataContracts;

namespace MyDemoApp
{
    public class SuccessfulAppInsightsDependencyFilter : ITelemetryProcessor
    {
        private ITelemetryProcessor Next { get; set; }

        public SuccessfulAppInsightsDependencyFilter(ITelemetryProcessor next)
        {
            Next = next;
        }

        public void Process(ITelemetry telemetry)
        {
            if (IsSuccessDependency(telemetry))
            {
                return;
            }

            Next.Process(telemetry);
        }

        private static bool IsSuccessDependency(ITelemetry telemetry)
        {
            if (telemetry is not DependencyTelemetry dependencyTelemetry)
            {
                return false;
            }

            return dependencyTelemetry.Success is true;
        }
    }
}

Then, if we are using AddApplicationInsightsTelemetryWorkerService then we need to define a Microsoft.ApplicationInsights.WorkerService.ITelemetryProcessorFactory implementation as shown below.

using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.ApplicationInsights.WorkerService;

namespace MyDemoApp
{
    public class SuccessfulAppInsightsWorkerDependencyFilterFactory : ITelemetryProcessorFactory
    {
        public ITelemetryProcessor Create(ITelemetryProcessor telemetryProcessor)
        {
            return new SuccessfulAppInsightsDependencyFilter(telemetryProcessor);
        }
    }
}

We can inject the factoy and use app insights deppendecy filter as shown below with Microsoft.Extensions.Hosting.IHostBuilder .

hostBuilder
	.UseConsoleLifetime()
        .ConfigureServices((serviceCollection) =>
        {
        	serviceCollection
                	.AddSingleton<ITelemetryProcessorFactory, SuccessfulAppInsightsWorkerDependencyFilterFactory>()
                        .AddApplicationInsightsTelemetryWorkerService();
	})
        .ConfigureLogging(loggingBuilder =>
        	loggingBuilder.AddApplicationInsights());

If we are using AddApplicationInsightsTelemetry then we need to define Microsoft.ApplicationInsights.AspNetCore.ITelemetryProcessorFactory as shown below.

using Microsoft.ApplicationInsights.AspNetCore;
using Microsoft.ApplicationInsights.Extensibility;

namespace MyDemoApp
{
    public class SuccessfulAppInsightsApiDependencyFilterFactory : ITelemetryProcessorFactory
    {
        public ITelemetryProcessor Create(ITelemetryProcessor telemetryProcessor)
        {
            return new SuccessfulAppInsightsDependencyFilter(telemetryProcessor);
        }
    }
}

We can inject the factoy and use app insights deppendecy filter as shown below with Microsoft.AspNetCore.Builder.WebApplicationBuilder .

builder.Services
	.AddSingleton<ITelemetryProcessorFactory, SuccessfulAppInsightsApiDependencyFilterFactory>()
        .AddApplicationInsightsTelemetry();

builder.Logging.AddApplicationInsights();