Integrate Datadog logs, APM, and custom metrics

Nullplatform supports Datadog as a logs and metrics provider. In this section, we'll discuss how you can leverage Datadog's capabilities while working with nullplatform.

Grant nullplatform access to Datadog

To give access to nullplatform you need to create the following NRN keys

datadog.apiKeyAuth: <apikey>,
datadog.appKeyAuth": <appkey>

Then to configure datadog logs of datadog metrics as provider, in any branch of NRN you can specify

global.logProvider: "datadoglogs"
global.metricsProvider: "datadoglogs"

Send scope logs to Datadog

When you set Datadog as your logging provider, nullplatform will configure the underlying logging infrastructure of your instances to directly send logs to Datadog. By default, this will include specific tags and a service name.

info

At this time, the Datadog logs integration is available for container-based scopes only (not serverless).

Instance-based scopes

To enable Datadog for your instance-based scopes you have to:

Get a Datadog API key.
Set your Datadog API key on your organization's NRN under the aws.datadogApiKey key.

PATCH /nrn/organization=1234
{
    "aws.datadogApiKey": "my-api-key-from-datadog"
}

tip

To learn more about placing API calls to nullplatform read the API section. To learn more about how NRN works you can read the Working with NRN page.

Kubernetes-based scopes

In the null-metrics update the env section of the nullplatform-log-controller container as follows:

          env:
            - name: DATADOG_LOGS_ENABLED
              value: 'true'
            - name: DATADOG_PERFORMANCE_METRICS_ENABLED
              value: 'true'
            - name: DATADOG_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DATADOG_API_KEY

Before applying the null-metrics configuration, you need to create a Kubernetes secret named datadog-secret that contains your Datadog API key.

Use the following template to create your secret:

apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  namespace: nullplatform-tools
type: Opaque
data:
  DATADOG_API_KEY: <your key>

Enable Log

Once you have configured Datadog as Backend for our logs, we must enable it at scope level as shown below.

Custom tags

You can send custom tags to Datadog by creating environment variables named NP_DD_<tag-name> where the tag value will be the value / content of that environment variable.

For instance, if you set the environment variable NP_DD_ENV=test, it will be relayed to Datadog as env:test.

Remember that environment variables for this purpose can be easily created by adding an application parameter from your application's dashboard page.

Send nullplatform metrics to Datadog

By enabling Datadog as metric provider in your scope, nullplatform will automatically send special logs to your datadog account that can be transformed in very detailed metrics about your scope.

If you configure the following metrics, those can be shown also in the nullplatform frontend when you access to the performance section of your scope:

nullplatform.scope.memory_usage_kb
- query: source:nullplatform-sys_agg
- measure: @user_memory_kb
- Group by
  - @account
  - @account_id
  - @namespace
  - @namespace_id
  - @application
  - @application_id
  - @scope
  - @scope_id
  - @deployment_id
  - @instance_id
nullplatform.scope.memory_usage_percentage
- query: source:nullplatform-sys_agg
- measure: @memory_usage_percentage
- Group by
  - @account
  - @account_id
  - @namespace
  - @namespace_id
  - @application
  - @application_id
  - @scope
  - @scope_id
  - @deployment_id
  - @instance_id
nullplatform.scope.cpu_usage_percentage
- query: source:nullplatform-sys_agg
- measure: @cpu_usage_percentage
- Group by
  - @account
  - @account_id
  - @namespace
  - @namespace_id
  - @application
  - @application_id
  - @scope
  - @scope_id
  - @deployment_id
  - @instance_id
nullplatform.scope.request_count
- query: source:nullplatform-http_agg
- measure: @counter
- Group by
  - @account
  - @account_id
  - @namespace
  - @namespace_id
  - @application
  - @application_id
  - @scope
  - @scope_id
  - @deployment_id
  - @instance_id
  - @quality
  - @code
  - @is_healthcheck
nullplatform.scope.response_time
- query: source:nullplatform-http_agg
- measure: @response_time
- Group by
  - @account
  - @account_id
  - @namespace
  - @namespace_id
  - @application
  - @application_id
  - @scope
  - @scope_id
  - @deployment_id
  - @instance_id
  - @quality
  - @code
  - @is_healthcheck

Recommendation: set a proxy agent to deliver custom metrics & APM

To transmit custom metrics and APM data to Datadog we recommend you configure a Proxy Agent. This setup lets your application send data directly to the agent minimizing the number of host agents needed, streamlining your architecture.

Here is a diagram of the proposed architecture:

proxy agent architecture

For optimal performance and reliability, we recommend using a Network Load Balancer (NLB) with at least two agents.

Grant nullplatform access to Datadog​

Send scope logs to Datadog​

Instance-based scopes​

Kubernetes-based scopes​

Enable Log​

Custom tags​

Send nullplatform metrics to Datadog​

Recommendation: set a proxy agent to deliver custom metrics & APM​