Datadog Observability

Overview

Datadog is a SaaS observability platform providing unified monitoring across infrastructure, applications, logs, and user experience. It offers AI-powered anomaly detection, 1000+ integrations, and OpenTelemetry compatibility.

Core Capabilities:

APM: Distributed tracing with automatic instrumentation for 8+ languages
Infrastructure: Host, container, and cloud service monitoring
Logs: Centralized collection with processing pipelines and 15-month retention
Metrics: Custom metrics via DogStatsD with cardinality management
Synthetics: Proactive API and browser testing from 29+ global locations
RUM: Frontend performance with Core Web Vitals and session replay

When to Use This Skill

Activate when:

Setting up production monitoring and observability
Implementing distributed tracing across microservices
Configuring log aggregation and analysis pipelines
Creating custom metrics and dashboards
Setting up alerting and anomaly detection
Optimizing Datadog costs

Do not use when:

Building with open-source stack (use Prometheus/Grafana instead)
Cost is primary concern and budget is limited
Need maximum customization over managed solution

Quick Start

1. Install Datadog Agent

Docker (simplest):

docker run -d --name dd-agent \
  -e DD_API_KEY=<YOUR_API_KEY> \
  -e DD_SITE="datadoghq.com" \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  gcr.io/datadoghq/agent:7

Kubernetes (Helm):

helm repo add datadog https://helm.datadoghq.com
helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_API_KEY> \
  --set datadog.apm.enabled=true \
  --set datadog.logs.enabled=true

2. Instrument Your Application

Python:

from ddtrace import tracer, patch_all

# Automatic instrumentation for common libraries
patch_all()

# Manual span for custom operations
with tracer.trace("custom.operation", service="my-service") as span:
    span.set_tag("user.id", user_id)
    # your code here

Node.js:

// Must be first import
const tracer = require('dd-trace').init({
  service: 'my-service',
  env: 'production',
  version: '1.0.0',
});

3. Verify in Datadog UI

Go to Infrastructure > Host Map to verify agent
Go to APM > Services to see traced services
Go to Logs > Search to verify log collection

Core Concepts

Tagging Strategy

Tags enable filtering, aggregation, and cost attribution. Use consistent tags across all telemetry.

Required Tags:

Tag	Purpose	Example
`env`	Environment	`env:production`
`service`	Service name	`service:api-gateway`
`version`	Deployment version	`version:1.2.3`
`team`	Owning team	`team:platform`

Avoid High-Cardinality Tags:

User IDs, request IDs, timestamps
Pod IDs in Kubernetes
Build numbers, commit hashes

Unified Observability

Datadog correlates metrics, traces, and logs automatically:

Traces include span tags that link to metrics
Logs inject trace IDs for correlation
Dashboards combine all data sources

Best Practices

Start Simple

Install Agent with basic configuration
Enable automatic instrumentation
Verify data in Datadog UI
Add custom spans/metrics as needed

Progressive Enhancement

Basic → APM tracing → Custom spans → Custom metrics → Profiling → RUM

Key Instrumentation Points

HTTP entry/exit points
Database queries
External service calls
Message queue operations
Business-critical flows

Common Mistakes

High-cardinality tags: Using user IDs or request IDs as tags creates millions of unique metrics
Missing log index quotas: Leads to unexpected bills from log volume spikes
Over-alerting: Creates alert fatigue; alert on symptoms, not causes
Missing service tags: Prevents correlation between metrics, traces, and logs
No sampling for high-volume traces: Ingests everything, causing cost explosion

Navigation

For detailed implementation:

Agent Installation: Docker, Kubernetes, Linux, Windows, and cloud-specific setup
APM Instrumentation: Python, Node.js, Go, Java instrumentation with code examples
Log Management: Pipelines, Grok parsing, standard attributes, archives
Custom Metrics: DogStatsD patterns, metric types, tagging best practices
Alerting: Monitor types, anomaly detection, alert hygiene
Cost Optimization: Metrics without Limits, sampling, index quotas
Kubernetes: DaemonSet, Cluster Agent, autodiscovery

Complementary Skills

When using this skill, consider these related skills (if deployed):

docker: Container instrumentation patterns
kubernetes: K8s-native monitoring patterns
python/nodejs/go: Language-specific APM setup

Resources

Official Documentation:

APM: https://docs.datadoghq.com/tracing/
Logs: https://docs.datadoghq.com/logs/
Metrics: https://docs.datadoghq.com/metrics/
DogStatsD: https://docs.datadoghq.com/developers/dogstatsd/

Cost Management:

Billing: https://docs.datadoghq.com/account_management/billing/
Usage Attribution: https://docs.datadoghq.com/account_management/billing/usage_attribution/

datadog-observability