Datadog APM

Distributed tracing, service maps, and performance analysis.

Requirements

Datadog Labs Pup should be installed. See Setup Pup if not.

Quick Start

pup auth login
pup apm services list
pup apm traces list --service api-gateway --duration 1h

Services

List Services

pup apm services list
pup apm services list --env production

Service Details

pup apm services get api-gateway --json

Service Map

# View dependencies
pup apm service-map --service api-gateway --json

Traces

Search Traces

# By service
pup apm traces list --service api-gateway --duration 1h

# Errors only
pup apm traces list --service api-gateway --status error

# Slow traces (>1s)
pup apm traces list --service api-gateway --min-duration 1000ms

# With specific tag
pup apm traces list --query "@http.url:/api/users"

Get Trace Detail

pup apm traces get <trace_id> --json

Key Metrics

Metric	What It Measures
`trace.http.request.hits`	Request count
`trace.http.request.duration`	Latency
`trace.http.request.errors`	Error count
`trace.http.request.apdex`	User satisfaction

⚠️ Trace Sampling

Not all traces are kept. Understand sampling:

Mode	What's Kept
Head-based	Random % at start
Error/Slow	All errors, slow traces
Retention	What's indexed (billed)

# Check retention filters
pup apm retention-filters list

Trace Retention Costs

Retention	Cost
Indexed spans	$$$ per million
Ingested spans	$ per million

Best practice: Only index what you need for search.

Service Level Objectives

Link APM to SLOs:

pup slos create \
  --name "API Latency p99 < 200ms" \
  --type metric \
  --numerator "sum:trace.http.request.hits{service:api,@duration:<200000000}" \
  --denominator "sum:trace.http.request.hits{service:api}" \
  --target 99.0

Common Queries

Goal	Query
Slowest endpoints	`avg:trace.http.request.duration{*} by {resource_name}`
Error rate	`sum:trace.http.request.errors{} / sum:trace.http.request.hits{}`
Throughput	`sum:trace.http.request.hits{*}.as_rate()`

Troubleshooting

Problem	Fix
No traces	Check ddtrace installed, DD_TRACE_ENABLED=true
Missing service	Verify DD_SERVICE env var
Traces not linked	Check trace headers propagated
High cardinality	Don't tag with user_id/request_id

dd-apm