Skills

opentelemetry

Installation
SKILL.md

OpenTelemetry with Grafana

Overview

OpenTelemetry (OTel) is a vendor-neutral framework for collecting observability data (metrics, logs, traces, profiles). Grafana Labs integrates it as a core strategy, offering a full stack to collect, ingest, store, analyze, and visualize telemetry data.

Four-Step Implementation Model

  1. Instrument - Add telemetry using Grafana SDKs, Beyla (eBPF), or upstream OTel SDKs
  2. Pipeline - Build processing infrastructure with Grafana Alloy or OTel Collector
  3. Ingest - Route data to Grafana Cloud OTLP endpoint or self-managed backends
  4. Analyze - Dashboards, alerts, Application Observability, Drilldown apps

Grafana Backends

Signal Backend
Metrics Grafana Mimir
Logs Grafana Loki
Traces Grafana Tempo
Profiles Grafana Pyroscope

OTLP Endpoint and Authentication

Grafana Cloud OTLP Endpoint

Grafana Cloud exposes a managed OTLP gateway endpoint:

https://otlp-gateway-<region>.grafana.net/otlp

Example regions: prod-us-east-0, prod-eu-west-0, prod-ap-southeast-0

Full example:

https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Authentication - Basic Auth

Grafana Cloud OTLP uses HTTP Basic Auth:

  • Username: Grafana Cloud Instance ID (numeric, e.g. 123456)
  • Password: Grafana Cloud API token (with MetricsPublisher, LogsPublisher, TracesPublisher roles)

Via environment variable (recommended)

# Base64-encode "instanceID:apiToken"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n '123456:glc_eyJ...' | base64)"

Via Alloy environment variables

export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp

Direct Send (no collector) - Environment Variables

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64(instanceID:apiToken)>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=production"

Instrumentation by Language

Go

Requirements: Go 1.22+

Install packages:

go get "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" \
  "go.opentelemetry.io/contrib/instrumentation/runtime" \
  "go.opentelemetry.io/otel" \
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \
  "go.opentelemetry.io/otel/sdk" \
  "go.opentelemetry.io/otel/sdk/metric"

Run with environment variables:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
go run .

See references/instrumentation.md for full Go code example.

Java (Grafana Distribution - JVM Agent)

Requirements: JDK 8+

Download: grafana-opentelemetry-java.jar from https://github.com/grafana/grafana-opentelemetry-java/releases

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=shoppingcart,service.namespace=ecommerce,deployment.environment=production" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
java -javaagent:/path/to/grafana-opentelemetry-java.jar -jar myapp.jar

Optional: Data saver mode (reduces metric cardinality):

export GRAFANA_OTEL_APPLICATION_OBSERVABILITY_METRICS=true

Debug:

export OTEL_JAVAAGENT_DEBUG=true
# Enable console output alongside OTLP
export OTEL_TRACES_EXPORTER=otlp,console
export OTEL_METRICS_EXPORTER=otlp,console
export OTEL_LOGS_EXPORTER=otlp,console

Node.js

Install:

npm install --save @opentelemetry/api
npm install --save @opentelemetry/auto-instrumentations-node

Run:

OTEL_TRACES_EXPORTER="otlp" \
OTEL_METRICS_EXPORTER="otlp" \
OTEL_LOGS_EXPORTER="otlp" \
OTEL_NODE_RESOURCE_DETECTORS="env,host,os" \
OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
node app.js

Warning: Bundlers like @vercel/ncc can break auto-instrumentation hooks.

See references/instrumentation.md for manual SDK setup example.

Python

Install:

pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
opentelemetry-instrument python app.py

Multi-process servers (Gunicorn, uWSGI): implement post-fork hooks to reinitialize OTel providers per worker.

.NET (Grafana Distribution)

Install NuGet:

dotnet add package Grafana.OpenTelemetry

ASP.NET Core setup:

using Grafana.OpenTelemetry;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry()
    .WithTracing(configure => configure.UseGrafana())
    .WithMetrics(configure => configure.UseGrafana());
builder.Logging.AddOpenTelemetry(options => options.UseGrafana());

Run:

OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64>" \
dotnet run

Requirements: .NET 6+ or .NET Framework 4.6.2+

See references/instrumentation.md for full .NET examples.

Beyla (eBPF - Language Agnostic)

Grafana Beyla instruments at the network layer - no code changes required, works with any language.

# Docker
docker run --rm -it \
  --privileged \
  -e BEYLA_SERVICE_NAME=myapp \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
  -v /sys/kernel/security:/sys/kernel/security \
  grafana/beyla

Verify with: curl http://localhost:9090/metrics

Full docs: https://grafana.com/docs/beyla/

Grafana Alloy Collector

Grafana Alloy is the recommended OTel Collector distribution. It combines upstream OTel Collector components with Prometheus exporters for infrastructure + application observability correlation.

Why Use a Collector?

  • Cost control: Aggregate, sample, and drop data before sending
  • Reliability: Buffer and retry on connection failures
  • Enrichment: Add resource attributes, transform, redact, and route data

Alloy Ports

Port Protocol Purpose
4317 gRPC OTLP gRPC receiver
4318 HTTP OTLP HTTP/protobuf receiver

Application -> Alloy -> Grafana Cloud

Application env vars (point to local Alloy):

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Alloy config env vars (Alloy -> Grafana Cloud):

export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...

See references/collector-config.md for full Alloy configuration.

Kubernetes Setup

Option 1: Grafana Kubernetes Monitoring Helm Chart (recommended)

The Grafana Kubernetes Monitoring Helm chart deploys Alloy with OTLP receivers pre-configured.

  1. Enable "OTLP Receivers" in the Cluster Configuration tab
  2. Get gRPC/HTTP endpoints from "Configure Application Instrumentation" section
  3. Point apps to the in-cluster Alloy endpoint:
export OTEL_EXPORTER_OTLP_ENDPOINT=<GRPC_ENDPOINT_FROM_HELM>
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Option 2: OpenTelemetry Operator

Install via official docs, then use Instrumentation CR for auto-injection:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otelcol:4317
  propagators:
    - tracecontext
    - baggage
  java:
    # Use Grafana distribution image
    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1
  nodejs: {}
  python: {}

Inject into pods with annotation:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    # or: inject-nodejs, inject-python, inject-dotnet

See references/collector-config.md for Kubernetes Alloy Helm values and OTel Collector YAML.

Sampling Strategies

Head-Based Sampling

Decision made at trace start - low overhead, may miss rare errors.

Environment variable (probability sampler):

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # 10% of traces

Alloy head sampling config:

otelcol.processor.probabilistic_sampler "default" {
  sampling_percentage = 10
  output {
    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

Tail-Based Sampling

Decision made after all spans collected - can sample based on outcome (e.g. keep all errors).

Alloy tail sampling config:

otelcol.processor.tail_sampling "default" {
  decision_wait            = "10s"
  num_traces               = 100000
  expected_new_traces_per_sec = 10

  policy {
    name = "keep-errors"
    type = "status_code"
    status_code {
      status_codes = ["ERROR"]
    }
  }

  policy {
    name = "probabilistic-sample"
    type = "probabilistic"
    probabilistic {
      sampling_percentage = 10
    }
  }

  output {
    traces = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

Key Environment Variables Reference

Variable Description Example
OTEL_EXPORTER_OTLP_ENDPOINT OTLP receiver URL https://otlp-gateway-prod-us-east-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_PROTOCOL Transport protocol grpc or http/protobuf
OTEL_EXPORTER_OTLP_HEADERS Auth headers Authorization=Basic <base64>
OTEL_RESOURCE_ATTRIBUTES Service metadata service.name=myapp,service.namespace=team,deployment.environment=prod
OTEL_TRACES_EXPORTER Trace exporter type otlp
OTEL_METRICS_EXPORTER Metrics exporter type otlp
OTEL_LOGS_EXPORTER Logs exporter type otlp
OTEL_SERVICE_NAME Service name (shorthand) myapp
OTEL_TRACES_SAMPLER Sampler type parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG Sampler argument 0.1 (10%)

Key Resource Attributes

Attribute Purpose Example
service.name Service identifier shoppingcart
service.namespace Groups related services ecommerce
deployment.environment Environment tier production, staging
service.version App version 1.2.3

Useful Links

Related skills

More from grafana/skills

Installs
206
Repository
grafana/skills
GitHub Stars
35
First Seen
Apr 14, 2026
Security Audits
Gen Agent Trust HubWarn
SocketPass
SnykWarn