google-agents-cli-observability
Set up tracing, logging, and monitoring for deployed ADK agents across Cloud Trace, BigQuery, and third-party platforms.
- Four observability tiers: Cloud Trace (always enabled, distributed tracing), Prompt-Response Logging (GenAI interactions to GCS/BigQuery), BigQuery Agent Analytics (structured agent events), and third-party integrations (AgentOps, Phoenix, MLflow, Weave, Freeplay, and others)
- For Agent Runtime deployments, run
agents-cli infra single-projectbefore first deploy to provision Terraform-managed infrastructure (service account, GCS bucket, BigQuery dataset); post-deployment setup requires manual IAM and env var configuration - Cloud Trace works out of the box with OpenTelemetry spans tracking invocation flow, LLM calls, and tool execution; accessible via Cloud Console Trace explorer
- Prompt-response logging is privacy-preserving by default (metadata only via
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT); disabled locally unlessLOGS_BUCKET_NAMEis set - Includes troubleshooting guide covering missing traces, privacy misconfiguration, BigQuery setup, and cost optimization strategies
ADK Observability Guide
Cloud Trace works out of the box — no infrastructure needed. Prompt-response logging and BigQuery Agent Analytics require Terraform-provisioned infrastructure (service account, GCS bucket, BigQuery dataset). Run
agents-cli infra single-project --project PROJECT_IDto provision these resources. Seereferences/cloud-trace-and-logging.mdfor details, env vars, and verification commands. If your project isn't scaffolded yet, see/google-agents-cli-scaffoldfirst.
Order of operations for agent_runtime deployments
For deployment_target = agent_runtime, run agents-cli infra single-project before the first agents-cli deploy. The Terraform module owns the entire Reasoning Engine resource (service account, deployment spec, env vars), so applying it after an SDK-based deploy creates a state mismatch Terraform can't reconcile without taking ownership of the whole resource.
Already ran agents-cli deploy? Two options:
- Switch to Terraform-managed — delete the SDK-deployed Reasoning Engine, then run
agents-cli infra single-projectandagents-cli deploy(sessions and in-flight state are lost). - Keep the SDK-deployed instance — skip
infra single-projectand set the observability env vars by re-runningagents-cli deploy --update-env-vars "KEY=VALUE,..."; deploy matches the existing Reasoning Engine by display name and updates it in place, preserving env vars set outside the deploy. You must also grant its service account the telemetry IAM roles the Terraform module would otherwise provision:roles/storage.admin(write completions to the logs bucket),roles/logging.logWriter,roles/cloudtrace.agent, plusroles/bigquery.dataOwner+roles/bigquery.jobUserwhen scaffolded with--bq-analytics. The full set lives indeployment/terraform/single-project/iam.tf(fromapp_sa_roles) andtelemetry.tf. Terraform-managed env vars aren't available in this mode.
Reference Files
| File | Contents |
|---|---|
references/cloud-trace-and-logging.md |
Scaffolded project details — Terraform-provisioned resources, environment variables, verification commands, enabling/disabling locally |
references/bigquery-agent-analytics.md |
BQ Agent Analytics plugin — enabling, key features, GCS offloading, tool provenance |