k8s-prometheus-troubleshooting
Prometheus troubleshooting
Find out why Prometheus does not scrape a target, does not load a rule, or does not alert, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
With the Prometheus Operator, a Prometheus custom resource selects
ServiceMonitors, PodMonitors, Probes, ScrapeConfigs and
PrometheusRules by label and namespace, and the operator turns them into
Prometheus' configuration. A ServiceMonitor selects Services by label and names one of
their ports. Something can go wrong at each step: the selection, the Service, the port,
the scrape itself.