Prometheus with K8s

Prometheus with K8s

Monitoring Kubernetes (K8s) clusters is essential for ensuring application health, performance, and uptime. Prometheus is one of the most widely adopted open-source tools for collecting, storing, and querying metrics from Kubernetes. In this guide, we'll walk you through how Prometheus integrates with Kubernetes, why it's effective, and how to get it up and running using both Helm and manual configuration.

🔍 Why Use Prometheus with Kubernetes?

Kubernetes orchestrates a lot of moving parts — pods, services, deployments, nodes — and Prometheus helps you keep an eye on all of them by:

• Scraping metrics from K8s-native endpoints (like kubelet, API server, cAdvisor)
• Alerting when thresholds are breached (CPU spikes, pod restarts)
• Querying data via PromQL (Prometheus Query Language)
• Visualizing data with Grafana dashboards

The best part? Prometheus is cloud-agnostic and can run inside any Kubernetes cluster.

⚙️ Installing Prometheus using Helm

Helm is the easiest way to install Prometheus with recommended defaults:

# Step 1: Add Prometheus Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Step 2: Update local chart list
helm repo update

# Step 3: Install Prometheus stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

This chart sets up Prometheus, Alertmanager, node exporters, kube-state-metrics, and Grafana all in one go.

📡 How Prometheus Collects Metrics in Kubernetes

Prometheus scrapes metrics using a pull-based model. In a Kubernetes environment, it discovers targets using the built-in service discovery features.

Key endpoints Prometheus scrapes in K8s:

• /metrics from application pods
• Node exporter for system metrics
• Kubelet for node and pod-level stats
• cAdvisor for container-level metrics
• kube-state-metrics for resource status (like pod status, deployment count)

📝 Sample ServiceMonitor Example

If you want Prometheus to scrape metrics from a custom app running on port 8080 with metrics exposed at /metrics, you can define a ServiceMonitor like this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

Make sure the service for your app exposes the right port and is labeled accordingly.

🔍 Accessing Prometheus & Grafana UI

Once installed, expose the UIs using port forwarding:

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090

kubectl port-forward -n monitoring svc/prometheus-grafana 3000

Then access Prometheus at http://localhost:9090 and Grafana at http://localhost:3000. The default Grafana login is usually admin/admin.

🧪 Writing Basic PromQL Queries

Here are a few sample PromQL queries you can run inside the Prometheus UI:

# CPU usage of a pod
rate(container_cpu_usage_seconds_total{pod="my-app"}[1m])

# Memory usage by namespace
sum(container_memory_usage_bytes) by (namespace)

# Number of restarts
kube_pod_container_status_restarts_total

PromQL allows powerful aggregations, filters, and time-series visualizations.

📣 Alerting with Prometheus

Alert rules can be defined inside Prometheus or using the Helm chart values:

groups:
- name: example-alert
  rules:
  - alert: HighPodRestarts
    expr: increase(kube_pod_container_status_restarts_total[5m]) > 3
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Pod restarts exceed threshold"

These alerts can be routed to Slack, email, PagerDuty, etc., using Alertmanager.

📌 Best Practices

• Use relabeling to reduce noisy metrics and scrape targets.
• Monitor Prometheus itself — it's just another pod.
• Set up recording rules to pre-aggregate frequently queried data.
• Use persistent storage for Prometheus data (PVCs).

✅ Conclusion

Prometheus with Kubernetes provides deep insights into your application and infrastructure. Whether you want to debug performance issues, track resource utilization, or set up automated alerts, Prometheus can handle it all. By leveraging Helm, ServiceMonitors, and PromQL, you can monitor even complex microservice environments with confidence.