Monitoring Basics: Getting Started with Prometheus and Grafana

You cannot fix what you cannot see. When a service slows down or an instance runs out of disk, the difference between a calm afternoon and a stressful one usually comes down to whether you had visibility before things broke. Prometheus and Grafana are a common, approachable pairing for turning raw system signals into dashboards and alerts you can rely on.

How the pieces fit together

• Prometheus scrapes metrics from targets on a schedule and stores them as time series.
• Exporters expose metrics in a format Prometheus understands — node_exporter for hosts, plus many for databases and services.
• Grafana queries Prometheus and turns those series into dashboards.
• Alertmanager receives alerts from Prometheus and routes them to email, chat or paging.

A minimal Prometheus configuration

Prometheus is configured declaratively in a YAML file. A good starting point is to scrape Prometheus itself and a single node exporter, then grow from there as you add services.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]

Running the stack quickly with Docker

For a first look, Compose is the fastest way to get Prometheus, node_exporter and Grafana running together on one machine.

services:
  prometheus:
    image: prom/prometheus
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
  node-exporter:
    image: prom/node-exporter
    ports: ["9100:9100"]
  grafana:
    image: grafana/grafana
    ports: ["3000:3000"]

From metrics to insight in Grafana

Once data is flowing, add Prometheus as a data source in Grafana and build a small dashboard for the signals that matter most: CPU, memory, disk usage and request latency. Resist the urge to graph everything on day one — a focused dashboard is far more useful than a wall of panels nobody reads.

Prometheus queries use PromQL. A common example is turning a raw counter into a per-second rate, which is what you almost always want to visualise.

# CPU busy percentage per instance
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Alert on symptoms, not noise

Good alerts describe user-facing pain, not every transient blip. Start with a few high-signal rules — disk nearly full, service down, latency above a threshold for several minutes — and use a for clause so brief spikes do not page anyone.

groups:
  - name: basics
    rules:
      - alert: HostHighDiskUsage
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) > 0.9
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Disk usage above 90% on {{ $labels.instance }}"

Where to go next

• Add exporters for the databases and services you actually run.
• Version-control your Prometheus config and Grafana dashboards.
• Tune alert thresholds as you learn what “normal” looks like for your systems.

Start small, keep the dashboards focused, and let real incidents guide what you add next. A little visibility, set up well, goes a very long way.