Setting Up Prometheus with Docker Compose

This guide provides a step-by-step approach to set up Prometheus with Docker Compose, including troubleshooting common issues.

Step 1: Clean Up Docker Environment

First, ensure all existing containers and networks are removed to start fresh:

docker-compose down -v
docker system prune -f
docker volume prune -f

Step 2: Verify and Correct Configuration Files

Ensure the docker-compose.yml and prometheus.yml files are correctly formatted and contain no errors.

docker-compose.yml

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert.rules:/etc/prometheus/alert.rules
    networks:
      - monitoring

  node_exporter:
    image: prom/node-exporter:latest
    container_name: node_exporter
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

prometheus.yml

global:
  scrape_interval: 15s

rule_files:
  - /etc/prometheus/alert.rules

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

alert.rules

Create a file named alert.rules if it doesn't exist:

groups:
  - name: example
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

Step 3: Restart Docker Services

Restart the Docker services to apply the changes:

docker-compose up --build -d

Step 4: Verify Configuration and Connectivity

Check the container logs to ensure Prometheus started correctly:

docker-compose logs prometheus

Step 5: Test Prometheus API

Test the Prometheus API to ensure it is working correctly.

Check Prometheus Targets

curl http://localhost:9090/api/v1/targets

Query Node Exporter Status

curl 'http://localhost:9090/api/v1/query?query=up{job="node_exporter"}'

If the above command returns a parsing error, use the URL-encoded query:

curl 'http://localhost:9090/api/v1/query?query=up%7Bjob%3D%22node_exporter%22%7D'

Summary

By ensuring the configuration files are correctly formatted and starting fresh with the Docker environment, you should be able to resolve issues and get Prometheus running smoothly. If you encounter any further issues, check the container logs and configuration for errors.

Additional Troubleshooting Tips

Ensure Prometheus is Running: Verify that Prometheus is running by checking the Docker container status with docker ps.
Check Network Configuration: Ensure the Docker network is correctly set up and that containers are connected to the monitoring network.
Access Prometheus Web UI: Navigate to http://localhost:9090 in your web browser to access the Prometheus web interface and verify the targets and query metrics.

This document should serve as a comprehensive reference for setting up and troubleshooting Prometheus with Docker Compose.

3.3 KiB Raw Blame History