medusa/the_information_nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
the_information_nexus/tech_docs/prometheus.md
medusa aa0500e728 Update tech_docs/prometheus.md
2024-05-29 19:14:01 +00:00

23 KiB
Raw Permalink Blame History

To add Grafana to your setup, we need to extend the docker-compose.yml file and configure Grafana to use Prometheus as a data source. Here are the steps:

Step 1: Extend docker-compose.yml to Include Grafana

Add the Grafana service to your docker-compose.yml file:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert.rules:/etc/prometheus/alert.rules
    networks:
      - monitoring

  node_exporter:
    image: prom/node-exporter:latest
    container_name: node_exporter
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    networks:
      - monitoring
    volumes:
      - grafana-storage:/var/lib/grafana

networks:
  monitoring:
    driver: bridge

volumes:
  grafana-storage:

Step 2: Restart Docker Services

Restart your Docker services to include Grafana:

docker-compose down
docker-compose up -d

Step 3: Configure Grafana

  1. Access Grafana: Open your web browser and go to http://localhost:3000 (Grafana default credentials: admin/admin).

  2. Add Prometheus Data Source:

    • Go to Configuration > Data Sources > Add data source.
    • Select Prometheus.
    • Set the URL to http://prometheus:9090 and save.

  3. Create a Dashboard:

    • Create a new dashboard.
    • Add new panels to visualize metrics from Node Exporter, such as CPU usage, memory usage, disk usage, etc.

Example PromQL Queries for Grafana Panels

  • CPU Usage:

    100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

  • Memory Usage:

    (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

  • Disk Usage:

    100 - ((node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs|rootfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|rootfs"}) * 100)

Step 4: Verify the Setup

  1. Check Grafana Dashboard: Open Grafana at http://localhost:3000 and verify that you can see the metrics from your Linux systems.

  2. Check Prometheus Targets: Open Prometheus at http://localhost:9090/targets to ensure all targets are being scraped correctly.

Summary

By adding Grafana to your Docker Compose setup and configuring it to use Prometheus as a data source, you can create powerful dashboards to visualize metrics from your Linux systems. This provides a comprehensive monitoring solution using Prometheus and Grafana. If you have any questions or need further assistance, feel free to ask!

Key Metrics and KPIs to Monitor

  1. CPU Usage
  2. Memory Usage
  3. Disk Usage
  4. Network Traffic
  5. System Load
  6. Uptime
  7. Temperature (if available)

Step-by-Step Guide to Create a Grafana Dashboard

Step 1: Access Grafana

Open your web browser and go to http://localhost:3000 (Grafana default credentials: admin/admin).

Step 2: Add Prometheus Data Source

  1. Configuration > Data Sources > Add data source
  2. Select Prometheus
  3. Set the URL to http://prometheus:9090 and save

Step 3: Create a New Dashboard

  1. Dashboard > New Dashboard > Add a New Panel

Step 4: Add Panels with PromQL Queries

Here are the important metrics and their corresponding PromQL queries:

  1. CPU Usage

    • Panel Title: CPU Usage (%)
    • PromQL Query: 
      100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

  2. Memory Usage

    • Panel Title: Memory Usage (%)
    • PromQL Query: 
      (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

  3. Disk Usage

    • Panel Title: Disk Usage (%)
    • PromQL Query: 
      100 - ((node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs|rootfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|rootfs"}) * 100)

  4. Network Traffic

    • Panel Title: Network Inbound Traffic (Bytes/s)

    • PromQL Query:

      rate(node_network_receive_bytes_total[5m])

    • Panel Title: Network Outbound Traffic (Bytes/s)

    • PromQL Query:

      rate(node_network_transmit_bytes_total[5m])

  5. System Load

    • Panel Title: System Load (1m)
    • PromQL Query: 
      node_load1

  6. Uptime

    • Panel Title: System Uptime (seconds)
    • PromQL Query: 
      node_time_seconds - node_boot_time_seconds

  7. Temperature (if available)

    • Panel Title: CPU Temperature (°C)
    • PromQL Query: 
      node_hwmon_temp_celsius

Example Panel Configurations

CPU Usage Panel

  1. Add a new panel.
  2. Set the title to "CPU Usage (%)".
  3. Enter the PromQL query: 
    100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
  4. Set visualization type to "Graph".
  5. Customize the visualization settings (e.g., set y-axis unit to percentage).

Memory Usage Panel

  1. Add a new panel.
  2. Set the title to "Memory Usage (%)".
  3. Enter the PromQL query: 
    (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100
  4. Set visualization type to "Graph".
  5. Customize the visualization settings (e.g., set y-axis unit to percentage).

Disk Usage Panel

  1. Add a new panel.
  2. Set the title to "Disk Usage (%)".
  3. Enter the PromQL query: 
    100 - ((node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs|rootfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|rootfs"}) * 100)
  4. Set visualization type to "Graph".
  5. Customize the visualization settings (e.g., set y-axis unit to percentage).

Network Traffic Panels

Inbound Traffic

  1. Add a new panel.
  2. Set the title to "Network Inbound Traffic (Bytes/s)".
  3. Enter the PromQL query: 
    rate(node_network_receive_bytes_total[5m])
  4. Set visualization type to "Graph".
  5. Customize the visualization settings (e.g., set y-axis unit to bytes/sec).

Outbound Traffic

  1. Add a new panel.
  2. Set the title to "Network Outbound Traffic (Bytes/s)".
  3. Enter the PromQL query: 
    rate(node_network_transmit_bytes_total[5m])
  4. Set visualization type to "Graph".
  5. Customize the visualization settings (e.g., set y-axis unit to bytes/sec).

System Load Panel

  1. Add a new panel.
  2. Set the title to "System Load (1m)".
  3. Enter the PromQL query: 
    node_load1
  4. Set visualization type to "Graph".
  5. Customize the visualization settings (e.g., set y-axis unit to none).

Uptime Panel

  1. Add a new panel.
  2. Set the title to "System Uptime (seconds)".
  3. Enter the PromQL query: 
    node_time_seconds - node_boot_time_seconds
  4. Set visualization type to "Stat".
  5. Customize the visualization settings (e.g., set y-axis unit to seconds).

Temperature Panel (if available)

  1. Add a new panel.
  2. Set the title to "CPU Temperature (°C)".
  3. Enter the PromQL query: 
    node_hwmon_temp_celsius
  4. Set visualization type to "Graph".
  5. Customize the visualization settings (e.g., set y-axis unit to degrees Celsius).

Summary

By setting up these panels in Grafana, you'll have a comprehensive dashboard displaying key metrics and KPIs for your Linux systems. This will provide valuable insights into the performance and health of your infrastructure.

If you have specific metrics or additional customizations you'd like to include, feel free to ask!

Directory Structure

We'll organize the directories under /volume1/docker/prometheus as follows:

/volume1/docker/prometheus
├── alertmanager
│   └── config.yml
├── grafana
├── loki
│   └── local-config.yaml
├── prometheus
│   └── prometheus.yml
├── promtail
│   └── config.yml
├── docker-compose.yml

Docker Compose File (docker-compose.yml)

Update the docker-compose.yml with absolute paths:

---
version: '3.8'

networks:
  loki:
  monitoring:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/prometheus:/etc/prometheus
      - prom_data:/prometheus
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    ports:
      - 9093:9093
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/config.yml'
    networks:
      - monitoring
    depends_on:
      - prometheus

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - '3030:3000'
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/grafana:/var/lib/grafana
    networks:
      - monitoring
      - loki
    depends_on:
      - prometheus

  node_exporter:
    image: prom/node-exporter
    container_name: node_exporter
    ports:
      - 9100:9100
    restart: unless-stopped
    networks:
      - monitoring
    depends_on:
      - prometheus

  loki:
    image: grafana/loki:2.6.0
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - /volume1/docker/prometheus/loki:/etc/loki
      - loki_data:/loki
    networks:
      - loki
    depends_on:
      - promtail

  promtail:
    image: grafana/promtail:2.6.0
    container_name: promtail
    volumes:
      - /var/log:/var/log
      - /volume1/docker/prometheus/promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki
    depends_on:
      - loki

volumes:
  prom_data:
  grafana-storage:
  loki_data:

Prometheus Configuration File (prometheus.yml)

Create the prometheus.yml file in the /volume1/docker/prometheus/prometheus directory:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

Alertmanager Configuration File (config.yml)

Create the config.yml file in the /volume1/docker/prometheus/alertmanager directory:

global:
  resolve_timeout: 5m

route:
  receiver: 'default'

receivers:
  - name: 'default'

Loki Configuration File (local-config.yaml)

Create the local-config.yaml file in the /volume1/docker/prometheus/loki directory:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Promtail Configuration File (config.yml)

Create the config.yml file in the /volume1/docker/prometheus/promtail directory:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Configuring Docker to Route Logs to Loki

  1. Install the Docker Loki plugin:

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

  2. Configure Docker daemon to use Loki:

    Edit or create the Docker daemon configuration file (dockerd.json):

    Synology:

    sudo vi var/packages/ContainerManager/etc/dockerd.json

    Add the following content:

    {
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push"
  }
}

  3. Restart Docker daemon:

    Synology:

    sudo synopkgctl stop ContainerManager
sudo synopkgctl start ContainerManager

Configuring Grafana to Query Logs

  1. Open Grafana:

    Navigate to http://192.168.1.67:3000.

  2. Log in:

    Use the default credentials (admin/admin) and change the password upon first login.

  3. Add Loki as a Data Source:

    • Go to Configuration > Data Sources > Add data source.
    • Choose Loki.
    • Set the URL to http://loki:3100.
    • Click Save & Test.

Deploying the Stack

With all the configuration files in place, navigate to the /volume1/docker/prometheus directory and start the stack using Docker Compose:

cd /volume1/docker/prometheus
docker-compose up -d

Conclusion

You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!

Yes, using --- at the beginning of a YAML document is a good practice as it explicitly denotes the start of a document. Here is the revised Docker Compose file with the inclusion of ---:

Docker Compose File (docker-compose.yml)

---
version: '3.8'

networks:
  loki:
  monitoring:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/prometheus:/etc/prometheus
      - /volume1/docker/prometheus/prom_data:/prometheus
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    ports:
      - 9093:9093
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/config.yml'
    networks:
      - monitoring

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/grafana:/var/lib/grafana
    networks:
      - monitoring
      - loki

  node_exporter:
    image: prom/node-exporter
    container_name: node_exporter
    ports:
      - 9100:9100
    restart: unless-stopped
    networks:
      - monitoring

  loki:
    image: grafana/loki:2.6.0
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - /volume1/docker/prometheus/loki:/etc/loki
      - /volume1/docker/prometheus/loki_data:/loki
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.0
    container_name: promtail
    volumes:
      - /var/log:/var/log
      - /volume1/docker/prometheus/promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

volumes:
  prom_data:
  grafana-storage:
  loki_data:

Updated Directory Structure

Here is a reminder of the directory structure:

/volume1/docker/prometheus
├── alertmanager
│   └── config.yml
├── grafana
├── loki
│   └── local-config.yaml
├── prometheus
│   └── prometheus.yml
├── promtail
│   └── config.yml
├── docker-compose.yml

Deploying the Stack

With all the configuration files in place, navigate to the /volume1/docker/prometheus directory and start the stack using Docker Compose:

cd /volume1/docker/prometheus
docker-compose up -d

Configuring Docker to Route Logs to Loki

  1. Install the Docker Loki plugin:

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

  2. Configure Docker daemon to use Loki:

    Edit or create the Docker daemon configuration file (dockerd.json):

    Synology:

    sudo nano /var/packages/Docker/etc/dockerd.json

    Add the following content:

    {
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push"
  }
}

  3. Restart Docker daemon:

    Synology:

    sudo synopkgctl stop Docker
sudo synopkgctl start Docker

Configuring Grafana to Query Logs

  1. Open Grafana:

    Navigate to http://<your-synology-ip>:3000.

  2. Log in:

    Use the default credentials (admin/admin) and change the password upon first login.

  3. Add Loki as a Data Source:

    • Go to Configuration > Data Sources > Add data source.
    • Choose Loki.
    • Set the URL to http://loki:3100.
    • Click Save & Test.

Conclusion

You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!

Adding Loki and Promtail to the Docker Compose Setup

Here’s an updated docker-compose.yml file that includes Loki and Promtail:

---
version: '3.8'

networks:
  loki:
  monitoring:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    ports:
      - 9093:9093
    restart: unless-stopped
    volumes:
      - ./alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/config.yml'
    networks:
      - monitoring

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - grafana-storage:/var/lib/grafana
    networks:
      - monitoring
      - loki

  node_exporter:
    image: prom/node-exporter
    container_name: node_exporter
    ports:
      - 9100:9100
    restart: unless-stopped
    networks:
      - monitoring

  loki:
    image: grafana/loki:2.6.0
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki:/etc/loki
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.0
    container_name: promtail
    volumes:
      - /var/log:/var/log
      - ./promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

volumes:
  prom_data:
  grafana-storage:

Loki Configuration File

Create a local-config.yaml file in the loki directory with the following content:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Promtail Configuration File

Create a config.yml file in the promtail directory with the following content:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Configuring Docker to Route Logs to Loki

  1. Install the Docker Loki plugin:

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

  2. Configure Docker daemon to use Loki:

    Edit or create the Docker daemon configuration file (dockerd.json):

    Linux default location:

    sudo nano /etc/docker/daemon.json

    Synology:

    sudo nano /var/packages/Docker/etc/dockerd.json

    Add the following content:

    {
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push"
  }
}

  3. Restart Docker daemon:

    Linux:

    sudo systemctl restart docker

    Synology:

    sudo synopkgctl stop Docker
sudo synopkgctl start Docker

Configuring Grafana to Query Logs

  1. Open Grafana:

    Navigate to http://<your-synology-ip>:3000.

  2. Log in:

    Use the default credentials (admin/admin) and change the password upon first login.

  3. Add Loki as a Data Source:

    • Go to Configuration > Data Sources > Add data source.
    • Choose Loki.
    • Set the URL to http://loki:3100.
    • Click Save & Test.

Deploying the Stack

With all the configuration files in place, navigate to the /volume1/docker/prometheus directory and start the stack using Docker Compose:

cd /volume1/docker/prometheus
docker-compose up -d

Conclusion

You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!