medusa/the_information_nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
be89a7b3b4cd609d2425cca499d27d9062198fc6
the_information_nexus/tech_docs/prometheus.md
medusa be89a7b3b4 Update tech_docs/prometheus.md
2024-05-29 15:45:43 +00:00

15 KiB
Raw Blame History

Directory Structure

We'll organize the directories under /volume1/docker/prometheus as follows:

/volume1/docker/prometheus
├── alertmanager
│   └── config.yml
├── grafana
├── loki
│   └── local-config.yaml
├── prometheus
│   └── prometheus.yml
├── promtail
│   └── config.yml
├── docker-compose.yml

Docker Compose File (docker-compose.yml)

Update the docker-compose.yml with absolute paths:

version: '3.8'

networks:
  loki:
  monitoring:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/prometheus:/etc/prometheus
      - /volume1/docker/prometheus/prom_data:/prometheus
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    ports:
      - 9093:9093
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/config.yml'
    networks:
      - monitoring

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/grafana:/var/lib/grafana
    networks:
      - monitoring
      - loki

  node_exporter:
    image: prom/node-exporter
    container_name: node_exporter
    ports:
      - 9100:9100
    restart: unless-stopped
    networks:
      - monitoring

  loki:
    image: grafana/loki:2.6.0
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - /volume1/docker/prometheus/loki:/etc/loki
      - /volume1/docker/prometheus/loki_data:/loki
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.0
    container_name: promtail
    volumes:
      - /var/log:/var/log
      - /volume1/docker/prometheus/promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

volumes:
  prom_data:
  grafana-storage:
  loki_data:

Prometheus Configuration File (prometheus.yml)

Create the prometheus.yml file in the /volume1/docker/prometheus/prometheus directory:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

Alertmanager Configuration File (config.yml)

Create the config.yml file in the /volume1/docker/prometheus/alertmanager directory:

global:
  resolve_timeout: 5m

route:
  receiver: 'default'

receivers:
  - name: 'default'

Loki Configuration File (local-config.yaml)

Create the local-config.yaml file in the /volume1/docker/prometheus/loki directory:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Promtail Configuration File (config.yml)

Create the config.yml file in the /volume1/docker/prometheus/promtail directory:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Configuring Docker to Route Logs to Loki

  1. Install the Docker Loki plugin:

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

  2. Configure Docker daemon to use Loki:

    Edit or create the Docker daemon configuration file (dockerd.json):

    Synology:

    sudo vi var/packages/ContainerManager/etc/dockerd.json

    Add the following content:

    {
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push"
  }
}

  3. Restart Docker daemon:

    Synology:

    sudo synopkgctl stop ContainerManager
sudo synopkgctl start ContainerManager

Configuring Grafana to Query Logs

  1. Open Grafana:

    Navigate to http://192.168.1.67:3000.

  2. Log in:

    Use the default credentials (admin/admin) and change the password upon first login.

  3. Add Loki as a Data Source:

    • Go to Configuration > Data Sources > Add data source.
    • Choose Loki.
    • Set the URL to http://loki:3100.
    • Click Save & Test.

Deploying the Stack

With all the configuration files in place, navigate to the /volume1/docker/prometheus directory and start the stack using Docker Compose:

cd /volume1/docker/prometheus
docker-compose up -d

Conclusion

You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!

Yes, using --- at the beginning of a YAML document is a good practice as it explicitly denotes the start of a document. Here is the revised Docker Compose file with the inclusion of ---:

Docker Compose File (docker-compose.yml)

---
version: '3.8'

networks:
  loki:
  monitoring:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/prometheus:/etc/prometheus
      - /volume1/docker/prometheus/prom_data:/prometheus
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    ports:
      - 9093:9093
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/config.yml'
    networks:
      - monitoring

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - /volume1/docker/prometheus/grafana:/var/lib/grafana
    networks:
      - monitoring
      - loki

  node_exporter:
    image: prom/node-exporter
    container_name: node_exporter
    ports:
      - 9100:9100
    restart: unless-stopped
    networks:
      - monitoring

  loki:
    image: grafana/loki:2.6.0
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - /volume1/docker/prometheus/loki:/etc/loki
      - /volume1/docker/prometheus/loki_data:/loki
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.0
    container_name: promtail
    volumes:
      - /var/log:/var/log
      - /volume1/docker/prometheus/promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

volumes:
  prom_data:
  grafana-storage:
  loki_data:

Updated Directory Structure

Here is a reminder of the directory structure:

/volume1/docker/prometheus
├── alertmanager
│   └── config.yml
├── grafana
├── loki
│   └── local-config.yaml
├── prometheus
│   └── prometheus.yml
├── promtail
│   └── config.yml
├── docker-compose.yml

Deploying the Stack

With all the configuration files in place, navigate to the /volume1/docker/prometheus directory and start the stack using Docker Compose:

cd /volume1/docker/prometheus
docker-compose up -d

Configuring Docker to Route Logs to Loki

  1. Install the Docker Loki plugin:

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

  2. Configure Docker daemon to use Loki:

    Edit or create the Docker daemon configuration file (dockerd.json):

    Synology:

    sudo nano /var/packages/Docker/etc/dockerd.json

    Add the following content:

    {
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push"
  }
}

  3. Restart Docker daemon:

    Synology:

    sudo synopkgctl stop Docker
sudo synopkgctl start Docker

Configuring Grafana to Query Logs

  1. Open Grafana:

    Navigate to http://<your-synology-ip>:3000.

  2. Log in:

    Use the default credentials (admin/admin) and change the password upon first login.

  3. Add Loki as a Data Source:

    • Go to Configuration > Data Sources > Add data source.
    • Choose Loki.
    • Set the URL to http://loki:3100.
    • Click Save & Test.

Conclusion

You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!

Adding Loki and Promtail to the Docker Compose Setup

Here’s an updated docker-compose.yml file that includes Loki and Promtail:

---
version: '3.8'

networks:
  loki:
  monitoring:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    ports:
      - 9093:9093
    restart: unless-stopped
    volumes:
      - ./alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/config.yml'
    networks:
      - monitoring

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - '3000:3000'
    restart: unless-stopped
    volumes:
      - grafana-storage:/var/lib/grafana
    networks:
      - monitoring
      - loki

  node_exporter:
    image: prom/node-exporter
    container_name: node_exporter
    ports:
      - 9100:9100
    restart: unless-stopped
    networks:
      - monitoring

  loki:
    image: grafana/loki:2.6.0
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki:/etc/loki
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.0
    container_name: promtail
    volumes:
      - /var/log:/var/log
      - ./promtail:/etc/promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

volumes:
  prom_data:
  grafana-storage:

Loki Configuration File

Create a local-config.yaml file in the loki directory with the following content:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Promtail Configuration File

Create a config.yml file in the promtail directory with the following content:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Configuring Docker to Route Logs to Loki

  1. Install the Docker Loki plugin:

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

  2. Configure Docker daemon to use Loki:

    Edit or create the Docker daemon configuration file (dockerd.json):

    Linux default location:

    sudo nano /etc/docker/daemon.json

    Synology:

    sudo nano /var/packages/Docker/etc/dockerd.json

    Add the following content:

    {
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push"
  }
}

  3. Restart Docker daemon:

    Linux:

    sudo systemctl restart docker

    Synology:

    sudo synopkgctl stop Docker
sudo synopkgctl start Docker

Configuring Grafana to Query Logs

  1. Open Grafana:

    Navigate to http://<your-synology-ip>:3000.

  2. Log in:

    Use the default credentials (admin/admin) and change the password upon first login.

  3. Add Loki as a Data Source:

    • Go to Configuration > Data Sources > Add data source.
    • Choose Loki.
    • Set the URL to http://loki:3100.
    • Click Save & Test.

Deploying the Stack

With all the configuration files in place, navigate to the /volume1/docker/prometheus directory and start the stack using Docker Compose:

cd /volume1/docker/prometheus
docker-compose up -d

Conclusion

You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!