Add tech_docs/prometheus.md

2024-05-27 13:44:32 +00:00
parent a57c6f4b5a
commit 0982bb210a
1 changed files with 241 additions and 0 deletions
--- a/tech_docs/prometheus.md
+++ b/tech_docs/prometheus.md
@@ -0,0 +1,241 @@
+Sure, let's integrate Loki into the existing Prometheus and Grafana setup, and also configure Docker to route logs to Loki.
+
+### Adding Loki and Promtail to the Docker Compose Setup
+
+Here’s an updated `docker-compose.yml` file that includes Loki and Promtail:
+
+```yaml
+version: '3.8'
+
+networks:
+  loki:
+  monitoring:
+    driver: bridge
+
+services:
+  prometheus:
+    image: prom/prometheus
+    container_name: prometheus
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+    ports:
+      - 9090:9090
+    restart: unless-stopped
+    volumes:
+      - ./prometheus:/etc/prometheus
+      - prom_data:/prometheus
+    networks:
+      - monitoring
+
+  alertmanager:
+    image: prom/alertmanager
+    container_name: alertmanager
+    ports:
+      - 9093:9093
+    restart: unless-stopped
+    volumes:
+      - ./alertmanager:/etc/alertmanager
+    command:
+      - '--config.file=/etc/alertmanager/config.yml'
+    networks:
+      - monitoring
+
+  grafana:
+    image: grafana/grafana
+    container_name: grafana
+    ports:
+      - '3000:3000'
+    restart: unless-stopped
+    volumes:
+      - grafana-storage:/var/lib/grafana
+    networks:
+      - monitoring
+      - loki
+
+  node_exporter:
+    image: prom/node-exporter
+    container_name: node_exporter
+    ports:
+      - 9100:9100
+    restart: unless-stopped
+    networks:
+      - monitoring
+
+  loki:
+    image: grafana/loki:2.6.0
+    container_name: loki
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/local-config.yaml
+    volumes:
+      - ./loki:/etc/loki
+    networks:
+      - loki
+
+  promtail:
+    image: grafana/promtail:2.6.0
+    container_name: promtail
+    volumes:
+      - /var/log:/var/log
+      - ./promtail:/etc/promtail
+    command: -config.file=/etc/promtail/config.yml
+    networks:
+      - loki
+
+volumes:
+  prom_data:
+  grafana-storage:
+```
+
+### Loki Configuration File
+
+Create a `local-config.yaml` file in the `loki` directory with the following content:
+
+```yaml
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+
+ingester:
+  lifecycler:
+    ring:
+      kvstore:
+        store: inmemory
+      replication_factor: 1
+  chunk_idle_period: 5m
+  chunk_retain_period: 30s
+  max_transfer_retries: 0
+
+schema_config:
+  configs:
+    - from: 2020-10-24
+      store: boltdb-shipper
+      object_store: filesystem
+      schema: v11
+      index:
+        prefix: index_
+        period: 24h
+
+storage_config:
+  boltdb_shipper:
+    active_index_directory: /loki/index
+    cache_location: /loki/cache
+    cache_ttl: 24h
+    shared_store: filesystem
+  filesystem:
+    directory: /loki/chunks
+
+limits_config:
+  enforce_metric_name: false
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+
+chunk_store_config:
+  max_look_back_period: 0s
+
+table_manager:
+  retention_deletes_enabled: false
+  retention_period: 0s
+```
+
+### Promtail Configuration File
+
+Create a `config.yml` file in the `promtail` directory with the following content:
+
+```yaml
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://loki:3100/loki/api/v1/push
+
+scrape_configs:
+- job_name: system
+  static_configs:
+  - targets:
+      - localhost
+    labels:
+      job: varlogs
+      __path__: /var/log/*log
+```
+
+### Configuring Docker to Route Logs to Loki
+
+1. **Install the Docker Loki plugin:**
+
+   ```bash
+   docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
+   ```
+
+2. **Configure Docker daemon to use Loki:**
+
+   Edit or create the Docker daemon configuration file (`dockerd.json`):
+
+   **Linux default location:**
+   ```bash
+   sudo nano /etc/docker/daemon.json
+   ```
+
+   **Synology:**
+   ```bash
+   sudo nano /var/packages/Docker/etc/dockerd.json
+   ```
+
+   Add the following content:
+
+   ```json
+   {
+     "log-driver": "loki",
+     "log-opts": {
+       "loki-url": "http://localhost:3100/loki/api/v1/push"
+     }
+   }
+   ```
+
+3. **Restart Docker daemon:**
+
+   **Linux:**
+   ```bash
+   sudo systemctl restart docker
+   ```
+
+   **Synology:**
+   ```bash
+   sudo synopkgctl stop Docker
+   sudo synopkgctl start Docker
+   ```
+
+### Configuring Grafana to Query Logs
+
+1. **Open Grafana:**
+
+   Navigate to `http://<your-synology-ip>:3000`.
+
+2. **Log in:**
+
+   Use the default credentials (admin/admin) and change the password upon first login.
+
+3. **Add Loki as a Data Source:**
+
+   - Go to Configuration > Data Sources > Add data source.
+   - Choose Loki.
+   - Set the URL to `http://loki:3100`.
+   - Click Save & Test.
+
+### Deploying the Stack
+
+With all the configuration files in place, navigate to the `/volume1/docker/prometheus` directory and start the stack using Docker Compose:
+
+```bash
+cd /volume1/docker/prometheus
+docker-compose up -d
+```
+
+### Conclusion
+
+You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!