Update tech_docs/prometheus.md
This commit is contained in:
@@ -1,10 +1,477 @@
|
||||
Sure, let's integrate Loki into the existing Prometheus and Grafana setup, and also configure Docker to route logs to Loki.
|
||||
### Directory Structure
|
||||
|
||||
We'll organize the directories under `/volume1/docker/prometheus` as follows:
|
||||
|
||||
```plaintext
|
||||
/volume1/docker/prometheus
|
||||
├── alertmanager
|
||||
│ └── config.yml
|
||||
├── grafana
|
||||
├── loki
|
||||
│ └── local-config.yaml
|
||||
├── prometheus
|
||||
│ └── prometheus.yml
|
||||
├── promtail
|
||||
│ └── config.yml
|
||||
├── docker-compose.yml
|
||||
```
|
||||
|
||||
### Docker Compose File (docker-compose.yml)
|
||||
|
||||
Update the `docker-compose.yml` with absolute paths:
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
networks:
|
||||
loki:
|
||||
monitoring:
|
||||
driver: bridge
|
||||
|
||||
services:
|
||||
prometheus:
|
||||
image: prom/prometheus
|
||||
container_name: prometheus
|
||||
command:
|
||||
- '--config.file=/etc/prometheus/prometheus.yml'
|
||||
ports:
|
||||
- 9090:9090
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/prometheus:/etc/prometheus
|
||||
- /volume1/docker/prometheus/prom_data:/prometheus
|
||||
networks:
|
||||
- monitoring
|
||||
|
||||
alertmanager:
|
||||
image: prom/alertmanager
|
||||
container_name: alertmanager
|
||||
ports:
|
||||
- 9093:9093
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/alertmanager:/etc/alertmanager
|
||||
command:
|
||||
- '--config.file=/etc/alertmanager/config.yml'
|
||||
networks:
|
||||
- monitoring
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana
|
||||
container_name: grafana
|
||||
ports:
|
||||
- '3000:3000'
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/grafana:/var/lib/grafana
|
||||
networks:
|
||||
- monitoring
|
||||
- loki
|
||||
|
||||
node_exporter:
|
||||
image: prom/node-exporter
|
||||
container_name: node_exporter
|
||||
ports:
|
||||
- 9100:9100
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- monitoring
|
||||
|
||||
loki:
|
||||
image: grafana/loki:2.6.0
|
||||
container_name: loki
|
||||
ports:
|
||||
- "3100:3100"
|
||||
command: -config.file=/etc/loki/local-config.yaml
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/loki:/etc/loki
|
||||
- /volume1/docker/prometheus/loki_data:/loki
|
||||
networks:
|
||||
- loki
|
||||
|
||||
promtail:
|
||||
image: grafana/promtail:2.6.0
|
||||
container_name: promtail
|
||||
volumes:
|
||||
- /var/log:/var/log
|
||||
- /volume1/docker/prometheus/promtail:/etc/promtail
|
||||
command: -config.file=/etc/promtail/config.yml
|
||||
networks:
|
||||
- loki
|
||||
|
||||
volumes:
|
||||
prom_data:
|
||||
grafana-storage:
|
||||
loki_data:
|
||||
```
|
||||
|
||||
### Prometheus Configuration File (prometheus.yml)
|
||||
|
||||
Create the `prometheus.yml` file in the `/volume1/docker/prometheus/prometheus` directory:
|
||||
|
||||
```yaml
|
||||
global:
|
||||
scrape_interval: 15s
|
||||
scrape_timeout: 10s
|
||||
evaluation_interval: 15s
|
||||
|
||||
alerting:
|
||||
alertmanagers:
|
||||
- static_configs:
|
||||
- targets: ['alertmanager:9093']
|
||||
|
||||
scrape_configs:
|
||||
- job_name: 'prometheus'
|
||||
static_configs:
|
||||
- targets: ['localhost:9090']
|
||||
|
||||
- job_name: 'node_exporter'
|
||||
static_configs:
|
||||
- targets: ['node_exporter:9100']
|
||||
```
|
||||
|
||||
### Alertmanager Configuration File (config.yml)
|
||||
|
||||
Create the `config.yml` file in the `/volume1/docker/prometheus/alertmanager` directory:
|
||||
|
||||
```yaml
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
|
||||
route:
|
||||
receiver: 'default'
|
||||
|
||||
receivers:
|
||||
- name: 'default'
|
||||
```
|
||||
|
||||
### Loki Configuration File (local-config.yaml)
|
||||
|
||||
Create the `local-config.yaml` file in the `/volume1/docker/prometheus/loki` directory:
|
||||
|
||||
```yaml
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
|
||||
ingester:
|
||||
lifecycler:
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
replication_factor: 1
|
||||
chunk_idle_period: 5m
|
||||
chunk_retain_period: 30s
|
||||
max_transfer_retries: 0
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2020-10-24
|
||||
store: boltdb-shipper
|
||||
object_store: filesystem
|
||||
schema: v11
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
storage_config:
|
||||
boltdb_shipper:
|
||||
active_index_directory: /loki/index
|
||||
cache_location: /loki/cache
|
||||
cache_ttl: 24h
|
||||
shared_store: filesystem
|
||||
filesystem:
|
||||
directory: /loki/chunks
|
||||
|
||||
limits_config:
|
||||
enforce_metric_name: false
|
||||
reject_old_samples: true
|
||||
reject_old_samples_max_age: 168h
|
||||
|
||||
chunk_store_config:
|
||||
max_look_back_period: 0s
|
||||
|
||||
table_manager:
|
||||
retention_deletes_enabled: false
|
||||
retention_period: 0s
|
||||
```
|
||||
|
||||
### Promtail Configuration File (config.yml)
|
||||
|
||||
Create the `config.yml` file in the `/volume1/docker/prometheus/promtail` directory:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
|
||||
positions:
|
||||
filename: /tmp/positions.yaml
|
||||
|
||||
clients:
|
||||
- url: http://loki:3100/loki/api/v1/push
|
||||
|
||||
scrape_configs:
|
||||
- job_name: system
|
||||
static_configs:
|
||||
- targets:
|
||||
- localhost
|
||||
labels:
|
||||
job: varlogs
|
||||
__path__: /var/log/*log
|
||||
```
|
||||
|
||||
### Configuring Docker to Route Logs to Loki
|
||||
|
||||
1. **Install the Docker Loki plugin:**
|
||||
|
||||
```bash
|
||||
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
|
||||
```
|
||||
|
||||
2. **Configure Docker daemon to use Loki:**
|
||||
|
||||
Edit or create the Docker daemon configuration file (`dockerd.json`):
|
||||
|
||||
**Synology:**
|
||||
```bash
|
||||
sudo nano /var/packages/Docker/etc/dockerd.json
|
||||
```
|
||||
|
||||
Add the following content:
|
||||
|
||||
```json
|
||||
{
|
||||
"log-driver": "loki",
|
||||
"log-opts": {
|
||||
"loki-url": "http://localhost:3100/loki/api/v1/push"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. **Restart Docker daemon:**
|
||||
|
||||
**Synology:**
|
||||
```bash
|
||||
sudo synopkgctl stop Docker
|
||||
sudo synopkgctl start Docker
|
||||
```
|
||||
|
||||
### Configuring Grafana to Query Logs
|
||||
|
||||
1. **Open Grafana:**
|
||||
|
||||
Navigate to `http://<your-synology-ip>:3000`.
|
||||
|
||||
2. **Log in:**
|
||||
|
||||
Use the default credentials (admin/admin) and change the password upon first login.
|
||||
|
||||
3. **Add Loki as a Data Source:**
|
||||
|
||||
- Go to Configuration > Data Sources > Add data source.
|
||||
- Choose Loki.
|
||||
- Set the URL to `http://loki:3100`.
|
||||
- Click Save & Test.
|
||||
|
||||
### Deploying the Stack
|
||||
|
||||
With all the configuration files in place, navigate to the `/volume1/docker/prometheus` directory and start the stack using Docker Compose:
|
||||
|
||||
```bash
|
||||
cd /volume1/docker/prometheus
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Conclusion
|
||||
|
||||
You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!
|
||||
|
||||
---
|
||||
|
||||
Yes, using `---` at the beginning of a YAML document is a good practice as it explicitly denotes the start of a document. Here is the revised Docker Compose file with the inclusion of `---`:
|
||||
|
||||
### Docker Compose File (docker-compose.yml)
|
||||
|
||||
```yaml
|
||||
---
|
||||
version: '3.8'
|
||||
|
||||
networks:
|
||||
loki:
|
||||
monitoring:
|
||||
driver: bridge
|
||||
|
||||
services:
|
||||
prometheus:
|
||||
image: prom/prometheus
|
||||
container_name: prometheus
|
||||
command:
|
||||
- '--config.file=/etc/prometheus/prometheus.yml'
|
||||
ports:
|
||||
- 9090:9090
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/prometheus:/etc/prometheus
|
||||
- /volume1/docker/prometheus/prom_data:/prometheus
|
||||
networks:
|
||||
- monitoring
|
||||
|
||||
alertmanager:
|
||||
image: prom/alertmanager
|
||||
container_name: alertmanager
|
||||
ports:
|
||||
- 9093:9093
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/alertmanager:/etc/alertmanager
|
||||
command:
|
||||
- '--config.file=/etc/alertmanager/config.yml'
|
||||
networks:
|
||||
- monitoring
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana
|
||||
container_name: grafana
|
||||
ports:
|
||||
- '3000:3000'
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/grafana:/var/lib/grafana
|
||||
networks:
|
||||
- monitoring
|
||||
- loki
|
||||
|
||||
node_exporter:
|
||||
image: prom/node-exporter
|
||||
container_name: node_exporter
|
||||
ports:
|
||||
- 9100:9100
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- monitoring
|
||||
|
||||
loki:
|
||||
image: grafana/loki:2.6.0
|
||||
container_name: loki
|
||||
ports:
|
||||
- "3100:3100"
|
||||
command: -config.file=/etc/loki/local-config.yaml
|
||||
volumes:
|
||||
- /volume1/docker/prometheus/loki:/etc/loki
|
||||
- /volume1/docker/prometheus/loki_data:/loki
|
||||
networks:
|
||||
- loki
|
||||
|
||||
promtail:
|
||||
image: grafana/promtail:2.6.0
|
||||
container_name: promtail
|
||||
volumes:
|
||||
- /var/log:/var/log
|
||||
- /volume1/docker/prometheus/promtail:/etc/promtail
|
||||
command: -config.file=/etc/promtail/config.yml
|
||||
networks:
|
||||
- loki
|
||||
|
||||
volumes:
|
||||
prom_data:
|
||||
grafana-storage:
|
||||
loki_data:
|
||||
```
|
||||
|
||||
### Updated Directory Structure
|
||||
|
||||
Here is a reminder of the directory structure:
|
||||
|
||||
```plaintext
|
||||
/volume1/docker/prometheus
|
||||
├── alertmanager
|
||||
│ └── config.yml
|
||||
├── grafana
|
||||
├── loki
|
||||
│ └── local-config.yaml
|
||||
├── prometheus
|
||||
│ └── prometheus.yml
|
||||
├── promtail
|
||||
│ └── config.yml
|
||||
├── docker-compose.yml
|
||||
```
|
||||
|
||||
### Deploying the Stack
|
||||
|
||||
With all the configuration files in place, navigate to the `/volume1/docker/prometheus` directory and start the stack using Docker Compose:
|
||||
|
||||
```bash
|
||||
cd /volume1/docker/prometheus
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Configuring Docker to Route Logs to Loki
|
||||
|
||||
1. **Install the Docker Loki plugin:**
|
||||
|
||||
```bash
|
||||
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
|
||||
```
|
||||
|
||||
2. **Configure Docker daemon to use Loki:**
|
||||
|
||||
Edit or create the Docker daemon configuration file (`dockerd.json`):
|
||||
|
||||
**Synology:**
|
||||
```bash
|
||||
sudo nano /var/packages/Docker/etc/dockerd.json
|
||||
```
|
||||
|
||||
Add the following content:
|
||||
|
||||
```json
|
||||
{
|
||||
"log-driver": "loki",
|
||||
"log-opts": {
|
||||
"loki-url": "http://localhost:3100/loki/api/v1/push"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. **Restart Docker daemon:**
|
||||
|
||||
**Synology:**
|
||||
```bash
|
||||
sudo synopkgctl stop Docker
|
||||
sudo synopkgctl start Docker
|
||||
```
|
||||
|
||||
### Configuring Grafana to Query Logs
|
||||
|
||||
1. **Open Grafana:**
|
||||
|
||||
Navigate to `http://<your-synology-ip>:3000`.
|
||||
|
||||
2. **Log in:**
|
||||
|
||||
Use the default credentials (admin/admin) and change the password upon first login.
|
||||
|
||||
3. **Add Loki as a Data Source:**
|
||||
|
||||
- Go to Configuration > Data Sources > Add data source.
|
||||
- Choose Loki.
|
||||
- Set the URL to `http://loki:3100`.
|
||||
- Click Save & Test.
|
||||
|
||||
### Conclusion
|
||||
|
||||
You now have a comprehensive observability stack including Prometheus, Grafana, Loki, and Promtail running on your Synology NAS. This setup provides monitoring, alerting, and log aggregation capabilities. If you need any more specific configurations or have any questions, feel free to ask!
|
||||
|
||||
---
|
||||
|
||||
### Adding Loki and Promtail to the Docker Compose Setup
|
||||
|
||||
Here’s an updated `docker-compose.yml` file that includes Loki and Promtail:
|
||||
|
||||
```yaml
|
||||
---
|
||||
version: '3.8'
|
||||
|
||||
networks:
|
||||
|
||||
Reference in New Issue
Block a user