Prometheus + Grafana + Exporters = Server Monitor

Grafana
功能強大且美觀的Dashboard
Prometheus
用來作為數據收集及儲存，透過設定至各 exporter 端"拉"資料回來
exporter
Prometheus exporter

環境建置

使用 docker-compose 時，會依所在目錄建立 ${path}-default docker network
純粹為了賞心悅目 建立一個 monitor-net 集中堆放 Grafana , Prometheus , xxxx exporter

建立 docker network

docker network create monitor-net

Grafana + Prometheus docker-compose.yml

version: '3' 
services: 
  grafana:
    image: grafana/grafana
    ports:
     - 9000:9000
     - 9010:3000
    environment:
     - GF_SERVER_ROOT_URL= 
     - GF_SECURITY_ADMIN_PASSWORD=
    volumes:  # copy grafana.ini to /etc/grafana
     - /etc/grafana/grafana.ini:/etc/grafana/grafana.ini
     - /var/lib/grafana:/var/lib/grafana
     - /var/log/grafana:/var/log/grafana
    container_name: my-grafana
    restart: always
    networks:
      - myonitornet
  prometheus:
    image: prom/prometheus
    ports:
     - 9090:9090
    volumes:   # copy prometheus.yml to /etc/prometheus
     - /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    command: --config.file=/etc/prometheus/prometheus.yml
    container_name: my-prometheus
    restart: always
    networks:
      - myonitornet
networks:
  myonitornet:
    external:
      name: monitor-net

Others exporter docker-compose.yml

version: '3' # 目前使用的版本，可以參考官網
services: 
  nodeexporter:
    image: prom/node-exporter
    ports: 
     - 9100:9100
    container_name: my-nodeexporter
    restart: always
    networks:
        - myonitornet
  cadivor:
    image: google/cadvisor
    ports:
      - 9111:8080
    container_name: my-cadivor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    restart: always
    networks:
        - myonitornet
  redisexporter:
    image: oliver006/redis_exporter
    ports:
     - 9121:9121
    container_name: my-redisexporter
    environment:
      - REDIS_ADDR=# ${redis domain}:${redis port}
      - REDIS_PASSWORD=# ${redis password}
    restart: always
    networks:
        - myonitornet
  mysqlexporter:
    image: prom/mysqld-exporter
    ports:
     - 9131:9104
    environment:
      # ${user name}@${password}@)${domain}:${port}}/${database}
      - DATA_SOURCE_NAME=
    container_name: my-mysqlexporter
    restart: always
    networks:
        - myonitornet
  esexporter:
    image: justwatch/elasticsearch_exporter:1.1.0
    ports:
      - 9141:9114
    container_name: my-esexporter
    command:
      - '--es.uri=# ${es address}:${es port}
    restart: always
    networks:
        - myonitornet
networks:
  myonitornet:
    external:
      name: monitor-net
  

環境設定

Prometheus Config

Examples

global:
    scrape_interval: 15s
    external_labels:
      monitor: 'my-monitor'
  scrape_configs:
    - job_name: 'prometheus'
      scrape_interval: 5s
      static_configs:
        - targets: ['${prometheus server}:9090']
    - job_name: 'node resources'
      scrape_interval: 10s
      tls_config:
        insecure_skip_verify: true
      static_configs:
        - targets: ['${server 1}:9100']
        - targets: ['${server 2}:9100']
    - job_name: 'cadvisor'
      scrape_interval: 10s
      static_configs:
        - targets: ['${server 1}:9111']
        - targets: ['${server 2}:9111']
    - job_name: 'redis resources'
      scrape_interval: 10s
      static_configs:
        - targets: ['${server 1}:9121']
        - targets: ['${server 2}:9121']
    - job_name: 'mysql resources'
      scrape_interval: 10s
      static_configs:
        - targets: ['${server 1}:9131']
    - job_name: 'elasticsearch resources'
      scrape_interval: 10s
      static_configs:
        - targets: ['${server 1}:9141']
    

Grafana Config