跳到主要內容

Kibana(1/2)


前面已經提了
FluentD(1/2)
FluentD(2/2)
Elasticsearch
最後一哩路就是視覺化 Kibana 出場

Dockerfile



FROM kibana:7.2.0


Build Image Shell Script


docker build -t mykibana .


kibana.yml Sample


# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: [${ES HOST}:${PORT}]

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.ssl.verification_mode in Elasticsearch is set to either certificate or full.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

Docker run Shell Script


# import! 先將 自己的kibana.yml copy to /opt/kibana/kibana.yml 
docker run -it -d --name mykibana -p 5601:5601 -v /opt/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml  mykibana


補充

與 ES 一樣的問題,不想付錢 就只能自己想辦法提高安全性
Cloud Service VM instance 都可透過網路設定 限制 IP 存取 or 利用Nginx 做 Basic Authentication
參考來源



# 先使用 htpasswd 建立帳號密碼檔 
$>htpasswd -c /password/store/path/.htpasswd ${username}

# 如果沒有安裝 apache 工具 ,也可透過線上工具 
Htpasswd Generator
將產生的資料存入 .htpasswd



server {
    
     ... 略 

    location / {
        auth_basic "Administrator’s Area";
        auth_basic_user_file /password/store/path/.htpasswd;
        
         ... 略 

    }

}

留言

這個網誌中的熱門文章

在 GCP 新增 VM 執行個體的儲存空間

GCP 文件連結 建立 GCP VM instance 時,預設磁碟空間為10GB,可在建立時自行調整, 也可另外新增永久磁碟供VM 掛載使用 在建立永久磁區前,請先參考  儲存空間儲存空間的類型及效能 地區與區域的區別為: 地區 是 區域 的集合 並且參考其  磁碟定價 來做為你要新增儲存空間的依據 以下以新增區域永久磁碟做說明 1. 增加新磁碟 前往 VM 執行個體   頁面 VM 執行個體詳細資料 點擊 編輯 在磁碟標籤標籤下 點擊 增加新磁碟 指定名稱、類型、來源 等屬性後 點擊儲存 2. 格式化和掛接新磁碟 //在終端機中,使用 lsblk 指令列出連接到執行個體的磁碟,並搜尋您要格式化和掛接的磁碟 sudo lsblk //格式化磁碟 以下用 sdb 取代 [DEVICE_ID] sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/[DEVICE_ID] //建立做為新磁碟掛接點的目錄 以下用 extra 取代 [MNT_DIR] sudo mkdir -p /mnt/disks/[MNT_DIR] //使用掛接工具,將磁碟掛接到已啟用 discard 選項的執行個體 sudo mount -o discard,defaults /dev/[DEVICE_ID] /mnt/disks/[MNT_DIR] //設定裝置上的讀取與寫入權限。在此範例中,授予所有使用者對裝置的寫入存取權 sudo chmod a+w /mnt/disks/[MNT_DIR] //可用單一指令完成此步驟。舉例來說,下列指令會在 /etc/fstab 中建立項目,以永久磁碟的 UUID 將 /dev/sdb 永久磁碟掛接到 /mnt/disks/[MNT_DIR] echo UUID=`sudo blkid -s UUID -o value /dev/[DEVICE_ID]` /mnt/disks/[MNT_DIR] ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab 補充 將 Docker 遷移至新磁碟 參考來源 //編

DotNet Core 專案部署腳本

DotNet core SDK 首先在 Server 上準備編譯環境 Dockerfile #2.2 3.0 3.1 FROM mcr.microsoft.com/dotnet/core/sdk:3.1 RUN mkdir /web WORKDIR /web build docker image shell script docker build -t dotnetcoresdk:3.1 . start docker container shell script docker run -it -d \ --name dotnet-core-sdk-3.1 \ -v /opt/web:/web \ dotnetcoresdk:3.1 Jenkins Execute shell script on remote hosting using ssh #切換至專案目錄 cd /opt/web/project/path #取得最新版本 git pull #切換至專案目錄 && 刷新 Dotnet Library docker exec -i dotnet-core-sdk-3.1 bash -c "cd project/path && dotnet restore" #切換至專案目錄 && 刪除上一次編譯的檔案 && 編譯 docker exec -i dotnet-core-sdk-3.1 bash -c "cd project/path && rm -rf bin/Release && dotnet publish -c Release" #docker-compose.yml 參 DotNet core Runtime Section #!--rmi all 將原本執行的容器關閉並移除Image docker-compose down --rmi all #將新版程式包入 Image 並開始容器 docker-compose up -d DotNet core Runtime 專案中包含 Dockerfile & docker-compose.yml d

FluentD 實作 Nginx Access Log 補充

FluentD 實作 Nginx Access Log 補充 前一篇針對 FluentD 安裝 及 Nginx Access log format 設定提供範例 本篇補充 1. 將 access_log 存入 MySQL 2. 針對Input 加工,ex 解析 Path 拆成不同欄位,在傳入 Output 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log 將 access_log 存入 MySQL <worker 0> <source> ... 略 </source> <match nginx.web.access> @type copy ... 略 <store> @type sql host ${MySQL Host address} port ${MySQL Port} adapter mysql2 database ${MySQL Database} username ${MySQL User Name} password ${MySQL Password} <table> table ${MySQL table} column_mapping 'logtime:logtime,method:method,path:path,code:code,size:size,resptime:resptime,token:token,path_url:path_url,timestamp:created_at' </table> </store> </match> </worker> 針對Input 加工,ex 解析 Path 拆成不同欄位,在傳入 Output 情境: 以下 access log 範例,需要針對 Query Parameter 拆解並存入新欄位,以利分析. [27/Dec/2019:07:14:10