碼農的學習手札

FluentD 實作 Error Log 本篇將介紹使用 DotNet 專案 log4net 套件，紀錄的 log 針對 Error Level 的訊息透過FluentD 提取出來 在紀錄中 增加 trace ID 設入 MongoDB , 及加入 Slack 通知 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log FluentD 實作 Nginx Access Log 補充 log4net <?xml version="1.0" encoding="utf-8" ?> <configuration> <log4net> <appender name="All" type="log4net.Appender.RollingFileAppender"> <file value="/var/log/my.log" /> <appendToFile value="true" /> <rollingStyle value="Size" /> <datePattern value="yyyy-MM-dd" /> <maximumFileSize value="5MB" /> <maxSizeRollBackups value="10" /> <staticLogFileName value="true" /> <PreserveLogFileNameExtension value="true" /> <layout type="log4net.Layout.PatternLayout"> <conversionPattern value="[%date] [...

實作 Nginx Access Log 透過 FluentD 收集 /var/log/nginx/nginx_web.access.log 日誌 過濾不必要的紀錄 輸出到MongoDB & Elasticsearch 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log 補充 fluent.conf # workers parameter for specifying the number of workers <system> workers 1 </system> <worker 0> <source> @type tail path /var/log/nginx/nginx_web.access.log pos_file /var/log/td-agent/nginx_web.access.log.pos tag nginx.web.access format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<logtime>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")(?: "(?<custom_field1>[^\"]*)" "(?< custom_field2>[^\"]*)" "(?< custom_field3>[^\"]*)")?$/ time_format %d/%b/%Y:%H:%M:%S %z </source> # 濾掉不必要的存取紀錄 <fil...

安裝完畢後 立馬登入Kibana 瞧瞧 1. 建立 Index Pattern ${domain:port}/app/kibana#/management/kibana/index_patterns?_g=() 2. Discover Data ${domain:port}/app/kibana#/discover 3. Create New Visualization ${domain:port}/app/kibana#/visualize 情境1 顯示最長被存取的文章 前 10 筆統計 [Data Table] 1. split table > field ${path} > advanced > include > ${/api/article} 2. split row > field $(article id) 情境2 哪段時間是 API 存取巔峰 [Vertical] 1. x-Axis > field ${timestamp} > interval (10m) 情境3 Http request fail rate [Pie] 1. split slices > field ${http status code} 情境4 特定關鍵字的統計 [Metric] Filter語法可參考 Kibana 上方工具列的 Filter 選項產生 1. split group > aggregation = Filters > filter1 {"match_phrase":{"path":"/tracking/url/path"}} 4. 將建立的 visualization 加入 Dashboard ${domain:port}/app/kibana#/dashboard

前面已經提了 FluentD(1/2) FluentD(2/2) Elasticsearch 最後一哩路就是視覺化 Kibana 出場 Dockerfile FROM kibana:7.2.0 Build Image Shell Script docker build -t mykibana . kibana.yml Sample # Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "0.0.0.0" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite ...

Elasticsearch Install Dockerfile FROM elastic/elasticsearch:7.2.0 Build Image Shell Script docker build -t myelasticsearch . Prepare yours ES config copy elasticsearch.yml jvm.options log4j2.properties to /es/conf/path Docker start Shell Script # 由於沒有安全保護，因此設計上希望 ES 不對外，使用 --network 串在已建立的 docker network 中 docker run -it -d --name myelasticsearch --restart=unless-stopped --network=my-dev_mydemonet -v /es/conf/path:/usr/share/elasticsearch/config -v /es/data/path:/usr/share/elasticsearch/data -v /es/plugin/path:/usr/share/elasticsearch/plugins -e "discovery.type=single-node" -p 9300:9300 -p 9200:9200 myelasticsearch 安裝中文分詞 參考來源 docker exec -it myelasticsearch sh elasticsearch-plugin install analysis-smartcn 補充 定時刪除Index Shell Script 由於架構中 ES 屬於 短期、快速 查詢使用，僅保留七日的紀錄，因此需準備一組 Script 每日清理過期 Index 可以使用任何排程工具執行以下 Script idx_date=$(date +%Y.%m.%d -d "$DATE - 7 day") echo "curl -XDELETE http://localhost:9200/fluent...

FluentD 實作 Nginx Access Log 補充 前一篇針對 FluentD 安裝 及 Nginx Access log format 設定提供範例 本篇補充 1. 將 access_log 存入 MySQL 2. 針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log 將 access_log 存入 MySQL <worker 0> <source> ... 略 </source> <match nginx.web.access> @type copy ... 略 <store> @type sql host ${MySQL Host address} port ${MySQL Port} adapter mysql2 database ${MySQL Database} username ${MySQL User Name} password ${MySQL Password} <table> table ${MySQL table} column_mapping 'logtime:logtime,method:method,path:path,code:code,size:size,resptime:resptime,token:token,path_url:path_url,timestamp:created_at' </table> </store> </match> </worker> 針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output 情境: 以下 access log 範例，需要針對 Query Parameter 拆解並存入新欄位，以利分析． [27/Dec/2019:07:14:10 ...

FluentD 高效、統一的日誌收集器 延伸閱讀 FluentD 實作 Nginx Access Log FluentD 實作 Nginx Access Log 補充 FluentD 安裝 Dockerfile FROM fluent/fluentd:v1.8.1-1.0 # Use root account to use apk USER root # below RUN includes plugin as examples elasticsearch is not required # you may customize including plugins as you wish RUN apk add --no-cache --update --virtual .build-deps \ sudo build-base ruby-dev \ && apk add mariadb-dev \ && sudo gem install fluent-plugin-elasticsearch \ && sudo gem install fluent-plugin-mongo \ && sudo gem install fluent-plugin-sql \ && sudo gem install mysql2 -v 0.5.2 \ && sudo gem sources --clear-all \ && apk del .build-deps \ && rm -rf /home/fluent/.gem/ruby/2.5.0/cache/*.gem VOLUME ["/fluentd/etc","/fluentd/log","/var/log"] docker-compose.yml version: '3' services: fluentd: build: context: . dockerfile: ./Dockerfile image: my/flue...