碼農的學習手札

實作 Nginx Access Log 透過 FluentD 收集 /var/log/nginx/nginx_web.access.log 日誌 過濾不必要的紀錄 輸出到MongoDB & Elasticsearch 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log 補充 fluent.conf # workers parameter for specifying the number of workers <system> workers 1 </system> <worker 0> <source> @type tail path /var/log/nginx/nginx_web.access.log pos_file /var/log/td-agent/nginx_web.access.log.pos tag nginx.web.access format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<logtime>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")(?: "(?<custom_field1>[^\"]*)" "(?< custom_field2>[^\"]*)" "(?< custom_field3>[^\"]*)")?$/ time_format %d/%b/%Y:%H:%M:%S %z </source> # 濾掉不必要的存取紀錄 <fil...

FluentD 實作 Nginx Access Log 補充 前一篇針對 FluentD 安裝 及 Nginx Access log format 設定提供範例 本篇補充 1. 將 access_log 存入 MySQL 2. 針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log 將 access_log 存入 MySQL <worker 0> <source> ... 略 </source> <match nginx.web.access> @type copy ... 略 <store> @type sql host ${MySQL Host address} port ${MySQL Port} adapter mysql2 database ${MySQL Database} username ${MySQL User Name} password ${MySQL Password} <table> table ${MySQL table} column_mapping 'logtime:logtime,method:method,path:path,code:code,size:size,resptime:resptime,token:token,path_url:path_url,timestamp:created_at' </table> </store> </match> </worker> 針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output 情境: 以下 access log 範例，需要針對 Query Parameter 拆解並存入新欄位，以利分析． [27/Dec/2019:07:14:10 ...

FluentD 高效、統一的日誌收集器 延伸閱讀 FluentD 實作 Nginx Access Log FluentD 實作 Nginx Access Log 補充 FluentD 安裝 Dockerfile FROM fluent/fluentd:v1.8.1-1.0 # Use root account to use apk USER root # below RUN includes plugin as examples elasticsearch is not required # you may customize including plugins as you wish RUN apk add --no-cache --update --virtual .build-deps \ sudo build-base ruby-dev \ && apk add mariadb-dev \ && sudo gem install fluent-plugin-elasticsearch \ && sudo gem install fluent-plugin-mongo \ && sudo gem install fluent-plugin-sql \ && sudo gem install mysql2 -v 0.5.2 \ && sudo gem sources --clear-all \ && apk del .build-deps \ && rm -rf /home/fluent/.gem/ruby/2.5.0/cache/*.gem VOLUME ["/fluentd/etc","/fluentd/log","/var/log"] docker-compose.yml version: '3' services: fluentd: build: context: . dockerfile: ./Dockerfile image: my/flue...