碼農的學習手札

FluentD 實作 Error Log 本篇將介紹使用 DotNet 專案 log4net 套件，紀錄的 log 針對 Error Level 的訊息透過FluentD 提取出來 在紀錄中 增加 trace ID 設入 MongoDB , 及加入 Slack 通知 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log FluentD 實作 Nginx Access Log 補充 log4net <?xml version="1.0" encoding="utf-8" ?> <configuration> <log4net> <appender name="All" type="log4net.Appender.RollingFileAppender"> <file value="/var/log/my.log" /> <appendToFile value="true" /> <rollingStyle value="Size" /> <datePattern value="yyyy-MM-dd" /> <maximumFileSize value="5MB" /> <maxSizeRollBackups value="10" /> <staticLogFileName value="true" /> <PreserveLogFileNameExtension value="true" /> <layout type="log4net.Layout.PatternLayout"> <conversionPattern value="[%date] [...

實作 Nginx Access Log 透過 FluentD 收集 /var/log/nginx/nginx_web.access.log 日誌 過濾不必要的紀錄 輸出到MongoDB & Elasticsearch 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log 補充 fluent.conf # workers parameter for specifying the number of workers <system> workers 1 </system> <worker 0> <source> @type tail path /var/log/nginx/nginx_web.access.log pos_file /var/log/td-agent/nginx_web.access.log.pos tag nginx.web.access format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<logtime>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")(?: "(?<custom_field1>[^\"]*)" "(?< custom_field2>[^\"]*)" "(?< custom_field3>[^\"]*)")?$/ time_format %d/%b/%Y:%H:%M:%S %z </source> # 濾掉不必要的存取紀錄 <fil...

安裝請參考 Grafana 設定 1. Datasource > add > Prometheus > url ${http://172.28.1.3:9090} > Save & Test 2. Users Invite > ${email} 3. Plugins Pie Chart 4. Dashboards Grafana 資源 Grafana Dashboard Grafana Plugins 套用別人建立好的 Dashboard 在 https://grafana.com/grafana/dashboards 過濾 Data Source 為 Prometheus 的套件 找到適合的套件記下 ID 在自己的 Grafana => Create => Import 5. import! Alerting 參考資料 add notification channel use slack as example ( Slack 新增 app 請參考 ) 取得 App Token 取得 Incoming Webhooks URL 填入 URL ${webhook URL} & token token 為 xoxb- 6. 建立磁碟空間警報 Add Panel > Add Query # Queries (node_filesystem_avail_bytes/node_filesystem_size_bytes) * 100 # Visualization # import ! user "Graph" # Alert just like this

Create New App 1. Create New App 填入 App Name 及 選擇 workspace 2. 維護 Base Information 維護 App Name & Short description & App icon & background color 3. 維護 Oauth & Permissions Add an Oauth Scope incoming-webhook  channels:read This scope lets an app "View basic information about public channels in the workspace" chat:write:bot This scope lets an app "Send messages as your slack app" 4. Install App into your workspace Code Example Send messages example DotNet Code Send Message 2 Slack private void send2stalck(string path,string query,string header,string body,string message) { Dictionary dictQuery = new Dictionary (); dictQuery.Add("channel", @" slack channel "); dictQuery.Add("token", @" slack token "); dictQuery.Add("text", message); JArray jaFields = new JArray(); var joEnv = new JObject() { { "title", "env" }, { "value", ...

Grafana 功能強大且美觀的Dashboard Prometheus 用來作為數據收集及儲存，透過設定至各 exporter 端"拉"資料回來 exporter Prometheus exporter 環境建置 使用 docker-compose 時，會依所在目錄建立 ${path}-default docker network 純粹為了賞心悅目 建立一個 monitor-net 集中堆放 Grafana , Prometheus , xxxx exporter 建立 docker network docker network create monitor-net Grafana + Prometheus docker-compose.yml version: '3' services: grafana: image: grafana/grafana ports: - 9000:9000 - 9010:3000 environment: - GF_SERVER_ROOT_URL= - GF_SECURITY_ADMIN_PASSWORD= volumes: # copy grafana.ini to /etc/grafana - /etc/grafana/grafana.ini:/etc/grafana/grafana.ini - /var/lib/grafana:/var/lib/grafana - /var/log/grafana:/var/log/grafana container_name: my-grafana restart: always networks: - myonitornet prometheus: image: prom/prometheus ports: - 9090:9090 volumes: # copy prometheus.yml to /etc/prometheus - /etc/prometheus/prometheus.yml:/etc/prome...

安裝完畢後 立馬登入Kibana 瞧瞧 1. 建立 Index Pattern ${domain:port}/app/kibana#/management/kibana/index_patterns?_g=() 2. Discover Data ${domain:port}/app/kibana#/discover 3. Create New Visualization ${domain:port}/app/kibana#/visualize 情境1 顯示最長被存取的文章 前 10 筆統計 [Data Table] 1. split table > field ${path} > advanced > include > ${/api/article} 2. split row > field $(article id) 情境2 哪段時間是 API 存取巔峰 [Vertical] 1. x-Axis > field ${timestamp} > interval (10m) 情境3 Http request fail rate [Pie] 1. split slices > field ${http status code} 情境4 特定關鍵字的統計 [Metric] Filter語法可參考 Kibana 上方工具列的 Filter 選項產生 1. split group > aggregation = Filters > filter1 {"match_phrase":{"path":"/tracking/url/path"}} 4. 將建立的 visualization 加入 Dashboard ${domain:port}/app/kibana#/dashboard

前面已經提了 FluentD(1/2) FluentD(2/2) Elasticsearch 最後一哩路就是視覺化 Kibana 出場 Dockerfile FROM kibana:7.2.0 Build Image Shell Script docker build -t mykibana . kibana.yml Sample # Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "0.0.0.0" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite ...

Elasticsearch Install Dockerfile FROM elastic/elasticsearch:7.2.0 Build Image Shell Script docker build -t myelasticsearch . Prepare yours ES config copy elasticsearch.yml jvm.options log4j2.properties to /es/conf/path Docker start Shell Script # 由於沒有安全保護，因此設計上希望 ES 不對外，使用 --network 串在已建立的 docker network 中 docker run -it -d --name myelasticsearch --restart=unless-stopped --network=my-dev_mydemonet -v /es/conf/path:/usr/share/elasticsearch/config -v /es/data/path:/usr/share/elasticsearch/data -v /es/plugin/path:/usr/share/elasticsearch/plugins -e "discovery.type=single-node" -p 9300:9300 -p 9200:9200 myelasticsearch 安裝中文分詞 參考來源 docker exec -it myelasticsearch sh elasticsearch-plugin install analysis-smartcn 補充 定時刪除Index Shell Script 由於架構中 ES 屬於 短期、快速 查詢使用，僅保留七日的紀錄，因此需準備一組 Script 每日清理過期 Index 可以使用任何排程工具執行以下 Script idx_date=$(date +%Y.%m.%d -d "$DATE - 7 day") echo "curl -XDELETE http://localhost:9200/fluent...

FluentD 實作 Nginx Access Log 補充 前一篇針對 FluentD 安裝 及 Nginx Access log format 設定提供範例 本篇補充 1. 將 access_log 存入 MySQL 2. 針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output 延伸閱讀 FluentD 參數說明 FluentD 實作 Nginx Access Log 將 access_log 存入 MySQL <worker 0> <source> ... 略 </source> <match nginx.web.access> @type copy ... 略 <store> @type sql host ${MySQL Host address} port ${MySQL Port} adapter mysql2 database ${MySQL Database} username ${MySQL User Name} password ${MySQL Password} <table> table ${MySQL table} column_mapping 'logtime:logtime,method:method,path:path,code:code,size:size,resptime:resptime,token:token,path_url:path_url,timestamp:created_at' </table> </store> </match> </worker> 針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output 情境: 以下 access log 範例，需要針對 Query Parameter 拆解並存入新欄位，以利分析． [27/Dec/2019:07:14:10 ...

FluentD 高效、統一的日誌收集器 延伸閱讀 FluentD 實作 Nginx Access Log FluentD 實作 Nginx Access Log 補充 FluentD 安裝 Dockerfile FROM fluent/fluentd:v1.8.1-1.0 # Use root account to use apk USER root # below RUN includes plugin as examples elasticsearch is not required # you may customize including plugins as you wish RUN apk add --no-cache --update --virtual .build-deps \ sudo build-base ruby-dev \ && apk add mariadb-dev \ && sudo gem install fluent-plugin-elasticsearch \ && sudo gem install fluent-plugin-mongo \ && sudo gem install fluent-plugin-sql \ && sudo gem install mysql2 -v 0.5.2 \ && sudo gem sources --clear-all \ && apk del .build-deps \ && rm -rf /home/fluent/.gem/ruby/2.5.0/cache/*.gem VOLUME ["/fluentd/etc","/fluentd/log","/var/log"] docker-compose.yml version: '3' services: fluentd: build: context: . dockerfile: ./Dockerfile image: my/flue...