FluentD 實作 Nginx Access Log 補充

---

前一篇針對 FluentD 安裝 及 Nginx Access log format 設定提供範例

本篇補充
1. 將 access_log 存入 MySQL
2. 針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output

延伸閱讀

---

FluentD 參數說明
FluentD 實作 Nginx Access Log

將 access_log 存入 MySQL

<worker 0>
  <source>
  
   ... 略

  </source>
  <match nginx.web.access>
    @type copy
     
     ... 略
  
    <store>
      @type sql
      host  ${MySQL Host address}
      port  ${MySQL Port}
      adapter mysql2
      database  ${MySQL Database}
      username  ${MySQL User Name}
      password  ${MySQL Password}
      <table>
        table  ${MySQL table}
        column_mapping 'logtime:logtime,method:method,path:path,code:code,size:size,resptime:resptime,token:token,path_url:path_url,timestamp:created_at'
      </table>
    </store>
  </match>
</worker>

針對Input 加工，ex 解析 Path 拆成不同欄位，在傳入 Output

情境: 以下 access log 範例，需要針對 Query Parameter 拆解並存入新欄位，以利分析．
[27/Dec/2019:07:14:10 +0000] "GET /path/to/url?out_id=somevalue&uri=somevalue HTTP/1.1" 302 608 0.057 "" "Mozilla/5.0 (Linux; Android 8.1.0; Redmi Note 5 Build/OPM1.171019.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/79.0.3945.93 Mobile Safari/537.36 Line/9.22.2/IAB" "" "" "" "" "upstream: 127.0.0.1:32081"

<worker 0>
  <source>
  
   ... 略

  </source>
   
  <filter nginx.web.access>
    @type parser
    key_name path
    reserve_data true   # import! 將原本的 Input 一併傳到 Ouptput 
    <parse>
        @type regexp
        expression /^(?<path_url>([^\?]+))+(.out_id=(?<token>[^&#]*).uri=(?<target>[^#]*)|)$/
    </parse>
  </filter>

  <match nginx.web.access>
     
     ... 略
  
  </match>
</worker>