Fluentd+Elasticsearch+Kibana采集nginx日志

安装过程

安装Fluentd

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
systemctl start td-agent

安装插件

td-agent-gem install fluent-plugin-elasticsearch
td-agent-gem install fluent-plugin-typecast

安装Elasticsearch

1. 下载软件包 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.tar.gz

2. 创建组和用户

   1. groupadd elsearch
   2. useradd elsearch -g elsearch -p elsearch
   3. chown -R elsearch:elsearch /usr/local/src/elsearch

3. (可选)elasticsearch插件安装（github上面检查版本对应）

   • ./bin/plugin install mobz/elasticsearch-head
   • ./bin/plugin install lmenezes/elasticsearch-kopf/v2.1.1

4. 编辑/etc/sysctl.conf

   • 增加 vm.max_map_count=655360
   • 查看是否生效 sysctl -p

5. 编辑/etc/security/limits.conf，在文件尾添加以下内容

   • * soft nofile 65536
   • * hard /nofile 65536

6. 编辑elasticsearch的配置文件 config/elasticsearch.yml,修改配置一下项

   • network.host: 0.0.0.0

7. 切换用户elsearch 启动elasticsearch

   • su elsearch -c bin/elasticsearch
   • 如果要以守护进程方式运行加上参数-d

8. 验证elasticsearch启动是否成功

   • curl localhost:9200

安装Kibana

1. 下载解压到/opt/目录
   • wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-linux-x86_64.tar.gz
2. 修改配置文件 config/kibana.yml
   • server.port: 5601
   • server.host: “0.0.0.0”
   • elasticsearch.url: “http://localhost:9200"
3. 运行kibana
   • bin/kibana

配置fluentd

1. 编辑配置文件 /etc/td-agent/td-agent.conf

<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/nginx/access.log.pos
  tag nginx.access.log
  format /^(?<ip>\S+)\s-\s-\s\[(?<time>[^\]]*)\]\s"(?<url>[^\"]+)"\s(?<status>\d+)\s(?<size>\d+)\s"(?<ref>[^\"]+)"\s"(?<agent>[^\"]+)"\s"(?<cookie_unb>[^\"]+)"$/
  time_format %d/%b/%Y:%H:%M:%S %z
</source>

<match nginx.access.log>
  @type elasticsearch
  host 192.168.137.52
  port 9200
  logstash_format true
  flush_interval 10s
  #index_name fluentd
  #type_name fluentd
</match>

2. 创建pos_file
   • touch /var/log/nginx/access.log.pos
3. 编辑权限
   • chmod 755 /var/log/nginx
   • chmod a+rw /var/log/nginx/access.log.pos
4. 重启fluentd
   • systemctl restart fluentd
5. 检查fluentd日志
   • tail -f /var/log/td-agent/td-agent.log
6. 用kibana查看日志

参考资料：

• 正则表达式：http://fluentular.herokuapp.com/
• http://blog.csdn.net/chenhaifeng2016/article/details/78683812
• http://blog.csdn.net/qq_27252133/article/details/53520416