5.1 Tomcat日志收集思路
我们只需要安装好tomcat,然后将 tomcat 修改为 json 格式日志,在使用 filebeat 进行收集即可;
5.2 Tomcat日志收集架构图
5.3 Tomcat访问日志收集实践
5.3.1 安装Tomcat
mkdir -p /soft/ && cd /soft
wget =
tar xf apache-tomcat-9.0.26.tar.gz
ln -s /soft/apache-tomcat-9.0.26 /soft/tomcat
5.3.2 修改日志为JSON
修改tomcat server.xml文件,修改日志格式
<Host name="" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="json_elk_log" suffix=".txt"
pattern="
{"clientip":"%h","
ClientUser":"%l","
authenticated":"%u","
AccessTime":"%t","
method":"%r","
status":"%s","
SendBytes":"%b","
Query?string":"%q","
partner":"%{Referer}i","
AgentVersion":"%{User-Agent}i"}" />
</Host>
启动tomcat
/soft/tomcat/bin/startup.sh
检查访问日志是否为json格式
cat /soft/tomcat/logs/json_elk_log.2021-10-30.txt
{"clientip":"10.0.0.1"," ClientUser":"-"," authenticated":"-"," AccessTime":"[30/Oct/2021:11:08:45 +0800]","
method":"GET / HTTP/1.1"," status":"200"," SendBytes":"200"," Query?string":""," partner":"-","
AgentVersion":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"}
5.3.3 配置filebeat
cat filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths: /soft/tomcat/logs/json_elk_log*.txt
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
output.elasticsearch:
hosts: ["172.16.1.161:9200","172.16.1.162:9200","172.16.1.163:9200"]
index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
: "tomcat" #定义模板名称
setup.template.pattern: "tomcat-*" #定义模板的匹配索引名称
systemctl restart filebeat
5.3.4 配置kibana
kibana中创建tomcat-access索并展示
5.4 Tomcat错误日志收集实践
5.4.1 错误日志特点
1.报错信息比较多。
2.报错信息分很多行
5.4.2 错误日志收集思路
例1: Tomcat正常日志是以 “日期” 开头的。而报错日志中间的错误都不是以 “日期” 开头的。
所以我们可以匹配以 “日期” 开头的一直到下一个日期出现则为一个事件日志。
例2: Elasticsearch正常日志是以 [] 开头的。而报错日志中间的错误信息不是以 [] 开头,所以可以匹配以 [开头的行,一直到下一个 [开头的出现则为一个事件日志。
5.4.3 Tomcat错误日志收集实践
cat filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths: /soft/tomcat/logs/json_elk_log*.txt
json.keys_under_root: true # 默认为False; 就是将所有的日志记录到Message字段;true不存储至Message字段
json.overwrite_keys: true # 会覆盖掉Message字段的内容,然后使用自行定义的Json格式的Key作为字段,来存储对应的值
tags: ["access"]
- type: log
enabled: true
paths: /soft/tomcat/logs/catalina.out
tags: ["error"]
multiline.pattern: '^\d{2}' #匹配以两个数字开头的
multiline.negate: true
multiline.match: after
multiline.max_lines: 1000 # 最大的合并行数 默认合并的数量是500
output.elasticsearch:
hosts: ["172.16.1.161:9200","172.16.1.162:9200","172.16.1.163:9200"]
indices:
- index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}" #自定义索引名称
when.contains:
tags: "access"
- index: "tomcat-error-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "error"
setup.ilm.enabled: false
: "tomcat" #定义模板名称
setup.template.pattern: "tomcat-*" #定义模板的匹配索引名称
5.4.4 ES错误日志收集实践
cat filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths: /var/logs/elasticsearch/my-es.log
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
output.elasticsearch:
hosts: ["172.16.1.161:9200","172.16.1.162:9200","172.16.1.163:9200"]
index: "es-%{[agent.version]}-%{+yyyy.MM.dd}" #自定义索引名称
setup.ilm.enabled: false
: "es" #定义模板名称
setup.template.pattern: "es-*" #定义模板的匹配索引名称