{"id":536,"date":"2014-10-23T19:53:46","date_gmt":"2014-10-23T17:53:46","guid":{"rendered":"http:\/\/blog.johannes-beck.name\/?p=536"},"modified":"2014-11-30T12:21:33","modified_gmt":"2014-11-30T11:21:33","slug":"shipping-tomcat-logs-to-logstash","status":"publish","type":"post","link":"https:\/\/blog.johannes-beck.name\/?p=536","title":{"rendered":"Shipping tomcat logs to logstash"},"content":{"rendered":"

In my previous post<\/a> I explained how to setup Logstash<\/a>. This time we will expand it to ship the log files from Tomcat<\/a>.<\/p>\n

I assume you are using Fedora \/ RHEL \/ Centos as OS, and have installed tomcat from the RPMS which are provided by the repository. The tutorial can easily adapted to different Linux distributions or tomcat installations, only file locations will change.<\/p>\n

As the first step we will change Tomcat’s logging behavior and disable the built-in log rotation. In \/etc\/tomcat\/server.xml you need to change the section at the very bottom of the configuration:<\/p>\n


\n<Valve className=\"org.apache.catalina.valves.AccessLogValve\"
\n  directory=\"logs\" prefix=\"access\"
\n  suffix=\".log\" rotatable=\"false\" resolveHosts=\"false\"
\n  pattern=\"%h %l %u %t "%r" %s %b\" \/>
\n<\/code><\/p>\n

and disable the log rotation in \/etc\/tomcat\/logging.properties<\/p>\n


\n1catalina.org.apache.juli.FileHandler.level = FINE
\n1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}\/logs
\n1catalina.org.apache.juli.FileHandler.prefix = catalina
\n1catalina.org.apache.juli.FileHandler.rotatable = false
\n2localhost.org.apache.juli.FileHandler.level = FINE
\n2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}\/logs
\n2localhost.org.apache.juli.FileHandler.prefix = localhost
\n2localhost.org.apache.juli.FileHandler.rotatable = false
\n3manager.org.apache.juli.FileHandler.level = FINE
\n3manager.org.apache.juli.FileHandler.directory = ${catalina.base}\/logs
\n3manager.org.apache.juli.FileHandler.prefix = manager
\n3manager.org.apache.juli.FileHandler.rotatable = false
\n4host-manager.org.apache.juli.FileHandler.level = FINE
\n4host-manager.org.apache.juli.FileHandler.directory = ${catalina.base}\/logs
\n4host-manager.org.apache.juli.FileHandler.prefix = host-manager
\n4host-manager.org.apache.juli.FileHandler.rotatable = false
\n<\/code><\/p>\n

Now the log files in \/var\/log\/tomcat\/ will be written into a files with constant names like access.log, manager.log etc.
\nTo prevent that the files grow infinitely, we’ll use logrotate.<\/p>\n

yum install logrotate<\/code> <\/p>\n

And add a configuration for tomcat at \/etc\/logrotate.d\/tomcat<\/p>\n


\n\/var\/log\/tomcat\/catalina.out {
\n  copytruncate
\n  daily
\n  rotate 7
\n  compress
\n  missingok
\n  create 0644 tomcat tomcat
\n}
\n\/var\/log\/tomcat\/*.log {
\n  copytruncate
\n  daily
\n  rotate 7
\n  compress
\n  missingok
\n  create 0644 tomcat tomcat
\n}
\n<\/code><\/p>\n

You may want to use different users and retention periods.<\/p>\n

Now we will configure the logstash-forwarder to ship the current file; in \/opt\/logstash-forwarder\/ setup a configuration file logstash-forwarder.conf<\/p>\n


\n{
\n \"network\": {
\n  \"servers\": [ \"my-logstash-server.example.org:9998\" ],
\n  \"ssl certificate\": \"\/etc\/pki\/tls\/certs\/client01.crt\",
\n  \"ssl key\": \"\/etc\/pki\/tls\/private\/client01.key\",
\n  \"ssl ca\": \"\/etc\/pki\/CA\/cacert.pem\"
\n },
\n \"files\": [
\n  {
\n   \"paths\": [ \"\/var\/log\/tomcat\/access.log\" ],
\n   \"fields\": { \"type\": \"apache-access\" }
\n  }
\n ]
\n}
\n<\/code><\/p>\n

and you’ll need to setup the logstash-forwarder as a daemon as explained in the last post. Now it will transmit the log files (in this example only the access.log) to logstash. To be able to parse the file correctly, you’ll need to configure the server with a parser for the type apache-access (which is similar to Apache HTTPD access.log). The first post has a working setup for this. <\/p>\n

If you want to ship the tomcat’s server log, or your application logs you will need to configure different parsers depending on the log format.<\/p>\n","protected":false},"excerpt":{"rendered":"

In my previous post I explained how to setup Logstash. This time we will expand it to ship the log files from Tomcat. I assume you are using Fedora \/ RHEL \/ Centos as OS, and have installed tomcat from the RPMS which are provided by the repository. The tutorial can easily adapted to different […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,36],"tags":[51,55],"class_list":["post-536","post","type-post","status-publish","format-standard","hentry","category-java","category-linux","tag-logstash","tag-tomcat"],"_links":{"self":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts\/536","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=536"}],"version-history":[{"count":6,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts\/536\/revisions"}],"predecessor-version":[{"id":542,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts\/536\/revisions\/542"}],"wp:attachment":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=536"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=536"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=536"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}