{"id":526,"date":"2014-10-10T22:06:16","date_gmt":"2014-10-10T20:06:16","guid":{"rendered":"http:\/\/blog.johannes-beck.name\/?p=526"},"modified":"2014-10-10T22:07:04","modified_gmt":"2014-10-10T20:07:04","slug":"logstash-forwarder-as-a-lightweight-log-shipper","status":"publish","type":"post","link":"https:\/\/blog.johannes-beck.name\/?p=526","title":{"rendered":"logstash-forwarder as a lightweight log shipper"},"content":{"rendered":"

This may be the first of a series of posts with my experiences around Elasticsearch<\/a> and Logstash<\/a>, but it is kind of a piece in the middle. For me it was a missing link between a lot of very helpful and detailled tutorials you’ll find elsewhere. <\/p>\n

Logstash is great for managing events and logs, but kind of weak when it comes to shipping logs to the central hub. The default tool provided by logstash is a Java based client which needs to be installed on all client machines, but this comes with the drawback of memory and cpu consumption for the Java VM. <\/p>\n

logstash-forwarder<\/a> is an alternative which is more lightweight, but you’ll need to do some setup on your own. My experiences are based on Fedora 20, you’ll need to adapt some pieces on other Linux distributions. <\/p>\n

First we’ll need to build logstash-forwarder and package it as an RPM. This needs to be done only once, the setup is needed for all clients.<\/p>\n


\nyum install golang ruby ruby-devel rubygems
\ngit clone https:\/\/github.com\/elasticsearch\/logstash-forwarder.git
\ncd logstash-forwarder
\ngo build
\ngem install fpm
\nmake rpm
\n<\/code><\/p>\n

This should give you a RPM file with the latest release. Copy it to the client, and install it. The example files assume the logstash is running on my-logstash-server.example.com and the client is client01.example.com and runs an Apache HTTP.<\/p>\n


\nyum install logstash-forwarder-0.3.1-1.i686.rpm
\n<\/code><\/p>\n

Next we need a configuration file in \/opt\/logstash-forwarder\/logstash-forwarder.conf:<\/p>\n


\n{
\n \"network\": {
\n  \"servers\": [ \"my-logstash-server.example.org:9998\" ],
\n   \"ssl certificate\": \"\/etc\/pki\/tls\/certs\/client01.crt\",
\n   \"ssl key\": \"\/etc\/pki\/tls\/private\/client01key\",
\n   \"ssl ca\": \"\/etc\/pki\/CA\/cacert.pem\"
\n },
\n \"files\": [
\n  {
\n   \"paths\": [ \"\/var\/log\/httpd\/access.log\" ],
\n   \"fields\": { \"type\": \"apache-access\" }
\n  }
\n ]
\n}
\n<\/code><\/p>\n

As you see you need to create a SSL certificate for the client. The init script which is provided by logstash-forwarder is useless for Fedora \/ Redhat \/ Centos based systems, so we will use supervisord<\/a> as a replacement.<\/p>\n


\nyum install supervisor
\nsystemctl enable supervisord
\n<\/code><\/p>\n

and place a control file at \/etc\/supervisord.d\/logstash-forwarder.ini<\/p>\n


\n[program:logstash-forwarder]
\ncommand=\/opt\/logstash-forwarder\/bin\/logstash-forwarder.sh -config=logstash-forwarder.conf
\nstdout_logfile=NONE
\nstderr_logfile=NONE
\nautorestart=true
\nautostart=true
\nuser=root
\ndirectory=\/opt\/logstash-forwarder
\n<\/code><\/p>\n

We can start this now with systemctl start supervisord, but we’ll need to configure the receiver at the server side too.
\nAssumption is that logstash and elasticsearch are already up and running at my-logstash-server.example.org.
\nPlace a new config file at \/etc\/logstash\/conf.d\/logstash-lumberjack.conf<\/p>\n


\ninput {
\n lumberjack {
\n  # The port to listen on
\n  port => 9998
\n  # The paths to your ssl cert and key
\n  ssl_certificate => \"\/etc\/logstash\/my-logstash-server.crt\"
\n  ssl_key => \"\/etc\/logstash\/my-logstash-server.key\"
\n  # Set this to whatever you want.
\n  type => \"somelogs\"
\n }
\n}
\nfilter {
\n if [type] == \"apache-access\" {
\n  grok {
\n   match => {\"message\" => \"%{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}\" }
\n   match => {\"message\" => \"%{COMMONAPACHELOG}\" }
\n  }
\n }
\n}
\n<\/code><\/p>\n

Again you’ll need a SSL certificate and the private key should only be readable by logstash.
\nThe filter lines I used here work well for Apache HTTP and Apache Tomcat (how to configure this will be covered in another post).<\/p>\n

Now restart both supervisor on the client and logstash on the server, and you’ll see your logs incoming to the Elasticsearch.<\/p>\n","protected":false},"excerpt":{"rendered":"

This may be the first of a series of posts with my experiences around Elasticsearch and Logstash, but it is kind of a piece in the middle. For me it was a missing link between a lot of very helpful and detailled tutorials you’ll find elsewhere. Logstash is great for managing events and logs, but […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[36,3],"tags":[54,53,51,52],"class_list":["post-526","post","type-post","status-publish","format-standard","hentry","category-linux","category-software-development","tag-elasticsearch","tag-httpd","tag-logstash","tag-supervisord"],"_links":{"self":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts\/526","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=526"}],"version-history":[{"count":8,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts\/526\/revisions"}],"predecessor-version":[{"id":534,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=\/wp\/v2\/posts\/526\/revisions\/534"}],"wp:attachment":[{"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=526"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=526"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.johannes-beck.name\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=526"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}