logstash-forwarder as a lightweight log shipper

This may be the first of a series of posts with my experiences around Elasticsearch and Logstash, but it is kind of a piece in the middle. For me it was a missing link between a lot of very helpful and detailled tutorials you’ll find elsewhere.

Logstash is great for managing events and logs, but kind of weak when it comes to shipping logs to the central hub. The default tool provided by logstash is a Java based client which needs to be installed on all client machines, but this comes with the drawback of memory and cpu consumption for the Java VM.

logstash-forwarder is an alternative which is more lightweight, but you’ll need to do some setup on your own. My experiences are based on Fedora 20, you’ll need to adapt some pieces on other Linux distributions.

First we’ll need to build logstash-forwarder and package it as an RPM. This needs to be done only once, the setup is needed for all clients.

yum install golang ruby ruby-devel rubygems
git clone https://github.com/elasticsearch/logstash-forwarder.git
cd logstash-forwarder
go build
gem install fpm
make rpm

This should give you a RPM file with the latest release. Copy it to the client, and install it. The example files assume the logstash is running on my-logstash-server.example.com and the client is client01.example.com and runs an Apache HTTP.

yum install logstash-forwarder-0.3.1-1.i686.rpm

Next we need a configuration file in /opt/logstash-forwarder/logstash-forwarder.conf:

 "network": {
  "servers": [ "my-logstash-server.example.org:9998" ],
   "ssl certificate": "/etc/pki/tls/certs/client01.crt",
   "ssl key": "/etc/pki/tls/private/client01key",
   "ssl ca": "/etc/pki/CA/cacert.pem"
 "files": [
   "paths": [ "/var/log/httpd/access.log" ],
   "fields": { "type": "apache-access" }

As you see you need to create a SSL certificate for the client. The init script which is provided by logstash-forwarder is useless for Fedora / Redhat / Centos based systems, so we will use supervisord as a replacement.

yum install supervisor
systemctl enable supervisord

and place a control file at /etc/supervisord.d/logstash-forwarder.ini

command=/opt/logstash-forwarder/bin/logstash-forwarder.sh -config=logstash-forwarder.conf

We can start this now with systemctl start supervisord, but we’ll need to configure the receiver at the server side too.
Assumption is that logstash and elasticsearch are already up and running at my-logstash-server.example.org.
Place a new config file at /etc/logstash/conf.d/logstash-lumberjack.conf

input {
 lumberjack {
  # The port to listen on
  port => 9998
  # The paths to your ssl cert and key
  ssl_certificate => "/etc/logstash/my-logstash-server.crt"
  ssl_key => "/etc/logstash/my-logstash-server.key"
  # Set this to whatever you want.
  type => "somelogs"
filter {
 if [type] == "apache-access" {
  grok {
   match => {"message" => "%{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}" }
   match => {"message" => "%{COMMONAPACHELOG}" }

Again you’ll need a SSL certificate and the private key should only be readable by logstash.
The filter lines I used here work well for Apache HTTP and Apache Tomcat (how to configure this will be covered in another post).

Now restart both supervisor on the client and logstash on the server, and you’ll see your logs incoming to the Elasticsearch.

Tagged : , , ,

Leave a Reply