logstash logoToday we are happy to announce version 1.0 of our Icinga Output Plugin for Logstash! It allows you to process check results, send notifications and manage downtimes by calling the Icinga API directly from Logstash. Furthermore, the Icinga output plugin for Logstash can be used in a high available manner, making sure you don’t lose any data.

Logstash

Logstash is a data processing pipeline. Logs and events are either actively collected or received from third party resources like Syslog or the Elastic Beats. Filters transform the data, usually by splitting the logs into separate fields and by adding additional information. The processed events are often stored for long term in Elasticsearch where they are used for analysis and visualisation.

Even though the Elasticsearch output is the most used one, it’s by far not the only method how Logstash can transmit data. There is an
email output, slack, redis, pagerduty and many more. Combined with conditionals, you can send certain data to certain outputs.

Icinga Output Plugin

The Icinga Output Plugin aims to build a bridge between your logging management and your monitoring. It can run various actions on your Icinga server by calling the Icinga API. Therefore, you don’t have to run Logstash on the same server as Icinga. Since the Icinga API requires SSL encryption and authentication, it’s save to send the data through the network.

logstash icinga combination

Post check results

The state of a host or a service is usually determined by running checks. However, Icinga also accepts check results from third party resources via the API. Lets say you receive logs of a certain criticality. With this plugin you can set the state of an Icinga host or service as desired. This works also the other way around, where you would set the state to OK.

Send custom notifications

Icinga is often used for notification management, where defined contacts get notified when hosts or services have problems. These notifications can be triggered manually by sending a custom notification. This feature is often used to just inform users about something, without warning them about an outage or critical services.

Add and remove downtimes

A service or host in a downtime won’t notify anyone even if reaches a problem state. With this plugin, downtimes can be created and removed dynamically based on events happening in your infrastructure. When your server starts the backup, you can create the downtime for it and remove it again when its done. This leaves you with a downtime that is exactly as long as it needs to be.

Add and remove comments

Comments are a nice way of leaving notes for hosts or services. Users can see them in the web interface.

High Availability

The plugin can be configured with multiple Icinga API endpoints. When an action is send and the API is not available, the Icinga Output Plugin will send the same action to the next host in the list. This goes on continuously until one of the configured API endpoints accepts the request. Each host is tried only once per request to prevent loops.

Installation

Installing Logstash plugins is pretty easy. All plugins are packed as Ruby gems and can be installed from rubygems.org. Logstash provides a CLI command to manage plugins. The documentation is currently available on GitHub.

user@localhost ~ $ /usr/share/logstash/bin/logstash-plugin install logstash-output-icinga

Examples

Process check results based on syslog severity
    filter {
      if [syslog_severity] == "error" {
        mutate {
          replace => { "exit_status" => "2" }
        }
      }
    }
    output {
      icinga {
        host           => "demo.icinga.com"
        user           => "icinga"
        password       => "supersecret"
        action         => "process-check-result"
        action_config  => {
          exit_status   => "%{exit_status}"
          plugin_output => "%{message}"
        }
        icinga_host    => "%{hostname}"
        icinga_service => "dummy"
      }
    }
Set a downtime for two hours, starting from now
    filter {
      ruby { code => "event.set('start_time', Time.now.to_i)" }
      ruby { code => "event.set('end_time', Time.now.to_i + 7200)" }
    }
    output {
      icinga {
        host           => "demo"
        user           => "icinga"
        password       => "supersecret"
        action         => "schedule-downtime"
        action_config  => {
          author     => "logstash"
          comment    => "Downtime set by Logstash Output"
          start_time => "%{start_time}"
          end_time   => "%{end_time}"
        }
        icinga_host    => "%{hostname}"
        icinga_service => "dummy"
      }
    }