Icinga Blog

Release announcements, development news and monitoring goodies

Advisory for latest security updates on RHEL 7

Community members told us today that Icinga 2 stopped working with the most recent RedHat Enterprise Linux 7 Kernel update 3.10.0-514.21.2. This update includes a security patch for the stack guard vulnerability.

Update 2017-06-20 19:55 Europe/Berlin: CentOS 7 is currently rolling the kernel update and affected too. Upstream bug report has been created.

Update 2017-06-21 11:30 Europe/Berlin: RedHat’s kernel team is investigating on this possible regression. We are in touch with them. Meanwhile apply the quickfix below.

Update 2017-06-21 19:20 Europe/Berlin: https://bugzilla.redhat.com/show_bug.cgi?id=1463241 is now public.

Update 2017-06-22 14:45 Europe/Berlin: We are investigating on the Icinga 2 side how to better handle rlimits. v2.7 is postponed until the Kernel issue is fully resolved. Other distributions might be still affected, there’s ongoing investigation on the oss-sec mailing lists.

Update 2017-06-23 10:45 Europe/Berlin: RedHat provided us with a fixed test package and our tests went fine. Please open a support case at RedHat to receive “accelerated fixes” or to get a test binary package. You also raise awareness by doing so, this should help getting an official release sooner.

 

Analysis

We’ve analysed the issue and reproduced the issue on RHEL7. Debian Jessie and Stretch also released a security update for CVE-2017-1000364 but Icinga 2 does not crash.

Icinga 2 reduces the default stack size from 8 MB to 256 KB for spawned threads. This is to avoid huge memory reservation and troubles with swap overcommit being disabled.

We consider this behaviour a bug inside the RHEL Kernel and have therefore created an upstream issue (hidden by default).

 

Quickfix

If you are planning to update your RHEL/CentOS system, you can apply this workaround: Add “–no-stack-rlimit” to ExecStart in your systemd configuration file. In order to change this permanently copy the existing systemd service file and then apply the changes.

cp /lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service

vim /etc/systemd/system/icinga2.service
ExecStart=/usr/sbin/icinga2 daemon -d -e ${ICINGA2_ERROR_LOG} --no-stack-rlimit

systemctl daemon-reload

Edit 2017-06-20 17:55 Europe/Berlin: You’ll also need to patch the /usr/lib/icinga2/prepare-dirs script to use –no-stack-rlimit parameter.

-ICINGA2_USER=`$DAEMON variable get --current RunAsUser`
+ICINGA2_USER=`$DAEMON variable get --current RunAsUser --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsUser variable. Error '$ICINGA2_USER'. Exiting."
         exit 6
 fi

-ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup`
+ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsGroup variable. Error '$ICINGA2_GROUP'. Exiting."
         exit 6

 

A simpler workaround for the icinga2 shell wrapper in /usr/sbin/icinga2 is mentioned in #5367:

vim /usr/sbin/icinga2

#exec $ICINGA2_BIN "$@"
exec $ICINGA2_BIN --no-stack-rlimit "$@"

 

The long term solution will be applied in Icinga 2 v2.7 by making this setting a Systemd/SysVInit configuration variable.

RHEL 6 is not affected in our ongoing tests.

If you are running RHEL 6, edit the /etc/init.d/icinga2 init script:

cp /etc/init.d/icinga2 /etc/init.d/icinga2.orig

vim /etc/init.d/icinga2

if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -d -e $ICINGA2_ERROR_LOG --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then
...
if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -C --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then

 

How to monitor your MySQL server(s) with Icinga 2

Are you responsible for one or more MySQL servers in your company or organisation? Do you run a MySQL replication setup or multiple MySQL instances on your machine? Then you probably want to know if they are performing as well as they could do and if you’re getting the most out of your hardware. Monitoring your MySQL server(s) provides valuable information — both for avoiding and dealing with problems. If you’re looking for a free and open-source solution, then it’s time to learn our tips and tricks for monitoring MySQL with Icinga 2.

This document introduces the necessary monitoring plugins and shows some Icinga 2 configuration examples for monitoring various MySQL setups, from single servers to master/slave setups with replication, to multiple MySQL instances on a single server.

Tip: All sample listings can also apply to MariaDB and Percona Server. So, whenever we talk about MySQL, this includes the two spin-offs. read more…

Icinga Performance Monitoring and Analysis

Icinga 2 is a feature “monster”. You can do so much more with it than just “check” and “notify”. Forward your performance data into metric systems such as Graphite or InfluxDB, add the IDO database backend for beautiful dashboards in Icinga Web 2 or connect to the REST API and have Dashing present the latest stats in your office.

After all, Icinga 2 runs as an application on your server and will suffer from outages, full disks, load and memory issues and what not. It shouldn’t happen but what if?

read more…

Monthly Snap May: Events, Integrations & Community Updates

This time we’re focusing on many cool integrations, past and upcoming events and even more with and around Icinga.

We’ve celebrated eight years Icinga in May – hooray!

Currently we are working on Icinga 2 v2.7 to be released in June. This will include certain enhancements for performance and metrics. I’m writing a blog post soon. We are also pushing resources into Icinga Exchange and our new backend. Many things cooking under the hood, and soon to be shared with insights.

One of them is new build infrastructure which has been released this week. Built with Jenkins and Puppet and developed in the open. All involved scripts, modules and a Vagrant test environment can be found on GitHub. You’ll also recognise a new theme on packages.icinga.com :)

read more…

Icinga 2 and AlertOps – a perfect match

AlertOps logoIcinga 2 can generate its own alerts when a host or service has reached a certain state (hosts: UP, DOWN, or UNREACHABLE, services: OK, WARNING, CRITICAL, or UNKNOWN). With the right configuration the monitoring software sends out emails, text or instant messages, etc. to users or user groups. (Check out our documentation to learn more about Notification objects.) As soon as you want to organise notifications for more than a few people or several teams with different on-call duties, things can become a bit uncomfortable in Icinga 2. read more…