Advisory for latest security updates on RHEL 7

Community members told us today that Icinga 2 stopped working with the most recent RedHat Enterprise Linux 7 Kernel update 3.10.0-514.21.2. This update includes a security patch for the stack guard vulnerability.

Update 2017-06-20 19:55 Europe/Berlin: CentOS 7 is currently rolling the kernel update and affected too. Upstream bug report has been created.

Update 2017-06-21 11:30 Europe/Berlin: RedHat’s kernel team is investigating on this possible regression. We are in touch with them. Meanwhile apply the quickfix below.

Update 2017-06-21 19:20 Europe/Berlin: https://bugzilla.redhat.com/show_bug.cgi?id=1463241 is now public.

Update 2017-06-22 14:45 Europe/Berlin: We are investigating on the Icinga 2 side how to better handle rlimits. v2.7 is postponed until the Kernel issue is fully resolved. Other distributions might be still affected, there’s ongoing investigation on the oss-sec mailing lists.

Update 2017-06-23 10:45 Europe/Berlin: RedHat provided us with a fixed test package and our tests went fine. Please open a support case at RedHat to receive “accelerated fixes” or to get a test binary package. You also raise awareness by doing so, this should help getting an official release sooner.

 

Analysis

We’ve analysed the issue and reproduced the issue on RHEL7. Debian Jessie and Stretch also released a security update for CVE-2017-1000364 but Icinga 2 does not crash.

Icinga 2 reduces the default stack size from 8 MB to 256 KB for spawned threads. This is to avoid huge memory reservation and troubles with swap overcommit being disabled.

We consider this behaviour a bug inside the RHEL Kernel and have therefore created an upstream issue (hidden by default).

 

Quickfix

If you are planning to update your RHEL/CentOS system, you can apply this workaround: Add “–no-stack-rlimit” to ExecStart in your systemd configuration file. In order to change this permanently copy the existing systemd service file and then apply the changes.

cp /lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service

vim /etc/systemd/system/icinga2.service
ExecStart=/usr/sbin/icinga2 daemon -d -e ${ICINGA2_ERROR_LOG} --no-stack-rlimit

systemctl daemon-reload

Edit 2017-06-20 17:55 Europe/Berlin: You’ll also need to patch the /usr/lib/icinga2/prepare-dirs script to use –no-stack-rlimit parameter.

-ICINGA2_USER=`$DAEMON variable get --current RunAsUser`
+ICINGA2_USER=`$DAEMON variable get --current RunAsUser --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsUser variable. Error '$ICINGA2_USER'. Exiting."
         exit 6
 fi

-ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup`
+ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsGroup variable. Error '$ICINGA2_GROUP'. Exiting."
         exit 6

 

A simpler workaround for the icinga2 shell wrapper in /usr/sbin/icinga2 is mentioned in #5367:

vim /usr/sbin/icinga2

#exec $ICINGA2_BIN "$@"
exec $ICINGA2_BIN --no-stack-rlimit "$@"

 

The long term solution will be applied in Icinga 2 v2.7 by making this setting a Systemd/SysVInit configuration variable.

RHEL 6 is not affected in our ongoing tests.

If you are running RHEL 6, edit the /etc/init.d/icinga2 init script:

cp /etc/init.d/icinga2 /etc/init.d/icinga2.orig

vim /etc/init.d/icinga2

if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -d -e $ICINGA2_ERROR_LOG --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then
...
if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -C --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then

 

Icinga Performance Monitoring and Analysis

Icinga 2 is a feature “monster”. You can do so much more with it than just “check” and “notify”. Forward your performance data into metric systems such as Graphite or InfluxDB, add the IDO database backend for beautiful dashboards in Icinga Web 2 or connect to the REST API and have Dashing present the latest stats in your office.

After all, Icinga 2 runs as an application on your server and will suffer from outages, full disks, load and memory issues and what not. It shouldn’t happen but what if?

(more…)

Monthly Snap May: Events, Integrations & Community Updates

This time we’re focusing on many cool integrations, past and upcoming events and even more with and around Icinga.

We’ve celebrated eight years Icinga in May – hooray!

Currently we are working on Icinga 2 v2.7 to be released in June. This will include certain enhancements for performance and metrics. I’m writing a blog post soon. We are also pushing resources into Icinga Exchange and our new backend. Many things cooking under the hood, and soon to be shared with insights.

One of them is new build infrastructure which has been released this week. Built with Jenkins and Puppet and developed in the open. All involved scripts, modules and a Vagrant test environment can be found on GitHub. You’ll also recognise a new theme on packages.icinga.com :)

(more…)

8 years #icingalove

8 years ago, May 2009 all we knew – create a better monitoring tool and work with our community and their demands. Integrate long awaited patches and functionality and care about feedback. Community members from all around the globe joined that vision. Germany, Austria, Italy, UK, India, USA, Australia, Brazil, Belgium,… many of us never worked in an open source project, and we try to make things right. Passionate with lots of emotional discussions, still putting our ideas to the next level.

(more…)

Monthly Snap April: Vagrant boxes, Logstash & Foreman

Greetings from sunny Upper Austria – I’m enjoying some days back at home, away from lovely Nuremberg. It is getting warmer over here after we’ve successfully welcomed Winter again during Easter ;-)

What happened with Icinga in April? Blerim blogged about the new log processor for Logstash and further introduced automated monitoring with Foreman. The webinar was moved to upcoming Thursday, 4.5.2017 16:00-17:00 +02:00.

We’ve also revamped our Vagrant boxes a bit, more details in this blog post. Our fellow community member Carsten takes the Grafana module for Icinga Web 2 to the next level, make sure to check it out! :)

Go for a walk outside, and meet a baby cow like our “Icinga”, if you can. Your monitoring keeps you safe to enjoy nature and the hopefully lovely weather at your place. See you soon! :-)