Advisory for latest security updates on RHEL 7

Community members told us today that Icinga 2 stopped working with the most recent RedHat Enterprise Linux 7 Kernel update 3.10.0-514.21.2. This update includes a security patch for the stack guard vulnerability.

Update 2017-06-20 19:55 Europe/Berlin: CentOS 7 is currently rolling the kernel update and affected too. Upstream bug report has been created.

Update 2017-06-21 11:30 Europe/Berlin: RedHat’s kernel team is investigating on this possible regression. We are in touch with them. Meanwhile apply the quickfix below.

Update 2017-06-21 19:20 Europe/Berlin: https://bugzilla.redhat.com/show_bug.cgi?id=1463241 is now public.

Update 2017-06-22 14:45 Europe/Berlin: We are investigating on the Icinga 2 side how to better handle rlimits. v2.7 is postponed until the Kernel issue is fully resolved. Other distributions might be still affected, there’s ongoing investigation on the oss-sec mailing lists.

Update 2017-06-23 10:45 Europe/Berlin: RedHat provided us with a fixed test package and our tests went fine. Please open a support case at RedHat to receive “accelerated fixes” or to get a test binary package. You also raise awareness by doing so, this should help getting an official release sooner.

 

Analysis

We’ve analysed the issue and reproduced the issue on RHEL7. Debian Jessie and Stretch also released a security update for CVE-2017-1000364 but Icinga 2 does not crash.

Icinga 2 reduces the default stack size from 8 MB to 256 KB for spawned threads. This is to avoid huge memory reservation and troubles with swap overcommit being disabled.

We consider this behaviour a bug inside the RHEL Kernel and have therefore created an upstream issue (hidden by default).

 

Quickfix

If you are planning to update your RHEL/CentOS system, you can apply this workaround: Add “–no-stack-rlimit” to ExecStart in your systemd configuration file. In order to change this permanently copy the existing systemd service file and then apply the changes.

cp /lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service

vim /etc/systemd/system/icinga2.service
ExecStart=/usr/sbin/icinga2 daemon -d -e ${ICINGA2_ERROR_LOG} --no-stack-rlimit

systemctl daemon-reload

Edit 2017-06-20 17:55 Europe/Berlin: You’ll also need to patch the /usr/lib/icinga2/prepare-dirs script to use –no-stack-rlimit parameter.

-ICINGA2_USER=`$DAEMON variable get --current RunAsUser`
+ICINGA2_USER=`$DAEMON variable get --current RunAsUser --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsUser variable. Error '$ICINGA2_USER'. Exiting."
         exit 6
 fi

-ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup`
+ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsGroup variable. Error '$ICINGA2_GROUP'. Exiting."
         exit 6

 

A simpler workaround for the icinga2 shell wrapper in /usr/sbin/icinga2 is mentioned in #5367:

vim /usr/sbin/icinga2

#exec $ICINGA2_BIN "$@"
exec $ICINGA2_BIN --no-stack-rlimit "$@"

 

The long term solution will be applied in Icinga 2 v2.7 by making this setting a Systemd/SysVInit configuration variable.

RHEL 6 is not affected in our ongoing tests.

If you are running RHEL 6, edit the /etc/init.d/icinga2 init script:

cp /etc/init.d/icinga2 /etc/init.d/icinga2.orig

vim /etc/init.d/icinga2

if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -d -e $ICINGA2_ERROR_LOG --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then
...
if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -C --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then

 

Icinga Performance Monitoring and Analysis

Icinga 2 is a feature “monster”. You can do so much more with it than just “check” and “notify”. Forward your performance data into metric systems such as Graphite or InfluxDB, add the IDO database backend for beautiful dashboards in Icinga Web 2 or connect to the REST API and have Dashing present the latest stats in your office.

After all, Icinga 2 runs as an application on your server and will suffer from outages, full disks, load and memory issues and what not. It shouldn’t happen but what if?

(more…)

Icinga 2 and AlertOps – a perfect match

AlertOps logoIcinga 2 can generate its own alerts when a host or service has reached a certain state (hosts: UP, DOWN, or UNREACHABLE, services: OK, WARNING, CRITICAL, or UNKNOWN). With the right configuration the monitoring software sends out emails, text or instant messages, etc. to users or user groups. (Check out our documentation to learn more about Notification objects.) As soon as you want to organise notifications for more than a few people or several teams with different on-call duties, things can become a bit uncomfortable in Icinga 2. (more…)

Vagrant box updates: Elastic Stack, InfluxDB & Icinga Web 2 modules

A while back we’ve written about changes inside our Vagrant box demo environments – and many things happened ever since.

There are a couple of new Icinga Web 2 modules directly integrated into the Vagrant boxes (Director, Grafana, Cube, Globe). In terms of metrics and event collecting we’ve integrated Elastic Stack with Icingabeat and also InfluxDB with Grafana. We are happy to release v1.3.0 today.

(more…)

Process Icinga Logs with Logstash

Central log management has always been a topic for almost every sysadmin. In the past we used a central syslog server to collect all logs from clients and store them in plain text files. Instead of searching the logs on 10 web servers, the sysadmin had to run just a single grep command on one machine. When someone managed to hack into your server, he probably wasn’t fast or clever enough to disable the remote logging. He did what he did, but your logs were save for later analysis. In an ideal world even network hardware would send their logs to this central syslog instance. Well, in an ideal world the “syslog” format would be the same on each device. In an ideal world developers would use reasonable and standardised timestamp formats. Instead, we have to write regexes for the umpteenth time to parse this stuff. Sigh. (more…)