Community members told us today that Icinga 2 stopped working with the most recent RedHat Enterprise Linux 7 Kernel update 3.10.0-514.21.2. This update includes a security patch for the stack guard vulnerability.

Update 2017-06-20 19:55 Europe/Berlin: CentOS 7 is currently rolling the kernel update and affected too. Upstream bug report has been created.

Update 2017-06-21 11:30 Europe/Berlin: RedHat’s kernel team is investigating on this possible regression. We are in touch with them. Meanwhile apply the quickfix below.

Update 2017-06-21 19:20 Europe/Berlin: https://bugzilla.redhat.com/show_bug.cgi?id=1463241 is now public.

Update 2017-06-22 14:45 Europe/Berlin: We are investigating on the Icinga 2 side how to better handle rlimits. v2.7 is postponed until the Kernel issue is fully resolved. Other distributions might be still affected, there’s ongoing investigation on the oss-sec mailing lists.

Update 2017-06-23 10:45 Europe/Berlin: RedHat provided us with a fixed test package and our tests went fine. Please open a support case at RedHat to receive “accelerated fixes” or to get a test binary package. You also raise awareness by doing so, this should help getting an official release sooner.

 

Analysis

We’ve analysed the issue and reproduced the issue on RHEL7. Debian Jessie and Stretch also released a security update for CVE-2017-1000364 but Icinga 2 does not crash.

Icinga 2 reduces the default stack size from 8 MB to 256 KB for spawned threads. This is to avoid huge memory reservation and troubles with swap overcommit being disabled.

We consider this behaviour a bug inside the RHEL Kernel and have therefore created an upstream issue (hidden by default).

 

Quickfix

If you are planning to update your RHEL/CentOS system, you can apply this workaround: Add “–no-stack-rlimit” to ExecStart in your systemd configuration file. In order to change this permanently copy the existing systemd service file and then apply the changes.

cp /lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service

vim /etc/systemd/system/icinga2.service
ExecStart=/usr/sbin/icinga2 daemon -d -e ${ICINGA2_ERROR_LOG} --no-stack-rlimit

systemctl daemon-reload

Edit 2017-06-20 17:55 Europe/Berlin: You’ll also need to patch the /usr/lib/icinga2/prepare-dirs script to use –no-stack-rlimit parameter.

-ICINGA2_USER=`$DAEMON variable get --current RunAsUser`
+ICINGA2_USER=`$DAEMON variable get --current RunAsUser --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsUser variable. Error '$ICINGA2_USER'. Exiting."
         exit 6
 fi

-ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup`
+ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsGroup variable. Error '$ICINGA2_GROUP'. Exiting."
         exit 6

 

A simpler workaround for the icinga2 shell wrapper in /usr/sbin/icinga2 is mentioned in #5367:

vim /usr/sbin/icinga2

#exec $ICINGA2_BIN "$@"
exec $ICINGA2_BIN --no-stack-rlimit "$@"

 

The long term solution will be applied in Icinga 2 v2.7 by making this setting a Systemd/SysVInit configuration variable.

RHEL 6 is not affected in our ongoing tests.

If you are running RHEL 6, edit the /etc/init.d/icinga2 init script:

cp /etc/init.d/icinga2 /etc/init.d/icinga2.orig

vim /etc/init.d/icinga2

if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -d -e $ICINGA2_ERROR_LOG --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then
...
if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -C --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then