Why is `kernel.core_pattern` not effective on my Ubuntu 18.04 VM after rebooting?

by yaobin.wen

My Intention

Simply put: I wanted to change the coredump pattern to /var/tmp/core.%h.%e.%t, but after I set it successfully on the command line, the pattern got rolled back after a reboot.

What I Did

I ran the following ansible command:

ansible -bKvvv -m "sysctl" -a "name=kernel.core_pattern state=present value='/var/tmp/core.%h.%e.%t' reload=yes" localhost

and it succeeded with the output:

localhost | CHANGED => {
    "changed": true,
    "invocation": {
        "module_args": {
            "ignoreerrors": false,
            "name": "kernel.core_pattern",
            "reload": true,
            "state": "present",
            "sysctl_file": "/etc/sysctl.conf",
            "sysctl_set": false,
            "value": "/var/tmp/core.%h.%e.%t"
        }
    }
}

I also checked the current configuration:

vagrant@ubuntu-bionic:/etc$ sysctl kernel.core_pattern
kernel.core_pattern = /var/tmp/core.%h.%e.%t

I also confirmed that the core_pattern was written into /etc/sysctl.conf:

vagrant@ubuntu-bionic:~$ grep "core_pattern" /etc/sysctl.conf
kernel.core_pattern=/var/tmp/core.%h.%e.%t

Then I rebooted my VM. After rebooting, the currently effective core_pattern got rolled back:

vagrant@ubuntu-bionic:~$ sysctl kernel.core_pattern
kernel.core_pattern = |/usr/share/apport/apport %p %s %c %d %P %E

Investigation & Result

After further investigation, I found kernel.core_pattern was overwritten by the package apport at system booting.

I should have thought of checking apport when I saw the string “/usr/share/apport/apport”. I wasn’t sure if apport was actually the one that overwrites kernel.core_pattern, so I decided to take a look at its source code.

By running apt-cache policy apport, I found the version I was using was 2.20.9-0ubuntu7.24:

vagrant@ubuntu-bionic:~$ apt-cache policy apport
apport:
  Installed: 2.20.9-0ubuntu7.24
  Candidate: 2.20.9-0ubuntu7.24
  Version table:
 *** 2.20.9-0ubuntu7.24 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.20.9-0ubuntu7 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
vagrant@ubuntu-bionic:~$

Then on apport’s LaunchPad page, I found its download link: apport_2.20.9-0ubuntu7.tar.gz.

Searching in the source code, I found the file debian/apport.init has the following content:

do_start()
{
    ...
    ...
    echo "|$AGENT %p %s %c %d %P" > /proc/sys/kernel/core_pattern
    echo 2 > /proc/sys/fs/suid_dumpable
}

So I ran dpkg -L apport on my VM to see if there was this file:

vagrant@ubuntu-bionic:~$ dpkg -L apport | grep init
/etc/init.d
/etc/init.d/apport

There wasn’t any file with the exactly matching file name, but I decided to take a look at /etc/init.d/apport and found this was the file I was looking for:

do_start()
{
        ...
        ...
        echo "|$AGENT %p %s %c %d %P %E" > /proc/sys/kernel/core_pattern
        echo 2 > /proc/sys/fs/suid_dumpable
}

I knew that /etc/sysctl.conf is read by the service systemd-sysctl, so the next thing I wanted to determine was whether systemd-sysctl was started before apport.

I searched if there was any apport-related service and found one:

vagrant@ubuntu-bionic:~$ systemctl list-units | grep apport
apport.service                                                                      loaded active exited    LSB: automatic crash report generation

I then checked their inter-dependency and found apport.service is started after systemd-sysctl.service:

vagrant@ubuntu-bionic:~$ systemctl list-dependencies apport.service
apport.service
● ├─system.slice
● └─sysinit.target
...
●   ├─systemd-sysctl.service
...
...
vagrant@ubuntu-bionic:~$

That means systemd-sysctl.service correctly reads my setting of kernel.core_pattern in /etc/sysctl.conf but the setting is then immediately overwritten by apport.service.

To verify this, I added two lines in /etc/init.d/apport:

do_start()
{
        ...
        ...
        # NOTE(ywen): Write the current value to a file.
        sysctl kernel.core_pattern > /home/vagrant/sysctl.kernel.core_pattern.txt

        echo "|$AGENT %p %s %c %d %P %E" > /proc/sys/kernel/core_pattern
        echo 2 > /proc/sys/fs/suid_dumpable

        # NOTE(ywen): Append the current value to the file.
        sysctl kernel.core_pattern >> /home/vagrant/sysctl.kernel.core_pattern.txt
}

Then I rebooted the VM. When the VM was up again, I found the file /home/vagrant/sysctl.kernel.core_pattern.txt had the following content:

vagrant@ubuntu-bionic:~$ cat sysctl.kernel.core_pattern.txt
kernel.core_pattern = /var/tmp/core.%h.%e.%t
kernel.core_pattern = |/usr/share/apport/apport %p %s %c %d %P %E
vagrant@ubuntu-bionic:~$

So, indeed, my setting was correctly read but then overwritten by apport.

(Originally I posted this article here.)