4. Segmentation faults in apache2

Segmentation faults in apache2

  • genesis74

    Hi @Manfred,

    • segfaults only happening on apache shutdown
    • yes, I am pretty sure that all Z-Push connections cause a segfault on shutdown; at least during several tests
    • no, segfaults only appear on apache shutdown, not when it is running
    • no, until now I did not generate core dumps
    • following apache modules are enabled:
      • core_module (static)
      • so_module (static)
      • watchdog_module (static)
      • http_module (static)
      • log_config_module (static)
      • logio_module (static)
      • version_module (static)
      • unixd_module (static)
      • access_compat_module (shared)
      • alias_module (shared)
      • auth_basic_module (shared)
      • authn_core_module (shared)
      • authn_file_module (shared)
      • authz_core_module (shared)
      • authz_host_module (shared)
      • authz_user_module (shared)
      • autoindex_module (shared)
      • cgi_module (shared)
      • deflate_module (shared)
      • dir_module (shared)
      • env_module (shared)
      • filter_module (shared)
      • headers_module (shared)
      • mime_module (shared)
      • mpm_prefork_module (shared)
      • negotiation_module (shared)
      • php7_module (shared)
      • reqtimeout_module (shared)
      • rewrite_module (shared)
      • setenvif_module (shared)
      • socache_shmcb_module (shared)
      • ssl_module (shared)
      • status_module (shared)
    0
  • Manfred
    Kopano

    Hi genesis74,

    could you provide us a core dump?

    Manfred

    0
  • genesis74

    Hi @Manfred,

    I am sorry but I seem to be unable to generate a core dump. Tried for hours now. Giving up. Don’t want to get off-topic here with discussing how core dump can be generated but maybe you can provide me with a step-by-step guide. Followed the official apache way (/usr/share/doc/apache2/README.backtrace) and tried nearly everything I found on the net with no help.

    0
  • Manfred
    Kopano

    Hi genesis74,

    If I remember correctly the last time I needed a core dump I followed those links: https://serverfault.com/questions/470407/how-to-get-a-core-dump-from-apache-when-segfaulting/470410#470410 and https://askubuntu.com/questions/612532/how-to-get-an-apache-core-dump .

    Alternatively you could start apache2 in single thread mode, attach with gdb to the process and then force for the segfault: https://httpd.apache.org/dev/debugging.html.

    Manfred

    0
  • genesis74

    Hi @Manfred,

    appreciate your help. Those links you posted were part of my unsuccessful tries yesterday. Today I found this instruction which finally led to core dumps being produced. I suspect the most important part is to set /proc/sys/kernel/core_pattern correctly (was pointing to apport in my Ubuntu 18.04 server environment).

    However, here is the gdb output:

    (gdb) bt full
#0  0x00007f77bf0a6ca0 in ?? ()
No symbol table info available.
#1  0x00007f77d53dbdda in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#2  0x00007f77d53dbe5a in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#3  0x00007f77d53d7b26 in zend_hash_index_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#4  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#5  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#6  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#7  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#8  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#9  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#10 0x00007f77d53c41fd in _zval_dtor_func () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#11 0x00007f77d53b8e5a in destroy_zend_class () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#12 0x00007f77d53d7e44 in zend_hash_destroy () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#13 0x00007f77d53c5fb1 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#14 0x00007f77d536210b in php_module_shutdown () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#15 0x00007f77d53621c9 in php_module_shutdown_wrapper () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#16 0x00007f77d5478971 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#17 0x00007f77d8cf7e2e in run_cleanups (cref=<optimized out>) at ./memory/unix/apr_pools.c:2629
        c = <optimized out>
        c = <optimized out>
#18 apr_pool_destroy (pool=0x7f77d9493028) at ./memory/unix/apr_pools.c:1000
        active = <optimized out>
        allocator = <optimized out>
#19 0x00007f77d59e4283 in clean_child_exit (code=code@entry=0) at prefork.c:227
No locals.
#20 0x00007f77d59e42bb in just_die (sig=<optimized out>) at prefork.c:355
No locals.
#21 0x00007f77d53f298c in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#22 0x00007f77d53f2b62 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#23 <signal handler called>
No locals.
#24 0x00007f77d8ac9f85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ffe2b32ffe0, expected=0, futex_word=0x55cfcd322818) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        __ret = -4
        oldtype = 0
        err = <optimized out>
        oldtype = <optimized out>
        err = <optimized out>
        __ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#25 __pthread_cond_wait_common (abstime=0x7ffe2b32ffe0, mutex=0x55cfcd3227c8, cond=0x55cfcd3227f0) at pthread_cond_wait.c:539
        spin = 0
        buffer = {__routine = 0x7f77d8ac9690 <__condvar_cleanup_waiting>, __arg = 0x7ffe2b32ff30, __canceltype = -852350944, __prev = 0x0}
        cbuffer = {wseq = 0, cond = 0x55cfcd3227f0, mutex = 0x55cfcd3227c8, private = 0}
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = <optimized out>
        seq = 0
        private = <optimized out>
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        rt = <optimized out>
        s = <optimized out>
#26 __pthread_cond_timedwait (cond=0x55cfcd3227f0, mutex=0x55cfcd3227c8, abstime=0x7ffe2b32ffe0) at pthread_cond_wait.c:667
No locals.
#27 0x00007f77c3401b80 in ?? ()
No symbol table info available.
#28 0x00007ffe2b32ffe0 in ?? ()
No symbol table info available.
#29 0x00002e07f698254e in ?? ()
No symbol table info available.
#30 0x00007ffe2b330044 in ?? ()
No symbol table info available.
#31 0x00007ffe2b330058 in ?? ()
No symbol table info available.
#32 0x0000000000000002 in ?? ()
No symbol table info available.
#33 0x0000000000000000 in ?? ()
No symbol table info available.
    0
  • Manfred
    Kopano

    Hi genesis74,

    I’m not the gdb expert, but I suppose you’ll have to install dbg packages for apache and php in order the dump to be helpful.

    Manfred

    0
  • genesis74

    Hi @Manfred,

    well, until you asked me if I could provide a core dump I never touched gdb. Tried to install dbg packages for apache and php - but that doesn’t seem to help. Moreover, apache is now throwing segfaults not only on shutdown but also during runtime. So I think I have to further shuffel with mods and packages hopefully finding out what the problem is. Thank you very much for your help. If I find a solution I will post it here.

    0
  • genesis74

    I switched my apache server from mpm_prefork to php-fpm. This solved the segfault problem. But there are still errors on shutdown so the root cause still seems to be present. I am pretty sure that somehow it is connected to a problem with apache shutdown while z-push connections are open. This is what is noted in error.log on apache2 shutdown:

    [Sun Feb 10 17:31:55.057810 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:31:57.060256 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:31:59.062464 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:32:01.064844 2019] [core:error] [pid 14957:tid 140151787768768] AH00046: child process 14959 still did not exit, sending a SIGKILL

    The example above represents an apache shutdown while only one z-push connection was open. For every additional open z-push connection these lines appear with the respective PID. Maybe this is a hint where the problem might be. At least this is better than experiencing segfaults. But still I would prefer to find the cause for this.

    0
  • genesis74

    Hi @Manfred,

    may I come back to this issue … I returned to mpm_prefork and still searching for a solution for this. Found the following:

    • on apache2 server status page all Z-Push connections are marked as ‚W‘ which means ‚sending reply‘
    • when stopping apache2 with ‚systemctl stop apache2‘ the segfaults appear for every PID/connection in that ‚W‘ state
    • when stopping apache2 with ‚apachectl graceful-stop‘ those connections are then marked as ‚G‘ (‚gracefully finishing‘) and apache2 needs a long time to shut down (did not check how many minutes yet - maybe connection lifetime?)

    I am pretty sure that for some reason Z-Push connections cannot be closed gracefully and are held open even if apache2 is trying to close down (gracefully). When not using the graceful option connections are closed but cause a segfault.

    Do you have any idea on that? Can anyone reproduce that or am I the only one with this problem? Is it normal behaviour that connections stay in the ‚W‘ state on apache even though it is just pinging the respective device?

    0
  • Manfred
    Kopano

    Hi genesis74,

    the devices keep the Ping-connections open, maybe that’s the reason why their status is “W”. However I’m not an apache expert, I suggest you to contact the Kopano Support regarding this issue.

    Manfred

    0
  • genesis74

    Hi @Manfred,

    I did not contact support yet. Too many possibilities I wanted to rule out. What I have done - just to keep you informed:

    • Complete reinstall of Ubuntu 18.04.2 on a test system with reinstall of Kopano 8.6.9 as well as 8.7.0 with Z-Push 2.4.5
    • Tested with PHP 7.3 - this included compiling and installing the latest Kopano community version in order to get mapi.so running

    It is hard to believe that I am the only one with this problem since I built up the system from scratch ending with the same segmentation faults again. Is still no one here experiencing the same problems? Nobody using Ubuntu 18.04.2 as base for a Kopano system? Or the other way round: Anybody using Ubuntu 18.04.2 but not experiencing this issue?

    0
  • genesis74

    With Debian 9, PHP 7.0, Apache 2.4.25 and same installation steps as with Ubuntu 18.04: No segmentation faults.

    Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?

    0 Manfred 1 Reply
  • Manfred
    Kopano

    Hi genesis74,
    @genesis74 said in Segmentation faults in apache2:

    Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?

    Are you referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions? I’ve updated the page to include Ubuntu 18.04 as well. Z-Push supports PHP 7.2 since version 2.4.3 and if I remember correctly that’s when we added Ubuntu 18.04 repository.

    There are a couple of hundred Z-Push installations on Ubuntu 18.04, so I suppose if it wasn’t working, we would have heard by now.

    If you want to be sure which distributions Z-Push supports, it’s better to check the repo page: http://repo.z-hub.io/z-push:/final/.

    Greets, Manfred

    0
  • genesis74

    Hi @Manfred,

    yes, I was referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions. I was still trying to find the reason for these segfaults. While my system seems to be running fine under Debian I don’t believe it’s the PHP version that makes the difference. At least it did not make any difference when I installed PHP 7.0 or 7.3 on Ubuntu. I did not try to downgrade Apache. However, migration to Debian fixed the problem for me. A little bit dissatisfactory, but ok.

    0
Log in to reply