Segmentation faults in apache2


  • Kopano

    Hi genesis74,

    could you provide us a core dump?

    Manfred



  • Hi @Manfred,

    I am sorry but I seem to be unable to generate a core dump. Tried for hours now. Giving up. Don’t want to get off-topic here with discussing how core dump can be generated but maybe you can provide me with a step-by-step guide. Followed the official apache way (/usr/share/doc/apache2/README.backtrace) and tried nearly everything I found on the net with no help.


  • Kopano

    Hi genesis74,

    If I remember correctly the last time I needed a core dump I followed those links: https://serverfault.com/questions/470407/how-to-get-a-core-dump-from-apache-when-segfaulting/470410#470410 and https://askubuntu.com/questions/612532/how-to-get-an-apache-core-dump .

    Alternatively you could start apache2 in single thread mode, attach with gdb to the process and then force for the segfault: https://httpd.apache.org/dev/debugging.html.

    Manfred



  • Hi @Manfred,

    appreciate your help. Those links you posted were part of my unsuccessful tries yesterday. Today I found this instruction which finally led to core dumps being produced. I suspect the most important part is to set /proc/sys/kernel/core_pattern correctly (was pointing to apport in my Ubuntu 18.04 server environment).

    However, here is the gdb output:

    (gdb) bt full
    #0  0x00007f77bf0a6ca0 in ?? ()
    No symbol table info available.
    #1  0x00007f77d53dbdda in ?? () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #2  0x00007f77d53dbe5a in ?? () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #3  0x00007f77d53d7b26 in zend_hash_index_del () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #4  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #5  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #6  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #7  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #8  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #9  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #10 0x00007f77d53c41fd in _zval_dtor_func () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #11 0x00007f77d53b8e5a in destroy_zend_class () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #12 0x00007f77d53d7e44 in zend_hash_destroy () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #13 0x00007f77d53c5fb1 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #14 0x00007f77d536210b in php_module_shutdown () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #15 0x00007f77d53621c9 in php_module_shutdown_wrapper () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #16 0x00007f77d5478971 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #17 0x00007f77d8cf7e2e in run_cleanups (cref=<optimized out>) at ./memory/unix/apr_pools.c:2629
            c = <optimized out>
            c = <optimized out>
    #18 apr_pool_destroy (pool=0x7f77d9493028) at ./memory/unix/apr_pools.c:1000
            active = <optimized out>
            allocator = <optimized out>
    #19 0x00007f77d59e4283 in clean_child_exit (code=code@entry=0) at prefork.c:227
    No locals.
    #20 0x00007f77d59e42bb in just_die (sig=<optimized out>) at prefork.c:355
    No locals.
    #21 0x00007f77d53f298c in ?? () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #22 0x00007f77d53f2b62 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
    No symbol table info available.
    #23 <signal handler called>
    No locals.
    #24 0x00007f77d8ac9f85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ffe2b32ffe0, expected=0, futex_word=0x55cfcd322818) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
            __ret = -4
            oldtype = 0
            err = <optimized out>
            oldtype = <optimized out>
            err = <optimized out>
            __ret = <optimized out>
            resultvar = <optimized out>
            __arg6 = <optimized out>
            __arg5 = <optimized out>
            __arg4 = <optimized out>
            __arg3 = <optimized out>
            __arg2 = <optimized out>
            __arg1 = <optimized out>
            _a6 = <optimized out>
            _a5 = <optimized out>
            _a4 = <optimized out>
            _a3 = <optimized out>
            _a2 = <optimized out>
            _a1 = <optimized out>
    #25 __pthread_cond_wait_common (abstime=0x7ffe2b32ffe0, mutex=0x55cfcd3227c8, cond=0x55cfcd3227f0) at pthread_cond_wait.c:539
            spin = 0
            buffer = {__routine = 0x7f77d8ac9690 <__condvar_cleanup_waiting>, __arg = 0x7ffe2b32ff30, __canceltype = -852350944, __prev = 0x0}
            cbuffer = {wseq = 0, cond = 0x55cfcd3227f0, mutex = 0x55cfcd3227c8, private = 0}
            err = <optimized out>
            g = 0
            flags = <optimized out>
            g1_start = <optimized out>
            maxspin = 0
            signals = <optimized out>
            result = 0
            wseq = <optimized out>
            seq = 0
            private = <optimized out>
            maxspin = <optimized out>
            err = <optimized out>
            result = <optimized out>
            wseq = <optimized out>
            g = <optimized out>
            seq = <optimized out>
            flags = <optimized out>
            private = <optimized out>
            signals = <optimized out>
            g1_start = <optimized out>
            spin = <optimized out>
            buffer = <optimized out>
            cbuffer = <optimized out>
            rt = <optimized out>
            s = <optimized out>
    #26 __pthread_cond_timedwait (cond=0x55cfcd3227f0, mutex=0x55cfcd3227c8, abstime=0x7ffe2b32ffe0) at pthread_cond_wait.c:667
    No locals.
    #27 0x00007f77c3401b80 in ?? ()
    No symbol table info available.
    #28 0x00007ffe2b32ffe0 in ?? ()
    No symbol table info available.
    #29 0x00002e07f698254e in ?? ()
    No symbol table info available.
    #30 0x00007ffe2b330044 in ?? ()
    No symbol table info available.
    #31 0x00007ffe2b330058 in ?? ()
    No symbol table info available.
    #32 0x0000000000000002 in ?? ()
    No symbol table info available.
    #33 0x0000000000000000 in ?? ()
    No symbol table info available.
    

  • Kopano

    Hi genesis74,

    I’m not the gdb expert, but I suppose you’ll have to install dbg packages for apache and php in order the dump to be helpful.

    Manfred



  • Hi @Manfred,

    well, until you asked me if I could provide a core dump I never touched gdb. Tried to install dbg packages for apache and php - but that doesn’t seem to help. Moreover, apache is now throwing segfaults not only on shutdown but also during runtime. So I think I have to further shuffel with mods and packages hopefully finding out what the problem is. Thank you very much for your help. If I find a solution I will post it here.



  • I switched my apache server from mpm_prefork to php-fpm. This solved the segfault problem. But there are still errors on shutdown so the root cause still seems to be present. I am pretty sure that somehow it is connected to a problem with apache shutdown while z-push connections are open. This is what is noted in error.log on apache2 shutdown:

    [Sun Feb 10 17:31:55.057810 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
    [Sun Feb 10 17:31:57.060256 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
    [Sun Feb 10 17:31:59.062464 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
    [Sun Feb 10 17:32:01.064844 2019] [core:error] [pid 14957:tid 140151787768768] AH00046: child process 14959 still did not exit, sending a SIGKILL
    

    The example above represents an apache shutdown while only one z-push connection was open. For every additional open z-push connection these lines appear with the respective PID. Maybe this is a hint where the problem might be. At least this is better than experiencing segfaults. But still I would prefer to find the cause for this.



  • Hi @Manfred,

    may I come back to this issue … I returned to mpm_prefork and still searching for a solution for this. Found the following:

    • on apache2 server status page all Z-Push connections are marked as ‚W‘ which means ‚sending reply‘
    • when stopping apache2 with ‚systemctl stop apache2‘ the segfaults appear for every PID/connection in that ‚W‘ state
    • when stopping apache2 with ‚apachectl graceful-stop‘ those connections are then marked as ‚G‘ (‚gracefully finishing‘) and apache2 needs a long time to shut down (did not check how many minutes yet - maybe connection lifetime?)

    I am pretty sure that for some reason Z-Push connections cannot be closed gracefully and are held open even if apache2 is trying to close down (gracefully). When not using the graceful option connections are closed but cause a segfault.

    Do you have any idea on that? Can anyone reproduce that or am I the only one with this problem? Is it normal behaviour that connections stay in the ‚W‘ state on apache even though it is just pinging the respective device?


  • Kopano

    Hi genesis74,

    the devices keep the Ping-connections open, maybe that’s the reason why their status is “W”. However I’m not an apache expert, I suggest you to contact the Kopano Support regarding this issue.

    Manfred



  • Hi @Manfred,

    I did not contact support yet. Too many possibilities I wanted to rule out. What I have done - just to keep you informed:

    • Complete reinstall of Ubuntu 18.04.2 on a test system with reinstall of Kopano 8.6.9 as well as 8.7.0 with Z-Push 2.4.5
    • Tested with PHP 7.3 - this included compiling and installing the latest Kopano community version in order to get mapi.so running

    It is hard to believe that I am the only one with this problem since I built up the system from scratch ending with the same segmentation faults again. Is still no one here experiencing the same problems? Nobody using Ubuntu 18.04.2 as base for a Kopano system? Or the other way round: Anybody using Ubuntu 18.04.2 but not experiencing this issue?



  • With Debian 9, PHP 7.0, Apache 2.4.25 and same installation steps as with Ubuntu 18.04: No segmentation faults.

    Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?


  • Kopano

    Hi genesis74,
    @genesis74 said in Segmentation faults in apache2:

    Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?

    Are you referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions? I’ve updated the page to include Ubuntu 18.04 as well. Z-Push supports PHP 7.2 since version 2.4.3 and if I remember correctly that’s when we added Ubuntu 18.04 repository.

    There are a couple of hundred Z-Push installations on Ubuntu 18.04, so I suppose if it wasn’t working, we would have heard by now.

    If you want to be sure which distributions Z-Push supports, it’s better to check the repo page: http://repo.z-hub.io/z-push:/final/.

    Greets, Manfred



  • Hi @Manfred,

    yes, I was referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions. I was still trying to find the reason for these segfaults. While my system seems to be running fine under Debian I don’t believe it’s the PHP version that makes the difference. At least it did not make any difference when I installed PHP 7.0 or 7.3 on Ubuntu. I did not try to downgrade Apache. However, migration to Debian fixed the problem for me. A little bit dissatisfactory, but ok.


Log in to reply