Navigation

    Kopano
    • Register
    • Login
    • Search
    • Categories
    • Get Official Kopano Support
    • Recent
    Statement regarding the closure of the Kopano community forum and the end of the community edition

    Segmentation faults in apache2

    Z-Push when using Kopano
    3
    20
    5501
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • genesis74
      genesis74 @Manfred last edited by

      Hi @Manfred,

      • segfaults only happening on apache shutdown
      • yes, I am pretty sure that all Z-Push connections cause a segfault on shutdown; at least during several tests
      • no, segfaults only appear on apache shutdown, not when it is running
      • no, until now I did not generate core dumps
      • following apache modules are enabled:
        • core_module (static)
        • so_module (static)
        • watchdog_module (static)
        • http_module (static)
        • log_config_module (static)
        • logio_module (static)
        • version_module (static)
        • unixd_module (static)
        • access_compat_module (shared)
        • alias_module (shared)
        • auth_basic_module (shared)
        • authn_core_module (shared)
        • authn_file_module (shared)
        • authz_core_module (shared)
        • authz_host_module (shared)
        • authz_user_module (shared)
        • autoindex_module (shared)
        • cgi_module (shared)
        • deflate_module (shared)
        • dir_module (shared)
        • env_module (shared)
        • filter_module (shared)
        • headers_module (shared)
        • mime_module (shared)
        • mpm_prefork_module (shared)
        • negotiation_module (shared)
        • php7_module (shared)
        • reqtimeout_module (shared)
        • rewrite_module (shared)
        • setenvif_module (shared)
        • socache_shmcb_module (shared)
        • ssl_module (shared)
        • status_module (shared)
      1 Reply Last reply Reply Quote 0
      • Manfred
        Manfred Kopano last edited by

        Hi genesis74,

        could you provide us a core dump?

        Manfred

        1 Reply Last reply Reply Quote 0
        • genesis74
          genesis74 last edited by

          Hi @Manfred,

          I am sorry but I seem to be unable to generate a core dump. Tried for hours now. Giving up. Don’t want to get off-topic here with discussing how core dump can be generated but maybe you can provide me with a step-by-step guide. Followed the official apache way (/usr/share/doc/apache2/README.backtrace) and tried nearly everything I found on the net with no help.

          1 Reply Last reply Reply Quote 0
          • Manfred
            Manfred Kopano last edited by

            Hi genesis74,

            If I remember correctly the last time I needed a core dump I followed those links: https://serverfault.com/questions/470407/how-to-get-a-core-dump-from-apache-when-segfaulting/470410#470410 and https://askubuntu.com/questions/612532/how-to-get-an-apache-core-dump .

            Alternatively you could start apache2 in single thread mode, attach with gdb to the process and then force for the segfault: https://httpd.apache.org/dev/debugging.html.

            Manfred

            1 Reply Last reply Reply Quote 0
            • genesis74
              genesis74 last edited by

              Hi @Manfred,

              appreciate your help. Those links you posted were part of my unsuccessful tries yesterday. Today I found this instruction which finally led to core dumps being produced. I suspect the most important part is to set /proc/sys/kernel/core_pattern correctly (was pointing to apport in my Ubuntu 18.04 server environment).

              However, here is the gdb output:

              (gdb) bt full
#0  0x00007f77bf0a6ca0 in ?? ()
No symbol table info available.
#1  0x00007f77d53dbdda in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#2  0x00007f77d53dbe5a in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#3  0x00007f77d53d7b26 in zend_hash_index_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#4  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#5  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#6  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#7  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#8  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#9  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#10 0x00007f77d53c41fd in _zval_dtor_func () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#11 0x00007f77d53b8e5a in destroy_zend_class () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#12 0x00007f77d53d7e44 in zend_hash_destroy () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#13 0x00007f77d53c5fb1 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#14 0x00007f77d536210b in php_module_shutdown () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#15 0x00007f77d53621c9 in php_module_shutdown_wrapper () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#16 0x00007f77d5478971 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#17 0x00007f77d8cf7e2e in run_cleanups (cref=<optimized out>) at ./memory/unix/apr_pools.c:2629
        c = <optimized out>
        c = <optimized out>
#18 apr_pool_destroy (pool=0x7f77d9493028) at ./memory/unix/apr_pools.c:1000
        active = <optimized out>
        allocator = <optimized out>
#19 0x00007f77d59e4283 in clean_child_exit (code=code@entry=0) at prefork.c:227
No locals.
#20 0x00007f77d59e42bb in just_die (sig=<optimized out>) at prefork.c:355
No locals.
#21 0x00007f77d53f298c in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#22 0x00007f77d53f2b62 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#23 <signal handler called>
No locals.
#24 0x00007f77d8ac9f85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ffe2b32ffe0, expected=0, futex_word=0x55cfcd322818) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        __ret = -4
        oldtype = 0
        err = <optimized out>
        oldtype = <optimized out>
        err = <optimized out>
        __ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#25 __pthread_cond_wait_common (abstime=0x7ffe2b32ffe0, mutex=0x55cfcd3227c8, cond=0x55cfcd3227f0) at pthread_cond_wait.c:539
        spin = 0
        buffer = {__routine = 0x7f77d8ac9690 <__condvar_cleanup_waiting>, __arg = 0x7ffe2b32ff30, __canceltype = -852350944, __prev = 0x0}
        cbuffer = {wseq = 0, cond = 0x55cfcd3227f0, mutex = 0x55cfcd3227c8, private = 0}
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = <optimized out>
        seq = 0
        private = <optimized out>
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        rt = <optimized out>
        s = <optimized out>
#26 __pthread_cond_timedwait (cond=0x55cfcd3227f0, mutex=0x55cfcd3227c8, abstime=0x7ffe2b32ffe0) at pthread_cond_wait.c:667
No locals.
#27 0x00007f77c3401b80 in ?? ()
No symbol table info available.
#28 0x00007ffe2b32ffe0 in ?? ()
No symbol table info available.
#29 0x00002e07f698254e in ?? ()
No symbol table info available.
#30 0x00007ffe2b330044 in ?? ()
No symbol table info available.
#31 0x00007ffe2b330058 in ?? ()
No symbol table info available.
#32 0x0000000000000002 in ?? ()
No symbol table info available.
#33 0x0000000000000000 in ?? ()
No symbol table info available.
              1 Reply Last reply Reply Quote 0
              • Manfred
                Manfred Kopano last edited by

                Hi genesis74,

                I’m not the gdb expert, but I suppose you’ll have to install dbg packages for apache and php in order the dump to be helpful.

                Manfred

                1 Reply Last reply Reply Quote 0
                • genesis74
                  genesis74 last edited by

                  Hi @Manfred,

                  well, until you asked me if I could provide a core dump I never touched gdb. Tried to install dbg packages for apache and php - but that doesn’t seem to help. Moreover, apache is now throwing segfaults not only on shutdown but also during runtime. So I think I have to further shuffel with mods and packages hopefully finding out what the problem is. Thank you very much for your help. If I find a solution I will post it here.

                  1 Reply Last reply Reply Quote 0
                  • genesis74
                    genesis74 last edited by

                    I switched my apache server from mpm_prefork to php-fpm. This solved the segfault problem. But there are still errors on shutdown so the root cause still seems to be present. I am pretty sure that somehow it is connected to a problem with apache shutdown while z-push connections are open. This is what is noted in error.log on apache2 shutdown:

                    [Sun Feb 10 17:31:55.057810 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:31:57.060256 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:31:59.062464 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:32:01.064844 2019] [core:error] [pid 14957:tid 140151787768768] AH00046: child process 14959 still did not exit, sending a SIGKILL

                    The example above represents an apache shutdown while only one z-push connection was open. For every additional open z-push connection these lines appear with the respective PID. Maybe this is a hint where the problem might be. At least this is better than experiencing segfaults. But still I would prefer to find the cause for this.

                    1 Reply Last reply Reply Quote 0
                    • genesis74
                      genesis74 last edited by

                      Hi @Manfred,

                      may I come back to this issue … I returned to mpm_prefork and still searching for a solution for this. Found the following:

                      • on apache2 server status page all Z-Push connections are marked as ‚W‘ which means ‚sending reply‘
                      • when stopping apache2 with ‚systemctl stop apache2‘ the segfaults appear for every PID/connection in that ‚W‘ state
                      • when stopping apache2 with ‚apachectl graceful-stop‘ those connections are then marked as ‚G‘ (‚gracefully finishing‘) and apache2 needs a long time to shut down (did not check how many minutes yet - maybe connection lifetime?)

                      I am pretty sure that for some reason Z-Push connections cannot be closed gracefully and are held open even if apache2 is trying to close down (gracefully). When not using the graceful option connections are closed but cause a segfault.

                      Do you have any idea on that? Can anyone reproduce that or am I the only one with this problem? Is it normal behaviour that connections stay in the ‚W‘ state on apache even though it is just pinging the respective device?

                      1 Reply Last reply Reply Quote 0
                      • Manfred
                        Manfred Kopano last edited by

                        Hi genesis74,

                        the devices keep the Ping-connections open, maybe that’s the reason why their status is “W”. However I’m not an apache expert, I suggest you to contact the Kopano Support regarding this issue.

                        Manfred

                        1 Reply Last reply Reply Quote 0
                        • genesis74
                          genesis74 last edited by

                          Hi @Manfred,

                          I did not contact support yet. Too many possibilities I wanted to rule out. What I have done - just to keep you informed:

                          • Complete reinstall of Ubuntu 18.04.2 on a test system with reinstall of Kopano 8.6.9 as well as 8.7.0 with Z-Push 2.4.5
                          • Tested with PHP 7.3 - this included compiling and installing the latest Kopano community version in order to get mapi.so running

                          It is hard to believe that I am the only one with this problem since I built up the system from scratch ending with the same segmentation faults again. Is still no one here experiencing the same problems? Nobody using Ubuntu 18.04.2 as base for a Kopano system? Or the other way round: Anybody using Ubuntu 18.04.2 but not experiencing this issue?

                          1 Reply Last reply Reply Quote 0
                          • genesis74
                            genesis74 last edited by

                            With Debian 9, PHP 7.0, Apache 2.4.25 and same installation steps as with Ubuntu 18.04: No segmentation faults.

                            Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?

                            Manfred 1 Reply Last reply Reply Quote 0
                            • Manfred
                              Manfred Kopano @genesis74 last edited by

                              Hi genesis74,
                              @genesis74 said in Segmentation faults in apache2:

                              Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?

                              Are you referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions? I’ve updated the page to include Ubuntu 18.04 as well. Z-Push supports PHP 7.2 since version 2.4.3 and if I remember correctly that’s when we added Ubuntu 18.04 repository.

                              There are a couple of hundred Z-Push installations on Ubuntu 18.04, so I suppose if it wasn’t working, we would have heard by now.

                              If you want to be sure which distributions Z-Push supports, it’s better to check the repo page: http://repo.z-hub.io/z-push:/final/.

                              Greets, Manfred

                              1 Reply Last reply Reply Quote 0
                              • genesis74
                                genesis74 last edited by

                                Hi @Manfred,

                                yes, I was referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions. I was still trying to find the reason for these segfaults. While my system seems to be running fine under Debian I don’t believe it’s the PHP version that makes the difference. At least it did not make any difference when I installed PHP 7.0 or 7.3 on Ubuntu. I did not try to downgrade Apache. However, migration to Debian fixed the problem for me. A little bit dissatisfactory, but ok.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post