Segmentation faults in apache2

Manfred

Hi genesis74,

did you compare pids of apache segfaults with those in Z-Push log to check on which requests it happened?

Manfred

genesis74

the pids shown in the apache segfault errors always relate to Z-Push connections and their pids. I double checked that with apache server-status page as well as z-push-top and z-push.log (running on debug level). There are quite some other open connections on the apache server, none of them causing segfaults on apache shutdown. This happens only on apache shutdown. See the following z-push.log:

04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] refpolkey:'zzzzz', sent polkey:'zzzzz'
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] ASDevice->GetHierarchyCache(): HierarchyCache is up - Cached objects: 16
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] ExportChangesICS->Config() initialized with state: 0x1b1800004e0f2e00
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] ExportChangesICS->InitializeExporter() successfully. 5 changes ready to sync for 'hierarchy'.
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] MAPIProvider->GetStoreProps(): Getting store properties.
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] MAPIProvider->getInboxProps(): Getting inbox properties.
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] ChangesMemoryWrapper->ImportFolderChange(): Change for folder 'Entwürfe' will not be sent as modification is not relevant.
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] ChangesMemoryWrapper->ImportFolderChange(): Change for folder 'Gesendete Objekte' will not be sent as modification is not relevant.
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] ChangesMemoryWrapper->ImportFolderChange(): Change for folder 'Gelöschte Objekte' will not be sent as modification is not relevant.
04/02/2019 14:53:49 [ 2868] [DEBUG] [user3] ChangesMemoryWrapper->ImportFolderChange(): Change for folder 'Posteingang' will not be sent as modification is not relevant.
04/02/2019 14:54:14 [30319] [ INFO] [user2] StatusException: SyncCollections->CheckForChanges(): Timeout forced after 630s from 860s due to other process - code: 3 - file: /usr/share/z-push/lib/core/synccollections.php:579
04/02/2019 14:54:14 [30319] [DEBUG] [user2] WBXMLEncoder->startWBXML() type: vnd.ms-sync.wbxml
04/02/2019 14:54:14 [30319] [DEBUG] [user2] WBXMLEncoder->endTag() WBXML output completed
04/02/2019 14:54:14 [30319] [DEBUG] [user2] LoopDetection->ProcessLoopDetectionTerminate()
04/02/2019 14:54:14 [30319] [ INFO] [user2] cmd='Ping' memory='6.01 MiB/2.00 MiB' time='630.27s' devType='iPad' devId='yyyyyyyyyyy' getUser='user2' from='xx.xx.xx.xx' idle='630s' version='2.4.5+0-0' method='POST' httpcode='200'
04/02/2019 14:54:14 [30319] [DEBUG] [user2] -------- End
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] -------- Start
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] cmd='Ping' devType='iPhone' devId='xxxxxxxx' getUser='user1' from='xx.xx.xx.xx' version='2.4.5+0-0' method='POST'
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] Used timezone 'Europe/Berlin'
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] ZPush::GetBackend(): trying autoload backend 'BackendKopano'
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] BackendKopano using PHP-MAPI version: 8.7.0 - PHP version: 7.2.10-0ubuntu0.18.04.1
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] SqlStateMachine(): init
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] SqlStateMachine->checkDbAndTables(): Database and tables exist.
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] SqlStateMachine->GetStateVersion(): supporting version '2'
04/02/2019 14:54:55 [ 3521] [DEBUG] [user1] Request::ProcessHeaders() ASVersion: 14.1

apache2 error.log at time of shutdown/restart (14:54:54):

[Mon Feb 04 14:54:54.435083 2019] [core:notice] [pid 17839] AH00051: child pid 3051 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Mon Feb 04 14:54:54.435242 2019] [core:notice] [pid 17839] AH00051: child pid 615 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Mon Feb 04 14:54:54.435273 2019] [core:notice] [pid 17839] AH00051: child pid 2868 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Mon Feb 04 14:54:54.435305 2019] [core:notice] [pid 17839] AH00051: child pid 633 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Mon Feb 04 14:54:54.435323 2019] [core:notice] [pid 17839] AH00051: child pid 2028 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Mon Feb 04 14:54:54.435344 2019] [core:notice] [pid 17839] AH00051: child pid 1795 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Mon Feb 04 14:54:54.435385 2019] [core:notice] [pid 17839] AH00051: child pid 26160 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Mon Feb 04 14:54:54.435430 2019] [core:info] [pid 17839] AH00096: removed PID file /var/run/apache2/apache2.pid (pid=17839)
[Mon Feb 04 14:54:54.435436 2019] [mpm_prefork:notice] [pid 17839] AH00169: caught SIGTERM, shutting down

Note in this case pid 2868. All other pids exiting with segfault shown in apache2 error.log can be found earlier in the z-push.log as well - I didn’t quote them because of the log size.

I have been reinstalling z-push, apache2, php, most of the php-modules and tweaking several php parameters as proposed on the net with no help.

Manfred

Hi genesis74,

do the segfaults happen only on apache shutdown? Do all Z-Push connections cause a segfault on shutdown or only some? Are there any when apache is running? Did you generate apache core dumps? What apache mods do you have enabled?

Manfred

genesis74

Hi @Manfred,

segfaults only happening on apache shutdown
yes, I am pretty sure that all Z-Push connections cause a segfault on shutdown; at least during several tests
no, segfaults only appear on apache shutdown, not when it is running
no, until now I did not generate core dumps
following apache modules are enabled:
- core_module (static)
- so_module (static)
- watchdog_module (static)
- http_module (static)
- log_config_module (static)
- logio_module (static)
- version_module (static)
- unixd_module (static)
- access_compat_module (shared)
- alias_module (shared)
- auth_basic_module (shared)
- authn_core_module (shared)
- authn_file_module (shared)
- authz_core_module (shared)
- authz_host_module (shared)
- authz_user_module (shared)
- autoindex_module (shared)
- cgi_module (shared)
- deflate_module (shared)
- dir_module (shared)
- env_module (shared)
- filter_module (shared)
- headers_module (shared)
- mime_module (shared)
- mpm_prefork_module (shared)
- negotiation_module (shared)
- php7_module (shared)
- reqtimeout_module (shared)
- rewrite_module (shared)
- setenvif_module (shared)
- socache_shmcb_module (shared)
- ssl_module (shared)
- status_module (shared)

Manfred

Hi genesis74,

could you provide us a core dump?

Manfred

genesis74

Hi @Manfred,

I am sorry but I seem to be unable to generate a core dump. Tried for hours now. Giving up. Don’t want to get off-topic here with discussing how core dump can be generated but maybe you can provide me with a step-by-step guide. Followed the official apache way (/usr/share/doc/apache2/README.backtrace) and tried nearly everything I found on the net with no help.

Manfred

Hi genesis74,

If I remember correctly the last time I needed a core dump I followed those links: https://serverfault.com/questions/470407/how-to-get-a-core-dump-from-apache-when-segfaulting/470410#470410 and https://askubuntu.com/questions/612532/how-to-get-an-apache-core-dump .

Alternatively you could start apache2 in single thread mode, attach with gdb to the process and then force for the segfault: https://httpd.apache.org/dev/debugging.html.

Manfred

genesis74

Hi @Manfred,

appreciate your help. Those links you posted were part of my unsuccessful tries yesterday. Today I found this instruction which finally led to core dumps being produced. I suspect the most important part is to set /proc/sys/kernel/core_pattern correctly (was pointing to apport in my Ubuntu 18.04 server environment).

However, here is the gdb output:

(gdb) bt full
#0  0x00007f77bf0a6ca0 in ?? ()
No symbol table info available.
#1  0x00007f77d53dbdda in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#2  0x00007f77d53dbe5a in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#3  0x00007f77d53d7b26 in zend_hash_index_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#4  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#5  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#6  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#7  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#8  0x00007f77d54001e5 in zend_object_std_dtor () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#9  0x00007f77d54057f7 in zend_objects_store_del () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#10 0x00007f77d53c41fd in _zval_dtor_func () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#11 0x00007f77d53b8e5a in destroy_zend_class () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#12 0x00007f77d53d7e44 in zend_hash_destroy () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#13 0x00007f77d53c5fb1 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#14 0x00007f77d536210b in php_module_shutdown () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#15 0x00007f77d53621c9 in php_module_shutdown_wrapper () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#16 0x00007f77d5478971 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#17 0x00007f77d8cf7e2e in run_cleanups (cref=<optimized out>) at ./memory/unix/apr_pools.c:2629
        c = <optimized out>
        c = <optimized out>
#18 apr_pool_destroy (pool=0x7f77d9493028) at ./memory/unix/apr_pools.c:1000
        active = <optimized out>
        allocator = <optimized out>
#19 0x00007f77d59e4283 in clean_child_exit (code=code@entry=0) at prefork.c:227
No locals.
#20 0x00007f77d59e42bb in just_die (sig=<optimized out>) at prefork.c:355
No locals.
#21 0x00007f77d53f298c in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#22 0x00007f77d53f2b62 in ?? () from /usr/lib/apache2/modules/libphp7.2.so
No symbol table info available.
#23 <signal handler called>
No locals.
#24 0x00007f77d8ac9f85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ffe2b32ffe0, expected=0, futex_word=0x55cfcd322818) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        __ret = -4
        oldtype = 0
        err = <optimized out>
        oldtype = <optimized out>
        err = <optimized out>
        __ret = <optimized out>
        resultvar = <optimized out>
        __arg6 = <optimized out>
        __arg5 = <optimized out>
        __arg4 = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a6 = <optimized out>
        _a5 = <optimized out>
        _a4 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
#25 __pthread_cond_wait_common (abstime=0x7ffe2b32ffe0, mutex=0x55cfcd3227c8, cond=0x55cfcd3227f0) at pthread_cond_wait.c:539
        spin = 0
        buffer = {__routine = 0x7f77d8ac9690 <__condvar_cleanup_waiting>, __arg = 0x7ffe2b32ff30, __canceltype = -852350944, __prev = 0x0}
        cbuffer = {wseq = 0, cond = 0x55cfcd3227f0, mutex = 0x55cfcd3227c8, private = 0}
        err = <optimized out>
        g = 0
        flags = <optimized out>
        g1_start = <optimized out>
        maxspin = 0
        signals = <optimized out>
        result = 0
        wseq = <optimized out>
        seq = 0
        private = <optimized out>
        maxspin = <optimized out>
        err = <optimized out>
        result = <optimized out>
        wseq = <optimized out>
        g = <optimized out>
        seq = <optimized out>
        flags = <optimized out>
        private = <optimized out>
        signals = <optimized out>
        g1_start = <optimized out>
        spin = <optimized out>
        buffer = <optimized out>
        cbuffer = <optimized out>
        rt = <optimized out>
        s = <optimized out>
#26 __pthread_cond_timedwait (cond=0x55cfcd3227f0, mutex=0x55cfcd3227c8, abstime=0x7ffe2b32ffe0) at pthread_cond_wait.c:667
No locals.
#27 0x00007f77c3401b80 in ?? ()
No symbol table info available.
#28 0x00007ffe2b32ffe0 in ?? ()
No symbol table info available.
#29 0x00002e07f698254e in ?? ()
No symbol table info available.
#30 0x00007ffe2b330044 in ?? ()
No symbol table info available.
#31 0x00007ffe2b330058 in ?? ()
No symbol table info available.
#32 0x0000000000000002 in ?? ()
No symbol table info available.
#33 0x0000000000000000 in ?? ()
No symbol table info available.

Manfred

Hi genesis74,

I’m not the gdb expert, but I suppose you’ll have to install dbg packages for apache and php in order the dump to be helpful.

Manfred

genesis74

Hi @Manfred,

well, until you asked me if I could provide a core dump I never touched gdb. Tried to install dbg packages for apache and php - but that doesn’t seem to help. Moreover, apache is now throwing segfaults not only on shutdown but also during runtime. So I think I have to further shuffel with mods and packages hopefully finding out what the problem is. Thank you very much for your help. If I find a solution I will post it here.

genesis74

I switched my apache server from mpm_prefork to php-fpm. This solved the segfault problem. But there are still errors on shutdown so the root cause still seems to be present. I am pretty sure that somehow it is connected to a problem with apache shutdown while z-push connections are open. This is what is noted in error.log on apache2 shutdown:

[Sun Feb 10 17:31:55.057810 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:31:57.060256 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:31:59.062464 2019] [core:warn] [pid 14957:tid 140151787768768] AH00045: child process 14959 still did not exit, sending a SIGTERM
[Sun Feb 10 17:32:01.064844 2019] [core:error] [pid 14957:tid 140151787768768] AH00046: child process 14959 still did not exit, sending a SIGKILL

The example above represents an apache shutdown while only one z-push connection was open. For every additional open z-push connection these lines appear with the respective PID. Maybe this is a hint where the problem might be. At least this is better than experiencing segfaults. But still I would prefer to find the cause for this.

genesis74

Hi @Manfred,

may I come back to this issue … I returned to mpm_prefork and still searching for a solution for this. Found the following:

on apache2 server status page all Z-Push connections are marked as ‚W‘ which means ‚sending reply‘
when stopping apache2 with ‚systemctl stop apache2‘ the segfaults appear for every PID/connection in that ‚W‘ state
when stopping apache2 with ‚apachectl graceful-stop‘ those connections are then marked as ‚G‘ (‚gracefully finishing‘) and apache2 needs a long time to shut down (did not check how many minutes yet - maybe connection lifetime?)

I am pretty sure that for some reason Z-Push connections cannot be closed gracefully and are held open even if apache2 is trying to close down (gracefully). When not using the graceful option connections are closed but cause a segfault.

Do you have any idea on that? Can anyone reproduce that or am I the only one with this problem? Is it normal behaviour that connections stay in the ‚W‘ state on apache even though it is just pinging the respective device?

Manfred

Hi genesis74,

the devices keep the Ping-connections open, maybe that’s the reason why their status is “W”. However I’m not an apache expert, I suggest you to contact the Kopano Support regarding this issue.

Manfred

genesis74

Hi @Manfred,

I did not contact support yet. Too many possibilities I wanted to rule out. What I have done - just to keep you informed:

Complete reinstall of Ubuntu 18.04.2 on a test system with reinstall of Kopano 8.6.9 as well as 8.7.0 with Z-Push 2.4.5
Tested with PHP 7.3 - this included compiling and installing the latest Kopano community version in order to get mapi.so running

It is hard to believe that I am the only one with this problem since I built up the system from scratch ending with the same segmentation faults again. Is still no one here experiencing the same problems? Nobody using Ubuntu 18.04.2 as base for a Kopano system? Or the other way round: Anybody using Ubuntu 18.04.2 but not experiencing this issue?

genesis74

With Debian 9, PHP 7.0, Apache 2.4.25 and same installation steps as with Ubuntu 18.04: No segmentation faults.

Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?

Manfred

Hi genesis74,
@genesis74 said in Segmentation faults in apache2:

Kopano claims to be tested with Ubuntu 18.04. How about Z-Push? According to the wiki the latest supported Ubuntu Version is 16.04. Is this still valid?

Are you referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions? I’ve updated the page to include Ubuntu 18.04 as well. Z-Push supports PHP 7.2 since version 2.4.3 and if I remember correctly that’s when we added Ubuntu 18.04 repository.

There are a couple of hundred Z-Push installations on Ubuntu 18.04, so I suppose if it wasn’t working, we would have heard by now.

If you want to be sure which distributions Z-Push supports, it’s better to check the repo page: http://repo.z-hub.io/z-push:/final/.

Greets, Manfred

genesis74

Hi @Manfred,

yes, I was referring to https://wiki.z-hub.io/display/ZP/Installation#Installation-Distributions. I was still trying to find the reason for these segfaults. While my system seems to be running fine under Debian I don’t believe it’s the PHP version that makes the difference. At least it did not make any difference when I installed PHP 7.0 or 7.3 on Ubuntu. I did not try to downgrade Apache. However, migration to Debian fixed the problem for me. A little bit dissatisfactory, but ok.