libkcclient.so causing Apache Segmentation fault

Date Seen
16.10.2017

Versions
Ubuntu 16.04.3 LTS
Kopano 8.4.90.492
Apache 2.4.18
MySQL 5.7.19

Bug Description
Since upgrading to version 8.4.90.492 i am constantly receiving the following error message:

AH00051: child pid XXXX exit signal Segmentation fault (11), possible coredump in /var/cache/apache2/core

Here is a Apache coredump:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f78c6ae4e88 in ?? () from /usr/lib/kopano/libkcclient.so
[Current thread is 1 (Thread 0x7f78ee178780 (LWP 2017))]
(gdb) bt
#0  0x00007f78c6ae4e88 in ?? () from /usr/lib/kopano/libkcclient.so
#1  0x00007f78c6ae3d9f in ?? () from /usr/lib/kopano/libkcclient.so
#2  0x00007f78c6ae3f49 in ?? () from /usr/lib/kopano/libkcclient.so
#3  0x00007f78d91f978b in KC::ECUnknown::Suicide() ()
   from /usr/lib/libkcmapi.so.0
#4  0x00007f78d91f9b70 in KC::ECUnknown::Release() ()
   from /usr/lib/libkcmapi.so.0
#5  0x00007f78c6a95fc4 in ?? () from /usr/lib/kopano/libkcclient.so
#6  0x00007f78c6a96039 in ?? () from /usr/lib/kopano/libkcclient.so
#7  0x00007f78d91f978b in KC::ECUnknown::Suicide() ()
   from /usr/lib/libkcmapi.so.0
#8  0x00007f78d91f9b70 in KC::ECUnknown::Release() ()
   from /usr/lib/libkcmapi.so.0
#9  0x00007f78d944b179 in ?? () from /usr/lib/libmapi.so.1
#10 0x00007f78d94446ea in ?? () from /usr/lib/libmapi.so.1
#11 0x00007f78d9444749 in ?? () from /usr/lib/libmapi.so.1
#12 0x00007f78d943785a in ?? () from /usr/lib/libmapi.so.1
#13 0x00007f78e8e7b341 in ?? () from /usr/lib/apache2/modules/libphp7.0.so
#14 0x00007f78e8e7b3ca in list_entry_destructor ()
   from /usr/lib/apache2/modules/libphp7.0.so
#15 0x00007f78e8e772a6 in zend_hash_index_del ()
   from /usr/lib/apache2/modules/libphp7.0.so
#16 0x00007f78e8e9de7b in zend_object_std_dtor ()
---Type <return> to continue, or q <return> to quit---
   from /usr/lib/apache2/modules/libphp7.0.so
#17 0x00007f78e8ea2f8a in zend_objects_store_del ()
   from /usr/lib/apache2/modules/libphp7.0.so
#18 0x00007f78e8e59a33 in zend_cleanup_user_class_data ()
   from /usr/lib/apache2/modules/libphp7.0.so
#19 0x00007f78e8e55d20 in shutdown_executor ()
   from /usr/lib/apache2/modules/libphp7.0.so
#20 0x00007f78e8e660fb in zend_deactivate ()
   from /usr/lib/apache2/modules/libphp7.0.so
#21 0x00007f78e8e04132 in php_request_shutdown ()
   from /usr/lib/apache2/modules/libphp7.0.so
#22 0x00007f78e8efcb47 in ?? () from /usr/lib/apache2/modules/libphp7.0.so
#23 0x0000561a6aff6c80 in ap_run_handler (r=r@entry=0x7f78edf970a0)
    at config.c:169
#24 0x0000561a6aff7206 in ap_invoke_handler (r=r@entry=0x7f78edf970a0)
    at config.c:433
#25 0x0000561a6b00e6a2 in ap_process_async_request (r=r@entry=0x7f78edf970a0)
    at http_request.c:410
#26 0x0000561a6b00e850 in ap_process_request (r=r@entry=0x7f78edf970a0)
    at http_request.c:445
#27 0x0000561a6b00a91e in ap_process_http_sync_connection (c=0x7f78ee089290)
    at http_core.c:210
#28 ap_process_http_connection (c=0x7f78ee089290) at http_core.c:251
---Type <return> to continue, or q <return> to quit---
#29 0x0000561a6b000bf0 in ap_run_process_connection (c=c@entry=0x7f78ee089290)
    at connection.c:41
#30 0x0000561a6b001000 in ap_process_connection (c=c@entry=0x7f78ee089290,
    csd=<optimized out>) at connection.c:213
#31 0x00007f78e94517e9 in child_main (child_num_arg=child_num_arg@entry=5,
    child_bucket=child_bucket@entry=0) at prefork.c:723
#32 0x00007f78e9451a74 in make_child (s=0x7f78ee155470, slot=5, bucket=0)
    at prefork.c:824
#33 0x00007f78e94528c5 in perform_idle_server_maintenance (p=<optimized out>)
    at prefork.c:932
#34 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
    s=<optimized out>) at prefork.c:1128
#35 0x0000561a6afd946e in ap_run_mpm (pconf=0x7f78ee183028,
    plog=0x7f78ee150028, s=0x7f78ee155470) at mpm_common.c:94
#36 0x0000561a6afd2400 in main (argc=3, argv=0x7ffeb39a6d38) at main.c:777

Severity
Major

Steps to Reproduce
I cannot reproduce the error, seems it has something todo with z-push.

Actual Behavior
Z-Push synchronization periodically stops

Expected Behavior
Z-Push synchronization is working, no erros are reported

Troubleshooting/Testing Steps Attempted
Apache coredump attached, a similar ticket was reported here --> https://jira.kopano.io/browse/KC-259

Workaround
No working workaround found yet.

Hello @darootler ,

can zou provide a wbxml of around the time the segfault was triggered in apache?

Hi @fbartels

I have to cut off the log because i cannot post all the lines due to forum limitations.

Here is the wbxml log:

Oct 19 13:26:23 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
Oct 19 13:26:23 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:23 +0200] "GET / HTTP/1.1" 200 7871 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?service=fingerprint HTTP/1.1" 200 - "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Oct 19 13:26:24 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
Oct 19 13:26:24 server kopano-server[1066]: message repeated 5 times: [ Error while connecting to search on "file:///var/run/kopano/search.sock"]
Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?service=fingerprint&type=keepalive HTTP/1.1" 200 - "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Oct 19 13:26:24 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
Oct 19 13:26:24 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 3020 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 1247 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Oct 19 13:26:26 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
Oct 19 13:26:26 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
Oct 19 13:26:26 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:26 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 337 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Oct 19 13:26:44 server z-push/core[7680]: [INFO][USER] StatusException: SyncCollections->CheckForChanges(): Timeout forced after 42s from 600s due to other process - code: 3 - file: /usr/share/z-push/lib/core/synccollections.php:579
Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] WBXMLEncoder->startWBXML() type: vnd.ms-sync.wbxml
Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O  <Ping:Ping>
Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O   <Ping:Status>
Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O   1
Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O   </Ping:Status>
Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O  </Ping:Ping>
Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] WBXMLEncoder->endTag() WBXML output completed
Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] WBXML-OUT: AwFqAAANRUcDMQABAQ==
Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] WBXML-IN :
Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] LoopDetection->ProcessLoopDetectionTerminate()
Oct 19 13:26:44 server z-push/core[7680]: [INFO][USER] cmd='Ping' memory='2.37 MiB/2.00 MiB' time='43.15s' devType='iPhone' devId='88imeq36lh03d2gbjkh272j5ko' getUser='USER' from='IPADDRESS' idle='42s' version='2.3.8+0-0' method='POST' httpcode='200'
Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] -------- End
Oct 19 13:26:44 server apache: [activesync.external.domain] IPADDRESS - USER [19/Oct/2017:13:26:01 +0200] "POST /Microsoft-Server-ActiveSync?User=USER&DeviceId=88IMEQ36LH03D2GBJKH272J5KO&DeviceType=iPhone&Cmd=Ping HTTP/1.1" 200 13 "-" "Apple-iPhone9C3/1501.402"
Oct 19 13:26:54 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:54 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 113 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Oct 19 13:26:55 server apache: [Thu Oct 19 13:26:55.867922 2017] [core:notice] [pid 7670] AH00051: child pid 7679 exit signal Segmentation fault (11), possible coredump in /etc/apache2

I just recognized that there is a problem with kopano-search and kopano-presence too.

Regards
Richard

HI @darootler ,

the segfault is logged 11 seconds after the last z-push log message. are you certain that the segfault and z-push are related?

You could disable the z-push vhost to rule this one out (and afterwards do the same with the webapp).

ps: if the log size is a problem you could still uploade it for example to https://pastebin.com/

Hi @fbartels

no i am not sure that the segfault is related to z-push. It was just a guess. I also don’t know what “libkcclient.so” is for, that’s why i opened a thread here.

Well i don’t want to disable z-push because i am using it.

Regards
Richard

Hi @fbartels,

i disabled the kopano webapp vhost because i am using it very rarely.

I‘ll report back.

Regards
Richard

Hi @fbartels,

i just received another segfault from apache. So in my opinion this error is related to z-push because the webapp vhost was disabled.

Regards
Richard

I would say you should generate a coredump and send it to Kopano.
The linked ticket is old, fixed & closed and probably not very helpful.

@sebastian said in libkcclient.so causing Apache Segmentation fault:

I would say you should generate a coredump and send it to Kopano.
The linked ticket is old, fixed & closed and probably not very helpful.

Okay, how do send it to Kopano? Mail? Ticket?

Regards
Richard

I am not receiving this error since uprading to version 8.4.90.550.

Regards
Richard