libkcclient.so causing Apache Segmentation fault



  • Date Seen
    16.10.2017

    Versions
    Ubuntu 16.04.3 LTS
    Kopano 8.4.90.492
    Apache 2.4.18
    MySQL 5.7.19

    Bug Description
    Since upgrading to version 8.4.90.492 i am constantly receiving the following error message:

    AH00051: child pid XXXX exit signal Segmentation fault (11), possible coredump in /var/cache/apache2/core
    

    Here is a Apache coredump:

    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Core was generated by `/usr/sbin/apache2 -k start'.
    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  0x00007f78c6ae4e88 in ?? () from /usr/lib/kopano/libkcclient.so
    [Current thread is 1 (Thread 0x7f78ee178780 (LWP 2017))]
    (gdb) bt
    #0  0x00007f78c6ae4e88 in ?? () from /usr/lib/kopano/libkcclient.so
    #1  0x00007f78c6ae3d9f in ?? () from /usr/lib/kopano/libkcclient.so
    #2  0x00007f78c6ae3f49 in ?? () from /usr/lib/kopano/libkcclient.so
    #3  0x00007f78d91f978b in KC::ECUnknown::Suicide() ()
       from /usr/lib/libkcmapi.so.0
    #4  0x00007f78d91f9b70 in KC::ECUnknown::Release() ()
       from /usr/lib/libkcmapi.so.0
    #5  0x00007f78c6a95fc4 in ?? () from /usr/lib/kopano/libkcclient.so
    #6  0x00007f78c6a96039 in ?? () from /usr/lib/kopano/libkcclient.so
    #7  0x00007f78d91f978b in KC::ECUnknown::Suicide() ()
       from /usr/lib/libkcmapi.so.0
    #8  0x00007f78d91f9b70 in KC::ECUnknown::Release() ()
       from /usr/lib/libkcmapi.so.0
    #9  0x00007f78d944b179 in ?? () from /usr/lib/libmapi.so.1
    #10 0x00007f78d94446ea in ?? () from /usr/lib/libmapi.so.1
    #11 0x00007f78d9444749 in ?? () from /usr/lib/libmapi.so.1
    #12 0x00007f78d943785a in ?? () from /usr/lib/libmapi.so.1
    #13 0x00007f78e8e7b341 in ?? () from /usr/lib/apache2/modules/libphp7.0.so
    #14 0x00007f78e8e7b3ca in list_entry_destructor ()
       from /usr/lib/apache2/modules/libphp7.0.so
    #15 0x00007f78e8e772a6 in zend_hash_index_del ()
       from /usr/lib/apache2/modules/libphp7.0.so
    #16 0x00007f78e8e9de7b in zend_object_std_dtor ()
    ---Type <return> to continue, or q <return> to quit---
       from /usr/lib/apache2/modules/libphp7.0.so
    #17 0x00007f78e8ea2f8a in zend_objects_store_del ()
       from /usr/lib/apache2/modules/libphp7.0.so
    #18 0x00007f78e8e59a33 in zend_cleanup_user_class_data ()
       from /usr/lib/apache2/modules/libphp7.0.so
    #19 0x00007f78e8e55d20 in shutdown_executor ()
       from /usr/lib/apache2/modules/libphp7.0.so
    #20 0x00007f78e8e660fb in zend_deactivate ()
       from /usr/lib/apache2/modules/libphp7.0.so
    #21 0x00007f78e8e04132 in php_request_shutdown ()
       from /usr/lib/apache2/modules/libphp7.0.so
    #22 0x00007f78e8efcb47 in ?? () from /usr/lib/apache2/modules/libphp7.0.so
    #23 0x0000561a6aff6c80 in ap_run_handler (r=r@entry=0x7f78edf970a0)
        at config.c:169
    #24 0x0000561a6aff7206 in ap_invoke_handler (r=r@entry=0x7f78edf970a0)
        at config.c:433
    #25 0x0000561a6b00e6a2 in ap_process_async_request (r=r@entry=0x7f78edf970a0)
        at http_request.c:410
    #26 0x0000561a6b00e850 in ap_process_request (r=r@entry=0x7f78edf970a0)
        at http_request.c:445
    #27 0x0000561a6b00a91e in ap_process_http_sync_connection (c=0x7f78ee089290)
        at http_core.c:210
    #28 ap_process_http_connection (c=0x7f78ee089290) at http_core.c:251
    ---Type <return> to continue, or q <return> to quit---
    #29 0x0000561a6b000bf0 in ap_run_process_connection (c=c@entry=0x7f78ee089290)
        at connection.c:41
    #30 0x0000561a6b001000 in ap_process_connection (c=c@entry=0x7f78ee089290,
        csd=<optimized out>) at connection.c:213
    #31 0x00007f78e94517e9 in child_main (child_num_arg=child_num_arg@entry=5,
        child_bucket=child_bucket@entry=0) at prefork.c:723
    #32 0x00007f78e9451a74 in make_child (s=0x7f78ee155470, slot=5, bucket=0)
        at prefork.c:824
    #33 0x00007f78e94528c5 in perform_idle_server_maintenance (p=<optimized out>)
        at prefork.c:932
    #34 prefork_run (_pconf=<optimized out>, plog=<optimized out>,
        s=<optimized out>) at prefork.c:1128
    #35 0x0000561a6afd946e in ap_run_mpm (pconf=0x7f78ee183028,
        plog=0x7f78ee150028, s=0x7f78ee155470) at mpm_common.c:94
    #36 0x0000561a6afd2400 in main (argc=3, argv=0x7ffeb39a6d38) at main.c:777
    

    Severity
    Major

    Steps to Reproduce
    I cannot reproduce the error, seems it has something todo with z-push.

    Actual Behavior
    Z-Push synchronization periodically stops

    Expected Behavior
    Z-Push synchronization is working, no erros are reported

    Troubleshooting/Testing Steps Attempted
    Apache coredump attached, a similar ticket was reported here --> https://jira.kopano.io/browse/KC-259

    Workaround
    No working workaround found yet.


  • Kopano

    Hello @darootler ,

    can zou provide a wbxml of around the time the segfault was triggered in apache?



  • Hi @fbartels

    I have to cut off the log because i cannot post all the lines due to forum limitations.

    Here is the wbxml log:

    Oct 19 13:26:23 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
    Oct 19 13:26:23 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
    Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:23 +0200] "GET / HTTP/1.1" 200 7871 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
    Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?service=fingerprint HTTP/1.1" 200 - "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
    Oct 19 13:26:24 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
    Oct 19 13:26:24 server kopano-server[1066]: message repeated 5 times: [ Error while connecting to search on "file:///var/run/kopano/search.sock"]
    Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?service=fingerprint&type=keepalive HTTP/1.1" 200 - "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
    Oct 19 13:26:24 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
    Oct 19 13:26:24 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
    Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 3020 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
    Oct 19 13:26:24 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:24 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 1247 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
    Oct 19 13:26:26 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
    Oct 19 13:26:26 server kopano-server[1066]: Error while connecting to search on "file:///var/run/kopano/search.sock"
    Oct 19 13:26:26 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:26 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 337 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
    Oct 19 13:26:44 server z-push/core[7680]: [INFO][USER] StatusException: SyncCollections->CheckForChanges(): Timeout forced after 42s from 600s due to other process - code: 3 - file: /usr/share/z-push/lib/core/synccollections.php:579
    Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] WBXMLEncoder->startWBXML() type: vnd.ms-sync.wbxml
    Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O  <Ping:Ping>
    Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O   <Ping:Status>
    Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O   1
    Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O   </Ping:Status>
    Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] O  </Ping:Ping>
    Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] WBXMLEncoder->endTag() WBXML output completed
    Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] WBXML-OUT: AwFqAAANRUcDMQABAQ==
    Oct 19 13:26:44 server z-push/core[7680]: [WBXML][USER] WBXML-IN :
    Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] LoopDetection->ProcessLoopDetectionTerminate()
    Oct 19 13:26:44 server z-push/core[7680]: [INFO][USER] cmd='Ping' memory='2.37 MiB/2.00 MiB' time='43.15s' devType='iPhone' devId='88imeq36lh03d2gbjkh272j5ko' getUser='USER' from='IPADDRESS' idle='42s' version='2.3.8+0-0' method='POST' httpcode='200'
    Oct 19 13:26:44 server z-push/core[7680]: [DEBUG][USER] -------- End
    Oct 19 13:26:44 server apache: [activesync.external.domain] IPADDRESS - USER [19/Oct/2017:13:26:01 +0200] "POST /Microsoft-Server-ActiveSync?User=USER&DeviceId=88IMEQ36LH03D2GBJKH272J5KO&DeviceType=iPhone&Cmd=Ping HTTP/1.1" 200 13 "-" "Apple-iPhone9C3/1501.402"
    Oct 19 13:26:54 server apache: [webmail.external.domain] 192.168.1.4 - USER@internal.domain [19/Oct/2017:13:26:54 +0200] "POST /kopano.php?subsystem=webapp_1508412384436 HTTP/1.1" 200 113 "https://webmail.external.domain/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
    Oct 19 13:26:55 server apache: [Thu Oct 19 13:26:55.867922 2017] [core:notice] [pid 7670] AH00051: child pid 7679 exit signal Segmentation fault (11), possible coredump in /etc/apache2
    

    I just recognized that there is a problem with kopano-search and kopano-presence too.

    Regards
    Richard


  • Kopano

    HI @darootler ,

    the segfault is logged 11 seconds after the last z-push log message. are you certain that the segfault and z-push are related?

    You could disable the z-push vhost to rule this one out (and afterwards do the same with the webapp).

    ps: if the log size is a problem you could still uploade it for example to https://pastebin.com/



  • Hi @fbartels

    no i am not sure that the segfault is related to z-push. It was just a guess. I also don’t know what “libkcclient.so” is for, that’s why i opened a thread here.

    Well i don’t want to disable z-push because i am using it.

    Regards
    Richard



  • Hi @fbartels,

    i disabled the kopano webapp vhost because i am using it very rarely.

    I‘ll report back.

    Regards
    Richard



  • Hi @fbartels,

    i just received another segfault from apache. So in my opinion this error is related to z-push because the webapp vhost was disabled.

    Regards
    Richard


  • Kopano

    I would say you should generate a coredump and send it to Kopano.
    The linked ticket is old, fixed & closed and probably not very helpful.



  • @sebastian said in libkcclient.so causing Apache Segmentation fault:

    I would say you should generate a coredump and send it to Kopano.
    The linked ticket is old, fixed & closed and probably not very helpful.

    Okay, how do send it to Kopano? Mail? Ticket?

    Regards
    Richard



  • I am not receiving this error since uprading to version 8.4.90.550.

    Regards
    Richard


Log in to reply
 

Looks like your connection to Kopano Community Forum was lost, please wait while we try to reconnect.