Gateway Process dies



  • Hi there,

    after an update of Kopano, I noticed the gateway-process die:

    Tue Jan 29 21:46:35 2019: [ZGateway IMAPs|T670] [error  ] Client disconnected
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [error  ] Previous message logged 25 times
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] ----------------------------------------------------------------------
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Fatal error detected. Please report all following information.
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Application kopano-gateway version: 8.7.80
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] OS: Linux, release: 4.15.0-39-generic, version: #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018, hardware: x86_64
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Thread name: NotifyThread
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Peak RSS: 560852
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Pid 2474 caught SIGABRT (6), out of memory or unhandled exception, traceback:
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Backtrace:
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x46940) [0x7f39b29b0940]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x3008d) [0x7f39b299a08d]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x19f) [0x7f39b299a33f]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f39b275d890]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f39b1a40e97]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f39b1a42801]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #6. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c8b7) [0x7f39b20978b7]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #7. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92a06) [0x7f39b209da06]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #8. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92a41) [0x7f39b209da41]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #9. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92c74) [0x7f39b209dc74]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #10. /usr/sbin/kopano-gateway(+0x11edf) [0x55f64118bedf]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #11. /usr/sbin/kopano-gateway(+0x2e7e8) [0x55f6411a87e8]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #12. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xacd3f) [0x7f39a7260d3f]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #13. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xb03fe) [0x7f39a72643fe]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #14. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f39b27526db]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] #15. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f39b1b2388f]
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Signal errno: Success, signal code: -6
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Sender pid: 2474, sender uid: 0, si_status: 0
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] Signal value: 0, faulting address: 0x9aa
    Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit   ] When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version.
    
    

    No memory issue, about 10GB free. Installed version: core-8.7.80.529-Ubuntu_18.04-amd64

    Any ideas?

    Thx & best
    Stephan



  • same here, happened at least once
    8.7.80 on Debian 9 “Stretch”

    codMon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] Pid 362 caught SIGABRT (6), out of memory or unhandled exception, traceback:
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] Backtrace:
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x487e0) [0x7f33870927e0]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x31d5d) [0x7f338707bd5d]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x1bd) [0x7f338707c03d]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f3386e3e0c0]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7f338393efff]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f338394042a]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #6. /lib/x86_64-linux-gnu/libc.so.6(+0x70c00) [0x7f338397cc00]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #7. /lib/x86_64-linux-gnu/libc.so.6(__libc_fatal+0x22) [0x7f338397cc42]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #8. /lib/x86_64-linux-gnu/libc.so.6(__netlink_assert_response+0x95) [0x7f3383a11315]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #9. /lib/x86_64-linux-gnu/libc.so.6(+0x104a02) [0x7f3383a10a02]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #10. /lib/x86_64-linux-gnu/libc.so.6(getaddrinfo+0x8c9) [0x7f33839e1aa9]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #11. /usr/lib/libgsoapssl++-kopano-2.8.74.so(+0x3063e) [0x7f337b64c63e]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #12. /usr/lib/libgsoapssl++-kopano-2.8.74.so(+0x2adf3) [0x7f337b646df3]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #13. /usr/lib/libgsoapssl++-kopano-2.8.74.so(soap_connect_command+0x12c) [0x7f337b64703c]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #14. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xdaadd) [0x7f337bc9badd]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #15. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xbbdee) [0x7f337bc7cdee]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #16. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0x9ab43) [0x7f337bc5bb43]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #17. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xa209a) [0x7f337bc6309a]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #18. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC9ECUnknown7SuicideEv+0x1b) [0x7f338708261b]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #19. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC9ECUnknown7ReleaseEv+0xaf) [0x7f3387082bef]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #20. /usr/sbin/kopano-gateway(+0xe5b8) [0x55c163be95b8]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #21. /usr/sbin/kopano-gateway(+0x155e6) [0x55c163bf05e6]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #22. /usr/sbin/kopano-gateway(+0x15909) [0x55c163bf0909]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #23. /usr/sbin/kopano-gateway(+0xcb92) [0x55c163be7b92]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #24. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f3386e34494]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] #25. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f33839f4acf]
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] Signal errno: Success, signal code: -6
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] Sender pid: 362, sender uid: 999, si_status: 0
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] Signal value: 0, faulting address: 0x3e70000016a
    Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit   ] When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version.
    e_text
    


  • @Stephan said in Gateway Process dies:

    Any ideas?

    i fixed this by : backup /etc/kopano

    remove kopano.
    Now go searching for leftovers of kopano.
    apt install mlocate && updatedb
    locate kopano

    Clean these leftovers up.

    install kopano

    and its fixed.

    Worked for me 3 times.



  • @thctlo said in Gateway Process dies:

    apt install mlocate && updatedb

    Thx a lot!

    Do you know anything about the root cause or a less invasive approach?

    I’m a bit nervous, uninstalling a production server ,-)



  • @Stephan

    The root of the problem, yes, if you have used the daily packages, ( the community packages), some of these have some left overs, multiple .py files, after removing kopano, the left overs needs to be removed.
    Then you can install them again.

    Just dont purge, and caputer all package which are removed.



  • @fbartels: will there an official fix for the gatewaycrash? We had this issue today for the second time.


  • Kopano

    @irrlicht I have not yet investigated this, but I doubt the gateway is crashing because of some leftover .py files.

    If you have a subscription I recommend to get in contact with our support so they can help you investigate.



  • quick&dirty-solution for Debian Stretch (systemd), hope it works

    /lib/systemd/system/kopano-gateway.service
    

    add:
    Restart=always

    [Unit]
    Description=Kopano Groupware Core IMAP/POP3 Gateway
    Documentation=man:kopano-gateway(8) man:kopano-gateway.cfg(5)
    
    [Service]
    Type=simple
    ExecStart=/usr/sbin/kopano-gateway -F
    ExecReload=/bin/kill -HUP $MAINPID
    Restart=always
    [Install]
    WantedBy=multi-user.target
    

    After edit:

    systemctl daemon-reload
    

    @fbartels: perhaps this would be a good default setting?


  • Kopano

    @irrlicht probably cannot hurt. If you want you can open a pr with your desired changes.

    You can find the file at https://github.com/Kopano-dev/kopano-core/blob/master/installer/linux/kopano-dagent.service
    The easiest is to open a pr on Github and we just pull this into our internal git. Alternatively you could follow the instructions at https://stash.kopano.io/projects/KC/repos/kopanocore/browse/CONTRIBUTING.md.



  • @irrlicht said in Gateway Process dies:

    uick&dirty-solution for Debian Stretch (systemd), hope it works

    If you go that way…

    systemctl edit kopano-gateway or systemctl edit --full kopano-gateway
    And use the override files.

    Dont touch the files in /lib/systemd/system/
    it’s not needed.



  • Fix is working, kopano-gateway crashed once again
    /var/log/syslog

    Feb  6 20:50:06 kopano_www_km22-01 kopano-gateway[21672]: Unexpected error 9 on netlink descriptor 99
    Feb  6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Main process exited, code=killed, status=6/ABRT
    Feb  6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Unit entered failed state.
    Feb  6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Failed with result 'signal'.
    Feb  6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Service hold-off time over, scheduling restart.
    Feb  6 20:50:06 kopano_www_km22-01 systemd[1]: Stopped Kopano Groupware Core IMAP/POP3 Gateway.
    
    

Log in to reply