Gateway Process dies
-
Hi there,
after an update of Kopano, I noticed the gateway-process die:
Tue Jan 29 21:46:35 2019: [ZGateway IMAPs|T670] [error ] Client disconnected Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [error ] Previous message logged 25 times Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] ---------------------------------------------------------------------- Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Fatal error detected. Please report all following information. Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Application kopano-gateway version: 8.7.80 Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] OS: Linux, release: 4.15.0-39-generic, version: #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018, hardware: x86_64 Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Thread name: NotifyThread Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Peak RSS: 560852 Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Pid 2474 caught SIGABRT (6), out of memory or unhandled exception, traceback: Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Backtrace: Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x46940) [0x7f39b29b0940] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x3008d) [0x7f39b299a08d] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x19f) [0x7f39b299a33f] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f39b275d890] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f39b1a40e97] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f39b1a42801] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #6. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c8b7) [0x7f39b20978b7] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #7. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92a06) [0x7f39b209da06] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #8. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92a41) [0x7f39b209da41] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #9. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92c74) [0x7f39b209dc74] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #10. /usr/sbin/kopano-gateway(+0x11edf) [0x55f64118bedf] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #11. /usr/sbin/kopano-gateway(+0x2e7e8) [0x55f6411a87e8] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #12. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xacd3f) [0x7f39a7260d3f] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #13. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xb03fe) [0x7f39a72643fe] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #14. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f39b27526db] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] #15. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f39b1b2388f] Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Signal errno: Success, signal code: -6 Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Sender pid: 2474, sender uid: 0, si_status: 0 Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] Signal value: 0, faulting address: 0x9aa Tue Jan 29 22:40:33 2019: [NotifyThread|T2164] [crit ] When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version.
No memory issue, about 10GB free. Installed version: core-8.7.80.529-Ubuntu_18.04-amd64
Any ideas?
Thx & best
Stephan -
same here, happened at least once
8.7.80 on Debian 9 “Stretch”codMon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] Pid 362 caught SIGABRT (6), out of memory or unhandled exception, traceback: Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] Backtrace: Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x487e0) [0x7f33870927e0] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x31d5d) [0x7f338707bd5d] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x1bd) [0x7f338707c03d] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f3386e3e0c0] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7f338393efff] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f338394042a] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #6. /lib/x86_64-linux-gnu/libc.so.6(+0x70c00) [0x7f338397cc00] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #7. /lib/x86_64-linux-gnu/libc.so.6(__libc_fatal+0x22) [0x7f338397cc42] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #8. /lib/x86_64-linux-gnu/libc.so.6(__netlink_assert_response+0x95) [0x7f3383a11315] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #9. /lib/x86_64-linux-gnu/libc.so.6(+0x104a02) [0x7f3383a10a02] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #10. /lib/x86_64-linux-gnu/libc.so.6(getaddrinfo+0x8c9) [0x7f33839e1aa9] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #11. /usr/lib/libgsoapssl++-kopano-2.8.74.so(+0x3063e) [0x7f337b64c63e] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #12. /usr/lib/libgsoapssl++-kopano-2.8.74.so(+0x2adf3) [0x7f337b646df3] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #13. /usr/lib/libgsoapssl++-kopano-2.8.74.so(soap_connect_command+0x12c) [0x7f337b64703c] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #14. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xdaadd) [0x7f337bc9badd] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #15. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xbbdee) [0x7f337bc7cdee] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #16. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0x9ab43) [0x7f337bc5bb43] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #17. /usr/lib/x86_64-linux-gnu/kopano/libkcclient.so(+0xa209a) [0x7f337bc6309a] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #18. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC9ECUnknown7SuicideEv+0x1b) [0x7f338708261b] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #19. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC9ECUnknown7ReleaseEv+0xaf) [0x7f3387082bef] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #20. /usr/sbin/kopano-gateway(+0xe5b8) [0x55c163be95b8] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #21. /usr/sbin/kopano-gateway(+0x155e6) [0x55c163bf05e6] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #22. /usr/sbin/kopano-gateway(+0x15909) [0x55c163bf0909] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #23. /usr/sbin/kopano-gateway(+0xcb92) [0x55c163be7b92] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #24. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f3386e34494] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] #25. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f33839f4acf] Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] Signal errno: Success, signal code: -6 Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] Sender pid: 362, sender uid: 999, si_status: 0 Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] Signal value: 0, faulting address: 0x3e70000016a Mon Jan 28 18:51:47 2019: [ZGateway IMAP|T4910] [crit ] When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version. e_text
-
@Stephan said in Gateway Process dies:
Any ideas?
i fixed this by : backup /etc/kopano
remove kopano.
Now go searching for leftovers of kopano.
apt install mlocate && updatedb
locate kopanoClean these leftovers up.
install kopano
and its fixed.
Worked for me 3 times.
-
@thctlo said in Gateway Process dies:
apt install mlocate && updatedb
Thx a lot!
Do you know anything about the root cause or a less invasive approach?
I’m a bit nervous, uninstalling a production server ,-)
-
The root of the problem, yes, if you have used the daily packages, ( the community packages), some of these have some left overs, multiple .py files, after removing kopano, the left overs needs to be removed.
Then you can install them again.Just dont purge, and caputer all package which are removed.
-
@fbartels: will there an official fix for the gatewaycrash? We had this issue today for the second time.
-
@irrlicht I have not yet investigated this, but I doubt the gateway is crashing because of some leftover .py files.
If you have a subscription I recommend to get in contact with our support so they can help you investigate.
-
quick&dirty-solution for Debian Stretch (systemd), hope it works
/lib/systemd/system/kopano-gateway.service
add:
Restart=always[Unit] Description=Kopano Groupware Core IMAP/POP3 Gateway Documentation=man:kopano-gateway(8) man:kopano-gateway.cfg(5) [Service] Type=simple ExecStart=/usr/sbin/kopano-gateway -F ExecReload=/bin/kill -HUP $MAINPID Restart=always [Install] WantedBy=multi-user.target
After edit:
systemctl daemon-reload
@fbartels: perhaps this would be a good default setting?
-
@irrlicht probably cannot hurt. If you want you can open a pr with your desired changes.
You can find the file at https://github.com/Kopano-dev/kopano-core/blob/master/installer/linux/kopano-dagent.service
The easiest is to open a pr on Github and we just pull this into our internal git. Alternatively you could follow the instructions at https://stash.kopano.io/projects/KC/repos/kopanocore/browse/CONTRIBUTING.md. -
@irrlicht said in Gateway Process dies:
uick&dirty-solution for Debian Stretch (systemd), hope it works
If you go that way…
systemctl edit kopano-gateway
orsystemctl edit --full kopano-gateway
And use the override files.Dont touch the files in
/lib/systemd/system/
it’s not needed. -
Fix is working, kopano-gateway crashed once again
/var/log/syslogFeb 6 20:50:06 kopano_www_km22-01 kopano-gateway[21672]: Unexpected error 9 on netlink descriptor 99 Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Main process exited, code=killed, status=6/ABRT Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Unit entered failed state. Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Failed with result 'signal'. Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Service hold-off time over, scheduling restart. Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: Stopped Kopano Groupware Core IMAP/POP3 Gateway.