kopano-server crashes under load with build core-8.6.80.1350_0+212-Ubuntu_16.04-amd64

Gerald

I tried updating from
core-8.6.80.865_0+123-Ubuntu_16.04-amd64
to
core-8.6.80.1350_0+212-Ubuntu_16.04-amd64

but kopano-server crashes as soon as it is being used (atfer 15 minutes max, probably much sooner depending on load)… I don’t know what user behavior causes it, but it can run without crashing at night, but as soon as someone is in the office it crashes. I’m using Office 2010 and Office 2016 clients as well as Z-Push (but mobile phones and ipad also sync during the night so they probably are not causing this)

Without debug logging, not much to see. With logging for kopano-server turned up to 6:

Aug 03 08:39:48 m.myhost.net kopano-server[2915]: No certificate in SSL connection.
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Accepted incoming SSL connection from 192.168.88.17
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: ----------------------------------------------------------------------
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Fatal error detected. Please report all following information.
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Application kopano-server version: 8.6.80
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: OS: Linux, release: 4.4.0-131-generic, version: #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018, hardware: x86_64
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Thread name: z-s: 192.168.88
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Peak RSS: 172596
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Pid 2915 caught SIGSEGV (11), traceback:
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Backtrace:
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x45890) [0x7f760ee91890]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x34d3d) [0x7f760ee80d3d]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x1bd) [                                                                                                          0x7f760ee8100d]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f760cc81390]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #4. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x8a244) [0x7f760eae4244]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #5. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN11KCmdService12tableFindRowEmjjjP13restrictTablePj+0x255) [0x7f760eb8084                                                                                                          5]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #6. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x3e8bf) [0x7f760ea988bf]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #7. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN11KCmdService8dispatchEP4soap+0x2537) [0x7f760ebf5607]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #8. /usr/sbin/kopano-server() [0x411e7e]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #9. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f760cc776ba]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: #10. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f760c41541d]
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Signal errno: Success, signal code: 1
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Sender pid: 8, sender uid: 0, si_status: -19076208
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: Signal value: -19076208, faulting address: 0x8
Aug 03 08:39:48 m.myhost.net kopano-server[2915]: When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano versio                                                                                                          n.
Aug 03 08:39:48 m.myhost.net systemd[1]: kopano-server.service: Main process exited, code=killed, status=11/SEGV
Aug 03 08:39:48 m.myhost.net systemd[1]: kopano-server.service: Unit entered failed state.
Aug 03 08:39:48 m.myhost.net systemd[1]: kopano-server.service: Failed with result 'signal'.

I don’t have these crashes on another smaller server with the same version, but this one has only VERY few users and no Outlook 2010 legacy clients… So it might be releated to those accessing the server?

tjoen

@Gerald
In Fedora-28 it doesn’t start
https://forum.kopano.io/topic/1522/db-backend-kopano-cli-mapi-error-80040111-mapi_e_logon_failed/10

Gerald

@tjoen
This does not look too similar to me. Can you please turn on debug logging when it crashes so we can see if the traceback looks similar or completely different?

tjoen

@gerald
I have only installed kopano on Fedora because of the MAPI-error. Confirmed with 1248. Failed testing on 1350 because server doesn’t start. But on my own system 1350 starts but gives MAPI-error

jengelh

This post is deleted!

jengelh

A fix should show up in 8.6.80.1426.

tjoen

kopanocore-8.6.80.1420-1.x86_64 running now, no crashes with simple tests
dagent works with some testmails sent bij telnet localhost 25 (Exim as MTA)
gateway works because I have received those mails in evolution
spooler not tested because using SMTP server. Folder sent items is empty

Gerald

Is compiling from source the only way to get Build 1420 or Build 1426?

On the FTP in the core directory: https://download.kopano.io/community/core:/
the files are updated daily but the version is stuck at Build 1350.

tjoen

I guess that with a subscription you can get access to the more private parts of the ftp-server
Compile it yourself only if you need to. As far as I can see you need the devel packages of gsoap, libkcoidc, libs3, tidy-html5, gss, libntlm, gsasl and vmime.
You also need to uninstall the old kopano-devel packages because otherwise it links to the old version

fbartels

Hi @Gerald ,

I have checked our build system and while the nightly release jobs did succeed over the last two weeks the initial job that triggers a new build seems to have gotten lost somewhere. I have triggered a new build manually so tomorrow there should be new packages and we are going to have a look why the initial trigger did get lost.

Gerald

@fbartels
Thank you, new build has appeared and I’m updating to it right now.

Gerald

May I bug you again about the daily builds @fbartels ?
No new builds since August 14th - 9 days ago…

Joost

@gerald said in kopano-server crashes under load with build core-8.6.80.1350_0+212-Ubuntu_16.04-amd64:

May I bug you again about the daily builds @fbartels ?
No new builds since August 14th - 9 days ago…

An upgrade issue in packaging was blocking new releases.

The issue should be fixed and a new release will be available as soon as the release-pipeline is completed successfully. (Expect new packages within the hour.)

- Joost