Kopano-server crashes
-
Just made an update to the actual nightly version:
root@kopano:~# apt-cache policy kopano-server kopano-server: Installiert: 8.6.81.228-0+37.1 Installationskandidat: 8.6.81.228-0+37.1 Versionstabelle: *** 8.6.81.228-0+37.1 100 100 /var/lib/dpkg/statusand:
root@kopano:~# dpkg -l | grep kopano ii kopano-backup 8.6.81.228-0+37.1 amd64 Utility to back up and restore Kopano stores ii kopano-bash-completion 8.6.81.228-0+37.1 amd64 bash TAB completion for Kopano Core command-line utilities ii kopano-client 8.6.81.228-0+37.1 amd64 Kopano MAPI provider library ii kopano-client-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-client ii kopano-common 8.6.81.228-0+37.1 amd64 Shared files for Kopano services ii kopano-contacts 8.6.81.228-0+37.1 amd64 MAPI provider adding contact folders in the addressbook ii kopano-dagent 8.6.81.228-0+37.1 amd64 E-Mail Delivery Agent for Kopano Core ii kopano-dagent-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-dagent ii kopano-dbg 8.6.81.228-0+37.1 amd64 Debug symbols for Kopano Core ii kopano-dev 8.6.81.228-0+37.1 amd64 C++ Development files for Kopano Core ii kopano-gateway 8.6.81.228-0+37.1 amd64 POP3 and IMAP Gateway for Kopano Core ii kopano-gateway-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-gateway ii kopano-ical 8.6.81.228-0+37.1 amd64 ICal and CalDAV Gateway for Kopano Core ii kopano-ical-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-ical ii kopano-lang 8.6.81.228-0+37.1 all Translations for Kopano Core components ii kopano-migration-imap 8.6.81.228-0+37.1 amd64 Utility to migrate between IMAP mailboxes ii kopano-migration-pst 8.6.81.228-0+37.1 amd64 Utility to import PST files ii kopano-monitor 8.6.81.228-0+37.1 amd64 Quota Monitor for Kopano Core ii kopano-monitor-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-monitor ii kopano-presence 8.6.81.228-0+37.1 amd64 kopano presence daemon ii kopano-python-utils 8.6.81.228-0+37.1 amd64 Additional Python-based command-line utils for Kopano Core ii kopano-search 8.6.81.228-0+37.1 amd64 Indexed search engine for Kopano Core ii kopano-server 8.6.81.228-0+37.1 amd64 Server component for Kopano Core ii kopano-server-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-server ii kopano-server-packages 8.6.81.228-0+37.1 all Metapackage to install the entire Kopano Core stack ii kopano-spamd 8.6.81.228-0+37.1 amd64 ICS-driven spam learning daemon for Kopano/SpamAssasin ii kopano-spooler 8.6.81.228-0+37.1 amd64 E-mail Spooler for Kopano Core ii kopano-spooler-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-spooler ii kopano-utils 8.6.81.228-0+37.1 amd64 Admin command-line utils for Kopano Core ii kopano-utils-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-utils ii kopano-webapp 3.4.22.1724+981.1 all New and improved WebApp for Kopano ii kopano-webapp-plugin-contactfax 3.4.22.1724+981.1 all Kopano WebApp fax plugin ii kopano-webapp-plugin-desktopnotifications 2.0.1.5+16.2 all Kopano WebApp Desktop notifications plugin ii kopano-webapp-plugin-filepreviewer 2.0.0.15+10.8 all Kopano File previewer plugin ii kopano-webapp-plugin-files 2.1.5.295+194.1 all Adds Files functionality to Kopano enabling access to WebDAV and other files backends. ii kopano-webapp-plugin-filesbackend-owncloud 2.1.1.84+61.1 all Adds Owncloud specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-filesbackend-smb 2.1.1.48+49.1 all Adds Samba specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-folderwidgets 3.4.22.1724+981.1 all Kopano WebApp folder widgets plugin ii kopano-webapp-plugin-gmaps 3.4.22.1724+981.1 all Kopano WebApp google maps plugin ii kopano-webapp-plugin-intranet 0.1+32.8 all This plugin adds one or more buttons in the top menu bar which can be used to open a webpage inside Kopano WebApp. ii kopano-webapp-plugin-mattermost 0.1+17.2 all Integrates Mattermost into WebApp ii kopano-webapp-plugin-mdm 2.1.0.106+26.1 all Kopano WebApp MDM plugin ii kopano-webapp-plugin-pimfolder 3.4.22.1724+981.1 all Kopano WebApp personal inbox plugin ii kopano-webapp-plugin-quickitems 3.4.22.1724+981.1 all Kopano WebApp quick items plugin ii kopano-webapp-plugin-spell 2.0.0.17+34.1 all Kopano WebApp Spellchecker plugin ii kopano-webapp-plugin-spell-de-at 2.0.0.4+28.1 all Kopano WebApp Spellchecker German (Austrian) dictionary plugin ii kopano-webapp-plugin-spell-de-ch 2.0.0.5+27.1 all Kopano WebApp Spellchecker German (Swiss) dictionary plugin ii kopano-webapp-plugin-spell-de-de 2.0.0.3+28.1 all Kopano WebApp Spellchecker German dictionary plugin ii kopano-webapp-plugin-spell-en 2.0.0.1+28.1 all Kopano WebApp Spellchecker English dictionary plugin ii kopano-webapp-plugin-spell-en-gb 2.0.0.1+27.1 all Kopano WebApp Spellchecker English (GB) dictionary plugin ii kopano-webapp-plugin-spell-es 2.0.0.1+27.1 all Kopano WebApp Spellchecker Spanish dictionary plugin ii kopano-webapp-plugin-spell-fr 2.0.0.1+28.1 all Kopano WebApp Spellchecker French dictionary plugin ii kopano-webapp-plugin-spell-it 1.0.0+40.1 all Kopano WebApp Spellchecker Italian dictionary plugin ii kopano-webapp-plugin-spell-nl 2.0.0.1+29.1 all Kopano WebApp Spellchecker Dutch dictionary plugin ii kopano-webapp-plugin-spell-pl-pl 2.0.0.0+33.1 all Kopano WebApp Spellchecker Polish dictionary plugin ii kopano-webapp-plugin-titlecounter 3.4.22.1724+981.1 all Kopano WebApp Titlecounter plugin ii kopano-webapp-plugin-webappmanual 3.4.22.1724+981.1 all Kopano WebApp Manual plugin ii kopano-webapp-plugin-zdeveloper 3.4.22.1724+981.1 all Kopano WebApp developer plugin ii libgsoap-kopano-2.8.62 2.8.62-0+2.1 amd64 Runtime libraries for gSOAP ii libvmime-kopano1 0.9.2.50+11.1 amd64 a C++ mail library (runtime library) ii libvmime-kopano1-dbgsym 0.9.2.50+11.1 amd64 Debug symbols for libvmime-kopano1 ii python-kopano 8.6.81.228-0+37.1 all High-level Python 2 bindings for Kopano ii python3-kopano 8.6.81.228-0+37.1 all High-level Python 3 bindings for Kopano ii z-push-backend-kopano 2.4.4+0-0 all Z-Push Kopano backend ii z-push-kopano 2.4.4+0-0 all Z-Push for KopanoLet’s see if this changes anything.
I’m still not completely desperate :-)Regards,
Alex
-
Unfortunately not. kopano-server still crashing every 4 - 5 hours after flooding the whole RAM and SWAP memory. :-(
-
hello @Sniks6,
i had a similar issue on a testsystem, resolving was quite questionable :)
first i made the mariadb tuning for the system, second i reduce the index-process to 1 worker for first indexing.
then i only started search, wait 5 minutes till it complains about missing kopano-server, then start it, without gateway or dagent.
Wait for the first indexing, this worked fine in my case but used a lot of time.If this works, stop everything, incrase the indexing worker to standard value (i think it was 5) and start everything.
The stats of this vm where:
VMware machine
4 cores
8GB ram
scsi sas drives
CentOS 7 64bit
mariadb - 5.5.60-1
kopano-server - 8.6.4.0
no webapp, since its an external machineregards,
Coffee_is_life
-
Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.
depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?
8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?
-
Hi @ all,
Thank you very much for your anwers!
Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.
That’s true. Perhaps I just didn’t start all services at the time I made the mysqldump.
depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?
You’re right. The search service always uses much cpu und I think after two weeks it still hasn’t finished the indexing. :-(
I stopped kopano-search 2 hours ago and at the moment the cpu and memory usage is very low:top - 15:24:59 up 6 days, 4 min, 3 users, load average: 0,03, 0,06, 0,01 Tasks: 123 total, 1 running, 122 sleeping, 0 stopped, 0 zombie %Cpu(s): 0,2 us, 0,3 sy, 0,0 ni, 99,5 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8045376 total, 5219748 free, 564032 used, 2261596 buff/cache KiB Swap: 4143100 total, 4040092 free, 103008 used. 7173136 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 158017 mysql 20 0 726648 243176 17020 S 0,0 3,0 1:32.18 mysqld 161987 kopano 20 0 360232 71796 18200 S 0,0 0,9 0:12.51 kopano-server 165057 www-data 20 0 734928 43668 35932 S 0,0 0,5 0:00.76 apache2 163614 www-data 20 0 740548 43224 33404 S 0,0 0,5 0:00.92 apache2 108383 www-data 20 0 740392 43132 33628 S 0,3 0,5 0:02.50 apache2 164767 www-data 20 0 735112 42924 34632 S 0,0 0,5 0:00.82 apache2 168432 www-data 20 0 735064 42452 34652 S 0,0 0,5 0:00.52 apache2 164391 www-data 20 0 739752 42276 33036 S 0,0 0,5 0:00.64 apache2 165992 www-data 20 0 735012 40456 33280 S 0,0 0,5 0:00.50 apache2 168150 www-data 20 0 734292 40408 33536 S 0,0 0,5 0:00.36 apache2 103010 www-data 20 0 734872 40388 32832 S 0,0 0,5 0:02.11 apache2 164770 www-data 20 0 734660 40136 33020 S 0,0 0,5 0:00.64 apache2 165990 www-data 20 0 734628 38988 32424 S 0,0 0,5 0:00.44 apache2 170618 www-data 20 0 666752 33664 28192 S 0,0 0,4 0:00.02 apache2 170622 www-data 20 0 666752 33664 28192 S 0,0 0,4 0:00.02 apache2 162160 kopano 20 0 230108 20472 18368 S 0,0 0,3 0:00.24 kopano-spooler 162138 kopano 20 0 160660 16596 14668 S 0,0 0,2 0:00.08 kopano-dagent 247 root 20 0 70680 9724 9492 S 0,0 0,1 1:38.56 systemd-journal 146108 root 20 0 101436 6904 5900 S 0,0 0,1 0:00.22 sshd 170192 root 20 0 101432 6856 5852 S 0,0 0,1 0:00.22 sshd8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?
I already changed the cache to:
cache_cell_size = 2GUnfortunately without success.
As of now with the disabled kopano-search it looks good. Is there a way to find out what causes this mem flooding when indexing by kopano-search?
Regards,
Alex -
Good Morning everyone,
now, after 17 hours it looks good kopano-server did not crash since then.
So I think we found the problem: kopano-search is causing the memory flood in kopano-server.But why? As Coffee_is_life wrote on his setup after completing the index process everything worked fine. Perhaps this is the same with my setup. But my indexing is running for two weeks now. :-(
Do you have any ideas what to tweek in this case?
-
After 4 days without kopano-search kopano-server did not crash anymore.
But the memory keeps still rising much more slowly…
Any thoughts?
-
I also (still?) have this problem. Currently using 8.6.81.171-0+25.1 on Ubuntu 18.04 x64.
[Fri Sep 28 19:51:29 2018] Out of memory: Kill process 1240 (kopano-server) score 902 or sacrifice child
[Fri Sep 28 19:51:29 2018] Killed process 1240 (kopano-server) total-vm:26514968kB, anon-rss:10946196kB, file-rss:0kB, shmem-rss:0kBBy the time it had crashed it was running only for 11 hours or so - after having been restarted due to a kopano-server crash the day before. The server should have enough memory for normal use (12GB) and an additional 16GB Swap is available. Especially with the swap I really see no reason at all for it to run out of memory.
I will now try to update to 8.6.81.435-0+75.1 in the hope that this out of memory bug has been fixed in the meantime?
In addition I’ve also cleaned the /var/lib/kopano/search/ directory and restarted the indexing process. Maybe index is corrupted?
Update: Has not changed/fixed anything. Server still crashing about once a day. Redoing the index didn’t help, neither did disabling the indexing of attachments.
I’ve added the following to the kopano-server.service file for a temporary fix:
[add in the service section]
Restart=on-failure
RestartSec=10s[add in the unit section]
StartLimitIntervalSec=300s
StartLimitBurst=2 -
Thank you for your message.
Here is a graph of the memory consumption:
- In the first half you see the memory rising very quickly till the server process crashed. It restarted automatically after modfying the kopano-server.service file.
- In the second half (~ week 38) I didn’t start the kopano-search process. The memory is still rising but much slower.
So I think it is not a problem with the search process. It seems that the search process simply speeds up the rising of the memory usage. If I had more users or much more interaction with the server process, perhaps the memory would be rising faster again.
So it looks like a problem in the kopano-server.
Just for interest, what hardware do you use, Gerald? Is it virtualised hardware? If yes which kind (VMware, HyperV,…)?
Regards,
Alex
-
@sniks6 said in Kopano-server crashes:
Just for interest, what hardware do you use, Gerald? Is it virtualised hardware? If yes which kind (VMware, HyperV,…)?
I’m using VMware ESXi (Free Edition), currently version 6.5.0 U1 (Build 7967591). The VM has the open-vm-tools package installed.
I don’t think the behavior would change on physical hardware though.Update: crashed again:
[Tue Oct 2 12:01:22 2018] Out of memory: Kill process 2038 (kopano-server) score 848 or sacrifice child
but my modifications to automatically restart the process worked so I guess nobody even noticed.Just 6 hours later memory usage for the kopano-server process was again up to 5.6GB, after restarting kopano-server memory usage is down to 350MB. This thing is leaking like crazy! At this rate even throwing 32GB at it would not really help for long.
Update2: regarding kopano on physical hardware
I just checked the kopano I’m using for my personal email, which is on physical hardware (Atom-based Quadcore NUC with 8GB RAM) and here kopano-server is hogging about 3GB of memory after running for 48 hours under low load (about 3 users with 3 devices each). Also an Ubuntu 18.04 system by the way. -
@Gerald: Thanks for your infos. :-)
Isn’t there anybody who could give us a hint, where we could tweak or check what could cause this memory problem?
-
@sniks6 said in Kopano-server crashes:
Isn’t there anybody who could give us a hint, where we could tweak or check what could cause this memory problem?
This does not look like there is something to tweak, but this rather is a memory leak somewhere in the master branch. We want to branch of a release soon, so we eventually will need to look into it.
I’ve created https://jira.kopano.io/browse/KC-1299 to track progress.
-
Thanks for your message, Felix.
If it is helpful, I could provide you more info from my system.I am happy to help.
-
We have plugged a first mem leak over the weekend, the fix should be included with the next update of the nightly packages: https://stash.kopano.io/projects/KC/repos/kopanocore/commits/57dd3acd8e04695137cd2f86bc9f173cbf242aa1
-
we fixed an important memory leak in master only two weeks ago… looking at the timing of above comments, it seems could have well been the cause of some of these issues?
commit 30b9996155fcad78d41cf79aa399867252b75e96
Merge: 4d16ca5c7 57dd3acd8
Author: Joost Hopmans j.hopmans@kopano.com
Date: Fri Oct 5 16:16:32 2018 +0200Merge pull request #2264 in KC/kopanocore from ~JENGELHARDT/kc:leak to master * commit '57dd3acd8e04695137cd2f86bc9f173cbf242aa1': libserver: plug a memory leak in loadObjectyeah, when doing initial syncing with search, that would probably uncover a memleak much faster than under average load…
-
Hi Mark,
thank you for your answer. Right now I am on holidays. But when I am back I will try the actual nightly build and report about it.Regards, Alex
-
Hi there,
just came back from my holidays.
I wanted to install the latest nightly build “core-8.7.80.12_0+11-Debian_9.0-amd64”.Unfortunately there is libkcpyplug0 missing. So I could not run the spooler or dagent.
Could you please include the libkcpyplug0 again?
Thanks.
-
that was fixed on Oct 24.
» tar -tf core-8.7.80.12_0+11-Debian_9.0-amd64.tar.gz | grep pypl core-8.7.80.12_0+11-Debian_9.0-amd64/./libkcpyplug0-dbgsym_8.7.80.12-0+11.1_amd64.deb core-8.7.80.12_0+11-Debian_9.0-amd64/./libkcpyplug0_8.7.80.12-0+11.1_amd64.deb
