Kopano-server crashes
-
Nothing special.
The only thing I noticed in /var/log/mail.log is
Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to commit message: disk error (80040116) Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to deliver message to "alex": disk error (80040116) Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to deliver all messages for server "default:"Don’t know what this is? I just set dagent log to 5, to get more info about this.
The weird thing is, that everything worked for two days without any problems. And then after I restarted the server because I moved it to its designated place on my floor, it began again. Everytime I restart der server service it immediately begins flooding the memory:
Crazy… :-(
-
I found out about the disk error message. There was one corrupted message in the postfix queue.
A “postqueue -f” and the error showed up again in the log.
After deleting the message from the postfix queue the kopano-dagent error vanished. -
The problem still exists. I don’t see anything weird or any errors in the logs.
Here is the output of kopano-stats --system:
0x3001001E 0x6740001E 0x6741001E ----------------------------------------------------------------------------------------------------------------------------------------- server_start_date Time when the server was started Di Sep 11 09:48:00 2018 cache_purge_date Time when the cache was cleared config_reload_date Time when the configuration file was reloaded / logrotation (SIGHUP) connections Number of handled incoming connections 436494 max_socket Highest socket number used 1057 redirections Number of redirected requests 0 soap_request Number of soap requests handled by server 2212637 response_time Response time of soap requests handled in milliseconds (includes time in queue) 14092788 processing_time Time taken to process soap requests in milliseconds (wallclock time) 14017893 searchfld_loaded Total number of searchfolders 10 searchfld_threads Current number of running searchfolder threads 0 searchupd_retry The number of times a search folder update was restarted 0 searchupd_fail The number of failed search folder updates after retrying 0 sql_connect Number of connections made to SQL server 432837 sql_select Number of SQL Select commands executed 9358564 sql_insert Number of SQL Insert commands executed 4369 sql_update Number of SQL Update commands executed 2298 sql_delete Number of SQL Delete commands executed 1163 sql_connect_fail Number of failed connections made to SQL server 0 sql_select_fail Number of failed SQL Select commands 0 sql_insert_fail Number of failed SQL Insert commands 0 sql_update_fail Number of failed SQL Update commands 0 sql_delete_fail Number of failed SQL Delete commands 0 sql_last_fail_time Timestamp of last failed SQL command mwops MAPI Write Operations 125 mrops MAPI Read Operations 873475 deferred_fetches Number rows retrieved via deferred write table 4657 deferred_merges Number of merges applied to the deferred write table 4 deferred_records Number records merged in the deferred write table 80 row_reads Number of table rows read in row order 4712 counter_resyncs Number of time a counter resync was required 0 login_password Number of logins through password authentication 2357 login_ssl Number of logins through SSL certificate authentication 0 login_sso Number of logins through Single Sign-on 0 login_unix Number of logins through Unix socket 433715 login_failed Number of failed logins 0 sessions_created Number of created sessions 436072 sessions_deleted Number of deleted sessions 435995 sessions_timeout Number of timed-out sessions 59 sess_int_created Number of created internal sessions 172 sess_int_deleted Number of deleted internal sessions 172 sess_grp_created Number of created sessiongroups 2167 sess_grp_deleted Number of deleted sessiongroups 2160 ldap_connect Number of connections made to LDAP server 0 ldap_reconnect Number of re-connections made to LDAP server 0 ldap_connect_fail Number of failed connections made to LDAP server 0 ldap_connect_time Total duration of connections made to LDAP server 0 ldap_max_connect Longest connection time made to LDAP server 0 ldap_auth Number of LDAP authentications 0 ldap_auth_fail Number of failed authentications 0 ldap_auth_time Total authentication time 0 ldap_max_auth Longest duration of authentication made to LDAP server 0 ldap_avg_auth Average duration of authentication made to LDAP server 0 ldap_search Number of searches made to LDAP server 0 ldap_search_fail Number of failed searches made to LDAP server 0 ldap_search_time Total duration of LDAP searches 0 ldap_max_search Longest duration of LDAP search 0 index_search_errors Number of failed indexer queries 0 index_search_max Maximum duration of an indexed search query 1070 index_search_avg Average duration of an indexed search query 1070 search_indexed Number of indexed searches performed 572 search_database Number of database searches performed 0 cache_acl_items Cache acl items 0 cache_acl_size Cache acl size 0 cache_acl_maxsz Cache acl maximum size 1048576 cache_acl_req Cache acl requests 0 cache_acl_hit Cache acl hits 0 cache_quota_items Cache quota items 1 cache_quota_size Cache quota size 104 cache_quota_maxsz Cache quota maximum size 1048576 cache_quota_req Cache quota requests 1238 cache_quota_hit Cache quota hits 990 cache_uquota_items Cache uquota items 0 cache_uquota_size Cache uquota size 0 cache_uquota_maxsz Cache uquota maximum size 1048576 cache_uquota_req Cache uquota requests 0 cache_uquota_hit Cache uquota hits 0 cache_extern_items Cache extern items 3 cache_extern_size Cache extern size 501 cache_extern_maxsz Cache extern maximum size 26214400 cache_extern_req Cache extern requests 6467 cache_extern_hit Cache extern hits 3828 cache_userid_items Cache userid items 3 cache_userid_size Cache userid size 456 cache_userid_maxsz Cache userid maximum size 26214400 cache_userid_req Cache userid requests 21191 cache_userid_hit Cache userid hits 21174 cache_abinfo_items Cache abinfo items 3 cache_abinfo_size Cache abinfo size 552 cache_abinfo_maxsz Cache abinfo maximum size 26214400 cache_abinfo_req Cache abinfo requests 12624 cache_abinfo_hit Cache abinfo hits 12621 cache_server_items Cache server items 0 cache_server_size Cache server size 0 cache_server_maxsz Cache server maximum size 1048576 cache_server_req Cache server requests 0 cache_server_hit Cache server hits 0 cache_store_items Cache store items 10903 cache_store_size Cache store size 1046688 cache_store_maxsz Cache store maximum size 1048576 cache_store_req Cache store requests 5788327 cache_store_hit Cache store hits 5346082 cache_obj_items Cache obj items 136117 cache_obj_size Cache obj size 11978296 cache_obj_maxsz Cache obj maximum size 16777216 cache_obj_req Cache obj requests 6469036 cache_obj_hit Cache obj hits 6334828 cache_cell_items Cache cell items 447448 cache_cell_size Cache cell size 1964950625 cache_cell_maxsz Cache cell maximum size 2147483648 cache_cell_req Cache cell requests 36588037 cache_cell_hit Cache cell hits 36073239 cache_index1_items Cache index1 items 246612 cache_index1_size Cache index1 size 32306108 cache_index1_maxsz Cache index1 maximum size 33554432 cache_index1_req Cache index1 requests 4162193 cache_index1_hit Cache index1 hits 2443995 cache_index2_items Cache index2 items 259385 cache_index2_size Cache index2 size 31904214 cache_index2_maxsz Cache index2 maximum size 33554432 cache_index2_req Cache index2 requests 5690998 cache_index2_hit Cache index2 hits 4491006 sessions Number of sessions 18 sessions_size Memory usage of sessions 1296 sessiongroups Number of session groups 7 sessiongroups_size Memory usage of session groups 5784 persist_conn Persistent connections 0 persist_conn_size Memory usage of persistent connections 0 persist_sess Persistent sessions 0 persist_sess_size Memory usage of persistent sessions 0 tables_subscr Tables subscribed 1 tables_subscr_size Memory usage of subscribed tables 72 object_subscr Objects subscribed 3 object_subscr_size Memory usage of subscribed objects 192 searchfld_stores Number of stores in use by search folders 3 searchfld_folders Number of folders in use by search folders 10 searchfld_events Number of events waiting for searchfolder updates 0 searchfld_size Memory usage of search folders 8703 queuelen Current queue length 0 queueage Age of the front queue item 0.000 threads Number of threads running to process items 0 threads_idle Number of idle threads 8 usercnt_active Number of active users 3 usercnt_nonactive Number of total non-active objects 0 usercnt_na_user Number of non-active users 0 usercnt_room Number of rooms 0 usercnt_equipment Number of equipment 0 usercnt_contact Number of contacts 0 tc_allocated Current allocated memory by TCMalloc 6587321000 tc_reserved Bytes of system memory reserved by TCMalloc 6811803648 tc_page_map_free Number of bytes in free, mapped pages in page heap 0 tc_page_unmap_free Number of bytes in free, unmapped pages in page heap (released to OS) 47865856 tc_threadcache_max A limit to how much memory TCMalloc dedicates for small objects 33554432 tc_threadcache_cur Current allocated memory in bytes for thread cache 26003304 pt_allocated Current allocated memory by libc ptmalloc, in bytes -2002612232 userplugin User plugin used dbAnd here is the output of top:
top - 14:12:31 up 2 days, 4:50, 2 users, load average: 1,45, 1,45, 1,37 Tasks: 131 total, 2 running, 129 sleeping, 0 stopped, 0 zombie %Cpu(s): 18,5 us, 7,8 sy, 0,0 ni, 69,5 id, 3,8 wa, 0,0 hi, 0,5 si, 0,0 st KiB Mem : 8045368 total, 130000 free, 7294192 used, 621176 buff/cache KiB Swap: 4143100 total, 3929428 free, 213672 used. 457920 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 92741 mysql 20 0 755712 182892 0 S 41,6 2,3 514:22.10 mysqld 154878 kopano 20 0 7020140 6,412g 17236 S 39,6 83,6 80:59.98 kopano-server 125893 kopano 20 0 513720 127600 4920 R 22,8 1,6 204:06.06 kopano-search 111218 root 20 0 0 0 0 S 1,0 0,0 0:00.02 kworker/3:0 119974 root 20 0 44924 3556 2892 R 1,0 0,0 0:00.09 top 1 root 20 0 204592 3508 2276 S 0,0 0,0 1:00.06 systemd 2 root 20 0 0 0 0 S 0,0 0,0 0:00.08 kthreadd 3 root 20 0 0 0 0 S 0,0 0,0 0:07.52 ksoftirqd/0 5 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/0:0H 6 root 20 0 0 0 0 S 0,0 0,0 0:00.28 kworker/u480:0 8 root 20 0 0 0 0 S 0,0 0,0 3:50.28 rcu_sched 9 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcu_bh 10 root rt 0 0 0 0 S 0,0 0,0 0:00.84 migration/0 11 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 lru-add-drain 12 root rt 0 0 0 0 S 0,0 0,0 0:00.39 watchdog/0 13 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/0 14 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/1 15 root rt 0 0 0 0 S 0,0 0,0 0:00.40 watchdog/1 16 root rt 0 0 0 0 S 0,0 0,0 0:00.85 migration/1 17 root 20 0 0 0 0 S 0,0 0,0 0:04.95 ksoftirqd/1 19 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/1:0H 20 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/2 21 root rt 0 0 0 0 S 0,0 0,0 0:00.43 watchdog/2 22 root rt 0 0 0 0 S 0,0 0,0 0:00.80 migration/2 23 root 20 0 0 0 0 S 0,0 0,0 0:10.55 ksoftirqd/2 25 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/2:0H 27 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/3 28 root rt 0 0 0 0 S 0,0 0,0 0:00.45 watchdog/3 29 root rt 0 0 0 0 S 0,0 0,0 0:00.83 migration/3 30 root 20 0 0 0 0 S 0,0 0,0 0:04.34 ksoftirqd/3 32 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/3:0H 33 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kdevtmpfs 34 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 netns 35 root 20 0 0 0 0 S 0,0 0,0 0:00.06 khungtaskd 36 root 20 0 0 0 0 S 0,0 0,0 0:00.00 oom_reaper 37 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 writeback 38 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kcompactd0 39 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kcompactd1 41 root 25 5 0 0 0 S 0,0 0,0 0:00.00 ksmd 42 root 39 19 0 0 0 S 0,0 0,0 0:07.14 khugepaged 43 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 crypto 44 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kintegrityd 45 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 bioset 46 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kblockd 47 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 devfreq_wq 48 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 watchdogd 49 root 20 0 0 0 0 S 0,0 0,0 4:18.34 kswapd0 50 root 20 0 0 0 0 S 0,0 0,0 4:50.38 kswapd1 51 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 vmstat 63 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kthrotld 69 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 ipv6_addrconf 99 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 hv_vmbus_con 118 root 20 0 0 0 0 S 0,0 0,0 0:00.00 scsi_eh_0 120 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 scsi_tmf_0 122 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 bioset 141 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 bioset 176 root 0 -20 0 0 0 S 0,0 0,0 0:37.67 kworker/0:1H 177 root 0 -20 0 0 0 S 0,0 0,0 0:35.76 kworker/1:1H 185 root 0 -20 0 0 0 S 0,0 0,0 0:35.09 kworker/3:1H 190 root 0 -20 0 0 0 S 0,0 0,0 0:45.28 kworker/2:1HAll I see is high CPU usage because of the kopano-search process.
Shutting down the search process ends the high CPU usage. But the memory usage keeps rising.Does anybody have a hint for me? What could I check or do to get the system running without restarting every 5 hours.
Thanks,
Alex
-
hi, can you post an output of :
dpkg -l | grep kopano apt-cache policy kopano-serverand did you upgrade this version from a 8.5?
then read : https://forum.kopano.io/topic/1305/crash-kopano-core-upgrade-from-8-5-81-288-0-53-1-to-8-6-80-645-0-68-1/18
after cleaning up the left overs, im out of problems. -
Hi,
thank you for your message.
here are the requested outputs:root@kopano:~# dpkg -l | grep kopano ii kopano-backup 8.6.81.198-0+35.1 amd64 Utility to back up and restore Kopano stores ii kopano-bash-completion 8.6.81.198-0+35.1 amd64 bash TAB completion for Kopano Core command-line utilities ii kopano-client 8.6.81.198-0+35.1 amd64 Kopano MAPI provider library ii kopano-client-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-client ii kopano-common 8.6.81.198-0+35.1 amd64 Shared files for Kopano services ii kopano-contacts 8.6.81.198-0+35.1 amd64 MAPI provider adding contact folders in the addressbook ii kopano-dagent 8.6.81.198-0+35.1 amd64 E-Mail Delivery Agent for Kopano Core ii kopano-dagent-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-dagent ii kopano-dbg 8.6.81.198-0+35.1 amd64 Debug symbols for Kopano Core ii kopano-dev 8.6.81.198-0+35.1 amd64 C++ Development files for Kopano Core ii kopano-gateway 8.6.81.198-0+35.1 amd64 POP3 and IMAP Gateway for Kopano Core ii kopano-gateway-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-gateway ii kopano-ical 8.6.81.198-0+35.1 amd64 ICal and CalDAV Gateway for Kopano Core ii kopano-ical-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-ical ii kopano-lang 8.6.81.198-0+35.1 all Translations for Kopano Core components ii kopano-migration-imap 8.6.81.198-0+35.1 amd64 Utility to migrate between IMAP mailboxes ii kopano-migration-pst 8.6.81.198-0+35.1 amd64 Utility to import PST files ii kopano-monitor 8.6.81.198-0+35.1 amd64 Quota Monitor for Kopano Core ii kopano-monitor-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-monitor ii kopano-presence 8.6.81.198-0+35.1 amd64 kopano presence daemon ii kopano-python-utils 8.6.81.198-0+35.1 amd64 Additional Python-based command-line utils for Kopano Core ii kopano-search 8.6.81.198-0+35.1 amd64 Indexed search engine for Kopano Core ii kopano-server 8.6.81.198-0+35.1 amd64 Server component for Kopano Core ii kopano-server-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-server ii kopano-server-packages 8.6.81.198-0+35.1 all Metapackage to install the entire Kopano Core stack ii kopano-spamd 8.6.81.198-0+35.1 amd64 ICS-driven spam learning daemon for Kopano/SpamAssasin ii kopano-spooler 8.6.81.198-0+35.1 amd64 E-mail Spooler for Kopano Core ii kopano-spooler-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-spooler ii kopano-utils 8.6.81.198-0+35.1 amd64 Admin command-line utils for Kopano Core ii kopano-utils-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-utils ii kopano-webapp 3.4.22.1724+981.1 all New and improved WebApp for Kopano ii kopano-webapp-plugin-contactfax 3.4.22.1724+981.1 all Kopano WebApp fax plugin ii kopano-webapp-plugin-desktopnotifications 2.0.1.5+16.2 all Kopano WebApp Desktop notifications plugin ii kopano-webapp-plugin-filepreviewer 2.0.0.15+10.8 all Kopano File previewer plugin ii kopano-webapp-plugin-files 2.1.5.295+194.1 all Adds Files functionality to Kopano enabling access to WebDAV and other files backends. ii kopano-webapp-plugin-filesbackend-owncloud 2.1.1.84+61.1 all Adds Owncloud specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-filesbackend-smb 2.1.1.48+49.1 all Adds Samba specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-folderwidgets 3.4.22.1724+981.1 all Kopano WebApp folder widgets plugin ii kopano-webapp-plugin-gmaps 3.4.22.1724+981.1 all Kopano WebApp google maps plugin ii kopano-webapp-plugin-intranet 0.1+32.8 all This plugin adds one or more buttons in the top menu bar which can be used to open a webpage inside Kopano WebApp. ii kopano-webapp-plugin-mattermost 0.1+17.2 all Integrates Mattermost into WebApp ii kopano-webapp-plugin-mdm 2.1.0.106+26.1 all Kopano WebApp MDM plugin ii kopano-webapp-plugin-pimfolder 3.4.22.1724+981.1 all Kopano WebApp personal inbox plugin ii kopano-webapp-plugin-quickitems 3.4.22.1724+981.1 all Kopano WebApp quick items plugin ii kopano-webapp-plugin-spell 2.0.0.17+34.1 all Kopano WebApp Spellchecker plugin ii kopano-webapp-plugin-spell-de-at 2.0.0.4+28.1 all Kopano WebApp Spellchecker German (Austrian) dictionary plugin ii kopano-webapp-plugin-spell-de-ch 2.0.0.5+27.1 all Kopano WebApp Spellchecker German (Swiss) dictionary plugin ii kopano-webapp-plugin-spell-de-de 2.0.0.3+28.1 all Kopano WebApp Spellchecker German dictionary plugin ii kopano-webapp-plugin-spell-en 2.0.0.1+28.1 all Kopano WebApp Spellchecker English dictionary plugin ii kopano-webapp-plugin-spell-en-gb 2.0.0.1+27.1 all Kopano WebApp Spellchecker English (GB) dictionary plugin ii kopano-webapp-plugin-spell-es 2.0.0.1+27.1 all Kopano WebApp Spellchecker Spanish dictionary plugin ii kopano-webapp-plugin-spell-fr 2.0.0.1+28.1 all Kopano WebApp Spellchecker French dictionary plugin ii kopano-webapp-plugin-spell-it 1.0.0+40.1 all Kopano WebApp Spellchecker Italian dictionary plugin ii kopano-webapp-plugin-spell-nl 2.0.0.1+29.1 all Kopano WebApp Spellchecker Dutch dictionary plugin ii kopano-webapp-plugin-spell-pl-pl 2.0.0.0+33.1 all Kopano WebApp Spellchecker Polish dictionary plugin ii kopano-webapp-plugin-titlecounter 3.4.22.1724+981.1 all Kopano WebApp Titlecounter plugin ii kopano-webapp-plugin-webappmanual 3.4.22.1724+981.1 all Kopano WebApp Manual plugin ii kopano-webapp-plugin-zdeveloper 3.4.22.1724+981.1 all Kopano WebApp developer plugin ii libgsoap-kopano-2.8.62 2.8.62-0+2.1 amd64 Runtime libraries for gSOAP ii libvmime-kopano1 0.9.2.50+11.1 amd64 a C++ mail library (runtime library) ii libvmime-kopano1-dbgsym 0.9.2.50+11.1 amd64 Debug symbols for libvmime-kopano1 ii python-kopano 8.6.81.198-0+35.1 all High-level Python 2 bindings for Kopano ii python3-kopano 8.6.81.198-0+35.1 all High-level Python 3 bindings for Kopano ii z-push-backend-kopano 2.4.4+0-0 all Z-Push Kopano backend ii z-push-kopano 2.4.4+0-0 all Z-Push for Kopanoroot@kopano:~# apt-cache policy kopano-server kopano-server: Installiert: 8.6.81.198-0+35.1 Installationskandidat: 8.6.81.198-0+35.1 Versionstabelle: *** 8.6.81.198-0+35.1 100 100 /var/lib/dpkg/statusHopefully this helps finding the problem.
Thank you very much in advance.Regards,
Alex
P.S.: Actually there is always a high CPU usage of kopano-search, kopano-server and mysqld:
top - 00:03:01 up 2 days, 8:42, 1 user, load average: 1,31, 1,38, 1,41 Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie %Cpu(s): 20,4 us, 0,9 sy, 0,0 ni, 69,3 id, 9,3 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8045376 total, 140868 free, 7740960 used, 163548 buff/cache KiB Swap: 4143100 total, 3174024 free, 969076 used. 69448 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 43607 kopano 20 0 481320 89756 5544 D 41,7 1,1 665:50.80 kopano-search 32208 kopano 20 0 8185608 6,879g 976 S 22,8 89,7 86:45.45 kopano-server 714 mysql 20 0 763904 170444 0 S 20,5 2,1 1322:28 mysqld 90822 root 20 0 44928 3648 2988 R 0,7 0,0 0:00.04 top 175 root 0 -20 0 0 0 S 0,3 0,0 0:46.38 kworker/3:1H 1 root 20 0 204692 3024 2056 S 0,0 0,0 0:24.93 systemd 2 root 20 0 0 0 0 S 0,0 0,0 0:00.07 kthreadd 3 root 20 0 0 0 0 S 0,0 0,0 0:08.63 ksoftirqd/0Perhaps this takes some days till index-process is over. All in all about 80 Gigabyte of attachments and MySQL DB was migrated before.
-
Just made an update to the actual nightly version:
root@kopano:~# apt-cache policy kopano-server kopano-server: Installiert: 8.6.81.228-0+37.1 Installationskandidat: 8.6.81.228-0+37.1 Versionstabelle: *** 8.6.81.228-0+37.1 100 100 /var/lib/dpkg/statusand:
root@kopano:~# dpkg -l | grep kopano ii kopano-backup 8.6.81.228-0+37.1 amd64 Utility to back up and restore Kopano stores ii kopano-bash-completion 8.6.81.228-0+37.1 amd64 bash TAB completion for Kopano Core command-line utilities ii kopano-client 8.6.81.228-0+37.1 amd64 Kopano MAPI provider library ii kopano-client-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-client ii kopano-common 8.6.81.228-0+37.1 amd64 Shared files for Kopano services ii kopano-contacts 8.6.81.228-0+37.1 amd64 MAPI provider adding contact folders in the addressbook ii kopano-dagent 8.6.81.228-0+37.1 amd64 E-Mail Delivery Agent for Kopano Core ii kopano-dagent-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-dagent ii kopano-dbg 8.6.81.228-0+37.1 amd64 Debug symbols for Kopano Core ii kopano-dev 8.6.81.228-0+37.1 amd64 C++ Development files for Kopano Core ii kopano-gateway 8.6.81.228-0+37.1 amd64 POP3 and IMAP Gateway for Kopano Core ii kopano-gateway-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-gateway ii kopano-ical 8.6.81.228-0+37.1 amd64 ICal and CalDAV Gateway for Kopano Core ii kopano-ical-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-ical ii kopano-lang 8.6.81.228-0+37.1 all Translations for Kopano Core components ii kopano-migration-imap 8.6.81.228-0+37.1 amd64 Utility to migrate between IMAP mailboxes ii kopano-migration-pst 8.6.81.228-0+37.1 amd64 Utility to import PST files ii kopano-monitor 8.6.81.228-0+37.1 amd64 Quota Monitor for Kopano Core ii kopano-monitor-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-monitor ii kopano-presence 8.6.81.228-0+37.1 amd64 kopano presence daemon ii kopano-python-utils 8.6.81.228-0+37.1 amd64 Additional Python-based command-line utils for Kopano Core ii kopano-search 8.6.81.228-0+37.1 amd64 Indexed search engine for Kopano Core ii kopano-server 8.6.81.228-0+37.1 amd64 Server component for Kopano Core ii kopano-server-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-server ii kopano-server-packages 8.6.81.228-0+37.1 all Metapackage to install the entire Kopano Core stack ii kopano-spamd 8.6.81.228-0+37.1 amd64 ICS-driven spam learning daemon for Kopano/SpamAssasin ii kopano-spooler 8.6.81.228-0+37.1 amd64 E-mail Spooler for Kopano Core ii kopano-spooler-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-spooler ii kopano-utils 8.6.81.228-0+37.1 amd64 Admin command-line utils for Kopano Core ii kopano-utils-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-utils ii kopano-webapp 3.4.22.1724+981.1 all New and improved WebApp for Kopano ii kopano-webapp-plugin-contactfax 3.4.22.1724+981.1 all Kopano WebApp fax plugin ii kopano-webapp-plugin-desktopnotifications 2.0.1.5+16.2 all Kopano WebApp Desktop notifications plugin ii kopano-webapp-plugin-filepreviewer 2.0.0.15+10.8 all Kopano File previewer plugin ii kopano-webapp-plugin-files 2.1.5.295+194.1 all Adds Files functionality to Kopano enabling access to WebDAV and other files backends. ii kopano-webapp-plugin-filesbackend-owncloud 2.1.1.84+61.1 all Adds Owncloud specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-filesbackend-smb 2.1.1.48+49.1 all Adds Samba specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-folderwidgets 3.4.22.1724+981.1 all Kopano WebApp folder widgets plugin ii kopano-webapp-plugin-gmaps 3.4.22.1724+981.1 all Kopano WebApp google maps plugin ii kopano-webapp-plugin-intranet 0.1+32.8 all This plugin adds one or more buttons in the top menu bar which can be used to open a webpage inside Kopano WebApp. ii kopano-webapp-plugin-mattermost 0.1+17.2 all Integrates Mattermost into WebApp ii kopano-webapp-plugin-mdm 2.1.0.106+26.1 all Kopano WebApp MDM plugin ii kopano-webapp-plugin-pimfolder 3.4.22.1724+981.1 all Kopano WebApp personal inbox plugin ii kopano-webapp-plugin-quickitems 3.4.22.1724+981.1 all Kopano WebApp quick items plugin ii kopano-webapp-plugin-spell 2.0.0.17+34.1 all Kopano WebApp Spellchecker plugin ii kopano-webapp-plugin-spell-de-at 2.0.0.4+28.1 all Kopano WebApp Spellchecker German (Austrian) dictionary plugin ii kopano-webapp-plugin-spell-de-ch 2.0.0.5+27.1 all Kopano WebApp Spellchecker German (Swiss) dictionary plugin ii kopano-webapp-plugin-spell-de-de 2.0.0.3+28.1 all Kopano WebApp Spellchecker German dictionary plugin ii kopano-webapp-plugin-spell-en 2.0.0.1+28.1 all Kopano WebApp Spellchecker English dictionary plugin ii kopano-webapp-plugin-spell-en-gb 2.0.0.1+27.1 all Kopano WebApp Spellchecker English (GB) dictionary plugin ii kopano-webapp-plugin-spell-es 2.0.0.1+27.1 all Kopano WebApp Spellchecker Spanish dictionary plugin ii kopano-webapp-plugin-spell-fr 2.0.0.1+28.1 all Kopano WebApp Spellchecker French dictionary plugin ii kopano-webapp-plugin-spell-it 1.0.0+40.1 all Kopano WebApp Spellchecker Italian dictionary plugin ii kopano-webapp-plugin-spell-nl 2.0.0.1+29.1 all Kopano WebApp Spellchecker Dutch dictionary plugin ii kopano-webapp-plugin-spell-pl-pl 2.0.0.0+33.1 all Kopano WebApp Spellchecker Polish dictionary plugin ii kopano-webapp-plugin-titlecounter 3.4.22.1724+981.1 all Kopano WebApp Titlecounter plugin ii kopano-webapp-plugin-webappmanual 3.4.22.1724+981.1 all Kopano WebApp Manual plugin ii kopano-webapp-plugin-zdeveloper 3.4.22.1724+981.1 all Kopano WebApp developer plugin ii libgsoap-kopano-2.8.62 2.8.62-0+2.1 amd64 Runtime libraries for gSOAP ii libvmime-kopano1 0.9.2.50+11.1 amd64 a C++ mail library (runtime library) ii libvmime-kopano1-dbgsym 0.9.2.50+11.1 amd64 Debug symbols for libvmime-kopano1 ii python-kopano 8.6.81.228-0+37.1 all High-level Python 2 bindings for Kopano ii python3-kopano 8.6.81.228-0+37.1 all High-level Python 3 bindings for Kopano ii z-push-backend-kopano 2.4.4+0-0 all Z-Push Kopano backend ii z-push-kopano 2.4.4+0-0 all Z-Push for KopanoLet’s see if this changes anything.
I’m still not completely desperate :-)Regards,
Alex
-
Unfortunately not. kopano-server still crashing every 4 - 5 hours after flooding the whole RAM and SWAP memory. :-(
-
hello @Sniks6,
i had a similar issue on a testsystem, resolving was quite questionable :)
first i made the mariadb tuning for the system, second i reduce the index-process to 1 worker for first indexing.
then i only started search, wait 5 minutes till it complains about missing kopano-server, then start it, without gateway or dagent.
Wait for the first indexing, this worked fine in my case but used a lot of time.If this works, stop everything, incrase the indexing worker to standard value (i think it was 5) and start everything.
The stats of this vm where:
VMware machine
4 cores
8GB ram
scsi sas drives
CentOS 7 64bit
mariadb - 5.5.60-1
kopano-server - 8.6.4.0
no webapp, since its an external machineregards,
Coffee_is_life
-
Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.
depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?
8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?
-
Hi @ all,
Thank you very much for your anwers!
Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.
That’s true. Perhaps I just didn’t start all services at the time I made the mysqldump.
depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?
You’re right. The search service always uses much cpu und I think after two weeks it still hasn’t finished the indexing. :-(
I stopped kopano-search 2 hours ago and at the moment the cpu and memory usage is very low:top - 15:24:59 up 6 days, 4 min, 3 users, load average: 0,03, 0,06, 0,01 Tasks: 123 total, 1 running, 122 sleeping, 0 stopped, 0 zombie %Cpu(s): 0,2 us, 0,3 sy, 0,0 ni, 99,5 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8045376 total, 5219748 free, 564032 used, 2261596 buff/cache KiB Swap: 4143100 total, 4040092 free, 103008 used. 7173136 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 158017 mysql 20 0 726648 243176 17020 S 0,0 3,0 1:32.18 mysqld 161987 kopano 20 0 360232 71796 18200 S 0,0 0,9 0:12.51 kopano-server 165057 www-data 20 0 734928 43668 35932 S 0,0 0,5 0:00.76 apache2 163614 www-data 20 0 740548 43224 33404 S 0,0 0,5 0:00.92 apache2 108383 www-data 20 0 740392 43132 33628 S 0,3 0,5 0:02.50 apache2 164767 www-data 20 0 735112 42924 34632 S 0,0 0,5 0:00.82 apache2 168432 www-data 20 0 735064 42452 34652 S 0,0 0,5 0:00.52 apache2 164391 www-data 20 0 739752 42276 33036 S 0,0 0,5 0:00.64 apache2 165992 www-data 20 0 735012 40456 33280 S 0,0 0,5 0:00.50 apache2 168150 www-data 20 0 734292 40408 33536 S 0,0 0,5 0:00.36 apache2 103010 www-data 20 0 734872 40388 32832 S 0,0 0,5 0:02.11 apache2 164770 www-data 20 0 734660 40136 33020 S 0,0 0,5 0:00.64 apache2 165990 www-data 20 0 734628 38988 32424 S 0,0 0,5 0:00.44 apache2 170618 www-data 20 0 666752 33664 28192 S 0,0 0,4 0:00.02 apache2 170622 www-data 20 0 666752 33664 28192 S 0,0 0,4 0:00.02 apache2 162160 kopano 20 0 230108 20472 18368 S 0,0 0,3 0:00.24 kopano-spooler 162138 kopano 20 0 160660 16596 14668 S 0,0 0,2 0:00.08 kopano-dagent 247 root 20 0 70680 9724 9492 S 0,0 0,1 1:38.56 systemd-journal 146108 root 20 0 101436 6904 5900 S 0,0 0,1 0:00.22 sshd 170192 root 20 0 101432 6856 5852 S 0,0 0,1 0:00.22 sshd8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?
I already changed the cache to:
cache_cell_size = 2GUnfortunately without success.
As of now with the disabled kopano-search it looks good. Is there a way to find out what causes this mem flooding when indexing by kopano-search?
Regards,
Alex -
Good Morning everyone,
now, after 17 hours it looks good kopano-server did not crash since then.
So I think we found the problem: kopano-search is causing the memory flood in kopano-server.But why? As Coffee_is_life wrote on his setup after completing the index process everything worked fine. Perhaps this is the same with my setup. But my indexing is running for two weeks now. :-(
Do you have any ideas what to tweek in this case?
-
After 4 days without kopano-search kopano-server did not crash anymore.
But the memory keeps still rising much more slowly…
Any thoughts?
-
I also (still?) have this problem. Currently using 8.6.81.171-0+25.1 on Ubuntu 18.04 x64.
[Fri Sep 28 19:51:29 2018] Out of memory: Kill process 1240 (kopano-server) score 902 or sacrifice child
[Fri Sep 28 19:51:29 2018] Killed process 1240 (kopano-server) total-vm:26514968kB, anon-rss:10946196kB, file-rss:0kB, shmem-rss:0kBBy the time it had crashed it was running only for 11 hours or so - after having been restarted due to a kopano-server crash the day before. The server should have enough memory for normal use (12GB) and an additional 16GB Swap is available. Especially with the swap I really see no reason at all for it to run out of memory.
I will now try to update to 8.6.81.435-0+75.1 in the hope that this out of memory bug has been fixed in the meantime?
In addition I’ve also cleaned the /var/lib/kopano/search/ directory and restarted the indexing process. Maybe index is corrupted?
Update: Has not changed/fixed anything. Server still crashing about once a day. Redoing the index didn’t help, neither did disabling the indexing of attachments.
I’ve added the following to the kopano-server.service file for a temporary fix:
[add in the service section]
Restart=on-failure
RestartSec=10s[add in the unit section]
StartLimitIntervalSec=300s
StartLimitBurst=2 -
Thank you for your message.
Here is a graph of the memory consumption:
- In the first half you see the memory rising very quickly till the server process crashed. It restarted automatically after modfying the kopano-server.service file.
- In the second half (~ week 38) I didn’t start the kopano-search process. The memory is still rising but much slower.
So I think it is not a problem with the search process. It seems that the search process simply speeds up the rising of the memory usage. If I had more users or much more interaction with the server process, perhaps the memory would be rising faster again.
So it looks like a problem in the kopano-server.
Just for interest, what hardware do you use, Gerald? Is it virtualised hardware? If yes which kind (VMware, HyperV,…)?
Regards,
Alex
-
@sniks6 said in Kopano-server crashes:
Just for interest, what hardware do you use, Gerald? Is it virtualised hardware? If yes which kind (VMware, HyperV,…)?
I’m using VMware ESXi (Free Edition), currently version 6.5.0 U1 (Build 7967591). The VM has the open-vm-tools package installed.
I don’t think the behavior would change on physical hardware though.Update: crashed again:
[Tue Oct 2 12:01:22 2018] Out of memory: Kill process 2038 (kopano-server) score 848 or sacrifice child
but my modifications to automatically restart the process worked so I guess nobody even noticed.Just 6 hours later memory usage for the kopano-server process was again up to 5.6GB, after restarting kopano-server memory usage is down to 350MB. This thing is leaking like crazy! At this rate even throwing 32GB at it would not really help for long.
Update2: regarding kopano on physical hardware
I just checked the kopano I’m using for my personal email, which is on physical hardware (Atom-based Quadcore NUC with 8GB RAM) and here kopano-server is hogging about 3GB of memory after running for 48 hours under low load (about 3 users with 3 devices each). Also an Ubuntu 18.04 system by the way. -
@Gerald: Thanks for your infos. :-)
Isn’t there anybody who could give us a hint, where we could tweak or check what could cause this memory problem?
-
@sniks6 said in Kopano-server crashes:
Isn’t there anybody who could give us a hint, where we could tweak or check what could cause this memory problem?
This does not look like there is something to tweak, but this rather is a memory leak somewhere in the master branch. We want to branch of a release soon, so we eventually will need to look into it.
I’ve created https://jira.kopano.io/browse/KC-1299 to track progress.
-
Thanks for your message, Felix.
If it is helpful, I could provide you more info from my system.I am happy to help.
-
We have plugged a first mem leak over the weekend, the fix should be included with the next update of the nightly packages: https://stash.kopano.io/projects/KC/repos/kopanocore/commits/57dd3acd8e04695137cd2f86bc9f173cbf242aa1
-
we fixed an important memory leak in master only two weeks ago… looking at the timing of above comments, it seems could have well been the cause of some of these issues?
commit 30b9996155fcad78d41cf79aa399867252b75e96
Merge: 4d16ca5c7 57dd3acd8
Author: Joost Hopmans j.hopmans@kopano.com
Date: Fri Oct 5 16:16:32 2018 +0200Merge pull request #2264 in KC/kopanocore from ~JENGELHARDT/kc:leak to master * commit '57dd3acd8e04695137cd2f86bc9f173cbf242aa1': libserver: plug a memory leak in loadObjectyeah, when doing initial syncing with search, that would probably uncover a memleak much faster than under average load…
