Navigation

    Kopano
    • Register
    • Login
    • Search
    • Categories
    • Get Official Kopano Support
    • Recent
    Statement regarding the closure of the Kopano community forum and the end of the community edition

    Kopano-server crashes

    Kopano Groupware Core
    9
    37
    4916
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Sniks6
      Sniks6 last edited by

      … and then it happened again. This afternoon:

      .
      .
      .
      Sep  9 17:14:01 kopano kopano-server[947]: Start syncedmessages table clean up
      Sep  9 17:14:01 kopano kopano-server[947]: syncedmessages table clean up done, 0 entries removed
      Sep  9 17:14:39 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für  (37985430 Bytes).
      Sep  9 17:14:39 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für  (110571564 Bytes).
      Sep  9 17:14:40 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für  (371923599 Bytes).
      Sep  9 17:15:41 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für (37985430 Bytes).
      Sep  9 17:15:41 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes).
      Sep  9 17:15:42 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für  (371923599 Bytes).
      Sep  9 17:16:43 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für  (37985430 Bytes).
      Sep  9 17:16:43 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für  (110571564 Bytes).
      Sep  9 17:16:44 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für (371923599 Bytes).
      Sep  9 17:17:45 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für  (37985430 Bytes).
      Sep  9 17:17:45 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes).
      Sep  9 17:17:46 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für  (371923599 Bytes).
      Sep  9 17:18:12 kopano kopano-server[947]: ECFileAttachment::LoadAttachmentInstance(SOAP): Out of memory while reading "/var/lib/kopano/attachments/1/1/56811.gz"
      Sep  9 17:18:12 kopano kopano-server[947]: SerializeObject failed with error code 0x80000028 for object 128416
      Sep  9 17:18:29 kopano kopano-server[947]: ECFileAttachment::LoadAttachmentInstance(SOAP): Out of memory while reading "/var/lib/kopano/attachments/6/7/56876.gz"
      Sep  9 17:18:29 kopano kopano-server[947]: SerializeObject failed with error code 0x80000028 for object 128522
      Sep  9 17:18:41 kopano kopano-server[947]: ----------------------------------------------------------------------
      Sep  9 17:18:41 kopano kopano-server[947]: Fatal error detected. Please report all following information.
      Sep  9 17:18:41 kopano kopano-server[947]: Application kopano-server version: 8.6.81
      Sep  9 17:18:41 kopano kopano-server[947]: OS: Linux, release: 4.9.0-8-amd64, version: #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21), hardware: x86_64
      Sep  9 17:18:41 kopano kopano-server[947]: Thread name: ECThreadPool
      Sep  9 17:18:41 kopano kopano-server[947]: Peak RSS: 7316596
      Sep  9 17:18:41 kopano kopano-server[947]: Pid 947 caught SIGABRT (6), out of memory or unhandled exception, traceback:
      Sep  9 17:18:41 kopano kopano-server[947]: Backtrace:
      Sep  9 17:18:41 kopano kopano-server[947]: #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x472c0) [0x7f3a273582c0]
      Sep  9 17:18:41 kopano kopano-server[947]: #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x362ed) [0x7f3a273472ed]
      Sep  9 17:18:41 kopano kopano-server[947]: #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x1bd) [0x7f3a273475cd]
      Sep  9 17:18:41 kopano kopano-server[947]: #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f3a21e3d0c0]
      Sep  9 17:18:41 kopano kopano-server[947]: #4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7f3a21222fff]
      Sep  9 17:18:41 kopano kopano-server[947]: #5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f3a2122442a]
      Sep  9 17:18:41 kopano kopano-server[947]: #6. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x15d) [0x7f3a21b3b0ad]
      Sep  9 17:18:41 kopano kopano-server[947]: #7. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f066) [0x7f3a21b39066]
      Sep  9 17:18:41 kopano kopano-server[947]: #8. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f0b1) [0x7f3a21b390b1]
      Sep  9 17:18:41 kopano kopano-server[947]: #9. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f2c9) [0x7f3a21b392c9]
      Sep  9 17:18:41 kopano kopano-server[947]: #10. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x37ef2) [0x7f3a26f4eef2]
      Sep  9 17:18:41 kopano kopano-server[947]: #11. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x5320f) [0x7f3a26f6a20f]
      Sep  9 17:18:41 kopano kopano-server[947]: #12. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x55eee) [0x7f3a26f6ceee]
      Sep  9 17:18:41 kopano kopano-server[947]: #13. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x1242d7) [0x7f3a2703b2d7]
      Sep  9 17:18:41 kopano kopano-server[947]: #14. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x12751c) [0x7f3a2703e51c]
      Sep  9 17:18:41 kopano kopano-server[947]: #15. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x188f9b) [0x7f3a2709ff9b]
      Sep  9 17:18:41 kopano kopano-server[947]: #16. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC14ECWaitableTask7executeEv+0x84) [0x7f3a27349c04]
      Sep  9 17:18:41 kopano kopano-server[947]: #17. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x3929c) [0x7f3a2734a29c]
      Sep  9 17:18:41 kopano kopano-server[947]: #18. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f3a21e33494]
      Sep  9 17:18:41 kopano kopano-server[947]: #19. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3a212d8acf]
      Sep  9 17:18:41 kopano kopano-server[947]: Signal errno: Erfolg, signal code: -6
      Sep  9 17:18:41 kopano kopano-server[947]: Sender pid: 947, sender uid: 999, si_status: 0
      Sep  9 17:18:41 kopano kopano-server[947]: Signal value: 0, faulting address: 0x3e7000003b3
      Sep  9 17:18:41 kopano kopano-server[947]: When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version.
      Sep  9 17:18:46 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für   (37985430 Bytes).
      Sep  9 17:18:47 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes).
      Sep  9 17:18:47 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für  (371923599 Bytes).
      Sep  9 17:19:34 kopano kopano-spooler[514]: Unable to get outgoing queue count: network error (80040115)
      Sep  9 17:19:34 kopano kopano-spooler[514]: ProcessQueue: ProcessAllEntries failed: network error (80040115)
      Sep  9 17:19:34 kopano kopano-spooler[514]: Server connection lost. Reconnecting in 3 seconds...
      Sep  9 17:19:37 kopano kopano-spooler[514]: gsoap connect: ()
      Sep  9 17:19:37 kopano kopano-spooler[514]: HrLogon server "default:" user "SYSTEM": network error
      .
      .
      .
      

      CheckMK logged the memory usage:

      alt text

      You can see the memory usage rise and then the crash of the server process and the memory usage returns to normal.

      1 Reply Last reply Reply Quote 0
      • marcus.bauch
        marcus.bauch last edited by

        …are there any messages in messages, syslog or daemon (/var/log/*) ?

        1 Reply Last reply Reply Quote 0
        • Sniks6
          Sniks6 last edited by

          Nothing special.

          The only thing I noticed in /var/log/mail.log is

          Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to commit message: disk error (80040116)
          Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to deliver message to "alex": disk error (80040116)
          Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to deliver all messages for server "default:"
          

          Don’t know what this is? I just set dagent log to 5, to get more info about this.

          The weird thing is, that everything worked for two days without any problems. And then after I restarted the server because I moved it to its designated place on my floor, it began again. Everytime I restart der server service it immediately begins flooding the memory:

          alt text

          Crazy… :-(

          1 Reply Last reply Reply Quote 0
          • Sniks6
            Sniks6 last edited by Sniks6

            I found out about the disk error message. There was one corrupted message in the postfix queue.
            A “postqueue -f” and the error showed up again in the log.
            After deleting the message from the postfix queue the kopano-dagent error vanished.

            1 Reply Last reply Reply Quote 1
            • Sniks6
              Sniks6 last edited by

              The problem still exists. I don’t see anything weird or any errors in the logs.

              Here is the output of kopano-stats --system:

                      0x3001001E              0x6740001E                                                                      0x6741001E
                      -----------------------------------------------------------------------------------------------------------------------------------------
                      server_start_date       Time when the server was started                                                Di Sep 11 09:48:00 2018
                      cache_purge_date        Time when the cache was cleared
                      config_reload_date      Time when the configuration file was reloaded / logrotation (SIGHUP)
                      connections             Number of handled incoming connections                                          436494
                      max_socket              Highest socket number used                                                      1057
                      redirections            Number of redirected requests                                                   0
                      soap_request            Number of soap requests handled by server                                       2212637
                      response_time           Response time of soap requests handled in milliseconds (includes time in queue) 14092788
                      processing_time         Time taken to process soap requests in milliseconds (wallclock time)            14017893
                      searchfld_loaded        Total number of searchfolders                                                   10
                      searchfld_threads       Current number of running searchfolder threads                                  0
                      searchupd_retry         The number of times a search folder update was restarted                        0
                      searchupd_fail          The number of failed search folder updates after retrying                       0
                      sql_connect             Number of connections made to SQL server                                        432837
                      sql_select              Number of SQL Select commands executed                                          9358564
                      sql_insert              Number of SQL Insert commands executed                                          4369
                      sql_update              Number of SQL Update commands executed                                          2298
                      sql_delete              Number of SQL Delete commands executed                                          1163
                      sql_connect_fail        Number of failed connections made to SQL server                                 0
                      sql_select_fail         Number of failed SQL Select commands                                            0
                      sql_insert_fail         Number of failed SQL Insert commands                                            0
                      sql_update_fail         Number of failed SQL Update commands                                            0
                      sql_delete_fail         Number of failed SQL Delete commands                                            0
                      sql_last_fail_time      Timestamp of last failed SQL command
                      mwops                   MAPI Write Operations                                                           125
                      mrops                   MAPI Read Operations                                                            873475
                      deferred_fetches        Number rows retrieved via deferred write table                                  4657
                      deferred_merges         Number of merges applied to the deferred write table                            4
                      deferred_records        Number records merged in the deferred write table                               80
                      row_reads               Number of table rows read in row order                                          4712
                      counter_resyncs         Number of time a counter resync was required                                    0
                      login_password          Number of logins through password authentication                                2357
                      login_ssl               Number of logins through SSL certificate authentication                         0
                      login_sso               Number of logins through Single Sign-on                                         0
                      login_unix              Number of logins through Unix socket                                            433715
                      login_failed            Number of failed logins                                                         0
                      sessions_created        Number of created sessions                                                      436072
                      sessions_deleted        Number of deleted sessions                                                      435995
                      sessions_timeout        Number of timed-out sessions                                                    59
                      sess_int_created        Number of created internal sessions                                             172
                      sess_int_deleted        Number of deleted internal sessions                                             172
                      sess_grp_created        Number of created sessiongroups                                                 2167
                      sess_grp_deleted        Number of deleted sessiongroups                                                 2160
                      ldap_connect            Number of connections made to LDAP server                                       0
                      ldap_reconnect          Number of re-connections made to LDAP server                                    0
                      ldap_connect_fail       Number of failed connections made to LDAP server                                0
                      ldap_connect_time       Total duration of connections made to LDAP server                               0
                      ldap_max_connect        Longest connection time made to LDAP server                                     0
                      ldap_auth               Number of LDAP authentications                                                  0
                      ldap_auth_fail          Number of failed authentications                                                0
                      ldap_auth_time          Total authentication time                                                       0
                      ldap_max_auth           Longest duration of authentication made to LDAP server                          0
                      ldap_avg_auth           Average duration of authentication made to LDAP server                          0
                      ldap_search             Number of searches made to LDAP server                                          0
                      ldap_search_fail        Number of failed searches made to LDAP server                                   0
                      ldap_search_time        Total duration of LDAP searches                                                 0
                      ldap_max_search         Longest duration of LDAP search                                                 0
                      index_search_errors     Number of failed indexer queries                                                0
                      index_search_max        Maximum duration of an indexed search query                                     1070
                      index_search_avg        Average duration of an indexed search query                                     1070
                      search_indexed          Number of indexed searches performed                                            572
                      search_database         Number of database searches performed                                           0
                      cache_acl_items         Cache acl items                                                                 0
                      cache_acl_size          Cache acl size                                                                  0
                      cache_acl_maxsz         Cache acl maximum size                                                          1048576
                      cache_acl_req           Cache acl requests                                                              0
                      cache_acl_hit           Cache acl hits                                                                  0
                      cache_quota_items       Cache quota items                                                               1
                      cache_quota_size        Cache quota size                                                                104
                      cache_quota_maxsz       Cache quota maximum size                                                        1048576
                      cache_quota_req         Cache quota requests                                                            1238
                      cache_quota_hit         Cache quota hits                                                                990
                      cache_uquota_items      Cache uquota items                                                              0
                      cache_uquota_size       Cache uquota size                                                               0
                      cache_uquota_maxsz      Cache uquota maximum size                                                       1048576
                      cache_uquota_req        Cache uquota requests                                                           0
                      cache_uquota_hit        Cache uquota hits                                                               0
                      cache_extern_items      Cache extern items                                                              3
                      cache_extern_size       Cache extern size                                                               501
                      cache_extern_maxsz      Cache extern maximum size                                                       26214400
                      cache_extern_req        Cache extern requests                                                           6467
                      cache_extern_hit        Cache extern hits                                                               3828
                      cache_userid_items      Cache userid items                                                              3
                      cache_userid_size       Cache userid size                                                               456
                      cache_userid_maxsz      Cache userid maximum size                                                       26214400
                      cache_userid_req        Cache userid requests                                                           21191
                      cache_userid_hit        Cache userid hits                                                               21174
                      cache_abinfo_items      Cache abinfo items                                                              3
                      cache_abinfo_size       Cache abinfo size                                                               552
                      cache_abinfo_maxsz      Cache abinfo maximum size                                                       26214400
                      cache_abinfo_req        Cache abinfo requests                                                           12624
                      cache_abinfo_hit        Cache abinfo hits                                                               12621
                      cache_server_items      Cache server items                                                              0
                      cache_server_size       Cache server size                                                               0
                      cache_server_maxsz      Cache server maximum size                                                       1048576
                      cache_server_req        Cache server requests                                                           0
                      cache_server_hit        Cache server hits                                                               0
                      cache_store_items       Cache store items                                                               10903
                      cache_store_size        Cache store size                                                                1046688
                      cache_store_maxsz       Cache store maximum size                                                        1048576
                      cache_store_req         Cache store requests                                                            5788327
                      cache_store_hit         Cache store hits                                                                5346082
                      cache_obj_items         Cache obj items                                                                 136117
                      cache_obj_size          Cache obj size                                                                  11978296
                      cache_obj_maxsz         Cache obj maximum size                                                          16777216
                      cache_obj_req           Cache obj requests                                                              6469036
                      cache_obj_hit           Cache obj hits                                                                  6334828
                      cache_cell_items        Cache cell items                                                                447448
                      cache_cell_size         Cache cell size                                                                 1964950625
                      cache_cell_maxsz        Cache cell maximum size                                                         2147483648
                      cache_cell_req          Cache cell requests                                                             36588037
                      cache_cell_hit          Cache cell hits                                                                 36073239
                      cache_index1_items      Cache index1 items                                                              246612
                      cache_index1_size       Cache index1 size                                                               32306108
                      cache_index1_maxsz      Cache index1 maximum size                                                       33554432
                      cache_index1_req        Cache index1 requests                                                           4162193
                      cache_index1_hit        Cache index1 hits                                                               2443995
                      cache_index2_items      Cache index2 items                                                              259385
                      cache_index2_size       Cache index2 size                                                               31904214
                      cache_index2_maxsz      Cache index2 maximum size                                                       33554432
                      cache_index2_req        Cache index2 requests                                                           5690998
                      cache_index2_hit        Cache index2 hits                                                               4491006
                      sessions                Number of sessions                                                              18
                      sessions_size           Memory usage of sessions                                                        1296
                      sessiongroups           Number of session groups                                                        7
                      sessiongroups_size      Memory usage of session groups                                                  5784
                      persist_conn            Persistent connections                                                          0
                      persist_conn_size       Memory usage of persistent connections                                          0
                      persist_sess            Persistent sessions                                                             0
                      persist_sess_size       Memory usage of persistent sessions                                             0
                      tables_subscr           Tables subscribed                                                               1
                      tables_subscr_size      Memory usage of subscribed tables                                               72
                      object_subscr           Objects subscribed                                                              3
                      object_subscr_size      Memory usage of subscribed objects                                              192
                      searchfld_stores        Number of stores in use by search folders                                       3
                      searchfld_folders       Number of folders in use by search folders                                      10
                      searchfld_events        Number of events waiting for searchfolder updates                               0
                      searchfld_size          Memory usage of search folders                                                  8703
                      queuelen                Current queue length                                                            0
                      queueage                Age of the front queue item                                                     0.000
                      threads                 Number of threads running to process items                                      0
                      threads_idle            Number of idle threads                                                          8
                      usercnt_active          Number of active users                                                          3
                      usercnt_nonactive       Number of total non-active objects                                              0
                      usercnt_na_user         Number of non-active users                                                      0
                      usercnt_room            Number of rooms                                                                 0
                      usercnt_equipment       Number of equipment                                                             0
                      usercnt_contact         Number of contacts                                                              0
                      tc_allocated            Current allocated memory by TCMalloc                                            6587321000
                      tc_reserved             Bytes of system memory reserved by TCMalloc                                     6811803648
                      tc_page_map_free        Number of bytes in free, mapped pages in page heap                              0
                      tc_page_unmap_free      Number of bytes in free, unmapped pages in page heap (released to OS)           47865856
                      tc_threadcache_max      A limit to how much memory TCMalloc dedicates for small objects                 33554432
                      tc_threadcache_cur      Current allocated memory in bytes for thread cache                              26003304
                      pt_allocated            Current allocated memory by libc ptmalloc, in bytes                             -2002612232
                      userplugin              User plugin used                                                                db
              
              

              And here is the output of top:

              top - 14:12:31 up 2 days,  4:50,  2 users,  load average: 1,45, 1,45, 1,37
              Tasks: 131 total,   2 running, 129 sleeping,   0 stopped,   0 zombie
              %Cpu(s): 18,5 us,  7,8 sy,  0,0 ni, 69,5 id,  3,8 wa,  0,0 hi,  0,5 si,  0,0 st
              KiB Mem :  8045368 total,   130000 free,  7294192 used,   621176 buff/cache
              KiB Swap:  4143100 total,  3929428 free,   213672 used.   457920 avail Mem
              
                 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
               92741 mysql     20   0  755712 182892      0 S  41,6  2,3 514:22.10 mysqld
              154878 kopano    20   0 7020140 6,412g  17236 S  39,6 83,6  80:59.98 kopano-server
              125893 kopano    20   0  513720 127600   4920 R  22,8  1,6 204:06.06 kopano-search
              111218 root      20   0       0      0      0 S   1,0  0,0   0:00.02 kworker/3:0
              119974 root      20   0   44924   3556   2892 R   1,0  0,0   0:00.09 top
                   1 root      20   0  204592   3508   2276 S   0,0  0,0   1:00.06 systemd
                   2 root      20   0       0      0      0 S   0,0  0,0   0:00.08 kthreadd
                   3 root      20   0       0      0      0 S   0,0  0,0   0:07.52 ksoftirqd/0
                   5 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kworker/0:0H
                   6 root      20   0       0      0      0 S   0,0  0,0   0:00.28 kworker/u480:0
                   8 root      20   0       0      0      0 S   0,0  0,0   3:50.28 rcu_sched
                   9 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcu_bh
                  10 root      rt   0       0      0      0 S   0,0  0,0   0:00.84 migration/0
                  11 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 lru-add-drain
                  12 root      rt   0       0      0      0 S   0,0  0,0   0:00.39 watchdog/0
                  13 root      20   0       0      0      0 S   0,0  0,0   0:00.00 cpuhp/0
                  14 root      20   0       0      0      0 S   0,0  0,0   0:00.00 cpuhp/1
                  15 root      rt   0       0      0      0 S   0,0  0,0   0:00.40 watchdog/1
                  16 root      rt   0       0      0      0 S   0,0  0,0   0:00.85 migration/1
                  17 root      20   0       0      0      0 S   0,0  0,0   0:04.95 ksoftirqd/1
                  19 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kworker/1:0H
                  20 root      20   0       0      0      0 S   0,0  0,0   0:00.00 cpuhp/2
                  21 root      rt   0       0      0      0 S   0,0  0,0   0:00.43 watchdog/2
                  22 root      rt   0       0      0      0 S   0,0  0,0   0:00.80 migration/2
                  23 root      20   0       0      0      0 S   0,0  0,0   0:10.55 ksoftirqd/2
                  25 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kworker/2:0H
                  27 root      20   0       0      0      0 S   0,0  0,0   0:00.00 cpuhp/3
                  28 root      rt   0       0      0      0 S   0,0  0,0   0:00.45 watchdog/3
                  29 root      rt   0       0      0      0 S   0,0  0,0   0:00.83 migration/3
                  30 root      20   0       0      0      0 S   0,0  0,0   0:04.34 ksoftirqd/3
                  32 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kworker/3:0H
                  33 root      20   0       0      0      0 S   0,0  0,0   0:00.00 kdevtmpfs
                  34 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 netns
                  35 root      20   0       0      0      0 S   0,0  0,0   0:00.06 khungtaskd
                  36 root      20   0       0      0      0 S   0,0  0,0   0:00.00 oom_reaper
                  37 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 writeback
                  38 root      20   0       0      0      0 S   0,0  0,0   0:00.00 kcompactd0
                  39 root      20   0       0      0      0 S   0,0  0,0   0:00.00 kcompactd1
                  41 root      25   5       0      0      0 S   0,0  0,0   0:00.00 ksmd
                  42 root      39  19       0      0      0 S   0,0  0,0   0:07.14 khugepaged
                  43 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 crypto
                  44 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kintegrityd
                  45 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 bioset
                  46 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kblockd
                  47 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 devfreq_wq
                  48 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 watchdogd
                  49 root      20   0       0      0      0 S   0,0  0,0   4:18.34 kswapd0
                  50 root      20   0       0      0      0 S   0,0  0,0   4:50.38 kswapd1
                  51 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 vmstat
                  63 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kthrotld
                  69 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 ipv6_addrconf
                  99 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 hv_vmbus_con
                 118 root      20   0       0      0      0 S   0,0  0,0   0:00.00 scsi_eh_0
                 120 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 scsi_tmf_0
                 122 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 bioset
                 141 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 bioset
                 176 root       0 -20       0      0      0 S   0,0  0,0   0:37.67 kworker/0:1H
                 177 root       0 -20       0      0      0 S   0,0  0,0   0:35.76 kworker/1:1H
                 185 root       0 -20       0      0      0 S   0,0  0,0   0:35.09 kworker/3:1H
                 190 root       0 -20       0      0      0 S   0,0  0,0   0:45.28 kworker/2:1H
              

              All I see is high CPU usage because of the kopano-search process.
              Shutting down the search process ends the high CPU usage. But the memory usage keeps rising.

              Does anybody have a hint for me? What could I check or do to get the system running without restarting every 5 hours.

              Thanks,

              Alex

              1 Reply Last reply Reply Quote 0
              • thctlo
                thctlo last edited by thctlo

                hi, can you post an output of :

                dpkg -l | grep kopano
                apt-cache policy kopano-server
                

                and did you upgrade this version from a 8.5?
                then read : https://forum.kopano.io/topic/1305/crash-kopano-core-upgrade-from-8-5-81-288-0-53-1-to-8-6-80-645-0-68-1/18
                after cleaning up the left overs, im out of problems.

                1 Reply Last reply Reply Quote 0
                • Sniks6
                  Sniks6 last edited by Sniks6

                  Hi,

                  thank you for your message.
                  here are the requested outputs:

                  root@kopano:~# dpkg -l | grep kopano
                  ii  kopano-backup                              8.6.81.198-0+35.1                          amd64        Utility to back up and restore Kopano stores
                  ii  kopano-bash-completion                     8.6.81.198-0+35.1                          amd64        bash TAB completion for Kopano Core command-line utilities
                  ii  kopano-client                              8.6.81.198-0+35.1                          amd64        Kopano MAPI provider library
                  ii  kopano-client-dbgsym                       8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-client
                  ii  kopano-common                              8.6.81.198-0+35.1                          amd64        Shared files for Kopano services
                  ii  kopano-contacts                            8.6.81.198-0+35.1                          amd64        MAPI provider adding contact folders in the addressbook
                  ii  kopano-dagent                              8.6.81.198-0+35.1                          amd64        E-Mail Delivery Agent for Kopano Core
                  ii  kopano-dagent-dbgsym                       8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-dagent
                  ii  kopano-dbg                                 8.6.81.198-0+35.1                          amd64        Debug symbols for Kopano Core
                  ii  kopano-dev                                 8.6.81.198-0+35.1                          amd64        C++ Development files for Kopano Core
                  ii  kopano-gateway                             8.6.81.198-0+35.1                          amd64        POP3 and IMAP Gateway for Kopano Core
                  ii  kopano-gateway-dbgsym                      8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-gateway
                  ii  kopano-ical                                8.6.81.198-0+35.1                          amd64        ICal and CalDAV Gateway for Kopano Core
                  ii  kopano-ical-dbgsym                         8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-ical
                  ii  kopano-lang                                8.6.81.198-0+35.1                          all          Translations for Kopano Core components
                  ii  kopano-migration-imap                      8.6.81.198-0+35.1                          amd64        Utility to migrate between IMAP mailboxes
                  ii  kopano-migration-pst                       8.6.81.198-0+35.1                          amd64        Utility to import PST files
                  ii  kopano-monitor                             8.6.81.198-0+35.1                          amd64        Quota Monitor for Kopano Core
                  ii  kopano-monitor-dbgsym                      8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-monitor
                  ii  kopano-presence                            8.6.81.198-0+35.1                          amd64        kopano presence daemon
                  ii  kopano-python-utils                        8.6.81.198-0+35.1                          amd64        Additional Python-based command-line utils for Kopano Core
                  ii  kopano-search                              8.6.81.198-0+35.1                          amd64        Indexed search engine for Kopano Core
                  ii  kopano-server                              8.6.81.198-0+35.1                          amd64        Server component for Kopano Core
                  ii  kopano-server-dbgsym                       8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-server
                  ii  kopano-server-packages                     8.6.81.198-0+35.1                          all          Metapackage to install the entire Kopano Core stack
                  ii  kopano-spamd                               8.6.81.198-0+35.1                          amd64        ICS-driven spam learning daemon for Kopano/SpamAssasin
                  ii  kopano-spooler                             8.6.81.198-0+35.1                          amd64        E-mail Spooler for Kopano Core
                  ii  kopano-spooler-dbgsym                      8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-spooler
                  ii  kopano-utils                               8.6.81.198-0+35.1                          amd64        Admin command-line utils for Kopano Core
                  ii  kopano-utils-dbgsym                        8.6.81.198-0+35.1                          amd64        Debug symbols for kopano-utils
                  ii  kopano-webapp                              3.4.22.1724+981.1                          all          New and improved WebApp for Kopano
                  ii  kopano-webapp-plugin-contactfax            3.4.22.1724+981.1                          all          Kopano WebApp fax plugin
                  ii  kopano-webapp-plugin-desktopnotifications  2.0.1.5+16.2                               all          Kopano WebApp Desktop notifications plugin
                  ii  kopano-webapp-plugin-filepreviewer         2.0.0.15+10.8                              all          Kopano File previewer plugin
                  ii  kopano-webapp-plugin-files                 2.1.5.295+194.1                            all          Adds Files functionality to Kopano enabling access to WebDAV and other files backends.
                  ii  kopano-webapp-plugin-filesbackend-owncloud 2.1.1.84+61.1                              all          Adds Owncloud specific functionality to Kopano Files plugin.
                  ii  kopano-webapp-plugin-filesbackend-smb      2.1.1.48+49.1                              all          Adds Samba specific functionality to Kopano Files plugin.
                  ii  kopano-webapp-plugin-folderwidgets         3.4.22.1724+981.1                          all          Kopano WebApp folder widgets plugin
                  ii  kopano-webapp-plugin-gmaps                 3.4.22.1724+981.1                          all          Kopano WebApp google maps plugin
                  ii  kopano-webapp-plugin-intranet              0.1+32.8                                   all          This plugin adds one or more buttons in the top menu bar which can be used to open a webpage inside Kopano WebApp.
                  ii  kopano-webapp-plugin-mattermost            0.1+17.2                                   all          Integrates Mattermost into WebApp
                  ii  kopano-webapp-plugin-mdm                   2.1.0.106+26.1                             all          Kopano WebApp MDM plugin
                  ii  kopano-webapp-plugin-pimfolder             3.4.22.1724+981.1                          all          Kopano WebApp personal inbox plugin
                  ii  kopano-webapp-plugin-quickitems            3.4.22.1724+981.1                          all          Kopano WebApp quick items plugin
                  ii  kopano-webapp-plugin-spell                 2.0.0.17+34.1                              all          Kopano WebApp Spellchecker plugin
                  ii  kopano-webapp-plugin-spell-de-at           2.0.0.4+28.1                               all          Kopano WebApp Spellchecker German (Austrian) dictionary plugin
                  ii  kopano-webapp-plugin-spell-de-ch           2.0.0.5+27.1                               all          Kopano WebApp Spellchecker German (Swiss) dictionary plugin
                  ii  kopano-webapp-plugin-spell-de-de           2.0.0.3+28.1                               all          Kopano WebApp Spellchecker German dictionary plugin
                  ii  kopano-webapp-plugin-spell-en              2.0.0.1+28.1                               all          Kopano WebApp Spellchecker English dictionary plugin
                  ii  kopano-webapp-plugin-spell-en-gb           2.0.0.1+27.1                               all          Kopano WebApp Spellchecker English (GB) dictionary plugin
                  ii  kopano-webapp-plugin-spell-es              2.0.0.1+27.1                               all          Kopano WebApp Spellchecker Spanish dictionary plugin
                  ii  kopano-webapp-plugin-spell-fr              2.0.0.1+28.1                               all          Kopano WebApp Spellchecker French dictionary plugin
                  ii  kopano-webapp-plugin-spell-it              1.0.0+40.1                                 all          Kopano WebApp Spellchecker Italian dictionary plugin
                  ii  kopano-webapp-plugin-spell-nl              2.0.0.1+29.1                               all          Kopano WebApp Spellchecker Dutch dictionary plugin
                  ii  kopano-webapp-plugin-spell-pl-pl           2.0.0.0+33.1                               all          Kopano WebApp Spellchecker Polish dictionary plugin
                  ii  kopano-webapp-plugin-titlecounter          3.4.22.1724+981.1                          all          Kopano WebApp Titlecounter plugin
                  ii  kopano-webapp-plugin-webappmanual          3.4.22.1724+981.1                          all          Kopano WebApp Manual plugin
                  ii  kopano-webapp-plugin-zdeveloper            3.4.22.1724+981.1                          all          Kopano WebApp developer plugin
                  ii  libgsoap-kopano-2.8.62                     2.8.62-0+2.1                               amd64        Runtime libraries for gSOAP
                  ii  libvmime-kopano1                           0.9.2.50+11.1                              amd64        a C++ mail library (runtime library)
                  ii  libvmime-kopano1-dbgsym                    0.9.2.50+11.1                              amd64        Debug symbols for libvmime-kopano1
                  ii  python-kopano                              8.6.81.198-0+35.1                          all          High-level Python 2 bindings for Kopano
                  ii  python3-kopano                             8.6.81.198-0+35.1                          all          High-level Python 3 bindings for Kopano
                  ii  z-push-backend-kopano                      2.4.4+0-0                                  all          Z-Push Kopano backend
                  ii  z-push-kopano                              2.4.4+0-0                                  all          Z-Push for Kopano
                  
                  root@kopano:~# apt-cache policy kopano-server
                  kopano-server:
                    Installiert:           8.6.81.198-0+35.1
                    Installationskandidat: 8.6.81.198-0+35.1
                    Versionstabelle:
                   *** 8.6.81.198-0+35.1 100
                          100 /var/lib/dpkg/status
                  
                  

                  Hopefully this helps finding the problem.
                  Thank you very much in advance.

                  Regards,

                  Alex

                  P.S.: Actually there is always a high CPU usage of kopano-search, kopano-server and mysqld:

                  top - 00:03:01 up 2 days,  8:42,  1 user,  load average: 1,31, 1,38, 1,41
                  Tasks: 127 total,   1 running, 126 sleeping,   0 stopped,   0 zombie
                  %Cpu(s): 20,4 us,  0,9 sy,  0,0 ni, 69,3 id,  9,3 wa,  0,0 hi,  0,0 si,  0,0 st
                  KiB Mem :  8045376 total,   140868 free,  7740960 used,   163548 buff/cache
                  KiB Swap:  4143100 total,  3174024 free,   969076 used.    69448 avail Mem
                  
                     PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                   43607 kopano    20   0  481320  89756   5544 D  41,7  1,1 665:50.80 kopano-search
                   32208 kopano    20   0 8185608 6,879g    976 S  22,8 89,7  86:45.45 kopano-server
                     714 mysql     20   0  763904 170444      0 S  20,5  2,1   1322:28 mysqld
                   90822 root      20   0   44928   3648   2988 R   0,7  0,0   0:00.04 top
                     175 root       0 -20       0      0      0 S   0,3  0,0   0:46.38 kworker/3:1H
                       1 root      20   0  204692   3024   2056 S   0,0  0,0   0:24.93 systemd
                       2 root      20   0       0      0      0 S   0,0  0,0   0:00.07 kthreadd
                       3 root      20   0       0      0      0 S   0,0  0,0   0:08.63 ksoftirqd/0
                  

                  Perhaps this takes some days till index-process is over. All in all about 80 Gigabyte of attachments and MySQL DB was migrated before.

                  1 Reply Last reply Reply Quote 0
                  • Sniks6
                    Sniks6 last edited by

                    Just made an update to the actual nightly version:

                    root@kopano:~# apt-cache policy kopano-server
                    kopano-server:
                      Installiert:           8.6.81.228-0+37.1
                      Installationskandidat: 8.6.81.228-0+37.1
                      Versionstabelle:
                     *** 8.6.81.228-0+37.1 100
                            100 /var/lib/dpkg/status
                    

                    and:

                    root@kopano:~# dpkg -l | grep kopano
                    ii  kopano-backup                              8.6.81.228-0+37.1                          amd64        Utility to back up and restore Kopano stores
                    ii  kopano-bash-completion                     8.6.81.228-0+37.1                          amd64        bash TAB completion for Kopano Core command-line utilities
                    ii  kopano-client                              8.6.81.228-0+37.1                          amd64        Kopano MAPI provider library
                    ii  kopano-client-dbgsym                       8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-client
                    ii  kopano-common                              8.6.81.228-0+37.1                          amd64        Shared files for Kopano services
                    ii  kopano-contacts                            8.6.81.228-0+37.1                          amd64        MAPI provider adding contact folders in the addressbook
                    ii  kopano-dagent                              8.6.81.228-0+37.1                          amd64        E-Mail Delivery Agent for Kopano Core
                    ii  kopano-dagent-dbgsym                       8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-dagent
                    ii  kopano-dbg                                 8.6.81.228-0+37.1                          amd64        Debug symbols for Kopano Core
                    ii  kopano-dev                                 8.6.81.228-0+37.1                          amd64        C++ Development files for Kopano Core
                    ii  kopano-gateway                             8.6.81.228-0+37.1                          amd64        POP3 and IMAP Gateway for Kopano Core
                    ii  kopano-gateway-dbgsym                      8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-gateway
                    ii  kopano-ical                                8.6.81.228-0+37.1                          amd64        ICal and CalDAV Gateway for Kopano Core
                    ii  kopano-ical-dbgsym                         8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-ical
                    ii  kopano-lang                                8.6.81.228-0+37.1                          all          Translations for Kopano Core components
                    ii  kopano-migration-imap                      8.6.81.228-0+37.1                          amd64        Utility to migrate between IMAP mailboxes
                    ii  kopano-migration-pst                       8.6.81.228-0+37.1                          amd64        Utility to import PST files
                    ii  kopano-monitor                             8.6.81.228-0+37.1                          amd64        Quota Monitor for Kopano Core
                    ii  kopano-monitor-dbgsym                      8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-monitor
                    ii  kopano-presence                            8.6.81.228-0+37.1                          amd64        kopano presence daemon
                    ii  kopano-python-utils                        8.6.81.228-0+37.1                          amd64        Additional Python-based command-line utils for Kopano Core
                    ii  kopano-search                              8.6.81.228-0+37.1                          amd64        Indexed search engine for Kopano Core
                    ii  kopano-server                              8.6.81.228-0+37.1                          amd64        Server component for Kopano Core
                    ii  kopano-server-dbgsym                       8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-server
                    ii  kopano-server-packages                     8.6.81.228-0+37.1                          all          Metapackage to install the entire Kopano Core stack
                    ii  kopano-spamd                               8.6.81.228-0+37.1                          amd64        ICS-driven spam learning daemon for Kopano/SpamAssasin
                    ii  kopano-spooler                             8.6.81.228-0+37.1                          amd64        E-mail Spooler for Kopano Core
                    ii  kopano-spooler-dbgsym                      8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-spooler
                    ii  kopano-utils                               8.6.81.228-0+37.1                          amd64        Admin command-line utils for Kopano Core
                    ii  kopano-utils-dbgsym                        8.6.81.228-0+37.1                          amd64        Debug symbols for kopano-utils
                    ii  kopano-webapp                              3.4.22.1724+981.1                          all          New and improved WebApp for Kopano
                    ii  kopano-webapp-plugin-contactfax            3.4.22.1724+981.1                          all          Kopano WebApp fax plugin
                    ii  kopano-webapp-plugin-desktopnotifications  2.0.1.5+16.2                               all          Kopano WebApp Desktop notifications plugin
                    ii  kopano-webapp-plugin-filepreviewer         2.0.0.15+10.8                              all          Kopano File previewer plugin
                    ii  kopano-webapp-plugin-files                 2.1.5.295+194.1                            all          Adds Files functionality to Kopano enabling access to WebDAV and other files backends.
                    ii  kopano-webapp-plugin-filesbackend-owncloud 2.1.1.84+61.1                              all          Adds Owncloud specific functionality to Kopano Files plugin.
                    ii  kopano-webapp-plugin-filesbackend-smb      2.1.1.48+49.1                              all          Adds Samba specific functionality to Kopano Files plugin.
                    ii  kopano-webapp-plugin-folderwidgets         3.4.22.1724+981.1                          all          Kopano WebApp folder widgets plugin
                    ii  kopano-webapp-plugin-gmaps                 3.4.22.1724+981.1                          all          Kopano WebApp google maps plugin
                    ii  kopano-webapp-plugin-intranet              0.1+32.8                                   all          This plugin adds one or more buttons in the top menu bar which can be used to open a webpage inside Kopano WebApp.
                    ii  kopano-webapp-plugin-mattermost            0.1+17.2                                   all          Integrates Mattermost into WebApp
                    ii  kopano-webapp-plugin-mdm                   2.1.0.106+26.1                             all          Kopano WebApp MDM plugin
                    ii  kopano-webapp-plugin-pimfolder             3.4.22.1724+981.1                          all          Kopano WebApp personal inbox plugin
                    ii  kopano-webapp-plugin-quickitems            3.4.22.1724+981.1                          all          Kopano WebApp quick items plugin
                    ii  kopano-webapp-plugin-spell                 2.0.0.17+34.1                              all          Kopano WebApp Spellchecker plugin
                    ii  kopano-webapp-plugin-spell-de-at           2.0.0.4+28.1                               all          Kopano WebApp Spellchecker German (Austrian) dictionary plugin
                    ii  kopano-webapp-plugin-spell-de-ch           2.0.0.5+27.1                               all          Kopano WebApp Spellchecker German (Swiss) dictionary plugin
                    ii  kopano-webapp-plugin-spell-de-de           2.0.0.3+28.1                               all          Kopano WebApp Spellchecker German dictionary plugin
                    ii  kopano-webapp-plugin-spell-en              2.0.0.1+28.1                               all          Kopano WebApp Spellchecker English dictionary plugin
                    ii  kopano-webapp-plugin-spell-en-gb           2.0.0.1+27.1                               all          Kopano WebApp Spellchecker English (GB) dictionary plugin
                    ii  kopano-webapp-plugin-spell-es              2.0.0.1+27.1                               all          Kopano WebApp Spellchecker Spanish dictionary plugin
                    ii  kopano-webapp-plugin-spell-fr              2.0.0.1+28.1                               all          Kopano WebApp Spellchecker French dictionary plugin
                    ii  kopano-webapp-plugin-spell-it              1.0.0+40.1                                 all          Kopano WebApp Spellchecker Italian dictionary plugin
                    ii  kopano-webapp-plugin-spell-nl              2.0.0.1+29.1                               all          Kopano WebApp Spellchecker Dutch dictionary plugin
                    ii  kopano-webapp-plugin-spell-pl-pl           2.0.0.0+33.1                               all          Kopano WebApp Spellchecker Polish dictionary plugin
                    ii  kopano-webapp-plugin-titlecounter          3.4.22.1724+981.1                          all          Kopano WebApp Titlecounter plugin
                    ii  kopano-webapp-plugin-webappmanual          3.4.22.1724+981.1                          all          Kopano WebApp Manual plugin
                    ii  kopano-webapp-plugin-zdeveloper            3.4.22.1724+981.1                          all          Kopano WebApp developer plugin
                    ii  libgsoap-kopano-2.8.62                     2.8.62-0+2.1                               amd64        Runtime libraries for gSOAP
                    ii  libvmime-kopano1                           0.9.2.50+11.1                              amd64        a C++ mail library (runtime library)
                    ii  libvmime-kopano1-dbgsym                    0.9.2.50+11.1                              amd64        Debug symbols for libvmime-kopano1
                    ii  python-kopano                              8.6.81.228-0+37.1                          all          High-level Python 2 bindings for Kopano
                    ii  python3-kopano                             8.6.81.228-0+37.1                          all          High-level Python 3 bindings for Kopano
                    ii  z-push-backend-kopano                      2.4.4+0-0                                  all          Z-Push Kopano backend
                    ii  z-push-kopano                              2.4.4+0-0                                  all          Z-Push for Kopano
                    

                    Let’s see if this changes anything.
                    I’m still not completely desperate :-)

                    Regards,

                    Alex

                    1 Reply Last reply Reply Quote 0
                    • Sniks6
                      Sniks6 last edited by

                      Unfortunately not. kopano-server still crashing every 4 - 5 hours after flooding the whole RAM and SWAP memory. :-(

                      1 Reply Last reply Reply Quote 0
                      • Coffee_is_life
                        Coffee_is_life last edited by

                        hello @Sniks6,

                        i had a similar issue on a testsystem, resolving was quite questionable :)
                        first i made the mariadb tuning for the system, second i reduce the index-process to 1 worker for first indexing.
                        then i only started search, wait 5 minutes till it complains about missing kopano-server, then start it, without gateway or dagent.
                        Wait for the first indexing, this worked fine in my case but used a lot of time.

                        If this works, stop everything, incrase the indexing worker to standard value (i think it was 5) and start everything.

                        The stats of this vm where:
                        VMware machine
                        4 cores
                        8GB ram
                        scsi sas drives
                        CentOS 7 64bit
                        mariadb - 5.5.60-1
                        kopano-server - 8.6.4.0
                        no webapp, since its an external machine

                        regards,

                        Coffee_is_life

                        1 Reply Last reply Reply Quote 0
                        • fbartels
                          fbartels Kopano last edited by

                          Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.

                          depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?

                          8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?

                          Regards Felix

                          Resources:
                          https://kopano.com/blog/how-to-get-kopano/
                          https://documentation.kopano.io/
                          https://kb.kopano.io/

                          Support overview:
                          https://kopano.com/support/

                          1 Reply Last reply Reply Quote 0
                          • Sniks6
                            Sniks6 last edited by

                            Hi @ all,

                            Thank you very much for your anwers!

                            Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.

                            That’s true. Perhaps I just didn’t start all services at the time I made the mysqldump.

                            depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?

                            You’re right. The search service always uses much cpu und I think after two weeks it still hasn’t finished the indexing. :-(
                            I stopped kopano-search 2 hours ago and at the moment the cpu and memory usage is very low:

                            top - 15:24:59 up 6 days, 4 min,  3 users,  load average: 0,03, 0,06, 0,01
                            Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
                            %Cpu(s):  0,2 us,  0,3 sy,  0,0 ni, 99,5 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
                            KiB Mem :  8045376 total,  5219748 free,   564032 used,  2261596 buff/cache
                            KiB Swap:  4143100 total,  4040092 free,   103008 used.  7173136 avail Mem
                            
                               PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                            158017 mysql     20   0  726648 243176  17020 S   0,0  3,0   1:32.18 mysqld
                            161987 kopano    20   0  360232  71796  18200 S   0,0  0,9   0:12.51 kopano-server
                            165057 www-data  20   0  734928  43668  35932 S   0,0  0,5   0:00.76 apache2
                            163614 www-data  20   0  740548  43224  33404 S   0,0  0,5   0:00.92 apache2
                            108383 www-data  20   0  740392  43132  33628 S   0,3  0,5   0:02.50 apache2
                            164767 www-data  20   0  735112  42924  34632 S   0,0  0,5   0:00.82 apache2
                            168432 www-data  20   0  735064  42452  34652 S   0,0  0,5   0:00.52 apache2
                            164391 www-data  20   0  739752  42276  33036 S   0,0  0,5   0:00.64 apache2
                            165992 www-data  20   0  735012  40456  33280 S   0,0  0,5   0:00.50 apache2
                            168150 www-data  20   0  734292  40408  33536 S   0,0  0,5   0:00.36 apache2
                            103010 www-data  20   0  734872  40388  32832 S   0,0  0,5   0:02.11 apache2
                            164770 www-data  20   0  734660  40136  33020 S   0,0  0,5   0:00.64 apache2
                            165990 www-data  20   0  734628  38988  32424 S   0,0  0,5   0:00.44 apache2
                            170618 www-data  20   0  666752  33664  28192 S   0,0  0,4   0:00.02 apache2
                            170622 www-data  20   0  666752  33664  28192 S   0,0  0,4   0:00.02 apache2
                            162160 kopano    20   0  230108  20472  18368 S   0,0  0,3   0:00.24 kopano-spooler
                            162138 kopano    20   0  160660  16596  14668 S   0,0  0,2   0:00.08 kopano-dagent
                               247 root      20   0   70680   9724   9492 S   0,0  0,1   1:38.56 systemd-journal
                            146108 root      20   0  101436   6904   5900 S   0,0  0,1   0:00.22 sshd
                            170192 root      20   0  101432   6856   5852 S   0,0  0,1   0:00.22 sshd
                            

                            8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?

                            I already changed the cache to:

                            cache_cell_size = 2G
                            

                            Unfortunately without success.

                            As of now with the disabled kopano-search it looks good. Is there a way to find out what causes this mem flooding when indexing by kopano-search?

                            Regards,
                            Alex

                            1 Reply Last reply Reply Quote 0
                            • Sniks6
                              Sniks6 last edited by

                              Good Morning everyone,

                              now, after 17 hours it looks good kopano-server did not crash since then.
                              So I think we found the problem: kopano-search is causing the memory flood in kopano-server.

                              But why? As Coffee_is_life wrote on his setup after completing the index process everything worked fine. Perhaps this is the same with my setup. But my indexing is running for two weeks now. :-(

                              Do you have any ideas what to tweek in this case?

                              1 Reply Last reply Reply Quote 0
                              • Sniks6
                                Sniks6 last edited by

                                After 4 days without kopano-search kopano-server did not crash anymore.
                                But the memory keeps still rising much more slowly…

                                0_1537712475900_image.png

                                Any thoughts?

                                1 Reply Last reply Reply Quote 0
                                • Gerald
                                  Gerald last edited by Gerald

                                  I also (still?) have this problem. Currently using 8.6.81.171-0+25.1 on Ubuntu 18.04 x64.

                                  [Fri Sep 28 19:51:29 2018] Out of memory: Kill process 1240 (kopano-server) score 902 or sacrifice child
                                  [Fri Sep 28 19:51:29 2018] Killed process 1240 (kopano-server) total-vm:26514968kB, anon-rss:10946196kB, file-rss:0kB, shmem-rss:0kB

                                  By the time it had crashed it was running only for 11 hours or so - after having been restarted due to a kopano-server crash the day before. The server should have enough memory for normal use (12GB) and an additional 16GB Swap is available. Especially with the swap I really see no reason at all for it to run out of memory.

                                  I will now try to update to 8.6.81.435-0+75.1 in the hope that this out of memory bug has been fixed in the meantime?
                                  In addition I’ve also cleaned the /var/lib/kopano/search/ directory and restarted the indexing process. Maybe index is corrupted?


                                  Update: Has not changed/fixed anything. Server still crashing about once a day. Redoing the index didn’t help, neither did disabling the indexing of attachments.

                                  I’ve added the following to the kopano-server.service file for a temporary fix:

                                  [add in the service section]
                                  Restart=on-failure
                                  RestartSec=10s

                                  [add in the unit section]
                                  StartLimitIntervalSec=300s
                                  StartLimitBurst=2

                                  1 Reply Last reply Reply Quote 0
                                  • Sniks6
                                    Sniks6 last edited by

                                    Thank you for your message.

                                    Here is a graph of the memory consumption:
                                    0_1538403954418_memoryusage01.png

                                    • In the first half you see the memory rising very quickly till the server process crashed. It restarted automatically after modfying the kopano-server.service file.
                                    • In the second half (~ week 38) I didn’t start the kopano-search process. The memory is still rising but much slower.

                                    So I think it is not a problem with the search process. It seems that the search process simply speeds up the rising of the memory usage. If I had more users or much more interaction with the server process, perhaps the memory would be rising faster again.

                                    So it looks like a problem in the kopano-server.

                                    Just for interest, what hardware do you use, Gerald? Is it virtualised hardware? If yes which kind (VMware, HyperV,…)?

                                    Regards,

                                    Alex

                                    Gerald 1 Reply Last reply Reply Quote 0
                                    • Gerald
                                      Gerald @Sniks6 last edited by Gerald

                                      @sniks6 said in Kopano-server crashes:

                                      Just for interest, what hardware do you use, Gerald? Is it virtualised hardware? If yes which kind (VMware, HyperV,…)?

                                      I’m using VMware ESXi (Free Edition), currently version 6.5.0 U1 (Build 7967591). The VM has the open-vm-tools package installed.
                                      I don’t think the behavior would change on physical hardware though.

                                      Update: crashed again:
                                      [Tue Oct 2 12:01:22 2018] Out of memory: Kill process 2038 (kopano-server) score 848 or sacrifice child
                                      but my modifications to automatically restart the process worked so I guess nobody even noticed.

                                      Just 6 hours later memory usage for the kopano-server process was again up to 5.6GB, after restarting kopano-server memory usage is down to 350MB. This thing is leaking like crazy! At this rate even throwing 32GB at it would not really help for long.

                                      Update2: regarding kopano on physical hardware
                                      I just checked the kopano I’m using for my personal email, which is on physical hardware (Atom-based Quadcore NUC with 8GB RAM) and here kopano-server is hogging about 3GB of memory after running for 48 hours under low load (about 3 users with 3 devices each). Also an Ubuntu 18.04 system by the way.

                                      1 Reply Last reply Reply Quote 0
                                      • Sniks6
                                        Sniks6 last edited by

                                        @Gerald: Thanks for your infos. :-)

                                        Isn’t there anybody who could give us a hint, where we could tweak or check what could cause this memory problem?

                                        fbartels 1 Reply Last reply Reply Quote 0
                                        • fbartels
                                          fbartels Kopano @Sniks6 last edited by

                                          @sniks6 said in Kopano-server crashes:

                                          Isn’t there anybody who could give us a hint, where we could tweak or check what could cause this memory problem?

                                          This does not look like there is something to tweak, but this rather is a memory leak somewhere in the master branch. We want to branch of a release soon, so we eventually will need to look into it.

                                          I’ve created https://jira.kopano.io/browse/KC-1299 to track progress.

                                          Regards Felix

                                          Resources:
                                          https://kopano.com/blog/how-to-get-kopano/
                                          https://documentation.kopano.io/
                                          https://kb.kopano.io/

                                          Support overview:
                                          https://kopano.com/support/

                                          1 Reply Last reply Reply Quote 0
                                          • Sniks6
                                            Sniks6 last edited by

                                            Thanks for your message, Felix.
                                            If it is helpful, I could provide you more info from my system.

                                            I am happy to help.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post