Kopano-server crashes
-
Thanks four your answer. I’m using a Windows 2016 Server HyperV Core for virtualisation.
Is there a way I could reduce the number of kopano-server processes? The start immediately after running “systemctl start kopano-server”.
Btw it’s the same with the mysqld processes. They all start right after running “systemctl start mysql”. Hmmm…Kopano config is pretty standard. Here you go:
# If a directive is not used (i.e. commented out), the built-in server default # is used, so to disable certain features, the empty string value must explicitly be # set on them. ############################################################## # SERVER SETTINGS # Space-separated list of address:port specifiers for where the server should # listen for unencrypted connections. # # "*:236" — port 236, all protocols # "[::]:236" — port 236 on IPv6 only # "[2001:db8::1]:236" — port 236 on specific address only # To disable unencrypted connections, set to an empty value. # server_listen = *:236 # Space-separated list of paths for where to listen for AF_LOCAL # connections. # To disable, set to an empty value. # #server_pipe_name = /var/run/kopano/server.sock # Space-separated list of paths for where to listen for AF_LOCAL # connections to handle with raised priority. # To disable, set to an empty value. # #server_pipe_priority = /var/run/kopano/prio.sock # Name for identifying the server in a multi-server environment #server_name = # Override the hostname of this server, used by Kerberos SSO if enabled #server_hostname = # Database engine (mysql) #database_engine = mysql # Allow connections from normal users through the Unix socket #allow_local_users = yes # local admin users who can connect to any store (use this for the kopano-dagent) # field is SPACE separated # eg: local_admin_users = root vmail # Default: root local_admin_users = root kopano # With owner_auto_full_access=true, the owner of an object is, in addition to # any ACLs, implicitly granted "read", "create-in" and "is-visible" rights, # even if the store belongs to another user, for example, when it has been # moved via kopano-storeadm -D,-A. # #owner_auto_full_access = true # e-mail address of the Kopano System user #system_email_address = postmaster@localhost # drop privileges and run the process as this user #run_as_user = kopano # drop privileges and run the process as this group #run_as_group = kopano # create a pid file for stopping the service via the init.d scripts #pid_file = /var/run/kopano/server.pid # run server in this path (when not using the -F switch) #running_path = /var/lib/kopano/empty # Use given allocator library. Values like libtcmalloc.so.4, # libtcmalloc_minimal.so.4 and libjemalloc.so.2 would work. #allocator_library = libtcmalloc_minimal.so.4 # create memory coredumps upon crash [no, systemdefault, yes] #coredump_enabled = systemdefault # session timeout for clients. Values lower than 300 will be upped to 300 # automatically. If the server hears nothing from a client in session_timeout # seconds, then the session is killed. #session_timeout = 300 # for temporary files #tmp_path = /tmp ############################################################## # LOG SETTINGS # Logging method (syslog, file), syslog facility is 'mail' #log_method = auto # Logfile (for log_method = file, '-' for stderr) #log_file = - # Loglevel (0(none), 1(crit), 2(err), 3(warn), 4(notice), 5(info), 6(debug)) #log_level = 3 log_level = 5 # Log timestamp - prefix each log line with timestamp in 'file' logging mode #log_timestamp = yes # Buffer logging in what sized blocks. 0 for line-buffered (syslog-style). #log_buffer_size = 0 ############################################################## # AUDIT LOG SETTINGS # Audit logging is by default not enabled #audit_log_enabled = no # Audit logging method (syslog, file), syslog facility is 'authpriv' #audit_log_method = syslog # Audit logfile (for log_method = file, '-' for stderr) #audit_log_file = /var/log/kopano/audit.log # Audit loglevel (0=no logging, 1=full logging) #audit_log_level = 1 # Audit log timestamp - prefix each log line with timestamp in 'file' logging mode #audit_log_timestamp = 1 ############################################################## # MYSQL SETTINGS (for database_engine = mysql) # MySQL hostname to connect to for database access mysql_host = localhost # MySQL port to connect with (usually 3306) mysql_port = 3306 # The user under which we connect with MySQL mysql_user = kopano # The password for the user (leave empty for no password) mysql_password = XXXXXXXXXXXXX # Override the default MySQL socket to access mysql locally # Works only if the mysql_host value is empty or 'localhost' #mysql_socket = # Database to connect to mysql_database = kopano # MySQL engine, per default it is InnoDB, which is the only supported # database engine. It is NOT advised to use a non-transactional engine # like MyISAM, Aria, etc. #mysql_engine = InnoDB # Where to place attachments. Value can be 'database', 'files' or 's3' attachment_storage = files # Enable fsync as method to make sure attachments are stored on disk where # supported and will not be buffered by OS and/or filesystem. Please note # this setting will lower attachment write performance depending on your # environment but enhances data safety with disaster recovery. # Only affects 'files' attachment storage backend. #attachment_files_fsync = yes # When attachment_storage is 'files', use this path to store the files # When attachment_storage is 's3', use this path to set a prefix to all # attachment data of a certain cluster, for example 'attach' attachment_path = /var/lib/kopano/attachments # Compression level for attachments when attachment_storage is 'files'. # Set compression level for attachments disabled=0, max=9 attachment_compression = 6 ############################################################## # S3 STORAGE SETTINGS (for attachment_storage = s3) # The hostname of the entry point to the S3 cloud where the bucket is located # If you are using minio or another S3 compatible implementation that # is using another port, you can specify the port with hostname:port. #attachment_s3_hostname = s3-eu-west-1.amazonaws.com # The region where the bucket is located, e.g. "eu-west-1" #attachment_s3_region = # The protocol that should be used to connect to S3, 'http' or 'https' (preferred) #attachment_s3_protocol = # The URL style of the bucket, "virtualhost" or "path" #attachment_s3_uristyle = # The access key id of your S3 account #attachment_s3_accesskeyid = # The secret access key of your S3 account #attachment_s3_secretaccesskey = # The bucket name in which the files will be stored #attachment_s3_bucketname = ############################################################## # SSL SETTINGS # Space-separated list of address:port specifiers for where to listen for # TLS-encrypted connections. # # "*:237" — port 237, all protocols # "[::]:237" — port 237 on IPv6 only # "[2001:db8::1]:237" — port 237 on specific address only # To disable encrypted connections, set to an empty value. # #server_listen_tls = # Required Server certificate, contains the certificate and the private key parts #server_ssl_key_file = /etc/kopano/ssl/server.pem # Password of Server certificate #server_ssl_key_pass = # Required Certificate Authority of server #server_ssl_ca_file = /etc/kopano/ssl/cacert.pem # Path with CA certificates, e.g. /etc/ssl/certs #server_ssl_ca_path = # SSL protocols to use, space-separated list of protocols # (SSLv3 TLSv1 TLSv1.1 TLSv1.2); prefix with ! to lock out a protocol. #server_ssl_protocols = # SSL ciphers to use, set to 'ALL' for backward compatibility #server_ssl_ciphers = ALL:!LOW:!SSLv2:!EXP:!aNULL # Prefer the server's order of SSL ciphers over client's #server_ssl_prefer_server_ciphers = no # Path of SSL Public keys of clients #sslkeys_path = /etc/kopano/sslkeys ############################################################## # THREAD SETTINGS # Number of server threads #threads = 8 # Watchdog frequency. The number of watchdog checks per second. #watchdog_frequency = 1 # Watchdog max age. The maximum age in ms of a task before a # new thread is started. #watchdog_max_age = 500 # SOAP recv timeout value (time between requests) #server_recv_timeout = 5 # SOAP read timeout value (time during requests) #server_read_timeout = 60 # SOAP send timeout value #server_send_timeout = 60 ############################################################## # OTHER SETTINGS # Softdelete clean cycle (in days) 0=never running #softdelete_lifetime = 30 # Sync lifetime, removes all changes remembered for a client after x days of inactivity #sync_lifetime = 90 # Set to 'yes' if you have Kerberos, NTLM or OpenID Connect correctly configured for single sign-on #enable_sso = no # Set to 'yes' if you want to show the GAB to your users #enable_gab = yes # Authentication can be through plugin (default, recommended), pam or kerberos #auth_method = plugin # If auth_method is set to pam, you should provide the pam service name #pam_service = passwd ############################################################# # CACHE SETTINGS # # To see the live cache usage, use 'kopano-stats --system'. # Size in bytes of the 'cell' cache (should be set as high as you can afford to set it) #cache_cell_size = 256M # Size in bytes of the 'object' cache #cache_object_size = 16M # Size in bytes of the 'indexed object' cache #cache_indexedobject_size = 32M # Size in bytes of the userquota details #cache_quota_size = 1M # Lifetime for userquota details #cache_quota_lifetime = 1 # Size in bytes of the acl cache #cache_acl_size = 1M # Size in bytes of the store id/guid cache #cache_store_size = 1M # Size in bytes of the 'user id' cache (this is allocated twice) #cache_user_size = 1M # Size in bytes of the 'user details' cache #cache_userdetails_size = 25M # Lifetime for user details #cache_userdetails_lifetime = 0 # Size in bytes of the server details (multiserver setups only) #cache_server_size = 1M # Lifetime for server details (multiserver setups only) #cache_server_lifetime = 30 ############################################################## # QUOTA SETTINGS # The default Warning Quota Level. Set to 0 to disable this level. # The user will receive an email when this level is reached. Value is in Mb. Default value is 0. #quota_warn = 0 # The default Soft Quota Level. Set to 0 to disable this level. # The user will still receive mail, but sending new mail is prohibited, until objects are removed from the store. # VALUE is in Mb. Default value is 0. #quota_soft = 0 # The default Hard Quota Level. Set to 0 to disable this level. # The user can not receive and send mail, until objects are removed from the store. # Value is in Mb. Default value is 0. #quota_hard = 0 # The default Warning Quota Level for multitenant public stores. Set to 0 to disable this level. # The tenant administrator will receive an email when this level is reached. Value is in Mb. Default value is 0. #companyquota_warn = 0 ############################################################## # USER PLUGIN SETTINGS # Name of the plugin that handles users # Required, default = db # Values: ldap, unix, db #user_plugin = db # configuration file of the user plugin, examples can be found in /usr/share/doc/kopano/example-config #user_plugin_config = /etc/kopano/ldap.cfg # scripts which create stores for users from an external source # used for ldap and unix plugins only #createuser_script = /usr/lib/kopano/userscripts/createuser #deleteuser_script = /usr/lib/kopano/userscripts/deleteuser #creategroup_script = /usr/lib/kopano/userscripts/creategroup #deletegroup_script = /usr/lib/kopano/userscripts/deletegroup #createcompany_script = /usr/lib/kopano/userscripts/createcompany #deletecompany_script = /usr/lib/kopano/userscripts/deletecompany # Set this option to 'yes' to skip the creation and deletion of new users # The action will be logged, so you can see if your changes to the plugin # configuration are correct. #user_safe_mode = no ############################################################## # OPENID CONNECT SETTINGS # Enable OpenID Connect Issuer Identifier # When set, the server attempts OIDC discovery using the configured issuer # identifier on startup. An Issuer Identifier is a case sensitive URL using the # https scheme that contains scheme, host, and optionally, port number and path # components. This no Issuer Identifier is set, OIDC support is disabled. #kcoidc_issuer_identifier = # Disable TLS validation for OpenID Connect requests # When set to yes, TLS certificate validation is skipped for all requests # related to OpenID connect. This is insecure and should not be used in # production setups. #kcoidc_insecure_skip_verify = no # Timeout in seconds when to give up OpenID Connect discovery # When the OIDC initialize timeout is reached, server continues startup without # OIDC and all OIDC validation will fail until the discovery completes. When # set to 0, the server startup does not wait for OIDC discovery at all. #kcoidc_initialize_timeout = 60 ############################################################## # MISC SETTINGS # Enable multi-tenancy environment # When set to true it is possible to create tenants within the # kopano instance and assign all users and groups to particular # tenants. # When set to false, the normal single-tenancy environment is created. #enable_hosted_kopano = false # Enable multi-server environment # When set to true it is possible to place users and tenants on # specific servers. # When set to false, the normal single-server environment is created. #enable_distributed_kopano = false # Display format of store name # Allowed variables: # %u Username # %f Full name # %c Tenant's name #storename_format = %f # Loginname format (for Multi-tenancy installations) # When the user does not login through a system-wide unique # username (like the email address) a unique name is created # by combining the username and the tenantname. # With this configuration option you can set how the # loginname should be built up. # # Note: Do not use the = character in the format. # # Allowed variables: # %u Username # %c Teantname # #loginname_format = %u # Everyone is a special internal group, which contains every user and group # You may want to disable this group from the Global Addressbook by setting # this option to 'yes'. Administrators will still be able to see the group. #hide_everyone = no # System is a special internal user, which has super-admin privileges # You may want to disable this user from the Global Addressbook by setting # this option to 'yes'. Administrators will still be able to see the user. #hide_system = yes # Use Indexing service for faster searching. # Enabling this option requires the kopano-search service to # be running. #search_enabled = yes # Path to the kopano-search service, this option is only required # if the server is going to make use of the indexing service. #search_socket = file:///var/run/kopano/search.sock # Time (in seconds) to wait for a connection to the kopano-search service # before terminating the indexed search request. #search_timeout = 10 # Allow enhanced ICS operations to speedup synchronization with cached profiles. #enable_enhanced_ics = yes # SQL Procedures allow for some optimized queries when streaming with enhanced ICS. # This is default disabled because you must set 'thread_stack = 256k' in your # MySQL server config under the [mysqld] tag and restart your MySQL server. #enable_sql_procedures = no # Synchronize GAB users on every open of the GAB (otherwise, only on # kopano-admin --sync) #sync_gab_realtime = yes # Disable features for users. This list is space separated. # Currently valid values: imap pop3 mobile outlook webapp #disabled_features = imap pop3 disabled_features = pop3 # Maximum number of deferred records in total #max_deferred_records = 0 # Maximum number of deferred records per folder #max_deferred_records_folder = 20 # Restrict the permissions that admins receive to folder permissions only. Please # read the server.cfg manpage before enabling this option so you really understand # the implications #restrict_admin_permissions = no # The maximum level of attachment recursion; Defines the number of # attachment-in-attachment in-attachment levels are allowed when saving and # replicating objects in the database. If you really want a higher level of # recursion than about 20, you probably have to increase MySQL's stack_size # to allow replication to work properly. #embedded_attachment_limit = 20 # Header to detect whether a connection has been received through a proxy. The # value of the header is not inspected. If the header exists then the connection # is taken to be received via a proxy. An empty value disables proxy detection # and the value of '*' is used to indicate that all connections are proxied #proxy_header = # Enable/disable reminders for shared stores #shared_reminders = yesAnd here is the debian 9 standard /etc/mysql/mariadb.conf.d/50-server.cnf:
# # These groups are read by MariaDB server. # Use it for options that only the server (but not clients) should see # # See the examples of server my.cnf files in /usr/share/mysql/ # # this is read by the standalone daemon and embedded servers [server] # this is only for the mysqld standalone daemon [mysqld] # # * Basic Settings # user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql skip-external-locking # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. bind-address = 127.0.0.1 # # * Fine Tuning # key_buffer_size = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 # This replaces the startup script and checks MyISAM tables if needed # the first time they are touched myisam_recover_options = BACKUP #max_connections = 100 #table_cache = 64 #thread_concurrency = 10 # # * Query Cache Configuration # query_cache_limit = 1M query_cache_size = 16M # # * Logging and Replication # # Both location gets rotated by the cronjob. # Be aware that this log type is a performance killer. # As of 5.1 you can enable the log at runtime! #general_log_file = /var/log/mysql/mysql.log #general_log = 1 # # Error log - should be very few entries. # log_error = /var/log/mysql/error.log # # Enable the slow query log to see queries with especially long duration #slow_query_log_file = /var/log/mysql/mariadb-slow.log #long_query_time = 10 #log_slow_rate_limit = 1000 #log_slow_verbosity = query_plan #log-queries-not-using-indexes # # The following can be used as easy to replay backup logs or for replication. # note: if you are setting up a replication slave, see README.Debian about # other settings you may need to change. #server-id = 1 #log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 100M #binlog_do_db = include_database_name #binlog_ignore_db = exclude_database_name # # * InnoDB # # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/. # Read the manual for more InnoDB related options. There are many! # # * Security Features # # Read the manual, too, if you want chroot! # chroot = /var/lib/mysql/ # # For generating SSL certificates you can use for example the GUI tool "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pem # # Accept only connections using the latest and most secure TLS protocol version. # ..when MariaDB is compiled with OpenSSL: # ssl-cipher=TLSv1.2 # ..when MariaDB is compiled with YaSSL (default in Debian): # ssl=on # # * Character sets # # MySQL/MariaDB default is Latin1, but in Debian we rather default to the full # utf8 4-byte character set. See also client.cnf # character-set-server = utf8mb4 collation-server = utf8mb4_general_ci # # * Unix socket authentication plugin is built-in since 10.0.22-6 # # Needed so the root database user can authenticate without a password but # only when running as the unix root user. # # Also available for other users if required. # See https://mariadb.com/kb/en/unix_socket-authentication-plugin/ # this is only for embedded server [embedded] # This group is only read by MariaDB servers, not by MySQL. # If you use the same .cnf file for MySQL and MariaDB, # you can put MariaDB-only options here [mariadb] # This group is only read by MariaDB-10.1 servers. # If you use the same .cnf file for MariaDB of different versions, # use this group for options that older servers don't understand [mariadb-10.1]Thanks again!
Regards,
Alex
-
After a little research I found out that htop is showing all threads of one process by default.
Here is a picture of htop after diabling it. Just one process as you said. :-)
-
Hej Alex.
Okay, this looks pretty simple ;-)
I don’t have much XP with HyperV. Maybe there is a bottleneck within.
Some other questions first…- Which hardware (SCSI, SAS, FC, SATA or iSCSI? RAIDn?) is underlaying for use with the VMs? Specially for Kopano?
- Had HyperV created one single file for the VM? (i think yes) …thinprovisioned? This meen “sparse files”?
- And, how many users do you have on the system?
-
Hi Marcus,
at the moment everthing looks good. The memory usage will stay in normal range.
To answer your questions:
Which hardware (SCSI, SAS, FC, SATA or iSCSI? RAIDn?) is underlaying for use with the VMs? Specially for Kopano?
Intel RAID Controller RS2BL040
3 x WDC WD5003ABYX 0 500GB HDDs
RAID 5Had HyperV created one single file for the VM? (i think yes) …thinprovisioned? This meen “sparse files”?
Yes, one VHDX file
And, how many users do you have on the system?
Just three users. It’s a small system for my business-partner and my wife. :-)
Anyways the MariaDB shows 5GB in phpmyadmin the attachements folder is 84GB:root@kopano:~# du -sh /var/lib/kopano/attachments 84G /var/lib/kopano/attachmentsI moved from an old Zarafa installation via kopano-migration-imap to the kopano system.
That’s why the DB is that big right from the start.But just as I said , at the moment the system runs good.
-
… and then it happened again. This afternoon:
. . . Sep 9 17:14:01 kopano kopano-server[947]: Start syncedmessages table clean up Sep 9 17:14:01 kopano kopano-server[947]: syncedmessages table clean up done, 0 entries removed Sep 9 17:14:39 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für (37985430 Bytes). Sep 9 17:14:39 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes). Sep 9 17:14:40 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für (371923599 Bytes). Sep 9 17:15:41 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für (37985430 Bytes). Sep 9 17:15:41 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes). Sep 9 17:15:42 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für (371923599 Bytes). Sep 9 17:16:43 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für (37985430 Bytes). Sep 9 17:16:43 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes). Sep 9 17:16:44 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für (371923599 Bytes). Sep 9 17:17:45 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für (37985430 Bytes). Sep 9 17:17:45 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes). Sep 9 17:17:46 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für (371923599 Bytes). Sep 9 17:18:12 kopano kopano-server[947]: ECFileAttachment::LoadAttachmentInstance(SOAP): Out of memory while reading "/var/lib/kopano/attachments/1/1/56811.gz" Sep 9 17:18:12 kopano kopano-server[947]: SerializeObject failed with error code 0x80000028 for object 128416 Sep 9 17:18:29 kopano kopano-server[947]: ECFileAttachment::LoadAttachmentInstance(SOAP): Out of memory while reading "/var/lib/kopano/attachments/6/7/56876.gz" Sep 9 17:18:29 kopano kopano-server[947]: SerializeObject failed with error code 0x80000028 for object 128522 Sep 9 17:18:41 kopano kopano-server[947]: ---------------------------------------------------------------------- Sep 9 17:18:41 kopano kopano-server[947]: Fatal error detected. Please report all following information. Sep 9 17:18:41 kopano kopano-server[947]: Application kopano-server version: 8.6.81 Sep 9 17:18:41 kopano kopano-server[947]: OS: Linux, release: 4.9.0-8-amd64, version: #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21), hardware: x86_64 Sep 9 17:18:41 kopano kopano-server[947]: Thread name: ECThreadPool Sep 9 17:18:41 kopano kopano-server[947]: Peak RSS: 7316596 Sep 9 17:18:41 kopano kopano-server[947]: Pid 947 caught SIGABRT (6), out of memory or unhandled exception, traceback: Sep 9 17:18:41 kopano kopano-server[947]: Backtrace: Sep 9 17:18:41 kopano kopano-server[947]: #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x472c0) [0x7f3a273582c0] Sep 9 17:18:41 kopano kopano-server[947]: #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x362ed) [0x7f3a273472ed] Sep 9 17:18:41 kopano kopano-server[947]: #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x1bd) [0x7f3a273475cd] Sep 9 17:18:41 kopano kopano-server[947]: #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f3a21e3d0c0] Sep 9 17:18:41 kopano kopano-server[947]: #4. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7f3a21222fff] Sep 9 17:18:41 kopano kopano-server[947]: #5. /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f3a2122442a] Sep 9 17:18:41 kopano kopano-server[947]: #6. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x15d) [0x7f3a21b3b0ad] Sep 9 17:18:41 kopano kopano-server[947]: #7. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f066) [0x7f3a21b39066] Sep 9 17:18:41 kopano kopano-server[947]: #8. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f0b1) [0x7f3a21b390b1] Sep 9 17:18:41 kopano kopano-server[947]: #9. /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8f2c9) [0x7f3a21b392c9] Sep 9 17:18:41 kopano kopano-server[947]: #10. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x37ef2) [0x7f3a26f4eef2] Sep 9 17:18:41 kopano kopano-server[947]: #11. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x5320f) [0x7f3a26f6a20f] Sep 9 17:18:41 kopano kopano-server[947]: #12. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x55eee) [0x7f3a26f6ceee] Sep 9 17:18:41 kopano kopano-server[947]: #13. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x1242d7) [0x7f3a2703b2d7] Sep 9 17:18:41 kopano kopano-server[947]: #14. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x12751c) [0x7f3a2703e51c] Sep 9 17:18:41 kopano kopano-server[947]: #15. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x188f9b) [0x7f3a2709ff9b] Sep 9 17:18:41 kopano kopano-server[947]: #16. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC14ECWaitableTask7executeEv+0x84) [0x7f3a27349c04] Sep 9 17:18:41 kopano kopano-server[947]: #17. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x3929c) [0x7f3a2734a29c] Sep 9 17:18:41 kopano kopano-server[947]: #18. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f3a21e33494] Sep 9 17:18:41 kopano kopano-server[947]: #19. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3a212d8acf] Sep 9 17:18:41 kopano kopano-server[947]: Signal errno: Erfolg, signal code: -6 Sep 9 17:18:41 kopano kopano-server[947]: Sender pid: 947, sender uid: 999, si_status: 0 Sep 9 17:18:41 kopano kopano-server[947]: Signal value: 0, faulting address: 0x3e7000003b3 Sep 9 17:18:41 kopano kopano-server[947]: When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version. Sep 9 17:18:46 kopano fetchmail[936]: 717 Nachrichten (717 gesehene) für (37985430 Bytes). Sep 9 17:18:47 kopano fetchmail[936]: 389 Nachrichten (389 gesehene) für (110571564 Bytes). Sep 9 17:18:47 kopano fetchmail[936]: 488 Nachrichten (488 gesehene) für (371923599 Bytes). Sep 9 17:19:34 kopano kopano-spooler[514]: Unable to get outgoing queue count: network error (80040115) Sep 9 17:19:34 kopano kopano-spooler[514]: ProcessQueue: ProcessAllEntries failed: network error (80040115) Sep 9 17:19:34 kopano kopano-spooler[514]: Server connection lost. Reconnecting in 3 seconds... Sep 9 17:19:37 kopano kopano-spooler[514]: gsoap connect: () Sep 9 17:19:37 kopano kopano-spooler[514]: HrLogon server "default:" user "SYSTEM": network error . . .CheckMK logged the memory usage:
You can see the memory usage rise and then the crash of the server process and the memory usage returns to normal.
-
…are there any messages in messages, syslog or daemon (/var/log/*) ?
-
Nothing special.
The only thing I noticed in /var/log/mail.log is
Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to commit message: disk error (80040116) Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to deliver message to "alex": disk error (80040116) Sep 10 09:50:49 kopano kopano-dagent[207613]: Unable to deliver all messages for server "default:"Don’t know what this is? I just set dagent log to 5, to get more info about this.
The weird thing is, that everything worked for two days without any problems. And then after I restarted the server because I moved it to its designated place on my floor, it began again. Everytime I restart der server service it immediately begins flooding the memory:
Crazy… :-(
-
I found out about the disk error message. There was one corrupted message in the postfix queue.
A “postqueue -f” and the error showed up again in the log.
After deleting the message from the postfix queue the kopano-dagent error vanished. -
The problem still exists. I don’t see anything weird or any errors in the logs.
Here is the output of kopano-stats --system:
0x3001001E 0x6740001E 0x6741001E ----------------------------------------------------------------------------------------------------------------------------------------- server_start_date Time when the server was started Di Sep 11 09:48:00 2018 cache_purge_date Time when the cache was cleared config_reload_date Time when the configuration file was reloaded / logrotation (SIGHUP) connections Number of handled incoming connections 436494 max_socket Highest socket number used 1057 redirections Number of redirected requests 0 soap_request Number of soap requests handled by server 2212637 response_time Response time of soap requests handled in milliseconds (includes time in queue) 14092788 processing_time Time taken to process soap requests in milliseconds (wallclock time) 14017893 searchfld_loaded Total number of searchfolders 10 searchfld_threads Current number of running searchfolder threads 0 searchupd_retry The number of times a search folder update was restarted 0 searchupd_fail The number of failed search folder updates after retrying 0 sql_connect Number of connections made to SQL server 432837 sql_select Number of SQL Select commands executed 9358564 sql_insert Number of SQL Insert commands executed 4369 sql_update Number of SQL Update commands executed 2298 sql_delete Number of SQL Delete commands executed 1163 sql_connect_fail Number of failed connections made to SQL server 0 sql_select_fail Number of failed SQL Select commands 0 sql_insert_fail Number of failed SQL Insert commands 0 sql_update_fail Number of failed SQL Update commands 0 sql_delete_fail Number of failed SQL Delete commands 0 sql_last_fail_time Timestamp of last failed SQL command mwops MAPI Write Operations 125 mrops MAPI Read Operations 873475 deferred_fetches Number rows retrieved via deferred write table 4657 deferred_merges Number of merges applied to the deferred write table 4 deferred_records Number records merged in the deferred write table 80 row_reads Number of table rows read in row order 4712 counter_resyncs Number of time a counter resync was required 0 login_password Number of logins through password authentication 2357 login_ssl Number of logins through SSL certificate authentication 0 login_sso Number of logins through Single Sign-on 0 login_unix Number of logins through Unix socket 433715 login_failed Number of failed logins 0 sessions_created Number of created sessions 436072 sessions_deleted Number of deleted sessions 435995 sessions_timeout Number of timed-out sessions 59 sess_int_created Number of created internal sessions 172 sess_int_deleted Number of deleted internal sessions 172 sess_grp_created Number of created sessiongroups 2167 sess_grp_deleted Number of deleted sessiongroups 2160 ldap_connect Number of connections made to LDAP server 0 ldap_reconnect Number of re-connections made to LDAP server 0 ldap_connect_fail Number of failed connections made to LDAP server 0 ldap_connect_time Total duration of connections made to LDAP server 0 ldap_max_connect Longest connection time made to LDAP server 0 ldap_auth Number of LDAP authentications 0 ldap_auth_fail Number of failed authentications 0 ldap_auth_time Total authentication time 0 ldap_max_auth Longest duration of authentication made to LDAP server 0 ldap_avg_auth Average duration of authentication made to LDAP server 0 ldap_search Number of searches made to LDAP server 0 ldap_search_fail Number of failed searches made to LDAP server 0 ldap_search_time Total duration of LDAP searches 0 ldap_max_search Longest duration of LDAP search 0 index_search_errors Number of failed indexer queries 0 index_search_max Maximum duration of an indexed search query 1070 index_search_avg Average duration of an indexed search query 1070 search_indexed Number of indexed searches performed 572 search_database Number of database searches performed 0 cache_acl_items Cache acl items 0 cache_acl_size Cache acl size 0 cache_acl_maxsz Cache acl maximum size 1048576 cache_acl_req Cache acl requests 0 cache_acl_hit Cache acl hits 0 cache_quota_items Cache quota items 1 cache_quota_size Cache quota size 104 cache_quota_maxsz Cache quota maximum size 1048576 cache_quota_req Cache quota requests 1238 cache_quota_hit Cache quota hits 990 cache_uquota_items Cache uquota items 0 cache_uquota_size Cache uquota size 0 cache_uquota_maxsz Cache uquota maximum size 1048576 cache_uquota_req Cache uquota requests 0 cache_uquota_hit Cache uquota hits 0 cache_extern_items Cache extern items 3 cache_extern_size Cache extern size 501 cache_extern_maxsz Cache extern maximum size 26214400 cache_extern_req Cache extern requests 6467 cache_extern_hit Cache extern hits 3828 cache_userid_items Cache userid items 3 cache_userid_size Cache userid size 456 cache_userid_maxsz Cache userid maximum size 26214400 cache_userid_req Cache userid requests 21191 cache_userid_hit Cache userid hits 21174 cache_abinfo_items Cache abinfo items 3 cache_abinfo_size Cache abinfo size 552 cache_abinfo_maxsz Cache abinfo maximum size 26214400 cache_abinfo_req Cache abinfo requests 12624 cache_abinfo_hit Cache abinfo hits 12621 cache_server_items Cache server items 0 cache_server_size Cache server size 0 cache_server_maxsz Cache server maximum size 1048576 cache_server_req Cache server requests 0 cache_server_hit Cache server hits 0 cache_store_items Cache store items 10903 cache_store_size Cache store size 1046688 cache_store_maxsz Cache store maximum size 1048576 cache_store_req Cache store requests 5788327 cache_store_hit Cache store hits 5346082 cache_obj_items Cache obj items 136117 cache_obj_size Cache obj size 11978296 cache_obj_maxsz Cache obj maximum size 16777216 cache_obj_req Cache obj requests 6469036 cache_obj_hit Cache obj hits 6334828 cache_cell_items Cache cell items 447448 cache_cell_size Cache cell size 1964950625 cache_cell_maxsz Cache cell maximum size 2147483648 cache_cell_req Cache cell requests 36588037 cache_cell_hit Cache cell hits 36073239 cache_index1_items Cache index1 items 246612 cache_index1_size Cache index1 size 32306108 cache_index1_maxsz Cache index1 maximum size 33554432 cache_index1_req Cache index1 requests 4162193 cache_index1_hit Cache index1 hits 2443995 cache_index2_items Cache index2 items 259385 cache_index2_size Cache index2 size 31904214 cache_index2_maxsz Cache index2 maximum size 33554432 cache_index2_req Cache index2 requests 5690998 cache_index2_hit Cache index2 hits 4491006 sessions Number of sessions 18 sessions_size Memory usage of sessions 1296 sessiongroups Number of session groups 7 sessiongroups_size Memory usage of session groups 5784 persist_conn Persistent connections 0 persist_conn_size Memory usage of persistent connections 0 persist_sess Persistent sessions 0 persist_sess_size Memory usage of persistent sessions 0 tables_subscr Tables subscribed 1 tables_subscr_size Memory usage of subscribed tables 72 object_subscr Objects subscribed 3 object_subscr_size Memory usage of subscribed objects 192 searchfld_stores Number of stores in use by search folders 3 searchfld_folders Number of folders in use by search folders 10 searchfld_events Number of events waiting for searchfolder updates 0 searchfld_size Memory usage of search folders 8703 queuelen Current queue length 0 queueage Age of the front queue item 0.000 threads Number of threads running to process items 0 threads_idle Number of idle threads 8 usercnt_active Number of active users 3 usercnt_nonactive Number of total non-active objects 0 usercnt_na_user Number of non-active users 0 usercnt_room Number of rooms 0 usercnt_equipment Number of equipment 0 usercnt_contact Number of contacts 0 tc_allocated Current allocated memory by TCMalloc 6587321000 tc_reserved Bytes of system memory reserved by TCMalloc 6811803648 tc_page_map_free Number of bytes in free, mapped pages in page heap 0 tc_page_unmap_free Number of bytes in free, unmapped pages in page heap (released to OS) 47865856 tc_threadcache_max A limit to how much memory TCMalloc dedicates for small objects 33554432 tc_threadcache_cur Current allocated memory in bytes for thread cache 26003304 pt_allocated Current allocated memory by libc ptmalloc, in bytes -2002612232 userplugin User plugin used dbAnd here is the output of top:
top - 14:12:31 up 2 days, 4:50, 2 users, load average: 1,45, 1,45, 1,37 Tasks: 131 total, 2 running, 129 sleeping, 0 stopped, 0 zombie %Cpu(s): 18,5 us, 7,8 sy, 0,0 ni, 69,5 id, 3,8 wa, 0,0 hi, 0,5 si, 0,0 st KiB Mem : 8045368 total, 130000 free, 7294192 used, 621176 buff/cache KiB Swap: 4143100 total, 3929428 free, 213672 used. 457920 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 92741 mysql 20 0 755712 182892 0 S 41,6 2,3 514:22.10 mysqld 154878 kopano 20 0 7020140 6,412g 17236 S 39,6 83,6 80:59.98 kopano-server 125893 kopano 20 0 513720 127600 4920 R 22,8 1,6 204:06.06 kopano-search 111218 root 20 0 0 0 0 S 1,0 0,0 0:00.02 kworker/3:0 119974 root 20 0 44924 3556 2892 R 1,0 0,0 0:00.09 top 1 root 20 0 204592 3508 2276 S 0,0 0,0 1:00.06 systemd 2 root 20 0 0 0 0 S 0,0 0,0 0:00.08 kthreadd 3 root 20 0 0 0 0 S 0,0 0,0 0:07.52 ksoftirqd/0 5 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/0:0H 6 root 20 0 0 0 0 S 0,0 0,0 0:00.28 kworker/u480:0 8 root 20 0 0 0 0 S 0,0 0,0 3:50.28 rcu_sched 9 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcu_bh 10 root rt 0 0 0 0 S 0,0 0,0 0:00.84 migration/0 11 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 lru-add-drain 12 root rt 0 0 0 0 S 0,0 0,0 0:00.39 watchdog/0 13 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/0 14 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/1 15 root rt 0 0 0 0 S 0,0 0,0 0:00.40 watchdog/1 16 root rt 0 0 0 0 S 0,0 0,0 0:00.85 migration/1 17 root 20 0 0 0 0 S 0,0 0,0 0:04.95 ksoftirqd/1 19 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/1:0H 20 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/2 21 root rt 0 0 0 0 S 0,0 0,0 0:00.43 watchdog/2 22 root rt 0 0 0 0 S 0,0 0,0 0:00.80 migration/2 23 root 20 0 0 0 0 S 0,0 0,0 0:10.55 ksoftirqd/2 25 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/2:0H 27 root 20 0 0 0 0 S 0,0 0,0 0:00.00 cpuhp/3 28 root rt 0 0 0 0 S 0,0 0,0 0:00.45 watchdog/3 29 root rt 0 0 0 0 S 0,0 0,0 0:00.83 migration/3 30 root 20 0 0 0 0 S 0,0 0,0 0:04.34 ksoftirqd/3 32 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/3:0H 33 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kdevtmpfs 34 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 netns 35 root 20 0 0 0 0 S 0,0 0,0 0:00.06 khungtaskd 36 root 20 0 0 0 0 S 0,0 0,0 0:00.00 oom_reaper 37 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 writeback 38 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kcompactd0 39 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kcompactd1 41 root 25 5 0 0 0 S 0,0 0,0 0:00.00 ksmd 42 root 39 19 0 0 0 S 0,0 0,0 0:07.14 khugepaged 43 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 crypto 44 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kintegrityd 45 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 bioset 46 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kblockd 47 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 devfreq_wq 48 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 watchdogd 49 root 20 0 0 0 0 S 0,0 0,0 4:18.34 kswapd0 50 root 20 0 0 0 0 S 0,0 0,0 4:50.38 kswapd1 51 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 vmstat 63 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kthrotld 69 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 ipv6_addrconf 99 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 hv_vmbus_con 118 root 20 0 0 0 0 S 0,0 0,0 0:00.00 scsi_eh_0 120 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 scsi_tmf_0 122 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 bioset 141 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 bioset 176 root 0 -20 0 0 0 S 0,0 0,0 0:37.67 kworker/0:1H 177 root 0 -20 0 0 0 S 0,0 0,0 0:35.76 kworker/1:1H 185 root 0 -20 0 0 0 S 0,0 0,0 0:35.09 kworker/3:1H 190 root 0 -20 0 0 0 S 0,0 0,0 0:45.28 kworker/2:1HAll I see is high CPU usage because of the kopano-search process.
Shutting down the search process ends the high CPU usage. But the memory usage keeps rising.Does anybody have a hint for me? What could I check or do to get the system running without restarting every 5 hours.
Thanks,
Alex
-
hi, can you post an output of :
dpkg -l | grep kopano apt-cache policy kopano-serverand did you upgrade this version from a 8.5?
then read : https://forum.kopano.io/topic/1305/crash-kopano-core-upgrade-from-8-5-81-288-0-53-1-to-8-6-80-645-0-68-1/18
after cleaning up the left overs, im out of problems. -
Hi,
thank you for your message.
here are the requested outputs:root@kopano:~# dpkg -l | grep kopano ii kopano-backup 8.6.81.198-0+35.1 amd64 Utility to back up and restore Kopano stores ii kopano-bash-completion 8.6.81.198-0+35.1 amd64 bash TAB completion for Kopano Core command-line utilities ii kopano-client 8.6.81.198-0+35.1 amd64 Kopano MAPI provider library ii kopano-client-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-client ii kopano-common 8.6.81.198-0+35.1 amd64 Shared files for Kopano services ii kopano-contacts 8.6.81.198-0+35.1 amd64 MAPI provider adding contact folders in the addressbook ii kopano-dagent 8.6.81.198-0+35.1 amd64 E-Mail Delivery Agent for Kopano Core ii kopano-dagent-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-dagent ii kopano-dbg 8.6.81.198-0+35.1 amd64 Debug symbols for Kopano Core ii kopano-dev 8.6.81.198-0+35.1 amd64 C++ Development files for Kopano Core ii kopano-gateway 8.6.81.198-0+35.1 amd64 POP3 and IMAP Gateway for Kopano Core ii kopano-gateway-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-gateway ii kopano-ical 8.6.81.198-0+35.1 amd64 ICal and CalDAV Gateway for Kopano Core ii kopano-ical-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-ical ii kopano-lang 8.6.81.198-0+35.1 all Translations for Kopano Core components ii kopano-migration-imap 8.6.81.198-0+35.1 amd64 Utility to migrate between IMAP mailboxes ii kopano-migration-pst 8.6.81.198-0+35.1 amd64 Utility to import PST files ii kopano-monitor 8.6.81.198-0+35.1 amd64 Quota Monitor for Kopano Core ii kopano-monitor-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-monitor ii kopano-presence 8.6.81.198-0+35.1 amd64 kopano presence daemon ii kopano-python-utils 8.6.81.198-0+35.1 amd64 Additional Python-based command-line utils for Kopano Core ii kopano-search 8.6.81.198-0+35.1 amd64 Indexed search engine for Kopano Core ii kopano-server 8.6.81.198-0+35.1 amd64 Server component for Kopano Core ii kopano-server-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-server ii kopano-server-packages 8.6.81.198-0+35.1 all Metapackage to install the entire Kopano Core stack ii kopano-spamd 8.6.81.198-0+35.1 amd64 ICS-driven spam learning daemon for Kopano/SpamAssasin ii kopano-spooler 8.6.81.198-0+35.1 amd64 E-mail Spooler for Kopano Core ii kopano-spooler-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-spooler ii kopano-utils 8.6.81.198-0+35.1 amd64 Admin command-line utils for Kopano Core ii kopano-utils-dbgsym 8.6.81.198-0+35.1 amd64 Debug symbols for kopano-utils ii kopano-webapp 3.4.22.1724+981.1 all New and improved WebApp for Kopano ii kopano-webapp-plugin-contactfax 3.4.22.1724+981.1 all Kopano WebApp fax plugin ii kopano-webapp-plugin-desktopnotifications 2.0.1.5+16.2 all Kopano WebApp Desktop notifications plugin ii kopano-webapp-plugin-filepreviewer 2.0.0.15+10.8 all Kopano File previewer plugin ii kopano-webapp-plugin-files 2.1.5.295+194.1 all Adds Files functionality to Kopano enabling access to WebDAV and other files backends. ii kopano-webapp-plugin-filesbackend-owncloud 2.1.1.84+61.1 all Adds Owncloud specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-filesbackend-smb 2.1.1.48+49.1 all Adds Samba specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-folderwidgets 3.4.22.1724+981.1 all Kopano WebApp folder widgets plugin ii kopano-webapp-plugin-gmaps 3.4.22.1724+981.1 all Kopano WebApp google maps plugin ii kopano-webapp-plugin-intranet 0.1+32.8 all This plugin adds one or more buttons in the top menu bar which can be used to open a webpage inside Kopano WebApp. ii kopano-webapp-plugin-mattermost 0.1+17.2 all Integrates Mattermost into WebApp ii kopano-webapp-plugin-mdm 2.1.0.106+26.1 all Kopano WebApp MDM plugin ii kopano-webapp-plugin-pimfolder 3.4.22.1724+981.1 all Kopano WebApp personal inbox plugin ii kopano-webapp-plugin-quickitems 3.4.22.1724+981.1 all Kopano WebApp quick items plugin ii kopano-webapp-plugin-spell 2.0.0.17+34.1 all Kopano WebApp Spellchecker plugin ii kopano-webapp-plugin-spell-de-at 2.0.0.4+28.1 all Kopano WebApp Spellchecker German (Austrian) dictionary plugin ii kopano-webapp-plugin-spell-de-ch 2.0.0.5+27.1 all Kopano WebApp Spellchecker German (Swiss) dictionary plugin ii kopano-webapp-plugin-spell-de-de 2.0.0.3+28.1 all Kopano WebApp Spellchecker German dictionary plugin ii kopano-webapp-plugin-spell-en 2.0.0.1+28.1 all Kopano WebApp Spellchecker English dictionary plugin ii kopano-webapp-plugin-spell-en-gb 2.0.0.1+27.1 all Kopano WebApp Spellchecker English (GB) dictionary plugin ii kopano-webapp-plugin-spell-es 2.0.0.1+27.1 all Kopano WebApp Spellchecker Spanish dictionary plugin ii kopano-webapp-plugin-spell-fr 2.0.0.1+28.1 all Kopano WebApp Spellchecker French dictionary plugin ii kopano-webapp-plugin-spell-it 1.0.0+40.1 all Kopano WebApp Spellchecker Italian dictionary plugin ii kopano-webapp-plugin-spell-nl 2.0.0.1+29.1 all Kopano WebApp Spellchecker Dutch dictionary plugin ii kopano-webapp-plugin-spell-pl-pl 2.0.0.0+33.1 all Kopano WebApp Spellchecker Polish dictionary plugin ii kopano-webapp-plugin-titlecounter 3.4.22.1724+981.1 all Kopano WebApp Titlecounter plugin ii kopano-webapp-plugin-webappmanual 3.4.22.1724+981.1 all Kopano WebApp Manual plugin ii kopano-webapp-plugin-zdeveloper 3.4.22.1724+981.1 all Kopano WebApp developer plugin ii libgsoap-kopano-2.8.62 2.8.62-0+2.1 amd64 Runtime libraries for gSOAP ii libvmime-kopano1 0.9.2.50+11.1 amd64 a C++ mail library (runtime library) ii libvmime-kopano1-dbgsym 0.9.2.50+11.1 amd64 Debug symbols for libvmime-kopano1 ii python-kopano 8.6.81.198-0+35.1 all High-level Python 2 bindings for Kopano ii python3-kopano 8.6.81.198-0+35.1 all High-level Python 3 bindings for Kopano ii z-push-backend-kopano 2.4.4+0-0 all Z-Push Kopano backend ii z-push-kopano 2.4.4+0-0 all Z-Push for Kopanoroot@kopano:~# apt-cache policy kopano-server kopano-server: Installiert: 8.6.81.198-0+35.1 Installationskandidat: 8.6.81.198-0+35.1 Versionstabelle: *** 8.6.81.198-0+35.1 100 100 /var/lib/dpkg/statusHopefully this helps finding the problem.
Thank you very much in advance.Regards,
Alex
P.S.: Actually there is always a high CPU usage of kopano-search, kopano-server and mysqld:
top - 00:03:01 up 2 days, 8:42, 1 user, load average: 1,31, 1,38, 1,41 Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie %Cpu(s): 20,4 us, 0,9 sy, 0,0 ni, 69,3 id, 9,3 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8045376 total, 140868 free, 7740960 used, 163548 buff/cache KiB Swap: 4143100 total, 3174024 free, 969076 used. 69448 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 43607 kopano 20 0 481320 89756 5544 D 41,7 1,1 665:50.80 kopano-search 32208 kopano 20 0 8185608 6,879g 976 S 22,8 89,7 86:45.45 kopano-server 714 mysql 20 0 763904 170444 0 S 20,5 2,1 1322:28 mysqld 90822 root 20 0 44928 3648 2988 R 0,7 0,0 0:00.04 top 175 root 0 -20 0 0 0 S 0,3 0,0 0:46.38 kworker/3:1H 1 root 20 0 204692 3024 2056 S 0,0 0,0 0:24.93 systemd 2 root 20 0 0 0 0 S 0,0 0,0 0:00.07 kthreadd 3 root 20 0 0 0 0 S 0,0 0,0 0:08.63 ksoftirqd/0Perhaps this takes some days till index-process is over. All in all about 80 Gigabyte of attachments and MySQL DB was migrated before.
-
Just made an update to the actual nightly version:
root@kopano:~# apt-cache policy kopano-server kopano-server: Installiert: 8.6.81.228-0+37.1 Installationskandidat: 8.6.81.228-0+37.1 Versionstabelle: *** 8.6.81.228-0+37.1 100 100 /var/lib/dpkg/statusand:
root@kopano:~# dpkg -l | grep kopano ii kopano-backup 8.6.81.228-0+37.1 amd64 Utility to back up and restore Kopano stores ii kopano-bash-completion 8.6.81.228-0+37.1 amd64 bash TAB completion for Kopano Core command-line utilities ii kopano-client 8.6.81.228-0+37.1 amd64 Kopano MAPI provider library ii kopano-client-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-client ii kopano-common 8.6.81.228-0+37.1 amd64 Shared files for Kopano services ii kopano-contacts 8.6.81.228-0+37.1 amd64 MAPI provider adding contact folders in the addressbook ii kopano-dagent 8.6.81.228-0+37.1 amd64 E-Mail Delivery Agent for Kopano Core ii kopano-dagent-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-dagent ii kopano-dbg 8.6.81.228-0+37.1 amd64 Debug symbols for Kopano Core ii kopano-dev 8.6.81.228-0+37.1 amd64 C++ Development files for Kopano Core ii kopano-gateway 8.6.81.228-0+37.1 amd64 POP3 and IMAP Gateway for Kopano Core ii kopano-gateway-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-gateway ii kopano-ical 8.6.81.228-0+37.1 amd64 ICal and CalDAV Gateway for Kopano Core ii kopano-ical-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-ical ii kopano-lang 8.6.81.228-0+37.1 all Translations for Kopano Core components ii kopano-migration-imap 8.6.81.228-0+37.1 amd64 Utility to migrate between IMAP mailboxes ii kopano-migration-pst 8.6.81.228-0+37.1 amd64 Utility to import PST files ii kopano-monitor 8.6.81.228-0+37.1 amd64 Quota Monitor for Kopano Core ii kopano-monitor-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-monitor ii kopano-presence 8.6.81.228-0+37.1 amd64 kopano presence daemon ii kopano-python-utils 8.6.81.228-0+37.1 amd64 Additional Python-based command-line utils for Kopano Core ii kopano-search 8.6.81.228-0+37.1 amd64 Indexed search engine for Kopano Core ii kopano-server 8.6.81.228-0+37.1 amd64 Server component for Kopano Core ii kopano-server-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-server ii kopano-server-packages 8.6.81.228-0+37.1 all Metapackage to install the entire Kopano Core stack ii kopano-spamd 8.6.81.228-0+37.1 amd64 ICS-driven spam learning daemon for Kopano/SpamAssasin ii kopano-spooler 8.6.81.228-0+37.1 amd64 E-mail Spooler for Kopano Core ii kopano-spooler-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-spooler ii kopano-utils 8.6.81.228-0+37.1 amd64 Admin command-line utils for Kopano Core ii kopano-utils-dbgsym 8.6.81.228-0+37.1 amd64 Debug symbols for kopano-utils ii kopano-webapp 3.4.22.1724+981.1 all New and improved WebApp for Kopano ii kopano-webapp-plugin-contactfax 3.4.22.1724+981.1 all Kopano WebApp fax plugin ii kopano-webapp-plugin-desktopnotifications 2.0.1.5+16.2 all Kopano WebApp Desktop notifications plugin ii kopano-webapp-plugin-filepreviewer 2.0.0.15+10.8 all Kopano File previewer plugin ii kopano-webapp-plugin-files 2.1.5.295+194.1 all Adds Files functionality to Kopano enabling access to WebDAV and other files backends. ii kopano-webapp-plugin-filesbackend-owncloud 2.1.1.84+61.1 all Adds Owncloud specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-filesbackend-smb 2.1.1.48+49.1 all Adds Samba specific functionality to Kopano Files plugin. ii kopano-webapp-plugin-folderwidgets 3.4.22.1724+981.1 all Kopano WebApp folder widgets plugin ii kopano-webapp-plugin-gmaps 3.4.22.1724+981.1 all Kopano WebApp google maps plugin ii kopano-webapp-plugin-intranet 0.1+32.8 all This plugin adds one or more buttons in the top menu bar which can be used to open a webpage inside Kopano WebApp. ii kopano-webapp-plugin-mattermost 0.1+17.2 all Integrates Mattermost into WebApp ii kopano-webapp-plugin-mdm 2.1.0.106+26.1 all Kopano WebApp MDM plugin ii kopano-webapp-plugin-pimfolder 3.4.22.1724+981.1 all Kopano WebApp personal inbox plugin ii kopano-webapp-plugin-quickitems 3.4.22.1724+981.1 all Kopano WebApp quick items plugin ii kopano-webapp-plugin-spell 2.0.0.17+34.1 all Kopano WebApp Spellchecker plugin ii kopano-webapp-plugin-spell-de-at 2.0.0.4+28.1 all Kopano WebApp Spellchecker German (Austrian) dictionary plugin ii kopano-webapp-plugin-spell-de-ch 2.0.0.5+27.1 all Kopano WebApp Spellchecker German (Swiss) dictionary plugin ii kopano-webapp-plugin-spell-de-de 2.0.0.3+28.1 all Kopano WebApp Spellchecker German dictionary plugin ii kopano-webapp-plugin-spell-en 2.0.0.1+28.1 all Kopano WebApp Spellchecker English dictionary plugin ii kopano-webapp-plugin-spell-en-gb 2.0.0.1+27.1 all Kopano WebApp Spellchecker English (GB) dictionary plugin ii kopano-webapp-plugin-spell-es 2.0.0.1+27.1 all Kopano WebApp Spellchecker Spanish dictionary plugin ii kopano-webapp-plugin-spell-fr 2.0.0.1+28.1 all Kopano WebApp Spellchecker French dictionary plugin ii kopano-webapp-plugin-spell-it 1.0.0+40.1 all Kopano WebApp Spellchecker Italian dictionary plugin ii kopano-webapp-plugin-spell-nl 2.0.0.1+29.1 all Kopano WebApp Spellchecker Dutch dictionary plugin ii kopano-webapp-plugin-spell-pl-pl 2.0.0.0+33.1 all Kopano WebApp Spellchecker Polish dictionary plugin ii kopano-webapp-plugin-titlecounter 3.4.22.1724+981.1 all Kopano WebApp Titlecounter plugin ii kopano-webapp-plugin-webappmanual 3.4.22.1724+981.1 all Kopano WebApp Manual plugin ii kopano-webapp-plugin-zdeveloper 3.4.22.1724+981.1 all Kopano WebApp developer plugin ii libgsoap-kopano-2.8.62 2.8.62-0+2.1 amd64 Runtime libraries for gSOAP ii libvmime-kopano1 0.9.2.50+11.1 amd64 a C++ mail library (runtime library) ii libvmime-kopano1-dbgsym 0.9.2.50+11.1 amd64 Debug symbols for libvmime-kopano1 ii python-kopano 8.6.81.228-0+37.1 all High-level Python 2 bindings for Kopano ii python3-kopano 8.6.81.228-0+37.1 all High-level Python 3 bindings for Kopano ii z-push-backend-kopano 2.4.4+0-0 all Z-Push Kopano backend ii z-push-kopano 2.4.4+0-0 all Z-Push for KopanoLet’s see if this changes anything.
I’m still not completely desperate :-)Regards,
Alex
-
Unfortunately not. kopano-server still crashing every 4 - 5 hours after flooding the whole RAM and SWAP memory. :-(
-
hello @Sniks6,
i had a similar issue on a testsystem, resolving was quite questionable :)
first i made the mariadb tuning for the system, second i reduce the index-process to 1 worker for first indexing.
then i only started search, wait 5 minutes till it complains about missing kopano-server, then start it, without gateway or dagent.
Wait for the first indexing, this worked fine in my case but used a lot of time.If this works, stop everything, incrase the indexing worker to standard value (i think it was 5) and start everything.
The stats of this vm where:
VMware machine
4 cores
8GB ram
scsi sas drives
CentOS 7 64bit
mariadb - 5.5.60-1
kopano-server - 8.6.4.0
no webapp, since its an external machineregards,
Coffee_is_life
-
Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.
depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?
8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?
-
Hi @ all,
Thank you very much for your anwers!
Its does not really make sense, that the growth in ram usage stopped, after a mysqldump was performed.
That’s true. Perhaps I just didn’t start all services at the time I made the mysqldump.
depending on tuning it could take a while until the initial search index has been completed. does the growth im ram usage also occur if you stop kopano-search?
You’re right. The search service always uses much cpu und I think after two weeks it still hasn’t finished the indexing. :-(
I stopped kopano-search 2 hours ago and at the moment the cpu and memory usage is very low:top - 15:24:59 up 6 days, 4 min, 3 users, load average: 0,03, 0,06, 0,01 Tasks: 123 total, 1 running, 122 sleeping, 0 stopped, 0 zombie %Cpu(s): 0,2 us, 0,3 sy, 0,0 ni, 99,5 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8045376 total, 5219748 free, 564032 used, 2261596 buff/cache KiB Swap: 4143100 total, 4040092 free, 103008 used. 7173136 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 158017 mysql 20 0 726648 243176 17020 S 0,0 3,0 1:32.18 mysqld 161987 kopano 20 0 360232 71796 18200 S 0,0 0,9 0:12.51 kopano-server 165057 www-data 20 0 734928 43668 35932 S 0,0 0,5 0:00.76 apache2 163614 www-data 20 0 740548 43224 33404 S 0,0 0,5 0:00.92 apache2 108383 www-data 20 0 740392 43132 33628 S 0,3 0,5 0:02.50 apache2 164767 www-data 20 0 735112 42924 34632 S 0,0 0,5 0:00.82 apache2 168432 www-data 20 0 735064 42452 34652 S 0,0 0,5 0:00.52 apache2 164391 www-data 20 0 739752 42276 33036 S 0,0 0,5 0:00.64 apache2 165992 www-data 20 0 735012 40456 33280 S 0,0 0,5 0:00.50 apache2 168150 www-data 20 0 734292 40408 33536 S 0,0 0,5 0:00.36 apache2 103010 www-data 20 0 734872 40388 32832 S 0,0 0,5 0:02.11 apache2 164770 www-data 20 0 734660 40136 33020 S 0,0 0,5 0:00.64 apache2 165990 www-data 20 0 734628 38988 32424 S 0,0 0,5 0:00.44 apache2 170618 www-data 20 0 666752 33664 28192 S 0,0 0,4 0:00.02 apache2 170622 www-data 20 0 666752 33664 28192 S 0,0 0,4 0:00.02 apache2 162160 kopano 20 0 230108 20472 18368 S 0,0 0,3 0:00.24 kopano-spooler 162138 kopano 20 0 160660 16596 14668 S 0,0 0,2 0:00.08 kopano-dagent 247 root 20 0 70680 9724 9492 S 0,0 0,1 1:38.56 systemd-journal 146108 root 20 0 101436 6904 5900 S 0,0 0,1 0:00.22 sshd 170192 root 20 0 101432 6856 5852 S 0,0 0,1 0:00.22 sshd8.7 has the possibility to perform tuning in kopano-server on its own. maybe it configures to high values for you? what happens if you manually set the options for the caches in server.cfg?
I already changed the cache to:
cache_cell_size = 2GUnfortunately without success.
As of now with the disabled kopano-search it looks good. Is there a way to find out what causes this mem flooding when indexing by kopano-search?
Regards,
Alex -
Good Morning everyone,
now, after 17 hours it looks good kopano-server did not crash since then.
So I think we found the problem: kopano-search is causing the memory flood in kopano-server.But why? As Coffee_is_life wrote on his setup after completing the index process everything worked fine. Perhaps this is the same with my setup. But my indexing is running for two weeks now. :-(
Do you have any ideas what to tweek in this case?
-
After 4 days without kopano-search kopano-server did not crash anymore.
But the memory keeps still rising much more slowly…
Any thoughts?
-
I also (still?) have this problem. Currently using 8.6.81.171-0+25.1 on Ubuntu 18.04 x64.
[Fri Sep 28 19:51:29 2018] Out of memory: Kill process 1240 (kopano-server) score 902 or sacrifice child
[Fri Sep 28 19:51:29 2018] Killed process 1240 (kopano-server) total-vm:26514968kB, anon-rss:10946196kB, file-rss:0kB, shmem-rss:0kBBy the time it had crashed it was running only for 11 hours or so - after having been restarted due to a kopano-server crash the day before. The server should have enough memory for normal use (12GB) and an additional 16GB Swap is available. Especially with the swap I really see no reason at all for it to run out of memory.
I will now try to update to 8.6.81.435-0+75.1 in the hope that this out of memory bug has been fixed in the meantime?
In addition I’ve also cleaned the /var/lib/kopano/search/ directory and restarted the indexing process. Maybe index is corrupted?
Update: Has not changed/fixed anything. Server still crashing about once a day. Redoing the index didn’t help, neither did disabling the indexing of attachments.
I’ve added the following to the kopano-server.service file for a temporary fix:
[add in the service section]
Restart=on-failure
RestartSec=10s[add in the unit section]
StartLimitIntervalSec=300s
StartLimitBurst=2 -
Thank you for your message.
Here is a graph of the memory consumption:
- In the first half you see the memory rising very quickly till the server process crashed. It restarted automatically after modfying the kopano-server.service file.
- In the second half (~ week 38) I didn’t start the kopano-search process. The memory is still rising but much slower.
So I think it is not a problem with the search process. It seems that the search process simply speeds up the rising of the memory usage. If I had more users or much more interaction with the server process, perhaps the memory would be rising faster again.
So it looks like a problem in the kopano-server.
Just for interest, what hardware do you use, Gerald? Is it virtualised hardware? If yes which kind (VMware, HyperV,…)?
Regards,
Alex
