Problems after upgrade KC 8.7.9 to 8.7.12
-
Hi,
i upgraded my Ubuntu 18.04 from KC8.7.9 to KC8.7.12 .Since KC8.7.9 the server was running without any problems.
With 8.7.12 the server is up for some time ( <24h) problems start like:
==> /var/log/kopano/server.log <== Fri May 1 09:28:38 2020: [error ] Previous message logged 69 times Fri May 1 09:28:38 2020: [warning] SQL [00000024] info: Try to reconnect Fri May 1 09:28:38 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Fri May 1 09:28:38 2020: [error ] SQL [00000000] Failed: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug, Query Size: 81, Query: "SELECT properties.val_ulong FROM properties WHERE hierarchyid = 467873 FOR UPDATE" Fri May 1 09:28:38 2020: [error ] KDatabase::DoSelect(): query failed: "MYSQL not initialized", query: SELECT properties.val_ulong FROM properties WHERE hierarchyid = 467873 FOR UPDATE Fri May 1 09:28:38 2020: [error ] ECSearchFolders::ResetResults(): SELECT failed: disk error (80000007) Fri May 1 09:28:38 2020: [error ] ECSearchFolders::Search() ResetResults failed: disk error (80000007) Fri May 1 09:28:44 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug==> /var/log/kopano/search.log <== 2020-05-01 09:28:44,706 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=4/5) 2020-05-01 09:28:51,117 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=5/5) 2020-05-01 09:28:57,524 - search - ERROR - Too many retries, skipping change 2020-05-01 09:28:57,527 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=0/5) 2020-05-01 09:29:03,937 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=1/5)- Search in the webapp give no result
- kopano-spooler was unable to deliver your mail to the/some of the recipient(s).
somehow to me it looks like it has to do something with the DB Connection. When i restart kopano-server everything works again for some time.
Things i have done so far:
root@xxx:/home/xxx# mysqlcheck --all-databases -u root -p Enter password: kopano.abchanges OK kopano.acl OK kopano.changes OK kopano.deferredupdate OK kopano.hierarchy OK kopano.indexedproperties OK kopano.lob OK kopano.mvproperties OK kopano.names OK kopano.object OK kopano.objectmvproperty OK kopano.objectproperty OK kopano.objectrelation OK kopano.outgoingqueue OK kopano.properties OK kopano.receivefolder OK kopano.searchresults OK kopano.settings OK kopano.singleinstances OK kopano.stores OK kopano.syncedmessages OK kopano.syncs OK kopano.tproperties OK kopano.users OK kopano.versions OK mysql.columns_priv OK mysql.db OK mysql.engine_cost OK mysql.event OK mysql.func OK mysql.general_log OK mysql.gtid_executed OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.innodb_index_stats OK mysql.innodb_table_stats OK mysql.ndb_binlog_index OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.proxies_priv OK mysql.server_cost OK mysql.servers OK mysql.slave_master_info OK mysql.slave_relay_log_info OK mysql.slave_worker_info OK mysql.slow_log OK mysql.tables_priv OK mysql.time_zone OK mysql.time_zone_leap_second OK mysql.time_zone_name OK mysql.time_zone_transition OK mysql.time_zone_transition_type OK mysql.user OK sys.sys_config OKkopano-search --reindexkopano-server --restart-searchesroot@xxx:/home/xxx# free -m total used free shared buff/cache available Mem: 4096 2384 898 10 812 1701 Swap: 0 0 0root@xxx:/home/xxx# df -h Filesystem Size Used Avail Use% Mounted on /dev/ploop26596p1 303G 82G 207G 29% / none 2,0G 0 2,0G 0% /sys/fs/cgroup none 2,0G 0 2,0G 0% /dev tmpfs 2,0G 52K 2,0G 1% /dev/shm tmpfs 2,0G 1,2M 2,0G 1% /run tmpfs 5,0M 0 5,0M 0% /run/lock none 2,0G 0 2,0G 0% /run/shmAny ideas where i should look or what i could do next?
-
https://stash.kopano.io/projects/KC/repos/kopanocore/pull-requests/3104/overview
edit: https://stash.kopano.io/users/jengelhardt/repos/kc/commits?until=refs%2Fheads%2Fks-45601
-
@jengelh said in Problems after upgrade KC 8.7.9 to 8.7.12:
https://stash.kopano.io/projects/KC/repos/kopanocore/pull-requests/3104/overview
sorry but i cannot access that link even after registering at bitbucket…
-
I see a similar error. Kopano investigates in this issue.
-
as a workaround i stopped kopano-search.
So far the rest of the server is running.
Update: it only slows down the time during the errors
-
Hi,
now my mail was offline for for two nights with this bug. As work around I created a script to search for this error and restart kopano-server if this error accoures.
Download link for this script: https://www.hofstaedtler.com/tmp/check_kopano_mysql.tgz
- Extract the .tgz archive in /
- Modify the recipients mail address
Hope this script helps.
Walter
-
I’m using the Kopano Contributor Edition.
Is there a way to get the 8.7.9 version from the download.kopano.io/supported/ download site ? -
xthbx,
I guess, only with a valid license code.
-
i wasnt clear with my post. I got the Kopano Contributor Edition.
And i found the 8.7.9 tar in the archives.Can anybody explain to me how i do the rollback to 8.7.9?
Do i have to uncomment the repository and then manualy install every package from the 8.7.9 tar? -
I have the kopano version myself but not the problem because my server is not always on but are there any news?
-
8.7.13 is now available in the pre-final download directory. Your feedback would be appreciated.
See https://forum.kopano.io/topic/3215/kopano-core-8-7-13-pre-final for more information.
Regards Felix
Resources:
https://kopano.com/blog/how-to-get-kopano/
https://documentation.kopano.io/
https://kb.kopano.io/Support overview:
https://kopano.com/support/ -
@fbartels thanks
i gave it a try and keep you updated
Update after 1 day since upgrade 8.7.13: still no problems. Looks great
-
@fbartels i’m sorry but the Problem is still there. It just happened again.
==> /var/log/kopano/server.log <== Wed May 27 16:00:33 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Wed May 27 16:00:33 2020: [error ] Unable to get database connection: MYSQL not initialized Wed May 27 16:00:33 2020: [error ] ECSearchFolders::Search() GetStore failed: disk error (80000007) Wed May 27 16:00:40 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Wed May 27 16:00:40 2020: [error ] Unable to get database connection: MYSQL not initialized Wed May 27 16:00:40 2020: [error ] ECSearchFolders::Search() GetStore failed: disk error (80000007) Wed May 27 16:00:45 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Wed May 27 16:00:45 2020: [error ] Unable to get database connection: MYSQL not initialized Wed May 27 16:00:45 2020: [error ] ECSearchFolders::Search() GetStore failed: disk error (80000007) Wed May 27 16:01:00 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug -
ER_CANT_CREATE_THREAD ... eng "Can't create a new thread (errno %M); if you are not out of available memory, you can consult the manual for a pos>–>
## sql/mysqld.cc:void create_thread_to_handle_connection(CONNECT *connect) /* Create new thread to handle connection */ inc_thread_created(); DBUG_PRINT("info",(("creating thread %lu"), (ulong) connect->thread_id)); connect->prior_thr_create_utime= microsecond_interval_timer(); if ((error= mysql_thread_create(key_thread_one_connection, &connect->real_id, &connection_attrib, handle_one_connection, (void*) connect))) { /* purecov: begin inspected */ DBUG_PRINT("error", ("Can't create thread to handle request (error %d)", error)); my_snprintf(error_message_buff, sizeof(error_message_buff), ER_DEFAULT(ER_CANT_CREATE_THREAD), error); connect->close_with_error(ER_CANT_CREATE_THREAD, error_message_buff, ER_OUT_OF_RESOURCES); DBUG_VOID_RETURN; /* purecov: end */ }–>
#ifdef HAVE_PSI_THREAD_INTERFACE #define mysql_thread_create(K, P1, P2, P3, P4) \ inline_mysql_thread_create(K, P1, P2, P3, P4) #else #define mysql_thread_create(K, P1, P2, P3, P4) \ pthread_create(P1, P2, P3, P4) #endifSo no check for connection limit exceeded (at this point), this is directly the result of pthread_create. If pthread_create fails, then you are running into some ulimit of your mysqld/system. Without further analysis of the Linux installation, I at this time conclude the fault is with your mysqld process, or the ulimits and cgroups that it has been configured with.
-
/usr/include/asm-generic/errno-base.h:#define EAGAIN 11 /* Try again */man pthread_create: EAGAIN Insufficient resources to create another thread. EAGAIN A system-imposed limit on the number of threads was encountered. There are a number of limits that may trig- ger this error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of pro- cesses and threads for a real user ID, was reached; the kernel's system-wide limit on the number of processes and threads, /proc/sys/kernel/threads-max, was reached (see proc(5)); or the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)). -
hmm strange the server worked fine for months with KC 8.7.9. I guess i have to keep looking. And i’m not a Linux pro so i dont really know where to look next.
-
mysql> show processlist; +----+--------+-----------+--------+---------+-------+----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+--------+-----------+--------+---------+-------+----------+------------------+ | 6 | kopano | localhost | kopano | Sleep | 89 | | NULL | | 7 | kopano | localhost | kopano | Sleep | 15 | | NULL | | 8 | kopano | localhost | kopano | Sleep | 5 | | NULL | | 9 | kopano | localhost | kopano | Sleep | 10 | | NULL | | 10 | kopano | localhost | kopano | Sleep | 10 | | NULL | | 11 | kopano | localhost | kopano | Sleep | 35 | | NULL | | 12 | kopano | localhost | kopano | Sleep | 5 | | NULL | | 13 | kopano | localhost | kopano | Sleep | 15 | | NULL | | 16 | kopano | localhost | kopano | Sleep | 1761 | | NULL | | 55 | kopano | localhost | kopano | Sleep | 28726 | | NULL | | 56 | kopano | localhost | kopano | Sleep | 28666 | | NULL | | 57 | kopano | localhost | kopano | Sleep | 25905 | | NULL | | 58 | kopano | localhost | kopano | Sleep | 25124 | | NULL | | 59 | kopano | localhost | kopano | Sleep | 25064 | | NULL | | 62 | kopano | localhost | kopano | Sleep | 25 | | NULL | | 63 | kopano | localhost | kopano | Sleep | 22307 | | NULL | | 64 | kopano | localhost | kopano | Sleep | 21527 | | NULL | | 65 | kopano | localhost | kopano | Sleep | 21467 | | NULL | | 66 | kopano | localhost | kopano | Sleep | 18704 | | NULL | | 67 | kopano | localhost | kopano | Sleep | 17923 | | NULL | | 68 | kopano | localhost | kopano | Sleep | 17863 | | NULL | | 76 | kopano | localhost | kopano | Sleep | 15106 | | NULL | | 77 | kopano | localhost | kopano | Sleep | 14326 | | NULL | | 78 | kopano | localhost | kopano | Sleep | 14266 | | NULL | | 81 | kopano | localhost | kopano | Sleep | 4307 | | NULL | | 82 | kopano | localhost | kopano | Sleep | 3527 | | NULL | | 83 | kopano | localhost | kopano | Sleep | 3467 | | NULL | | 98 | kopano | localhost | kopano | Sleep | 704 | | NULL | | 99 | root | localhost | NULL | Query | 0 | starting | show processlist | +----+--------+-----------+--------+---------+-------+----------+------------------+ 29 rows in set (0.00 sec)is it a normal behavior that with every new search in webapp a new process is generated and not reset ?
here is my mysqld.cnf:
# # The MySQL database server configuration file. # # You can copy this to one of: # - "/etc/mysql/my.cnf" to set global options, # - "~/.my.cnf" to set user-specific options. # # One can use all long options that the program supports. # Run program with --help to get a list of available options and with # --print-defaults to see which it would actually understand and use. # # For explanations see # http://dev.mysql.com/doc/mysql/en/server-system-variables.html # This will be passed to all mysql clients # It has been reported that passwords should be enclosed with ticks/quotes # escpecially if they contain "#" chars... # Remember to edit /etc/mysql/debian.cnf when changing the socket location. # Here is entries for some specific programs # The following values assume you have at least 32M ram [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] # # * Basic Settings # user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql skip-external-locking # # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. bind-address = 127.0.0.1 # # * Fine Tuning # ##KopanoDB innodb_buffer_pool_size = 768M innodb_log_file_size = 192M innodb_log_buffer_size = 48M max_connections = 500 innodb_file_per_table ##KopanoDB key_buffer_size = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 # This replaces the startup script and checks MyISAM tables if needed # the first time they are touched myisam-recover-options = BACKUP #max_connections = 100 #table_open_cache = 64 #thread_concurrency = 10 # # * Query Cache Configuration # query_cache_limit = 1M query_cache_size = 16M # # * Logging and Replication # # Both location gets rotated by the cronjob. # Be aware that this log type is a performance killer. # As of 5.1 you can enable the log at runtime! #general_log_file = /var/log/mysql/mysql.log #general_log = 1 # # Error log - should be very few entries. # log_error = /var/log/mysql/error.log # # Here you can see queries with especially long duration #slow_query_log = 1 #slow_query_log_file = /var/log/mysql/mysql-slow.log #slow_query_log = 1 #slow_query_log_file = /var/log/mysql/mysql-slow.log #long_query_time = 2 #log-queries-not-using-indexes # # The following can be used as easy to replay backup logs or for replication. # note: if you are setting up a replication slave, see README.Debian about # other settings you may need to change. #server-id = 1 #log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 100M #binlog_do_db = include_database_name #binlog_ignore_db = include_database_name # # * InnoDB # # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/. # Read the manual for more InnoDB related options. There are many! # # * Security Features # # Read the manual, too, if you want chroot! # chroot = /var/lib/mysql/ # # For generating SSL certificates I recommend the OpenSSL GUI "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pemroot@xxx:/home/xxx# free -m total used free shared buff/cache available Mem: 4096 2167 1576 10 352 1917 Swap: 0 0 0 -
@fbartels i just installed 8.7.14 (pre-final) and keep you updated…
-
so far it looks better:
mysql> show processlist; +-----+--------+-----------+--------+---------+------+----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +-----+--------+-----------+--------+---------+------+----------+------------------+ | 6 | kopano | localhost | kopano | Sleep | 111 | | NULL | | 7 | kopano | localhost | kopano | Sleep | 9 | | NULL | | 8 | kopano | localhost | kopano | Sleep | 29 | | NULL | | 9 | kopano | localhost | kopano | Sleep | 19 | | NULL | | 10 | kopano | localhost | kopano | Sleep | 34 | | NULL | | 11 | kopano | localhost | kopano | Sleep | 4 | | NULL | | 12 | kopano | localhost | kopano | Sleep | 29 | | NULL | | 13 | kopano | localhost | kopano | Sleep | 9 | | NULL | | 14 | kopano | localhost | kopano | Sleep | 4 | | NULL | | 16 | kopano | localhost | kopano | Sleep | 218 | | NULL | | 110 | root | localhost | NULL | Query | 0 | starting | show processlist | +-----+--------+-----------+--------+---------+------+----------+------------------+ 11 rows in set (0.00 sec) -
@fbartels it looks great. No problem since installation. Still only 11 connections.
