Problems after upgrade KC 8.7.9 to 8.7.12
-
@fbartels thanks
i gave it a try and keep you updated
Update after 1 day since upgrade 8.7.13: still no problems. Looks great
-
@fbartels i’m sorry but the Problem is still there. It just happened again.
==> /var/log/kopano/server.log <== Wed May 27 16:00:33 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Wed May 27 16:00:33 2020: [error ] Unable to get database connection: MYSQL not initialized Wed May 27 16:00:33 2020: [error ] ECSearchFolders::Search() GetStore failed: disk error (80000007) Wed May 27 16:00:40 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Wed May 27 16:00:40 2020: [error ] Unable to get database connection: MYSQL not initialized Wed May 27 16:00:40 2020: [error ] ECSearchFolders::Search() GetStore failed: disk error (80000007) Wed May 27 16:00:45 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Wed May 27 16:00:45 2020: [error ] Unable to get database connection: MYSQL not initialized Wed May 27 16:00:45 2020: [error ] ECSearchFolders::Search() GetStore failed: disk error (80000007) Wed May 27 16:01:00 2020: [error ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug -
ER_CANT_CREATE_THREAD ... eng "Can't create a new thread (errno %M); if you are not out of available memory, you can consult the manual for a pos>–>
## sql/mysqld.cc:void create_thread_to_handle_connection(CONNECT *connect) /* Create new thread to handle connection */ inc_thread_created(); DBUG_PRINT("info",(("creating thread %lu"), (ulong) connect->thread_id)); connect->prior_thr_create_utime= microsecond_interval_timer(); if ((error= mysql_thread_create(key_thread_one_connection, &connect->real_id, &connection_attrib, handle_one_connection, (void*) connect))) { /* purecov: begin inspected */ DBUG_PRINT("error", ("Can't create thread to handle request (error %d)", error)); my_snprintf(error_message_buff, sizeof(error_message_buff), ER_DEFAULT(ER_CANT_CREATE_THREAD), error); connect->close_with_error(ER_CANT_CREATE_THREAD, error_message_buff, ER_OUT_OF_RESOURCES); DBUG_VOID_RETURN; /* purecov: end */ }–>
#ifdef HAVE_PSI_THREAD_INTERFACE #define mysql_thread_create(K, P1, P2, P3, P4) \ inline_mysql_thread_create(K, P1, P2, P3, P4) #else #define mysql_thread_create(K, P1, P2, P3, P4) \ pthread_create(P1, P2, P3, P4) #endifSo no check for connection limit exceeded (at this point), this is directly the result of pthread_create. If pthread_create fails, then you are running into some ulimit of your mysqld/system. Without further analysis of the Linux installation, I at this time conclude the fault is with your mysqld process, or the ulimits and cgroups that it has been configured with.
-
/usr/include/asm-generic/errno-base.h:#define EAGAIN 11 /* Try again */man pthread_create: EAGAIN Insufficient resources to create another thread. EAGAIN A system-imposed limit on the number of threads was encountered. There are a number of limits that may trig- ger this error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of pro- cesses and threads for a real user ID, was reached; the kernel's system-wide limit on the number of processes and threads, /proc/sys/kernel/threads-max, was reached (see proc(5)); or the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)). -
hmm strange the server worked fine for months with KC 8.7.9. I guess i have to keep looking. And i’m not a Linux pro so i dont really know where to look next.
-
mysql> show processlist; +----+--------+-----------+--------+---------+-------+----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+--------+-----------+--------+---------+-------+----------+------------------+ | 6 | kopano | localhost | kopano | Sleep | 89 | | NULL | | 7 | kopano | localhost | kopano | Sleep | 15 | | NULL | | 8 | kopano | localhost | kopano | Sleep | 5 | | NULL | | 9 | kopano | localhost | kopano | Sleep | 10 | | NULL | | 10 | kopano | localhost | kopano | Sleep | 10 | | NULL | | 11 | kopano | localhost | kopano | Sleep | 35 | | NULL | | 12 | kopano | localhost | kopano | Sleep | 5 | | NULL | | 13 | kopano | localhost | kopano | Sleep | 15 | | NULL | | 16 | kopano | localhost | kopano | Sleep | 1761 | | NULL | | 55 | kopano | localhost | kopano | Sleep | 28726 | | NULL | | 56 | kopano | localhost | kopano | Sleep | 28666 | | NULL | | 57 | kopano | localhost | kopano | Sleep | 25905 | | NULL | | 58 | kopano | localhost | kopano | Sleep | 25124 | | NULL | | 59 | kopano | localhost | kopano | Sleep | 25064 | | NULL | | 62 | kopano | localhost | kopano | Sleep | 25 | | NULL | | 63 | kopano | localhost | kopano | Sleep | 22307 | | NULL | | 64 | kopano | localhost | kopano | Sleep | 21527 | | NULL | | 65 | kopano | localhost | kopano | Sleep | 21467 | | NULL | | 66 | kopano | localhost | kopano | Sleep | 18704 | | NULL | | 67 | kopano | localhost | kopano | Sleep | 17923 | | NULL | | 68 | kopano | localhost | kopano | Sleep | 17863 | | NULL | | 76 | kopano | localhost | kopano | Sleep | 15106 | | NULL | | 77 | kopano | localhost | kopano | Sleep | 14326 | | NULL | | 78 | kopano | localhost | kopano | Sleep | 14266 | | NULL | | 81 | kopano | localhost | kopano | Sleep | 4307 | | NULL | | 82 | kopano | localhost | kopano | Sleep | 3527 | | NULL | | 83 | kopano | localhost | kopano | Sleep | 3467 | | NULL | | 98 | kopano | localhost | kopano | Sleep | 704 | | NULL | | 99 | root | localhost | NULL | Query | 0 | starting | show processlist | +----+--------+-----------+--------+---------+-------+----------+------------------+ 29 rows in set (0.00 sec)is it a normal behavior that with every new search in webapp a new process is generated and not reset ?
here is my mysqld.cnf:
# # The MySQL database server configuration file. # # You can copy this to one of: # - "/etc/mysql/my.cnf" to set global options, # - "~/.my.cnf" to set user-specific options. # # One can use all long options that the program supports. # Run program with --help to get a list of available options and with # --print-defaults to see which it would actually understand and use. # # For explanations see # http://dev.mysql.com/doc/mysql/en/server-system-variables.html # This will be passed to all mysql clients # It has been reported that passwords should be enclosed with ticks/quotes # escpecially if they contain "#" chars... # Remember to edit /etc/mysql/debian.cnf when changing the socket location. # Here is entries for some specific programs # The following values assume you have at least 32M ram [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] # # * Basic Settings # user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc-messages-dir = /usr/share/mysql skip-external-locking # # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. bind-address = 127.0.0.1 # # * Fine Tuning # ##KopanoDB innodb_buffer_pool_size = 768M innodb_log_file_size = 192M innodb_log_buffer_size = 48M max_connections = 500 innodb_file_per_table ##KopanoDB key_buffer_size = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 # This replaces the startup script and checks MyISAM tables if needed # the first time they are touched myisam-recover-options = BACKUP #max_connections = 100 #table_open_cache = 64 #thread_concurrency = 10 # # * Query Cache Configuration # query_cache_limit = 1M query_cache_size = 16M # # * Logging and Replication # # Both location gets rotated by the cronjob. # Be aware that this log type is a performance killer. # As of 5.1 you can enable the log at runtime! #general_log_file = /var/log/mysql/mysql.log #general_log = 1 # # Error log - should be very few entries. # log_error = /var/log/mysql/error.log # # Here you can see queries with especially long duration #slow_query_log = 1 #slow_query_log_file = /var/log/mysql/mysql-slow.log #slow_query_log = 1 #slow_query_log_file = /var/log/mysql/mysql-slow.log #long_query_time = 2 #log-queries-not-using-indexes # # The following can be used as easy to replay backup logs or for replication. # note: if you are setting up a replication slave, see README.Debian about # other settings you may need to change. #server-id = 1 #log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 100M #binlog_do_db = include_database_name #binlog_ignore_db = include_database_name # # * InnoDB # # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/. # Read the manual for more InnoDB related options. There are many! # # * Security Features # # Read the manual, too, if you want chroot! # chroot = /var/lib/mysql/ # # For generating SSL certificates I recommend the OpenSSL GUI "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pemroot@xxx:/home/xxx# free -m total used free shared buff/cache available Mem: 4096 2167 1576 10 352 1917 Swap: 0 0 0 -
@fbartels i just installed 8.7.14 (pre-final) and keep you updated…
-
so far it looks better:
mysql> show processlist; +-----+--------+-----------+--------+---------+------+----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +-----+--------+-----------+--------+---------+------+----------+------------------+ | 6 | kopano | localhost | kopano | Sleep | 111 | | NULL | | 7 | kopano | localhost | kopano | Sleep | 9 | | NULL | | 8 | kopano | localhost | kopano | Sleep | 29 | | NULL | | 9 | kopano | localhost | kopano | Sleep | 19 | | NULL | | 10 | kopano | localhost | kopano | Sleep | 34 | | NULL | | 11 | kopano | localhost | kopano | Sleep | 4 | | NULL | | 12 | kopano | localhost | kopano | Sleep | 29 | | NULL | | 13 | kopano | localhost | kopano | Sleep | 9 | | NULL | | 14 | kopano | localhost | kopano | Sleep | 4 | | NULL | | 16 | kopano | localhost | kopano | Sleep | 218 | | NULL | | 110 | root | localhost | NULL | Query | 0 | starting | show processlist | +-----+--------+-----------+--------+---------+------+----------+------------------+ 11 rows in set (0.00 sec) -
@fbartels it looks great. No problem since installation. Still only 11 connections.
-
@xthbx thanks for you feedback
