Problems after upgrade KC 8.7.9 to 8.7.12

jengelh

https://stash.kopano.io/projects/KC/repos/kopanocore/pull-requests/3104/overview

edit: https://stash.kopano.io/users/jengelhardt/repos/kc/commits?until=refs%2Fheads%2Fks-45601

xthbx

@jengelh said in Problems after upgrade KC 8.7.9 to 8.7.12:

https://stash.kopano.io/projects/KC/repos/kopanocore/pull-requests/3104/overview

sorry but i cannot access that link even after registering at bitbucket…

I see a similar error. Kopano investigates in this issue.

xthbx

as a workaround i stopped kopano-search.

So far the rest of the server is running.

Update: it only slows down the time during the errors

Hi,

now my mail was offline for for two nights with this bug. As work around I created a script to search for this error and restart kopano-server if this error accoures.

Download link for this script: https://www.hofstaedtler.com/tmp/check_kopano_mysql.tgz

Extract the .tgz archive in /
Modify the recipients mail address

Hope this script helps.

Walter

xthbx

I’m using the Kopano Contributor Edition.
Is there a way to get the 8.7.9 version from the download.kopano.io/supported/ download site ?

xthbx,

I guess, only with a valid license code.

xthbx

i wasnt clear with my post. I got the Kopano Contributor Edition.
And i found the 8.7.9 tar in the archives.

Can anybody explain to me how i do the rollback to 8.7.9?
Do i have to uncomment the repository and then manualy install every package from the 8.7.9 tar?

Sinux

I have the kopano version myself but not the problem because my server is not always on but are there any news?

fbartels

Hi @xthbx, @Sinux.

8.7.13 is now available in the pre-final download directory. Your feedback would be appreciated.

See https://forum.kopano.io/topic/3215/kopano-core-8-7-13-pre-final for more information.

xthbx

@fbartels thanks

i gave it a try and keep you updated

Update after 1 day since upgrade 8.7.13: still no problems. Looks great

xthbx

@fbartels i’m sorry but the Problem is still there. It just happened again.

==> /var/log/kopano/server.log <==
Wed May 27 16:00:33 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
Wed May 27 16:00:33 2020: [error  ] Unable to get database connection: MYSQL not initialized
Wed May 27 16:00:33 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
Wed May 27 16:00:40 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
Wed May 27 16:00:40 2020: [error  ] Unable to get database connection: MYSQL not initialized
Wed May 27 16:00:40 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
Wed May 27 16:00:45 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
Wed May 27 16:00:45 2020: [error  ] Unable to get database connection: MYSQL not initialized
Wed May 27 16:00:45 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
Wed May 27 16:01:00 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

jengelh

ER_CANT_CREATE_THREAD  
...
        eng "Can't create a new thread (errno %M); if you are not out of available memory, you can consult the manual for a pos>

–>

## sql/mysqld.cc:void create_thread_to_handle_connection(CONNECT *connect)

  /* Create new thread to handle connection */
  inc_thread_created();
  DBUG_PRINT("info",(("creating thread %lu"), (ulong) connect->thread_id));
  connect->prior_thr_create_utime= microsecond_interval_timer();

  if ((error= mysql_thread_create(key_thread_one_connection,
                                  &connect->real_id, &connection_attrib,
                                  handle_one_connection, (void*) connect)))
  {
    /* purecov: begin inspected */
    DBUG_PRINT("error", ("Can't create thread to handle request (error %d)",
                error));
    my_snprintf(error_message_buff, sizeof(error_message_buff),
                ER_DEFAULT(ER_CANT_CREATE_THREAD), error);
    connect->close_with_error(ER_CANT_CREATE_THREAD, error_message_buff,
                              ER_OUT_OF_RESOURCES);
    DBUG_VOID_RETURN;
    /* purecov: end */
  }

–>

#ifdef HAVE_PSI_THREAD_INTERFACE
  #define mysql_thread_create(K, P1, P2, P3, P4) \
    inline_mysql_thread_create(K, P1, P2, P3, P4)
#else
  #define mysql_thread_create(K, P1, P2, P3, P4) \
    pthread_create(P1, P2, P3, P4)
#endif

So no check for connection limit exceeded (at this point), this is directly the result of pthread_create. If pthread_create fails, then you are running into some ulimit of your mysqld/system. Without further analysis of the Linux installation, I at this time conclude the fault is with your mysqld process, or the ulimits and cgroups that it has been configured with.

jengelh

/usr/include/asm-generic/errno-base.h:#define   EAGAIN          11      /* Try again */

man pthread_create:

       EAGAIN Insufficient resources to create another thread.

       EAGAIN A system-imposed limit on the number of threads was encountered.  There are a number of limits that may  trig-
              ger  this  error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of pro-
              cesses and threads for a real user ID, was reached; the kernel's system-wide limit on the number of  processes
              and  threads,  /proc/sys/kernel/threads-max,  was  reached  (see  proc(5));  or  the  maximum  number of PIDs,
              /proc/sys/kernel/pid_max, was reached (see proc(5)).

xthbx

hmm strange the server worked fine for months with KC 8.7.9. I guess i have to keep looking. And i’m not a Linux pro so i dont really know where to look next.

xthbx

mysql> show processlist;
+----+--------+-----------+--------+---------+-------+----------+------------------+
| Id | User   | Host      | db     | Command | Time  | State    | Info             |
+----+--------+-----------+--------+---------+-------+----------+------------------+
|  6 | kopano | localhost | kopano | Sleep   |    89 |          | NULL             |
|  7 | kopano | localhost | kopano | Sleep   |    15 |          | NULL             |
|  8 | kopano | localhost | kopano | Sleep   |     5 |          | NULL             |
|  9 | kopano | localhost | kopano | Sleep   |    10 |          | NULL             |
| 10 | kopano | localhost | kopano | Sleep   |    10 |          | NULL             |
| 11 | kopano | localhost | kopano | Sleep   |    35 |          | NULL             |
| 12 | kopano | localhost | kopano | Sleep   |     5 |          | NULL             |
| 13 | kopano | localhost | kopano | Sleep   |    15 |          | NULL             |
| 16 | kopano | localhost | kopano | Sleep   |  1761 |          | NULL             |
| 55 | kopano | localhost | kopano | Sleep   | 28726 |          | NULL             |
| 56 | kopano | localhost | kopano | Sleep   | 28666 |          | NULL             |
| 57 | kopano | localhost | kopano | Sleep   | 25905 |          | NULL             |
| 58 | kopano | localhost | kopano | Sleep   | 25124 |          | NULL             |
| 59 | kopano | localhost | kopano | Sleep   | 25064 |          | NULL             |
| 62 | kopano | localhost | kopano | Sleep   |    25 |          | NULL             |
| 63 | kopano | localhost | kopano | Sleep   | 22307 |          | NULL             |
| 64 | kopano | localhost | kopano | Sleep   | 21527 |          | NULL             |
| 65 | kopano | localhost | kopano | Sleep   | 21467 |          | NULL             |
| 66 | kopano | localhost | kopano | Sleep   | 18704 |          | NULL             |
| 67 | kopano | localhost | kopano | Sleep   | 17923 |          | NULL             |
| 68 | kopano | localhost | kopano | Sleep   | 17863 |          | NULL             |
| 76 | kopano | localhost | kopano | Sleep   | 15106 |          | NULL             |
| 77 | kopano | localhost | kopano | Sleep   | 14326 |          | NULL             |
| 78 | kopano | localhost | kopano | Sleep   | 14266 |          | NULL             |
| 81 | kopano | localhost | kopano | Sleep   |  4307 |          | NULL             |
| 82 | kopano | localhost | kopano | Sleep   |  3527 |          | NULL             |
| 83 | kopano | localhost | kopano | Sleep   |  3467 |          | NULL             |
| 98 | kopano | localhost | kopano | Sleep   |   704 |          | NULL             |
| 99 | root   | localhost | NULL   | Query   |     0 | starting | show processlist |
+----+--------+-----------+--------+---------+-------+----------+------------------+
29 rows in set (0.00 sec)

is it a normal behavior that with every new search in webapp a new process is generated and not reset ?

here is my mysqld.cnf:

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]
#
# * Basic Settings
#
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 127.0.0.1
#
# * Fine Tuning
#
##KopanoDB
innodb_buffer_pool_size = 768M
innodb_log_file_size = 192M
innodb_log_buffer_size = 48M
max_connections = 500
innodb_file_per_table
##KopanoDB
key_buffer_size         = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options  = BACKUP
#max_connections        = 100
#table_open_cache       = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#slow_query_log         = 1
#slow_query_log_file    = /var/log/mysql/mysql-slow.log
#slow_query_log         = 1
#slow_query_log_file    = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id              = 1
#log_bin                        = /var/log/mysql/mysql-bin.log
expire_logs_days        = 10
max_binlog_size   = 100M
#binlog_do_db           = include_database_name
#binlog_ignore_db       = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem

root@xxx:/home/xxx# free -m
              total        used        free      shared  buff/cache   available
Mem:           4096        2167        1576          10         352        1917
Swap:             0           0           0

xthbx

@fbartels i just installed 8.7.14 (pre-final) and keep you updated…

xthbx

so far it looks better:

mysql> show processlist;
+-----+--------+-----------+--------+---------+------+----------+------------------+
| Id  | User   | Host      | db     | Command | Time | State    | Info             |
+-----+--------+-----------+--------+---------+------+----------+------------------+
|   6 | kopano | localhost | kopano | Sleep   |  111 |          | NULL             |
|   7 | kopano | localhost | kopano | Sleep   |    9 |          | NULL             |
|   8 | kopano | localhost | kopano | Sleep   |   29 |          | NULL             |
|   9 | kopano | localhost | kopano | Sleep   |   19 |          | NULL             |
|  10 | kopano | localhost | kopano | Sleep   |   34 |          | NULL             |
|  11 | kopano | localhost | kopano | Sleep   |    4 |          | NULL             |
|  12 | kopano | localhost | kopano | Sleep   |   29 |          | NULL             |
|  13 | kopano | localhost | kopano | Sleep   |    9 |          | NULL             |
|  14 | kopano | localhost | kopano | Sleep   |    4 |          | NULL             |
|  16 | kopano | localhost | kopano | Sleep   |  218 |          | NULL             |
| 110 | root   | localhost | NULL   | Query   |    0 | starting | show processlist |
+-----+--------+-----------+--------+---------+------+----------+------------------+
11 rows in set (0.00 sec)

xthbx

@fbartels it looks great. No problem since installation. Still only 11 connections.

fbartels

@xthbx thanks for you feedback