Problems after upgrade KC 8.7.9 to 8.7.12


  • Hi,

    i upgraded my Ubuntu 18.04 from KC8.7.9 to KC8.7.12 .Since KC8.7.9 the server was running without any problems.

    With 8.7.12 the server is up for some time ( <24h) problems start like:

    ==> /var/log/kopano/server.log <==
    Fri May  1 09:28:38 2020: [error  ] Previous message logged 69 times
    Fri May  1 09:28:38 2020: [warning] SQL [00000024] info: Try to reconnect
    Fri May  1 09:28:38 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
    Fri May  1 09:28:38 2020: [error  ] SQL [00000000] Failed: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug, Query Size: 81, Query: "SELECT properties.val_ulong FROM properties WHERE hierarchyid = 467873 FOR UPDATE"
    Fri May  1 09:28:38 2020: [error  ] KDatabase::DoSelect(): query failed: "MYSQL not initialized", query: SELECT properties.val_ulong FROM properties WHERE hierarchyid = 467873 FOR UPDATE
    Fri May  1 09:28:38 2020: [error  ] ECSearchFolders::ResetResults(): SELECT failed: disk error (80000007)
    Fri May  1 09:28:38 2020: [error  ] ECSearchFolders::Search() ResetResults failed: disk error (80000007)
    Fri May  1 09:28:44 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
    
    ==> /var/log/kopano/search.log <==
    2020-05-01 09:28:44,706 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=4/5)
    2020-05-01 09:28:51,117 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=5/5)
    2020-05-01 09:28:57,524 - search - ERROR - Too many retries, skipping change
    2020-05-01 09:28:57,527 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=0/5)
    2020-05-01 09:29:03,937 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=1/5)
    
    
    
    • Search in the webapp give no result
    • kopano-spooler was unable to deliver your mail to the/some of the recipient(s).

    somehow to me it looks like it has to do something with the DB Connection. When i restart kopano-server everything works again for some time.

    Things i have done so far:

    root@xxx:/home/xxx# mysqlcheck --all-databases -u root -p
    Enter password:
    kopano.abchanges                                   OK
    kopano.acl                                         OK
    kopano.changes                                     OK
    kopano.deferredupdate                              OK
    kopano.hierarchy                                   OK
    kopano.indexedproperties                           OK
    kopano.lob                                         OK
    kopano.mvproperties                                OK
    kopano.names                                       OK
    kopano.object                                      OK
    kopano.objectmvproperty                            OK
    kopano.objectproperty                              OK
    kopano.objectrelation                              OK
    kopano.outgoingqueue                               OK
    kopano.properties                                  OK
    kopano.receivefolder                               OK
    kopano.searchresults                               OK
    kopano.settings                                    OK
    kopano.singleinstances                             OK
    kopano.stores                                      OK
    kopano.syncedmessages                              OK
    kopano.syncs                                       OK
    kopano.tproperties                                 OK
    kopano.users                                       OK
    kopano.versions                                    OK
    mysql.columns_priv                                 OK
    mysql.db                                           OK
    mysql.engine_cost                                  OK
    mysql.event                                        OK
    mysql.func                                         OK
    mysql.general_log                                  OK
    mysql.gtid_executed                                OK
    mysql.help_category                                OK
    mysql.help_keyword                                 OK
    mysql.help_relation                                OK
    mysql.help_topic                                   OK
    mysql.innodb_index_stats                           OK
    mysql.innodb_table_stats                           OK
    mysql.ndb_binlog_index                             OK
    mysql.plugin                                       OK
    mysql.proc                                         OK
    mysql.procs_priv                                   OK
    mysql.proxies_priv                                 OK
    mysql.server_cost                                  OK
    mysql.servers                                      OK
    mysql.slave_master_info                            OK
    mysql.slave_relay_log_info                         OK
    mysql.slave_worker_info                            OK
    mysql.slow_log                                     OK
    mysql.tables_priv                                  OK
    mysql.time_zone                                    OK
    mysql.time_zone_leap_second                        OK
    mysql.time_zone_name                               OK
    mysql.time_zone_transition                         OK
    mysql.time_zone_transition_type                    OK
    mysql.user                                         OK
    sys.sys_config                                     OK
    
    kopano-search --reindex
    
    kopano-server --restart-searches
    
    root@xxx:/home/xxx# free -m
                  total        used        free      shared  buff/cache   available
    Mem:           4096        2384         898          10         812        1701
    Swap:             0           0           0
    
    
    
    root@xxx:/home/xxx# df -h
    Filesystem         Size  Used Avail Use% Mounted on
    /dev/ploop26596p1  303G   82G  207G  29% /
    none               2,0G     0  2,0G   0% /sys/fs/cgroup
    none               2,0G     0  2,0G   0% /dev
    tmpfs              2,0G   52K  2,0G   1% /dev/shm
    tmpfs              2,0G  1,2M  2,0G   1% /run
    tmpfs              5,0M     0  5,0M   0% /run/lock
    none               2,0G     0  2,0G   0% /run/shm
    

    Any ideas where i should look or what i could do next?



  • @jengelh said in Problems after upgrade KC 8.7.9 to 8.7.12:

    https://stash.kopano.io/projects/KC/repos/kopanocore/pull-requests/3104/overview

    sorry but i cannot access that link even after registering at bitbucket…


  • I see a similar error. Kopano investigates in this issue.


  • as a workaround i stopped kopano-search.

    So far the rest of the server is running.

    Update: it only slows down the time during the errors


  • Hi,

    now my mail was offline for for two nights with this bug. As work around I created a script to search for this error and restart kopano-server if this error accoures.

    Download link for this script: https://www.hofstaedtler.com/tmp/check_kopano_mysql.tgz

    1. Extract the .tgz archive in /
    2. Modify the recipients mail address

    Hope this script helps.

    Walter


  • I’m using the Kopano Contributor Edition.
    Is there a way to get the 8.7.9 version from the download.kopano.io/supported/ download site ?


  • xthbx,

    I guess, only with a valid license code.


  • i wasnt clear with my post. I got the Kopano Contributor Edition.
    And i found the 8.7.9 tar in the archives.

    Can anybody explain to me how i do the rollback to 8.7.9?
    Do i have to uncomment the repository and then manualy install every package from the 8.7.9 tar?


  • I have the kopano version myself but not the problem because my server is not always on but are there any news?

  • Kopano

    Hi @xthbx, @Sinux.

    8.7.13 is now available in the pre-final download directory. Your feedback would be appreciated.

    See https://forum.kopano.io/topic/3215/kopano-core-8-7-13-pre-final for more information.


  • @fbartels thanks

    i gave it a try and keep you updated

    Update after 1 day since upgrade 8.7.13: still no problems. Looks great


  • @fbartels i’m sorry but the Problem is still there. It just happened again.

    ==> /var/log/kopano/server.log <==
    Wed May 27 16:00:33 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
    Wed May 27 16:00:33 2020: [error  ] Unable to get database connection: MYSQL not initialized
    Wed May 27 16:00:33 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
    Wed May 27 16:00:40 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
    Wed May 27 16:00:40 2020: [error  ] Unable to get database connection: MYSQL not initialized
    Wed May 27 16:00:40 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
    Wed May 27 16:00:45 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
    Wed May 27 16:00:45 2020: [error  ] Unable to get database connection: MYSQL not initialized
    Wed May 27 16:00:45 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
    Wed May 27 16:01:00 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
    

  • ER_CANT_CREATE_THREAD  
    ...
            eng "Can't create a new thread (errno %M); if you are not out of available memory, you can consult the manual for a pos>
    

    –>

    ## sql/mysqld.cc:void create_thread_to_handle_connection(CONNECT *connect)
    
      /* Create new thread to handle connection */
      inc_thread_created();
      DBUG_PRINT("info",(("creating thread %lu"), (ulong) connect->thread_id));
      connect->prior_thr_create_utime= microsecond_interval_timer();
    
      if ((error= mysql_thread_create(key_thread_one_connection,
                                      &connect->real_id, &connection_attrib,
                                      handle_one_connection, (void*) connect)))
      {
        /* purecov: begin inspected */
        DBUG_PRINT("error", ("Can't create thread to handle request (error %d)",
                    error));
        my_snprintf(error_message_buff, sizeof(error_message_buff),
                    ER_DEFAULT(ER_CANT_CREATE_THREAD), error);
        connect->close_with_error(ER_CANT_CREATE_THREAD, error_message_buff,
                                  ER_OUT_OF_RESOURCES);
        DBUG_VOID_RETURN;
        /* purecov: end */
      }
    

    –>

    #ifdef HAVE_PSI_THREAD_INTERFACE
      #define mysql_thread_create(K, P1, P2, P3, P4) \
        inline_mysql_thread_create(K, P1, P2, P3, P4)
    #else
      #define mysql_thread_create(K, P1, P2, P3, P4) \
        pthread_create(P1, P2, P3, P4)
    #endif
    

    So no check for connection limit exceeded (at this point), this is directly the result of pthread_create. If pthread_create fails, then you are running into some ulimit of your mysqld/system. Without further analysis of the Linux installation, I at this time conclude the fault is with your mysqld process, or the ulimits and cgroups that it has been configured with.


  • /usr/include/asm-generic/errno-base.h:#define   EAGAIN          11      /* Try again */
    
    man pthread_create:
    
           EAGAIN Insufficient resources to create another thread.
    
           EAGAIN A system-imposed limit on the number of threads was encountered.  There are a number of limits that may  trig-
                  ger  this  error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of pro-
                  cesses and threads for a real user ID, was reached; the kernel's system-wide limit on the number of  processes
                  and  threads,  /proc/sys/kernel/threads-max,  was  reached  (see  proc(5));  or  the  maximum  number of PIDs,
                  /proc/sys/kernel/pid_max, was reached (see proc(5)).
    

  • hmm strange the server worked fine for months with KC 8.7.9. I guess i have to keep looking. And i’m not a Linux pro so i dont really know where to look next.


  • mysql> show processlist;
    +----+--------+-----------+--------+---------+-------+----------+------------------+
    | Id | User   | Host      | db     | Command | Time  | State    | Info             |
    +----+--------+-----------+--------+---------+-------+----------+------------------+
    |  6 | kopano | localhost | kopano | Sleep   |    89 |          | NULL             |
    |  7 | kopano | localhost | kopano | Sleep   |    15 |          | NULL             |
    |  8 | kopano | localhost | kopano | Sleep   |     5 |          | NULL             |
    |  9 | kopano | localhost | kopano | Sleep   |    10 |          | NULL             |
    | 10 | kopano | localhost | kopano | Sleep   |    10 |          | NULL             |
    | 11 | kopano | localhost | kopano | Sleep   |    35 |          | NULL             |
    | 12 | kopano | localhost | kopano | Sleep   |     5 |          | NULL             |
    | 13 | kopano | localhost | kopano | Sleep   |    15 |          | NULL             |
    | 16 | kopano | localhost | kopano | Sleep   |  1761 |          | NULL             |
    | 55 | kopano | localhost | kopano | Sleep   | 28726 |          | NULL             |
    | 56 | kopano | localhost | kopano | Sleep   | 28666 |          | NULL             |
    | 57 | kopano | localhost | kopano | Sleep   | 25905 |          | NULL             |
    | 58 | kopano | localhost | kopano | Sleep   | 25124 |          | NULL             |
    | 59 | kopano | localhost | kopano | Sleep   | 25064 |          | NULL             |
    | 62 | kopano | localhost | kopano | Sleep   |    25 |          | NULL             |
    | 63 | kopano | localhost | kopano | Sleep   | 22307 |          | NULL             |
    | 64 | kopano | localhost | kopano | Sleep   | 21527 |          | NULL             |
    | 65 | kopano | localhost | kopano | Sleep   | 21467 |          | NULL             |
    | 66 | kopano | localhost | kopano | Sleep   | 18704 |          | NULL             |
    | 67 | kopano | localhost | kopano | Sleep   | 17923 |          | NULL             |
    | 68 | kopano | localhost | kopano | Sleep   | 17863 |          | NULL             |
    | 76 | kopano | localhost | kopano | Sleep   | 15106 |          | NULL             |
    | 77 | kopano | localhost | kopano | Sleep   | 14326 |          | NULL             |
    | 78 | kopano | localhost | kopano | Sleep   | 14266 |          | NULL             |
    | 81 | kopano | localhost | kopano | Sleep   |  4307 |          | NULL             |
    | 82 | kopano | localhost | kopano | Sleep   |  3527 |          | NULL             |
    | 83 | kopano | localhost | kopano | Sleep   |  3467 |          | NULL             |
    | 98 | kopano | localhost | kopano | Sleep   |   704 |          | NULL             |
    | 99 | root   | localhost | NULL   | Query   |     0 | starting | show processlist |
    +----+--------+-----------+--------+---------+-------+----------+------------------+
    29 rows in set (0.00 sec)
    

    is it a normal behavior that with every new search in webapp a new process is generated and not reset ?

    here is my mysqld.cnf:

    #
    # The MySQL database server configuration file.
    #
    # You can copy this to one of:
    # - "/etc/mysql/my.cnf" to set global options,
    # - "~/.my.cnf" to set user-specific options.
    #
    # One can use all long options that the program supports.
    # Run program with --help to get a list of available options and with
    # --print-defaults to see which it would actually understand and use.
    #
    # For explanations see
    # http://dev.mysql.com/doc/mysql/en/server-system-variables.html
    
    # This will be passed to all mysql clients
    # It has been reported that passwords should be enclosed with ticks/quotes
    # escpecially if they contain "#" chars...
    # Remember to edit /etc/mysql/debian.cnf when changing the socket location.
    
    # Here is entries for some specific programs
    # The following values assume you have at least 32M ram
    
    [mysqld_safe]
    socket          = /var/run/mysqld/mysqld.sock
    nice            = 0
    
    [mysqld]
    #
    # * Basic Settings
    #
    user            = mysql
    pid-file        = /var/run/mysqld/mysqld.pid
    socket          = /var/run/mysqld/mysqld.sock
    port            = 3306
    basedir         = /usr
    datadir         = /var/lib/mysql
    tmpdir          = /tmp
    lc-messages-dir = /usr/share/mysql
    skip-external-locking
    #
    # Instead of skip-networking the default is now to listen only on
    # localhost which is more compatible and is not less secure.
    bind-address            = 127.0.0.1
    #
    # * Fine Tuning
    #
    ##KopanoDB
    innodb_buffer_pool_size = 768M
    innodb_log_file_size = 192M
    innodb_log_buffer_size = 48M
    max_connections = 500
    innodb_file_per_table
    ##KopanoDB
    key_buffer_size         = 16M
    max_allowed_packet      = 16M
    thread_stack            = 192K
    thread_cache_size       = 8
    # This replaces the startup script and checks MyISAM tables if needed
    # the first time they are touched
    myisam-recover-options  = BACKUP
    #max_connections        = 100
    #table_open_cache       = 64
    #thread_concurrency     = 10
    #
    # * Query Cache Configuration
    #
    query_cache_limit       = 1M
    query_cache_size        = 16M
    #
    # * Logging and Replication
    #
    # Both location gets rotated by the cronjob.
    # Be aware that this log type is a performance killer.
    # As of 5.1 you can enable the log at runtime!
    #general_log_file        = /var/log/mysql/mysql.log
    #general_log             = 1
    #
    # Error log - should be very few entries.
    #
    log_error = /var/log/mysql/error.log
    #
    # Here you can see queries with especially long duration
    #slow_query_log         = 1
    #slow_query_log_file    = /var/log/mysql/mysql-slow.log
    #slow_query_log         = 1
    #slow_query_log_file    = /var/log/mysql/mysql-slow.log
    #long_query_time = 2
    #log-queries-not-using-indexes
    #
    # The following can be used as easy to replay backup logs or for replication.
    # note: if you are setting up a replication slave, see README.Debian about
    #       other settings you may need to change.
    #server-id              = 1
    #log_bin                        = /var/log/mysql/mysql-bin.log
    expire_logs_days        = 10
    max_binlog_size   = 100M
    #binlog_do_db           = include_database_name
    #binlog_ignore_db       = include_database_name
    #
    # * InnoDB
    #
    # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
    # Read the manual for more InnoDB related options. There are many!
    #
    # * Security Features
    #
    # Read the manual, too, if you want chroot!
    # chroot = /var/lib/mysql/
    #
    # For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
    #
    # ssl-ca=/etc/mysql/cacert.pem
    # ssl-cert=/etc/mysql/server-cert.pem
    # ssl-key=/etc/mysql/server-key.pem
    
    root@xxx:/home/xxx# free -m
                  total        used        free      shared  buff/cache   available
    Mem:           4096        2167        1576          10         352        1917
    Swap:             0           0           0
    

  • @fbartels i just installed 8.7.14 (pre-final) and keep you updated…


  • so far it looks better:

    mysql> show processlist;
    +-----+--------+-----------+--------+---------+------+----------+------------------+
    | Id  | User   | Host      | db     | Command | Time | State    | Info             |
    +-----+--------+-----------+--------+---------+------+----------+------------------+
    |   6 | kopano | localhost | kopano | Sleep   |  111 |          | NULL             |
    |   7 | kopano | localhost | kopano | Sleep   |    9 |          | NULL             |
    |   8 | kopano | localhost | kopano | Sleep   |   29 |          | NULL             |
    |   9 | kopano | localhost | kopano | Sleep   |   19 |          | NULL             |
    |  10 | kopano | localhost | kopano | Sleep   |   34 |          | NULL             |
    |  11 | kopano | localhost | kopano | Sleep   |    4 |          | NULL             |
    |  12 | kopano | localhost | kopano | Sleep   |   29 |          | NULL             |
    |  13 | kopano | localhost | kopano | Sleep   |    9 |          | NULL             |
    |  14 | kopano | localhost | kopano | Sleep   |    4 |          | NULL             |
    |  16 | kopano | localhost | kopano | Sleep   |  218 |          | NULL             |
    | 110 | root   | localhost | NULL   | Query   |    0 | starting | show processlist |
    +-----+--------+-----------+--------+---------+------+----------+------------------+
    11 rows in set (0.00 sec)
    

  • @fbartels it looks great. No problem since installation. Still only 11 connections.