Navigation

    Kopano
    • Register
    • Login
    • Search
    • Categories
    • Get Official Kopano Support
    • Recent
    Statement regarding the closure of the Kopano community forum and the end of the community edition

    Problems after upgrade KC 8.7.9 to 8.7.12

    Kopano Groupware Core
    5
    21
    1517
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • xthbx
      xthbx last edited by xthbx

      Hi,

      i upgraded my Ubuntu 18.04 from KC8.7.9 to KC8.7.12 .Since KC8.7.9 the server was running without any problems.

      With 8.7.12 the server is up for some time ( <24h) problems start like:

      ==> /var/log/kopano/server.log <==
      Fri May  1 09:28:38 2020: [error  ] Previous message logged 69 times
      Fri May  1 09:28:38 2020: [warning] SQL [00000024] info: Try to reconnect
      Fri May  1 09:28:38 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
      Fri May  1 09:28:38 2020: [error  ] SQL [00000000] Failed: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug, Query Size: 81, Query: "SELECT properties.val_ulong FROM properties WHERE hierarchyid = 467873 FOR UPDATE"
      Fri May  1 09:28:38 2020: [error  ] KDatabase::DoSelect(): query failed: "MYSQL not initialized", query: SELECT properties.val_ulong FROM properties WHERE hierarchyid = 467873 FOR UPDATE
      Fri May  1 09:28:38 2020: [error  ] ECSearchFolders::ResetResults(): SELECT failed: disk error (80000007)
      Fri May  1 09:28:38 2020: [error  ] ECSearchFolders::Search() ResetResults failed: disk error (80000007)
      Fri May  1 09:28:44 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
      
      ==> /var/log/kopano/search.log <==
      2020-05-01 09:28:44,706 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=4/5)
      2020-05-01 09:28:51,117 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=5/5)
      2020-05-01 09:28:57,524 - search - ERROR - Too many retries, skipping change
      2020-05-01 09:28:57,527 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=0/5)
      2020-05-01 09:29:03,937 - search - WARNING - Received a MAPI error or timeout (error=0x80040116, retry=1/5)
      
      
      
      • Search in the webapp give no result
      • kopano-spooler was unable to deliver your mail to the/some of the recipient(s).

      somehow to me it looks like it has to do something with the DB Connection. When i restart kopano-server everything works again for some time.

      Things i have done so far:

      root@xxx:/home/xxx# mysqlcheck --all-databases -u root -p
      Enter password:
      kopano.abchanges                                   OK
      kopano.acl                                         OK
      kopano.changes                                     OK
      kopano.deferredupdate                              OK
      kopano.hierarchy                                   OK
      kopano.indexedproperties                           OK
      kopano.lob                                         OK
      kopano.mvproperties                                OK
      kopano.names                                       OK
      kopano.object                                      OK
      kopano.objectmvproperty                            OK
      kopano.objectproperty                              OK
      kopano.objectrelation                              OK
      kopano.outgoingqueue                               OK
      kopano.properties                                  OK
      kopano.receivefolder                               OK
      kopano.searchresults                               OK
      kopano.settings                                    OK
      kopano.singleinstances                             OK
      kopano.stores                                      OK
      kopano.syncedmessages                              OK
      kopano.syncs                                       OK
      kopano.tproperties                                 OK
      kopano.users                                       OK
      kopano.versions                                    OK
      mysql.columns_priv                                 OK
      mysql.db                                           OK
      mysql.engine_cost                                  OK
      mysql.event                                        OK
      mysql.func                                         OK
      mysql.general_log                                  OK
      mysql.gtid_executed                                OK
      mysql.help_category                                OK
      mysql.help_keyword                                 OK
      mysql.help_relation                                OK
      mysql.help_topic                                   OK
      mysql.innodb_index_stats                           OK
      mysql.innodb_table_stats                           OK
      mysql.ndb_binlog_index                             OK
      mysql.plugin                                       OK
      mysql.proc                                         OK
      mysql.procs_priv                                   OK
      mysql.proxies_priv                                 OK
      mysql.server_cost                                  OK
      mysql.servers                                      OK
      mysql.slave_master_info                            OK
      mysql.slave_relay_log_info                         OK
      mysql.slave_worker_info                            OK
      mysql.slow_log                                     OK
      mysql.tables_priv                                  OK
      mysql.time_zone                                    OK
      mysql.time_zone_leap_second                        OK
      mysql.time_zone_name                               OK
      mysql.time_zone_transition                         OK
      mysql.time_zone_transition_type                    OK
      mysql.user                                         OK
      sys.sys_config                                     OK
      
      kopano-search --reindex
      
      kopano-server --restart-searches
      
      root@xxx:/home/xxx# free -m
                    total        used        free      shared  buff/cache   available
      Mem:           4096        2384         898          10         812        1701
      Swap:             0           0           0
      
      
      
      root@xxx:/home/xxx# df -h
      Filesystem         Size  Used Avail Use% Mounted on
      /dev/ploop26596p1  303G   82G  207G  29% /
      none               2,0G     0  2,0G   0% /sys/fs/cgroup
      none               2,0G     0  2,0G   0% /dev
      tmpfs              2,0G   52K  2,0G   1% /dev/shm
      tmpfs              2,0G  1,2M  2,0G   1% /run
      tmpfs              5,0M     0  5,0M   0% /run/lock
      none               2,0G     0  2,0G   0% /run/shm
      

      Any ideas where i should look or what i could do next?

      1 Reply Last reply Reply Quote 0
      • jengelh
        jengelh Banned last edited by jengelh

        https://stash.kopano.io/projects/KC/repos/kopanocore/pull-requests/3104/overview

        edit: https://stash.kopano.io/users/jengelhardt/repos/kc/commits?until=refs%2Fheads%2Fks-45601

        xthbx 1 Reply Last reply Reply Quote 0
        • xthbx
          xthbx @jengelh last edited by

          @jengelh said in Problems after upgrade KC 8.7.9 to 8.7.12:

          https://stash.kopano.io/projects/KC/repos/kopanocore/pull-requests/3104/overview

          sorry but i cannot access that link even after registering at bitbucket…

          1 Reply Last reply Reply Quote 0
          • A Former User
            A Former User last edited by

            I see a similar error. Kopano investigates in this issue.

            1 Reply Last reply Reply Quote 0
            • xthbx
              xthbx last edited by xthbx

              as a workaround i stopped kopano-search.

              So far the rest of the server is running.

              Update: it only slows down the time during the errors

              1 Reply Last reply Reply Quote 0
              • A Former User
                A Former User last edited by

                Hi,

                now my mail was offline for for two nights with this bug. As work around I created a script to search for this error and restart kopano-server if this error accoures.

                Download link for this script: https://www.hofstaedtler.com/tmp/check_kopano_mysql.tgz

                1. Extract the .tgz archive in /
                2. Modify the recipients mail address

                Hope this script helps.

                Walter

                1 Reply Last reply Reply Quote 0
                • xthbx
                  xthbx last edited by xthbx

                  I’m using the Kopano Contributor Edition.
                  Is there a way to get the 8.7.9 version from the download.kopano.io/supported/ download site ?

                  1 Reply Last reply Reply Quote 0
                  • A Former User
                    A Former User last edited by

                    xthbx,

                    I guess, only with a valid license code.

                    1 Reply Last reply Reply Quote 0
                    • xthbx
                      xthbx last edited by

                      i wasnt clear with my post. I got the Kopano Contributor Edition.
                      And i found the 8.7.9 tar in the archives.

                      Can anybody explain to me how i do the rollback to 8.7.9?
                      Do i have to uncomment the repository and then manualy install every package from the 8.7.9 tar?

                      Sinux 1 Reply Last reply Reply Quote 0
                      • Sinux
                        Sinux @xthbx last edited by

                        I have the kopano version myself but not the problem because my server is not always on but are there any news?

                        1 Reply Last reply Reply Quote 0
                        • fbartels
                          fbartels Kopano last edited by

                          Hi @xthbx, @Sinux.

                          8.7.13 is now available in the pre-final download directory. Your feedback would be appreciated.

                          See https://forum.kopano.io/topic/3215/kopano-core-8-7-13-pre-final for more information.

                          Regards Felix

                          Resources:
                          https://kopano.com/blog/how-to-get-kopano/
                          https://documentation.kopano.io/
                          https://kb.kopano.io/

                          Support overview:
                          https://kopano.com/support/

                          xthbx 2 Replies Last reply Reply Quote 0
                          • xthbx
                            xthbx @fbartels last edited by xthbx

                            @fbartels thanks

                            i gave it a try and keep you updated

                            Update after 1 day since upgrade 8.7.13: still no problems. Looks great

                            1 Reply Last reply Reply Quote 0
                            • xthbx
                              xthbx @fbartels last edited by xthbx

                              @fbartels i’m sorry but the Problem is still there. It just happened again.

                              ==> /var/log/kopano/server.log <==
                              Wed May 27 16:00:33 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
                              Wed May 27 16:00:33 2020: [error  ] Unable to get database connection: MYSQL not initialized
                              Wed May 27 16:00:33 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
                              Wed May 27 16:00:40 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
                              Wed May 27 16:00:40 2020: [error  ] Unable to get database connection: MYSQL not initialized
                              Wed May 27 16:00:40 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
                              Wed May 27 16:00:45 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
                              Wed May 27 16:00:45 2020: [error  ] Unable to get database connection: MYSQL not initialized
                              Wed May 27 16:00:45 2020: [error  ] ECSearchFolders::Search() GetStore failed: disk error (80000007)
                              Wed May 27 16:01:00 2020: [error  ] mysql_real_connect: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
                              
                              1 Reply Last reply Reply Quote 0
                              • jengelh
                                jengelh Banned last edited by

                                ER_CANT_CREATE_THREAD  
                                ...
                                        eng "Can't create a new thread (errno %M); if you are not out of available memory, you can consult the manual for a pos>
                                

                                –>

                                ## sql/mysqld.cc:void create_thread_to_handle_connection(CONNECT *connect)
                                
                                  /* Create new thread to handle connection */
                                  inc_thread_created();
                                  DBUG_PRINT("info",(("creating thread %lu"), (ulong) connect->thread_id));
                                  connect->prior_thr_create_utime= microsecond_interval_timer();
                                
                                  if ((error= mysql_thread_create(key_thread_one_connection,
                                                                  &connect->real_id, &connection_attrib,
                                                                  handle_one_connection, (void*) connect)))
                                  {
                                    /* purecov: begin inspected */
                                    DBUG_PRINT("error", ("Can't create thread to handle request (error %d)",
                                                error));
                                    my_snprintf(error_message_buff, sizeof(error_message_buff),
                                                ER_DEFAULT(ER_CANT_CREATE_THREAD), error);
                                    connect->close_with_error(ER_CANT_CREATE_THREAD, error_message_buff,
                                                              ER_OUT_OF_RESOURCES);
                                    DBUG_VOID_RETURN;
                                    /* purecov: end */
                                  }
                                

                                –>

                                #ifdef HAVE_PSI_THREAD_INTERFACE
                                  #define mysql_thread_create(K, P1, P2, P3, P4) \
                                    inline_mysql_thread_create(K, P1, P2, P3, P4)
                                #else
                                  #define mysql_thread_create(K, P1, P2, P3, P4) \
                                    pthread_create(P1, P2, P3, P4)
                                #endif
                                

                                So no check for connection limit exceeded (at this point), this is directly the result of pthread_create. If pthread_create fails, then you are running into some ulimit of your mysqld/system. Without further analysis of the Linux installation, I at this time conclude the fault is with your mysqld process, or the ulimits and cgroups that it has been configured with.

                                1 Reply Last reply Reply Quote 0
                                • jengelh
                                  jengelh Banned last edited by

                                  /usr/include/asm-generic/errno-base.h:#define   EAGAIN          11      /* Try again */
                                  
                                  man pthread_create:
                                  
                                         EAGAIN Insufficient resources to create another thread.
                                  
                                         EAGAIN A system-imposed limit on the number of threads was encountered.  There are a number of limits that may  trig-
                                                ger  this  error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of pro-
                                                cesses and threads for a real user ID, was reached; the kernel's system-wide limit on the number of  processes
                                                and  threads,  /proc/sys/kernel/threads-max,  was  reached  (see  proc(5));  or  the  maximum  number of PIDs,
                                                /proc/sys/kernel/pid_max, was reached (see proc(5)).
                                  
                                  1 Reply Last reply Reply Quote 0
                                  • xthbx
                                    xthbx last edited by

                                    hmm strange the server worked fine for months with KC 8.7.9. I guess i have to keep looking. And i’m not a Linux pro so i dont really know where to look next.

                                    1 Reply Last reply Reply Quote 0
                                    • xthbx
                                      xthbx last edited by xthbx

                                      mysql> show processlist;
                                      +----+--------+-----------+--------+---------+-------+----------+------------------+
                                      | Id | User   | Host      | db     | Command | Time  | State    | Info             |
                                      +----+--------+-----------+--------+---------+-------+----------+------------------+
                                      |  6 | kopano | localhost | kopano | Sleep   |    89 |          | NULL             |
                                      |  7 | kopano | localhost | kopano | Sleep   |    15 |          | NULL             |
                                      |  8 | kopano | localhost | kopano | Sleep   |     5 |          | NULL             |
                                      |  9 | kopano | localhost | kopano | Sleep   |    10 |          | NULL             |
                                      | 10 | kopano | localhost | kopano | Sleep   |    10 |          | NULL             |
                                      | 11 | kopano | localhost | kopano | Sleep   |    35 |          | NULL             |
                                      | 12 | kopano | localhost | kopano | Sleep   |     5 |          | NULL             |
                                      | 13 | kopano | localhost | kopano | Sleep   |    15 |          | NULL             |
                                      | 16 | kopano | localhost | kopano | Sleep   |  1761 |          | NULL             |
                                      | 55 | kopano | localhost | kopano | Sleep   | 28726 |          | NULL             |
                                      | 56 | kopano | localhost | kopano | Sleep   | 28666 |          | NULL             |
                                      | 57 | kopano | localhost | kopano | Sleep   | 25905 |          | NULL             |
                                      | 58 | kopano | localhost | kopano | Sleep   | 25124 |          | NULL             |
                                      | 59 | kopano | localhost | kopano | Sleep   | 25064 |          | NULL             |
                                      | 62 | kopano | localhost | kopano | Sleep   |    25 |          | NULL             |
                                      | 63 | kopano | localhost | kopano | Sleep   | 22307 |          | NULL             |
                                      | 64 | kopano | localhost | kopano | Sleep   | 21527 |          | NULL             |
                                      | 65 | kopano | localhost | kopano | Sleep   | 21467 |          | NULL             |
                                      | 66 | kopano | localhost | kopano | Sleep   | 18704 |          | NULL             |
                                      | 67 | kopano | localhost | kopano | Sleep   | 17923 |          | NULL             |
                                      | 68 | kopano | localhost | kopano | Sleep   | 17863 |          | NULL             |
                                      | 76 | kopano | localhost | kopano | Sleep   | 15106 |          | NULL             |
                                      | 77 | kopano | localhost | kopano | Sleep   | 14326 |          | NULL             |
                                      | 78 | kopano | localhost | kopano | Sleep   | 14266 |          | NULL             |
                                      | 81 | kopano | localhost | kopano | Sleep   |  4307 |          | NULL             |
                                      | 82 | kopano | localhost | kopano | Sleep   |  3527 |          | NULL             |
                                      | 83 | kopano | localhost | kopano | Sleep   |  3467 |          | NULL             |
                                      | 98 | kopano | localhost | kopano | Sleep   |   704 |          | NULL             |
                                      | 99 | root   | localhost | NULL   | Query   |     0 | starting | show processlist |
                                      +----+--------+-----------+--------+---------+-------+----------+------------------+
                                      29 rows in set (0.00 sec)
                                      

                                      is it a normal behavior that with every new search in webapp a new process is generated and not reset ?

                                      here is my mysqld.cnf:

                                      #
                                      # The MySQL database server configuration file.
                                      #
                                      # You can copy this to one of:
                                      # - "/etc/mysql/my.cnf" to set global options,
                                      # - "~/.my.cnf" to set user-specific options.
                                      #
                                      # One can use all long options that the program supports.
                                      # Run program with --help to get a list of available options and with
                                      # --print-defaults to see which it would actually understand and use.
                                      #
                                      # For explanations see
                                      # http://dev.mysql.com/doc/mysql/en/server-system-variables.html
                                      
                                      # This will be passed to all mysql clients
                                      # It has been reported that passwords should be enclosed with ticks/quotes
                                      # escpecially if they contain "#" chars...
                                      # Remember to edit /etc/mysql/debian.cnf when changing the socket location.
                                      
                                      # Here is entries for some specific programs
                                      # The following values assume you have at least 32M ram
                                      
                                      [mysqld_safe]
                                      socket          = /var/run/mysqld/mysqld.sock
                                      nice            = 0
                                      
                                      [mysqld]
                                      #
                                      # * Basic Settings
                                      #
                                      user            = mysql
                                      pid-file        = /var/run/mysqld/mysqld.pid
                                      socket          = /var/run/mysqld/mysqld.sock
                                      port            = 3306
                                      basedir         = /usr
                                      datadir         = /var/lib/mysql
                                      tmpdir          = /tmp
                                      lc-messages-dir = /usr/share/mysql
                                      skip-external-locking
                                      #
                                      # Instead of skip-networking the default is now to listen only on
                                      # localhost which is more compatible and is not less secure.
                                      bind-address            = 127.0.0.1
                                      #
                                      # * Fine Tuning
                                      #
                                      ##KopanoDB
                                      innodb_buffer_pool_size = 768M
                                      innodb_log_file_size = 192M
                                      innodb_log_buffer_size = 48M
                                      max_connections = 500
                                      innodb_file_per_table
                                      ##KopanoDB
                                      key_buffer_size         = 16M
                                      max_allowed_packet      = 16M
                                      thread_stack            = 192K
                                      thread_cache_size       = 8
                                      # This replaces the startup script and checks MyISAM tables if needed
                                      # the first time they are touched
                                      myisam-recover-options  = BACKUP
                                      #max_connections        = 100
                                      #table_open_cache       = 64
                                      #thread_concurrency     = 10
                                      #
                                      # * Query Cache Configuration
                                      #
                                      query_cache_limit       = 1M
                                      query_cache_size        = 16M
                                      #
                                      # * Logging and Replication
                                      #
                                      # Both location gets rotated by the cronjob.
                                      # Be aware that this log type is a performance killer.
                                      # As of 5.1 you can enable the log at runtime!
                                      #general_log_file        = /var/log/mysql/mysql.log
                                      #general_log             = 1
                                      #
                                      # Error log - should be very few entries.
                                      #
                                      log_error = /var/log/mysql/error.log
                                      #
                                      # Here you can see queries with especially long duration
                                      #slow_query_log         = 1
                                      #slow_query_log_file    = /var/log/mysql/mysql-slow.log
                                      #slow_query_log         = 1
                                      #slow_query_log_file    = /var/log/mysql/mysql-slow.log
                                      #long_query_time = 2
                                      #log-queries-not-using-indexes
                                      #
                                      # The following can be used as easy to replay backup logs or for replication.
                                      # note: if you are setting up a replication slave, see README.Debian about
                                      #       other settings you may need to change.
                                      #server-id              = 1
                                      #log_bin                        = /var/log/mysql/mysql-bin.log
                                      expire_logs_days        = 10
                                      max_binlog_size   = 100M
                                      #binlog_do_db           = include_database_name
                                      #binlog_ignore_db       = include_database_name
                                      #
                                      # * InnoDB
                                      #
                                      # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
                                      # Read the manual for more InnoDB related options. There are many!
                                      #
                                      # * Security Features
                                      #
                                      # Read the manual, too, if you want chroot!
                                      # chroot = /var/lib/mysql/
                                      #
                                      # For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
                                      #
                                      # ssl-ca=/etc/mysql/cacert.pem
                                      # ssl-cert=/etc/mysql/server-cert.pem
                                      # ssl-key=/etc/mysql/server-key.pem
                                      
                                      root@xxx:/home/xxx# free -m
                                                    total        used        free      shared  buff/cache   available
                                      Mem:           4096        2167        1576          10         352        1917
                                      Swap:             0           0           0
                                      
                                      1 Reply Last reply Reply Quote 0
                                      • xthbx
                                        xthbx last edited by xthbx

                                        @fbartels i just installed 8.7.14 (pre-final) and keep you updated…

                                        1 Reply Last reply Reply Quote 1
                                        • xthbx
                                          xthbx last edited by

                                          so far it looks better:

                                          mysql> show processlist;
                                          +-----+--------+-----------+--------+---------+------+----------+------------------+
                                          | Id  | User   | Host      | db     | Command | Time | State    | Info             |
                                          +-----+--------+-----------+--------+---------+------+----------+------------------+
                                          |   6 | kopano | localhost | kopano | Sleep   |  111 |          | NULL             |
                                          |   7 | kopano | localhost | kopano | Sleep   |    9 |          | NULL             |
                                          |   8 | kopano | localhost | kopano | Sleep   |   29 |          | NULL             |
                                          |   9 | kopano | localhost | kopano | Sleep   |   19 |          | NULL             |
                                          |  10 | kopano | localhost | kopano | Sleep   |   34 |          | NULL             |
                                          |  11 | kopano | localhost | kopano | Sleep   |    4 |          | NULL             |
                                          |  12 | kopano | localhost | kopano | Sleep   |   29 |          | NULL             |
                                          |  13 | kopano | localhost | kopano | Sleep   |    9 |          | NULL             |
                                          |  14 | kopano | localhost | kopano | Sleep   |    4 |          | NULL             |
                                          |  16 | kopano | localhost | kopano | Sleep   |  218 |          | NULL             |
                                          | 110 | root   | localhost | NULL   | Query   |    0 | starting | show processlist |
                                          +-----+--------+-----------+--------+---------+------+----------+------------------+
                                          11 rows in set (0.00 sec)
                                          
                                          xthbx 1 Reply Last reply Reply Quote 0
                                          • xthbx
                                            xthbx @xthbx last edited by

                                            @fbartels it looks great. No problem since installation. Still only 11 connections.

                                            fbartels 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post