mysql_real_connect: Too many connections, Unable to get database connection: MYSQL not initialized, DeserializeObject failed: disk error (80000007)
-
Yesterday, all of a sudden my long-running Kopano (core-10.0.1.190.e90604a-Debian_10-amd64.tar.gz) stopped working: It always killed my mySQL server (“too many connections”). After increasing the max_connections in my.cnf and no improvements, I found: https://forum.kopano.io/topic/3164/problems-after-upgrade-kc-8-7-9-to-8-7-12/19?loggedin=true
So I upgraded to the latest nightly (core-11.0.1.101.7cb69d8-Debian_10-amd64.tar.gz) but the problem persists :-( The linked post is a year old and older versions so I am not sure if this is even related.
After some time, my kopano server log is filled with:
2021-04-19T20:56:45.270655: [error ] mysql_real_connect: Too many connections 2021-04-19T20:56:45.270753: [error ] Unable to get database connection: MYSQL not initialized 2021-04-19T20:56:45.270799: [error ] DeserializeObject failed: disk error (80000007)Running SHOW PROCESSLIST on my mySQL server shows 518 rows (limit is currently set to 512), all from kopano.
Any advice?
-
Hi @divb,
I would recommend to check with
kopano-stats(for example with--top) to check what keeps your system busy.Regards Felix
Resources:
https://kopano.com/blog/how-to-get-kopano/
https://documentation.kopano.io/
https://kb.kopano.io/Support overview:
https://kopano.com/support/ -
@fbartels Ok, thank you.
This shows the process as “S” (sleep):
Last update: Wed Apr 21 06:33:20 2021 (1.0s since last) Sess: 7 Sess grp: 3 Users: 2 Hosts: 2 CPU: 0% QLen: 0 QAge: 0.000 RT: 0 ms SQL/s SEL: 1 UPD: 0 INS: 0 DEL: 0 Threads(idle): 2(6) MWOPS: 0 MROPS: 0 SOAP calls: 6 VERSION USERID PEER APP TIME CPUTIME CPU% NREQ STAT TASK 7,2,6,5 user 10.7.102.100 /(OUTLOOK.EXE) 1:02 0:00 0 0 S 11.0.1 SYSTEM *:pid-9450 stats/11.0.1(kopano 0:00 0:00 0 7 P 7,2,6,5 user 10.7.102.100 /(OUTLOOK.EXE) 0:00 0:00 0 0 7,2,6,5 user 10.7.102.100 /(OUTLOOK.EXE) 0:06 0:01 0 0 10.0.1 user 10.7.102.100 Apple-iPhone10C1/18 0:00 0:00 0 0 10.0.1 user 10.7.102.100 Apple-iPhone10C1/18 0:00 0:00 0 0 10.0.1 user 10.7.102.100 Apple-iPhone10C1/18 0:00 0:00 0 0After this, I checked hte command SHOW PROCESSLIST again on the mySQL server and found that it is STUCK with a query:
MariaDB [(none)]> show processlist; +-------+-------------+------------------------------+--------+---------+------+--------------------------+------------------------------------------------------------------------------------------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +-------+-------------+------------------------------+--------+---------+------+--------------------------+------------------------------------------------------------------------------------------------------+----------+ | 2 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 3 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 1 | system user | | NULL | Daemon | NULL | InnoDB purge coordinator | NULL | 0.000 | | 4 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 | | 5 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 | | 10481 | zarafa | kopano.int.example.net:50434 | zarafa | Sleep | 413 | | NULL | 0.000 | | 10483 | zarafa | kopano.int.example.net:50438 | zarafa | Sleep | 413 | | NULL | 0.000 | | 10484 | zarafa | kopano.int.example.net:50440 | zarafa | Sleep | 413 | | NULL | 0.000 | | 10485 | zarafa | kopano.int.example.net:50442 | zarafa | Sleep | 61 | | NULL | 0.000 | | 10486 | zarafa | kopano.int.example.net:50446 | zarafa | Sleep | 413 | | NULL | 0.000 | | 10493 | zarafa | kopano.int.example.net:50514 | zarafa | Sleep | 9 | | NULL | 0.000 | | 10508 | zarafa | kopano.int.example.net:50634 | zarafa | Sleep | 271 | | NULL | 0.000 | | 10511 | zarafa | kopano.int.example.net:50654 | zarafa | Query | 39 | Writing to net | SELECT 0, tag, properties.type, val_ulong, val_string, val_binary, val_double, val_longint, val_hi, | 0.000 | | 10543 | root | localhost | NULL | Query | 0 | Init | show processlist | 0.000 | +-------+-------------+------------------------------+--------+---------+------+--------------------------+------------------------------------------------------------------------------------------------------+----------+ 14 rows in set (0.000 sec)Could it be that the mariadb database has an issue? Is there any command to scan/fix for database issues?
EDIT:
- I backed up the complete database with mysqldump, recreated the database and all permissions and recovered the data. Problem persists.
- My mySQL server is on a different host. “Writing to net” seems to be network related. So I installed mariadb on the same host as kopano, completely created the database from scratch and imported the data from mysqldump. Connect via UNIX domain socket. Problem persists.
-
@divb said in mysql_real_connect: Too many connections, Unable to get database connection: MYSQL not initialized, DeserializeObject failed: disk error (80000007):
Problem persists.
Which “problem” exactly persists? Your original post talked about too many mysql connections, but the further messages show only a very low amount of processes.
Regards Felix
Resources:
https://kopano.com/blog/how-to-get-kopano/
https://documentation.kopano.io/
https://kb.kopano.io/Support overview:
https://kopano.com/support/ -
@fbartels said in mysql_real_connect: Too many connections, Unable to get database connection: MYSQL not initialized, DeserializeObject failed: disk error (80000007):
Which “problem” exactly persists? Your original post talked about too many mysql connections, but the further messages show only a very low amount of processes.
It seems that the problem is now that the query hangs, as described in my last answer.
(I should have opened a new thread for this but I think it’s relaed) -
@divb are you really sure its hanging? You can see from the processlist output that at that point in time it was only running for 39 seconds.
Regards Felix
Resources:
https://kopano.com/blog/how-to-get-kopano/
https://documentation.kopano.io/
https://kb.kopano.io/Support overview:
https://kopano.com/support/ -
@fbartels Yes, I am sure. I’ve been waiting for hours already today, while repeatedly issuing SHOW PROCESSLIST. The same query is always in there.
Also, when I sync via Outlook, synchronization hangs at “Syncing Drafts”.
-
@divb this exact query or another one with the same data? Like I said in your processlist it was only running for 39 seconds.
Regards Felix
Resources:
https://kopano.com/blog/how-to-get-kopano/
https://documentation.kopano.io/
https://kb.kopano.io/Support overview:
https://kopano.com/support/ -
@fbartels It’s always exactly this query (of course, I only see the beginning, “SELECT 0, tag, properties.type, val_ulong, val_string, val_binary, val_double, val_longint, val_hi,”, in the table).
The reason it shows only 39s is because I grabbed the data after I restarted everything (clients, mysql, kopano-server).
-
@fbartels Could you think of anything else to try that is non-destructive by any chance?
I’ve already upgraded to the latest version, installed mySQL from scatch and re-created database.
-
Locking here since you’ve opened https://forum.kopano.io/topic/3708/sql-query-brutally-hangs-errors-in-logs-any-help-appreciated for followup
