kopano-server crashes after Debian 10 updates
-
At least I think it is related since I got my 2 Kopano servers down this morning with the following log entries after reboot :
Mon Feb 10 06:54:05 2020: [warning] SQL [00000024] info: Try to reconnect Mon Feb 10 06:54:05 2020: [error ] KDatabase::Connect(): database access error Unknown error code (0x80000007), mysql error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) Mon Feb 10 06:54:05 2020: [crit ] ---------------------------------------------------------------------- Mon Feb 10 06:54:05 2020: [crit ] Fatal error detected. Please report all following information. Mon Feb 10 06:54:05 2020: [crit ] Application kopano-server version: 8.7.0 Mon Feb 10 06:54:05 2020: [crit ] OS: Linux, release: 4.19.0-6-amd64, version: #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11), hardware: x86_64 Mon Feb 10 06:54:05 2020: [crit ] Thread name: z-s: Mon Feb 10 06:54:05 2020: [crit ] Peak RSS: 58064 Mon Feb 10 06:54:05 2020: [crit ] Pid 953 caught SIGSEGV (11), traceback: Mon Feb 10 06:54:05 2020: [crit ] Backtrace: Mon Feb 10 06:54:05 2020: [crit ] #0. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x490f0) [0x7f1a73d3a0f0] Mon Feb 10 06:54:05 2020: [crit ] #1. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x333fd) [0x7f1a73d243fd] Mon Feb 10 06:54:05 2020: [crit ] #2. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x1a1) [0x7f1a73d24681] Mon Feb 10 06:54:05 2020: [crit ] #3. /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f1a71178730] Mon Feb 10 06:54:05 2020: [crit ] #4. /usr/lib/x86_64-linux-gnu/libmariadb.so.3(mariadb_reconnect+0x3a) [0x7f1a71789e4a] Mon Feb 10 06:54:05 2020: [crit ] #5. /usr/lib/x86_64-linux-gnu/libmariadb.so.3(+0x264d1) [0x7f1a7178a4d1] Mon Feb 10 06:54:05 2020: [crit ] #6. /usr/lib/x86_64-linux-gnu/libmariadb.so.3(mysql_real_query+0x60) [0x7f1a71788000] Mon Feb 10 06:54:05 2020: [crit ] #7. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN2KC9KDatabase5QueryERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x7f1a73b4c422] Mon Feb 10 06:54:05 2020: [crit ] #8. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN2KC10ECDatabase5QueryERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x15) [0x7f1a73b7b5c5] Mon Feb 10 06:54:05 2020: [crit ] #9. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN2KC9KDatabase8DoSelectERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS_9DB_RESULTEb+0x53) [0x7f1a73b4c9b3] Mon Feb 10 06:54:05 2020: [crit ] #10. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN2KC10ECDatabase8DoSelectERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS_9DB_RESULTEb+0x11) [0x7f1a73b7a5a1] Mon Feb 10 06:54:05 2020: [crit ] #11. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0xac03c) [0x7f1a73b9a03c] Mon Feb 10 06:54:05 2020: [crit ] #12. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN11KCmdService10getChangesEmRK17xsd__base64BinaryjjjjP13restrictTableP17icsChangeResponse+0x23a) [0x7f1a73c1d48a] Mon Feb 10 06:54:05 2020: [crit ] #13. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(+0x19fd89) [0x7f1a73c8dd89] Mon Feb 10 06:54:05 2020: [crit ] #14. /usr/lib/x86_64-linux-gnu/libkcserver.so.0(_ZN11KCmdService8dispatchEP4soap+0x2768) [0x7f1a73c931c8] Mon Feb 10 06:54:05 2020: [crit ] #15. /usr/sbin/kopano-server(+0x15310) [0x55b2a1fd7310] Mon Feb 10 06:54:05 2020: [crit ] #16. /usr/lib/x86_64-linux-gnu/libkcutil.so.0(+0x3ab2b) [0x7f1a73d2bb2b] Mon Feb 10 06:54:05 2020: [crit ] #17. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f1a7116dfa3] Mon Feb 10 06:54:05 2020: [crit ] #18. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f1a70d7d4cf] Mon Feb 10 06:54:05 2020: [crit ] Signal errno: Success, signal code: 128 Mon Feb 10 06:54:05 2020: [crit ] Sender pid: 0, sender uid: 0, si_status: 0 Mon Feb 10 06:54:05 2020: [crit ] Signal value: 0, faulting address: (nil) Mon Feb 10 06:54:05 2020: [crit ] When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version.
OS : Debian 10.3 (4.19.0-8-cloud-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux) and Kopano 8.7
I can’t see what is going wrong, might be OS related…
Joris.
-
MariaDB on Debian 10 is a bit unforgiving. To fix this, commit https://stash.kopano.io/projects/KC/repos/kopanocore/commits/2944c0e57103647a882a378eb796519030e013ed is required.
-
Thanks. I got one Kopano server back. The other doesn’t give the SQL error anymore neither but gets stuck on this :
Mon Feb 10 10:56:39 2020: [info ] Audit logging not enabled. Mon Feb 10 10:56:39 2020: [=======] Starting kopano-server version 8.7.0 (pid 2880 uid 0) Mon Feb 10 10:56:39 2020: [info ] Using epoll events Mon Feb 10 10:56:39 2020: [notice ] Listening for TCP connections on port 236 Mon Feb 10 10:56:39 2020: [notice ] Listening for priority pipe connections on /var/run/kopano/prio.sock Mon Feb 10 10:56:39 2020: [notice ] Listening for pipe connections on /var/run/kopano/server.sock Mon Feb 10 10:56:39 2020: [notice ] Coredumps are disabled via configuration file. Mon Feb 10 10:56:39 2020: [debug ] Reexecing /usr/sbin/kopano-server Mon Feb 10 10:56:39 2020: [notice ] K-1240: Failed to re-exec self: Permission denied. Continuing with standard allocator and/or restricted coredumps. Mon Feb 10 10:56:39 2020: [notice ] Connection to database 'kopanoserver' succeeded Mon Feb 10 10:56:39 2020: [error ] Unable to write attachments to the directory '/mnt/sdb/kopano/attachments' - Permission denied. Please check the directory and sub directories. Mon Feb 10 10:56:39 2020: [notice ] Waiting for mysql_server_end Mon Feb 10 10:56:39 2020: [notice ] Waiting for mysql_library_end Mon Feb 10 10:56:39 2020: [=======] Server shutdown complete.
This has nothing to do with the previous issue of course but still stops the kopano-server service from starting. I don’t understand, my permissions are ok and also tried resetting them with chown -R kopano:kopano /mnt/sdb/kopano/attachments but still the same… This worked before (server is in prod since a few months)…
-
I’ll close this one here and open a new one as this is not related to the original issue anymore.