dagent.log error cant send mails via dagent

Hello Forum,

today we got a problem with sending mails via dagent (programms connected via postfix directly still sending)
foun d this error 2 minutes before the first employee tells us he cant send mails:

Tue Nov  7 10:16:58 2017: [info   ] [10451] Rule Forward matches
Tue Nov  7 10:16:58 2017: [info   ] [10451] K-1901: proc_fwd_allowed: "<our-domain>" whitelist match on "*"
Tue Nov  7 10:16:58 2017: [error  ] [10451] Rule Forward: FORWARD Unable to send forward message: Unknown error code (27ad5290)
Tue Nov  7 10:16:58 2017: [crit   ] [10451] ----------------------------------------------------------------------
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Fatal error detected. Please report all following information.
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Application kopano-dagent version: 8.4.2.0
Tue Nov  7 10:16:58 2017: [crit   ] [10451] OS: Linux, release: 3.10.0-693.2.2.el7.x86_64, version: #1 SMP Tue Sep 12 22:26:13 UTC 2017, hardware: x86_64
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Thread name: kopano-dagent
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Peak RSS: 20304
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Pid 10451 caught SIGABRT (6), out of memory or unhandled exception, traceback:
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Backtrace:
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #0. /lib64/libkcutil.so.0(+0x29b40) [0x7fd23478ab40]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #1. /lib64/libkcutil.so.0(+0x22a0f) [0x7fd234783a0f]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #2. /lib64/libkcutil.so.0(_ZN2KC23generic_sigsegv_handlerEPNS_8ECLoggerEPKcS3_iPK9siginfo_tPKv+0x1d0) [0x7fd234783d80]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #3. /lib64/libpthread.so.0(+0xf5e0) [0x7fd23104c5e0]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #4. /lib64/libc.so.6(gsignal+0x37) [0x7fd22faa41f7]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #5. /lib64/libc.so.6(abort+0x148) [0x7fd22faa58e8]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #6. /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165) [0x7fd2303aaac5]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #7. /lib64/libstdc++.so.6(+0x5ea36) [0x7fd2303a8a36]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #8. /lib64/libstdc++.so.6(+0x5ea63) [0x7fd2303a8a63]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #9. /lib64/libstdc++.so.6(+0x5f5cf) [0x7fd2303a95cf]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #10. /usr/sbin/kopano-dagent() [0x42b19a]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #11. /usr/sbin/kopano-dagent() [0x414176]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #12. /usr/sbin/kopano-dagent() [0x4160b5]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #13. /usr/sbin/kopano-dagent() [0x417dee]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #14. /usr/sbin/kopano-dagent() [0x418c10]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #15. /usr/sbin/kopano-dagent() [0x41ab84]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #16. /lib64/libkcutil.so.0(_ZN2KC18unix_fork_functionEPFPvS0_ES0_iPi+0x6e) [0x7fd2347887ee]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #17. /usr/sbin/kopano-dagent() [0x40c190]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #18. /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd22fa90c05]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] #19. /usr/sbin/kopano-dagent() [0x40d2bf]
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Signal errno: Success, signal code: -6
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Sender pid: 10451, sender uid: 0, si_status: 0
Tue Nov  7 10:16:58 2017: [crit   ] [10451] Signal value: 0, faulting address: 0x28d3
Tue Nov  7 10:16:58 2017: [crit   ] [10451] When reporting this traceback, please include Linux distribution name (and version), system architecture and Kopano version.

//EDIT: just checked for rules of user “mailarchiv” (python kopano-rules.py --user mailarchiv --list)
nothing exists, so why does kopano want to use a “FORWARD”-rule?
Ram was used to 78%

All Versions etc are in my signature

dagent still deliver incoming mails to inboxes but wont accept sending mails

coffee_is_life

Hello @coffee_is_life ,

the dagent is not sending mail, but merely delivering them (into their respective mailbox). The only exception to this may be rules that forward messages (because rules are parsed before delivery). The spooler is the process that is sending out mails.

sadly the spooler just stopped sending, logs didnt go on:

Tue Nov  7 10:07:38 2017: [info   ] [ 8566] * Loading plugins started
Tue Nov  7 10:07:38 2017: [info   ] [ 8566] ** Checking plugins in /var/lib/kopano/spooler/plugins
Tue Nov  7 10:07:38 2017: [info   ] [ 8566] * Loading plugins done
Tue Nov  7 10:07:38 2017: [info   ] [ 8566] Sending e-mail for user abergen, size: 11342
Tue Nov  7 10:07:38 2017: [info   ] [ 8566] * PreSending processing started
Tue Nov  7 10:07:38 2017: [info   ] [ 8566] * PreSending processing done
Tue Nov  7 10:07:38 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:07:38 2017: [info   ] [25829] Spooler process started on PID 8570
Tue Nov  7 10:07:38 2017: [info   ] [ 8570] * Loading plugins started
Tue Nov  7 10:07:38 2017: [info   ] [ 8570] ** Checking plugins in /var/lib/kopano/spooler/plugins
Tue Nov  7 10:07:38 2017: [info   ] [ 8570] * Loading plugins done
Tue Nov  7 10:07:38 2017: [info   ] [ 8570] Sending e-mail for user fkarp, size: 226082
Tue Nov  7 10:07:38 2017: [info   ] [ 8570] * PreSending processing started
Tue Nov  7 10:07:38 2017: [info   ] [ 8570] * PreSending processing done              

< here is the break, but it just stops logging > - no messages between this entrys

Tue Nov  7 10:51:17 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:51:17 2017: [info   ] [  847] Logger process started on pid 910
Tue Nov  7 10:51:17 2017: [ notice] [  847] Starting kopano-spooler version 8.4.2.0 (pid 847)
Tue Nov  7 10:51:17 2017: [error  ] [  847] gsoap connect: ()
Tue Nov  7 10:51:17 2017: [error  ] [  847] M4LMsgServiceAdmin::ConfigureMsgService() MSGServiceEntry failed 80040115: network error
Tue Nov  7 10:51:17 2017: [crit   ] [  847] CreateProfileTemp(): ConfigureMsgService failed 80040115: network error
Tue Nov  7 10:51:17 2017: [warning] [  847] CreateProfileTemp failed: 80040115: network error
Tue Nov  7 10:51:17 2017: [error  ] [  847] Unable to open admin session. Error 0x80040115
Tue Nov  7 10:51:17 2017: [warning] [  847] Server connection lost. Reconnecting in 3 seconds...
Tue Nov  7 10:51:20 2017: [error  ] [  847] gsoap connect: ()
Tue Nov  7 10:51:20 2017: [error  ] [  847] M4LMsgServiceAdmin::ConfigureMsgService() MSGServiceEntry failed 80040115: network error
Tue Nov  7 10:51:20 2017: [crit   ] [  847] CreateProfileTemp(): ConfigureMsgService failed 80040115: network error
Tue Nov  7 10:51:20 2017: [warning] [  847] CreateProfileTemp failed: 80040115: network error
Tue Nov  7 10:51:20 2017: [error  ] [  847] Unable to open admin session. Error 0x80040115
Tue Nov  7 10:52:03 2017: [info   ] [  847] Connection to storage server succeeded after 2 retries
Tue Nov  7 10:52:03 2017: [info   ] [  847] Spooler process started on PID 2240
Tue Nov  7 10:52:03 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:52:04 2017: [info   ] [  847] Spooler process started on PID 2243
Tue Nov  7 10:52:04 2017: [info   ] [  847] Spooler process started on PID 2244
Tue Nov  7 10:52:04 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:52:05 2017: [info   ] [  847] Spooler process started on PID 2247
Tue Nov  7 10:52:05 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:52:05 2017: [info   ] [  847] Spooler process started on PID 2250
Tue Nov  7 10:52:05 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:52:05 2017: [info   ] [  847] Spooler process started on PID 2251
Tue Nov  7 10:52:06 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:52:06 2017: [info   ] [  847] Spooler process started on PID 2254
Tue Nov  7 10:52:06 2017: [info   ] [  847] Spooler process started on PID 2257
Tue Nov  7 10:52:06 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:52:06 2017: [info   ] [  847] Spooler process started on PID 2258
Tue Nov  7 10:52:06 2017: [notice ] Coredumps are disabled via configuration file.
Tue Nov  7 10:52:06 2017: [info   ] [  847] Spooler process started on PID 2262
Tue Nov  7 10:52:06 2017: [info   ] [ 2240] * Loading plugins started
Tue Nov  7 10:52:06 2017: [info   ] [ 2240] ** Checking plugins in /var/lib/kopano/spooler/plugins

checked messages.log and found following:

Nov  7 10:51:15 pwmail002 systemd: [/usr/lib/systemd/system/kopano-dagent.service.d/scl.conf:3] Unknown lvalue 'Envirnoment' in section 'Service'


/usr/lib/systemd/system/kopano-dagent.service.d/scl.conf:
[Service]
Environment=X_SCLS=rh-php56
Envirnoment=LD_LIBRARY_PATH=/opt/rh/rh-php56/root/usr/lib64
Environment=PATH=/usr/local/sbin:/usr/local/bin:/opt/rh/rh-php56/root/usr/sbin:/opt/rh/rh-php56/root/usr/bin:/usr/sbin:/usr/bin

i think some typos in environment can cause some failures - can i just adjust this varname?

coffee_is_life

Hi @coffee_is_life ,

a 8.4.3 release has been pushed that should fix this.

Hello @fbartels,

i updated to 8.4.3 but got some new problems:

Since i restored a vmware-snapshot i cant deliver the whole traceback, but i know which error was there:

After upgrading kopano-server started, loaded searchfolders and than got the following error, leading to a traceback and KC stopped.
Error:

plugin libkcserver-ldap.so has not the same version as the KC-Server

Upgrade process:
stopping postfix
stopping all kopano-services
create vmware-snapshot

yum --nogpgcheck update kopano*

==============================================================================================================================================================================================================================================
 Package                                                            Arch                                               Version                                                  Paketquelle                                             Größe
==============================================================================================================================================================================================================================================
Aktualisieren:
 kopano-backup                                                      x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                              37 k
 kopano-client                                                      x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             378 k
 kopano-common                                                      x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                              24 k
 kopano-contacts                                                    x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                              43 k
 kopano-dagent                                                      x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             143 k
 kopano-gateway                                                     x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             147 k
 kopano-ical                                                        x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             116 k
 kopano-lang                                                        x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             141 k
 kopano-monitor                                                     x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                              43 k
 kopano-presence                                                    x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                              20 k
 kopano-search                                                      x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                              47 k
 kopano-server                                                      x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             292 k
 kopano-server-packages                                             x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             8.4 k
 kopano-spooler                                                     x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                              92 k
 kopano-utils                                                       x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             176 k
Aktualisiert für Abhängigkeiten:
 php-mapi                                                           x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             189 k
 python-mapi                                                        x86_64                                             8.4.3.4-12.1                                             Kopano-Core                                             344 k

Transaktionsübersicht
==============================================================================================================================================================================================================================================

installed the packages and …
…started kopano-server
…started dagent
…started search
tailf /var/log/kopano/search.log -> coudnt connect to server.sock
check for kopano-server -> failed with the leading error

coffee_is_life

sooo,

cloned my vm, updated kopano via same way (yum update kopano*)

aaand got the same error when starting, but this time i can post the whole traceback:

Thu Nov  9 19:02:22 2017: [info   ] Using epoll events
Thu Nov  9 19:02:22 2017: [notice ] Listening for TCP connections on port 236
Thu Nov  9 19:02:22 2017: [error  ] Coredumps will not be generated: kopano-server requires the fs.suid_dumpable sysctl to contain the value 2, not 0.
Thu Nov  9 19:02:22 2017: [notice ] Listening for priority pipe connections on /var/run/kopano/prio.sock
Thu Nov  9 19:02:22 2017: [notice ] Listening for pipe connections on /var/run/kopano/server.sock
Thu Nov  9 19:02:22 2017: [notice ] Connection to database 'zarafa' succeeded
Thu Nov  9 19:02:24 2017: [notice ] Querying database for searchfolders. This may take a while.
Thu Nov  9 19:02:26 2017: [notice ] Loading search folders.
Thu Nov  9 19:02:41 2017: [notice ] Done loading search folders.
Thu Nov  9 19:02:41 2017: [notice ] Startup succeeded on pid 2233
Thu Nov  9 19:03:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:03:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:03:01 2017: [crit   ] Unable to instantiate user plugin
Thu Nov  9 19:03:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:03:01 2017: [crit   ] Unable to instantiate user plugin
Thu Nov  9 19:03:01 2017: [crit   ] Unable to instantiate user plugin
Thu Nov  9 19:03:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:03:01 2017: [crit   ] Unable to instantiate user plugin
Thu Nov  9 19:03:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:03:01 2017: [crit   ] Unable to instantiate user plugin
Thu Nov  9 19:03:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:03:01 2017: [crit   ] Unable to instantiate user plugin
Thu Nov  9 19:04:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:04:01 2017: [crit   ] Unable to instantiate user plugin
Thu Nov  9 19:04:01 2017: [crit   ] Version of the plugin "libkcserver-ldap.so" is not the same for the server. Expected 0x80420000 (8.4.2.0), plugin 0x80430004
Thu Nov  9 19:04:01 2017: [crit   ] ----------------------------------------------------------------------

furthermore i’ve installed 8.4.3.4-12 on a fresh server and there it works… so the update routine went wrong, new install routine is fine!

i will try to just copy this plugin from my fresh installation to my clone and will post the result here (i dont think it works because the expected result of version is 8.4.2.0, so i think the internal query is wrong)

(btw the database is called zarafa, because we’ve migrated from zarafa way earlier and just didnt renamed the db :) )

coffee_is_life

Hello,

after rolling back my vm i started the update-process with the new repo, doenst work either!
BUT: after updating libkc* (yum update libkc*) system is loading and updating the plugins…
seems to be missing dependencies in kopano-server update

coffee_is_life