gateway / server - handling connection problem

sky.x

Hi, we have mailserver setup with small number of clients/account up to 20. And gateway handle only IMAPs.

for some time we have problem with crashing kopano-server due high memory consuption and high number of proceses raised in time (10-18days)

Finaly Im found that there is some problem with connection handling, mainly between gateway and server.

Normaly we have HA setup for gateway with 3 nodes balanced with HAproxy.
And in this setup the kopano-server crash first due he reach limits.
When we disabled 2 nodes of gateway and connect last node directly to server, then after some time 3-6 days gateway hang due error:

[error  ] Unable to accept IMAPs socket connection: network error (80040115)
[error  ] Unable to accept(): Too many open files

It reach kernel limits which is in my system 8000.

There is lot of conenction in CLOSE_WAIT state

lsof -i :236 | grep CLOSE_WAIT | wc -l
3999
lsof -i :236 | grep ESTABLISHED | wc -l
4001

On kopano-server node there are only ESTABLISHED connection from gateway with the same number as on kopano-gateway node.

The connections number raising in time an no one is released.
The only solution is restart kopano-gateway to release connections correctly.

Version of software is:

OpenSUSE Tumbleweed
Kernel 5.10.4
kopano-gateway 10.0.6
kopano-server 10.0.6

Thank you for any help.

thctlo

@sky-x
Assumming your using systemd , sorry i dont know if Suse is using it…

https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process Properties

Try : systemctl edit kopano-server
add :

LimitNOFILE=infinity

And others kopano services might need it also.

sky.x

@thctlo
Thank you for suggestion.

But i dont think that raising limit to unlimit is the solution. As connections are not released correctly, at the end memory resources will be consumed.

thctlo

@sky-x said in gateway / server - handling connection problem:

As connections are not released correctly

i say, try it… your max connections… well,
For that user and its service, its around that 4k with systemd these days,
so. it wont hurt if you tried it.

su into the “service user of kopano” and run
ulimit -a

sky.x

@thctlo

yes i can try it …but it will reach limits again as connections are not released correctly and they grow up in days.

the best workaround which i found is restart gateway service every night, to release properly connections.

theoneandonly-vector

@sky-x
Hey there
I got the exact same problem.
I have 7492 close-wait currently.

the version I use currently:
CORE_VERSION=10.0.6.13