error connecting to IMAPs via gateway (core 8.6.81.416)

Hello fellow forum users,

after upgrading to 8.6.81.416 I can not connect to my gateway via IMAPs. I get connection refused running it remotely via

openssl s_client -connect domain.xx:993
connect: Operation timed out
connect:errno=60

running it on the server I get the following:

openssl s_client -connect 127.0.0.1:993
139664058627328:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:108:
139664058627328:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:109:
connect:errno=111

can anyone advice? Thank you.

This is my config:

#pop3s_listen =
imap_listen = *:143
imaps_listen = *:993

# Connection to the storage server.
# Please refer to the administrator manual or manpage why HTTP is used rather than the UNIX s$
server_socket = http://localhost:236/

# Set this value to a name to show in the logon greeting to clients.
# Leave empty to use DNS to find this name.
server_hostname =

# Whether to show the hostname in the logon greeting to clients.
server_hostname_greeting = no

# drop privileges and run the process as this user
#run_as_user = kopano
# drop privileges and run the process as this group
#run_as_group = kopano

# create a pid file for stopping the service via the init.d scripts
#pid_file = /var/run/kopano/gateway.pid

# run server in this path (when not using the -F switch)
#running_path = /var/lib/kopano/empty

# create memory coredumps upon crash [no, systemdefault, yes]
#coredump_enabled = systemdefault

# Only mail folder for IMAP or all subfolders (calendar, contacts, tasks, etc. too)
imap_only_mailfolders = yes

# Show Public folders for IMAP
imap_public_folders = yes

# IMAP clients may use IDLE command
imap_capability_idle = yes

# The maximum size of an email that can be uploaded to the gateway
imap_max_messagesize = 128M

# Internally issue the expunge command to directly delete e-mail marked for deletion in IMAP.
imap_expunge_on_delete = no

# Maximum count of allowed failed IMAP command counts per client
imap_max_fail_commands = 50

# Some MUAs are sending commands via idle causing the connection
# to reach imap_max_fail_commands and leaves the client in a
# broken state. The clients include Apple Mail. If you experience
# problems or uses Apple Mail set this option to yes
#imap_ignore_command_idle = no

# Disable all plaintext authentications unless SSL/TLS is used
disable_plaintext_auth = yes
# File with RSA key for SSL
ssl_private_key_file = /etc/letsencrypt/live/domain.xx/privkey.pem
#ssl_private_key_file = /etc/kopano/gateway/privkey.pem

#File with certificate for SSL
ssl_certificate_file = /etc/letsencrypt/live/domain.xx/fullchain.pem
#ssl_certificate_file = /etc/kopano/gateway/cert.pem

# Verify client certificate
ssl_verify_client = no

# Client verify file and/or path
#ssl_verify_file =
#ssl_verify_path =

# SSL protocols to use, space-separated list of protocols
# (SSLv3 TLSv1 TLSv1.1 TLSv1.2); prefix with ! to lock out a protocol.
#ssl_protocols =

# SSL ciphers to use, set to 'ALL' for backward compatibility
ssl_ciphers = ALL:!LOW:!SSLv2:!EXP:!aNULL

# Prefer the server's order of SSL ciphers over client's
ssl_prefer_server_ciphers = no
# Processes are potentially safer from a security point of view.
process_model = fork

# For temporary files.
tmp_path = /tmp

# Whether Gateway should filter HTML messages or not. Usually, WebApp
# takes care of this. Letting the gateways do this improves the user latency a
# bit, but uses more disk space. (yes/no)
#html_safety_filter = no

##############################################################
# GATEWAY LOG SETTINGS

# Logging method (syslog, file)
log_method = file

# Loglevel (0(none), 1(crit), 2(err), 3(warn), 4(notice), 5(info), 6(debug))
log_level = 6

# Logfile for log_method = file, use '-' for stderr
# Default: -
log_file = /var/log/kopano/gateway.log

# Log timestamp - prefix each log line with timestamp in 'file' logging mode
log_timestamp = yes

# Buffer logging in what sized blocks. 0 for line-buffered (syslog-style).
#log_buffer_size = 0

# Bypass authentification when connecting as an administrator to the UNIX socket.
#bypass_auth = no

it works by enabling the obsolete option:

imaps_enable = yes

I know the search in this forum isn’t great, but this has actually already been discussed recently: https://forum.kopano.io/topic/1750/kopano-gateway-don-t-listen-on-993

@fbartels
I am very sorry, I really did try searching. But couldn’t find it. Thank you. That solved the first problem. I know appear to have another issue with ssl.

I now get the following error using my same -unchanged- SSL Certificate from letsencrypt:

Fri Sep 28 13:21:12 2018: [=======] Starting kopano-gateway version 8.6.81 (pid 4271 uid 0)
Fri Sep 28 13:21:12 2018: [info   ] Coredump status left at system default.
Fri Sep 28 13:21:12 2018: [error  ] ECChannel::HrSetCtx(): cannot open key file
Fri Sep 28 13:21:12 2018: [error  ] Error loading SSL context, POP3S and IMAPS will be disabled
Fri Sep 28 13:21:12 2018: [debug  ] Reexecing /usr/sbin/kopano-gateway
Fri Sep 28 13:21:12 2018: [=======] Starting kopano-gateway version 8.6.81 (pid 4271 uid 999)
Fri Sep 28 13:21:12 2018: [info   ] Coredump status left at system default.
Fri Sep 28 13:21:12 2018: [error  ] ECChannel::HrSetCtx(): cannot open key file
Fri Sep 28 13:21:12 2018: [error  ] Error loading SSL context, POP3S and IMAPS will be disabled
Fri Sep 28 13:21:12 2018: [info   ] Logger process started on pid 4274

I know that I had a similar issue with kopano core a few months ago and I had to merge

 cat cert.pem privkey.pem >> kopano.pem

but using kopano.pem only gives me:

Fri Sep 28 13:19:12 2018: [error  ] SSL CTX private key file error: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch

@fbartels Hi Felix, could it be that the kopano-gateway has trouble opening the certificates if they are symbolic links? I can use the ssl context without problems by specifying the file directly

I had a similar problem a few weeks ago, I was also using a symlink. But it might also have just been the permissions. Something changed in kopano and if i remember correctly I changed the certificate files so that the kopano user could read them instead of just root. It was a bit weird and my memory of what I had done to fix it is cloudy. But it was something about either the symlink or the permissions or both :-)

And it was just for my IMAP server! The other components still worked fine when I encountered this issue…

@Gerald exactly, it is only the gateway. But it is a bit weird because server needs the cert.pem and privkey.pem in one but works with symlinks an gateway accepts them separate but not symlinks or has trouble with the rights. Maybe @fbartels can help…

symbolic links work for me:

root@ucs-4570 /etc/kopano
$ netstat -tulpen | grep kopano
tcp        0      0 0.0.0.0:236             0.0.0.0:*               LISTEN      0          615833     13797/kopano-server
tcp        0      0 0.0.0.0:237             0.0.0.0:*               LISTEN      0          615836     13797/kopano-server
tcp6       0      0 :::110                  :::*                    LISTEN      0          620285     14564/kopano-gatewa
tcp6       0      0 :::143                  :::*                    LISTEN      0          620295     14564/kopano-gatewa
tcp6       0      0 :::8080                 :::*                    LISTEN      0          616810     13925/kopano-ical
tcp6       0      0 :::2003                 :::*                    LISTEN      0          617348     13978/kopano-dagent
tcp6       0      0 :::8443                 :::*                    LISTEN      0          616815     13925/kopano-ical
tcp6       0      0 :::993                  :::*                    LISTEN      0          620300     14564/kopano-gatewa
tcp6       0      0 :::995                  :::*                    LISTEN      0          620290     14564/kopano-gatewa

root@ucs-4570 /etc/kopano
$ grep ssl gateway.cfg
# Warning: the value "ssl_private_key_file" has been set via UCR variable "kopano/cfg/gateway/ssl_private_key_file"
ssl_private_key_file = /etc/kopano/ssl/link-private.key
# Warning: the value "ssl_certificate_file" has been set via UCR variable "kopano/cfg/gateway/ssl_certificate_file"
ssl_certificate_file = /etc/kopano/ssl/link-cert.pem
#ssl_verify_client = no
#ssl_verify_file =
#ssl_verify_path =
#ssl_protocols =
#ssl_ciphers = ALL:!LOW:!SSLv2:!EXP:!aNULL
#ssl_prefer_server_ciphers = no

root@ucs-4570 /etc/kopano
$ ls -la /etc/kopano/ssl
total 28
drwxr-xr-x 2 kopano kopano 4096 Oct  4 08:33 .
drwxr-xr-x 6 root   root   4096 Oct  4 08:33 ..
-rw-r----- 1 kopano kopano 5488 Oct  3 16:59 cert.pem
lrwxrwxrwx 1 root   root      8 Oct  4 08:32 link-cert.pem -> cert.pem
lrwxrwxrwx 1 root   root     11 Oct  4 08:33 link-private.key -> private.key
lrwxrwxrwx 1 root   root     10 Oct  4 08:33 link-server.pem -> server.pem
-rw------- 1 kopano kopano 1679 Oct  3 16:59 private.key
-rw------- 1 kopano kopano 7167 Oct  3 16:59 server.pem

root@ucs-4570 /etc/kopano
$ dpkg -l | grep kopano-gateway
ii  kopano-gateway                                      8.6.81.475-0+86.1                                amd64        POP3 and IMAP Gateway for Kopano Core

@fbartels that is very strange, it does not work for me. I am using the standard letsencrypt folder structure. If I copy the files manually and not use a symlink it works.

@yep_dd you are probably missing permissions for the user your kopano-gateway is running with to access all of the folders leading to the actual certificate (to resolve the symlink)

@fbartels true it was a permission problem of the letsencrypt folder, but I wonder why Kopano Server runs without issues as kopano users using the same certificate? That’s why I didn’t expect that and also why did it run with a previous kopano-core build (8.6.80.752) was the last one I used.

That cannot really be, since the requirement to resolve the ssl certificate is the same across all services and has been there since (but that is a guesstimate, did not check changelogs) 8.4.x.

We’ve just upgraded one of our servers to Kopano 8.6.x and had problems with this. Can I ask why the behaviour was changed? I consider this a bug, as to enable SSL/TLS in any Kopany services I now need to make the private key files readable by the unprivileged kopano user. Isn’t the purpose of using an unprivileged user to avoid private information getting out in the event of a security issue? All other daemon processes that I know of use privileged access to open the key files on startup, keeping them secure, before transferring process ownership over to the unprivileged user. I suspect that many Kopano users use the same certificate for SMTP and HTTPS as they do for Kopano, which makes Kopano the weakest link as far as security is concerned.

@kitserve said in error connecting to IMAPs via gateway (core 8.6.81.416):

Isn’t the purpose of using an unprivileged user to avoid private information getting out in the event of a security issue?

No. Never really was. Not in Kopano, not in operating systems, not even in the analog life. Whoever holds the money is in a position to spend it.

use privileged access to open the key files on startup, keeping them secure, before transferring process ownership over to the unprivileged user.

Red herring. The contents are already in memory (gotta do the encryption somehow) — it could leak from there. This is even the most probable attack vector, since kopano-server and postfix do not open arbitrary paths in direct consequence of a client asking for it.
httpd is a different matter, because GET /index.html translates outright to /srv/www/htdocs/index.html in the general case. Moreover, you could still exploit the memory of the httpd process to get at the key. With mod_php being installed almost everywhere, that is the first thing I’d try to break…

With respect, I must disagree. I take your point that the keys must live in memory, but attacking the memory isn’t necessarily that easy. (Kopano does use ASLR, right?). By contrast, attacking the filesystem is trivial. The Kopano user can read the Kopano config files, which are in a known location, and read the location of the certificates from them. Of course Kopano, Postfix, etc, aren’t supposed to access arbitrary files, but bugs do happen. With all due respect to the Kopano devs, who’ve achieved great things over the years, I trust Postfix’s security record a lot more than Kopano’s.

Most Kopano processes have to launch as root in order to open privileged ports (236/237, POP3/S, IMAP/S) before dropping privileges to the Kopano user. Why not load the keys into memory at the same time, the same as EVERY other program does? Apache, Dovecot, Postfix, ProFTPd, etc all take this approach. What is the compelling reason that Kopano should do things differently? Are all of these other programs doing it wrong?

Quite apart from the (arguable) security issues, this is making my life as a sysadmin more difficult. We use Let’s Encrypt certificates, which are only readable by root. Unless we use separate dedicated certificates for Kopano to the ones that we use for everything else, our certificates will only be readable by root. All other daemon processes that use them work in the way I’ve described above. What is the recommendation to get around this issue? The only obvious options seem to be to run Kopano as root (not a good idea), write some kind of cron job to periodically make the key readable for the Kopano user (hacky), or use a separate certificate for Kopano (extra admin work).

What I’m really not seeing here is a positive reason why Kopano’s behaviour has been changed. It seems like a change for change’s sake that hasn’t been demonstrated to improve Kopano for its users or administrators, although perhaps it makes life easier for the developers. I am open to hearing a clear rationale of why things have changed, and a sensible way of working round the change. In the absence of either of those, I suspect that I’m not the only one who would greatly prefer that this particular change be reverted.

Kopano does use ASLR, right?

The KC build uses the distro defaults in that regard.

attacking the memory isn’t necessarily that easy.

Nor is it with files in “[daemons that] aren’t supposed to access arbitrary files”: To change the buffer that eventually contains the in-filesystem name, you would need, you guessed it, a memory attack at just the right time.

Postfix’s security record a lot more than Kopano’s.

Well what can I say, KC undoubtedly descended from a codebase that had its head sooo deep in Windows-isms…

All other daemon processes that use them work in the way I’ve described above. [certs owned by root]

Certs owned by root also render certificate reloading useless, instead necessiting a restart of the entire service. postfix and httpd work around this by keeping one of their processes running as root. KC on the other hand completely switches away from root after the sockets are obtained. Those installations which are not listening on a privileged port number, or are using socket-based activation, can even start it with an unprivileged random UID from the get-go; despite me not changing it specifically for the purpose of Docker, I am being told it unfolded nicely for that case.

What is the recommendation to get around this issue?

setfacl -m u:kopano:r /etc/that/sslcert.key

reason why Kopano’s behaviour has been changed.

Since you don’t trust KC as much as postfix, doing less of the initialization (openssl, mysql, gsoap, setrlimit) as root should fit your case. ;-)

Thanks, very informative, you raise good points. Unfortunately we don’t have ACL set up so it looks as though I’m going to have to get a separate certificate for Kopano. I appreciate you taking the time to explain why things have changed, even if I’m still not sure I support it!

anyhow. …

I know that I had a similar issue with kopano core a few months ago and I had to merge

 cat cert.pem privkey.pem >> kopano.pem

should be at least.

cat privkey.pem cert.pem >> kopano.pem

Always first key then cert in combined certificate.

ssl_private_key_file = /etc/letsencrypt/live/domain.xx/privkey.pem
#ssl_private_key_file = /etc/kopano/gateway/privkey.pem

#File with certificate for SSL
ssl_certificate_file = /etc/letsencrypt/live/domain.xx/cert.pem

The CA is already integrated in /etc/ssl/cert/ca-certificates.

Just point the CA to that file or point to one in letsencrypt.