Kopano setup recommendation

mlued

Greetings dear Kopano community,

we want to change our mailserver from MS Exchange to Kopano.
But before we make the big move, my boss wants a test system with a small group of users (early adopters (20-30).
To be future proof, we wanted a high availability system. In the documentation
Kopano recommends DRDB and pacemake. We had lots of trouble with DRDB in the past, so we set up a mariadb galera cluster (3 VMs).
Kopano is running also on these servers (Debian 10 with the shipped 8.7.0 Kopano Version, I know not ideal). The corresponding IPs for mariadb and kopano are handled by keepalived
Yesterday I tried to import a few mailboxes (kopano-migration-imap). To see progress I logged in to my account and I saw mails popping up.
After a while (I tried this and that) I decided to clean the mailboxes and try the import again. I disconnected the store with

kopano-storeadm -D -n username

And wiped it from the database with

kopano-storeadm -R <GUID>

While the new imap import started, I got the following message:

Host2 failure: Error login on [localhost] with user [username] auth [LOGIN]: 2 BAD Internal error: OpenECSession failed

In the kopano-server log I got lots of duplicate errors:

KDatabase::I_Update() query failed: "Duplicate entry.....

I then tried to fix the database issues with:

kopano-dbadm k-1216

But the errors persisted. My ultima ratio was to overwrite the database with an older backup. After that everything went smooth.

My question is, is it possible that I will experience a problem like this again on an import like this?
I can not risk to loose a lot of mails, when I have to recover an older backup because of such database inconsitencies.
The second question is, is there any experience with this combination of galera and kopano here?
And before the question will be raised, as soon as the test will proof a good and stable mailserver, we will buy kopano licenses and support.

greetings from germany

Maik

fbartels

Hi @mlued,

@mlued said in Kopano setup recommendation:

While the new imap import started, I got the following message:

Could it be that these log messages are from a client that was still connected at the time the old store was removed and therefore is still trying to open the old store? Do these messages go away after stopping kopano-gateway and restarting it after all clients have run into their individual timeouts?

@mlued said in Kopano setup recommendation:

Kopano recommends DRDB and pacemake. We had lots of trouble with DRDB in the past, so we set up a mariadb galera cluster (3 VMs).

In the end there are many ways to reach high availability and the key points should be to have some form of replication both for stored files (which is where attachments and imap data are stored) as well as for the database (which is where the actual mailbox data minus the attachments are stored). If you have a form of replicated storage on your network then its fine to use this directly instead of working with replicated file systems. The usage of Galera is also fine, but comes with the downside that you need at least three systems and each of them needs to hold a replica of all databases. Here it would be easier to just use MariaDB/MySQL Master/Master replication directly.

@mlued said in Kopano setup recommendation:

And before the question will be raised, as soon as the test will proof a good and stable mailserver, we will buy kopano licenses and support.

Its worthwhile though to restart your tests with an evaluation key and the actual supported packages. The version provided by Debian is gravely out of date and uses a different packaging layout compared to our own packages. So it will not be a case of simply swapping packages when doing the step towards an officially supported installation.

mlued

@fbartels
A long time passed, but now I am testing a evaluation license from kopano.
I really had to update a few installation routines, but all in all it went ok.
I even got a solution to my topic. Als I wrote, I removed the store from the user and deleted the then orphaned store.
But I forgot to create a new store with

kopano-storeadm -Cn <user>

I have one thing, that concerns me the most in the migration process.
We have an Microsoft Exchange server, with > 400 user, but the migration is pretty straight forward. (as far as I could test it, with some mailaccounts)
The second server is a dovecot mailserver. I tried two ways to migrate. At first, I used the mbox2kopano python script. It took forever and it dumped all mails in the inbox (not in subfolders as it is on the old server).
The next thing is kopano-migration-imap. It works nice but the speed is abysmal. Currently I have a test running, witch has an ETA of today 7 pm. The throughput is around 2.7-3 messages/second. Is this normal? I don’t know, how I should explain this to the users, when we will migrate their mailboxes and they should not use it for a whole day.

Is there a better (faster) way for the migration?

By the way, I changed the database to a master/master configuration and followed the recommendations from the documentation.

fbartels

@mlued said in Kopano setup recommendation:

with > 400 user

For these amounts of users I would rather recommend to talk to our support directly. They can also check server performance for you.

mlued

@fbartels
The import from Exchange is not that fast, but my own 300 MB mailbox took less than 30 min.
My question is rather, can I speed up the

kopano-migration-imap

import?

fbartels

@mlued said in Kopano setup recommendation:

can I speed up the […] import?

Most probably. In the end this all comes down to Innodb write performance.

PS: kopano-migration-imap is in the end just a packaged up version of imapsync so all its documented strategies for partial/time based migration apply here as well.

mculibk

@mlued
Hi there!

Just wondering if you managed to “solve” the slow imap import problem?
I’m facing the same issue… I have some 10 mailboxes to migrate but they are large, approx 700GB in total…
After some initial kopano-migrate-imap testing I get 1.5 msgs/sec at best but the average speed is closer to 0.5 msgs/sec… which is… well, really bad… (tried locally migrate from one kopano account to the other and from old system to kopano)

On the old server (also local “migration”) the rate is in the range of 20 msgs/sec which is a big difference.

During migration the kopano-gateway hovers at around 40-60% CPU usage which seems a little high… but I don’t know “what’s normal”…

The server is a VM with 16GB RAM, SSDs, 4x vCPU vs the old one has only 8GB RAM and 2x vCPU and not even SSDs…

Any suggestion/hint for optimizing IMAP import speed would be greatly appreciated.

Best regards,
M:Culibrk

mlued

@mculibk
In my case, I ended up buying licenses and reinstalling it on a Debian 10.
I used a script for concurrent migration, which does not help with the slow rate, but it helped the overall migration time.

#!/bin/bash

while read line; do 
kopano-storeadm -Cn $line -l de_DE.utf8 &

kopano-migration-imap --noauthmd5  --host1 <host1> --user1 $line --password1 <password1> --host2 localhost --user2 $line --password2 <password2> --useheader Message-ID --addheader --usecache --errorsmax 1000 --skipemptyfolders &
done < user.csv

In the user.csv were only the usernames.
I ended up using 32 GB RAM, for the migration process, but I don´t know, if that helped.
I discovered a bottleneck in my mysql settings.

innodb_buffer_pool_size = 4096kb
innodb_log_file_size = 1024kb

4 MB to 8 GB was a huge difference for the buffer_pool_size. And also 1 MB for the log_file_size seems a little bit slow.

innodb_buffer_pool_size = 8Gb
innodb_log_file_size = 2Gb

@mculibk said in Kopano setup recommendation:

@mlued
Hi there!

Just wondering if you managed to “solve” the slow imap import problem?
I’m facing the same issue… I have some 10 mailboxes to migrate but they are large, approx 700GB in total…
After some initial kopano-migrate-imap testing I get 1.5 msgs/sec at best but the average speed is closer to 0.5 msgs/sec… which is… well, really bad… (tried locally migrate from one kopano account to the other and from old system to kopano)

On the old server (also local “migration”) the rate is in the range of 20 msgs/sec which is a big difference.

During migration the kopano-gateway hovers at around 40-60% CPU usage which seems a little high… but I don’t know “what’s normal”…

The server is a VM with 16GB RAM, SSDs, 4x vCPU vs the old one has only 8GB RAM and 2x vCPU and not even SSDs…

Any suggestion/hint for optimizing IMAP import speed would be greatly appreciated.

Best regards,
M:Culibrk

mculibk

@mlued
Hi there!
Thanks for the feedback!

I have the license… but suppose that’s no the issue, right?

Soo… basically, it’s “normal” that the import to Kopano-gateway (imap) is so slow?
What is the rate you got during import? In what range? (> 5/sec?)
The VM I’m currently pre-testing the migration is not a monster but anyway… I tried with more ram, more CPUs but that make little/no difference at all…
Also tweaked a little the mysql parameters (very similar to yours) but there was no “noticeable” difference in import performance…

I noticed the CPU is rather “loaded” by the gateway (and mysqld) process…

The “problem” for me is that I have a lot (~1.5TB) of mail to transfer and that’ll take ages to complete…
Number of mboxes is small so parallelism will be limited…

Thanks for the info anyway…
Regards

mlued

@mculibk
My sugesstion, migrate right away (now) let it take its time. Then, when you really want to switch to kopano do a second migration (it is a delta, so it checks if the mail is already there). It will need its time, bit it is a fraction, because the mails, which are already in your new system, will be skipped.

mculibk

@mlued
Yes… that is/was the idea anyway… but it’s a “week of syncing”…

And then there is the “issue” of removing all the IMAP data once the import is done… so I’ll need at least 3TB (I’m sure it will be less but…) to migrate the whole “double” pile…

Regards