working with huge mailboxes is close to impossible

daudo

For our customer, the scenario looks like this:

They have 4 people working at their front desk, all of them have access to the office@… account plus their individual, personal accounts.

The office@… account is quite huge (about 20GB in mail) .

Setting up Outlook + KOE on those workstations is almost impossible, because the initial synchronization takes hours (vs. seconds when using the old MAPI client). Additionally, the server load is huge during this initial synchronization (vs. minimal with the old MAPI client).

Occasionally one of their backoffice workers also connects to the office@… account and and adds it via the “shared folders” feature.

This again takes hours to synchronize and produces huge server load (vs. none visible with the old MAP client).

Due to tight integration with third party software, working with the DeskApp/WebApp is no option.

Any ideas how to make this scale?

Udo

externa1

@daudo
why not adding the office account as IMAP Mailbox in outlook where the personal accounts can still stay with koe ?

rg
Christian

fbartels

Hi @daudo ,

@daudo said in working with huge mailboxes is close to impossible:

The office@… account is quite huge (about 20GB in mail) .

large accounts should generally be no problem. Sure the initial sync may take a while, but the again you hopefully only have to do it once.

@daudo said in working with huge mailboxes is close to impossible:

the initial synchronization takes hours (vs. seconds when using the old MAPI client).

You seem to confuse KOE with the old Zarafa mapi client. While the old zarafa client had an online mode that basically worked as a terminal for whatever was stored on the server, KOE/activesync requires all data to be downloaded to the local client first. In fact Zarafa also had a cached/offline mode and while I never measured it, the sync process with koe feels faster.

@daudo said in working with huge mailboxes is close to impossible:

Additionally, the server load is huge during this initial synchronization (vs. minimal with the old MAPI client).

It seems your should revisit the tuning parameters of kopano/mysql/your webserver. https://kb.kopano.io/display/WIKI/Tuning provides a good start, some more in depth guides can be obtained from our support.

Gerald

@fbartels said in working with huge mailboxes is close to impossible:

[…] the old zarafa client had an online mode that basically worked as a terminal for whatever was stored on the server, KOE/activesync requires all data to be downloaded to the local client first.

This was absolutely great about the old client. It was possible to have a 100GB shared Mailbox for all users that was easily accessible within the office (remote access was not so great) without the need for every single client machine to have a 100GB local copy of the shared mails. Now with Kopano/Activesync every single client PC will store a local copy of those 100GB, which will then also increase the backup storage needed for each of those clients from 30GB (normal Windows Installation) to 130GB (Win10+Mails).

And now if there are synchronsation problems you have to trigger a full resync of the 100GB. Not looking forward to doing this :-)
Summary: everything was better in the old days :cry:
(ok, not everything. Mobile use in Outlook with activesync is soooo much better than the old zarafa cached mode)

daudo

@fbartels said in working with huge mailboxes is close to impossible:

Hi @daudo ,

@daudo said in working with huge mailboxes is close to impossible:

The office@… account is quite huge (about 20GB in mail) .

large accounts should generally be no problem. Sure the initial sync may take a while, but the again you hopefully only have to do it once.

@daudo said in working with huge mailboxes is close to impossible:

the initial synchronization takes hours (vs. seconds when using the old MAPI client).

You seem to confuse KOE with the old Zarafa mapi client. While the old zarafa client had an online mode that basically worked as a terminal for whatever was stored on the server, KOE/activesync requires all data to be downloaded to the local client first. In fact Zarafa also had a cached/offline mode and while I never measured it, the sync process with koe feels faster.

@daudo said in working with huge mailboxes is close to impossible:

Additionally, the server load is huge during this initial synchronization (vs. minimal with the old MAPI client).

It seems your should revisit the tuning parameters of kopano/mysql/your webserver. https://kb.kopano.io/display/WIKI/Tuning provides a good start, some more in depth guides can be obtained from our support.

Thanks for your input. I’m very aware that the KOE and the old MAPI client are an entirely different approach to the same usecase.

Yes, database tuning may help a little, but I doubt that the time needed to synchronize will get significantly down from the current hours to a more decent value (like minutes).

Part of our problem is probably that we set up the front desks simultaneously, causing the same server load 4 times.

From a technical POV I understand why that is happening, from an user’s perspective this is more difficult to understand (not blaming Zarafa/Kopano for that, no worries). And user experience is pivot as we all know.

So far, the best idea to at least remove parts of the synchronization overhead seems to be to use IMAP instead, which again comes with the disadvantage that it lacks calendaring and tasks. So this might be usable for the back office people, but not for the front desks, they must have access to the calendar & tasks.

Thanks again

Udo

Sebastian

Would a smaller sync window for the emails be an option? E.g. synchronizing only 2 month of emails instead of the full 20 GB of data?

daudo

@sebastian

yes, that would be way better, but if I see it correctly, the only option is “1 month” vs. “all” in the ActiveSync account settings.

And 1 month is definitively not enough.

mcostan

Perhaps if that’s not available to choose from in Outlook, it could be a setting somewhere on the backend in the config in z-push where the users can force a particular sync time? I.e. in Outlook the user selects “Everything” but on the back end if it limited say for a specific user and a particular client, to a specific interval?

externa1

@daudo said in working with huge mailboxes is close to impossible:

So far, the best idea to at least remove parts of the synchronization overhead seems to be to use IMAP instead, which again comes with the disadvantage that it lacks calendaring and tasks. So this might be usable for the back office people, but not for the front desks, they must have access to the calendar & tasks.

Hi Udo,

Yes but the front desk also may be able to use imap for the shared mails, and can use the calendar and tasks over KOE Active Sync

rg
Christian

Gerald

Yes but the front desk also may be able to use imap for the shared mails, and can use the calendar and tasks over KOE Active Sync

AFAIK Outlook 2016 can either sync all of Contacts/Calendar/Mail or nothing. How do you use ActiveSync in Outlook 2016 to sync only the Calendar as you have suggested?

PS: Also, IMAP in Outlook 2016 will keep a local copy of the entire mail database. So if you have 100GB Mails in the shared account every client will need the necessary free disk space for that.

externa1

@gerald

Hi, with Kopano KOE (Kopano Outloook Extension) you can select only Calendar from a shared mailbox

0_1503404644905_5fc62653-683f-4621-97f9-ba1968d0a3bd-grafik.png

Also on IMAP account you can select 1,3,6,12 or 24 month of offline mail download or all

0_1503404600793_20ccba3a-3e23-4c84-812d-d9a0e8934638-grafik.png

rg
Christian

isol

Well the limitation that you need the disk space applies to ActiveSync and IMAP likewise (although with the latter you can limit the sync amount in detail, which we have done otherwise outlook spends too much time syncing IMAP accounts with many subfolders). The fact that all mails are stored locally means you need to treat large mailboxes carefully and plan migration of such accounts with disk space, performance, and sync time in mind.

After migrating a customer with many large mailboxes, I strongly suggest that if you want to use Kopano with Outlook and ActiveSync, then limit your Mailboxes to around 10GB and move the rest to an archival solution. In that way you can keep mailbox handling manageable also for the future - if the mailbox keeps growing, resyncing after e.g. reinstalling the client computer will take longer and longer.

Coffee_is_life

Hello all,

in my config.php from z-push there are some lines:

  154      // Global limitation of items to be synchronized
   155      // The mobile can define a sync back period for calendar and email items
   156      // For large stores with many items the time period could be limited to a max value
   157      // If the mobile transmits a wider time period, the defined max value is used
   158      // Applicable values:
   159      //   SYNC_FILTERTYPE_ALL (default, no limitation)
   160      //   SYNC_FILTERTYPE_1DAY, SYNC_FILTERTYPE_3DAYS, SYNC_FILTERTYPE_1WEEK, SYNC_FILTERTYPE_2WEEKS,
   161      //   SYNC_FILTERTYPE_1MONTH, SYNC_FILTERTYPE_3MONTHS, SYNC_FILTERTYPE_6MONTHS
   162      define('SYNC_FILTERTIME_MAX', SYNC_FILTERTYPE_6MONTHS);

location on centos: /usr/share/z-push/config.php

you can define the sync-filter serverside.

Coffee_is_life

mcostan

Thanks, I must have missed this. Is there any way to configure it by username? I.e. you could have the case where a big mailbox is only for a particular user and the others aren’t therefore they want the entire history.

Sebastian

@coffee_is_life said in working with huge mailboxes is close to impossible:

location on centos: /usr/share/z-push/config.php

If you install from packages you should always edit /etc/z-push/z-push.conf.php
(at the end it’s the same file, but still).

Cheers,
Sebastian