z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F

vincentvana

Finally found a solution. After looking for hours at the increased log level suggested by jengelh and messing with a lot of settings without much luck (even having to restore the whole DB at some point). It looks like my error was related to a corruption in a mailbox. By lowering the SYNC_MAX_ITEMS, in the z-push config, all the way down to 5, z-push started syncing that account on the three new devices (already connected devices were working fine). Maybe running into a corruption stops syncing of the whole batch, and if that batch contains part of the hierarchy, it never manages to set it up.

If that is the case, it can be “fixed” by adding this situation to the loopdetection. It would make z-push more robust when dealing with these errors in a backend.

Anyway, to permanently fix this problem for me:

kopano-backup user
kopano-storeadm -D -n user
kopano-storeadm -C -n user
kopano-backup --restore user

This did take the better part of a day.

Makes me wonder though, what changed in the last half year or so that caused this corruption/errors, or made z-push/kopano more sensitive? Connecting new devices worked fine a few months ago.

P.S. btrfs filesystem snapshots are a life saver. :-)

rolek

@jengelh Jan, I tried increasing the MariaDB value for max_allowed_packet to 64MB, but that didn’t fix the issue. (This was a few days ago). So I don’t think that’s the cause.

jengelh

That sounds familiar. Synchronization streams will currently abort on error (e.g. attachment strangely missing from database).

hugo21212

I had the same error synchronizing a folder in my android phone that had many and messages with big attachments. I store all attachments in the database to make backup easier (just do a mysqldump).

I use Debian 10 and increasing max_allowed_packet for mariadb didn’t work.

I used @vincentvana suggestion (backup of user + deleting / recreating store + restore user) and it worked for me. At least for now and I didn’t dare to resync the big folder - I may try this on the weekend, when I have more time to fix problems.

fbartels

Unfortunately its still unclear how to reproduce the issue.

I have just closed two topics around the same error, the other topics were https://forum.kopano.io/topic/2783/statusexception-exportchangesics-initializeexporter-error-mapi_exportchanges_config-failed-0x8004010f-code-12/ and https://forum.kopano.io/topic/1678/statusexception-exportchangesics-initializeexporter-error-mapi_exportchanges_config-failed-0xffffffff8004010f

vincentvana

It started happening again. My newly imported mailbox is again causing issues. I’m not sure what the problem is because there are no errors I understand.
I have now switched away from using z-push for mail, and instead am using imap for mail sync. This works flawlessly.
Z-push is still used for calendar/contacts.

In response to fbartels: What I believe to be the problem, is troubleshooting. Z-push does not provide the level we apparently need, as does kopano-server. I would expect a log line like “sync of this person on this device aborted due to problematic message x”, or “message x in folder y corrupt/not found”. If you want to reproduce it, just create a big mailbox folder, remove an attachment, and sync. But really, a “MAPI not found” should not abort the sync. It looks like that is the problem/bug.

jengelh

How did you remove that attachment?

Manfred

@vincentvana bumping jengelh’s question. How did you remove that attachment?

weini

I´m also still facing the same issue. I´m facing the issue for mail folders with many (starts with a few 1.000 up to 15.000) entries.

It might be interesting, that the issue only happens when I sync to my mobile (Samsung S8). Sync is limited to last 30 days for emails there. I´m syncing the same inbox via EAS also to Outlook without any time restriction, but that never had an issue.

What I did in the meantime, and what DIDN´T solve the issue:

backup user store with kopano-backup and restore it
play around with MariaDB parameters, esp. max_allowed_packet
rebuild tproperties completely with the script provided in the kopano wiki
use kopano-dbadm to (potentially) cleanup db schema

Only sucessful workaround was to move from MariaDB 10.3 to MySQL. The issue was gone there immediately. I used the Docker image to do this. However, having MySQL 8.0 and MariaDB 10.3 running at the same time (as I need MariaDB for some other stuff) is causing a high load on my server, so I want to avoid this scenario in the mid term.

Hint: If you want to go for a similar approach, do yourself a favor and don´t use MySQL 8.0, but version 5.7 instead. You can move tables / schemas between MariaDB 10.3 (and also 10.4) and MySQL 5.7 very easily via InnoDB tablespace import/export. However, if you use MySQL 8.0, this is a one way trip. Going back from MySQL 8.0 to MariaDB requires a full logical export / import which is quite time consuming.

weini

@rolek said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

I can report that the patch from https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=939751;filename=fix-zpush-sync-error.patch;msg=15 fixes the problem for us.

@fbartels: Did you integrate that patch into your codebase?

robgnu

I was also facing the same issue. I got a big log flooding with the same message. z-push.log was about 10 GBytes after a day.
I solved the issue, but I am not sure, what the real problem was. So I hope, these information will help anyone:

All users are connected to multiple Outlooks 365 via ActiveSync (no KOE)
Some of the users are connected via an Android Phone (Samsung)
The users have many (>4000) calendar entries.
Sync of calendar items between Outlook seems to work, but didn’t sync to the phones. Deleting and recreation of accounts on the phone didn’t solve the issue.
kopano-backup (backup + restore) did NOT solve the issues.
Changing mariadb parameters did NOT solve the issues.
Switching away from MariaDB to Mysql is currently not an option.

There were so many log entrys, that I decided to remove the affected usersaccounts in Outlook and recreated them. After resync the log flood was gone.

Currently working on Debian 10 (Kopano 8.7.85.0.a02e61d66-0+176.1 with z-push 2.5.1+0-0)

bye
Robert.

weini

@robgnu
What exactly do you mean by “remove affected useraccounts in Outlook”?
I delete the account in Outlook and removed it then in z-push. Afterwards, I connected Outlook again.
Unfortunately, this didn´t cure the issue on my side. Did you do any service restarts in between?

fbartels

@weini said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

@fbartels: Did you integrate that patch into your codebase?

No, since there was never an answer to https://forum.kopano.io/post/15294

weini

@weini said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

Unfortunately, this didn´t cure the issue on my side. Did you do any service restarts in between?

It seems I was a bit too fast: about 1h after I deleted the Outlook account (and recreated it), the errors are gone and my mobile is syncing the folders that it was formerly refusing to do so.

Fingers crossed…

robgnu

@weini
I removed the Outlook account and recreated the account.

Today I faced some issues again. Now I’m trying to backup/restore the accounts with kopano-backup into a new store. After that I request a resync via z-push-admin to the android devices.

I had luck with one account today. I’m running currently a batch job to recreate a bunch of users. I can tell more tomorrow…

There must be some deeper problems…

Robert.

robgnu

A small update:
After backup users and restoring to a new store the log flood is back:

There are more than 10 entries per second in the logfile (z-push-erro.log):

11/12/2019 06:17:56 [ 1748] [ERROR] [user] StatusException: ExportChangesICS->Config(): Error, exporter not available - code: 12 - file: /usr/share/z-push/backend/kopano/exporter.php:126

The same - more than 10 entries per second in z-push.log:

11/12/2019 06:22:15 [ 9049] [ INFO] [user] cmd='Sync' memory='2.32 MiB/2.00 MiB' time='0.02s' devType='WindowsOutlook' devId='4beabac1b19844dd946eefb2afec7a6c' getUser='user' from='2003:xxxx' idle='0s' version='2.5.1+0-0' method='POST' httpcode='200'

Sometimes this messages are logged multiple times:

11/12/2019 06:37:07 [26162] [ INFO] [user] NoHierarchyCacheAvailableException: Folderid 'Ub8896' is not fully synchronized on the device - code: 0 - file: /usr/share/z-push/lib/core/devicemanager.php:466
11/12/2019 06:37:07 [26162] [ INFO] [user] HandlePing(): unknown collection 'Ub8896', triggering HierarchySync
11/12/2019 06:37:07 [26162] [ INFO] [user] NoHierarchyCacheAvailableException: Folderid 'U70eec' is not fully synchronized on the device - code: 0 - file: /usr/share/z-push/lib/core/devicemanager.php:466
11/12/2019 06:40:11 [31742] [WARN] [user] Not possible to determine class of request. Request did not contain class and apparently there is an issue with the HierarchyCache.

Today I will “go around” and reconfigure the Outlook accounts.

Bye
Robert

fbartels

Hi @robgnu,

in another post you said you we’re using 8.7.7 which would be one of our supported builds. If you have a subscription please reach out to our support so we can have a look at your system.

robgnu

@fbartels
That’s correct. Some of my clients have 8.7.7 with a subscription. Some smaller clients are still using the community builds (installations since 2015 - Zarafa). Switching them to an official build/subscription is in discussion and on it’s way.
All 8.7.7. installations are still on Debian 9, because there are no official Debian 10 builds. :-(

The problem hits all installations with the community build after upgrading Debian 9 to 10 and 8.7.80.307 (stretch) to 8.7.85.0 (buster).

I can give another short update:
After restoring all users to a new store, the Android clients are fine. But some Outlook 365 did a big logflood (see my post above). Today I walked to each computer and recreated all active sync accounts. Currently everything works as before.

Bye
Robert

weini

Brief update from my side as well:

Issue came back in the mobile sync. I dropped the Outlook profile once again and recreated it. Reason is, that Outlook resync had some issues yesterday, so I wanted to give it a 2nd try. Currently, mobile sync is working fine angain and Outlook resync has completed.

From my perspective, the “pattern” looks as follows:

MariaDB as database
EAS profiles for Outlook and also for a mobile device
(possibly) “old” installations with many migration steps in between

weini

This time it worked for 2 days. Now folders do no longer sync on the mobile and the error message is back…
So not a sustainable workaround, at least not for me!