z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F

fbartels

@weini said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

@fbartels: Did you integrate that patch into your codebase?

No, since there was never an answer to https://forum.kopano.io/post/15294

weini

@weini said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

Unfortunately, this didn´t cure the issue on my side. Did you do any service restarts in between?

It seems I was a bit too fast: about 1h after I deleted the Outlook account (and recreated it), the errors are gone and my mobile is syncing the folders that it was formerly refusing to do so.

Fingers crossed…

robgnu

@weini
I removed the Outlook account and recreated the account.

Today I faced some issues again. Now I’m trying to backup/restore the accounts with kopano-backup into a new store. After that I request a resync via z-push-admin to the android devices.

I had luck with one account today. I’m running currently a batch job to recreate a bunch of users. I can tell more tomorrow…

There must be some deeper problems…

Robert.

robgnu

A small update:
After backup users and restoring to a new store the log flood is back:

There are more than 10 entries per second in the logfile (z-push-erro.log):

11/12/2019 06:17:56 [ 1748] [ERROR] [user] StatusException: ExportChangesICS->Config(): Error, exporter not available - code: 12 - file: /usr/share/z-push/backend/kopano/exporter.php:126

The same - more than 10 entries per second in z-push.log:

11/12/2019 06:22:15 [ 9049] [ INFO] [user] cmd='Sync' memory='2.32 MiB/2.00 MiB' time='0.02s' devType='WindowsOutlook' devId='4beabac1b19844dd946eefb2afec7a6c' getUser='user' from='2003:xxxx' idle='0s' version='2.5.1+0-0' method='POST' httpcode='200'

Sometimes this messages are logged multiple times:

11/12/2019 06:37:07 [26162] [ INFO] [user] NoHierarchyCacheAvailableException: Folderid 'Ub8896' is not fully synchronized on the device - code: 0 - file: /usr/share/z-push/lib/core/devicemanager.php:466
11/12/2019 06:37:07 [26162] [ INFO] [user] HandlePing(): unknown collection 'Ub8896', triggering HierarchySync
11/12/2019 06:37:07 [26162] [ INFO] [user] NoHierarchyCacheAvailableException: Folderid 'U70eec' is not fully synchronized on the device - code: 0 - file: /usr/share/z-push/lib/core/devicemanager.php:466
11/12/2019 06:40:11 [31742] [WARN] [user] Not possible to determine class of request. Request did not contain class and apparently there is an issue with the HierarchyCache.

Today I will “go around” and reconfigure the Outlook accounts.

Bye
Robert

fbartels

Hi @robgnu,

in another post you said you we’re using 8.7.7 which would be one of our supported builds. If you have a subscription please reach out to our support so we can have a look at your system.

robgnu

@fbartels
That’s correct. Some of my clients have 8.7.7 with a subscription. Some smaller clients are still using the community builds (installations since 2015 - Zarafa). Switching them to an official build/subscription is in discussion and on it’s way.
All 8.7.7. installations are still on Debian 9, because there are no official Debian 10 builds. :-(

The problem hits all installations with the community build after upgrading Debian 9 to 10 and 8.7.80.307 (stretch) to 8.7.85.0 (buster).

I can give another short update:
After restoring all users to a new store, the Android clients are fine. But some Outlook 365 did a big logflood (see my post above). Today I walked to each computer and recreated all active sync accounts. Currently everything works as before.

Bye
Robert

weini

Brief update from my side as well:

Issue came back in the mobile sync. I dropped the Outlook profile once again and recreated it. Reason is, that Outlook resync had some issues yesterday, so I wanted to give it a 2nd try. Currently, mobile sync is working fine angain and Outlook resync has completed.

From my perspective, the “pattern” looks as follows:

MariaDB as database
EAS profiles for Outlook and also for a mobile device
(possibly) “old” installations with many migration steps in between

weini

This time it worked for 2 days. Now folders do no longer sync on the mobile and the error message is back…
So not a sustainable workaround, at least not for me!

jengelh

The environment variable can be set to KOPANO_CLIENT_LOGLEVEL=0x800006 to get sync messages. That is something introduced in 3ac65333, so for those who feel lucky and in a dev mood…

hrobbie1981

Not sure if it helps, but I do only see this issue as long as my mobile device is set to sync “less than no Limit” days. Specifically on the iPhone in accounts if I set “Mail Days to Sync” to “No Limit” the error does not show up anymore…

Not really what I want, though, as my mailbox is huge…

robgnu

@hrobbie1981
I can not confirm. I see the issues on users where Android Phones (Samsung) and Outlook are connected. The sync limit is set to “All / No Limit”.
Users who using an iPhone doesn’t have any problems as much as I know.

Bye
Robert.

glotzkowsky

Hi. I got the same issue with an android mobile today after upgrading the SQL server VM from Debian 8 to Debian 10. My setup is:
Zarafa 7.2.6-10 (still on Debian 7)
Z-Push 2.2.8
DB: mariadb 10.3.18

For me setting “max_allowed_packet” in my.conf in the [mysqld] Section, resolved the issue immediately.

weini

@glotzkowsky said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

For me setting “max_allowed_packet” in my.conf in the [mysqld] Section, resolved the issue immediately.

I´d be more than happy, if this would be a sustainable solution…
Unfortunately, others already reported back in this thread, that increasing max_allowed_packet did not cure the issue for them.

To which value did you set “max_allowed_packet”?
Could you please keep us updated, if the issue is gone for you on a permanent base?

glotzkowsky

I set it to 64M, but unfortunately it does not solve the problems for all users. There is one user with > 30000 mails in his inbox. For this user the problem still exists.
Before the update, it was a Mysql 5.5 server with max_allowed_packet set to default (16M).

thctlo

@glotzkowsky 30k emails in the inbox. thats crazy.
It slows down everything…

read : https://support.microsoft.com/en-us/help/2768656/outlook-performance-issues-when-there-are-too-many-items-or-folders-in

MS advices max 5k per folder, so make that guy clean up its mail really, its that simple.

PRO123

@thctlo said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

MS advices max 5k per folder, so make that guy clean up its mail really, its that simple.

Sry, but this is not correct.

The Microsoft recommendation from your URL mentions “5,000 items per Calendar folder” but in general “100,000 items per folder” and “500 folders total”.

30k mails in the inbox should at least for outlook generally not be too big a problem, if one also thinks about archive solutions as for example Kopano Archiver.

PS: Is there actually already progress with the new Kopano Archiver (2.0) announced about 2 years ago? Auditing acceptability, especially in the areas of completeness and protection from change and tamper (denial of deletion and modification).

thctlo

Ah, sorry, yes, but on older clients it really helps, i suggest you try it.

My boss had also over 30k items, i/we cleaned it up, added most in subfolders and gone problems.
Costs nothing, worth a try.

Pozzo-Balbi

z-push version 2.5.1+0-118.1
kopano-server 8.7.85-2.4

I ran into the same problem after restoring emails into Inbox. Only after reducing the amount of emails to around 2600 in Inbox, did my iphone sync again. I just moved the excessive emails into sub-folders. Standard sync is set to 1 day. The other option to sync all emails was not viable.

weini

After I setup my kopano server completely new from the scratch, I looks like the issue is gone from me. I backuped my users as PSTs, dropped the database an attachment store and create a new one (via kopano-dbadm, as the automatic creation from the server failed). After importing the PSTs (and recreating all rules manually, as they didn´t come back with the import), the system is running smoothly.
I have to admit that my installation was very old, I always updated since version 7.0.x. When I realized, that I´ve some metadata corruption that prevented me from dropping ophaned stores (https://forum.kopano.io/topic/2888/can-t-delete-orphaned-store/6), I decided to do a new installation.

That was 2 weeks ago and so far, the system is running properly.

vincentvana

So, I’m back again. Sorry for the late response. To respond to the question about the attachment: I did not remove it myself. The backup and restore put it back in order enough for me.

Anyway, the issue resurfaced for me too. It started happening on a tablet that I only sync manually, and hadn’t done so in a few months. Tried to install mysql from the official repo, but ran into incompatibilities with the new mysql 8 password encoding. Long story short, don’t try this at home!

What did help for me was a …drumroll please: OPTIMIZE TABLE of the properties and tproperties tables (takes hours even for a relatively small db, don’t do this on a weekday).

So, I think I narrowed the problem down to performance. It is some sort of timeout between the components. The response from X simply takes too long.

That would explain the backup/restore fix (fresh import would be clean and all data would be together) and the resurfacing after a while.
It would explain the fact that some existing devices still sync. They don’t have too much to sync, while new devices won’t.
It would explain syncing failing on big mailboxes/folders only.

The problem is that I still don’t know where to look exactly. It is a long chain of client app -> network -> webserver(for some of your with loadbalancer) -> php engine -> z-push-server -> network -> kopano server -> network -> database. The timeout could be in any one of these parts, I would not know.

But for me, for now, stuff works as expected as long as I do a weekly OPTIMIZE TABLE during a weekend night. Be careful, I did not pay attention to table locking or storage size. Please check before you attempt this.
Oh, also tried playing with the fsync option of innodb but did not find a significant improvement there, but it can be a big risk for your data. AFAICS don’t waste your time there.