z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F

weini

No big difference for z-push, but kopano packages were worse:

kopano-spamd failed with errors on checking most of the new emails, mostly needed a restart
kopano-server dumped with backtrace sporadically approx. once a day

Lucky outcome for me is now, that the “orig” packages seem to work again. I´m now on:

kopano-server nightly build 8.7.82.12.6297bf8c5-0+294.1
z-push 2.5 from kopano repository (Debian_9.0)
Debian 10
MariaDB from Debian 10 repo

Did a complete resync dropping all state files. Error messages did occur while syncing, but sync completed and I´m now able to access all my folders.

nib

Very same problem here. (including an unsatisfactory workaround)

I run a combination of relatively new (but release versions) of

Kopano Core (8.7.x)
z-push (2.5.0-0) and
MariaDB10 (10.3.11).

While IMAP (via gateway) and z-push of Calendar & Contacts worked fine, Email Sync (via z-push) stopped working for an iOS device apparently due to the first folder (INBOX) growing to more than ~3000 objects.
Reason: When reducing number of emails in INBOX, sync immediately started working again.

All other (web-search-induced) hotfixes failed to show the desired effect.
Some of the attempted (and failed) fixes:

z-push-admin -a resync -t hierarchy -u …
complete removal of device in question, removal of account, recreate account
Reducing the email sync interval (from 1 month to 1 week to 3 days)
Changing in server.cfg enable_enhanced_ics = yes to no

The symptom(s) did not exactly coincide with a package upgrade, however, at the time of package upgrade (together with a certificate renewal), the lack of the most recent emails was noticed. Other email clients accessing the same Kopano-mailbox didn’t stop syncing/downloading all email.
Also, during all time, the sync of Calendar & Contacts through the same z-push connection were completely ok, both directions. A second account (for a second calendar only) also synced fine.

During the search for solutions, I also encountered posts, where only calendar sync was broken, so there is reason to believe that a more generic part of the sync code causes this issue as soon as too many objects (somewhere between 1000 - 3000) are bunched into one sync operation.
Other posts (I haven’t tried this solution) have been reporting that change of MariaDB10 to MySQL 8.x has (at least temporarily) resolved the issue.
First such posts (using the exact error code started appearing in 2018.

tl;dr:
Syncing of iOS devices (calendar, email, possibly contacts as well) through z-push/active sync with a MariaDB10 supported KopanoCore (>=8.6.?) fails to transfer/sync new objects to an activeSync Client (definitely iOS, maybe others) as soon as some threshold of objects to be synced is surpassed.

Solution:
Email: Moving emails away from a folder that has grown too big seems to work.
Calendar: Changing in server.cfg enable_enhanced_ics = yes to no may provide a temporary workaround.

Nib.

fbartels

Hi @nib,

@nib said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

growing to more than ~3000 objects.

ah, this could finally be the missing piece to also reproduce this. Will try to have a look next week.

If you want to speed this up @nib, from your posted versions it seems you have a valid subscription? If so then opening a support ticket and have our support reproduce this could be a bit faster.

weini

Brief update from my side: The “fix” was unfortunately not a permanent one. Sync breaks again after some time and the content of larger folders (email and also calendar) simply disapperas on the mobile. The error message is still present in the logs.

weini

@nib: Did you get any update via the support? Is there a fix on the way?

JPW_Kopano

Same problem for me.
Configuration:
Kopano & MariaDB from Debian 10 Repo
z-Push 2.3.7

Small mailboxes are synced. Big mailboxes are not.

nib

Apologies for the late reply, but sadly I don’t have a subscription. This is all based on the community version. I only run this server for my family privately on a VM on my NAS. I do have the luxury of access to several client systems which happened to be available for small tests.
I also didn’t try the change from MariaDB to MySQL as such a change on my “production” system would take quite some time for backups, which I didn’t want to invest after finding a “good-enough-for-the-moment” fix.

Nib

Beleggrodion

I have the same issue on one of our setups.

Kopano Core (8.7.5)
z-push (2.5.0+0-0)
MariaDB (10.4.6)

Currently their is only one user affected with this issue:

Folder: Calendar             Sync: Initialized   Status:  0% (0/10236)

The following setting had no effect:

enable_enhanced_ics = no

I also enabled the debugging for this device.

13/08/2019 13:36:12 [ 9528] [DEBUG] -------- Start
13/08/2019 13:36:12 [ 9528] [DEBUG] cmd='Sync' devType='Android' devId='android1263512214918' getUser='max.mustermann' from='123.123.123.123' version='2.5.0+0-0' method='POST'
13/08/2019 13:36:12 [ 9528] [DEBUG] Used timezone 'Europe/Zurich'
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::GetBackend(): trying autoload backend 'BackendKopano'
13/08/2019 13:36:12 [ 9528] [DEBUG] BackendKopano using PHP-MAPI version: 8.7.5 - PHP version: 7.2.19-0ubuntu0.18.04.1
13/08/2019 13:36:12 [ 9528] [DEBUG] Request::ProcessHeaders() ASVersion: 14.1
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::CommandNeedsProvisioning(0): true
13/08/2019 13:36:12 [ 9528] [DEBUG] FileStateMachine->GetState() read '4411' bytes from file: '/var/lib/z-push/9/3/android1263512214918-devicedata'
13/08/2019 13:36:12 [ 9528] [DEBUG] ASDevice data loaded for user: 'max.mustermann'
13/08/2019 13:36:12 [ 9528] [DEBUG] TopCollector(): Initialized mutexid Resource id #9 and memid Resource id #10.
13/08/2019 13:36:12 [ 9528] [DEBUG] TopCollector initialised with IPC provider 'IpcSharedMemoryProvider' with type '20'
13/08/2019 13:36:12 [ 9528] [DEBUG] LoopDetection(): Initialized mutexid Resource id #11 and memid Resource id #12.
13/08/2019 13:36:12 [ 9528] [DEBUG] LoopDetection initialised with IPC provider 'IpcSharedMemoryProvider' with type '1337'
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::HierarchyCommand(0): false
13/08/2019 13:36:12 [ 9528] [DEBUG] DeviceManager->ProvisioningRequired('538798535') saved device key '538798535': false
13/08/2019 13:36:12 [ 9528] [DEBUG] DeviceManager->getPolicyName(): determined policy name: 'default'
13/08/2019 13:36:12 [ 9528] [DEBUG] DeviceManager->getProvisioningPolicies(): loaded 'default' policy.
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::CommandNeedsAuthentication(0): true
13/08/2019 13:36:12 [ 9528] [DEBUG] KopanoBackend->Logon(): Trying to authenticate user 'max.mustermann'..
13/08/2019 13:36:12 [ 9528] [DEBUG] KopanoBackend->openMessageStore('max.mustermann'): Found 'DEFAULT' store: 'Resource id #15'
13/08/2019 13:36:12 [ 9528] [DEBUG] KopanoBackend->Logon(): User 'max.mustermann' is authenticated
13/08/2019 13:36:12 [ 9528] [DEBUG] Store supports properties containing Unicode characters.
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::CommandNeedsPlainInput(0): false
13/08/2019 13:36:12 [ 9528] [WBXML] I  <Synchronize>
13/08/2019 13:36:12 [ 9528] [WBXML] I   <Folders>
13/08/2019 13:36:12 [ 9528] [WBXML] I    <Folder>
13/08/2019 13:36:12 [ 9528] [WBXML] I     <SyncKey>
13/08/2019 13:36:12 [ 9528] [WBXML] I      0
13/08/2019 13:36:12 [ 9528] [WBXML] I     </SyncKey>
13/08/2019 13:36:12 [ 9528] [WBXML] I     <FolderId>
13/08/2019 13:36:12 [ 9528] [WBXML] I      U5e894
13/08/2019 13:36:12 [ 9528] [WBXML] I     </FolderId>
13/08/2019 13:36:12 [ 9528] [DEBUG] SyncParameters->RemoveSyncKey(): saved sync key removed
13/08/2019 13:36:12 [ 9528] [DEBUG] DeviceManager->GetBackendIdForFolderId(): folderid U5e894 => 0879775ab9d94a42901de37ae24a9a9ca40100000000
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::GetFolderClassFromFolderType('8'): Calendar
13/08/2019 13:36:12 [ 9528] [DEBUG] GetFolderClassFromCacheByID from Device Manager: 'Calendar' for id:'U5e894'
13/08/2019 13:36:12 [ 9528] [DEBUG] SyncCollections->AddCollection(): Folder id 'U5e894' : ref. PolicyKey '1', ref. Lifetime '10', last sync at ''
13/08/2019 13:36:12 [ 9528] [WBXML] I    </Folder>
13/08/2019 13:36:12 [ 9528] [DEBUG] HandleSync(): FilterType applied globally or specifically, using value: 7
13/08/2019 13:36:12 [ 9528] [DEBUG] HandleSync(): FilterType has changed (old: '', new: '7'), removing folderstat to force Exporter setup
13/08/2019 13:36:12 [ 9528] [WBXML] I   </Folders>
13/08/2019 13:36:12 [ 9528] [WBXML] I  </Synchronize>
13/08/2019 13:36:12 [ 9528] [DEBUG] HandleSync(): Start Output
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::GetAdditionalSyncFolderStore('0879775ab9d94a42901de37ae24a9a9ca40100000000'): 'false'
13/08/2019 13:36:12 [ 9528] [DEBUG] KopanoBackend->GetFolderStat() fetched status information of 28 folders for store 'max.mustermann'
13/08/2019 13:36:12 [ 9528] [DEBUG] SyncParameters->IsExporterRunRequired(): true - current: 1565693864/10236/0/13 - saved: false - expiring: not set
13/08/2019 13:36:12 [ 9528] [DEBUG] Sync->loadStates(): loading states for folder 'U5e894'
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::GetAdditionalSyncFolderStore('0879775ab9d94a42901de37ae24a9a9ca40100000000'): 'false'
13/08/2019 13:36:12 [ 9528] [DEBUG] ZPush::GetAdditionalSyncFolderStore('0879775ab9d94a42901de37ae24a9a9ca40100000000'): 'false'
13/08/2019 13:36:12 [ 9528] [DEBUG] ExportChangesICS->Config(): synching inital data
13/08/2019 13:36:12 [ 9528] [DEBUG] ExportChangesICS->Config() initialized with state: 0x0000000000000000
13/08/2019 13:36:12 [ 9528] [WARN] StatusException: ExportChangesICS->InitializeExporter(): Error, mapi_exportchanges_config() failed: 0xFFFFFFFF8004010F - code: 12 - file: /usr/share/z-push/backend/kopano/exporter.php:230
13/08/2019 13:36:12 [ 9528] [DEBUG] LoopDetection->ProcessLoopDetectionAddStatus: 'U5e894' with status 12
13/08/2019 13:36:12 [ 9528] [DEBUG] WBXMLEncoder->startWBXML() type: vnd.ms-sync.wbxml
13/08/2019 13:36:12 [ 9528] [WBXML] O  <Synchronize>
13/08/2019 13:36:12 [ 9528] [WBXML] O   <Status>
13/08/2019 13:36:12 [ 9528] [WBXML] O   12
13/08/2019 13:36:12 [ 9528] [WBXML] O   </Status>
13/08/2019 13:36:12 [ 9528] [WBXML] O  </Synchronize>
13/08/2019 13:36:12 [ 9528] [DEBUG] WBXMLEncoder->endTag() WBXML output completed
13/08/2019 13:36:12 [ 9528] [WBXML] WBXML-OUT: AwFqAEVOAzEyAAEB
13/08/2019 13:36:12 [ 9528] [WBXML] WBXML-IN : AwFqAEVcT0sDMAABUgNVNWU4OTQAAQEBAQ==
13/08/2019 13:36:12 [ 9528] [DEBUG] LoopDetection->ProcessLoopDetectionTerminate()
13/08/2019 13:36:12 [ 9528] [ INFO] cmd='Sync' memory='2.87 MiB/2.00 MiB' time='0.11s' devType='Android' devId='android1263512214918' getUser='max.mustermann' from='123.123.123.123' idle='0s' version='2.5.0+0-0' method='POST' httpcode='200'
13/08/2019 13:36:12 [ 9528] [DEBUG] -------- End

Their is not much information visible, only the same error as in the normal logfile.

Perhaps the problem could also be another source and not depends on the amount of entries? because the calendar affected user was some long time ago migrated from scalix to zarafa (via ical) and later migrated from an old zarafa to a kopano (also ical) some months ago. And now two week’s i updated the kopano to the version above.

The problem is not much prio at them moment, because it’s only one user which is affected.

I also cannot migrated at the moment back from MariaDB to mySQL, because all users together had a big database which needs some time to backup and restore.

badewaschl

Hi all,

I am also having this issue with one user. I tried to set enable_enhanced_ics = no but that did not help.

Next thing I ran z-push-admin -a remove and clearing all data on android, but to no avail.

Then I tried to clean up using kopano-fsck, but that reported the same 14 issues every time I ran it, claiming the issues were fixed. I ended up manually deleting the affected calendar entries. But the z-push issue still remained.

I even tried to use kopano-archiver as suggested above, but could not get it to work (see separate post).

My last desperate attempt was to copy all data to a new user using kopano-backup. About 1% of all emails, contacts and calendar entries got lost in the process (for example the tool seems to skip all contacts with a picture) - but also with the new user, the calendar disappeared from android after a few hours.

Now I am at my wit’s end. Does anybody have an idea what else I can do? Everything worked fine for years using zarafa, but ever since I upgraded to kopano the problems keep piling up. Was it maybe a bad idea to reuse the zarafa database?

I am using the latest kopano/z-push/apache packages from Debian 10.
On Android I connect using the Nine groupware client.

Regards

badewaschl

thctlo

Try this :

wget -qO - http://repo.z-hub.io/z-push:/final/Debian_10/Release.key | sudo apt-key add -
echo "deb http://repo.z-hub.io/z-push:/final/Debian_10 /" > /etc/apt/sources.list.d/z-push.list
apt update
apt dist-upgrade 
apt-get install z-push-common

Note: package z-push in debian = z-push-common on the z-push repo.
optional also install : z-push-config-apache z-push-config-apache-autodiscover

hugo21212

I have the same problem:

15/08/2019 13:00:51 [15452] [WARN] [x@y.z] StatusException: ExportChangesICS->InitializeExporter(): Error, mapi_exportchanges_config() failed: 0xFFFFFFFF8004010F - code: 12 - file: /usr/share/z-push/backend/kopano/exporter.php:230

But only for Android using Exchange and synchronizing the calendar.
If I use Outlook with ActiveSync, there is no problem. I can see the calendar and create entries

Installed packages:

ii  z-push-autodiscover                  2.5.1+0-0                     all          Z-Push autodiscover
ii  z-push-backend-caldav                2.5.1+0-0                     all          Z-Push caldav backend
ii  z-push-backend-kopano                2.5.1+0-0                     all          Z-Push Kopano backend
ii  z-push-common                        2.5.1+0-0                     all          open source implementation of the ActiveSync protocol
ii  z-push-config-apache                 2.5.1+0-0                     all          Z-Push apache configuration
ii  z-push-config-apache-autodiscover    2.5.1+0-0                     all          Z-Push autodiscover apache configuration
ii  z-push-ipc-sharedmemory              2.5.1+0-0                     all          Z-Push ipc shared memory provider

on Debian 10 Buster

And one thing: in Debian 10 there is no such thing as php7-mapi.

> apt install z-push-kopano z-push-backend-kopano

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 z-push-backend-kopano : Depends: php7-mapi but it is not installable or
                                  php5-mapi but it is not installable
E: Unable to correct problems, you have held broken packages.

only:

ii  php-mapi                             8.7.0-3                       amd64        Complete and feature rich groupware solution - PHP MAPI bindings

hugo21212

For me it helped to delete all old calender entries. Then synchronization worked.

It would be good to see, why it failed (corrupt entry, too many entries etc.).

thctlo

apt remove php-mapi --autoremove
apt install kopano-server-packages

Ow, before you remove php-mapi, verify the other packages of kopano also.

dpkg -l |grep “8.7.0-3”

Beleggrodion

@badewaschl said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

Then I tried to clean up using kopano-fsck, but that reported the same 14 issues every time I ran it, claiming the issues were fixed. I ended up manually deleting the affected calendar entries. But the z-push issue still remained.

Thanks. I forgot completely that this tool exists. I tried this on my side and it worked for the affected user :-)

badewaschl

@thctlo said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

apt install kopano-server-packages

# apt install kopano-server-packages
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package kopano-server-packages

thctlo

@badewaschl
Hai, then you are probely not have the local repo.

I suggest get this and run it, IF the packages are online again.
and i just looked at : https://download.kopano.io/community/
And its not, so reuse the older downloaded kopano files.

wget https://raw.githubusercontent.com/thctlo/Kopano/master/get-kopano-community.sh

get this script run it, ignore the errors then get the files your already downloaded before.
Put them in, based on :

BASE_FOLDER=$HOME/kopano-repo

# A subfolder in BASE_FOLDER.
KOPANO_EXTRACT2FOLDER="apt"

$HOME/kopano-repo/apt

And run the script again.
Now your able to use apt and kopano-server-packages should be available now.

fbartels

So far we have been unable to reproduce this issue. If running kopano-fsk indeed makes a difference, then it would be great if someone could provide a dump of such a store to feedback@kopano.com (or get in contact with our support for a remote session if you have a valid subscription).

Edit: since there is an influx of users saying that they are running an older release on Debian 10 (presumably installed from the downstream Debian repository) I would not recommend to mix our packages with the ones provided by the Debian Maintainers. They are using a different naming schema (and only our packages and releases are actively tested by us) and are not compatible with each other.

thctlo

@fbartels
There is a dependecy problem with php-mapi.
The debian kopano 8.7.0 should never have been release and i really advice everyone not to use it.
Just to buggy. i tried to get 8.7.3 in but Buster was frozen already.

if you have the debian kopano (8.7.0-3) installed and your upgrade to the “development/community” packages, your upgrade will fail due to a missing depend on php7-mapi ( from kopano-repo.)

If kopano adds a “Replaces” php-mapi in the packaging for packages php7-mapi
then this upgrade problem is fixed in the future.

Greetz,

Louis

fbartels

@thctlo said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

The debian kopano 8.7.0 should never have been release

You would need to complain on the Debian mailinglist about that. We have no influence on packages in the Debian repositories. For the Maintainers there it’s probably also more helpful to list actual problems instead of saying that it was all a mistake.

@thctlo said in z-push 2.4.5 sync issue - 0xFFFFFFFF8004010F:

If kopano adds a “Replaces” php-mapi in the packaging for packages php7-mapi
then this upgrade problem is fixed in the future.

I do doubt that a replaces in a single package would make a difference, the general advice is to not mix packages and when following our documentation to install our packages this will not happen.

thctlo

@fbartels
why complain at #Debian, its not complaining, im just giving a notice that the debian packages are not very good to use and its better to use the Kopano packages.

If i upgrade the kopano packages with debian packages, then yes, i can give them a notice but it’s visaversa.

If you upgrade Debian with Kopano packages, then the Kopano packages should take the “php-mapi” in count.

Debian changed php7-mapi to php-mapi, dont ask my why.
I have to do the same with packaging samba and the python2 too python3 changes.