z-push 2.4.0 with IMAP backend: Android Mails sometimes not syncing

Hi,

I’ve been trying to replace z-push 2.1.1 with 2.3.9 and, more recently, 2.4.0.
With both newer versions, arriving mails are sometimes pushed to the client, but sometimes not. The whole thing seemed totally random at first, but after several hours of testing I now narrowed it down to the following phenomenon:

If a new mail arrives while the Android device is online, the mail is pushed within a few seconds.

If a new mail arrives after the device has been offline for several hours, the mail is pushed to the device as soon as it goes online again.

If the Android device goes offline (for example WiFi disconnect), there is still a Ping command running on the z-push server until the timeout is reached. If a new mail arrives while that Ping command is still active, the mail is NOT pushed to the device after it goes online again.
But if another mail arrives now with the device being online, both mails are pushed within a few seconds.

So it seems that if there are any folder changes detected while a Ping command is still running, but the device is not reachable because it has been disconnected, these changes are never pushed until there are either some other changes in that folder (for example another mail) or a manual resync is triggered.

This seems kind of weird to me as it has always been working perfectly with z-push 2.1.1. Also it doesn’t seem to be an issue with the device because it happens both on Android 6.0 and 7.0 and both on Boxer and Nine.

I didn’t do any state migration from the old version or anything like that, just a plain new installation. The configuration files are pretty standard too, I didn’t change any of the default values except:

config.php:
define(‘LOGLEVEL’, LOGLEVEL_WBXML);
define(‘BACKEND_PROVIDER’, ‘BackendIMAP’);

backend/imap/config.php:
define(‘IMAP_FOLDER_CONFIGURED’, true);

And for now, I also set
define(‘PING_HIGHER_BOUND_LIFETIME’, 60);
which mitigates the problem somehow, but it doesn’t seem like a sufficient solution and also means that if a new mail arrives within 60 seconds after the device went offline, that mail will still not be pushed.

Logfile
10:05 device connecting to z-push server
10:06 device disconnecting, Ping command still running on z-push server, new mail arriving and apparently being recognized by z-push
10:08 device going online again and reconnecting, but mail not being pushed
10:09 second mail arriving, both mails are pushed to the device

Thanks,
Kevin

Hi Kevin,

did I understand correctly that you did a new Z-Push 2.4.0 installation and started the synchronisation from zero (meaning the state folder was empty and the device downloaded emails from the server again)?

When the device reconnects, does it issue Sync or Ping command?

Manfred

Hi Manfred,

yes, new z-push installation with empty state folder. I also removed the account from the Android device and cleared all app data before connecting to z-push 2.4.0.

The device issues a Ping command when reconnecting:

18/03/2018 10:08:08 [19992] [DEBUG] [testuser] cmd='Ping' devType='Android' devId='boxerabcd1234567800000' getUser='testuser' from='192.0.2.1' version='GIT' method='POST'

Thanks,
Kevin

Hi Kevin,

could you post WBXML log after reconnecting?

Have you tried the standard email app? Because the devices usually issue Sync and only afterwards Ping if they were offline.

Manfred

Hi Manfred,

there is a link to the full Logfile in my initial post. The device reconnects at 10:08:08.

In addition, here is a second logfile from the same where the email is pushed correctly: Logfile 2

The way I read it, in this 2nd logfile the procedure is as follows:

  • client connects using Ping command
  • server connects to the backend, looks for changes and finds a new mail in the INBOX folder
  • server sends “Ping:Status 2” to the client which means that it should sync the folder
  • client sends Sync command and gets the new mail

But in the 1st case where it does not work, the following seems to happen:

  • client connects using Ping command
  • client disconnects (for example because the WiFi connection is lost)
  • the Ping command is still active, at 10:06:56 a new mail arrives and the server sends “Ping:Status 2” to the client, but that command is never received because the client is offline
  • the client reconnects at 10:08:08 using a new Ping command, the server connects to the backend, doesn’t find any further changes and therefore does not send “Ping:Status 2” again.

I think this is the cause of the problem because the server should be aware that the client didn’t receive the previous “Ping:Status 2” command and therefore should send it again.

Here is a third logfile from z-push 2.1.3 where this procedure works as described: Logfile 3

In this logfile the server recognizes a new mail at 17:37:43 and sends “Ping:Status 2” to the client, but the client is offline and doesn’t receive it.
At 17:38:12 the client reconnects and sends a new Ping command, the server sends “Ping:Status 2” again.

So I think the question would be: How can I get z-push 2.4.0 to behave like z-push 2.1.3 is this situation? Maybe it’s just a few lines of code, but honestly I don’t know where to look.

Many Thanks,
Kevin

Ok, I think I found it. I changed the function HasChangesSink in the file backend/imap/imap.php from

    public function HasChangesSink() {
        return true;
    }

to

    public function HasChangesSink() {
        return false;
    }

It seems that now everything is working as expected.

Regards,
Kevin

Hi Kevin,

your hint that disabling ChangesSink solved the issue helped to narrow down why this happened. There are two causes for this issue and solving any of them would solve this problem.
For one the imap backend lacks an improvement in tracking changes in folders (known as folder stats). Another point is that there is no fallback in core if these folder stats are not available in the backend and so the Ping request doesn’t find changes while a device was offline. I’ve created JIRA issues to solve it: https://jira.z-hub.io/browse/ZP-1380 and https://jira.z-hub.io/browse/ZP-1381.

Thanks for reporting and investigating it.

Manfred

Log in to reply

Looks like your connection to Kopano Community Forum was lost, please wait while we try to reconnect.