z-push 2.4.0 with IMAP backend: Android Mails sometimes not syncing

kevbo

Hi,

I’ve been trying to replace z-push 2.1.1 with 2.3.9 and, more recently, 2.4.0.
With both newer versions, arriving mails are sometimes pushed to the client, but sometimes not. The whole thing seemed totally random at first, but after several hours of testing I now narrowed it down to the following phenomenon:

If a new mail arrives while the Android device is online, the mail is pushed within a few seconds.

If a new mail arrives after the device has been offline for several hours, the mail is pushed to the device as soon as it goes online again.

If the Android device goes offline (for example WiFi disconnect), there is still a Ping command running on the z-push server until the timeout is reached. If a new mail arrives while that Ping command is still active, the mail is NOT pushed to the device after it goes online again.
But if another mail arrives now with the device being online, both mails are pushed within a few seconds.

So it seems that if there are any folder changes detected while a Ping command is still running, but the device is not reachable because it has been disconnected, these changes are never pushed until there are either some other changes in that folder (for example another mail) or a manual resync is triggered.

This seems kind of weird to me as it has always been working perfectly with z-push 2.1.1. Also it doesn’t seem to be an issue with the device because it happens both on Android 6.0 and 7.0 and both on Boxer and Nine.

I didn’t do any state migration from the old version or anything like that, just a plain new installation. The configuration files are pretty standard too, I didn’t change any of the default values except:

config.php:
define(‘LOGLEVEL’, LOGLEVEL_WBXML);
define(‘BACKEND_PROVIDER’, ‘BackendIMAP’);

backend/imap/config.php:
define(‘IMAP_FOLDER_CONFIGURED’, true);

And for now, I also set
define(‘PING_HIGHER_BOUND_LIFETIME’, 60);
which mitigates the problem somehow, but it doesn’t seem like a sufficient solution and also means that if a new mail arrives within 60 seconds after the device went offline, that mail will still not be pushed.

Logfile
10:05 device connecting to z-push server
10:06 device disconnecting, Ping command still running on z-push server, new mail arriving and apparently being recognized by z-push
10:08 device going online again and reconnecting, but mail not being pushed
10:09 second mail arriving, both mails are pushed to the device

Thanks,
Kevin

Manfred

Hi Kevin,

did I understand correctly that you did a new Z-Push 2.4.0 installation and started the synchronisation from zero (meaning the state folder was empty and the device downloaded emails from the server again)?

When the device reconnects, does it issue Sync or Ping command?

Manfred

kevbo

Hi Manfred,

yes, new z-push installation with empty state folder. I also removed the account from the Android device and cleared all app data before connecting to z-push 2.4.0.

The device issues a Ping command when reconnecting:

18/03/2018 10:08:08 [19992] [DEBUG] [testuser] cmd='Ping' devType='Android' devId='boxerabcd1234567800000' getUser='testuser' from='192.0.2.1' version='GIT' method='POST'

Thanks,
Kevin

Manfred

Hi Kevin,

could you post WBXML log after reconnecting?

Have you tried the standard email app? Because the devices usually issue Sync and only afterwards Ping if they were offline.

Manfred

kevbo

Hi Manfred,

there is a link to the full Logfile in my initial post. The device reconnects at 10:08:08.

In addition, here is a second logfile from the same where the email is pushed correctly: Logfile 2

The way I read it, in this 2nd logfile the procedure is as follows:

client connects using Ping command
server connects to the backend, looks for changes and finds a new mail in the INBOX folder
server sends “Ping:Status 2” to the client which means that it should sync the folder
client sends Sync command and gets the new mail

But in the 1st case where it does not work, the following seems to happen:

client connects using Ping command
client disconnects (for example because the WiFi connection is lost)
the Ping command is still active, at 10:06:56 a new mail arrives and the server sends “Ping:Status 2” to the client, but that command is never received because the client is offline
the client reconnects at 10:08:08 using a new Ping command, the server connects to the backend, doesn’t find any further changes and therefore does not send “Ping:Status 2” again.

I think this is the cause of the problem because the server should be aware that the client didn’t receive the previous “Ping:Status 2” command and therefore should send it again.

Here is a third logfile from z-push 2.1.3 where this procedure works as described: Logfile 3

In this logfile the server recognizes a new mail at 17:37:43 and sends “Ping:Status 2” to the client, but the client is offline and doesn’t receive it.
At 17:38:12 the client reconnects and sends a new Ping command, the server sends “Ping:Status 2” again.

So I think the question would be: How can I get z-push 2.4.0 to behave like z-push 2.1.3 is this situation? Maybe it’s just a few lines of code, but honestly I don’t know where to look.

Many Thanks,
Kevin

kevbo

Ok, I think I found it. I changed the function HasChangesSink in the file backend/imap/imap.php from

    public function HasChangesSink() {
        return true;
    }

to

    public function HasChangesSink() {
        return false;
    }

It seems that now everything is working as expected.

Regards,
Kevin

Manfred

Hi Kevin,

your hint that disabling ChangesSink solved the issue helped to narrow down why this happened. There are two causes for this issue and solving any of them would solve this problem.
For one the imap backend lacks an improvement in tracking changes in folders (known as folder stats). Another point is that there is no fallback in core if these folder stats are not available in the backend and so the Ping request doesn’t find changes while a device was offline. I’ve created JIRA issues to solve it: https://jira.z-hub.io/browse/ZP-1380 and https://jira.z-hub.io/browse/ZP-1381.

Thanks for reporting and investigating it.

Manfred

kour1er

Just to follow on for this issue: I’m seeing what I think is a similar situation on iOS using Dovecot as the backend. When I alter backend/imap/imap.php as kevbo listed above it works for me too. The test situation is similar - iOS device connected via Z-Push to Dovecot. Push works fine. iOS Device goes offline (airplane mode for simulated testing). During this offline time, if a new email arrives, when the iOS device goes back online, no push is received. When another email is received, that is then pushed.

@Manfred I notice you say that folder stats are missing. Do you happen to know if there is anything I can do from a Dovecot perspective to add that ability? Or does that just not exist in Dovecot world? Thanks for any insight you can offer.

Manfred

Hi kour1er,

@kour1er said in z-push 2.4.0 with IMAP backend: Android Mails sometimes not syncing:

@Manfred I notice you say that folder stats are missing. Do you happen to know if there is anything I can do from a Dovecot perspective to add that ability? Or does that just not exist in Dovecot world? Thanks for any insight you can offer.

it’s a Z-Push issue, so there’s nothing to do on Dovecot side. The JIRA issues I mentioned in my posts need to be solved in order to fix this problem.

Manfred

kour1er

@Manfred cool - thanks for the prompt response! I’ll keep an eye on those JIRA issues and test as soon as you have time to work on them. Cheers

lt50

@manfred said in z-push 2.4.0 with IMAP backend: Android Mails sometimes not syncing:

it’s a Z-Push issue, so there’s nothing to do on Dovecot side. The JIRA issues I mentioned in my posts need to be solved in order to fix this problem.

Manfred

@Manfred Is setting HasChangesSink to false an ok work around or is there something else we can do temporarily while the issues in JIRA are being worked on?

On some Android devices that make use of Doze and Background like Google Pixels. It can result in temporary network disconnection with only periodic access anytime the phone is stationary with the screen off. This is resulting in the bug happening multiple times per hour on these devices.

Manfred

Hi lt50,

yes, that’s fine. ChangesSink is just a way of getting changes and disabling it will solve your issue in the meantime.

Manfred