z-push 2.4.0 with IMAP backend: Android Mails sometimes not syncing

kevbo

Hi Manfred,

yes, new z-push installation with empty state folder. I also removed the account from the Android device and cleared all app data before connecting to z-push 2.4.0.

The device issues a Ping command when reconnecting:

18/03/2018 10:08:08 [19992] [DEBUG] [testuser] cmd='Ping' devType='Android' devId='boxerabcd1234567800000' getUser='testuser' from='192.0.2.1' version='GIT' method='POST'

Thanks,
Kevin

Manfred

Hi Kevin,

could you post WBXML log after reconnecting?

Have you tried the standard email app? Because the devices usually issue Sync and only afterwards Ping if they were offline.

Manfred

kevbo

Hi Manfred,

there is a link to the full Logfile in my initial post. The device reconnects at 10:08:08.

In addition, here is a second logfile from the same where the email is pushed correctly: Logfile 2

The way I read it, in this 2nd logfile the procedure is as follows:

client connects using Ping command
server connects to the backend, looks for changes and finds a new mail in the INBOX folder
server sends “Ping:Status 2” to the client which means that it should sync the folder
client sends Sync command and gets the new mail

But in the 1st case where it does not work, the following seems to happen:

client connects using Ping command
client disconnects (for example because the WiFi connection is lost)
the Ping command is still active, at 10:06:56 a new mail arrives and the server sends “Ping:Status 2” to the client, but that command is never received because the client is offline
the client reconnects at 10:08:08 using a new Ping command, the server connects to the backend, doesn’t find any further changes and therefore does not send “Ping:Status 2” again.

I think this is the cause of the problem because the server should be aware that the client didn’t receive the previous “Ping:Status 2” command and therefore should send it again.

Here is a third logfile from z-push 2.1.3 where this procedure works as described: Logfile 3

In this logfile the server recognizes a new mail at 17:37:43 and sends “Ping:Status 2” to the client, but the client is offline and doesn’t receive it.
At 17:38:12 the client reconnects and sends a new Ping command, the server sends “Ping:Status 2” again.

So I think the question would be: How can I get z-push 2.4.0 to behave like z-push 2.1.3 is this situation? Maybe it’s just a few lines of code, but honestly I don’t know where to look.

Many Thanks,
Kevin

kevbo

Ok, I think I found it. I changed the function HasChangesSink in the file backend/imap/imap.php from

    public function HasChangesSink() {
        return true;
    }

to

    public function HasChangesSink() {
        return false;
    }

It seems that now everything is working as expected.

Regards,
Kevin

Manfred

Hi Kevin,

your hint that disabling ChangesSink solved the issue helped to narrow down why this happened. There are two causes for this issue and solving any of them would solve this problem.
For one the imap backend lacks an improvement in tracking changes in folders (known as folder stats). Another point is that there is no fallback in core if these folder stats are not available in the backend and so the Ping request doesn’t find changes while a device was offline. I’ve created JIRA issues to solve it: https://jira.z-hub.io/browse/ZP-1380 and https://jira.z-hub.io/browse/ZP-1381.

Thanks for reporting and investigating it.

Manfred

kour1er

Just to follow on for this issue: I’m seeing what I think is a similar situation on iOS using Dovecot as the backend. When I alter backend/imap/imap.php as kevbo listed above it works for me too. The test situation is similar - iOS device connected via Z-Push to Dovecot. Push works fine. iOS Device goes offline (airplane mode for simulated testing). During this offline time, if a new email arrives, when the iOS device goes back online, no push is received. When another email is received, that is then pushed.

@Manfred I notice you say that folder stats are missing. Do you happen to know if there is anything I can do from a Dovecot perspective to add that ability? Or does that just not exist in Dovecot world? Thanks for any insight you can offer.

Manfred

Hi kour1er,

@kour1er said in z-push 2.4.0 with IMAP backend: Android Mails sometimes not syncing:

@Manfred I notice you say that folder stats are missing. Do you happen to know if there is anything I can do from a Dovecot perspective to add that ability? Or does that just not exist in Dovecot world? Thanks for any insight you can offer.

it’s a Z-Push issue, so there’s nothing to do on Dovecot side. The JIRA issues I mentioned in my posts need to be solved in order to fix this problem.

Manfred

kour1er

@Manfred cool - thanks for the prompt response! I’ll keep an eye on those JIRA issues and test as soon as you have time to work on them. Cheers

lt50

@manfred said in z-push 2.4.0 with IMAP backend: Android Mails sometimes not syncing:

it’s a Z-Push issue, so there’s nothing to do on Dovecot side. The JIRA issues I mentioned in my posts need to be solved in order to fix this problem.

Manfred

@Manfred Is setting HasChangesSink to false an ok work around or is there something else we can do temporarily while the issues in JIRA are being worked on?

On some Android devices that make use of Doze and Background like Google Pixels. It can result in temporary network disconnection with only periodic access anytime the phone is stationary with the screen off. This is resulting in the bug happening multiple times per hour on these devices.

Manfred

Hi lt50,

yes, that’s fine. ChangesSink is just a way of getting changes and disabling it will solve your issue in the meantime.

Manfred