Z-Push no longer works after switch from Zarafa to Kopano and local install to packaged version

kitserve

We recently migrated our mail from Zarafa 7.2.6 to the latest version of Kopano, and also Z-Push from a manual tarball install to the packaged version from the repos. Since then it no longer works. Client devices just get a ‘Connection to server failed’ error message. There is nothing in /var/log/z-push/z-push-error.log. I’m able to access the ActiveSync url using GET in a browser with no problems. Making a connection request from a device shows the following in /var/log/z-push/z-push.log:

12/10/2017 19:20:57 [ 8638] [ INFO] [user] Options request
12/10/2017 19:20:57 [ 8638] [ INFO] [user] cmd='' memory='1.67 MiB/2.00 MiB' time='0.05s' devType='' devId='' getUser='user' from='1.2.3.4' idle='0s' version='2.3.8+0-0' method='OPTIONS' httpcode='200'

Nothing else appears. This looks related to https://forum.kopano.io/topic/601/z-push-connection-fails-on-imap-and-combined-backend/6, in that it fails without giving any useful debugging information, but there are some key differences. The system I’m referring to in this thread is Debian Jessie running Kopano as the backend and using PHP5 through FPM. The system in the other thread is Debian Stretch running an IMAP backend and using PHP7.0 through mod_php.

Any help with debugging would be gratefully received because frankly, I’m at a loss.

fbartels

Hi @kitserve ,

Could you post the versions you are using?

By default the packages expect that mod_php is used. Could you check if this would work?

Was it an in place upgrade or install on a new machine? Did you remove all the files of the old Z-Push installation? Can you check if all the configuration files are resolving correctly?

kitserve

Hi Felix,

I was mistaken, Z-Push is running through mod_php after the upgrade. FPM is not in use.

Debian Jessie 8.9
Apache 2.4.10-10+deb8u11
PHP 5.6.30+dfsg-0+deb8u1
Kopano 8.4.90.448-0+80.1
Z-Push 2.3.8+0-0

This was an in place upgrade. I believe that all of the old Z-Push files were removed, certainly they were in a different directory (/usr/local/share/z-push as opposed to the new location of /usr/share/z-push). We left the state files in place.

To make things more complicated, accounts on Android devices still seem to be working (or at least one of them). It only seems to be iOS devices that aren’t working. Here’s a snippet from the logs for an Android device.

13/10/2017 11:23:54 [16834] [ INFO] [user] cmd='Sync' memory='3.11 MiB/3.50 MiB' time='0.26s' devType='Android' devId='androidXXXXXX' getUser='user' from='1.2.3.4' idle='0s' version='2.3.8+0-0' method='POST' httpcode='200'
13/10/2017 11:23:54 [16834] [ INFO] [user] SyncCollections->CheckForChanges(): Waiting for store changes... (lifetime 1680 seconds)

It looks as though iOS devices aren’t getting their devType and devId through to Z-Push, although I don’t know if that’s a cause or a symptom of the problem.

kitserve

@fbartels said in Z-Push no longer works after switch from Zarafa to Kopano and local install to packaged version:

Can you check if all the configuration files are resolving correctly?

Oh, and what do you mean by this question?

fbartels

@kitserve said in Z-Push no longer works after switch from Zarafa to Kopano and local install to packaged version:

Oh, and what do you mean by this question?

there is a config.php in /usr/share/z-push, but this is actually a symlink to /etc/z-push. Same accounts for the backend configuration.

AnotherAndy

hi @kitserve ,
have you updated your IOS Deviced to the latest OS Version 11.03?
Apple fixed in 11.02 and 11.03 a lot of “Mail Server problems” which occurred in IOS 11.

Can you try the z-push with another client like windows mail oder outlook 2013++ ?
For more infos, I recommend to set the Debug levelin /etc/z-push/ z-push.conf.php higher.

Just for your info - I am running nearly the same environment.
Debian 8, PHp 5.6, Apache 2.4., Z-Push 2.3.8 nad kopano 8.3.4.12-0+23.1. I don’t have any IOS deviced but Android and Outlook are working fine.
Best regards

kitserve

@fbartels: I edited the config files in /etc/z-push, and I can confirm that the symlinks for config.php and policies.ini in /usr/share/z-push are correctly set.

@Merlin2104: We are an exclusively Linux shop, so I don’t have an easy way of testing with Window Mail or Outlook 2013/2016. I just tried increasing the log level to LOGLEVEL_WBXMLSTACK and recreating the account on an iPad mini 2, running the latest iOS 11.0.3. It failed with the error ‘Exchange Account - Unable to verify account information’. Log details are below:

15/10/2017 11:26:50 [11603] [DEBUG] [user] [] -------- Start
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] cmd='' devType='' devId='' getUser='user' from='1.2.3.4' version='2.3.8+0-0' method='OPTIONS'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] Used timezone 'Europe/London'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] BackendKopano using PHP-MAPI version: 8.4.90 - PHP version: 5.6.30-0+deb8u1
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] Request::ProcessHeaders() ASVersion: 14.0
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] KopanoBackend->Logon(): Trying to authenticate user 'user'..
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] KopanoBackend->openMessageStore('user'): Found 'DEFAULT' store: 'Resource id #17'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] KopanoBackend->Logon(): User 'user' is authenticated
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] Store supports properties containing Unicode characters.
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] NoPostRequestException: Options request - code: 1 - file: /usr/share/z-push/index.php:66
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] ZPush::GetSupportedProtocolVersions(): 12.0,12.1,14.0
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] ZPush::GetSupportedCommands(): Sync,SendMail,SmartForward,SmartReply,GetAttachment,GetHierarchy,CreateCollection,DeleteCollection,MoveCollection,FolderSync,FolderCreate,FolderDelete,FolderUpdate,MoveItems,GetItemEstimate,MeetingResponse,ResolveRecipients,ValidateCert,Provision,Search,Ping,Notify,ItemOperations,Settings
15/10/2017 11:26:50 [11603] [ INFO] [user] [] Options request
15/10/2017 11:26:50 [11603] [ INFO] [user] [] cmd='' memory='1.67 MiB/2.00 MiB' time='0.06s' devType='' devId='' getUser='user' from='1.2.3.4' idle='0s' version='2.3.8+0-0' method='OPTIONS' httpcode='200'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] -------- End

Perhaps the NoPostRequest Exception is related?

Thanks for your help so far.

liverpoolfcfan

Are you using a LetsEncrypt certificate?

I have seen this happen on an iOS device when connecting to a z-push server configured with a LetsEncrypt certificate. It seems that Apple do not trust them by default.

I just clicked on check details, and then accept (or whatever the option was called) and then it seemed to work OK

kitserve

Thanks, good suggestion but we’re using a paid Comodo certificate, not Let’s Encrypt. The setup worked previously with Zarafa and the certificate hasn’t been changed.

Manfred

Hi kitserve,

could you nevertheless check the certificate details as liverpoolfcfan described?

Are there any errors in apache logs?

Manfred

kitserve

I think liverpoolfcfan is referring to the step in the setup where the device attempts to access the autodiscover config. Telling the device to trust the certificate doesn’t help unfortunately, immediately after selecting ‘Trust’ I get the same ‘Unable to verify account information’ error.

Out of curiosity, I just set up autodiscover with a valid certificate to see if it would make a difference, but it didn’t help. In this case I don’t see any warnings about the certificate at all, it just goes straight to the verification error.

Here’s the Apache access log:

1.2.3.4 - user@example.com [19/Oct/2017:09:51:22 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6275 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user [19/Oct/2017:09:51:22 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6275 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user [19/Oct/2017:09:51:22 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6278 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user@example.com [19/Oct/2017:09:51:23 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6278 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user@example.com [19/Oct/2017:09:51:24 +0100] "OPTIONS /Microsoft-Server-ActiveSync HTTP/1.1" 200 5756 "-" "Apple-iPhone8C4/1501.432"

The Apache error log is empty. The Z-Push error log is empty. The main Z-Push access log contains the following, the same as before:

19/10/2017 09:51:24 [23472] [DEBUG] [user] [] -------- Start
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] cmd='' devType='' devId='' getUser='user' from='1.2.3.4' version='2.3.8+0-0' method='OPTIONS'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] Used timezone 'Europe/London'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] BackendKopano using PHP-MAPI version: 8.4.90 - PHP version: 5.6.30-0+deb8u1
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] Request::ProcessHeaders() ASVersion: 14.0
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] KopanoBackend->Logon(): Trying to authenticate user 'user'..
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] KopanoBackend->openMessageStore('user'): Found 'DEFAULT' store: 'Resource id #16'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] KopanoBackend->Logon(): User 'user' is authenticated
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] Store supports properties containing Unicode characters.
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] NoPostRequestException: Options request - code: 1 - file: /usr/share/z-push/index.php:66
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] ZPush::GetSupportedProtocolVersions(): 12.0,12.1,14.0
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] ZPush::GetSupportedCommands(): Sync,SendMail,SmartForward,SmartReply,GetAttachment,GetHierarchy,CreateCollection,DeleteCollection,MoveCollection,FolderSync,FolderCreate,FolderDelete,FolderUpdate,MoveItems,GetItemEstimate,MeetingResponse,ResolveRecipients,ValidateCert,Provision,Search,Ping,Notify,ItemOperations,Settings
19/10/2017 09:51:24 [23472] [ INFO] [user] [] Options request
19/10/2017 09:51:24 [23472] [ INFO] [user] [] cmd='' memory='1.67 MiB/2.00 MiB' time='0.06s' devType='' devId='' getUser='user' from='1.2.3.4' idle='0s' version='2.3.8+0-0' method='OPTIONS' httpcode='200'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] -------- End

Manfred

Hi kitserve,

the fact that there’s an OPTIONS request coming from the mobile device indicates that it is able to connect to the server. However I don’t have any idea why it won’t continue. It sounds silly, but have you tried turning the mobile off and on again and creating account then?

Manfred

kitserve

It’s generally worth asking the silly questions, but yes, I have tried that several times!

Manfred

Hi kitserve,

do you remember which Z-Push version was before? Have you tried to remove the device with z-push-admin and recreate account again?

Could you post the Z-Push config files of apache?

Manfred

kitserve

Sorry, I don’t remember the previous version but it was relatively recent. It was certainly upgraded within the last few months. Yes, I deleted all of my user data with z-push-admin and tried to add the device again. The Apache config contains the default Z-Push options. I just copy and pasted it back in from the official .deb to make sure we weren’t missing anything:

# Z-Push - ActiveSync over-the-air - default Apache configuration
<IfModule mod_alias.c>
    Alias /Microsoft-Server-ActiveSync /usr/share/z-push/index.php
</IfModule>

<Directory /usr/share/z-push>
    # Don't list a directory index, follow symlinks (maybe state dir is somewhere linked)
    DirectoryIndex index.php
    Options -Indexes +FollowSymLinks

    # Z-push requirements
    php_value magic_quotes_gpc off
    php_value magic_quotes_runtime off
    php_value register_globals off
    php_value short_open_tag on

    # Optional
    # php_value display_errors off

    # Setting memory limit higher (larger attachments)
    php_value memory_limit 128M
    
    # Security
    # Don't allow .htaccess Overrides, disallow access to files
    AllowOverride none
    <IfModule !mod_authz_core.c>
        Order allow,deny
        allow from all
    </IfModule>
    <IfModule mod_authz_core.c>
        Require all granted
    </IfModule> 

    <Files "config.php">
      <IfModule !mod_authz_core.c>
        Deny from All
      </IfModule>
      <IfModule mod_authz_core.c>
        Require all denied
      </IfModule>
    </Files>
</Directory>

Sebastian

I can only imagine that this is some configuration/permission issue… But as you said, Android is working so this seems not to be the case.

I have seen once that iOS completely refused to connect to the server. The only solution was there to remove the account from the device and to add it again.
You could try that (again).

Sorry, no further ideas.

kitserve

Things got even weirder. I just switched Z-Push over to a manual install in /usr/local/share/z-push using the latest tarball, and it worked with no problems. I diffed the main config and the Kopano backend config, and they are functionally identical. Totally perplexed by this. Perhaps it is some quirk of our server config but it’s hard to know what it could be. It’s frustrating that we can’t use the repo packaged version of Z-Push, but given that it’s working now and I have already sunk far too many hours into fixing it, I’m not going to spend more time on it now! Thanks for all your help.

kitserve

After testing it last night and verifying that everything was working, I got up this morning to discover that it had stopped working again! I don’t think words can express quite how frustrated I am with Z-Push at the moment!

fbartels

Hello @kitserve ,

if you have a subscription I would recommend to get into contact with our support to have someone take a closer look at your system.

Meanwhile you could also have a look at your Apache configuration. That it stopped working over night makes me think this has to do with the nightly log rotation (e.g. an Apache reload). Maybe you have vhosts conflicting with the z-push one?

AnotherAndy

Hi @kitserve ,
I understand that u’re really frustrated.
I can offer you to test it with windows mail or an android mail client as reference - if you send me a test-account.
Your z-push Config looks similar to mine - its just the default config from the repo.
best regards