Z-Push no longer works after switch from Zarafa to Kopano and local install to packaged version

fbartels

@kitserve said in Z-Push no longer works after switch from Zarafa to Kopano and local install to packaged version:

Oh, and what do you mean by this question?

there is a config.php in /usr/share/z-push, but this is actually a symlink to /etc/z-push. Same accounts for the backend configuration.

AnotherAndy

hi @kitserve ,
have you updated your IOS Deviced to the latest OS Version 11.03?
Apple fixed in 11.02 and 11.03 a lot of “Mail Server problems” which occurred in IOS 11.

Can you try the z-push with another client like windows mail oder outlook 2013++ ?
For more infos, I recommend to set the Debug levelin /etc/z-push/ z-push.conf.php higher.

Just for your info - I am running nearly the same environment.
Debian 8, PHp 5.6, Apache 2.4., Z-Push 2.3.8 nad kopano 8.3.4.12-0+23.1. I don’t have any IOS deviced but Android and Outlook are working fine.
Best regards

kitserve

@fbartels: I edited the config files in /etc/z-push, and I can confirm that the symlinks for config.php and policies.ini in /usr/share/z-push are correctly set.

@Merlin2104: We are an exclusively Linux shop, so I don’t have an easy way of testing with Window Mail or Outlook 2013/2016. I just tried increasing the log level to LOGLEVEL_WBXMLSTACK and recreating the account on an iPad mini 2, running the latest iOS 11.0.3. It failed with the error ‘Exchange Account - Unable to verify account information’. Log details are below:

15/10/2017 11:26:50 [11603] [DEBUG] [user] [] -------- Start
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] cmd='' devType='' devId='' getUser='user' from='1.2.3.4' version='2.3.8+0-0' method='OPTIONS'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] Used timezone 'Europe/London'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] BackendKopano using PHP-MAPI version: 8.4.90 - PHP version: 5.6.30-0+deb8u1
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] Request::ProcessHeaders() ASVersion: 14.0
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] KopanoBackend->Logon(): Trying to authenticate user 'user'..
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] KopanoBackend->openMessageStore('user'): Found 'DEFAULT' store: 'Resource id #17'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] KopanoBackend->Logon(): User 'user' is authenticated
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] Store supports properties containing Unicode characters.
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] NoPostRequestException: Options request - code: 1 - file: /usr/share/z-push/index.php:66
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] ZPush::GetSupportedProtocolVersions(): 12.0,12.1,14.0
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] ZPush::GetSupportedCommands(): Sync,SendMail,SmartForward,SmartReply,GetAttachment,GetHierarchy,CreateCollection,DeleteCollection,MoveCollection,FolderSync,FolderCreate,FolderDelete,FolderUpdate,MoveItems,GetItemEstimate,MeetingResponse,ResolveRecipients,ValidateCert,Provision,Search,Ping,Notify,ItemOperations,Settings
15/10/2017 11:26:50 [11603] [ INFO] [user] [] Options request
15/10/2017 11:26:50 [11603] [ INFO] [user] [] cmd='' memory='1.67 MiB/2.00 MiB' time='0.06s' devType='' devId='' getUser='user' from='1.2.3.4' idle='0s' version='2.3.8+0-0' method='OPTIONS' httpcode='200'
15/10/2017 11:26:50 [11603] [DEBUG] [user] [] -------- End

Perhaps the NoPostRequest Exception is related?

Thanks for your help so far.

liverpoolfcfan

Are you using a LetsEncrypt certificate?

I have seen this happen on an iOS device when connecting to a z-push server configured with a LetsEncrypt certificate. It seems that Apple do not trust them by default.

I just clicked on check details, and then accept (or whatever the option was called) and then it seemed to work OK

kitserve

Thanks, good suggestion but we’re using a paid Comodo certificate, not Let’s Encrypt. The setup worked previously with Zarafa and the certificate hasn’t been changed.

Manfred

Hi kitserve,

could you nevertheless check the certificate details as liverpoolfcfan described?

Are there any errors in apache logs?

Manfred

kitserve

I think liverpoolfcfan is referring to the step in the setup where the device attempts to access the autodiscover config. Telling the device to trust the certificate doesn’t help unfortunately, immediately after selecting ‘Trust’ I get the same ‘Unable to verify account information’ error.

Out of curiosity, I just set up autodiscover with a valid certificate to see if it would make a difference, but it didn’t help. In this case I don’t see any warnings about the certificate at all, it just goes straight to the verification error.

Here’s the Apache access log:

1.2.3.4 - user@example.com [19/Oct/2017:09:51:22 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6275 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user [19/Oct/2017:09:51:22 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6275 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user [19/Oct/2017:09:51:22 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6278 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user@example.com [19/Oct/2017:09:51:23 +0100] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 200 6278 "-" "Apple-iPhone8C4/1501.432"
1.2.3.4 - user@example.com [19/Oct/2017:09:51:24 +0100] "OPTIONS /Microsoft-Server-ActiveSync HTTP/1.1" 200 5756 "-" "Apple-iPhone8C4/1501.432"

The Apache error log is empty. The Z-Push error log is empty. The main Z-Push access log contains the following, the same as before:

19/10/2017 09:51:24 [23472] [DEBUG] [user] [] -------- Start
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] cmd='' devType='' devId='' getUser='user' from='1.2.3.4' version='2.3.8+0-0' method='OPTIONS'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] Used timezone 'Europe/London'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] BackendKopano using PHP-MAPI version: 8.4.90 - PHP version: 5.6.30-0+deb8u1
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] Request::ProcessHeaders() ASVersion: 14.0
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] KopanoBackend->Logon(): Trying to authenticate user 'user'..
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] KopanoBackend->openMessageStore('user'): Found 'DEFAULT' store: 'Resource id #16'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] KopanoBackend->Logon(): User 'user' is authenticated
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] Store supports properties containing Unicode characters.
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] NoPostRequestException: Options request - code: 1 - file: /usr/share/z-push/index.php:66
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] ZPush::GetSupportedProtocolVersions(): 12.0,12.1,14.0
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] ZPush::GetSupportedCommands(): Sync,SendMail,SmartForward,SmartReply,GetAttachment,GetHierarchy,CreateCollection,DeleteCollection,MoveCollection,FolderSync,FolderCreate,FolderDelete,FolderUpdate,MoveItems,GetItemEstimate,MeetingResponse,ResolveRecipients,ValidateCert,Provision,Search,Ping,Notify,ItemOperations,Settings
19/10/2017 09:51:24 [23472] [ INFO] [user] [] Options request
19/10/2017 09:51:24 [23472] [ INFO] [user] [] cmd='' memory='1.67 MiB/2.00 MiB' time='0.06s' devType='' devId='' getUser='user' from='1.2.3.4' idle='0s' version='2.3.8+0-0' method='OPTIONS' httpcode='200'
19/10/2017 09:51:24 [23472] [DEBUG] [user] [] -------- End

Manfred

Hi kitserve,

the fact that there’s an OPTIONS request coming from the mobile device indicates that it is able to connect to the server. However I don’t have any idea why it won’t continue. It sounds silly, but have you tried turning the mobile off and on again and creating account then?

Manfred

kitserve

It’s generally worth asking the silly questions, but yes, I have tried that several times!

Manfred

Hi kitserve,

do you remember which Z-Push version was before? Have you tried to remove the device with z-push-admin and recreate account again?

Could you post the Z-Push config files of apache?

Manfred

kitserve

Sorry, I don’t remember the previous version but it was relatively recent. It was certainly upgraded within the last few months. Yes, I deleted all of my user data with z-push-admin and tried to add the device again. The Apache config contains the default Z-Push options. I just copy and pasted it back in from the official .deb to make sure we weren’t missing anything:

# Z-Push - ActiveSync over-the-air - default Apache configuration
<IfModule mod_alias.c>
    Alias /Microsoft-Server-ActiveSync /usr/share/z-push/index.php
</IfModule>

<Directory /usr/share/z-push>
    # Don't list a directory index, follow symlinks (maybe state dir is somewhere linked)
    DirectoryIndex index.php
    Options -Indexes +FollowSymLinks

    # Z-push requirements
    php_value magic_quotes_gpc off
    php_value magic_quotes_runtime off
    php_value register_globals off
    php_value short_open_tag on

    # Optional
    # php_value display_errors off

    # Setting memory limit higher (larger attachments)
    php_value memory_limit 128M
    
    # Security
    # Don't allow .htaccess Overrides, disallow access to files
    AllowOverride none
    <IfModule !mod_authz_core.c>
        Order allow,deny
        allow from all
    </IfModule>
    <IfModule mod_authz_core.c>
        Require all granted
    </IfModule> 

    <Files "config.php">
      <IfModule !mod_authz_core.c>
        Deny from All
      </IfModule>
      <IfModule mod_authz_core.c>
        Require all denied
      </IfModule>
    </Files>
</Directory>

Sebastian

I can only imagine that this is some configuration/permission issue… But as you said, Android is working so this seems not to be the case.

I have seen once that iOS completely refused to connect to the server. The only solution was there to remove the account from the device and to add it again.
You could try that (again).

Sorry, no further ideas.

kitserve

Things got even weirder. I just switched Z-Push over to a manual install in /usr/local/share/z-push using the latest tarball, and it worked with no problems. I diffed the main config and the Kopano backend config, and they are functionally identical. Totally perplexed by this. Perhaps it is some quirk of our server config but it’s hard to know what it could be. It’s frustrating that we can’t use the repo packaged version of Z-Push, but given that it’s working now and I have already sunk far too many hours into fixing it, I’m not going to spend more time on it now! Thanks for all your help.

kitserve

After testing it last night and verifying that everything was working, I got up this morning to discover that it had stopped working again! I don’t think words can express quite how frustrated I am with Z-Push at the moment!

fbartels

Hello @kitserve ,

if you have a subscription I would recommend to get into contact with our support to have someone take a closer look at your system.

Meanwhile you could also have a look at your Apache configuration. That it stopped working over night makes me think this has to do with the nightly log rotation (e.g. an Apache reload). Maybe you have vhosts conflicting with the z-push one?

AnotherAndy

Hi @kitserve ,
I understand that u’re really frustrated.
I can offer you to test it with windows mail or an android mail client as reference - if you send me a test-account.
Your z-push Config looks similar to mine - its just the default config from the repo.
best regards

kitserve

More news. I’m not going to say that it’s fixed as I don’t want to be premature, but I made an interesting discovery. If I disable PHP’s opcache then things seem to start working again. Two devices now seem to be working for over an hour. I’m going to check again tomorrow and get the other users to test before I get too excited. My guess is that something device specific is getting cached, and that’s what’s causing things to break. My inital reports each time about devices working could be due to the first connection occurring when the cache is empty. I’ll update tomorrow about whether it’s still working this time.

@Merlin2104 Thanks, I appreciate the offer. Hopefully that won’t be necessary.

@fbartels Unfortunately we don’t have a subscription, otherwise I would have contacted support already! I’m 99% certain that our Apache vhosts are fine, as we didn’t change any of them, and only one of them is configured to access Z-Push.

kitserve

Okay, I can pretty confidently say that the problem is with opcache. Both the locally installed and packaged versions of Z-Push work when opcache is disabled. For some reason I can’t get opcache.blacklist_filename to play ball, so I have to completely disable opcache to get Z-Push to work. As we’re hosting a number of other sites I’ve had to proxy Z-Push through to FPM, with opcache disabled on FPM, and leave everything else running on mod_php so that it can still use opcache. Any ideas what might be causing the problem? I see from the Z-Push documentation that opcache is supported, and I haven’t noticed it causing problems with any of our other sites or services. Here is our relevant config from php.ini:

[opcache]
opcache.enable=1
;opcache.enable_cli=0
opcache.memory_consumption=256
;opcache.interned_strings_buffer=4
;opcache.max_accelerated_files=2000
;opcache.max_wasted_percentage=5
;opcache.use_cwd=1
;opcache.validate_timestamps=1
;opcache.revalidate_freq=2
;opcache.revalidate_path=0
;opcache.save_comments=1
;opcache.load_comments=1
;opcache.fast_shutdown=0
;opcache.enable_file_override=0
;opcache.optimization_level=0xffffffff
;opcache.inherited_hack=1
;opcache.dups_fix=0
;opcache.blacklist_filename=
;opcache.max_file_size=0
;opcache.consistency_checks=0
;opcache.force_restart_timeout=180
;opcache.error_log=
;opcache.log_verbosity_level=1
;opcache.preferred_memory_model=
;opcache.protect_memory=0

As you can see, everything is at the defaults except for the memory consumption.
PHP 5.6.30+dfsg-0+deb8u1
Apache 2.4.10-10+deb8u11

Manfred

Hi kitserve,

now that you mentioned opcache I remember that there were some users who had similar issues with Z-Push and opcache. However I’ve seen other servers with opcache enabled and iOS devices syncing without issues.

What apache modules do you have enabled? Maybe one of them interferes with opcache and Z-Push?

Manfred

kitserve

Maybe! I’ve no idea which might or might not conflict. Here are the modules:

user@host:~$ ls /etc/apache2/mods-enabled/
access_compat.load  authn_core.load       autoindex.conf  dir.load      headers.load      negotiation.load  proxy_fcgi.load  reqtimeout.load     ssl.load
actions.conf        authn_file.load       autoindex.load  env.load      mime.conf         pagespeed.conf    proxy_html.conf  rewrite.load        status.conf
actions.load        authz_core.load       cgi.load        expires.load  mime.load         pagespeed.load    proxy_html.load  setenvif.conf       status.load
alias.conf          authz_groupfile.load  deflate.conf    fastcgi.conf  mpm_prefork.conf  php5.conf         proxy_http.load  setenvif.load       xml2enc.load
alias.load          authz_host.load       deflate.load    fastcgi.load  mpm_prefork.load  php5.load         proxy.load       socache_shmcb.load
auth_basic.load     authz_user.load       dir.conf        filter.load   negotiation.conf  proxy.conf        reqtimeout.conf  ssl.conf