Kopano Gateway IDLE Issue Apple Mail



  • Hi,

    I encountered an issue with Apple Mail and the Gateway with IDLE.
    I know that rfc2177 states that the client is not supposed to sent command when in idling mode but it does not specify how the server should react if it does.

    Apple Mail sometimes seems to send commands even when it is in IDLE mode.
    Here is a debug output of the Kopanoe Gateway:

    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > 7.3032 OK FETCH completed
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 8.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > + waiting for notifications
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 9.3032 EXPUNGE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > * BAD still in idle state
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 10.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > + waiting for notifications
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 11.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > * BAD still in idle state
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 12.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > + waiting for notifications
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 13.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > * BAD still in idle state
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 14.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > + waiting for notifications
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 15.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > * BAD still in idle state
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 16.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > + waiting for notifications
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] < 17.3032 IDLE
    Fri Mar 10 18:58:25 2017: [debug  ] [15997] > * BAD still in idle state
    Fri Mar 10 19:03:23 2017: [debug  ] [15997] < DONE
    Fri Mar 10 19:03:23 2017: [debug  ] [15997] >  BAD was not idling
    Fri Mar 10 19:03:23 2017: [debug  ] [15997] < DONE
    

    Also it seems like you are terminating the IDLE session when you receive a command from the client which is not DONE (IMAP.cpp line 398):

    	ToUpper(strCommand);
    	if (isIdle()) {
    		hr = HrResponse(RESP_UNTAGGED, "BAD still in idle state");
    		HrDone(false); // false for no output		
    		return hr;
    	}
    

    This though leads to a broken connection and in the end to:

    Disconnecting client of user XXXX because too many (10) erroneous commands received, last reply:

    In my oppinion the correct way of dealing with this here is to simply ignore the client command and not terminate the IDLE state by calling HrDone(false).

    This leads to an unrecoverable situation and it does happen very frequently with our Apple Mail.

    Would be happy to get some feedback from you if you agree.

    Best and thanks
    Sven


  • Kopano

    Hi Sven

    @voleatech said in Kopano Gateway IDLE Issue Apple Mail:

    Hi,
    I encountered an issue with Apple Mail and the Gateway with IDLE.
    I know that rfc2177 states that the client is not supposed to sent command when in idling mode but it does not specify how the server should react if it does.
    Apple Mail sometimes seems to send commands even when it is in IDLE mode.

    Have you tested your modification? What happens whenever the EXPUNGE command succeed? Does it assume it is still in IDLE or does it issue yet another IDLE command. I could add a config option, but I would need to know the behavior of the client before I can do that.

    By the way, I find it a bit weird nobody reported similar malfunctioning with other IMAP servers (i.e. cyrus, dovecot, courier). I look briefly at the cyrus imapd code and it seems like they are handling it the same way.

    Bo



  • @bosim said in Kopano Gateway IDLE Issue Apple Mail:

    By the way, I find it a bit weird nobody reported similar malfunctioning with other IMAP servers

    Hi, I can confirm that Apple Mail is not running perfectly smooth with kopano-gateway. It relatively often stops synchronizing. This was already the case with zarafa-gateway.
    I’ve never digged into it, but I’m regularly seeing !-marks (indicating sync-problems) in the connected folders. Re-starting the application helps in this case. Duplicated autosave-draft-messages are another issue (also not necessarily Kopano’s culprit) .

    gateway-log-messages like:

    Mon Mar 13 08:40:27 2017: [error  ] [ 1230] Disconnecting client of user umgfoin because too many (10) erroneous commands received, last reply:
    Mon Mar 13 08:40:27 2017: [error  ] [ 1230]  BAD was not idling
    

    are quite common.

    This is just thought as feedback, I’ve never investigated root-cause and responsible party for these problems.
    ++umgfoin.



  • Hi,

    we had the same symptoms than @bosim with Zarafa Gateway so I finally looked into it.

    I looked at Cyrus and Dovecot as well and they do ignore the commands.
    You can see it in Cyrus at https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/idled.c function static void process_message line 130.
    They just break out of the processing function but they do not call remove_ientry to stop the idling for the client.

    I did not recompile the gateway to test out the change.
    Can I use the master code on an official Kopano 8.2 installation or will I get a version mismatch?

    Right now Apple Mail sends another IDLE after EXPUNGE and soon after a DONE.
    All these commands are then not working because the IDLE is not on in Kopano anymore and lead to the 10 erroneous commands.

    Apple Mail also does not seem to recognize that the Gateway closed the connection and is just waiting for changes.

    Best
    Sven


  • Kopano

    Hi Sven

    @voleatech said in Kopano Gateway IDLE Issue Apple Mail:

    I did not recompile the gateway to test out the change.
    Can I use the master code on an official Kopano 8.2 installation or will I get a version mismatch?

    You can use the code from git with the 8.2.1 tag to ensure you will not get into problems. Our master is too far ahead for using it.

    https://stash.kopano.io/projects/KC/repos/kopanocore/browse?at=refs%2Ftags%2Fkopanocore-8.2.1

    So just run

    $ git clone https://stash.kopano.io/scm/kc/kopanocore.git
    $ cd kopanocore
    $ git checkout kopanocore-8.2.1
    

    Thanks for helping out!

    Bo



  • Hi Bo,

    thanks I have the code checked out.
    Since I haven’t compiled Kopano before, can I just compile the gateway executable preferably to another location so I can copy it to the bin folder for testing?

    This way I can test it in our production server without deleting the current gateway.

    I would appreciate if you have the command for me to set the correct configuration and make command.

    Best and thanks
    Sven



  • Hi,

    I migrated this week one of our customers from zimbra to kopano and i run into the same issue as described here.

    He had errors when he moves mails and Mac Mail generate a local folder under “On My Mac” with the Name “Wiederhergestellte E-Mails” which contains mails who the customer deleted. Also some of his rules in mac mail fail when the shoul’d move one mail to a subfolder. I think also of this issue when i look into the logfiles.

    Is their currently only the workaround to patch the kopano-gateway with editing the source files of this version, or is something planned for the next release of the kopano-gateway?


  • Kopano

    @Beleggrodion you sadly don’t write which version you are currently using (8.1, 8.2, 8.3, 8.4?). My college involved here is currently on holiday and I could not find an issue in our Jira that seems related to this. But I know that there have been some further exchanges on this topic out of this forum, so what is stated here may very well be outdated.



  • I installed kopano 8.2.1.531 from the supported repository. And this was also the version which was mentioned above.

    See my error message below, which looks also similar:

    Thu Apr 20 08:34:56 2017: [error  ] [22556] Disconnecting client of user user99 because too many (10) erroneous commands received, last reply:
    Thu Apr 20 08:34:56 2017: [error  ] [22556] 561.4 NO UID FETCH failed
    

  • Kopano

    @Beleggrodion I’ve sent my college an email, will update once I hear back from him.



  • Recent commit c703531d7c0 massively improved IMAP-compatibility between Kopano-Gateway and Apple-Mail:

    Verified on Sierra and ElCapitan, I observed no more timeouts or disconnects on IDLE, sofar.
    Gateway-log is clean.

    Documentation for config-option imap_ignore_command_idle should be added to gateway.cfg

    Thanks !
    ++umgfoin.


  • Kopano

    @umgfoin said in Kopano Gateway IDLE Issue Apple Mail:

    Recent commit c703531d7c0 massively improved IMAP-compatibility between Kopano-Gateway and Apple-Mail:

    Verified on Sierra and ElCapitan, I observed no more timeouts or disconnects on IDLE, sofar.
    Gateway-log is clean.

    Documentation for config-option imap_ignore_command_idle should be added to gateway.cfg

    Thanks !
    ++umgfoin.

    Glad to hear! Will add it to the documentation.

    Bo


Log in to reply
 

Looks like your connection to Kopano Community Forum was lost, please wait while we try to reconnect.