[fixed] Python-kopano: Possible regression with commit c6cf914b672 in __init__.py Item::header(tagName)

  • Hi kopano-team,
    this is a crosspost with Zarafa-Forum “the nightly thread for Kopano Core 8.2.0”

    I discovered a problem with pyko ( >= core-8.3.0~568_4.1-RHEL_6_PHP_56-x86_64) while trying to evaluate a message-spam-tag using method Item::header('x-spam-flag'):

    Although, Item::headers() correctly returns the entire header-section including a “x-spam-flag”-tag, Item::header('x-spam-flag') returns 'None'.

    headers(self) in /usr/lib/python2.6/site-packages/kopano/__init__.py uses property PR_TRANSPORT_MESSAGE_HEADERS_W to retrieve the email’s transport-header section.
    There seems to be a problem with self.headers().get(name), if the headers are returned as multibyte-string.

    Since commit 90279dc46bf this code was moved to item.py, the below code assumes the previous state as __init__.py was not yet split up into several files.

    def header(self, name):
            """ Return transport message header with given name """
            return self.headers().get(name)
    def headers(self):
            """ Return transport message headers """
                message_headers = self.prop(PR_TRANSPORT_MESSAGE_HEADERS_W)
                headers = email.parser.Parser().parsestr(message_headers.value, headersonly=True)
                return headers
            except MAPIErrorNotFound:
                return {}

    This is possibly a regression introduced with commit c6cf914b672
    If we revert to PR_TRANSPORT_MESSAGE_HEADERS, which maps to PROP_TAG(PT_TSTRING, ...) everything works as expected.

    From mapitags.h:


    From MSDN:

    “…String properties are most commonly defined as PT_TSTRING. The PT_TSTRING property type conditionally compiles to one of the other string property types, depending on whether the UNICODE macro has been defined…”

    I believe the issue is Python 2.x related.
    The mail/header encoding has no influence.
    Verified with UTF-8, ASCII 7-and 8-Bit-encoded mails.

    Best regards,

  • Kopano

    Hello umgfoin,

    to make the circle perfect I asked a developer to look into this and he replied at https://forums.zarafa.com/showthread.php?12956-the-nightly-thread-for-Kopano-Core-8-2-0&p=57837&viewfull=1#post57837

  • @fbartels

    Hi Felix
    …just replied to Mark.

    Concerning loop-detection ;-) :
    Best to report here or in the Zarafa-forum?

  • As the problem is still present in and 8.3 beta:

    Root-cause is different behaviour of Python’s email-package in 2.x and 3.x: email.Parser().parsestr(str) can’t handle unicode-str in 2.x, but PR_TRANSPORT_MESSAGE_HEADERS_W always returns unicode-encoding. An additional encode()will do the magic and convert to standard-encoding, which should keep the result untouched for python 3.x, but convert to sgl.-byte-str in 2.x

    Alternativly, using the PT_TSTRING variant of PR_TRANSPORT_MESSAGE_HEADERSx might return the correct string-type dependant on platform/interpreter’s defaults - at least according to Microsoft’s intention in MAPI.

    As already mentioned, the current implementation completely breaks SPAM-handling based on header-tag evaluation for CentOS6/ Python 2.x and possibly other things.


    From e605186ff44bb075ce8440badcc5846998e06e95 Mon Sep 17 00:00:00 2001
    From: umgfoin <ask@me>
    Date: Wed, 22 Mar 2017 18:38:56 +0100
    Subject: [PATCH 1/1] swig: item.py: Fixed item::header() for python 2.6 compatibility
     swig/python/kopano/kopano/item.py |    2 +-
     1 files changed, 1 insertions(+), 1 deletions(-)
    diff --git a/swig/python/kopano/kopano/item.py b/swig/python/kopano/kopano/item.py
    index 1cdd266..d4a1fe2 100644
    --- a/swig/python/kopano/kopano/item.py
    +++ b/swig/python/kopano/kopano/item.py
    @@ -418,7 +418,7 @@ class Item(object):
                 message_headers = self.prop(PR_TRANSPORT_MESSAGE_HEADERS_W)
    -            headers = email.parser.Parser().parsestr(message_headers.value, headersonly=True)
    +            headers = email.parser.Parser().parsestr(message_headers.value.encode(), headersonly=True)
                 return headers
             except MAPIErrorNotFound:
                 return {}


  • Kopano

    Thanks for you patch @umgfoin. I have created https://jira.kopano.io/browse/KC-606 for this and already opened a pull request.

    PS: we recently added contributing instructions to our git repository, you can find them at https://stash.kopano.io/projects/KC/repos/kopanocore/browse/CONTRIBUTING.md.

  • KC-606 fixed with Commit 4715f7530bc.
    Nightly build core-8.4.0~181_44.1