[fixed] Python-kopano: Possible regression with commit c6cf914b672 in __init__.py Item::header(tagName)

umgfoin

Hi kopano-team,
this is a crosspost with Zarafa-Forum “the nightly thread for Kopano Core 8.2.0”

I discovered a problem with pyko ( >= core-8.3.0~568_4.1-RHEL_6_PHP_56-x86_64) while trying to evaluate a message-spam-tag using method Item::header('x-spam-flag'):

Although, Item::headers() correctly returns the entire header-section including a “x-spam-flag”-tag, Item::header('x-spam-flag') returns 'None'.

headers(self) in /usr/lib/python2.6/site-packages/kopano/__init__.py uses property PR_TRANSPORT_MESSAGE_HEADERS_W to retrieve the email’s transport-header section.
There seems to be a problem with self.headers().get(name), if the headers are returned as multibyte-string.

Since commit 90279dc46bf this code was moved to item.py, the below code assumes the previous state as __init__.py was not yet split up into several files.

def header(self, name):
        """ Return transport message header with given name """

        return self.headers().get(name)

def headers(self):
        """ Return transport message headers """

        try:
            message_headers = self.prop(PR_TRANSPORT_MESSAGE_HEADERS_W)
            headers = email.parser.Parser().parsestr(message_headers.value, headersonly=True)
            return headers
        except MAPIErrorNotFound:
            return {}

This is possibly a regression introduced with commit c6cf914b672
If we revert to PR_TRANSPORT_MESSAGE_HEADERS, which maps to PROP_TAG(PT_TSTRING, ...) everything works as expected.

From mapitags.h:

#define PR_TRANSPORT_MESSAGE_HEADERS PROP_TAG(PT_TSTRING, 0x007D)
#define PR_TRANSPORT_MESSAGE_HEADERS_W PROP_TAG(PT_UNICODE, 0x007D)
#define PR_TRANSPORT_MESSAGE_HEADERS_A PROP_TAG(PT_STRING8, 0x007D)

From MSDN:

“…String properties are most commonly defined as PT_TSTRING. The PT_TSTRING property type conditionally compiles to one of the other string property types, depending on whether the UNICODE macro has been defined…”

I believe the issue is Python 2.x related.
The mail/header encoding has no influence.
Verified with UTF-8, ASCII 7-and 8-Bit-encoded mails.

Best regards,
Umgfoin.

fbartels

Hello umgfoin,

to make the circle perfect I asked a developer to look into this and he replied at https://forums.zarafa.com/showthread.php?12956-the-nightly-thread-for-Kopano-Core-8-2-0&p=57837&viewfull=1#post57837

umgfoin

@fbartels

Hi Felix
…just replied to Mark.

Concerning loop-detection ;-) :
Best to report here or in the Zarafa-forum?

umgfoin

As the problem is still present in 8.4.0.111 and 8.3 beta:

Root-cause is different behaviour of Python’s email-package in 2.x and 3.x: email.Parser().parsestr(str) can’t handle unicode-str in 2.x, but PR_TRANSPORT_MESSAGE_HEADERS_W always returns unicode-encoding. An additional encode()will do the magic and convert to standard-encoding, which should keep the result untouched for python 3.x, but convert to sgl.-byte-str in 2.x

Alternativly, using the PT_TSTRING variant of PR_TRANSPORT_MESSAGE_HEADERSx might return the correct string-type dependant on platform/interpreter’s defaults - at least according to Microsoft’s intention in MAPI.

As already mentioned, the current implementation completely breaks SPAM-handling based on header-tag evaluation for CentOS6/ Python 2.x and possibly other things.

Patch:

From e605186ff44bb075ce8440badcc5846998e06e95 Mon Sep 17 00:00:00 2001
From: umgfoin <ask@me>
Date: Wed, 22 Mar 2017 18:38:56 +0100
Subject: [PATCH 1/1] swig: item.py: Fixed item::header() for python 2.6 compatibility

---
 swig/python/kopano/kopano/item.py |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/swig/python/kopano/kopano/item.py b/swig/python/kopano/kopano/item.py
index 1cdd266..d4a1fe2 100644
--- a/swig/python/kopano/kopano/item.py
+++ b/swig/python/kopano/kopano/item.py
@@ -418,7 +418,7 @@ class Item(object):

         try:
             message_headers = self.prop(PR_TRANSPORT_MESSAGE_HEADERS_W)
-            headers = email.parser.Parser().parsestr(message_headers.value, headersonly=True)
+            headers = email.parser.Parser().parsestr(message_headers.value.encode(), headersonly=True)
             return headers
         except MAPIErrorNotFound:
             return {}
--
1.7.1

++umgfoin.

fbartels

Thanks for you patch @umgfoin. I have created https://jira.kopano.io/browse/KC-606 for this and already opened a pull request.

PS: we recently added contributing instructions to our git repository, you can find them at https://stash.kopano.io/projects/KC/repos/kopanocore/browse/CONTRIBUTING.md.

umgfoin

KC-606 fixed with Commit 4715f7530bc.
Nightly build core-8.4.0~181_44.1
Thanks!