[Outlook 2013] sent items to addresses and shared contacts from public folder

baloan

I’ve successfully installed KOE (1279a62) with shared calendars on Outlook 2013. There are two challenges I observe:

in “Sent Items” email To adresses are garbled (zeus is the hostname where kopano and z-push run). This was fine when running Outlook 2013 without KOE.
Shared contacts from the public folders does not work. “Family contacts” remains empty in Outlook. Note that sharing a calendar from another user works fine.

Software stack:

Kopano 8.4.0_162, z-push-2.6.3 , Postfix 3.1.0
Platform: Ubuntu 16.04.2 LTS
Webserver: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g mod_wsgi/4.3.0 Python/2.7.12

Clients:

Microsoft Lumia 640 Dual SIM Windows Phone 10 (via z-push)
BQ Aquaris U Plus, Android 7.1.1 with TypeApp (via z-push)
Windows 10 Mail (via z-push),
Outlook 2013 (via z-push)
Kopano WebApp 3.3.0.566

Coffee_is_life

Hello @baloan,

i think the Family contacts are empty because you need to run “gab-sync.php -a sync” under “<path-to-z-push>/tools/gab-sync/”.

This will sync your public folders data from the ‘SYSTEM’-Store.

Make sure your configuration (/…/tools/gab-sync/config.php) is set to ‘SYSTEM’

Coffee_is_life

baloan

@Coffee_is_life I have run gab-sync -a sync manually, and set it up in the cron.daily and createusers.d. It has worked and made the Z-Push-KOE-GAB, i.e. the Global Adress Book folder, visible. I can see all my kopano local users in Outlook in the “Address Book for baloghs.de”.

baloan

I am using kopano_ol_extension_1279a62.tar.gz downloaded from download.kopano.io on May 4th, 2017. Have these issues been fixed in the in the github 1.3 branch?

Coffee_is_life

Im using ther same constellation with GAB and another contact folder named “IT-Contacts”
No Problems with contacts. Using KOE version 1.3.156

I think its worth trying on a test-maschine.

Coffee_is_life

Sebastian

These are two distinct issues.

We have seen the issue with the broken from address before, but last time it was related to the MAPI conversion library of KC that wasn’t able to handle this email properly. We could never reproduce or get enough data about it.
Do you have a reproducible use case? Are all emails from a certain mailbox inside your installation broken? Is it an external contact?
The issue with contacts of the public folder could very well be related to this ticket: https://jira.z-hub.io/browse/ZP-1220
This is already fixed and available in the pre-final (version 2.3.7beta0+43) and develop repositories. Could you upgrade your Z-Push to pre-final and try to close/reopen that shared folder? It should be fixed.

Cheers,
Sebastian

baloan

I can confirm z-push 2.3.7beta0+43 from http://repo.z-hub.io/z-push:/pre-final/Ubuntu_16.04/ resolves the issue with public folder contacts.

baloan

@Sebastian

Do you have a reproducible use case?

Only some To addresses in the Sent Folder are broken, some are not.

Are all emails from a certain mailbox inside your installation broken?

No, only some. Also in outbound emails with multiple recipients some are garbled, some are fine.

Is it an external contact?

Some external are broken, some are not.

Sebastian

OK, we would need such a message in a format we can reimport it and try to reproduce the case.
You would need to:

identify the sourcekey of such a message
export the message with the below python script
send us the dump(s)

To identify the message:

Put your z-push in wbxml log mode
do some action on the broken message, e.g. mark it as un/read or flag it in webapp
This generates some output in the log updating that message:

...
25/05/2017 20:10:43 [11680] [WBXML] [ol] O      <Modify>
25/05/2017 20:10:43 [11680] [WBXML] [ol] O       <ServerEntryId>
25/05/2017 20:10:43 [11680] [WBXML] [ol] O       U98828:99b867e0eac5424982dd39d507add46ba81000000000
25/05/2017 20:10:43 [11680] [WBXML] [ol] O       </ServerEntryId>
25/05/2017 20:10:43 [11680] [WBXML] [ol] O       <Data>
...

The important part here is 99b867e0eac5424982dd39d507add46ba81000000000. This is the source key of that message.

Create a short python script to export the message.
Copy the following code to a new file like export.py

import kopano
import sys

user, folder, sourcekey = sys.argv[1:]

server = kopano.Server()
user = server.user(user)

for item in user.folder(folder):
    if item.sourcekey == sourcekey:
        dump = item.dumps()
        open('dump', 'wb').write(dump)
        break

Then you execute this script with:

python export.py username Inbox 99b867e0eac5424982dd39d507add46ba81000000000

Where username is your username and Inbox is the folder that contains this message and then the previously identified sourcekey.
This should generate a dump file in the local directory containing all the data of that message.

If you have several cases (different dumps) would probably not hurt.

Thanks,
Sebastian

baloan

@Sebastian Enclosed a link to dump for the message. Also the python script should be modified

    if item.sourcekey == sourcekey.upper():

to accomodate for capitals in item.sourcekey. I found

28/05/2017 17:36:15 [31316] [WBXML] [andreas] [ol] I      <Modify>
28/05/2017 17:36:15 [31316] [WBXML] [andreas] [ol] I       <ServerEntryId>
28/05/2017 17:36:15 [31316] [WBXML] [andreas] [ol] I        Ub254d:f8803d323ad44440918be0673a3da44e02b501000000
28/05/2017 17:36:15 [31316] [WBXML] [andreas] [ol] I       </ServerEntryId>
28/05/2017 17:36:15 [31316] [WBXML] [andreas] [ol] I       <Data>

in the log while item.sourcekey was F8803D323AD44440918BE0673A3DA44E02B501000000.
In case you are curious about the binary content ;-)

Sebastian

Yeah, importing this dump into a test account of course works. I see the receiver just fine.
I guess you have set your GMX address as email of your GAB user?

The issue is certainly related to mapitoinet in combination with retrieving the user from the GAB. In my log I also see

MAPIProvider->getEmailAddressFromSearchKey(): fall back to PR_SEARCH_KEY to resolve user and get email address

This is to set the correct ActiveSync tag. If you also see this, it still doesn’t mean that it’s correct, because OL overwrites these values with the data received from the mail headers it receives, and the error is there (in the generation of these headers).

Do you still have the WBXML log when this message was synchronized to Outlook? Could you paste the complete WBXML-OUT line?

If not, you could just reimport the dump file using:

import sys
import kopano

server = kopano.Server()
user = server.user('username')

dump = open('dump', 'rb').read()
item = user.inbox.create_item(loads=dump)

print(user.inbox.count)

The message should then be synchronized again to your inbox and another WBXML-OUT is generated (please have WBXML debug enabled).

About the content: Das ist sogar in JEDER Hinsicht falsch, aber trotzdem witzig :)

baloan

@Sebastian Actually forgot to mention the WBXML log after setting the message flag in Outlook.

Sebastian

Ahm, I guess this is a different issue (ical)?

baloan

@Sebastian This one should be the right one.

Sebastian

yeah, that’s the right one but it’s not helping.

I would need the WBXML out when the message is synchronized for the first time to Outlook. The easiest to get this should be to re-import the dump into your inbox as a new email with the script above.

baloan

@Sebastian So I did

export.py andreas "Sent Items" F8803D323AD44440918BE0673A3DA44E02B501000000
root@zeus:~/issue# ll
total 168
drwxr-xr-x 2 root root   4096 May 30 18:43 ./
drwx------ 9 root root   4096 May 30 18:42 ../
-rw-r--r-- 1 root root 150861 May 30 18:43 dump
-rwxrwxr-x 1 root root    348 May 28 18:09 export.py*
-rw-r--r-- 1 root root     84 May 28 18:09 extract.sh
-rwxr-xr-x 1 root root    201 May 30 18:42 reimport.py*
root@zeus:~/issue# ./reimport.py
140

~~and then nothing happened. No update in Outlook, no activity in WBXML log.~~ Took awhile to find the messages in my Inbox sorted date descending. Find the latest log with WBXML-OUT and one more dump here .

Sebastian

So, yes… just confirmed this. The conversion from the the MAPI object back to a mime message is faulty and not able to fill in the correct from address, this is why this is happening. In the mime message the from header is already wrong. OL by default takes that value instead of the ActiveSync From (which is sent correctly).

I will discuss this with the core developers to see how we can debug this further.