kopano-backup: email headers incomplete after restore

fbartels

just gave this a quick try and for me this is in the mails I have checked 1:1 identical.

Just to be clear you are also talking about this here?
alt text

genesis74

yes, these are the email headers I am talking about.

Ok, but now I discovered something really strange.

I am using Thunderbird as an email client in parallel with WebApp. Whenever dealing with email headers and stuff, I prefer the old-school email client which can show the email source code. So that is the place where I discovered that parts of the email headers were missing. Since emails which I freshly received after the restore date had complete headers in Thunderbird I didn’t even think about that this might be a Thunderbird/IMAP thing.

But as I saw your screenshot I just had WebApp open, so I opened one of the restored emails in WebApp - surprisingly, all headers are there! Now, the situation is:

All emails that were restored from kopano-backup have headers partly missing if downloaded with Thunderbird via IMAP
Emails received after the restore date have complete headers when downloaded with Thunderbird via IMAP
With WebApp, all emails, regardless if they were restored or just received, have complete headers

Just deleted the email account in Thunderbird and set it up from scratch, so it downloaded the whole INBOX via IMAP again. Still the same: restored emails missing those special X-something-headers and DKIM-Signatures, emails received thereafter are fine.

So this could also be a kopano-gateway issue even though it has to be connected to kopano-backup somehow.

genesis74

I tried downloading email messages with another IMAP client (Evolution). Same outcome. So I would not expect the problem to be a Thunderbird related problem but rather an issue with IMAP/kopano-gateway.

fbartels

@genesis74 yes, I had this posting already open for some time, but did not yet find time to reply. You did not mention before that you were using imap ;-)

zarafa 6.40 introduces a new gateway which brought the possibility to store the original rfc message on disk (as an attachment) to gain performance when delivering these to clients. if these messages are not on disk the gateway will generate these out of the mapi data. when using kopano-backup we are not backing up (and therefore also not restoring) these rfc message.

So its not connected to the imap client, but is a general imap thing (connected with the fact that you restored your data from kopano-backup).

genesis74

Hi @fbartels,

thank you for that information. It brings light into this. Sorry I didn’t mention IMAP before - a good example why it is hard to keep description of issues short while not skipping information that might be relevant.

However, what I am still wondering about: If WebApp is able to show the complete headers out of the MAPI data, why doesn’t kopano-gateway deliver those headers via IMAP? Since the rfc messages are not present after the restore as an attachment it has to re-collect the information anyway (as does WebApp) … Maybe there should be a way to rebuild the “original” attachment-message during kopano-backup restore or maybe afterwards in the background/with a script.

fbartels

@genesis74 well there is https://stash.kopano.io/projects/KC/repos/kopanocore/browse/tools/python-scripts/optimize-imap.py.

@genesis74 said in kopano-backup: email headers incomplete after restore:

why doesn’t kopano-gateway deliver those headers via IMAP?

probably because no one thought it would be important.

genesis74

@fbartels This sounded like the perfect solution to me. But using the script results in the following:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/property_.py", line 175, in create_prop
    mapiobj.SetProps([SPropValue(proptag2, value)])
  File "/usr/lib/python3/dist-packages/MAPICore.py", line 404, in SetProps
    return _MAPICore.IMAPIProp_SetProps(self, cValues)
TypeError: expected bytes, str found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/doc/kopano-gateway/optimize-imap.py", line 56, in <module>
    main()
  File "/usr/share/doc/kopano-gateway/optimize-imap.py", line 52, in main
    generate_imap_message(item)
  File "/usr/share/doc/kopano-gateway/optimize-imap.py", line 25, in generate_imap_message
    item.create_prop(PR_EC_IMAP_BODYSTRUCTURE, bodystructure)
  File "/usr/lib/python3/dist-packages/kopano/properties.py", line 61, in create_prop
    return _prop.create_prop(self, self.mapiobj, proptag, value, proptype)
  File "/usr/lib/python3/dist-packages/kopano/property_.py", line 178, in create_prop
    raise Error('Could not create property, type and value did not match')
kopano.errors.Error: Could not create property, type and value did not match

According to the KC Administrator Manual (did some reverse searching on the script’s name) the script is called just with the name of the user. Am I missing any prerequisites not mentioned in the manual? I called the script just with the username as parameter.

fbartels

@genesis74 no, that is simply needs some updating for python3. If you call it through python2 it will run like expect.

genesis74

@fbartels hm … no luck with python2 as well:

Traceback (most recent call last):
  File "optimize-imap.py", line 7, in <module>
    from MAPI.Tags import PR_EC_IMAP_EMAIL_SIZE, PR_EC_IMAP_BODYSTRUCTURE, PR_EC_IMAP_BODY, PR_EC_IMAP_EMAIL, PT_STRING8
ImportError: No module named MAPI.Tags

Seems to be related to this KC-1149. Working after installing python-kopano.

fbartels

Hi @genesis74 ,

I have created https://jira.kopano.io/browse/KC-1406 to make the script compatible with python3.

But reading your reply you have now been able to execute it with python2, correct?

genesis74

Hi @fbartels,

yes, after installing python-kopano package I was able to run the script. After that and redownloading all email messages via IMAP the headers of about 1000 (the exact number of 1037 if they were processed in order of date - I didn’t check every single one of them ;-) ) of nearly 3000 emails in the inbox of one account were reconstructed (from the IMAP point of view). The rest of the emails in inbox as well as all emails in other folders look like before running the script (headers missing). Those reconstructed emails now seem to have all headers in them. MIME-Headers still different from the originals. I could not find any information in the logs.

Running the script again does not change anything.