kopano-backup: email headers incomplete after restore

genesis74

Hi there,

in order to solve a suspected problem with my kopano database (see this post) I decided to make a backup of the affected user and restore the backup into a new user store using kopano-backup. During Backup I received this warning (entryid altered):

2019-02-01 16:33:44,852 - backup - WARNING - no data found for attachment of item with entryid 00000000FxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxD62AB471A74C00000000

During restore, kopano-backup threw this error:

2019-02-01 12:24:00,272 - backup - ERROR - Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/log.py", line 103, in log_exc
    try: yield
  File "/usr/lib/python3/dist-packages/kopano_backup/__init__.py", line 743, in restore_folder
    item = folder.create_item(loads=data, attachments=not self.options.skip_attachments)
  File "/usr/lib/python3/dist-packages/kopano/folder.py", line 483, in create_item
    item = _item.Item(self, eml=eml, ics=ics, vcf=vcf, load=load, loads=loads, attachments=attachments, create=True, save=save)
  File "/usr/lib/python3/dist-packages/kopano/item.py", line 206, in __init__
    self.loads(loads, attachments=attachments)
  File "/usr/lib/python3/dist-packages/kopano/item.py", line 1193, in loads
    self._load(_pickle_loads(s), attachments)
  File "/usr/lib/python3/dist-packages/kopano/item.py", line 1183, in _load
    stream.Write(data)
  File "/usr/lib/python3/dist-packages/MAPICore.py", line 229, in Write
    return _MAPICore.ISequentialStream_Write(self, pv)
TypeError: in method 'ISequentialStream_Write', argument 2 of type 'void const *'

2019-02-01 12:26:15,963 - backup - WARNING - could not resolve rule target: missing argument to identify folder

Anyway, restore continued. After restore I checked the number of folders and of the items in all folders - it looks like all items were restored. I also did some random tests regarding email attachments without any findings. The only obvious drawback is that notes in the Notes-Folder have lost their date of creation (it has been set to the time of restore).

But as I looked at the source codes of the restored emails I noticed there were some major changes: all email headers starting with an X (i.e. X-Spam-Flag, which is added by my email provider to mark spam mails) are gone as well as DKIM-Signatures. Also, headers of MIME-parts look different:

Before:

--Apple-Mail-23xxxA7C-B957-4877-859B-F419xxxxx372
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

After:

--=_ZG_static
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

I do not expect major problems to arise out of this issue, but when using a tool like kopano-backup declared to be capable of brick-level restore I would not have expected such kind of changes in the restored data. Infact, even though only the headers are affected, in the end the email content is being altered during this process.

Is this behaviour of kopano-backup intentional? Is there any way around it?

Thanks in advance for any help.

Kopano Core 8.7.0
Ubuntu 18.04.1

fbartels

Hi @genesis74,

When kopano-backup is used mails will be serialised out and back in on restore, therefore the tickets will not be exact 1:1 copies of their former self. While I have not yet check if that has influence on the mail header (all the things you mentioned are stored in the header), but that could indeed be a possibility.

genesis74

Hi @fbartels,

Kopano seems to store email headers separately in its database so it possibly kopano-backup does not collect all the datasets which comprise the whole email (including the header) during backup. I am sorry to say, but I was not able to understand your last sentence - what could be indeed a possibility? Do you think this is worth to have a closer look at?

fbartels

Hi @genesis74 ,

I was sitting in a train musing about your issue. What I meant to say is exactly what you wrote yourself. Mails consist of multiple value internally (since we store everything as/through mapi). So it is possible that the headers will be different after restore (but again was just thinking about it did not do tests for this theory).

genesis74

Hi @fbartels,

ok, that part seems to be clear then. Anyway, I see no problem in rearranging the email headers on restore. But still wondering why kopano-backup does not backup all headers and why it is altering the MIME headers. It is just stripping all “X-something” and “DKIM-Signature: something” headers during backup (not during restore I assume after looking at the backup files) from all emails.

fbartels

Hi @genesis74 ,

just gave this a quick try and for me this is in the mails I have checked 1:1 identical.

Just to be clear you are also talking about this here?
alt text

genesis74

Hi @fbartels,

yes, these are the email headers I am talking about.

Ok, but now I discovered something really strange.

I am using Thunderbird as an email client in parallel with WebApp. Whenever dealing with email headers and stuff, I prefer the old-school email client which can show the email source code. So that is the place where I discovered that parts of the email headers were missing. Since emails which I freshly received after the restore date had complete headers in Thunderbird I didn’t even think about that this might be a Thunderbird/IMAP thing.

But as I saw your screenshot I just had WebApp open, so I opened one of the restored emails in WebApp - surprisingly, all headers are there! Now, the situation is:

All emails that were restored from kopano-backup have headers partly missing if downloaded with Thunderbird via IMAP
Emails received after the restore date have complete headers when downloaded with Thunderbird via IMAP
With WebApp, all emails, regardless if they were restored or just received, have complete headers

Just deleted the email account in Thunderbird and set it up from scratch, so it downloaded the whole INBOX via IMAP again. Still the same: restored emails missing those special X-something-headers and DKIM-Signatures, emails received thereafter are fine.

So this could also be a kopano-gateway issue even though it has to be connected to kopano-backup somehow.

genesis74

I tried downloading email messages with another IMAP client (Evolution). Same outcome. So I would not expect the problem to be a Thunderbird related problem but rather an issue with IMAP/kopano-gateway.

fbartels

@genesis74 yes, I had this posting already open for some time, but did not yet find time to reply. You did not mention before that you were using imap ;-)

zarafa 6.40 introduces a new gateway which brought the possibility to store the original rfc message on disk (as an attachment) to gain performance when delivering these to clients. if these messages are not on disk the gateway will generate these out of the mapi data. when using kopano-backup we are not backing up (and therefore also not restoring) these rfc message.

So its not connected to the imap client, but is a general imap thing (connected with the fact that you restored your data from kopano-backup).

genesis74

Hi @fbartels,

thank you for that information. It brings light into this. Sorry I didn’t mention IMAP before - a good example why it is hard to keep description of issues short while not skipping information that might be relevant.

However, what I am still wondering about: If WebApp is able to show the complete headers out of the MAPI data, why doesn’t kopano-gateway deliver those headers via IMAP? Since the rfc messages are not present after the restore as an attachment it has to re-collect the information anyway (as does WebApp) … Maybe there should be a way to rebuild the “original” attachment-message during kopano-backup restore or maybe afterwards in the background/with a script.

fbartels

@genesis74 well there is https://stash.kopano.io/projects/KC/repos/kopanocore/browse/tools/python-scripts/optimize-imap.py.

@genesis74 said in kopano-backup: email headers incomplete after restore:

why doesn’t kopano-gateway deliver those headers via IMAP?

probably because no one thought it would be important.

genesis74

@fbartels This sounded like the perfect solution to me. But using the script results in the following:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/property_.py", line 175, in create_prop
    mapiobj.SetProps([SPropValue(proptag2, value)])
  File "/usr/lib/python3/dist-packages/MAPICore.py", line 404, in SetProps
    return _MAPICore.IMAPIProp_SetProps(self, cValues)
TypeError: expected bytes, str found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/doc/kopano-gateway/optimize-imap.py", line 56, in <module>
    main()
  File "/usr/share/doc/kopano-gateway/optimize-imap.py", line 52, in main
    generate_imap_message(item)
  File "/usr/share/doc/kopano-gateway/optimize-imap.py", line 25, in generate_imap_message
    item.create_prop(PR_EC_IMAP_BODYSTRUCTURE, bodystructure)
  File "/usr/lib/python3/dist-packages/kopano/properties.py", line 61, in create_prop
    return _prop.create_prop(self, self.mapiobj, proptag, value, proptype)
  File "/usr/lib/python3/dist-packages/kopano/property_.py", line 178, in create_prop
    raise Error('Could not create property, type and value did not match')
kopano.errors.Error: Could not create property, type and value did not match

According to the KC Administrator Manual (did some reverse searching on the script’s name) the script is called just with the name of the user. Am I missing any prerequisites not mentioned in the manual? I called the script just with the username as parameter.

fbartels

@genesis74 no, that is simply needs some updating for python3. If you call it through python2 it will run like expect.

genesis74

@fbartels hm … no luck with python2 as well:

Traceback (most recent call last):
  File "optimize-imap.py", line 7, in <module>
    from MAPI.Tags import PR_EC_IMAP_EMAIL_SIZE, PR_EC_IMAP_BODYSTRUCTURE, PR_EC_IMAP_BODY, PR_EC_IMAP_EMAIL, PT_STRING8
ImportError: No module named MAPI.Tags

Seems to be related to this KC-1149. Working after installing python-kopano.

fbartels

Hi @genesis74 ,

I have created https://jira.kopano.io/browse/KC-1406 to make the script compatible with python3.

But reading your reply you have now been able to execute it with python2, correct?

genesis74

Hi @fbartels,

yes, after installing python-kopano package I was able to run the script. After that and redownloading all email messages via IMAP the headers of about 1000 (the exact number of 1037 if they were processed in order of date - I didn’t check every single one of them ;-) ) of nearly 3000 emails in the inbox of one account were reconstructed (from the IMAP point of view). The rest of the emails in inbox as well as all emails in other folders look like before running the script (headers missing). Those reconstructed emails now seem to have all headers in them. MIME-Headers still different from the originals. I could not find any information in the logs.

Running the script again does not change anything.