single mailbox backup / export

micro

Hi all,

is it possible to export a single user mailbox from site1 running Kopano version X and import it on a completely different site2 running Kopano version Y ?

eg:

kopano-admin --details johndoe

output the mailbox data of “John Doe” and its
Current store size: 500MB

I’d like somehow to export this mailbox and either import it in site2 or “attach” that data into an existing user account on site2, so I can view/use it.

How can this be done?

externa1

@micro
yes you can do that with kopano-backup.

see: man kopano-backup and kopano-backup –help

rg
Christian

dw2412

@micro said in single mailbox backup / export:

is it possible to export a single user mailbox from site1 running Kopano version X and import it on a completely different site2 running Kopano version Y ?
I’d like somehow to export this mailbox and either import it in site2 or “attach” that data into an existing user account on site2, so I can view/use it.

Depending what kopano version x and kopano version y means it should be possible to achive what you want by kopano-backup

How can this be done?

Just use kopano-backup and take an empty backup directory. Do a tar of all backed up data after the backup finished and copy it to site2 for restore.

Even if site1 and site2 are on completely different versions you could try to attach to site2 by using http/https and use the kopano backup from site 1 for the restore. Futhermore you could create some linux container in site2 and install just alle necessary packages to have kopano-backup running for restoring the data from site1.

Unsupported and not recommended Option: Use Outlook and Zarafa Client, do a PST Export and copy the PST to site2. There you could use kopano PST Import tool or Outlook and Zarafa Client to restore it.

Andreas

micro

Thank you all for the information. I did try the backup with

kopano-backup -u alice -O /foo/export

and I got 19 such lines as output:

2020-02-05 17:14:00,343 - backup - WARNING - no data found for attachment of item with entryid 00000000FA9A274B76D94FDE854730AC7541FC060100000005000000BAE448D39A3C4402A1CFF155384B689F00000000

for some reason there seems some attachments missing. However, I ran a

du -sh /foo/export/alice

which outputs me 980MB and …

compared to

kopano-admin --details alice

with 1150MB there seems missing attachments in size of about 170MB ? does that kopano-backup command only export the messages of this mailbox without attachments? my kopano configuration is holding the attachments under filesystem (in /var/lib/kopano/attachments/*)

And one additional question --> does this backup method include also the softdelete items ?

dw2412

@micro

seems to me that you have some issue with lost attachment. I guess you will not be able to recover these attachment.

Differences of 170MB are not only because of missing attachments. Maybe you have IMAP Feature enabled? This means for performance reasons additional Props being stored in database… Have you ever had Filesystem Issues on the machine running as site1?

You could recalc the store size of the user at origin site with “kopano-cli --reset-folder-count -u alice”

Andreas

micro

no known issues yet, no. Neither with Kopano nor in filesystem. What exactly does the proposed command do? reset-folder sounds kinda dangerous to me :P

dw2412

@micro

what about IMAP/Pop3 - have you got enabled any of them?

about reset-folder-count - it will actually not do what I expected from it, but it will not do bad things. Simply resetting the folder unread count. I thought reset folder count is what I’ve searched for when recalculating the store size of a user…

micro

no, not using IMAP/POP at all.

dw2412

That you don’t use it does not need to mean that you did not have it enabled ;-) Please have a look for disabled_features in server.cfg. Additionally you could have it disabled/enabled on per user base. Just have a look for alice by “kopano-admin --details alice” PR_EC_ENABLED_FEATURES or “kopano-cli -u alice” for Features.

Another reason for disappeared attachments… Did you try to do some hardlink or have you got a virus scanner running?

In general about attachments, you say that you have 19 lines that would actually mean an average of below 10 MB per File - not much if you ask me. Some users still intend to use a mailing system for file sharing ;-) Just kidding but 10 MB would be equal 3 high resolution pictures with 24 color bit depth taken by mobile phone camera…

micro

sorry, I meant I have disabled it:

/etc/kopano/server.cfg
[...]
disabled_features = imap pop3

but you’re right. This particular user seems to have IMAP enabled. I disabled it and retried the kopano-backup run. This resulted in the same output unfortunately, no change.

any other clues?

dw2412

@micro

If the IMAP/POP3 additonal attributes got once created, they are there… But this is just a bit of the 170 MB difference and depending much on the store size in Database. I would guess at worst it will be 20 MB. But still you have 150 MB missing in attachments - this is like I said actually possible just with 19 files and file size average will be 7-8MB. So more or less nothing a few pictures and a big powerpoint presentation could be what is missing.

Have you got a virus scanner running or have you tried some hardlinking of files? Maybe virus scanner found something and simply removed the attachment file. This would explain why they’re marked missing during backup.

Well and another little thing Kopano compresses attachments in filesystem. I’m not so sure if the backup has them compressed too or if they get compressed higher into the backup or with less compression rate. May be this would be some more or less megabytes parts in the difference that you have, too and the overall missing attachment size less depending on how compression is being used and where.

Andreas

micro

Hallo Andreas,

viele Dank. There’s running clamav on the postfix on this host. No idea but I will try to restore this data now another machine and see testwise if and which data/attachments I can access. Thank you so far for your assistance. Have a great day

micro

I did the restore on the other host.

$ kopano-backup --restore alice -u bob --recursive

2020-02-06 11:04:39,315 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B1D512D000000
2020-02-06 11:04:39,315 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B40CB2F000000
2020-02-06 11:04:39,316 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B426F32000000
2020-02-06 11:04:39,316 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B537745000000
2020-02-06 11:04:39,316 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B7FCE2B000000
2020-02-06 11:04:39,317 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B8B6432000000
2020-02-06 11:04:39,318 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BA79626000000
2020-02-06 11:04:39,318 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B148C35000000
2020-02-06 11:04:39,319 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B6B6842000000
[...]
2020-02-06 11:04:39,319 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B8A2E1E000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BB23726000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BB96849000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BC0A329000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BDD3C1E000000
2020-02-06 11:04:42,560 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BFFDC2A000000
2020-02-06 11:04:42,847 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BFFDC2A000000
2020-02-06 11:04:46,677 - backup - ERROR - Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/user.py", line 92, in __init__
    self._name, MAPI_UNICODE), MAPI_UNICODE)
  File "/usr/lib/python3/dist-packages/MAPICore.py", line 2169, in ResolveUserName
    return _MAPICore.IECServiceAdmin_ResolveUserName(self, lpszUserName, ulFlags)
MAPI.Struct.MAPIErrorNotFound: MAPI error 8004010F (MAPI_E_NOT_FOUND)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/permission.py", line 74, in _permissions_loads
    entryid = server.user(value).userid
  File "/usr/lib/python3/dist-packages/kopano/server.py", line 450, in user
    return _user.User(name, email=email, server=self, userid=userid)
  File "/usr/lib/python3/dist-packages/kopano/user.py", line 95, in __init__
    raise NotFoundError("no such user: '%s'" % self.name)
kopano.errors.NotFoundError: no such user: 'charlie'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/log.py", line 106, in log_exc
    try: yield
  File "/usr/lib/python3/dist-packages/kopano/permission.py", line 79, in _permissions_loads
    except kopano.NotFoundError:
NameError: name 'kopano' is not defined

when I execute on host2 “kopano-admin --details bob” after that restore, the store size shown is only 10MB lower than the output on host1 “kopano-admin --details alice”. There were lot of error lines like the shown “skipping duplicate item with sourcekey…” I also mentioned the error line "no such user: ‘charlie’ and don’t understand where this comes from. On host1 where I ran the backup command there is indeed a user called “charlie”. But what relation does “charlie” have with the backup I did of user “alice” and restored on host2 into user account “bob” ? hmmm

externa1

@micro
I think user charlie has some share rights on alice mailbox or vice versa - so this can be ignored if this user is not on the other site

rg

Christian

micro

its not on site2 so I don’t really care. But there were no shared mailbox attached on site1 when I ran the backup, weird!

dw2412

@micro

if size difference is less than 10MB my bet is that there are really some attachments missing (broken / removed by virus scanner after mail was delivered to user by dagent) .

The message about missing user might be because you have Invitiations / Delegated Tasks or similar where another user is the organizer or is being invited. This just means that simply the linkaged between both is broken because user is simply not there.

Andreas

externa1

@micro
this has not to be a shared mailbox, each individual user can share there whole mailbox or parts of it like calendar, inbox, contacts by themselve

rg
Christian

externa1

@micro said in single mailbox backup / export:

“skipping duplicate item with sourcekey…”

You can check in the backup data what items these are
kopano-backup --index /pathtobackupdata/user > user.csv

this is also needed/used to restore single items for a user

@micro said in single mailbox backup / export:

kopano-backup --restore alice -u bob --recursive

the --recursive option you do not need on whole mailbox restore - this is only for restoring named folders to restore the subfolders in that folders too - on full restore everything gets restored (mails, calendar, contacts, task, etc)

rg
Christian

micro

@externa1 said in single mailbox backup / export:

kopano-backup --index /pathtobackupdata/user > user.csv

Hi Christian and many thanks for the explanation and command examples. I just checked it on site2 with your suggested command. Then I compared to the error lines I received while the restore task.

Well, some of those “…duplicate item skipping…” ids are:

messages from postfix notification (clamav) with subject = “UNCHECKED contents in mail FROM LOCAL [127.0.0.1]:43022 <some.user@example.tld”
messages from postfix notification (clamav) with subject = "VIRUS (Doc.Dropper.Agent-5932512-0) in mail FROM LOCAL [127.0.0.1]:48061 info@example.tld
messages from pflogsumm (a postfix reporting tool executed on daily basis) with subject = “Cron root@myhost /usr/sbin/pflogsumm -d yesterday /var/log/mail.log --problems-first --rej-add-from --verbose-msg-detail -q”
but also tons of ordinary messages which I see in the restored mailbox on siteB although the error was shown. I did make some tests on about 5 of those messages. They are all on hostB.

as long as there are only a few megabytes missing, I’m feeling good :-)