single mailbox backup / export

micro

no, not using IMAP/POP at all.

dw2412

That you don’t use it does not need to mean that you did not have it enabled ;-) Please have a look for disabled_features in server.cfg. Additionally you could have it disabled/enabled on per user base. Just have a look for alice by “kopano-admin --details alice” PR_EC_ENABLED_FEATURES or “kopano-cli -u alice” for Features.

Another reason for disappeared attachments… Did you try to do some hardlink or have you got a virus scanner running?

In general about attachments, you say that you have 19 lines that would actually mean an average of below 10 MB per File - not much if you ask me. Some users still intend to use a mailing system for file sharing ;-) Just kidding but 10 MB would be equal 3 high resolution pictures with 24 color bit depth taken by mobile phone camera…

micro

sorry, I meant I have disabled it:

/etc/kopano/server.cfg
[...]
disabled_features = imap pop3

but you’re right. This particular user seems to have IMAP enabled. I disabled it and retried the kopano-backup run. This resulted in the same output unfortunately, no change.

any other clues?

dw2412

@micro

If the IMAP/POP3 additonal attributes got once created, they are there… But this is just a bit of the 170 MB difference and depending much on the store size in Database. I would guess at worst it will be 20 MB. But still you have 150 MB missing in attachments - this is like I said actually possible just with 19 files and file size average will be 7-8MB. So more or less nothing a few pictures and a big powerpoint presentation could be what is missing.

Have you got a virus scanner running or have you tried some hardlinking of files? Maybe virus scanner found something and simply removed the attachment file. This would explain why they’re marked missing during backup.

Well and another little thing Kopano compresses attachments in filesystem. I’m not so sure if the backup has them compressed too or if they get compressed higher into the backup or with less compression rate. May be this would be some more or less megabytes parts in the difference that you have, too and the overall missing attachment size less depending on how compression is being used and where.

Andreas

micro

Hallo Andreas,

viele Dank. There’s running clamav on the postfix on this host. No idea but I will try to restore this data now another machine and see testwise if and which data/attachments I can access. Thank you so far for your assistance. Have a great day

micro

I did the restore on the other host.

$ kopano-backup --restore alice -u bob --recursive

2020-02-06 11:04:39,315 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B1D512D000000
2020-02-06 11:04:39,315 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B40CB2F000000
2020-02-06 11:04:39,316 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B426F32000000
2020-02-06 11:04:39,316 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B537745000000
2020-02-06 11:04:39,316 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B7FCE2B000000
2020-02-06 11:04:39,317 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B8B6432000000
2020-02-06 11:04:39,318 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BA79626000000
2020-02-06 11:04:39,318 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B148C35000000
2020-02-06 11:04:39,319 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B6B6842000000
[...]
2020-02-06 11:04:39,319 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512B8A2E1E000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BB23726000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BB96849000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BC0A329000000
2020-02-06 11:04:39,320 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BDD3C1E000000
2020-02-06 11:04:42,560 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BFFDC2A000000
2020-02-06 11:04:42,847 - backup - WARNING - skipping duplicate item with sourcekey AF6C7BE951E8481AA4970BC328C3512BFFDC2A000000
2020-02-06 11:04:46,677 - backup - ERROR - Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/user.py", line 92, in __init__
    self._name, MAPI_UNICODE), MAPI_UNICODE)
  File "/usr/lib/python3/dist-packages/MAPICore.py", line 2169, in ResolveUserName
    return _MAPICore.IECServiceAdmin_ResolveUserName(self, lpszUserName, ulFlags)
MAPI.Struct.MAPIErrorNotFound: MAPI error 8004010F (MAPI_E_NOT_FOUND)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/permission.py", line 74, in _permissions_loads
    entryid = server.user(value).userid
  File "/usr/lib/python3/dist-packages/kopano/server.py", line 450, in user
    return _user.User(name, email=email, server=self, userid=userid)
  File "/usr/lib/python3/dist-packages/kopano/user.py", line 95, in __init__
    raise NotFoundError("no such user: '%s'" % self.name)
kopano.errors.NotFoundError: no such user: 'charlie'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kopano/log.py", line 106, in log_exc
    try: yield
  File "/usr/lib/python3/dist-packages/kopano/permission.py", line 79, in _permissions_loads
    except kopano.NotFoundError:
NameError: name 'kopano' is not defined

when I execute on host2 “kopano-admin --details bob” after that restore, the store size shown is only 10MB lower than the output on host1 “kopano-admin --details alice”. There were lot of error lines like the shown “skipping duplicate item with sourcekey…” I also mentioned the error line "no such user: ‘charlie’ and don’t understand where this comes from. On host1 where I ran the backup command there is indeed a user called “charlie”. But what relation does “charlie” have with the backup I did of user “alice” and restored on host2 into user account “bob” ? hmmm

externa1

@micro
I think user charlie has some share rights on alice mailbox or vice versa - so this can be ignored if this user is not on the other site

rg

Christian

micro

its not on site2 so I don’t really care. But there were no shared mailbox attached on site1 when I ran the backup, weird!

dw2412

@micro

if size difference is less than 10MB my bet is that there are really some attachments missing (broken / removed by virus scanner after mail was delivered to user by dagent) .

The message about missing user might be because you have Invitiations / Delegated Tasks or similar where another user is the organizer or is being invited. This just means that simply the linkaged between both is broken because user is simply not there.

Andreas

externa1

@micro
this has not to be a shared mailbox, each individual user can share there whole mailbox or parts of it like calendar, inbox, contacts by themselve

rg
Christian

externa1

@micro said in single mailbox backup / export:

“skipping duplicate item with sourcekey…”

You can check in the backup data what items these are
kopano-backup --index /pathtobackupdata/user > user.csv

this is also needed/used to restore single items for a user

@micro said in single mailbox backup / export:

kopano-backup --restore alice -u bob --recursive

the --recursive option you do not need on whole mailbox restore - this is only for restoring named folders to restore the subfolders in that folders too - on full restore everything gets restored (mails, calendar, contacts, task, etc)

rg
Christian

micro

@externa1 said in single mailbox backup / export:

kopano-backup --index /pathtobackupdata/user > user.csv

Hi Christian and many thanks for the explanation and command examples. I just checked it on site2 with your suggested command. Then I compared to the error lines I received while the restore task.

Well, some of those “…duplicate item skipping…” ids are:

messages from postfix notification (clamav) with subject = “UNCHECKED contents in mail FROM LOCAL [127.0.0.1]:43022 <some.user@example.tld”
messages from postfix notification (clamav) with subject = "VIRUS (Doc.Dropper.Agent-5932512-0) in mail FROM LOCAL [127.0.0.1]:48061 info@example.tld
messages from pflogsumm (a postfix reporting tool executed on daily basis) with subject = “Cron root@myhost /usr/sbin/pflogsumm -d yesterday /var/log/mail.log --problems-first --rej-add-from --verbose-msg-detail -q”
but also tons of ordinary messages which I see in the restored mailbox on siteB although the error was shown. I did make some tests on about 5 of those messages. They are all on hostB.

as long as there are only a few megabytes missing, I’m feeling good :-)