Mails with wrong content

mpetersen

Hi,
I made a fresh install of kopano 8.7.82 on Debian 10.2 with mariadb 10.4, only using dagent, spooler, search and server component, webapp is running on a different system. Then I made a full restore with kopano-backup on this new system with data from another system running kopano 8.7.80. The restore seems to be ok (no error messages). Attachements are stored outside in the file system and I copied them with rsync.
Now I have to problems:
1.) Running kopano-backup on the new system brings a lot of messages like "[error ] K-1562: cannot open attachment “/var/lib/kopano/attachments/x/x/x”: No such file or directory
2.) Sometimes a wrong content will be shown in folder “sent mails” when a new mail is created by deskapp or webmail. The mail will be shown with the right content on recipient site, but in “sent mails” the content is filled with the content of any old mail even from another kopano user ! Very, very strange. It also happened that a new mail will sent to the recipient with the attachements of another old mail (already a very precarious position …)

I switched log level to debug but there are no serious messages for this behavior. I try other things, aka updating to kopano 9.0.2, cleaning caches, building new search indices, running mysqlrepair but nothing put things really right.

So my questions are:
1.) Have someone similiar experience ?
2.) Are there any tools/scripts for checking data consistency ?
3.) How can I locate the mails with the missing attachements ?

I appreciate any hint :-)

Best regards
Martin

fbartels

Hi @mpetersen,

You mention deskapp/webapp once, but am I correct in guessing you are using an imap client?

It sounds a bit like you restored your attachment over already existing ones, maybe even ones that are deduplicated with hardlinks?

RFC mails displayed by imap clients are stored as attachments on disk and scrambling them on disk will lead to the observed behaviour.

thctlo

I suggest, verify you folder rights on /var/lib/kopano/attachments and a sub folder.
and if needed :

chmod -R 770 /var/lib/kopano/attachments 
chown -R kopano:kopano /var/lib/kopano/attachments

mpetersen

Hi @fbartels,

thank you for your fast answer.
In fact I used an imap client (thunderbird) on my old kopano system. With the new one I only use deskapp/webmail.
Do you know how I can fix the system to make it consistent again ? What will be the right way to migrate the data from the old to the new system ?

mpetersen

@thctlo : Thanks for your hint, but the folder rights are already checked and they are ok.

fbartels

@mpetersen said in Mails with wrong content:

In fact I used an imap client (thunderbird) on my old kopano system. With the new one I only use deskapp/webmail.

Do you see this switched content then also in WebApp?

@mpetersen said in Mails with wrong content:

Do you know how I can fix the system to make it consistent again ?

The files_v2 backend will make it harder for admins to create this mess in the future, but there is no migration path between the two backend types.

The cleanest and most complete migration path between two Kopano systems is always doing a mysql dump and copying /var/lib/kopano. Restoring with kopano-backup should work as well, though. Like already written before you should just not do a restore over existing data, especially when you have used hardlink in the past.

mpetersen

Yes, I see it also in WebApp.

So, when I will do it with a kopano-backup restore, the steps will be:

1.) make sure /var/lib/kopano exists and is empty at the new system
2.) user stores on the new system must be empty
2.) run kopano-backup restore on the new system
3.) run rsync -a from old kopano system to the new one

Is this ok so ? How can I delete existing user stores on the new system, with kopano-cli --remove store ?
Because I already use the new system, what will be the right way to get the actual state again: Cleaning all as mention above, then run a kopano-backup restore with the last backup from the old system, then a kopano-backup restore from the new system (skipping the existent entries). Or is it just enough to clean all and run a kopano-backup restore with the complete (scrambled ?) data from the new kopano system ?

In the moment I feel a little confused … What is the right time to do the rsync of the attachment folder: Before or after the kopano-backup --restore ??

fbartels

@mpetersen said in Mails with wrong content:

3.) run rsync -a from old kopano system to the new one

no, exactly not this one step. Kopano-backup already includes everything from your user. you just have to be aware that it will not be an identical clone, but merely a copy (since data needs to be serialised in and out, also creation timestamps will be of the time of the restore).

@mpetersen said in Mails with wrong content:

Because I already use the new system

I would discard all backups done after your restore and start with the machine from scratch.

mpetersen

No rsync ? On the old kopano system the attachments were already stored in the file system and if I understand aright from documentation the kopano-backup command only save the mails, but not the attachments. So I misunderstand this part and /var/lib/kopano must bot backup separately ?

fbartels

@mpetersen said in Mails with wrong content:

if I understand aright from documentation the kopano-backup command only save the mails, but not the attachments

No, that is indeed not the case. kopano-backup stores complete messages including their attachments.

Maybe you mixed up https://documentation.kopano.io/kopanocore_administrator_manual/backup_restore.html#brick-level-backups with the previous chapter?

mpetersen

My understanding came from https://documentation.kopano.io/kopanocore_administrator_manual/backup_restore.html#attachments-backup:
"When using the attachments storage outside the database, make sure that these attachments are also backed-up.
Some backup methods that can be used to backup the attachments:

Rsync"

When you mentioned “start from scratch” it means, that I will lost all my mails after the time of the restore from the last backup of the old system, that would be very silly …

fbartels

@mpetersen said in Mails with wrong content:

When using the attachments storage outside the database, make sure that these attachments are also backed-up.

that quote is in relation to chapter “10.2. Full database dump”, kopano-backup is explained in chapter “10.3. Brick-level backup”.

@mpetersen said in Mails with wrong content:

When you mentioned “start from scratch” it means, that I will lost all my mails after the time of the restore from the last backup of the old system, that would be very silly …

You mentioned that even newly created mails show inconsistencies. While you cannot apply a date filter when generating a backup with kopano-backup you could set one when restoring. But I am not sure if I would trust that data myself (since it was inconsistent before the backup).

mpetersen

Hi @fbartels
I started from scratch and run the restore from the last backup of the old kopano system:
/usr/sbin/kopano-backup --restore /var/backups/nfs/kvm/user -u user --recursive -l DEBUG

I got a lot of “[error ] Unable to open attachment “/var/lib/kopano/attachments/x/x/xxx{.gz}” for writing: Datei oder Verzeichnis nicht gefunden”

At the end there is no entry in /var/lib/kopano/attachments, simply nothing ! Is this ok ?

fbartels

@mpetersen said in Mails with wrong content:

At the end there is no entry in /var/lib/kopano/attachments, simply nothing ! Is this ok ?

The only way in which this was ok was when you actually have another path configured in your server.cfg…

@mpetersen said in Mails with wrong content:

Unable to open […] for writing

It sounds like the volume you are trying to write to does not permit you to write to it.

mpetersen

I gave up and switch back to the status of this morning.

The rights of the attachment folder are set correctly:
root@kopano:/var/lib/kopano/attachments# ls -al
total 48
drwxr-x— 12 kopano kopano 4096 Jan 2 21:39 .
drwxr-xr-x 11 kopano kopano 4096 Dec 28 19:51 …
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 0
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 1
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 2
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 3
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 4
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 5
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 6
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 7
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 8
drwxr-xr-x 22 kopano kopano 4096 Nov 9 2012 9
root@kopano:/var/lib/kopano/attachments#
And there are already files in there with actual date eg.
274502 4 -rw-r–r-- 1 kopano kopano 2899 Dec 31 15:03 ./6/6/14066.gz

I think I have to decide if I accept data loss from the past or some data corruption in the present. What a silly choice …

And I will switch my backup procedure from kopano-backup to lvm snapshot, which seems more reliable in my opinion.

Thanks for helping so far.

fbartels

@mpetersen said in Mails with wrong content:

The rights of the attachment folder are set correctly:

maybe your unterlying filesystem prevents writes?

@mpetersen said in Mails with wrong content:

What a silly choice …

your use of “silly” really irks me. I personally find it very silly to not test a migration and your backup strategy beforehand.

@mpetersen said in Mails with wrong content:

And I will switch my backup procedure from kopano-backup to lvm snapshot, which seems more reliable in my opinion.

You do you, but the downside of lvm snapshots is that you have to make sure that all temporary data has been written to disk before the snapshot’, else you risk creating inconsistent database records.

For a full disaster recovery I would always go for mysqldumps plus file system backups and kopano-backup to fill up any differences and be able to do partial restores for users.