Lack of documentation on how to use kopano-backup - especially regarding the purge function

Gerald

I’m trying to understand how kopano-backup actually works… The man page does unfortunately not go into a lot of detail and there so far has been little discussion on this topic in the forum. I tried to read some comments on bugs discussing development of kopano-backup to get some understanding of what is does.

For example, the man page says regarding --deletes YESNO
“The default when backing up is to maintain data which has been soft- or hard-deleted.”

So I would expect that when backing up with “–deletes no” that it would remove deleted information from the backup directory. It does not do this however… The basic behavior of kopano-backup seems to be “never delete any item and just always add to it”… Which is of course not so perfect because if you have a 150MB mail in your inbox and then you put in in /stuff/project_a/ a few days later your backup now contains the mail twice and twice the storage space is needed!

From what I found out, the only way to get rid of these duplicates in the backup seems to be the purge option? This is however not accepted as an option while doing the backup! It seems to be a separate process to be run later?

So how do I start --purge? There is absolutely no documentation on this, so i started trying stuff:

I was only able to start it for the directories of individual backed-up users, where it seems to do something (kopano-backup --purge 7 /mnt/backup-dir/user/)
When I start it for the entire backup (kopano-backup --purge 7 /mnt/backup-dir/) it seems to do nothing at all
I could write a script that iterates through the users in /mnt/backup-dir/ but even that would not make sure that stuff that is deleted is gone, because if a user was deleted in kopano this does not seem to make kopano-backup remove the backup for this user at all - with or without purge…

kopano-backup needs more documentation, especially in the “what does it actually do?” area, the “purge” command needs to be explained, and a “some basic examples of how to use it” somewhere either in the man page or in the kopano administrator manual would be useful… And some hints like “if you delete a user in kopano, you will manually need to remove the users backup because kopano-backup will not do so on its own”.

Regarding examples:
Currently the administrators guide recommends the use of filesystem snapshots in the target directory… Which is fine but since we have snapshot anyway the policy of “never delete any mail from the backup ever” does not seem to make sense - that’s what the snapshots are for…

Perhaps there should be two ways with examples how to use kopano-backup

with “delete stuff that was deleted” for filesystem destinations that provide snapshots
one with “just keep everything forever” for filesystem destinations that do not provide snapshots?

In case someone is interested regarding how I’m using it (posting this because there are so few examples):
I’m currently using crontab to run

kopano-backup --skip-deleted --worker-processes 1 --output-dir /mnt/btrfsBackup/kopano-backup/

3 days a week and afterwards rotate the btrfs snapshot using my own custom magic :relaxed: (which keeps and removes snapshots to satisfy my retention goals).

I had crashes with more than 1 worker-process but that was quite some time ago and may very well have been fixed. But incremental backups are fast anyway. This seems to work well, except for the missing purge of no longer existing items…

Gerald

I’m still wondering how kopano-backup is actually supposed to work. I’ve just looked at a couple of backup folders which were created during the past months by a weekly run of

/usr/sbin/kopano-backup --deletes no --skip-deleted --worker-processes 1 --output-dir /mnt/path/kopano-backup-dir/

The resulting backup directory contains 37GB of files.

When I make a completely new backup into a fresh directory, the backup is only 29GB big.
Which made me wonder why the backup that grew over time is bigger even though I specified “–deletes no”?

So I ran

 /usr/sbin/kopano-backup --purge 1  /mnt/btrfsBackup/test-old/
 /usr/sbin/kopano-backup --purge 0  /mnt/btrfsBackup/test-old/

which did not change anything. The “legacy” backup is always 8GB bigger than the freshly created backup and there seems to be no way to get those two back in sync except deleting the entire thing.

The new kopano-backup is really weird :confused:

fbartels

Hi @gerald ,

maybe you should run your commands with -l debug to see what the program is executing.

@gerald said in Lack of documentation on how to use kopano-backup - especially regarding the purge function:

if a user was deleted in kopano this does not seem to make kopano-backup remove the backup for this user at all - with or without purge…

from my experience admins want to keep the last backup set of a deleted user. just imagine the following:

you accidentally delete a user
run kopano-backup
and only then realise that you accidentally deleted the user and want to restore him
-> you could not do this if kopano-backup would purge backups of non existing users.

metagmbh

I would like also to know how the backup works. I experienced the same behavior as @Gerald .

The --purge function does exactly…nothing. I ran the kopano-backup on several users via

kopano-backup -u user -l debug --purge 1 /mnt/backup/user
...
2019-07-03 15:57:22,152 - backup - INFO - purged 0 folders and 0 items

and nothing got purged and I am pretty sure, that they have deleted objects (including me).

fbartels

Hello @metagmbh,

this seems to work for me:

2019-07-04 02:06:23,655 - backup - INFO - purged 4 folders and 6315 items

Can you get in touch with our support so we can better understand the versions used and how you create the initial backup?

metagmbh

Hi,

it works this way:

kopano-backup -u user -l debug --purge 0 /mnt/backup/user
...
2019-07-05 08:51:52,851 - backup - INFO - purged 1 folders and 5 items

I have to iterate through every user manually (or by script…)? I can not start the purge script with the root backup directory like:

kopano-backup -u user -l debug --purge 0 /mnt/backup

?

fbartels

@metagmbh said in Lack of documentation on how to use kopano-backup - especially regarding the purge function:

I can not start the purge script with the root backup directory

No, this is indeed not how this functionality is implemented. when specifying the PATH argument this always need to point directly to the backup of a given user. But in your first example you already specified a path that looked like a user backup, therefore I did not say anything about this.

metagmbh

But in your first example you already specified a path that looked like a user backup, therefore I did not say anything about this.

I used the single user in my example, because a backup of all users would take too long. So I tested it with just mine. The backup script runs everyday with all users.

If someone is interested how to iterate through every user, here my snippet

for D in /mnt/backup/*; do [ -d "${D}" ] && kopano-backup -l debug --purge 0 $D; done