Cleanup Attachments without mail reference
-
Hello Forum,
i ran into a diskspace problem because the old attachments are growning fast.
extended my logical volume to get some time :)We also got a Mailarchive, every mail is bcc’d to this and it stores all mails with attachments.
Mails older 180 days will be deleted on Kopano-Server. (if there is no marking)How do i delete the external stored attachments without mail-reference in db?
Coffee_is_life
-
Hi @Coffee_is_life ,
actually these should be automatically cleaned up (once the softdelete has expired).
-
OK, i did some brainwork and i think this thread can be closed…
explanation:
I copied the attachements around April 14th in order to switch to kopano and the new centos7 server - so all attachments older this date got the new timestamp.
I bet the check-script if attachments can be deleted will check the timestamp aswell… But there is no attachments older April 14th… but if i wait some time (Okt 15th should be the date) than all attachments that i copied are older 180 days and will get deleted.
correct me if this wont happen :) - and if thats the case, how do i manage this?
Coffee_is_life
-
@Coffee_is_life said in Cleanup Attachments without mail reference:
the check-script if attachments can be deleted will check the timestamp aswell
i don’t know what kind of script you are talking about.
-
“Script” is the wrong word, better the internal function to delete this old attachments will check for the timestamp
-
@Coffee_is_life there are no functions to check files on disk, but rather all attachments belonging to a mail will be purged, once it has been removed from the softdelete.
-
Then i can’t explain why the attachment-folder grows so much:
nearly 100GB in 1 1/2 months…
if the delete function is working correctly this shouldn’t happend - no more data than usually…
-
@Coffee_is_life do you have a cron job for the softdelete setup?
-
first time i hear about a cronjob, managing the softdelete setup… - so the answer is no
is there a guide or can you provide me how this should look like?
-
@Coffee_is_life said in Cleanup Attachments without mail reference:
is there a guide or can you provide me how this should look like?
there is an extensive tuning guide than you can request from our support. this also covers the manual trigger of softdelete (which is by the way just
kopano-admin --purge-softdelete 30). using a cron for this instead of the builtin functionality is recommended for installations above 100 users (and the internal trigger for it is currently broken, but fixed for 8.3.3 final - https://jira.kopano.io/browse/KC-638). -
Thanks, this will help a lot!
just to be sure, if i use “kopano-admin --purge-softdelete 180” once a day (21:00, 1h before the backup starts) than it will delete all items older 180 days including attachments, right?
Coffee_is_life
-
@Coffee_is_life said in Cleanup Attachments without mail reference:
it will delete all items older 180 days including attachments
no, it will delete items from the softdelete that are older than 180 days.
-
@fbartels said in Cleanup Attachments without mail reference:
no, it will delete items from the softdelete that are older than 180 days.
Makes sense :sweat_smile:
is there a way to easy delete all items older 180 days (or mark them as deleted) from db including attachments? - as i mention we got an archive system and wont need double data
Coffee_is_life
-
@Coffee_is_life said in Cleanup Attachments without mail reference:
is there a way to easy delete all items older 180 days (or mark them as deleted) from db
I don’t think there is a ready made solution for that yet, but this is certainly possible with python-kopano. If you open a support case we could collect your requirements, investigate and see how much effort it would be to create such a script for you.
-
I will see if we really need this function - since no script exists yet.
I did the kopano-admin --purge-softdelete 30 to clean up some data…
deleted nearly 13 GB, seems fine… but:since 13:00 it is doing nothing (0,0 CPU usage, 0,0 Mem usage), trying to start the purge again:
Fri Jul 28 14:14:24 2017: [info ] Start forced softdelete clean up Fri Jul 28 14:14:24 2017: [error ] Softdelete already runningThe command isnt running anymore (ps wwwaux |grep kopano-admin shows nothing)
Dont want to restart the server/service now so i dont know if it happens everytime…If yes, the cronjob also needs to restart the kopano-server but this is not an option, since the users are accessing it nearly 24/7.
Something i can do while server is running?
Coffee_is_life
-
@Coffee_is_life give it a bit more patience. even if its nut running anymore in the foreground, the task kicked off in the server may very well still be active.
and no you don’t need to restart kopano-server during or after running purge-softdelete.
-
Can we be sure that this has really been fixed in 8.3.3? I just had a case where (with a pre-8.3.3 version) I had no space left for attachments and after reading this post, I ran softdelete manually. However this seems to have created a hude load on the database (100 of 400 GB attachements deleted, so I suppose about 25% of database entries), i.e. softdelete has probably never worked since setting this up. And, what was a real problem, I then had users with empty inboxes that had to be resynced (might be a load issue). But I also saw entries like these in the z-push-error logs:
StatusException: PHPWrapper->ImportMessageDeletion(): Received 13708 remove requests from ICS for folder ‘34e26d5db12f428f8105c4110045f1a3612903000000’ (max. 1000 allowed). Triggering folder
re-sync. - code: 3 - file: /usr/share/z-push/backend/kopano/mapiphpwrapper.php:171But why should the final deletion of already deleted objects trigger anything at the z-push level??? These objects should not be user-visible anyway?
How can I check that softdelete is working as intended to avoid such accumulations in the future?
-
@isol said in Cleanup Attachments without mail reference:
Can we be sure that this has really been fixed in 8.3.3
yes, its in the changelog and the fix has been verified.
