@pwhooftman said in Running script throws login error:

Administrator@server:~$ python contact2csv.py

You’d have to run the second command with sudo as well.

Hi all,

as the amount of documentation about storing attachments in S3 storage is not much I’ve done some reverse engineering with a bit of try&error on my own and would like to share my earned knowledge with you. Please keep in mind that I’m not from the support stuff and just want to shed some light into the dark for other users.

Please do NOT copy & execute these commands without knowing what these commands do.
I will not take over any guarantee if you will result in data loss or destroy your environment.

I’ve done the successful migration from “files” to “s3” with kopano-server version is 8.7.80.307.3 using my notes below:

pkill kopano-server (stop kopano server) mkdir /tmp/migrate cp -ar /var/lib/kopano/attachments /tmp/migrate cd /tmp/migrate mkdir kopano-attachments find -type f -exec mv {} kopano-attachments/ \; find -type d -empty -delete cd kopano-attachments gzip -d *.gz mv /var/lib/kopano/attachments /var/lib/kopano/attachments_old``

upload the folder kopano-attachments to your s3 bucket.
Change the configuration as follow:

attachment_storage = s3 attachment_path = kopano-attachments (IMPORTANT: no leading / name of your subfolder in basket) #attachment_files_fsync = yes #attachment_compression = 6 attachment_s3_hostname = s3.eu-central-1.amazonaws.com attachment_s3_region = eu-central-1 attachment_s3_protocol = https attachment_s3_uristyle = path attachment_s3_accesskeyid = abc1234abc1234 attachment_s3_secretaccesskey = abc1234+abc1234+abc1234 attachment_s3_bucketname = mybasketname

At first start of kopano-server uses the parameter “ignore parameter” as attachment storage was moved

/usr/sbin/kopano-server -F --ignore-attachment-storage-conflict

Open some attachments within your webapp to see if they are downloading correctly.

After this stop the manually started kopano-server and start the service daemon again.

I hope these notes will help you to save your time and headache ;-)
If you like you can spent me a coffee ;-)

@fbartels, thanks for your hint about the folder structure.

Regards
Timo

Hi @fbartels
Thank you for the information. You’re absolutely right! One should read the documentation first…but because of the automatic updating process with the repositories in ubuntu it would be nice when there would be a hint in the executing messages.
You do a great job at Kopano! Thank you very much.

Hi fbartels,

I upgraded to version 8.7.80.587.10057a. After restart of kopano-server it told me to run “kopano-dbadm usmp”. After an other kopano-server restart, the server accepted the messages. It was either the upgrade to a newer version or the dbadm fix.

Thanks for your help.

Regards
Stefan

@s_wolff said in dagent 8.7.80.355 Unable to commit message: disk error (80040116) => Mailbox temporarily unavailable:

How did you solve this?

you have to upgrade to the latest version.

At least in core-8.7.80.529-Ubuntu_18.04-amd64 kopano-cachestat is working:

root@server:~# kopano-cachestat Kopano Cache Statistics Server start time: b'Wed Jan 30 12:17:52 2019' Current time : Fri Feb 8 10:56:01 2019 Cache Hit ratio Mem usage ratio cell ( 224838290/ 225394483) (99%) ( 203173926/ 2185232384) ( 9%) abinfo ( 5792911/ 5792966) (99%) ( 9568/ 26214400) ( 0%) obj ( 37307767/ 37341267) (99%) ( 5169472/ 67108864) ( 7%) store ( 12902815/ 12921698) (99%) ( 736032/ 1048576) (70%) server ( 0/ 0) (N/A) ( 0/ 1048576) ( 0%) userid ( 7773214/ 8078082) (96%) ( 7904/ 1048576) ( 0%) acl ( 215/ 221) (97%) ( 600/ 1048576) ( 0%) extern ( 2614014/ 2696750) (96%) ( 6536/ 1048576) ( 0%) uquota ( 883137/ 908505) (97%) ( 312/ 1048576) ( 0%) index2 ( 22745106/ 22963788) (99%) ( 6739144/ 33554432) (20%) quota ( 872037/ 931742) (93%) ( 728/ 1048576) ( 0%) index1 ( 2736582/ 16475802) (16%) ( 7150768/ 33554432) (21%)```

Seems to be an issue with the python-magic shipped with Ubuntu that is missing magic_open, see https://github.com/ahupp/python-magic.

Name Conflict There are, sadly, two libraries which use the module name magic. Both have been around for quite a while.If you are using this module and get an error using a method like open, your code is expecting the other one. Hopefully one day these will be reconciled.

Where can I find the module including magic_open for Ubuntu?

the DB is indeed on SSD already, but the attachments aren’t nor the index files themselves.

Just because the SSD raid array is very small… hence I have to check very carefully what’s on there to avoid running out of space…

Hi @fbartels,

yes, after installing python-kopano package I was able to run the script. After that and redownloading all email messages via IMAP the headers of about 1000 (the exact number of 1037 if they were processed in order of date - I didn’t check every single one of them ;-) ) of nearly 3000 emails in the inbox of one account were reconstructed (from the IMAP point of view). The rest of the emails in inbox as well as all emails in other folders look like before running the script (headers missing). Those reconstructed emails now seem to have all headers in them. MIME-Headers still different from the originals. I could not find any information in the logs.

Running the script again does not change anything.

Fix is working, kopano-gateway crashed once again
/var/log/syslog

Feb 6 20:50:06 kopano_www_km22-01 kopano-gateway[21672]: Unexpected error 9 on netlink descriptor 99 Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Main process exited, code=killed, status=6/ABRT Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Unit entered failed state. Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Failed with result 'signal'. Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: kopano-gateway.service: Service hold-off time over, scheduling restart. Feb 6 20:50:06 kopano_www_km22-01 systemd[1]: Stopped Kopano Groupware Core IMAP/POP3 Gateway.

@fbartels did so.

Hi @s_wolff ,

if you would be using ldap, then you could realise this with the so called “dynamic groups”.

I am confident that you can use python-kopano to read out (and modify) groups in a public folder, but since I am not a python developer I cannot tell you how.

Reply for myself, testing bit more, since i notice more of these backtraces in the forum.

If someone can check this also, this might help in fixing this problem.

if i run :

systemctl stop kopano-server systemctl start kopano-server

i can repeat this without any problems. No crashed.

if i use this script to stop/start like this :

/usr/local/bin/kopano-maint.sh stop
/usr/local/bin/kopano-maint.sh start

It also works fine. This is the script.

#!/usr/bin/env bash # Order matters ! SERVICES="postfix kopano-spooler kopano-dagent kopano-gateway kopano-ical kopano-search kopano-monitor kopano-server kopano-konnectd" CMD_SYSTEMCTL="$(which systemctl)" if [ -z "$1" ] then echo "Use stop/start to start/stop the needed services, exiting now.." exit 1 fi if [ "$1" = stop ] then for x in $SERVICES do ${CMD_SYSTEMCTL} "$1" "$x" done fi if [ "$1" = start ] then for x in $SERVICES do ${CMD_SYSTEMCTL} "$1" "$x" done fi

now if i use :
bash /usr/local/bin/kopano-maint.sh stop
bash /usr/local/bin/kopano-maint.sh start

now it im getting the backtrace.

If i run all commands in the same order as the script, but manualy it also works fine.

So i dont understand why this happing? anyone?

The default score for FROM_EXCESS_QP and TO_EXCESS_QP in Rspamd is +1.2

-- From that contains encoded characters while quoted-printable is not needed as all symbols are 7bit reconf['FROM_EXCESS_QP'] = { re = string.format('%s & !%s', from_encoded_qp, from_needs_mime), score = 1.2, description = 'From that contains encoded characters while quoted-printable is not needed as all symbols are 7bit', group = 'excessqp' } -- To that contains encoded characters while quoted-printable is not needed as all symbols are 7bit -- Final rule reconf['TO_EXCESS_QP'] = { re = string.format('%s & !%s', to_encoded_qp, to_needs_mime), score = 1.2, description = 'To that contains encoded characters while quoted-printable is not needed as all symbols are 7bit', group = 'excessqp' }

So every mail received from a sender via Kopano gets a score +2.4 added. This is a severe problem!

Have in mind that you can’t lower the score for FROM_EXCESS_QP in Rspamd because doing so would lower the score of mostly every real SPAM-mail.

@fbartels said in configuration defaults:

Also we are not including any mysql configuration magic in our own packages since we cannot be certain that the admin actually wants to have mysql running on the same host at all.

Ahh, yes, thats a good one.
Thanks for clearing this up.

Thanks, Felix!