Define ham and spam for spamassassin

BMWfan

Hello,

i was using a long time Zarafa and had the functionality in the webaccess frontend to make a right click on a spam or ham mail and to select the button “Spam” (for a spam mail) which will let learn my spam filter - this is spam - and sort it in the Junk Mail Folder or to select “kein Spam” (for ham mail) which will let learn my spam filter - this is no spam -.

I have already seen there is a deb package called kopano-spamd which i have already installed and seen a manual here https://stash.kopano.io/projects/KSC/repos/kopano-spamd/browse which informs me that since version 8.6 is another way to do this but not what i have to do to implement the new kopano-spamd which is developed for kopano 8.6 and higher. Could someone describe me this, perhaps also better ways to realize this if there are.

Kopano Version: 8.7.80
OS: Debian 9

Thanks in advance and best regards

Daniel

fbartels

The documentation for it can be found in the Kopano core manual: https://documentation.kopano.io/kopanocore_administrator_manual/special_kc_configurations.html#configuring-kopano-spamd-for-automatic-spam-ham-learning

BMWfan

@fbartels thank you.

I have installed the kopano-spamd deb package and made the configuration steps as described here: https://github.com/bkram/inotify-spamlearn
But if i run this command sudo systemctl enable inotify-spamlearn then i get this error message:

Failed to enable unit: File inotify-spamlearn.service: Invalid argument

Best Regards

Daniel

fbartels

Best guess: the instructions are missing a reload of the unit files (after you copied the systemd unit)

BMWfan

@fbartels said in Define ham and spam for spamassassin:

reload of the unit files (after you copied the systemd unit)

What do you mean with that? Or how should i proceed to do this?
A restart of kopano-core and kopano-spamd is already done without any positive progress.

Best Regards

Daniel

fbartels

Restarting any Kopano service won’t here you, since systemd is complaining it cannot find a service unit that you have copied from a third party git repo. What I meant with the “reloading” is that systemctl usually complains that you have to execute a reload command, when you manually installed a unit file. But if this is not displayed to you then my second best guess is that you copied the unit file to the wrong location.

BMWfan

@fbartels said in [But if this is not displayed to you then my second best guess is that you copied the unit file to the wrong location.

No, this is not displayed.

I downloaded the files via wget and copied it in the folder as explained in the manual.

/etc/systemd/system
inotify-spamlearn.service

/usr/local/sbin
inotify-spamlearn.py

/etc/kopano
inotify-spamlearn.cfg

then i tried to execute the command

/etc/systemd/system# systemctl enable inotify-spamlearn.service
Failed to enable unit: File inotify-spamlearn.service: Invalid argument

tried to start it:

/etc/systemd/system# systemctl start inotify-spamlearn.service
Failed to start inotify-spamlearn.service: Unit inotify-spamlearn.service is not loaded properly: Invalid argument.
See system logs and 'systemctl status inotify-spamlearn.service' for details.

sudo systemctl status inotify-spamlearn
● inotify-spamlearn.service
   Loaded: error (Reason: Invalid argument)
   Active: inactive (dead)

So, it should not be dependend on a wrong location, or what do you mean @fbartels ?

Best Regards

Daniel

fbartels

This works for me (Debian 9):

root@system ~/inotify-spamlearn
$ sudo cp inotify-spamlearn.service /etc/systemd/system

root@system ~/inotify-spamlearn
$ sudo systemctl status inotify-spamlearn.service
● inotify-spamlearn.service - Inotify Spamlearn
   Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

BMWfan

@fbartels ok, it seems to be that the download was not complete of the config files thats why i had this issues. Now i’am stucking at the next problem.
I have now all files again downloaded and replaced and gave the file /usr/local/sbin/inotify-spamlearn.py the following rights: rwx—r-x (User:root, Group: root) i hope this is correct.

I installed via pip the python module inotify

pip install inotify
Collecting inotify
  Using cached https://files.pythonhosted.org/packages/c7/fc/9728f1f708ecd5981007abe133d44fdcddf40915f8d13e12a140b77376ae/inotify-0.2.10-py2-none-any.whl
Requirement already satisfied: nose in /usr/local/lib/python2.7/dist-packages (from inotify) (1.3.7)
Installing collected packages: inotify
Successfully installed inotify-0.2.10

and tried to start inotify-spamlearn which gaves me no output.
But, if it try to check the status i see this:

sudo systemctl status inotify-spamlearn
● inotify-spamlearn.service - Inotify Spamlearn
   Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-12-29 13:57:46 CET; 6s ago
  Process: 7172 ExecStart=/usr/local/sbin/inotify-spamlearn.py (code=exited, status=1/FAILURE)
 Main PID: 7172 (code=exited, status=1/FAILURE)

Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]: Traceback (most recent call last):
Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]:   File "/usr/local/sbin/inotify-spamlearn.py", line 11, in <module>
Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]:     import inotify.adapters
Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]: ImportError: No module named 'inotify'

This is the line 11

import inotify.adapters

As you can see here, the module is installed:

pip list
Package         Version
--------------- -------
configparser    3.5.0
inotify         0.2.10
logging         0.4.9.6
MAPI            8.7.80
nose            1.3.7
pip             18.1
PyICU           1.9.5
python-dateutil 2.5.3
pytz            2016.7
pyzor           1.0.0
setuptools      33.1.1
six             1.10.0

Do you have any idea how i could fix this?

Best Regards

Daniel

fbartels

No, sorry not familiar with this script.

BMWfan

@fbartels thanks, nevertheless.

i found now the issue.

I have 2 python versions installed

$ python --version
Python 2.7.13

$ python3 --version
Python 3.5.3

As i explained before, i have installed the needed module via pip install inotify which install it for python 2.7

Requirement already satisfied: nose in /usr/local/lib/python2.7/dist-packages (from inotify) (1.3.7)

The requirements as i see in the python script inotify-spamlearn.py in the first line are amongst other things python 3

#!/usr/bin/env python3

So i had to find a way to install the needed module for python 3 what ive done like this:

$ apt-get install python3-setuptools
$ easy_install3 pip
$ pip-3.5 install inotify

it works:

$ systemctl status inotify-spamlearn.service
● inotify-spamlearn.service - Inotify Spamlearn
   Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-01-01 16:03:38 CET; 7s ago
 Main PID: 1062 (python3)
   CGroup: /system.slice/inotify-spamlearn.service
           └─1062 python3 /usr/local/sbin/inotify-spamlearn.py

Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Starting inotify-spamlearn.py
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spa
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Inotify learning started

Best Regards

Daniel

BMWfan

@fbartels perhaps you have a idea here. As you can see in the post before that my inotify-spamlearn is running.
If i try to mark a mail as spam in the webapp dashboard then i got the following output:

Jan 01 16:35:56 server inotify-spamlearn.py[1367]: INFO Inotify learning started
Jan 01 16:36:55 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 01 16:36:55 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin/user_prefs" is inaccessible: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /dev/null/.spamassassin/bayes.lock.server.server.1434 for /dev/null/.spamassassin/bayes.lock: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /dev/null/.spamassassin/bayes.lock.server.server.1434 for /dev/null/.spamassassin/bayes.lock: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/0A8EFD57433D41F983C4ED716DA7CE5D.eml: Learned tokens from 0 message(s) (1 message(s) examined)
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: INFO Removing file: /var/lib/kopano/spamd/spam/0A8EFD57433D41F983C4ED716DA7CE5D.eml

There is a path ~/.spamassassin which has amongst other things these file included:

root@system:~/.spamassassin# ls
user_prefs

But under /dev is null a file and not a directory

root@system:/dev# cd null
-bash: cd: null: Not a directory

Best Regards

Daniel

ashceryth

Hi @BMWfan

I think this is because the default home of the kopano user is /dev/null (you can check your /etc/passwd) and inotify-spamlearn is running as kopano user. It seems like ~/.spamassassin/ (where ~/ is the home directory of the kopano user - /dev/null) is the default directory used by the sa-learn command. You can specify the paths used by the sa-learn command in inotify-spamlearn.cfg or run inotify-spamlearn as different user (set in the systemd unit file).

Regards
Patrick

BMWfan

@ashceryth thanks for your response. I’am thinking about it what would be the best choice todo.
I’am not sure if it would help and would be good if i assign the kopano user a home directory, because /dev/null is standard if you dont set once i think. What have you done to get it work?

For kopano-spamd i assigned already kopano to the amavis group

gpasswd -a kopano amavis

ashceryth

I suggest to try to run inotify-spamlearn as the same user as amavis runs as.

So, edit the systemd unit file (/etc/systemd/system/inotify-spamlearn.service) and set:

User=amavis
Group=amavis

Run systemctl daemon-reload after editing and restart the service with systemctl restart inotify-spamlearn.

Check your spamd.cfg to make sure it uses the amavis group (the .eml files should then get the gid of the amavis group):

sa_group = amavis

Does it work then?

I don’t use the sa-learn command. I use Rspamd and let inotify-spamlearn send the mails via HTTP post requests with curl.

BMWfan

@ashceryth said in Define ham and spam for spamassassin:

systemctl restart inotify-spamlearn

No, this solves it not. The problem is after i change the sa_group at /etc/kopano/spamd.cfg to amavis then i get permission errors in the kopano-spamd process as here described: Kopano spamd error - PermissionError: [Errno 1] Operation not permitted

My configuration is now this:
/etc/kopano/spamd.cfg

# run as specific user
run_as_user         = kopano

# run as specific group
run_as_group        = kopano

# Spamassassin group
sa_group = kopano

/etc/systemd/system/inotify-spamlearn.service

[Service]
User=kopano
Group=kopano

but then again with this log entries:

Jan 12 16:13:10 euve264608 inotify-spamlearn.py[7348]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 12 16:13:10 euve264608 inotify-spamlearn.py[7348]: config: path "/dev/null/.spamassassin/user_prefs" is inaccessible: Not a directory
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.euve264608.serverprofi24.de.7427 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.euve264608.serverprofi24.de.7427 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/72C00ABE4488476D8AF626120D868211.eml: Learned tokens from 0 message(s) (1 message(s) examined)
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: INFO Removing file: /var/lib/kopano/spamd/spam/72C00ABE4488476D8AF626120D868211.eml

if i change only these entries from kopano to amavis
/etc/systemd/system/inotify-spamlearn.service

[Service]
User=amavis
Group=amavis

then i get this messages:

Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Starting inotify-spamlearn.py
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spam
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Inotify learning started
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.Servername.serverprofi24.de.7506 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.Servername.serverprofi24.de.7506 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml: Learned tokens from 0 message(s) (1 message(s) examined)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: INFO Removing file: /var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Exception in thread Inotify Handling:
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Traceback (most recent call last):
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 80, in inotified
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     process('/'.join([watch_path, filename]), spamcmd, delete, initiator)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 43, in process
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     os.remove(filename)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: PermissionError: [Errno 13] Permission denied: '/var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml'
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: During handling of the above exception, another exception occurred:
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Traceback (most recent call last):
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     self.run()
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/lib/python3.5/threading.py", line 862, in run
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     self._target(*self._args, **self._kwargs)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 85, in inotified
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     i.remove_watch([spam_dir, ham_dir])
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/lib/python3.5/dist-packages/inotify/adapters.py", line 109, in remove_watch
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     wd = self.__watches.get(path)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: TypeError: unhashable type: 'list'

These are the amavis and kopano rights under /etc/groups

amavis:x:123:clamav,kopano
kopano:x:999:kapi,konnect,amavis

ashceryth

@BMWfan said in Define ham and spam for spamassassin:

No, this solves it not. The problem is after i change the sa_group at /etc/kopano/spamd.cfg to amavis then i get permission errors in the kopano-spamd process as here described: Kopano spamd error - PermissionError: [Errno 1] Operation not permitted

Maybe because the problem described by @fbartels in this post insn’t solved for you yet?

Who is the owner/group and what are the permissions of the files in /var/lib/kopano/spamd/spam/ (just ls -la)?

And please test the sa-learn command manually as the desired user before configuring it with inotify-spamlearn to see if everything works as expected.

BMWfan

@ashceryth thanks for your reponse and hints.
Here is the suggested output:

$ /var/lib/kopano/spamd/spam# ls -la
total 8
drwxr-xr-x 2 kopano kopano 4096 Jan 12 17:05 .
drwxr-xr-x 4 kopano kopano 4096 Dec 27 10:32 ..

ashceryth

Well, at the moment your spam folder is empty. Normally the .eml files get deleted after learning by inotify-spamlearn. However, according to your log from above it wasn’t possible to remove them because of permission issues:

@BMWfan said in Define ham and spam for spamassassin:

Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 43, in process
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     os.remove(filename)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: PermissionError: [Errno 13] Permission denied: '/var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml'

That’s why I’m asking. Would be interesting to see the permissions of the .eml files themselves.

BMWfan

@ashceryth i stopped in the inotify-spamlearn service and here is the suggested output:

$ /var/lib/kopano/spamd/spam# ls -la
total 12
drwxr-xr-x 2 kopano kopano 4096 Jan 14 20:39 .
drwxr-xr-x 4 kopano kopano 4096 Dec 27 10:32 ..
-rw-rw-rw- 1 kopano kopano 3643 Jan 14 20:39 600F5F44181B4F2B865EAE2A45EC2573.eml