Define ham and spam for spamassassin

fbartels

The documentation for it can be found in the Kopano core manual: https://documentation.kopano.io/kopanocore_administrator_manual/special_kc_configurations.html#configuring-kopano-spamd-for-automatic-spam-ham-learning

BMWfan

@fbartels thank you.

I have installed the kopano-spamd deb package and made the configuration steps as described here: https://github.com/bkram/inotify-spamlearn
But if i run this command sudo systemctl enable inotify-spamlearn then i get this error message:

Failed to enable unit: File inotify-spamlearn.service: Invalid argument

Best Regards

Daniel

fbartels

Best guess: the instructions are missing a reload of the unit files (after you copied the systemd unit)

BMWfan

@fbartels said in Define ham and spam for spamassassin:

reload of the unit files (after you copied the systemd unit)

What do you mean with that? Or how should i proceed to do this?
A restart of kopano-core and kopano-spamd is already done without any positive progress.

Best Regards

Daniel

fbartels

Restarting any Kopano service won’t here you, since systemd is complaining it cannot find a service unit that you have copied from a third party git repo. What I meant with the “reloading” is that systemctl usually complains that you have to execute a reload command, when you manually installed a unit file. But if this is not displayed to you then my second best guess is that you copied the unit file to the wrong location.

BMWfan

@fbartels said in [But if this is not displayed to you then my second best guess is that you copied the unit file to the wrong location.

No, this is not displayed.

I downloaded the files via wget and copied it in the folder as explained in the manual.

/etc/systemd/system
inotify-spamlearn.service

/usr/local/sbin
inotify-spamlearn.py

/etc/kopano
inotify-spamlearn.cfg

then i tried to execute the command

/etc/systemd/system# systemctl enable inotify-spamlearn.service
Failed to enable unit: File inotify-spamlearn.service: Invalid argument

tried to start it:

/etc/systemd/system# systemctl start inotify-spamlearn.service
Failed to start inotify-spamlearn.service: Unit inotify-spamlearn.service is not loaded properly: Invalid argument.
See system logs and 'systemctl status inotify-spamlearn.service' for details.

sudo systemctl status inotify-spamlearn
● inotify-spamlearn.service
   Loaded: error (Reason: Invalid argument)
   Active: inactive (dead)

So, it should not be dependend on a wrong location, or what do you mean @fbartels ?

Best Regards

Daniel

fbartels

This works for me (Debian 9):

root@system ~/inotify-spamlearn
$ sudo cp inotify-spamlearn.service /etc/systemd/system

root@system ~/inotify-spamlearn
$ sudo systemctl status inotify-spamlearn.service
● inotify-spamlearn.service - Inotify Spamlearn
   Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

BMWfan

@fbartels ok, it seems to be that the download was not complete of the config files thats why i had this issues. Now i’am stucking at the next problem.
I have now all files again downloaded and replaced and gave the file /usr/local/sbin/inotify-spamlearn.py the following rights: rwx—r-x (User:root, Group: root) i hope this is correct.

I installed via pip the python module inotify

pip install inotify
Collecting inotify
  Using cached https://files.pythonhosted.org/packages/c7/fc/9728f1f708ecd5981007abe133d44fdcddf40915f8d13e12a140b77376ae/inotify-0.2.10-py2-none-any.whl
Requirement already satisfied: nose in /usr/local/lib/python2.7/dist-packages (from inotify) (1.3.7)
Installing collected packages: inotify
Successfully installed inotify-0.2.10

and tried to start inotify-spamlearn which gaves me no output.
But, if it try to check the status i see this:

sudo systemctl status inotify-spamlearn
● inotify-spamlearn.service - Inotify Spamlearn
   Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-12-29 13:57:46 CET; 6s ago
  Process: 7172 ExecStart=/usr/local/sbin/inotify-spamlearn.py (code=exited, status=1/FAILURE)
 Main PID: 7172 (code=exited, status=1/FAILURE)

Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]: Traceback (most recent call last):
Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]:   File "/usr/local/sbin/inotify-spamlearn.py", line 11, in <module>
Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]:     import inotify.adapters
Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]: ImportError: No module named 'inotify'

This is the line 11

import inotify.adapters

As you can see here, the module is installed:

pip list
Package         Version
--------------- -------
configparser    3.5.0
inotify         0.2.10
logging         0.4.9.6
MAPI            8.7.80
nose            1.3.7
pip             18.1
PyICU           1.9.5
python-dateutil 2.5.3
pytz            2016.7
pyzor           1.0.0
setuptools      33.1.1
six             1.10.0

Do you have any idea how i could fix this?

Best Regards

Daniel

fbartels

No, sorry not familiar with this script.

BMWfan

@fbartels thanks, nevertheless.

i found now the issue.

I have 2 python versions installed

$ python --version
Python 2.7.13

$ python3 --version
Python 3.5.3

As i explained before, i have installed the needed module via pip install inotify which install it for python 2.7

Requirement already satisfied: nose in /usr/local/lib/python2.7/dist-packages (from inotify) (1.3.7)

The requirements as i see in the python script inotify-spamlearn.py in the first line are amongst other things python 3

#!/usr/bin/env python3

So i had to find a way to install the needed module for python 3 what ive done like this:

$ apt-get install python3-setuptools
$ easy_install3 pip
$ pip-3.5 install inotify

it works:

$ systemctl status inotify-spamlearn.service
● inotify-spamlearn.service - Inotify Spamlearn
   Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-01-01 16:03:38 CET; 7s ago
 Main PID: 1062 (python3)
   CGroup: /system.slice/inotify-spamlearn.service
           └─1062 python3 /usr/local/sbin/inotify-spamlearn.py

Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Starting inotify-spamlearn.py
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spa
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham
Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Inotify learning started

Best Regards

Daniel

BMWfan

@fbartels perhaps you have a idea here. As you can see in the post before that my inotify-spamlearn is running.
If i try to mark a mail as spam in the webapp dashboard then i got the following output:

Jan 01 16:35:56 server inotify-spamlearn.py[1367]: INFO Inotify learning started
Jan 01 16:36:55 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 01 16:36:55 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin/user_prefs" is inaccessible: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /dev/null/.spamassassin/bayes.lock.server.server.1434 for /dev/null/.spamassassin/bayes.lock: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /dev/null/.spamassassin/bayes.lock.server.server.1434 for /dev/null/.spamassassin/bayes.lock: Not a directory
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/0A8EFD57433D41F983C4ED716DA7CE5D.eml: Learned tokens from 0 message(s) (1 message(s) examined)
Jan 01 16:36:56 server inotify-spamlearn.py[1367]: INFO Removing file: /var/lib/kopano/spamd/spam/0A8EFD57433D41F983C4ED716DA7CE5D.eml

There is a path ~/.spamassassin which has amongst other things these file included:

root@system:~/.spamassassin# ls
user_prefs

But under /dev is null a file and not a directory

root@system:/dev# cd null
-bash: cd: null: Not a directory

Best Regards

Daniel

ashceryth

Hi @BMWfan

I think this is because the default home of the kopano user is /dev/null (you can check your /etc/passwd) and inotify-spamlearn is running as kopano user. It seems like ~/.spamassassin/ (where ~/ is the home directory of the kopano user - /dev/null) is the default directory used by the sa-learn command. You can specify the paths used by the sa-learn command in inotify-spamlearn.cfg or run inotify-spamlearn as different user (set in the systemd unit file).

Regards
Patrick

BMWfan

@ashceryth thanks for your response. I’am thinking about it what would be the best choice todo.
I’am not sure if it would help and would be good if i assign the kopano user a home directory, because /dev/null is standard if you dont set once i think. What have you done to get it work?

For kopano-spamd i assigned already kopano to the amavis group

gpasswd -a kopano amavis

ashceryth

I suggest to try to run inotify-spamlearn as the same user as amavis runs as.

So, edit the systemd unit file (/etc/systemd/system/inotify-spamlearn.service) and set:

User=amavis
Group=amavis

Run systemctl daemon-reload after editing and restart the service with systemctl restart inotify-spamlearn.

Check your spamd.cfg to make sure it uses the amavis group (the .eml files should then get the gid of the amavis group):

sa_group = amavis

Does it work then?

I don’t use the sa-learn command. I use Rspamd and let inotify-spamlearn send the mails via HTTP post requests with curl.

BMWfan

@ashceryth said in Define ham and spam for spamassassin:

systemctl restart inotify-spamlearn

No, this solves it not. The problem is after i change the sa_group at /etc/kopano/spamd.cfg to amavis then i get permission errors in the kopano-spamd process as here described: Kopano spamd error - PermissionError: [Errno 1] Operation not permitted

My configuration is now this:
/etc/kopano/spamd.cfg

# run as specific user
run_as_user         = kopano

# run as specific group
run_as_group        = kopano

# Spamassassin group
sa_group = kopano

/etc/systemd/system/inotify-spamlearn.service

[Service]
User=kopano
Group=kopano

but then again with this log entries:

Jan 12 16:13:10 euve264608 inotify-spamlearn.py[7348]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
Jan 12 16:13:10 euve264608 inotify-spamlearn.py[7348]: config: path "/dev/null/.spamassassin/user_prefs" is inaccessible: Not a directory
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.euve264608.serverprofi24.de.7427 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.euve264608.serverprofi24.de.7427 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/72C00ABE4488476D8AF626120D868211.eml: Learned tokens from 0 message(s) (1 message(s) examined)
Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: INFO Removing file: /var/lib/kopano/spamd/spam/72C00ABE4488476D8AF626120D868211.eml

if i change only these entries from kopano to amavis
/etc/systemd/system/inotify-spamlearn.service

[Service]
User=amavis
Group=amavis

then i get this messages:

Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Starting inotify-spamlearn.py
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spam
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Inotify learning started
Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.Servername.serverprofi24.de.7506 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.Servername.serverprofi24.de.7506 for /var/lib/spamassassin/bayes.lock: Permission denied
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml: Learned tokens from 0 message(s) (1 message(s) examined)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: INFO Removing file: /var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Exception in thread Inotify Handling:
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Traceback (most recent call last):
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 80, in inotified
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     process('/'.join([watch_path, filename]), spamcmd, delete, initiator)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 43, in process
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     os.remove(filename)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: PermissionError: [Errno 13] Permission denied: '/var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml'
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: During handling of the above exception, another exception occurred:
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Traceback (most recent call last):
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     self.run()
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/lib/python3.5/threading.py", line 862, in run
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     self._target(*self._args, **self._kwargs)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 85, in inotified
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     i.remove_watch([spam_dir, ham_dir])
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/lib/python3.5/dist-packages/inotify/adapters.py", line 109, in remove_watch
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     wd = self.__watches.get(path)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: TypeError: unhashable type: 'list'

These are the amavis and kopano rights under /etc/groups

amavis:x:123:clamav,kopano
kopano:x:999:kapi,konnect,amavis

ashceryth

@BMWfan said in Define ham and spam for spamassassin:

No, this solves it not. The problem is after i change the sa_group at /etc/kopano/spamd.cfg to amavis then i get permission errors in the kopano-spamd process as here described: Kopano spamd error - PermissionError: [Errno 1] Operation not permitted

Maybe because the problem described by @fbartels in this post insn’t solved for you yet?

Who is the owner/group and what are the permissions of the files in /var/lib/kopano/spamd/spam/ (just ls -la)?

And please test the sa-learn command manually as the desired user before configuring it with inotify-spamlearn to see if everything works as expected.

BMWfan

@ashceryth thanks for your reponse and hints.
Here is the suggested output:

$ /var/lib/kopano/spamd/spam# ls -la
total 8
drwxr-xr-x 2 kopano kopano 4096 Jan 12 17:05 .
drwxr-xr-x 4 kopano kopano 4096 Dec 27 10:32 ..

ashceryth

Well, at the moment your spam folder is empty. Normally the .eml files get deleted after learning by inotify-spamlearn. However, according to your log from above it wasn’t possible to remove them because of permission issues:

@BMWfan said in Define ham and spam for spamassassin:

Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 43, in process
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     os.remove(filename)
Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: PermissionError: [Errno 13] Permission denied: '/var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml'

That’s why I’m asking. Would be interesting to see the permissions of the .eml files themselves.

BMWfan

@ashceryth i stopped in the inotify-spamlearn service and here is the suggested output:

$ /var/lib/kopano/spamd/spam# ls -la
total 12
drwxr-xr-x 2 kopano kopano 4096 Jan 14 20:39 .
drwxr-xr-x 4 kopano kopano 4096 Dec 27 10:32 ..
-rw-rw-rw- 1 kopano kopano 3643 Jan 14 20:39 600F5F44181B4F2B865EAE2A45EC2573.eml

ashceryth

The reason for the permission error messages is that your spam directory is not writable for the group. Without write permissions the .eml files cannot be removed by inotify-spamlearn running as amavis (member of kopano group).

Just run chmod g+w /var/lib/kopano/spamd/spam as well as chmod g+w /var/lib/kopano/spamd/ham and it should be working.

P.S. Having the .eml files world read and writable seems no to be optimal :-)