Define ham and spam for spamassassin



  • Hello,

    i was using a long time Zarafa and had the functionality in the webaccess frontend to make a right click on a spam or ham mail and to select the button “Spam” (for a spam mail) which will let learn my spam filter - this is spam - and sort it in the Junk Mail Folder or to select “kein Spam” (for ham mail) which will let learn my spam filter - this is no spam -.

    I have already seen there is a deb package called kopano-spamd which i have already installed and seen a manual here https://stash.kopano.io/projects/KSC/repos/kopano-spamd/browse which informs me that since version 8.6 is another way to do this but not what i have to do to implement the new kopano-spamd which is developed for kopano 8.6 and higher. Could someone describe me this, perhaps also better ways to realize this if there are.

    Kopano Version: 8.7.80
    OS: Debian 9

    Thanks in advance and best regards

    Daniel


  • Kopano



  • @fbartels thank you.

    I have installed the kopano-spamd deb package and made the configuration steps as described here: https://github.com/bkram/inotify-spamlearn
    But if i run this command sudo systemctl enable inotify-spamlearn then i get this error message:

    Failed to enable unit: File inotify-spamlearn.service: Invalid argument
    

    Best Regards

    Daniel


  • Kopano

    Best guess: the instructions are missing a reload of the unit files (after you copied the systemd unit)



  • @fbartels said in Define ham and spam for spamassassin:

    reload of the unit files (after you copied the systemd unit)

    What do you mean with that? Or how should i proceed to do this?
    A restart of kopano-core and kopano-spamd is already done without any positive progress.

    Best Regards

    Daniel


  • Kopano

    Restarting any Kopano service won’t here you, since systemd is complaining it cannot find a service unit that you have copied from a third party git repo. What I meant with the “reloading” is that systemctl usually complains that you have to execute a reload command, when you manually installed a unit file. But if this is not displayed to you then my second best guess is that you copied the unit file to the wrong location.



  • @fbartels said in [But if this is not displayed to you then my second best guess is that you copied the unit file to the wrong location.

    No, this is not displayed.

    I downloaded the files via wget and copied it in the folder as explained in the manual.

    /etc/systemd/system
    inotify-spamlearn.service
    
    /usr/local/sbin
    inotify-spamlearn.py
    
    /etc/kopano
    inotify-spamlearn.cfg
    

    then i tried to execute the command

    /etc/systemd/system# systemctl enable inotify-spamlearn.service
    Failed to enable unit: File inotify-spamlearn.service: Invalid argument
    

    tried to start it:

    /etc/systemd/system# systemctl start inotify-spamlearn.service
    Failed to start inotify-spamlearn.service: Unit inotify-spamlearn.service is not loaded properly: Invalid argument.
    See system logs and 'systemctl status inotify-spamlearn.service' for details.
    
    sudo systemctl status inotify-spamlearn
    ● inotify-spamlearn.service
       Loaded: error (Reason: Invalid argument)
       Active: inactive (dead)
    

    So, it should not be dependend on a wrong location, or what do you mean @fbartels ?

    Best Regards

    Daniel


  • Kopano

    This works for me (Debian 9):

    root@system ~/inotify-spamlearn
    $ sudo cp inotify-spamlearn.service /etc/systemd/system
    
    root@system ~/inotify-spamlearn
    $ sudo systemctl status inotify-spamlearn.service
    ● inotify-spamlearn.service - Inotify Spamlearn
       Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; disabled; vendor preset: enabled)
       Active: inactive (dead)
    


  • @fbartels ok, it seems to be that the download was not complete of the config files thats why i had this issues. Now i’am stucking at the next problem.
    I have now all files again downloaded and replaced and gave the file /usr/local/sbin/inotify-spamlearn.py the following rights: rwx—r-x (User:root, Group: root) i hope this is correct.

    I installed via pip the python module inotify

    pip install inotify
    Collecting inotify
      Using cached https://files.pythonhosted.org/packages/c7/fc/9728f1f708ecd5981007abe133d44fdcddf40915f8d13e12a140b77376ae/inotify-0.2.10-py2-none-any.whl
    Requirement already satisfied: nose in /usr/local/lib/python2.7/dist-packages (from inotify) (1.3.7)
    Installing collected packages: inotify
    Successfully installed inotify-0.2.10
    

    and tried to start inotify-spamlearn which gaves me no output.
    But, if it try to check the status i see this:

    sudo systemctl status inotify-spamlearn
    ● inotify-spamlearn.service - Inotify Spamlearn
       Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Sat 2018-12-29 13:57:46 CET; 6s ago
      Process: 7172 ExecStart=/usr/local/sbin/inotify-spamlearn.py (code=exited, status=1/FAILURE)
     Main PID: 7172 (code=exited, status=1/FAILURE)
    
    Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]: Traceback (most recent call last):
    Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]:   File "/usr/local/sbin/inotify-spamlearn.py", line 11, in <module>
    Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]:     import inotify.adapters
    Dec 29 13:57:46 euve264608 inotify-spamlearn.py[7172]: ImportError: No module named 'inotify'
    

    This is the line 11

    import inotify.adapters
    

    As you can see here, the module is installed:

    pip list
    Package         Version
    --------------- -------
    configparser    3.5.0
    inotify         0.2.10
    logging         0.4.9.6
    MAPI            8.7.80
    nose            1.3.7
    pip             18.1
    PyICU           1.9.5
    python-dateutil 2.5.3
    pytz            2016.7
    pyzor           1.0.0
    setuptools      33.1.1
    six             1.10.0
    

    Do you have any idea how i could fix this?

    Best Regards

    Daniel


  • Kopano

    No, sorry not familiar with this script.



  • @fbartels thanks, nevertheless.

    i found now the issue.

    I have 2 python versions installed

    $ python --version
    Python 2.7.13
    
    $ python3 --version
    Python 3.5.3
    

    As i explained before, i have installed the needed module via pip install inotify which install it for python 2.7

    Requirement already satisfied: nose in /usr/local/lib/python2.7/dist-packages (from inotify) (1.3.7)
    

    The requirements as i see in the python script inotify-spamlearn.py in the first line are amongst other things python 3

    #!/usr/bin/env python3
    

    So i had to find a way to install the needed module for python 3 what ive done like this:

    $ apt-get install python3-setuptools
    $ easy_install3 pip
    $ pip-3.5 install inotify
    

    it works:

    $ systemctl status inotify-spamlearn.service
    ● inotify-spamlearn.service - Inotify Spamlearn
       Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; enabled; vendor preset: enabled)
       Active: active (running) since Tue 2019-01-01 16:03:38 CET; 7s ago
     Main PID: 1062 (python3)
       CGroup: /system.slice/inotify-spamlearn.service
               └─1062 python3 /usr/local/sbin/inotify-spamlearn.py
    
    Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Starting inotify-spamlearn.py
    Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
    Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spa
    Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
    Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham
    Jan 01 16:03:38 euve264608 inotify-spamlearn.py[1062]: INFO Inotify learning started
    

    Best Regards

    Daniel



  • @fbartels perhaps you have a idea here. As you can see in the post before that my inotify-spamlearn is running.
    If i try to mark a mail as spam in the webapp dashboard then i got the following output:

    Jan 01 16:35:56 server inotify-spamlearn.py[1367]: INFO Inotify learning started
    Jan 01 16:36:55 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
    Jan 01 16:36:55 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin/user_prefs" is inaccessible: Not a directory
    Jan 01 16:36:56 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
    Jan 01 16:36:56 server inotify-spamlearn.py[1367]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
    Jan 01 16:36:56 server inotify-spamlearn.py[1367]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /dev/null/.spamassassin/bayes.lock.server.server.1434 for /dev/null/.spamassassin/bayes.lock: Not a directory
    Jan 01 16:36:56 server inotify-spamlearn.py[1367]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /dev/null/.spamassassin/bayes.lock.server.server.1434 for /dev/null/.spamassassin/bayes.lock: Not a directory
    Jan 01 16:36:56 server inotify-spamlearn.py[1367]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
    Jan 01 16:36:56 server inotify-spamlearn.py[1367]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/0A8EFD57433D41F983C4ED716DA7CE5D.eml: Learned tokens from 0 message(s) (1 message(s) examined)
    Jan 01 16:36:56 server inotify-spamlearn.py[1367]: INFO Removing file: /var/lib/kopano/spamd/spam/0A8EFD57433D41F983C4ED716DA7CE5D.eml
    

    There is a path ~/.spamassassin which has amongst other things these file included:

    root@system:~/.spamassassin# ls
    user_prefs
    

    But under /dev is null a file and not a directory

    root@system:/dev# cd null
    -bash: cd: null: Not a directory
    

    Best Regards

    Daniel



  • Hi @BMWfan

    I think this is because the default home of the kopano user is /dev/null (you can check your /etc/passwd) and inotify-spamlearn is running as kopano user. It seems like ~/.spamassassin/ (where ~/ is the home directory of the kopano user - /dev/null) is the default directory used by the sa-learn command. You can specify the paths used by the sa-learn command in inotify-spamlearn.cfg or run inotify-spamlearn as different user (set in the systemd unit file).

    Regards
    Patrick



  • @ashceryth thanks for your response. I’am thinking about it what would be the best choice todo.
    I’am not sure if it would help and would be good if i assign the kopano user a home directory, because /dev/null is standard if you dont set once i think. What have you done to get it work?

    For kopano-spamd i assigned already kopano to the amavis group

    gpasswd -a kopano amavis
    


  • I suggest to try to run inotify-spamlearn as the same user as amavis runs as.

    So, edit the systemd unit file (/etc/systemd/system/inotify-spamlearn.service) and set:

    User=amavis
    Group=amavis
    

    Run systemctl daemon-reload after editing and restart the service with systemctl restart inotify-spamlearn.

    Check your spamd.cfg to make sure it uses the amavis group (the .eml files should then get the gid of the amavis group):

    sa_group = amavis

    Does it work then?

    I don’t use the sa-learn command. I use Rspamd and let inotify-spamlearn send the mails via HTTP post requests with curl.



  • @ashceryth said in Define ham and spam for spamassassin:

    systemctl restart inotify-spamlearn

    No, this solves it not. The problem is after i change the sa_group at /etc/kopano/spamd.cfg to amavis then i get permission errors in the kopano-spamd process as here described: Kopano spamd error - PermissionError: [Errno 1] Operation not permitted

    My configuration is now this:
    /etc/kopano/spamd.cfg

    # run as specific user
    run_as_user         = kopano
    
    # run as specific group
    run_as_group        = kopano
    
    # Spamassassin group
    sa_group = kopano
    

    /etc/systemd/system/inotify-spamlearn.service

    [Service]
    User=kopano
    Group=kopano
    

    but then again with this log entries:

    Jan 12 16:13:10 euve264608 inotify-spamlearn.py[7348]: config: path "/dev/null/.spamassassin" is inaccessible: Not a directory
    Jan 12 16:13:10 euve264608 inotify-spamlearn.py[7348]: config: path "/dev/null/.spamassassin/user_prefs" is inaccessible: Not a directory
    Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.euve264608.serverprofi24.de.7427 for /var/lib/spamassassin/bayes.lock: Permission denied
    Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.euve264608.serverprofi24.de.7427 for /var/lib/spamassassin/bayes.lock: Permission denied
    Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
    Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/72C00ABE4488476D8AF626120D868211.eml: Learned tokens from 0 message(s) (1 message(s) examined)
    Jan 12 16:13:14 euve264608 inotify-spamlearn.py[7348]: INFO Removing file: /var/lib/kopano/spamd/spam/72C00ABE4488476D8AF626120D868211.eml
    

    if i change only these entries from kopano to amavis
    /etc/systemd/system/inotify-spamlearn.service

    [Service]
    User=amavis
    Group=amavis
    

    then i get this messages:

    Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Starting inotify-spamlearn.py
    Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
    Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spam
    Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
    Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Inotify learning started
    Jan 12 16:20:41 Servername inotify-spamlearn.py[7501]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: bayes: expire_old_tokens: locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.Servername.serverprofi24.de.7506 for /var/lib/spamassassin/bayes.lock: Permission denied
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create tmp lockfile /var/lib/spamassassin/bayes.lock.Servername.serverprofi24.de.7506 for /var/lib/spamassassin/bayes.lock: Permission denied
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 500.
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml: Learned tokens from 0 message(s) (1 message(s) examined)
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: INFO Removing file: /var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Exception in thread Inotify Handling:
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Traceback (most recent call last):
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 80, in inotified
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     process('/'.join([watch_path, filename]), spamcmd, delete, initiator)
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 43, in process
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     os.remove(filename)
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: PermissionError: [Errno 13] Permission denied: '/var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml'
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: During handling of the above exception, another exception occurred:
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: Traceback (most recent call last):
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     self.run()
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/lib/python3.5/threading.py", line 862, in run
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     self._target(*self._args, **self._kwargs)
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 85, in inotified
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     i.remove_watch([spam_dir, ham_dir])
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/lib/python3.5/dist-packages/inotify/adapters.py", line 109, in remove_watch
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     wd = self.__watches.get(path)
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: TypeError: unhashable type: 'list'
    

    These are the amavis and kopano rights under /etc/groups

    amavis:x:123:clamav,kopano
    kopano:x:999:kapi,konnect,amavis
    


  • @BMWfan said in Define ham and spam for spamassassin:

    No, this solves it not. The problem is after i change the sa_group at /etc/kopano/spamd.cfg to amavis then i get permission errors in the kopano-spamd process as here described: Kopano spamd error - PermissionError: [Errno 1] Operation not permitted

    Maybe because the problem described by @fbartels in this post insn’t solved for you yet?

    Who is the owner/group and what are the permissions of the files in /var/lib/kopano/spamd/spam/ (just ls -la)?

    And please test the sa-learn command manually as the desired user before configuring it with inotify-spamlearn to see if everything works as expected.



  • @ashceryth thanks for your reponse and hints.
    Here is the suggested output:

    $ /var/lib/kopano/spamd/spam# ls -la
    total 8
    drwxr-xr-x 2 kopano kopano 4096 Jan 12 17:05 .
    drwxr-xr-x 4 kopano kopano 4096 Dec 27 10:32 ..
    


  • Well, at the moment your spam folder is empty. Normally the .eml files get deleted after learning by inotify-spamlearn. However, according to your log from above it wasn’t possible to remove them because of permission issues:

    @BMWfan said in Define ham and spam for spamassassin:

    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:   File "/usr/local/sbin/inotify-spamlearn.py", line 43, in process
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]:     os.remove(filename)
    Jan 12 16:21:00 Servername inotify-spamlearn.py[7501]: PermissionError: [Errno 13] Permission denied: '/var/lib/kopano/spamd/spam/A5D6C47FA96743C1B2DA0FE7203DA793.eml'
    

    That’s why I’m asking. Would be interesting to see the permissions of the .eml files themselves.



  • @ashceryth i stopped in the inotify-spamlearn service and here is the suggested output:

    $ /var/lib/kopano/spamd/spam# ls -la
    total 12
    drwxr-xr-x 2 kopano kopano 4096 Jan 14 20:39 .
    drwxr-xr-x 4 kopano kopano 4096 Dec 27 10:32 ..
    -rw-rw-rw- 1 kopano kopano 3643 Jan 14 20:39 600F5F44181B4F2B865EAE2A45EC2573.eml
    

Log in to reply