Navigation

    Kopano
    • Register
    • Login
    • Search
    • Categories
    • Get Official Kopano Support
    • Recent
    Statement regarding the closure of the Kopano community forum and the end of the community edition

    need help with spam filter

    Kopano Groupware Core
    3
    4
    1300
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • pixel
      pixel last edited by

      Hi@all,

      i use Kopano on a UCS member server (4.4-6 errata803) Kopano is installed in the versions

      WebApp: 4.6.3.0+155.1
Kopano Core: 8.7.16
Z-Push: 2.6.0+0-0

      For the setup of kopano-spamd I followed these hints:

      https://documentation.kopano.io/kopanocore_administrator_manual/special_kc_configurations.html#configuring-kopano-spamd-for-automatic-spam-ham-learning

      https://github.com/bkram/inotify-spamlearn

      My steps of the installation:

      univention-install kopano-spamd
usermod -aG amavis kopano
univention-install python3-pip
pip3 install inotify
cp /etc/kopano/spamd.cfg /etc/kopano/spamd.cfg.org

      Edit /etc/kopano/spamd.cfg:

      run_as_group = amavis
learn_ham = yes

      The entire file:

      # run as specific user
#run_as_user         = amavis
# run as specific group
run_as_group        = amavis
# control pid file
#pid_file            =   /var/run/kopano/spamd.pid
# run server in this path (when not using the -F switch)
#running_path = /var/lib/kopano
# LOG SETTINGS
# Logging method (syslog, file)
log_method          =   syslog
# Loglevel (0(none), 1(crit), 2(err), 3(warn), 4(notice), 5(info), 6(debug))
#log_level           =   3
# Logfile for log_method = file, use '-' for stderr
#log_file            =   /var/log/kopano/spamd.log
# Log timestamp - prefix each log line with timestamp in 'file' logging mode
#log_timestamp       =   1
#spam_dir = /var/lib/kopano/spamd/spam
#spam_db = /var/lib/kopano/spamd/spam.db
# Learn ham, when the user moves emails from junk to inbox,
# enabled by default.
learn_ham = yes
# The dir where ham mails are written to which are later picked up
# by the sa-learn program
#ham_dir = /var/lib/kopano/spamd/ham
# Spamassassin group
sa_group = amavis
# Header tag for spam emails
#header_tag = X-Spam-Flag

      Files download from https://github.com/bkram/inotify-spamlearn:

      • inotify-spamlearn.cfg
      • inotify-spamlearn.py
      • inotify-spamlearn.service

      [inotify-spamlearn.cfg]

      [paths]
# path to look for spam emls
spam_dir = /var/lib/kopano/spamd/spam
# path to look for ham emls
ham_dir = /var/lib/kopano/spamd/ham

[spam]
# command to pipe the spam into
spamcmd = /usr/bin/sa-learn --spam
# command to pipe the ham into
hamcmd = /usr/bin/sa-learn --ham

[mode]
# should we delete emls after processing
delete = True
# should we scan and process existing spam/ham in the watch directories
scan = True
# after scanning and processing of existing spam/ham exit the script. (crontab mode)
oneshot = False

[logging]
# levels possible are DEBUG INFO WARN ERROR CRITICAL
loglevel = INFO
# empty means log to console, use journalctl to read the logs when run from systemd service
logfile =

      [inotify-spamlearn.service]

      [Unit]
Description=Inotify Spamlearn

[Service]
User=amavis
Group=amavis
Type=simple
ExecStart= /usr/local/sbin/inotify-spamlearn.py
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

      I have not modified the file inotify-spamlearn.py

      The next steps:

      cp inotify-spamlearn.cfg /etc/kopano
cp inotify-spamlearn.py /usr/local/sbin/
cp inotify-spamlearn.service /etc/systemd/system
chmod a+x /usr/local/sbin/inotify-spamlearn.py
chown -R kopano:amavis /var/lib/kopano/spamd
chmod g+w /var/lib/kopano/spamd/spam
chmod g+w /var/lib/kopano/spamd/ham
systemctl daemon-reload
      • Activate and start *
      systemctl enable inotify-spamlearn
systemctl start inotify-spamlearn
      • Check status *
      root@com01:~# systemctl status inotify-spamlearn
● inotify-spamlearn.service - Inotify Spamlearn
   Loaded: loaded (/etc/systemd/system/inotify-spamlearn.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-11-18 08:49:44 CET; 50min ago
 Main PID: 818 (python3)
    Tasks: 2 (limit: 4915)
   Memory: 11.5M
      CPU: 338ms
   CGroup: /system.slice/inotify-spamlearn.service
           └─818 python3 /usr/local/sbin/inotify-spamlearn.py

Nov 18 08:49:44 com01 systemd[1]: Started Inotify Spamlearn.
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Starting inotify-spamlearn.py
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spam
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Inotify learning started
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham

      to see what happens if I mark a mail as spam in the inbox and as HAM in the junk folder

      journalctl -u inotify-spamlearn -f
      • Example 1 *
      • I get about 3-6 spam mails a day. These always come from the same sender. But only in the field “FROM” there is always the same sender. But if you look at the mail in detail, there is a different address in the sender address each time.

      Here is the header of the file as it arrived today:

      Return-Path: <de@have.pecces.com>
Received: from com01.domain.internal (127.0.0.1:55520)
	by com01 (kopano-dagent) with LMTP;
	Sun, 15 Nov 2020 08:06:45 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
	by com01.domain.internal (Postfix) with ESMTP id DC20EDE1CF
	for <me@mydomain.de>; Sun, 15 Nov 2020 08:06:45 +0100 (CET)
X-Virus-Scanned: by amavisd-new-2.10.1 (20141025) (Debian) at domain.internal
X-Spam-Flag: NO
X-Spam-Score: 0.011
X-Spam-Level:
X-Spam-Status: No, score=0.011 tagged_above=-1000 required=5
	tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1,
	HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.1, SPF_HELO_NONE=0.001,
	SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled
Received: from com01.domain.internal ([127.0.0.1])
	by localhost (com01.domain.internal [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id GEK0ILSyLIpX for <me@mydomain.de>;
	Sun, 15 Nov 2020 08:06:44 +0100 (CET)
Received: from com01.domain.internal (localhost [127.0.0.1])
	by com01.domain.internal (Postfix) with ESMTP id A9C07DF703
	for <me@mydomain.de>; Sun, 15 Nov 2020 08:06:44 +0100 (CET)
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on
	dd44626.provider.com
X-Original-To: me.mueller@domain2.de.de
Delivered-To: ispuser@dd44626.provider.com
X-Greylist: delayed 7204 seconds by postgrey-1.34 at dd44626; Sun, 15 Nov 2020 08:06:02 CET
Authentication-Results: dd44626.provider.com;
	dkim=pass (1024-bit key; unprotected) header.d=pecces.com header.i=de@have.pecces.com header.b="X1w/g76p";
	dkim-atps=neutral
X-policyd-weight: using cached result; rate: -6.1
Received: from w0193f8d.provider.com [11.22.33.44]
	by com01.domain.internal with POP3 (fetchmail-6.3.26)
	for <me@mydomain.de> (single-drop); Sun, 15 Nov 2020 08:06:44 +0100 (CET)
Received: from host0.pecces.com (host0.pecces.com [146.59.199.89])
	by dd44626.provider.com (Postfix) with ESMTPS id EDA0AC03B22
	for <me.mueller@domain2.de.de>; Sun, 15 Nov 2020 08:06:02 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=mailer; d=pecces.com;
 h=Date:To:From:Reply-To:Subject:Message-ID:List-Unsubscribe:MIME-Version:
 Content-Type; i=de@have.pecces.com;
 bh=O6bTbsinAi8Lljk8YJgeSknxbDoZwYWF+GArhTsbnsI=;
 b=X1w/g76piY0LOrA0n6hLqRgnIMsrLaRHMzvhun0NmWrea5DSgKtey6hBUAKL0tig6fowuuhK8odx
   KN3B2iNREQruXe0ar9ZsAE/ZYuGYfbwAjdrmoCmZVMOgDJ8aw/H4n5+gD9e6QXwdqcDP6vyZt4pF
   AHl/QyxMOYm/PmYRPYE=
Date: Sun, 15 Nov 2020 13:05:56 +0800
To: me.mueller@domain2.de.de
From: Bellgs <de@have.pecces.com>
Reply-To: Bellgs <de@have.pecces.com>
Subject: =?UTF-8?Q?Etwas_muss_sich_=C3=A4ndern.?=
Message-ID: <ecdcd70f2a15ac1442aa8ffd634096b7@iZ0xi729wodf0k0eja0uqoZ>
X-Mailer: Sendy (https://sendy.co)
List-Unsubscribe: <http://r.sconfit.com/sendy/unsubscribe/Kwq81Rnr6gb44OU8IIT4um2gzwpGQy4AxPJZQSAIBjo/lWs5rtSyFNAsAaeQVJfpUA/1UXUIFyeVsCYfvqXYIy892kQ>
MIME-Version: 1.0
Content-Type: text/html; charset=UTF-8
X-KasLoop: ispuser

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head><meta name="ROBOTS" content="NOINDEX, NOFOLLOW"><meta name="referrer" content="no-referrer">
	<title></title>
	<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta name="viewport" content="initial-scale=1.0"><!-- So that mobile webkit will display zoomed in --><meta name="format-detection" content="telephone=no"><!-- disable auto telephone linking in iOS -->
</head>

      If I mark them as spam in Kopano the following happens on the server:

      -- Reboot --
Nov 18 08:49:44 com01 systemd[1]: Started Inotify Spamlearn.
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Starting inotify-spamlearn.py
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Looking for existing files in /var/lib/kopano/spamd/spam
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Finished looking for existing files in /var/lib/kopano/spamd/spam
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Looking for existing files in /var/lib/kopano/spamd/ham
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Inotify learning started
Nov 18 08:49:47 com01 inotify-spamlearn.py[818]: INFO Finished looking for existing files in /var/lib/kopano/spamd/ham
Nov 18 09:58:05 com01 inotify-spamlearn.py[818]: INFO Processing [Inotify] /var/lib/kopano/spamd/spam/B20D94D163FC40BE869DABE867ED020F.eml: Learned tokens from 1 message(s) (1 message(s) examined)
Nov 18 09:58:05 com01 inotify-spamlearn.py[818]: INFO Removing file: /var/lib/kopano/spamd/spam/B20D94D163FC40BE869DABE867ED020F.eml

      It looks to me like the spam filter uses this mail to learn. I mark these mails every day but the recognition does not get better.

      • Example 2 * but on a different server. Exactly the same UCS and Kopano version and exactly the same configuration of the components. This mail is a newsletter to which the user has subscribed.
      Return-Path: <musikblitz@musikpressedienst.de>
Received: from com01.foobar.internal (127.0.0.1:34188)
	by com01 (kopano-dagent) with LMTP;
	Thu, 10 Sep 2020 09:48:53 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
	by com01.foobar.internal (Postfix) with ESMTP id 2FF14803C55A
	for <max.mustermann@company.de>; Thu, 10 Sep 2020 09:48:53 +0200 (CEST)
X-Virus-Scanned: by amavisd-new-2.10.1 (20141025) (Debian) at foobar.internal
X-Spam-Flag: YES
X-Spam-Score: 5.211
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.211 tagged_above=-1000 required=5
	tests=[HDRS_LCASE_IMGONLY=0.1, HELO_MISC_IP=0.174,
	HTML_IMAGE_ONLY_08=1.781, HTML_MESSAGE=0.001, RDNS_NONE=1.274,
	SPF_NONE=0.001, TVD_SPACE_RATIO=0.001, TVD_SPACE_RATIO_MINFP=1.878,
	URIBL_BLOCKED=0.001] autolearn=disabled
Received: from com01.foobar.internal ([127.0.0.1])
	by localhost (com01.foobar.internal [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id zmBU9lTRm3QS for <max.mustermann@company.de>;
	Thu, 10 Sep 2020 09:48:49 +0200 (CEST)
Received: from com01.foobar.internal (localhost [127.0.0.1])
	by com01.foobar.internal (Postfix) with ESMTP id 2F18E801515C
	for <max.mustermann@company.de>; Thu, 10 Sep 2020 09:48:49 +0200 (CEST)
Received: from pop.provider.de [212.227.15.171]
	by com01.foobar.internal with POP3 (fetchmail-6.3.26)
	for <max.mustermann@company.de> (single-drop); Thu, 10 Sep 2020 09:48:49 +0200 (CEST)
Received: from [217.72.192.67] ([217.72.192.67]) by mx.kundenserver.de
 (mxeue109 [217.72.192.67]) with ESMTPS (Nemesis) id 1MtvMg-1kWZFA35S8-00uGlT
 for <max.mustermann@company.de>; Thu, 10 Sep 2020 09:48:04 +0200
Received: from mail6.melting-mind.de ([195.226.185.125]) by mx.kundenserver.de
 (mxeue109 [217.72.192.67]) with ESMTPS (Nemesis) id 1MHH8g-1kKh9A2yki-00DIXb
 for <max.mustermann@company-edition.de>; Thu, 10 Sep 2020 09:48:04
 +0200
Received: from localhost (localhost [127.0.0.1])
	by mail6.melting-mind.de (Postfix) with ESMTP id 045AA9A4C5F
	for <max.mustermann@company-edition.de>; Thu, 10 Sep 2020 09:48:02 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at mail6.melting-mind.de
Received: from mail6.melting-mind.de ([127.0.0.1])
	by localhost (mail6.melting-mind.de [127.0.0.1]) (amavisd-new, port 10026)
	with ESMTP id j-1cRs-Tl7fR
	for <max.mustermann@company-edition.de>;
	Thu, 10 Sep 2020 09:48:01 +0200 (CEST)
Received: from [192.168.1.77] (p5de1c80b.dip0.t-ipconnect.de [93.225.200.11])
	(Authenticated sender: blitz@musikpressedienst.de)
	by mail6.melting-mind.de (Postfix) with ESMTPA id 9E4CA9A4C7A
	for <max.mustermann@company-edition.de>; Thu, 10 Sep 2020 09:47:57 +0200 (CEST)
Message-Id: <Dpt5N0WLhq0q4gUWfLgrA92V0tm2t2WRQSc7K6CJ9OcF@musikpressedienst.de>
Mime-Version: 1.0
From: musikblitz@musikpressedienst.de
To: =?iso-8859-1?Q?"J=FCrgen_Rinschler"?= <max.mustermann@company-edition.de>
Subject: musikpressedienst meldet
Date: Thu, 10 Sep 2020 09:47:58 +0200
X-Bounce-Tracking-Info: <SsO8cmdlbglSaW5zY2hsZXIJQmVsbGEgTXVzaWNhCWp1ZXJnZW4ucmluc2NobGVyQGJlbGxhLW11c2ljYS1lZGl0aW9uLmRlCW11c2lrcHJlc3NlZGllbnN0IG1lbGRldAk0ODYwCW11c2lrYmxpdHoJMTIxCWJvdW5jZQlubwlubw==>
Content-type: multipart/alternative;
     Boundary="--=BOUNDARY_910947_JFCH_JLXH_HXPI_SCJI"
Envelope-To: <max.mustermann@company.de>
X-Spam-Flag: NO
X-UI-Filterresults: notjunk:1;V03:K0:JvLY7sqQ6mU=:r8p0Ic1Ye3TvUIG8q3fbwThFnE
 y247+brhcqVkgfDhO1QXB6iy9i+xeCHqpqa7fvN8FGvl1ksim3QevEFtvpFJ+LKpH6FIuu0sO
 ycUV5lfOePcODRtT92tSl5fWFPhiBE5cVu7nmhwbNijGqGEnzQUpbkYuhGcw8IT4C8pgPsmbC
 S3oRrHvTSPqovQ8TRqH1o44nnbfDpbaZmNJ2/8CSWHtG4tIirQVxEKt3L88ZHh2tjJ92OOVK4
 QYvRPFq15ang1Opq4OtXAdoYJB1Ne6bMgCOBK5wHg4hBmoevl7nNGM3BXtkPr4iG/YYeGVVmS
 72NvUl1t36kepqQb8HAZqjL/V5KpPR3h6zyUR+Ywbx0hSBwRooxGWFHdR/Sw0BDFOByeQeqL6
 XA4I8WLVaEPIB4sOWTuOmVXkBb0C1D24k6cj0KeHGOBpMyErsBmaBrwWLpEOEwGiRbi5W90+s
 4sbYT5yV9IxFoBYJQKKejEthjRBsYU98o4xG+gQI2lSpFmQem7qmbSobbF428ZnETcYov7K8X
 HiTAxYr0YS1Atz0aCquUXLTezB8LtplnLuDoWl0wJ/CxQg9TvUTgCKXIQ0/1IyXOfbz48c8/o
 p1Fdns76klSbezzbtrof7Eg7KqMM+D9/UNRwUtfHz8gqhTBMAi/Mj+znXPZMG3RttG2c26lkZ
 jfwaop3dZPTcD/qAusGMBL5pa0lApNo/8zO6M3PqUSQLJZJACvJS+bzam/wS9EJ3lsEmXbv0C
 ETJ1a6FqZ4MaGY2NPoOqCfQI7miPrc7+u6SdRQ5c/lB6UuqZO/HjY03RgN5yFLPs4ymnXlq3I
 A+K21Dm98JELZy0m0ym9+Q3dzea4sC9lG2rAxKWIQCRX7va3m1dnp8sE06hEODDmK1Wb2ff/C
 a1QnQukhPl4sTlediDLA2tulTLuxVCJ/mJGCi56v8gb7aJK2pcbmiID+PWoVPRtXJYNM03lwk
 O10iJYPXhEkzW9RUzdsE831ItZRAlyCST87mAcZ3Q9YwjsgvDpQzDFEsxJs7QQ6hBjIbnYHis
 IC7Niq3HnUZrsEcgQHaBR6iPemrEoMiKlaCCDNQ+IF9dWAeIZz+i0Zc+CHLazt0VB40gWJt9r
 OtKfx+ArKldD9/URZ+Z+4uDunhZRE7S/+40Rw36iI/ta2sN+hJqvP/hhUWxhekWcKvVDj9YDM
 84AgT7dbeXGX4ZGiJr+WmsMDdx8oXyy4ZPZ9jib3qfIbr823fri5uZDygDv40/q8i7xM91Yk7
 4RQHDpFHBHSFLRa2uPdFxEIaB9iBvRoWpVZdp8acKzs85GZEd/9QOVC/T4T/RLvR1RrD4Irke
 ZVqwPbfEnMAzY8uG22X38iJASk+wuHJOLbfPmBlKsqYiAYP0kCKVLG630nhwoVP+pp+MM7sym
 LqZ4DRVFzFhHS2VepON/fyTpMqlTp/BQdTAv9llhccUmBl+Hj3eH+yfLDP4jMICBPfl7mm890
 nzOHB+yubumJiGDF3CxMihQEd75rleMYj5fKKRjt0IOLPLH37+gIpqOn8EwS3Or2ZIFwZ7l+j
 GpBLJ16SKTJQ86zCIZ+XcdcqvfKWa1eadqHS/GHfBzHIi7/vlaAFMdzc2sU3Ri2dHDaceF0I/
 aYOxJi2sJFQSHW4WyLBO8EhgqmjSMCJyFZ8UBUJp2iXp7IeD4fADjuhQas8xjqGnQFqZn/xhx
 p+aSdnOHWcBR3QPgOV/5/Y/IDNZ+JTvKpibp230owyAVkx02KiQeuIohI58/wMfSkcP9byv62
 p4MFTs6bWNL34L63fU2ne5R9P/coUNSuUI4nwMUk5XBAFOMSb47MG9OF5btk6oV1aVto67WCL
 RTywqgwO/69NQYX1JidOPc3ed5aHt42hpFlsob9JKPlxyoyEqhUWqnVpruF0A2CaeZu5CazTJ
 h2VrczVum9yBIQkKkvtYD9yv+o9S8nFFVbxQPT9coBo5cGISb3qjVFV7FZVopcPuqRdourcju
 byUnvcyZA==

Die Nachricht ist im MIME-Format. Da Ihr E-Mail-Programm dieses Format nicht
unterstützt, werden einzelne oder alle Nachrichten nicht lesbar sein.

----=BOUNDARY_910947_JFCH_JLXH_HXPI_SCJI
Content-type: text/plain;
     charset=iso-8859-1; format=flowed
Content-transfer-encoding: quoted-printable


16=2ESeptember: Reeperbahn Festival startet - viel Programm
http://mailings=2Einfernoevents=2Ecom/m/13704303/
----=BOUNDARY_910947_JFCH_JLXH_HXPI_SCJI
Content-type: multipart/related;
     Boundary="--=BOUNDARY_910947_QDWV_HLHY_CYBN_XKVV"

----=BOUNDARY_910947_QDWV_HLHY_CYBN_XKVV
Content-type: text/html;
     charset=iso-8859-1
Content-transfer-encoding: quoted-printable

      When I move this mail from the Junk-Folder to the inbox nothing happens in the logfile.

      I have searched here in the forum but found no solution. What do I have to adjust to improve detection?

      with best
      pixel24

      tompf 1 Reply Last reply Reply Quote 0
      • tompf
        tompf @pixel last edited by

        @pixel
        Sorry I cant help you to improve detection, but I’d like to say thank you for summarizing all the relevant installation steps - I was looking for this quite a while …

        BR,
        Thomas

        pixel 1 Reply Last reply Reply Quote 0
        • pixel
          pixel @tompf last edited by

          @tompf You’re welcome :-)

          gentrix 1 Reply Last reply Reply Quote 0
          • gentrix
            gentrix @pixel last edited by

            @pixel I just set up spamd on Ubuntu and found that there’s a bug so that kopano-spamd creates the ham folder with root permissions so doesn’t have access to it after dropping priviledges. So the folder permissions have to be set manually to the correct user:group.

            1 Reply Last reply Reply Quote 0
            • First post
              Last post