kopano-spamd (8.6.2) is broken

irreleph4nt

Spamd - when compiled from source (master) - currently does not work. When I move a message that contains SPAM from the SPAM to inbox folder and back, I see the following in the logs:

2018-06-05 22:59:40,957 - spamd - INFO - starting spamd
2018-06-05 23:00:14,301 - spamd - ERROR - could not process change for entryid 0000000078FD421F260F48A48B41805356BC77EE01000000050000005545D8E8E5E74323AE02A2EAF2428AC400000000 ([SPropValue(0x67110003, 150), SPropValue(0x6715000A, 2147746063), SPropValue(0x0FFA0102, b'x\xfdB\x1f&\x0fH\xa4\x8bA\x80SV\xbcw\xee')]):
2018-06-05 23:00:14,304 - spamd - ERROR - Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/kopano/ics.py", line 118, in ImportMessageChange
    self.importer.update(item, flags)
  File "/usr/lib/python3.6/site-packages/kopano_spamd/__init__.py", line 82, in update
    self.learnham and self.was_spam(searchkey):
  File "/usr/lib/python3.6/site-packages/kopano_spamd/__init__.py", line 63, in was_spam
    return searchkey in db
  File "/usr/lib/python3.6/_collections_abc.py", line 666, in __contains__
    self[key]
  File "/usr/lib/python3.6/site-packages/bsddb3/__init__.py", line 239, in __getitem__
    return _DeadlockWrap(lambda: self.db[key])  # self.db[key]
  File "/usr/lib/python3.6/site-packages/bsddb3/dbutils.py", line 67, in DeadlockWrap
    return function(*_args, **_kwargs)
  File "/usr/lib/python3.6/site-packages/bsddb3/__init__.py", line 239, in <lambda>
    return _DeadlockWrap(lambda: self.db[key])  # self.db[key]
TypeError: Bytes or Integer object expected for key, str found

2018-06-05 23:00:18,366 - spamd - ERROR - could not process change for entryid 0000000078FD421F260F48A48B41805356BC77EE0100000005000000524D5EEDF1AD45EC9DC6F3C2479536F900000000 ([SPropValue(0x67110003, 150), SPropValue(0x6715000A, 2147746063), SPropValue(0x0FFA0102, b'x\xfdB\x1f&\x0fH\xa4\x8bA\x80SV\xbcw\xee')]):
2018-06-05 23:00:18,367 - spamd - ERROR - Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/kopano/ics.py", line 118, in ImportMessageChange
    self.importer.update(item, flags)
  File "/usr/lib/python3.6/site-packages/kopano_spamd/__init__.py", line 72, in update
    item.header(headertag).upper() != 'YES':
NameError: name 'headertag' is not defined

/etc/kopan/spam.cfg:

##############################################################
# SPAMD SERVICE SETTINGS

# run as specific user
run_as_user         = kopano

# run as specific group
run_as_group        = kopano

# control pid file
pid_file            =   /var/run/kopano/spamd.pid

# run server in this path (when not using the -F switch)
running_path = /var/lib/kopano

##############################################################
# LOG SETTINGS

# Logging method (syslog, file)
log_method          =   file

# Loglevel (0(none), 1(crit), 2(err), 3(warn), 4(notice), 5(info), 6(debug))
log_level           =   6

# Logfile for log_method = file, use '-' for stderr
log_file            =   /var/log/kopano/spamd.log

# Log timestamp - prefix each log line with timestamp in 'file' logging mode
log_timestamp       =   1

###############################################################
# SPAMD Specific settings

# The dir where spam mails are written to which are later picked up
# by the sa-learn program
spam_dir = /var/lib/kopano/spamd/spam

# Location for the database containing metadata on learned spam
spam_db = /var/lib/kopano/spamd/spam.db

# Learn ham, when the user moves emails from junk to inbox,
# enabled by default.
learn_ham = yes

# The dir where ham mails are written to which are later picked up
# by the sa-learn program
ham_dir = /var/lib/kopano/spamd/ham

# Spamassassin group
sa_group = amavis

# Header tag for spam emails
header_tag = X-Spam-Flag

server_socket = file:///var/run/kopano/server.sock

fbartels

Hi @irreleph4nt ,

in your headline you said 8.6.2, but in the first sentence say you compiled from master. Which version are you using exactly?

It seems this is linked to your usage of python 3.6. I tried to reproduce it on a system where we compile for 2.7 and did not get this error. Can you give it a quick check with an older python release as well?

irreleph4nt

Hi Felix,

I am sorry for not being clear. When the announcement came out that 8.6.2 was ready, I compiled from the source code hosted on your stash. I hence „tagged“ my package as 8.6.2. Hope that makes sense :S

I assume to check with phyton2 I‘d have to re-compile core? Can do but that won’t be quick. What‘s spamd‘s purpose and is it useful even when I have clamav and spamassassin installed? I assume it learns similar to SA. Just thinking if I really need it and whether going back to python2 is worth it.

fbartels

@irreleph4nt said in kopano-spamd (8.6.2) is broken:

I am sorry for not being clear. When the announcement came out that 8.6.2 was ready, I compiled from the source code hosted on your stash. I hence „tagged“ my package as 8.6.2. Hope that makes sense :S

Ah, then you should have a closer look at the different branches and tags on stash. If you are building from master, then you have something that is based on the 8.6.80 tag. If you want to compile one of our releases you should look in the related release branch kc-8.6.x and there build the latest tag.

@irreleph4nt said in kopano-spamd (8.6.2) is broken:

I assume to check with phyton2 I‘d have to re-compile core?

No, for the Python components this is afaik not necessary. We just set the shebang during packaging to match the correct pyhton version, so you’d just have to change it to something else.

@irreleph4nt said in kopano-spamd (8.6.2) is broken:

What‘s spamd‘s purpose

Its a helper tool to export messages that users move into their junk email folder and that you then (with a separate script) feed as spam or ham to spamassassin (or your other preferred solution that is capable of learning from eml files).

weini

I´m facing the same problem with a current nightly build (core-8.6.80.1055_0+156-Debian_9.0-amd64). I definitely don´t use python 3.x but python 2.7 instead.

Moving an email to junk leads to the following log:

Jun 27 17:06:58 nas2 kopano-spamd[4004]: 2018-06-27 17:06:58,704 - spamd - ERROR - could not process change for entryid 0000000056286673185E4F68AD47C3C76D4A4F9A01000000
05000000357950C9DE80412A930F0A0D9C8FE41400000000 ([SPropValue(0x67110003, 1125145L), SPropValue(0x67150003, 1004340L), SPropValue(0x0FFA0102, 'V(fs\x18^Oh\xadG\xc3\xc7m
JO\x9a')]):
Jun 27 17:06:58 nas2 kopano-spamd[4004]: 2018-06-27 17:06:58,729 - spamd - ERROR - Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kopano/ics.py", line 118, in ImportMessageChange
    self.importer.update(item, flags)
  File "/usr/lib/python2.7/dist-packages/kopano_spamd/__init__.py", line 81, in update
    elif item.folder == item.store.inbox and \
  File "/usr/lib/python2.7/dist-packages/kopano/store.py", line 194, in inbox
    return _folder.Folder(self, _benc(self.mapiobj.GetReceiveFolder(u'IPM', MAPI_UNICODE)[0]))
  File "/usr/lib/python2.7/dist-packages/MAPICore.py", line 611, in GetReceiveFolder
    return _MAPICore.IMsgStore_GetReceiveFolder(self, lpszMessageClass, ulFlags)
MAPIErrorNoSupport: MAPI error 80040102 (MAPI_E_NO_SUPPORT)

cburghardt

I don’t really get that logic. Kopano supports python3 if I am not mistaken. I am using packages for Suse Leap and they are only provided for python3. I get the same issue as above but already since 8.6.0.
Is there a workaround?

cburghardt

So what I did is to edit kopano_spamd/init.py and to convert the searchkey to binary:
self.learnham and self.was_spam(b’searchkey’)
self.mark_spam(b’searchkey’)

weini

Thanks for the hint!
I confirm, most of the error messages are gone, but not all:

Jul 26 08:22:36 nas2 kopano-spamd[25246]: 2018-07-26 08:22:36,940 - spamd - ERROR - could not process change for entryid 000000008D071BEA942F42619CEB29F47F6CC4CA010000000500000053F76E18C6A243A4A81AF24F0DB20A6900000000 ([SPropValue(0x67110003, 1130852L), SPropValue(0x67150003, 201L), SPropValue(0x0FFA0102, ‘\x8d\x07\x1b\xea\x94/Ba\x9c\xeb)\xf4\x7fl\xc4\xca’)]):
Jul 26 08:22:36 nas2 kopano-spamd[25246]: 2018-07-26 08:22:36,991 - spamd - ERROR - Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/kopano/ics.py”, line 118, in ImportMessageChange
self.importer.update(item, flags)
File “/usr/lib/python2.7/dist-packages/kopano_spamd/init.py”, line 72, in update
item.header(headertag).upper() != ‘YES’:
NameError: global name ‘headertag’ is not defined

However, I don´t get anything placed into “/var/lib/kopano/spamd/[ham|spam]”

fbartels

@weini said in kopano-spamd (8.6.2) is broken:

NameError: global name ‘headertag’ is not defined

this has been fixed for a month already (in https://stash.kopano.io/projects/KC/repos/kopanocore/commits/dc15ffc9334b228b54539796e175c9cf7fd24f46) you should update your version.

weini

Thanks for the hint! Upgraded to build 1248 and added the two tweaks from cburghardt mentioned above.
Now I´m getting:

Jul 28 08:42:54 nas2 kopano-spamd[53301]: 2018-07-28 08:42:54,200 - spamd - ERROR - could not process change for entryid 0000000056286673185E4F68AD47C3C76D4A4F9A01000000050000000211D3BEB2E64C178164EE317BA80F8F00000000 ([SPropValue(0x67110003, 1131213L), SPropValue(0x67150003, 121928L), SPropValue(0x0FFA0102, ‘V(fs\x18^Oh\xadG\xc3\xc7mJO\x9a’)]):
Jul 28 08:42:54 nas2 kopano-spamd[53301]: 2018-07-28 08:42:54,200 - spamd - ERROR - Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/kopano/ics.py”, line 118, in ImportMessageChange
self.importer.update(item, flags)
File “/usr/lib/python2.7/dist-packages/kopano_spamd/init.py”, line 81, in update
elif item.folder == item.store.inbox and
File “/usr/lib/python2.7/dist-packages/kopano/store.py”, line 194, in inbox
return _folder.Folder(self, _benc(self.mapiobj.GetReceiveFolder(u’IPM’, MAPI_UNICODE)[0]))
File “/usr/lib/python2.7/dist-packages/MAPICore.py”, line 611, in GetReceiveFolder
return _MAPICore.IMsgStore_GetReceiveFolder(self, lpszMessageClass, ulFlags)
MAPIErrorNoSupport: MAPI error 80040102 (MAPI_E_NO_SUPPORT)
Jul 28 09:36:17 nas2 kopano-spamd[53301]: 2018-07-28 09:36:17,180 - spamd - ERROR - could not process change for entryid 0000000056286673185E4F68AD47C3C76D4A4F9A01000000050000001E97A8A533C14154A28E4F33C1C8BDE700000000 ([SPropValue(0x67110003, 1131221L), SPropValue(0x67150003, 121928L), SPropValue(0x0FFA0102, ‘V(fs\x18^Oh\xadG\xc3\xc7mJO\x9a’)]):
Jul 28 09:36:17 nas2 kopano-spamd[53301]: 2018-07-28 09:36:17,181 - spamd - ERROR - Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/kopano/ics.py”, line 118, in ImportMessageChange
self.importer.update(item, flags)
File “/usr/lib/python2.7/dist-packages/kopano_spamd/init.py”, line 81, in update
elif item.folder == item.store.inbox and
File “/usr/lib/python2.7/dist-packages/kopano/store.py”, line 194, in inbox
return _folder.Folder(self, _benc(self.mapiobj.GetReceiveFolder(u’IPM’, MAPI_UNICODE)[0]))
File “/usr/lib/python2.7/dist-packages/MAPICore.py”, line 611, in GetReceiveFolder
return _MAPICore.IMsgStore_GetReceiveFolder(self, lpszMessageClass, ulFlags)
MAPIErrorNoSupport: MAPI error 80040102 (MAPI_E_NO_SUPPORT)

So still no luck, it´s getting frustrating.
Is there some wiki page or doc how to track such issues down?

Spiff

Hi,

Upgraded to latest 8.6 (nightly) today, and have a similar problem.
In my case it seems like the item.header(self.headertag) fails into a NoneType object that does not support upper().
Get the following error whenever I move a message into the Junk folder:

2018-07-28 16:55:27,197 - spamd - ERROR - could not process change for entryid 000000003872FA2265094788A871E21E74BB150101000000050000004FE6C1B1D13743A0ABFB9AB828B4FCC100000000 ([SPropValue(0x67110003, 148806L), SPropValue(0x67150003, 134L), SPropValue(0x0FFA0102, ‘8r\xfa"e\tG\x88\xa8q\xe2\x1et\xbb\x15\x01’)]):
2018-07-28 16:55:27,197 - spamd - ERROR - Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/kopano/ics.py”, line 118, in ImportMessageChange
self.importer.update(item, flags)
File “/usr/lib/python2.7/site-packages/kopano_spamd/init.py”, line 69, in update
item.header(self.headertag).upper() != ‘YES’:
AttributeError: ‘NoneType’ object has no attribute ‘upper’

Update:
Removed “.upper()” from the script for testing, and got past this problem.
Obviously when the header does NOT contain “X-Spam-Flag”, an empty object is returned, hence it breaks the code here.
-Please update code to avoid this issue (e.g. test validity and default to “NO” if property is not found)

Second observation, amavis seems to almost be a requirement. -I’m using Spamassassin with clamav. -What should i set the sa-group to when amavis doesn’t exist?

mark dufour

hi all,

sorry for the problems you are having. I will take a look shortly at the mentioned issues.

the header problem, should be easy to solve
running spamd with python3, as now happens in master
the GetReceiveFolder issue

I wouldn’t use b’searchkey’, as this results in the literal word ‘searchkey’…

mark dufour

hi all,

quick update: I made a PR which should address above issues. it should land in master within a few days.

we now treat a missing header as signal the mail should be seen as spam
made some python3 fixes, and made sure our tests test spamd with python2 and python3
made store.inbox return None for public stores, avoiding the GetReceiveFolder issue

will post an update once this PR makes it through QA.

mark.

weini

Hi Mark!
It´s really great to see progress here, after this was kind a dead end for some months.
Many thanks for taking care.

Small comment on the default handling if the spam tag is missing: Best option would be to have this configurable via the cfg file. While your proposal is obviously the best approach to be on the safe side, there are several spam solution besides amavis that only add the tag on spam emails and leave ham emails untouched.

Chris

mark dufour

let’s first check if everything is better now. the patches have been merged to our 8.7 branch. I will let you know once they hit master.