Filter Rules disappearing

bmaehr

@lherrmann Is the problem fixed? Should I try with a new version?

compsos

Hi
We have seen the same issue of rules and signatures missing from accounts in 8.6.80 in both Debian Stretch and Jessie.
Will run those tests to see what we can find.

bmaehr

I’m really disappointed.

I found out what’s the problem, sent you the data for the testcase and 2 months later it is still not fixed…

Coffee_is_life

@bmaehr said in Filter Rules disappearing:

I’m really disappointed.

I found out what’s the problem, sent you the data for the testcase and 2 months later it is still not fixed…

greate you found the problem, since you did, would you share the solution?

best regards
coffee_is_life

jengelh

I have created about 20 rules in the WebApp. And after the next mail ist processed by dagent the rules disappeared in the WebApp. But they can be displayed via kopano-rules.py --list. So they are still available.

Well if kopano-rules.py shows any part of it and WebApp does not, that would initially point to WebApp.

And after they disappeared in the WebApp they don’t show up in the dagent log, so they aren’t used any more.
After I delete the last rule

If they are not shown in WebApp, there is nothing to delete.

they show up again in the WebApp and in the dagent log. But they aren’t used by dagent. Every log entry says: Rule “***” does not match: not found (8004010f)

If it is what I think it is, then: This message means that destination folder of the rule action was not found.

freyuh

Hello,

now I can’t even print the rules with the kopano-rules.py script.
I get the following error message:

Traceback (most recent call last):
File “/usr/local/bin/kopano-rules.py”, line 1110, in <module>
main()
File “/usr/local/bin/kopano-rules.py”, line 1104, in main
kopano_rule()
File “/usr/local/bin/kopano-rules.py”, line 885, in kopano_rule
printrules(filters, user, server)
File “/usr/local/bin/kopano-rules.py”, line 665, in printrules
condition_message = convertcondition(rule[4].Value)
File “/usr/local/bin/kopano-rules.py”, line 311, in convertcondition
totalconditions = len(conditions.lpRes)
TypeError: object of type ‘SPropertyRestriction’ has no len()

Ulrich.

jengelh

There is a new program, you can use kopano-ibrule -S (see manpage for the -u option too) instead.

freyuh

Hello,

hmm, kopano-rules was easier to read / use …

Ulrich.

robing

@freyuh ,

We rewrote kopano-rules so it can handle imports from exchange and listing rules was not a high priority at the moment.
We are still busy with it and are working on a fix for listing the rules at the moment .

Robin

freyuh

And once again, after adding several rules, all of them disappeared …
The WebApp shows no rules and kopano-ibrule -S shows nothing.

WebApp: 3.4.23.1909+83.1
Kopano Core: 8.7.80
Z-Push: 2.4.4+0-0

Ulrich.

ashceryth

@fbartels @robing

Hi guys,

This is still an issue and definitely a massive showstopper. Today, I completely reinstalled my setup on a new machine and migrated the database to the new host. After the db import I checked the WebApp and all rules were gone. To test with a “clean” db I deleted all rules for my user in WebApp on my old setup and exported the database again. So there were no rules in the db dump. After reimporting the db on the new host I recreated all rules (~25) manually via WebApp. The rules didn’t disappear while adding them. I also checked via kopano-ibrule command while adding the rules. Everything seemed to be fine. However, 1 hour later as some mails arrived that would be matched by one of the rules all of them are gone again! They are not visible via WebApp nor via kopano-ibrule command anymore.

@lherrmann, for me the behaviour is a 100% reproduceable.

I can’t send you my rules backup beacuse kopano-backup --only-meta -u <username> -f Inbox does not show any output in the rules file after the rules dissapeared. Trying to export the rules on the old host (using an db version where rules still existed) throws the following error:

kopano-backup --only-meta -u <username> -f Inbox
[error  ] gsoap connect: connect failed in tcp_connect()
2018-12-02 19:53:55,598 - backup0 - ERROR - Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kopano/log.py", line 89, in log_exc
    try: yield
  File "/usr/lib/python2.7/dist-packages/kopano_backup/__init__.py", line 878, in dump_rules
    path = store.folder(entryid=_hex(f.text.decode('base64'))).path
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 240, in path
    parent = parent.parent
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 173, in parent
    parent_eid = _benc(self._get_fast(PR_PARENT_ENTRYID))
  File "/usr/lib/python2.7/dist-packages/kopano/compat.py", line 101, in benc
    return s.encode(_BIN_ENCODING).strip().upper()
AttributeError: 'NoneType' object has no attribute 'encode'

2018-12-02 19:53:55,598 - backup0 - ERROR - Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kopano/log.py", line 89, in log_exc
    try: yield
  File "/usr/lib/python2.7/dist-packages/kopano_backup/__init__.py", line 878, in dump_rules
    path = store.folder(entryid=_hex(f.text.decode('base64'))).path
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 240, in path
    parent = parent.parent
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 173, in parent
    parent_eid = _benc(self._get_fast(PR_PARENT_ENTRYID))
  File "/usr/lib/python2.7/dist-packages/kopano/compat.py", line 101, in benc
    return s.encode(_BIN_ENCODING).strip().upper()
AttributeError: 'NoneType' object has no attribute 'encode'

When using python kopano-rules.py --user <username> --list the rules get listed but some of them have an empty condition. Those with the empty condition are filtering by the destination address. So maybe that causes the issue?

Please, let me provide anything else you need for digging into that. Just tell me what you need, please.

Debian 9.6
KC: 8.7.80.169-0+44.1
KW: 3.4.25.1980+1095.1
MariaDB Server: 10.1.37-0+deb9u1

Regards,
Patrick

ashceryth

I recreated 4 rules with conditions on sender address and content of subject only. Those seem to work, as they get applied and don’t disappear. So it seems it has to do with the condition of the rule…

jengelh

The only way this is gonna get resolved is to track the table delete/modify event and see what client it was issued from, i.e. custom build and diagnosis.

ashceryth

@jengelh, I think there is nothing deleted in the database as all rules reappear when using the same database with an older (working) version of kopano-core, as my tests showed about half a year ago. So the rules seem to simply becoming invisible.

Christoph.Goth

The Problem seems so occure when “dumping” the rules for the dagent (?) to process the mails while arriving.

I’v made the same discovery as @ashceryth, the rules get listed as long as no mail arrives. After the first mail being proccessed after editing the ruleset they disappear in the webapp and kopani-ibrules and none (!) get processed anymore. Using python kopano-rules.py i can list and modify them, after deleting any Rule-Entry - so the rule-set gets smaller again - they reappear and get prosessed again! Ther seems to be a point of summary rulesize where the problem beginns.

My rules are mostly based on “if senders-Adress contains X, move to folder Y”.
It doesnt mather, if i use one very big rule oder a few smaller ones, after some numbers of entrys the problem occures again and again, only solution is to delete a rule entry so the rest is working again…

jengelh

I smell a cache issue. (I tested and can reproduce now; kopano-srvadm --purge-cache=all makes it visible.)

ashceryth

@jengelh Great that you are on to something now. You mean kopano-srvadm --purge-cache=all makes the rules visible again?

ashceryth

I cleared the cache with kopano-srvadm --clear-cache=all after the rules disappeared (~20 rules) and they didn’t came back. With kopano-backup --only-meta -u <username> -f Inbox the rules file was empty but with SELECT h.owner, val_binary FROM properties AS p INNER JOIN hierarchy AS h ON p.hierarchyid=h.id WHERE tag=0x3FE1 all rules were listed in the output. So everything was still in the db. I can go up to about ~15 simple rules (if sender address is X move to folder Y) until the rules disappear.

freyuh

My rules went back after running ‘kopano-cli --clear-cache’

freyuh

Okay, my rules came back after the “clear-cache”, but only one of them is working: it’s the one with the ‘subject-rule’.
All with ‘sent-to-rules’ don’t work …