Filter Rules disappearing

jengelh

There is a new program, you can use kopano-ibrule -S (see manpage for the -u option too) instead.

freyuh

Hello,

hmm, kopano-rules was easier to read / use …

Ulrich.

robing

We rewrote kopano-rules so it can handle imports from exchange and listing rules was not a high priority at the moment.
We are still busy with it and are working on a fix for listing the rules at the moment .

Robin

freyuh

And once again, after adding several rules, all of them disappeared …
The WebApp shows no rules and kopano-ibrule -S shows nothing.

WebApp: 3.4.23.1909+83.1
Kopano Core: 8.7.80
Z-Push: 2.4.4+0-0

Ulrich.

ashceryth

@fbartels @robing

Hi guys,

This is still an issue and definitely a massive showstopper. Today, I completely reinstalled my setup on a new machine and migrated the database to the new host. After the db import I checked the WebApp and all rules were gone. To test with a “clean” db I deleted all rules for my user in WebApp on my old setup and exported the database again. So there were no rules in the db dump. After reimporting the db on the new host I recreated all rules (~25) manually via WebApp. The rules didn’t disappear while adding them. I also checked via kopano-ibrule command while adding the rules. Everything seemed to be fine. However, 1 hour later as some mails arrived that would be matched by one of the rules all of them are gone again! They are not visible via WebApp nor via kopano-ibrule command anymore.

@lherrmann, for me the behaviour is a 100% reproduceable.

I can’t send you my rules backup beacuse kopano-backup --only-meta -u <username> -f Inbox does not show any output in the rules file after the rules dissapeared. Trying to export the rules on the old host (using an db version where rules still existed) throws the following error:

kopano-backup --only-meta -u <username> -f Inbox
[error  ] gsoap connect: connect failed in tcp_connect()
2018-12-02 19:53:55,598 - backup0 - ERROR - Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kopano/log.py", line 89, in log_exc
    try: yield
  File "/usr/lib/python2.7/dist-packages/kopano_backup/__init__.py", line 878, in dump_rules
    path = store.folder(entryid=_hex(f.text.decode('base64'))).path
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 240, in path
    parent = parent.parent
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 173, in parent
    parent_eid = _benc(self._get_fast(PR_PARENT_ENTRYID))
  File "/usr/lib/python2.7/dist-packages/kopano/compat.py", line 101, in benc
    return s.encode(_BIN_ENCODING).strip().upper()
AttributeError: 'NoneType' object has no attribute 'encode'

2018-12-02 19:53:55,598 - backup0 - ERROR - Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kopano/log.py", line 89, in log_exc
    try: yield
  File "/usr/lib/python2.7/dist-packages/kopano_backup/__init__.py", line 878, in dump_rules
    path = store.folder(entryid=_hex(f.text.decode('base64'))).path
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 240, in path
    parent = parent.parent
  File "/usr/lib/python2.7/dist-packages/kopano/folder.py", line 173, in parent
    parent_eid = _benc(self._get_fast(PR_PARENT_ENTRYID))
  File "/usr/lib/python2.7/dist-packages/kopano/compat.py", line 101, in benc
    return s.encode(_BIN_ENCODING).strip().upper()
AttributeError: 'NoneType' object has no attribute 'encode'

When using python kopano-rules.py --user <username> --list the rules get listed but some of them have an empty condition. Those with the empty condition are filtering by the destination address. So maybe that causes the issue?

Please, let me provide anything else you need for digging into that. Just tell me what you need, please.

Debian 9.6
KC: 8.7.80.169-0+44.1
KW: 3.4.25.1980+1095.1
MariaDB Server: 10.1.37-0+deb9u1

Regards,
Patrick

ashceryth

I recreated 4 rules with conditions on sender address and content of subject only. Those seem to work, as they get applied and don’t disappear. So it seems it has to do with the condition of the rule…

jengelh

The only way this is gonna get resolved is to track the table delete/modify event and see what client it was issued from, i.e. custom build and diagnosis.

ashceryth

@jengelh, I think there is nothing deleted in the database as all rules reappear when using the same database with an older (working) version of kopano-core, as my tests showed about half a year ago. So the rules seem to simply becoming invisible.

Christoph.Goth

The Problem seems so occure when “dumping” the rules for the dagent (?) to process the mails while arriving.

I’v made the same discovery as @ashceryth, the rules get listed as long as no mail arrives. After the first mail being proccessed after editing the ruleset they disappear in the webapp and kopani-ibrules and none (!) get processed anymore. Using python kopano-rules.py i can list and modify them, after deleting any Rule-Entry - so the rule-set gets smaller again - they reappear and get prosessed again! Ther seems to be a point of summary rulesize where the problem beginns.

My rules are mostly based on “if senders-Adress contains X, move to folder Y”.
It doesnt mather, if i use one very big rule oder a few smaller ones, after some numbers of entrys the problem occures again and again, only solution is to delete a rule entry so the rest is working again…

jengelh

I smell a cache issue. (I tested and can reproduce now; kopano-srvadm --purge-cache=all makes it visible.)

ashceryth

@jengelh Great that you are on to something now. You mean kopano-srvadm --purge-cache=all makes the rules visible again?

ashceryth

I cleared the cache with kopano-srvadm --clear-cache=all after the rules disappeared (~20 rules) and they didn’t came back. With kopano-backup --only-meta -u <username> -f Inbox the rules file was empty but with SELECT h.owner, val_binary FROM properties AS p INNER JOIN hierarchy AS h ON p.hierarchyid=h.id WHERE tag=0x3FE1 all rules were listed in the output. So everything was still in the db. I can go up to about ~15 simple rules (if sender address is X move to folder Y) until the rules disappear.

freyuh

My rules went back after running ‘kopano-cli --clear-cache’

freyuh

Okay, my rules came back after the “clear-cache”, but only one of them is working: it’s the one with the ‘subject-rule’.
All with ‘sent-to-rules’ don’t work …

jengelh

Anyway it’s fixed in 8.6.90.80 / 8.7.80.283 and newer.

bmaehr

@jengelh Thank you!

ashceryth

@jengelh

Thanks!

freyuh

@jengelh

Thanks!

freyuh

@jengelh said in Filter Rules disappearing:

Anyway it’s fixed in 8.6.90.80 / 8.7.80.283 and newer.

I have installed the update, but nothing has changed: only the ‘subject rule’ works …
Do I have to do someting more than installing the fix?

freyuh

Hello,

maybe the error is with the email-address?

I went through all email addresses and did a ‘save’ without changing the addresses.
Before the 'update I got following content:

 *  PR_RULE_CONDITION:
     *  RES_SUBRESTRICTION subobject PR_MESSAGE_RECIPIENTS
         *  RES_COMMENT
             *  Restriction:
                 *  RES_PROPERTY 0x300b0102 == binary("SMTP:ENBW@UELLERN.DE")
             *  tag 0x60000003 1
             *  tag 0x00010102 binary("SMTP:ENBW@UELLERN.DE")
             *  tag 0x0001001f "enbw@uellern.de <enbw@uellern.de>"
             *  PR_DISPLAY_TYPE 0

And after the ‘update’ I got:

 *  PR_RULE_CONDITION:
     *  RES_SUBRESTRICTION subobject PR_MESSAGE_RECIPIENTS
         *  RES_COMMENT
             *  Restriction:
                 *  RES_PROPERTY 0x300b0102 == binary("SMTP:ENBW@UELLERN.DE\0")
             *  tag 0x60000003 1
             *  tag 0x00010102 binary("SMTP:ENBW@UELLERN.DE\0")
             *  tag 0x0001001f "enbw@uellern.de <enbw@uellern.de>"
             *  PR_DISPLAY_TYPE 0

So the addresses have changed a little. There has been added a ‘\0’ after each address …

Now I’m waiting if ‘save’ did the job …