Filter Rules disappearing

lherrmann

I tried to reproduce it (updated from 8.6.2 to 8.6.80 in core), but my rules weren’t affected by the upgrade…

bmaehr

After upgrading to the latest nightly yesterday all my rules got lost.

Coffee_is_life

Hello @all

there is a script (https://stash.kopano.io/projects/KSC/repos/core-tools/browse/kopano-rules) which you can use to list all rules of a specific user. - Check if the rule is still shown.

coffee_is_life

bmaehr

@Coffee_is_life All gone.

Coffee_is_life

@bmaehr and i bet dagent log doesnt show lines like

[info   ] [32079] Rule <Some Rule_Name>  doesn't match: 0x8004010f

there must be somethink broken with the listing and processing of these rules, since they exist in DB.
even the python-script cant see or locate these rules

since you are using the nightly builds, i dont think you got subsription (if i’m wrong, best way would be to open a ticket)
this issue is huge and got a greate impact on daily workflow, so i think the devs will look into this asap - with and without ticket

coffee_is_life

bmaehr

dagent log is completely empty (created 3 months ago).
That’s the bad thing in kopano using the SQL db just as data heap: I can’t check if they are really there. I don’t think they will come back, but perhaps I get surprised.
The rule created with core-8.6.80.1018 didn’t get deleted after upgrading to core-8.6.80.1055, so it seams not to be a problem of the upgrade itself but more a problem of the format of the old rules.

Christoph.Goth

Same problem here, at least with one User. Another Users rules dont seem to be affected.

Problematical user recreated his rules, are visible with py-script but no more in webapp and dont get applyed :(

fbartels

Up until this point we were sadly not able to reproduce this issue. If you have any more information about the rules of the user, or with which master version this appeared first we would be happy to hear about this.

Christoph.Goth

I’ve upgraded from core-8.6.80.752_0+85-Debian_9.0-amd64 to core-8.6.80.1018_0+152-Debian_9.0-amd64, then the problem occured.
The User was greeted by the webapp configuration wizard, others not.
Afterwards, i upgraded to core-8.6.80.1055_0+156-Debian_9.0-amd64 with no change.

Then i recreated the rules in webapp, after switching windows in webapp they disappeared there again.

Using rules based on “Includes these words in the sender’s address” are showing up correctly in py-script, those using full senders address are having an empty Condition-Table entry. Maybe there the Problem comes from?

After Upgrade working users rules only use the first, working, condition.

Another thing i notices:
The problematic user is also missing the mail signature, setup via webapp.

bmaehr

@fbartels Can you provide a SQL Statement to list all rules from the database directly? So we could check if they are still there.

Christoph.Goth

After deleting the Rules with missing conditions, the remaining rules reappeeared in the Webapp, test if they’re working is pending.

Christoph.Goth

Update: Not working and rules disappeared again :(

jengelh

If you have any more information

Better yet, if you have any disk images, shell prompts, etc., …

SQL Statement to list all [dagent] rules from the database directly?

SELECT h.owner, val_binary FROM properties AS p INNER JOIN hierarchy AS h ON p.hierarchyid=h.id WHERE tag=0x3FE1

bmaehr

@jengelh Ok, I have executed the statement on my updated system and imported a database-dump from some months ago into an other database and executed the statement. I don’t think that I changed the rules since then - just created one rule after the update when all rules got lost.
Both are returning exactly one record with a binary field. The size of the field is very different: about 58kb on the old and about 2kb on the new.

I’m not really able to read the content of the binary, but I was able to find the sender addresses I use with the rules to sort the mail into different folders inside the binary blob.
In the old one there are many addresses (I expect all addresses of all rules) in the blob and in the new one only one for the one rule I created.

So it seams to me all rules get lost also in the database and/or the whole blob was thrown away.

Could I help you in some way to find the problem? Perhaps exporting my old blob in some way and sending it to you?

bmaehr

Update: I just saw, that the blob is an XML dokument.

So i started to recreate my rules over the Webapp. I entered my 6 rules and clicked to save. Then I went to Mail view and back to settings / rules and my rules where gone (they are still in database now and kopano-rules lists them, but not shown in the webapp).

Perhaps the number of sendes in a rule is the problem…

bmaehr

Update: I tried to recreate the rules with Webapp several times in different orders. And lost the rules two times after saving them, going to Mail and returning to Rules.
So I think the problem is the size of the whole blob. I think the limit is somewhere between 50kB and 62kB.

bmaehr

Update: I saved my rules an hour ago and they where there in the WebApp when I check the last time. I didn’t do anything since then, but now they are lost in tde WebApp. They are still displayed by kopano-rules.py, but they are not respected when processing mails.

bmaehr

@fbartels Is this issue fixed? As already said I"m able to reproduce it.

fbartels

Hi @bmaehr ,

no no luck reproducing it on our side and therefore also no fix for this. We have added two changes though to get some additional logging:

https://stash.kopano.io/projects/KC/repos/kopanocore/commits/5f7b0f351f78266a773d323f657b89ce34f60a61
https://stash.kopano.io/projects/KC/repos/kopanocore/commits/614469db65191788ef15d6416e7a83192a4afbf0

bmaehr

@fbartels I just have to add enough rules on my running instance and the rules get lost without any upgrade. I can provide you the information to reproduce it.