email rules don't match when the mail header is too large

mholschbach

Dear all,

I’m using email rules to sort mailing lists directly into specific folders. This is working fine for most of the mails and they are sorted correctly. But some mails stay in my inbox, even though they contain the header fields which I use in my rules.

All of these mails do have in common that the header itself is well above 8KB and the header field I’m looking for is not within the first 7 to 8 KB. Did anyone else observe this?

Any suggested configuration changes?

Best regards
Michael

umgfoin

@mholschbach
Yes, I can confirm this. Especially annoying for Google mails, featuring excessive transport-header orgies.
++umgfoin.

robing

@mholschbach @umgfoin ,

This is due the caching issue we encounter at the moment with 8.7.x , but it should be fixed with the latest master version.

For the supported packages there isn’t a release date known as we still need to fix some bugs first regarding this.

umgfoin

Hi @robing, are you sure about the cache-cause? The problem exists for long - since initial contribution of the feature (~11/2017).
It is still present in today’s master 8.7.80.893.

My suspicion is more in direction of possible truncation during PR_TRANSPORT_MESSAGE_HEADERS extraction or during the FL_SUBSTRING compare when evaluating restrictions. Anyway, I didn’t find time to dive into it.

++umgfoin.

mholschbach

Hi @umgfoin and @robing
Thank you for your answers.

I’m using kopano as part of univention’s ucs. The installed version is 8.6.8.10-0+23.1 as dpkg reports it. This supports @umgfoin 's assumption that it isn’t a new bug in 8.7.x

Yes, this bug is annoying. For a quick fix I’m considering to remove headers like “IronPort-PHdr”, “Autocrypt” etc. from within Postfix to reduce the overall header size for Kopano. Any other ideas?

Best regards
Michael

fbartels

Hi @mholschbach ,

thanks for your report I have created https://jira.kopano.io/browse/KC-1478 to keep track of it.

fbartels

Hi @mholschbach and @umgfoin,

we had another check and think behaviour should be covered by the changes introduced in https://jira.kopano.io/browse/KC-1443. Can you have a quick check to see if this now works as expected for you?

umgfoin

@fbartels said in email rules don't match when the mail header is too large:

if this now works as expected for you?

Hi Felix,
not sure, yet - I was on an earlier master-build (a6a08597471 / 2nd May 19) for some time, now - there the issue was still present, but seemed to appear less often, subjectively.
I observed that commit f902c9fc124: “php: transparently postload large properties” regularely segfaults php-fpm under many situations (e.g. replying to a mail from webapp). I had to revert it in order to run recent master.

++umgfoin.

fbartels

Just to be on the safe side, the above change was merged in https://stash.kopano.io/projects/KC/repos/kopanocore/commits/b025bcaaf733202fc462231dbfb14c95e4cef869

@umgfoin said in email rules don't match when the mail header is too large:

I observed that commit f902c9fc124: “php: transparently postload large properties” regularely segfaults php-fpm under many situations (e.g. replying to a mail from webapp)

Thanks for bringing this to my attention. I will check with our WebApp team if the seem something similar.

umgfoin

@fbartels said in email rules don't match when the mail header is too large:

Just to be on the safe side, the above change was merged it

If KC-1443 related (HrGetFullProp) was merged in March, then this would mean “not fixed” from my side.
To my understanding, HRESULT TestRestriction(const SRestriction *lpCondition, IMAPIProp *lpMessage,
const ECLocale &locale, ULONG ulLevel) is doing the compare-part of the rule-evaluation. Supposely we need the HrGetFullProp-variant there, too.

Thanks for bringing this to my attention. I will check with our WebApp team if the seem something similar.

Please point out my non standard environment: PHP 7 + Python 3.6 on CentOS6
++umgfoin.

umgfoin

@umgfoin said in email rules don't match when the mail header is too large:

Supposely we need the HrGetFullProp-variant there, too.

I’m testing this, right now and will report back asap.
++umgfoin.

umgfoin

@umgfoin said in email rules don't match when the mail header is too large:

To my understanding, HRESULT TestRestriction(const SRestriction *lpCondition, IMAPIProp *lpMessage,
const ECLocale &locale, ULONG ulLevel) is doing the compare-part of the rule-evaluation. Supposely we need the HrGetFullProp-variant there, too.

I’m testing this, right now and will report back asap.

Hi Felix @fbartels ,
test successful:
Substituting relevant GetOneProp-calls with GetFullProp in Utils::TestRestriction fixes the here discussed problem. ~~I’ll post a PR to Github later on.~~

PR here.

bg umgfoin.