email rules don't match when the mail header is too large

[umgfoin]

@mholschbach
Yes, I can confirm this. Especially annoying for Google mails, featuring excessive transport-header orgies.
++umgfoin.

[robing]

@mholschbach @umgfoin ,

This is due the caching issue we encounter at the moment with 8.7.x , but it should be fixed with the latest master version.

For the supported packages there isn’t a release date known as we still need to fix some bugs first regarding this.

[umgfoin]

Hi @robing, are you sure about the cache-cause? The problem exists for long - since initial contribution of the feature (~11/2017).
It is still present in today’s master 8.7.80.893.

My suspicion is more in direction of possible truncation during PR_TRANSPORT_MESSAGE_HEADERS extraction or during the FL_SUBSTRING compare when evaluating restrictions. Anyway, I didn’t find time to dive into it.

++umgfoin.

[mholschbach]

Hi @umgfoin and @robing
Thank you for your answers.

I’m using kopano as part of univention’s ucs. The installed version is 8.6.8.10-0+23.1 as dpkg reports it. This supports @umgfoin 's assumption that it isn’t a new bug in 8.7.x

Yes, this bug is annoying. For a quick fix I’m considering to remove headers like “IronPort-PHdr”, “Autocrypt” etc. from within Postfix to reduce the overall header size for Kopano. Any other ideas?

Best regards
Michael

[fbartels]

Hi @mholschbach ,

thanks for your report I have created https://jira.kopano.io/browse/KC-1478 to keep track of it.

[fbartels]

Hi @mholschbach and @umgfoin,

we had another check and think behaviour should be covered by the changes introduced in https://jira.kopano.io/browse/KC-1443. Can you have a quick check to see if this now works as expected for you?

[umgfoin]

@fbartels said in email rules don't match when the mail header is too large:

if this now works as expected for you?

Hi Felix,
not sure, yet - I was on an earlier master-build (a6a08597471 / 2nd May 19) for some time, now - there the issue was still present, but seemed to appear less often, subjectively.
I observed that commit f902c9fc124: “php: transparently postload large properties” regularely segfaults php-fpm under many situations (e.g. replying to a mail from webapp). I had to revert it in order to run recent master.

++umgfoin.

[fbartels]

Just to be on the safe side, the above change was merged in https://stash.kopano.io/projects/KC/repos/kopanocore/commits/b025bcaaf733202fc462231dbfb14c95e4cef869

@umgfoin said in email rules don't match when the mail header is too large:

I observed that commit f902c9fc124: “php: transparently postload large properties” regularely segfaults php-fpm under many situations (e.g. replying to a mail from webapp)

Thanks for bringing this to my attention. I will check with our WebApp team if the seem something similar.

[umgfoin]

@fbartels said in email rules don't match when the mail header is too large:

Just to be on the safe side, the above change was merged it

If KC-1443 related (HrGetFullProp) was merged in March, then this would mean “not fixed” from my side.
To my understanding, HRESULT TestRestriction(const SRestriction *lpCondition, IMAPIProp *lpMessage,
const ECLocale &locale, ULONG ulLevel) is doing the compare-part of the rule-evaluation. Supposely we need the HrGetFullProp-variant there, too.

Thanks for bringing this to my attention. I will check with our WebApp team if the seem something similar.

Please point out my non standard environment: PHP 7 + Python 3.6 on CentOS6
++umgfoin.

[umgfoin]

@umgfoin said in email rules don't match when the mail header is too large:

Supposely we need the HrGetFullProp-variant there, too.

I’m testing this, right now and will report back asap.
++umgfoin.

[umgfoin]

@umgfoin said in email rules don't match when the mail header is too large:

To my understanding, HRESULT TestRestriction(const SRestriction *lpCondition, IMAPIProp *lpMessage,
const ECLocale &locale, ULONG ulLevel) is doing the compare-part of the rule-evaluation. Supposely we need the HrGetFullProp-variant there, too.

I’m testing this, right now and will report back asap.

Hi Felix @fbartels ,
test successful:
Substituting relevant GetOneProp-calls with GetFullProp in Utils::TestRestriction fixes the here discussed problem. I’ll post a PR to Github later on.

PR here.

bg umgfoin.