Empty emails on Outlook 2016 & 2019 with Kopano 10

deajan

Hello,

I am connecting Outlook 2016 and Outlook 2019 via ActiveSync to Kopano, and some of the mails that appear normally in Webapp, appear empty in Outlook.

I’ve tried to make WBXML logs from z-push, but the only relevant part is:


31/08/2020 19:21:20 [  891] [DEBUG] [my@mail-user] BodyPreference types: 4
31/08/2020 19:21:20 [  891] [DEBUG] [my@mail-user] GetBodyPreferenceBestMatch: 4
31/08/2020 19:21:20 [  891] [DEBUG] [my@mail-user] bpo: truncation size:'0', allornone:'1', preview:'0'
31/08/2020 19:21:20 [  891] [ERROR] [my@mail-user] MAPIProvider->imtoinet(): got no stream or content from mapi_inetmapi_imtoinet()
31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O      <Add>
31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       <ServerEntryId>
31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       U883b8:e7b9279a462d4bd7939d02bbe2f016e5b24c00000000
31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       </ServerEntryId>
31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       <Data>
31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O        <POOMMAIL:To>
...

I’ve already tried two different outlook clients (outlook 2016 and outlook 2019), with and without KOE installed.
Running nightly builds, I’ve updated from core-10.0.6.13.1853765-Debian_10-amd64 to core-10.0.6.406.e263d46-Debian_10-amd64

Config:
Debian: 10.5 x64
Kopano: kopano-server 10.0.6.406.69083f1d7-0+217.1
Z-Push: z-push 2.5.2+0-0
PHP-MAPI: php7-mapi 10.0.6.406.69083f1d7-0+217.1
Python-MAPI: python3-mapi 10.0.6.406.69083f1d7-0+217.1 amd64

Some of the offending emails litteraly contain “test” and have subject “test”, so I don’t have a rule of thumb to know when this happens exactly.

I’ve outruled anything I could, now I’m stuck here.
I’ve googled around the error MAPIProvider->imtoinet(): got no stream or content from mapi_inetmapi_imtoinet() but I only find issues that happened 2 years ago, with Kopano 8.6x.

I am out of ideas, willing to share offending emails (if I get a way to extract them from the store). Willing to make logs, tests, whaterver is needed.

Please help :(

[EDIT] My mail setup has 3 home users, hence the nightly builds usage. Outlook client is mandatory for my better half… drives me crazy. Willing to pay kopano support just so I don’t get bugged, but not sure if I can simply downgrade bleeding edge to supported version. @fbartels ?[/EDIT]

Manfred

Hi @deajan,

where do these emails come from? Did you migrate from some other system to kopano? Or are you sending them from some other test system? Are you able to reproduce this behaviour reliably?

Manfred

deajan

Hello Manfred,

No migration involved, it’s a new server. The mails are from various senders, didn’t find any common property for those. Also, I didn’t yet find a way to reproduce it, but I think I may make a “diff” between kopano webapp and outlook, and ask my antispam to resend the problematic mails in order to be able to trigger the problem.

I can of course reproduce the MAPIProvider->imtoinet(): got no stream or content from mapi_inetmapi_imtoinet() error by triggering a full resync via KOE, which will recreate the empty mails on the outlook side.

Being a home system, I don’t mind installing beta versions if requested, or pull a z-push from git.

Thanks,
Orsiris.

Manfred

Hi Orsiris,

is it possible that your antispam manipulates body of some emails by removing data? My guess is that the emails are already “broken” in the Kopano database. Empty stream from mapi_inetmapi_imtoinet() is just the consequence and not the culprit.

Could you post the complete WBXML log of one such email being synced to Outlook?

Manfred

deajan

Manfred,

Yes, the antispam could add red text to links etc.
Sure I will post a WBXML log as soon as I identified a particular broken email (not easy), but I wonder why Kopano WebApp could show such a broken email without any problems ?

Orsiris.

Manfred

Hi Orsiris,

@deajan said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

Sure I will post a WBXML log as soon as I identified a particular broken email (not easy), but I wonder why Kopano WebApp could show such a broken email without any problems ?

Without going into too much details it’s because Webapp just gets the required properties (subject, body etc) from the database while Outlook requires full RFC-822 message and Kopano core has to put this message together.

It’s also possible to save the raw message. See kopano-dagent.cfg manual regarding log_raw_message for more information. And if I remember correctly it’s necessary to enable imap feature for the user to make use of it.

Manfred

deajan

Thanks for your help Manfred.

I have setup kopano-gateway and connected an outlook profile with an IMAP profile.
So the problematic mails appear both in Z-Push and IMAP sync modes, so I can outrule Z-push.

I’ve resubmitted failed mails from the antispam, they are sent as .msg attachment, and appear correct in Outlook.
I’ve then connected directly into the antispam via ssh, and resubmitted two mails for delivery with /usr/sbin/sendmail.postfix -t < /var/spool/Mailscanner/20200831/nospam/<mymailID>. Those mails are raw format.

The mails show in outlook without any problem, using z-push or imap.

I’ve also sent two test mails from another mail server… One came as empty message, the other came in ok.

I have no clue where to search now.
I have activated dagent raw logging, but since this is totally random, it will be searching a needle in a haystack.

I’m pretty sure I won’t need the WBXML logs since I outruled Z-Push, right ?

All I can do is leaving dagent raw logging enabled, and wait for the next missing mails, am I right ?

Best regards,
Orsiris.

Manfred

Hi Orsiris,

@deajan said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

I’ve also sent two test mails from another mail server… One came as empty message, the other came in ok.

Well, what was the difference between two emails? Maybe some special chars (e.g. German umlauts ä ö ü) in one? Or the content type (plain vs html)?

I’m pretty sure I won’t need the WBXML logs since I outruled Z-Push, right ?

Yes, if you can rule out Z-Push, then the WBXML log is not necessary.

All I can do is leaving dagent raw logging enabled, and wait for the next missing mails, am I right ?

Unfortunately I also don’t have any other idea. Maybe you could post the same question in the core section of the forum now that Z-Push or KOE seem not to be the ones breaking the emails.

Manfred

deajan

Hello Manfred,

Well, what was the difference between two emails? Maybe some special chars (e.g. German umlauts ä ö ü) in one? Or the content type (plain vs html)?

Both mails are litteraly the same, written by another kopano instance I had earlier. They have the same subject, and contain “test” and “test 2”, the latter being received without problems.

I also suspected some problems with mysql, and increased max_packet_size from 16MB to 128MB, but that shouldn’t be the problem, since my test emails should weight some KBs only.

Anyway, I’ve kept dagent raw logs on, and am currently waiting for the next problematic emails (non as of today). I still wonder why this problem is so random. I’ve looked at dmesg and various other log files just to make sure the system wasn’t faulty too.

Thanks for your help.
A last thing, I would like to “extract” a problematic mail from kopano DB, to check whether it is fully RFC822 compliant. Is there any kopano tool I can export a message as raw ?

Thanks,
Orsiris.

alexthetiger68

I am on the same OS and Kopano Version as deajan and experience the same empty mails in Outlook via ActiveSync (Z-Push).

osrelease               Pretty operating system name                                                    Debian GNU/Linux 10 (buster)
program_name            Program name                                                                    kopano-server
program_version         Program version                                                                 10.0.6
server_guid                                                                                             7636E72C7285493384B2CCC3487C6508

What is interesting, the same empty Mail is not only perfectly visible in Kopano Webapp, but also on another device synced via z-push (iphone) to the same account. I dont have the KOE extensions installed.

I did not see any error messages in the mysql logs.

The kopano-server 10.0.6.406.69083f1d7-0+217.1 seems to have at least two major issues, the lost connections to LDAP and these empty mails. Do you have an ETA when the nightly artefacts will be updated again so we can test more? Or maybe you can point me to the correct repository for kopano-core so I can compile the sources on my own. I only found a repo with tons of forks and am a bit lost as to what is the master repo.

deajan

@alexthetiger68 I don’t experience more of those empty messages for the moment (strange enough, it’s random).

Maybe it’s related to the following mysql settings I tuned

[mysqld]
max_allowed_packet=128M
innodb_log_file_size = 128M # Fix kopano-server: SQL [00000088] info: MySQL server has gone away. Reconnecting, see https://jira.kopano.io/browse/KC-1053

The second setting needs you to remove your ib_log files with

systemctl stop mariadb
mv /var/lib/mysql/ib_log* /root
# make changes in /etc/mysql/mariadb.conf.d/50-server.cnf
systemctl restart mariadb

Can you try these and report back if it worked for you ?
Thanks.

alexthetiger68

@deajan I’ve made the changes to the mysqld and will monitor this issue.

I’ve seen a Warning in the mysqld log, but right now I do not consider this to be a major problem.

Sep 08 10:59:21 fizzi2 systemd[1]: Starting MariaDB 10.3.23 database server…
Sep 08 10:59:21 fizzi2 mysqld[23436]: 2020-09-08 10:59:21 0 [Note] /usr/sbin/mysqld (mysqld 10.3.23-MariaDB-0+deb10u1) starting as process 23436 …
Sep 08 10:59:21 fizzi2 mysqld[23436]: 2020-09-08 10:59:21 0 [Warning] Could not increase number of max_open_files to more than 16364 (request: 32184)
Sep 08 10:59:22 fizzi2 systemd[1]: Started MariaDB 10.3.23 database server.

deajan

@alexthetiger68 said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

Could not increase number of max_open_files to more than 16364

How big is your setup ? (ie how many users / gigabytes of data)
Is your MySQL shared ?

Unless your mysql is huge, you should check why it has open more than 16k files.

If your usecase requires to, you should definitly tune mysql, see https://haydenjames.io/could-not-increase-number-of-max_open_files-to-more-than/

alexthetiger68

@deajan I’m currently running Kopano in a container on Proxmox. The database has been converted from Zarafa to Kopano a long time ago and is currently 46GB in size. I’ve made the switch from VM to container about two years ago and the system has always been running smooth, until I upgraded to from from 10.0.5.5.8827066e2-0+106.1 to core-10.0.6.406.e263d46-Debian_10-amd64 .

deajan

Honestly I run my personal Kopano with less than 25G, so I can’t tell how much connections you should have open at the same time.
But nevertheless, you should have a look at why you need more than 16K files open before tuning this setting. Maybe SHOW STATUS WHERE variable_name = 'Max_used_connections'; could help here.

Manfred

Hi Orsiris,

@deajan said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

A last thing, I would like to “extract” a problematic mail from kopano DB, to check whether it is fully RFC822 compliant. Is there any kopano tool I can export a message as raw ?

In webapp right click on the email and select “Export as” -> EML file(s).

Manfred

deajan

Thanks ;) I’ll check this ASAP.