Navigation

    Kopano
    • Register
    • Login
    • Search
    • Categories
    • Get Official Kopano Support
    • Recent
    Statement regarding the closure of the Kopano community forum and the end of the community edition

    Empty emails on Outlook 2016 & 2019 with Kopano 10

    Kopano OL Extension (KOE)
    3
    17
    1742
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • deajan
      deajan last edited by deajan

      Hello,

      I am connecting Outlook 2016 and Outlook 2019 via ActiveSync to Kopano, and some of the mails that appear normally in Webapp, appear empty in Outlook.

      I’ve tried to make WBXML logs from z-push, but the only relevant part is:

      
      31/08/2020 19:21:20 [  891] [DEBUG] [my@mail-user] BodyPreference types: 4
      31/08/2020 19:21:20 [  891] [DEBUG] [my@mail-user] GetBodyPreferenceBestMatch: 4
      31/08/2020 19:21:20 [  891] [DEBUG] [my@mail-user] bpo: truncation size:'0', allornone:'1', preview:'0'
      31/08/2020 19:21:20 [  891] [ERROR] [my@mail-user] MAPIProvider->imtoinet(): got no stream or content from mapi_inetmapi_imtoinet()
      31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O      <Add>
      31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       <ServerEntryId>
      31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       U883b8:e7b9279a462d4bd7939d02bbe2f016e5b24c00000000
      31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       </ServerEntryId>
      31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O       <Data>
      31/08/2020 19:21:20 [  891] [WBXML] [my@mail-user] O        <POOMMAIL:To>
      ...
      

      I’ve already tried two different outlook clients (outlook 2016 and outlook 2019), with and without KOE installed.
      Running nightly builds, I’ve updated from core-10.0.6.13.1853765-Debian_10-amd64 to core-10.0.6.406.e263d46-Debian_10-amd64

      Config:
      Debian: 10.5 x64
      Kopano: kopano-server 10.0.6.406.69083f1d7-0+217.1
      Z-Push: z-push 2.5.2+0-0
      PHP-MAPI: php7-mapi 10.0.6.406.69083f1d7-0+217.1
      Python-MAPI: python3-mapi 10.0.6.406.69083f1d7-0+217.1 amd64

      Some of the offending emails litteraly contain “test” and have subject “test”, so I don’t have a rule of thumb to know when this happens exactly.

      I’ve outruled anything I could, now I’m stuck here.
      I’ve googled around the error MAPIProvider->imtoinet(): got no stream or content from mapi_inetmapi_imtoinet() but I only find issues that happened 2 years ago, with Kopano 8.6x.

      I am out of ideas, willing to share offending emails (if I get a way to extract them from the store). Willing to make logs, tests, whaterver is needed.

      Please help :(

      [EDIT] My mail setup has 3 home users, hence the nightly builds usage. Outlook client is mandatory for my better half… drives me crazy. Willing to pay kopano support just so I don’t get bugged, but not sure if I can simply downgrade bleeding edge to supported version. @fbartels ?[/EDIT]

      1 Reply Last reply Reply Quote 0
      • Manfred
        Manfred Kopano last edited by

        Hi @deajan,

        where do these emails come from? Did you migrate from some other system to kopano? Or are you sending them from some other test system? Are you able to reproduce this behaviour reliably?

        Manfred

        1 Reply Last reply Reply Quote 0
        • deajan
          deajan last edited by

          Hello Manfred,

          No migration involved, it’s a new server. The mails are from various senders, didn’t find any common property for those. Also, I didn’t yet find a way to reproduce it, but I think I may make a “diff” between kopano webapp and outlook, and ask my antispam to resend the problematic mails in order to be able to trigger the problem.

          I can of course reproduce the MAPIProvider->imtoinet(): got no stream or content from mapi_inetmapi_imtoinet() error by triggering a full resync via KOE, which will recreate the empty mails on the outlook side.

          Being a home system, I don’t mind installing beta versions if requested, or pull a z-push from git.

          Thanks,
          Orsiris.

          1 Reply Last reply Reply Quote 0
          • Manfred
            Manfred Kopano last edited by

            Hi Orsiris,

            is it possible that your antispam manipulates body of some emails by removing data? My guess is that the emails are already “broken” in the Kopano database. Empty stream from mapi_inetmapi_imtoinet() is just the consequence and not the culprit.

            Could you post the complete WBXML log of one such email being synced to Outlook?

            Manfred

            1 Reply Last reply Reply Quote 0
            • deajan
              deajan last edited by

              Manfred,

              Yes, the antispam could add red text to links etc.
              Sure I will post a WBXML log as soon as I identified a particular broken email (not easy), but I wonder why Kopano WebApp could show such a broken email without any problems ?

              Orsiris.

              1 Reply Last reply Reply Quote 0
              • Manfred
                Manfred Kopano last edited by

                Hi Orsiris,

                @deajan said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

                Sure I will post a WBXML log as soon as I identified a particular broken email (not easy), but I wonder why Kopano WebApp could show such a broken email without any problems ?

                Without going into too much details it’s because Webapp just gets the required properties (subject, body etc) from the database while Outlook requires full RFC-822 message and Kopano core has to put this message together.

                It’s also possible to save the raw message. See kopano-dagent.cfg manual regarding log_raw_message for more information. And if I remember correctly it’s necessary to enable imap feature for the user to make use of it.

                Manfred

                1 Reply Last reply Reply Quote 0
                • deajan
                  deajan last edited by

                  Thanks for your help Manfred.

                  I have setup kopano-gateway and connected an outlook profile with an IMAP profile.
                  So the problematic mails appear both in Z-Push and IMAP sync modes, so I can outrule Z-push.

                  I’ve resubmitted failed mails from the antispam, they are sent as .msg attachment, and appear correct in Outlook.
                  I’ve then connected directly into the antispam via ssh, and resubmitted two mails for delivery with /usr/sbin/sendmail.postfix -t < /var/spool/Mailscanner/20200831/nospam/<mymailID>. Those mails are raw format.

                  The mails show in outlook without any problem, using z-push or imap.

                  I’ve also sent two test mails from another mail server… One came as empty message, the other came in ok.

                  I have no clue where to search now.
                  I have activated dagent raw logging, but since this is totally random, it will be searching a needle in a haystack.

                  I’m pretty sure I won’t need the WBXML logs since I outruled Z-Push, right ?

                  All I can do is leaving dagent raw logging enabled, and wait for the next missing mails, am I right ?

                  Best regards,
                  Orsiris.

                  1 Reply Last reply Reply Quote 0
                  • Manfred
                    Manfred Kopano last edited by

                    Hi Orsiris,

                    @deajan said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

                    I’ve also sent two test mails from another mail server… One came as empty message, the other came in ok.

                    Well, what was the difference between two emails? Maybe some special chars (e.g. German umlauts ä ö ü) in one? Or the content type (plain vs html)?

                    I’m pretty sure I won’t need the WBXML logs since I outruled Z-Push, right ?

                    Yes, if you can rule out Z-Push, then the WBXML log is not necessary.

                    All I can do is leaving dagent raw logging enabled, and wait for the next missing mails, am I right ?

                    Unfortunately I also don’t have any other idea. Maybe you could post the same question in the core section of the forum now that Z-Push or KOE seem not to be the ones breaking the emails.

                    Manfred

                    1 Reply Last reply Reply Quote 0
                    • deajan
                      deajan last edited by

                      Hello Manfred,

                      Well, what was the difference between two emails? Maybe some special chars (e.g. German umlauts ä ö ü) in one? Or the content type (plain vs html)?

                      Both mails are litteraly the same, written by another kopano instance I had earlier. They have the same subject, and contain “test” and “test 2”, the latter being received without problems.

                      I also suspected some problems with mysql, and increased max_packet_size from 16MB to 128MB, but that shouldn’t be the problem, since my test emails should weight some KBs only.

                      Anyway, I’ve kept dagent raw logs on, and am currently waiting for the next problematic emails (non as of today). I still wonder why this problem is so random. I’ve looked at dmesg and various other log files just to make sure the system wasn’t faulty too.

                      Thanks for your help.
                      A last thing, I would like to “extract” a problematic mail from kopano DB, to check whether it is fully RFC822 compliant. Is there any kopano tool I can export a message as raw ?

                      Thanks,
                      Orsiris.

                      Manfred 1 Reply Last reply Reply Quote 0
                      • alexthetiger68
                        alexthetiger68 last edited by

                        I am on the same OS and Kopano Version as deajan and experience the same empty mails in Outlook via ActiveSync (Z-Push).

                        osrelease               Pretty operating system name                                                    Debian GNU/Linux 10 (buster)
                        program_name            Program name                                                                    kopano-server
                        program_version         Program version                                                                 10.0.6
                        server_guid                                                                                             7636E72C7285493384B2CCC3487C6508
                        

                        What is interesting, the same empty Mail is not only perfectly visible in Kopano Webapp, but also on another device synced via z-push (iphone) to the same account. I dont have the KOE extensions installed.

                        I did not see any error messages in the mysql logs.

                        The kopano-server 10.0.6.406.69083f1d7-0+217.1 seems to have at least two major issues, the lost connections to LDAP and these empty mails. Do you have an ETA when the nightly artefacts will be updated again so we can test more? Or maybe you can point me to the correct repository for kopano-core so I can compile the sources on my own. I only found a repo with tons of forks and am a bit lost as to what is the master repo.

                        1 Reply Last reply Reply Quote 0
                        • deajan
                          deajan last edited by

                          @alexthetiger68 I don’t experience more of those empty messages for the moment (strange enough, it’s random).

                          Maybe it’s related to the following mysql settings I tuned

                          [mysqld]
                          max_allowed_packet=128M
                          innodb_log_file_size = 128M # Fix kopano-server: SQL [00000088] info: MySQL server has gone away. Reconnecting, see https://jira.kopano.io/browse/KC-1053
                          

                          The second setting needs you to remove your ib_log files with

                          systemctl stop mariadb
                          mv /var/lib/mysql/ib_log* /root
                          # make changes in /etc/mysql/mariadb.conf.d/50-server.cnf
                          systemctl restart mariadb
                          

                          Can you try these and report back if it worked for you ?
                          Thanks.

                          1 Reply Last reply Reply Quote 0
                          • alexthetiger68
                            alexthetiger68 last edited by

                            @deajan I’ve made the changes to the mysqld and will monitor this issue.

                            I’ve seen a Warning in the mysqld log, but right now I do not consider this to be a major problem.

                            Sep 08 10:59:21 fizzi2 systemd[1]: Starting MariaDB 10.3.23 database server…
                            Sep 08 10:59:21 fizzi2 mysqld[23436]: 2020-09-08 10:59:21 0 [Note] /usr/sbin/mysqld (mysqld 10.3.23-MariaDB-0+deb10u1) starting as process 23436 …
                            Sep 08 10:59:21 fizzi2 mysqld[23436]: 2020-09-08 10:59:21 0 [Warning] Could not increase number of max_open_files to more than 16364 (request: 32184)
                            Sep 08 10:59:22 fizzi2 systemd[1]: Started MariaDB 10.3.23 database server.

                            1 Reply Last reply Reply Quote 0
                            • deajan
                              deajan last edited by

                              @alexthetiger68 said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

                              Could not increase number of max_open_files to more than 16364

                              How big is your setup ? (ie how many users / gigabytes of data)
                              Is your MySQL shared ?

                              Unless your mysql is huge, you should check why it has open more than 16k files.

                              If your usecase requires to, you should definitly tune mysql, see https://haydenjames.io/could-not-increase-number-of-max_open_files-to-more-than/

                              alexthetiger68 1 Reply Last reply Reply Quote 0
                              • alexthetiger68
                                alexthetiger68 @deajan last edited by

                                @deajan I’m currently running Kopano in a container on Proxmox. The database has been converted from Zarafa to Kopano a long time ago and is currently 46GB in size. I’ve made the switch from VM to container about two years ago and the system has always been running smooth, until I upgraded to from from 10.0.5.5.8827066e2-0+106.1 to core-10.0.6.406.e263d46-Debian_10-amd64 .

                                1 Reply Last reply Reply Quote 0
                                • deajan
                                  deajan last edited by

                                  Honestly I run my personal Kopano with less than 25G, so I can’t tell how much connections you should have open at the same time.
                                  But nevertheless, you should have a look at why you need more than 16K files open before tuning this setting. Maybe SHOW STATUS WHERE variable_name = 'Max_used_connections'; could help here.

                                  1 Reply Last reply Reply Quote 0
                                  • Manfred
                                    Manfred Kopano @deajan last edited by

                                    Hi Orsiris,

                                    @deajan said in Empty emails on Outlook 2016 & 2019 with Kopano 10:

                                    A last thing, I would like to “extract” a problematic mail from kopano DB, to check whether it is fully RFC822 compliant. Is there any kopano tool I can export a message as raw ?

                                    In webapp right click on the email and select “Export as” -> EML file(s).

                                    Manfred

                                    1 Reply Last reply Reply Quote 0
                                    • deajan
                                      deajan last edited by

                                      Thanks ;) I’ll check this ASAP.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post