Kopano Search Returns MAPI Error -> entire Kopano server stops working

yep_DD

Hi guys,

I’ve recently experienced a strange behavior that I could trace to be related to Kopano Search. I’ve noticed the following flaws:

Kopano WebApp doesn’t load (rotating icon)
IMAP Client doesn’t load Mails
Z-Push client doesn’t show mail
Contacts on iOS devices lost

The error in the Kopano logs was always the following (search.log):

  File "/usr/lib/python2.7/dist-packages/kopano/ics.py", line 102, in ImportMessageChange
    item.store = _store.Store(mapiobj=mapistore, server=self.server)
  File "/usr/lib/python2.7/dist-packages/kopano/store.py", line 101, in __init__
    self._root = self.mapiobj.OpenEntry(None, None, 0)
  File "/usr/lib/python2.7/dist-packages/MAPICore.py", line 605, in OpenEntry
    return _MAPICore.IMsgStore_OpenEntry(self, cbEntryID, lpInterface, ulFlags)
MAPIErrorNetworkError: MAPI error 80040115 (MAPI_E_NETWORK_ERROR)

I updated to Kopano Core core-8.4.0~931_164.1-Debian_9.0-amd64 lately. When I noticed the error I tried the most recent core-8.4.90.448_0+80-Debian_9.0-amd64 but no changes.

I deleted the search registry under /var/lib/kopano/search with no positive outcome. Only when I deactived the complete search in server.cfg the entire Kopano system is working without flaws.

Can anyone help me on that?

fbartels

Hello @yep_dd ,

the error you are getting from search is a network error, which means kopano-search cannot contact kopano-server, so you rather have to add it to your list of symptoms instead of it being the cause for your symptoms.

A good point to start digging is looking at kopano-stats (the --top view is always nice to have open) while the error is happening. Actually looking into the server.log (maybe on an increased log_level) may also give some insights.

yep_DD

Hello @fbartels ,

thanks for the reply. The only thing I noticed in the server.log so far is that I when I am receiving a huge mail I get the following error:

  [error  ] SQL [00001573] result failed: BIGINT UNSIGNED value is out of range in '(`zarafa`.`properties`.`tag` - 0x8501)', Query: "SELECT 0,properties.tag,properties.type,properties.val_ulong,properties.val_string,properties.val_binary,properties.val_double,properties.val_longint,properties.val_hi,properties.val_lo, hierarchyid, names.nameid, names.namestring, names.guid FROM properties FORCE INDEX (PRIMARY) LEFT JOIN names ON (properties.tag-0x8501)=names.id WHERE hierarchyid=9 AND (tag <= 0x8500 OR names.id IS NOT NULL)"

It seems the SQL server (mariadb) can’t handle that request. I am not sure if that is related.

fbartels

Hi @yep_dd ,

ah that is indeed a good catch. I have created https://jira.kopano.io/browse/KC-852 to investigate this further.

yep_DD

Hi @fbartels ,

thank you. I saw the issue is a duplicate and of another one. But may be it is important to not just narrow it to RHEL because this happened on Debian and can be reproduced with a long logwatch mail sent every day.

Regs,

Stephan

yep_DD

Hi @fbartels ,

my Kopano server now completely stopped responding. The error logs show:

Mon Oct 16 07:49:37 2017: [error  ] SQL [00010431] result failed: BIGINT UNSIGNED value is out of range in '(`zarafa`.`properties`.`tag` - 0x8501)', Query: "SELECT 0,properties.tag,properties.type,properties.val_ulong,properties.val_string,properties.val_binary,properties.val_double,properties.val_longint,properties.val_hi,properties.val_lo, hierarchyid, names.nameid, names.namestring, names.guid FROM properties FORCE INDEX (PRIMARY) LEFT JOIN names ON (properties.tag-0x8501)=names.id WHERE hierarchyid=9 AND (tag <= 0x8500 OR names.id IS NOT NULL)"

Any idea?

fbartels

Hi @yep_dd ,

the required fix for this has been merged to master some days ago (https://stash.kopano.io/projects/KC/repos/kopanocore/commits/2d8b04dd8299aa52a60d4fd0c9e374c6fa94e7d8#common/database.cpp).

If I look at the timestamps with the community downloads the latest available package should include the fix for this.

Ihon

Hi,

I am using the Version »core-8.4.90.492_0+85-Ubuntu_16.04-amd64.tar.gz, the error is still occuring.
Can you please verify that the fix is included?

Thanks,

Ihon

yep_DD

I’ive just installed the lattest 492 release. Let’s see

PS:

dpkg: warning: downgrading libjansson4:amd64 from 2.9-1 to 2.7-1

I’ve been getting these since every Kopano update in the last months. Is there a reason Kopano is using 2.7-1?

Stephan

Ihon

Hi,

I am getting this error message multiple times:

Tue Oct 17 22:45:43 2017: [error ] SQL [00000200] result failed: BIGINT UNSIGNED value is out of range in ‘Zarafa.properties.tag - 0x8501’, Query: “SELECT 0,properties.tag,properties.type,properties.val_ulong,properties.val_string,properties.val_binary,properties.val_double,properties.val_longint,properties.val_hi,properties.val_lo, hierarchyid, names.nameid, names.namestring, names.guid FROM properties FORCE INDEX (PRIMARY) LEFT JOIN names ON (properties.tag-0x8501)=names.id WHERE hierarchyid=316249 AND (tag <= 0x8500 OR names.id IS NOT NULL)”

The server not working in that case anymore.

Regards,

Steffen

yep_DD

Hi @fbartels ,

I can confirm what @Ihon wrote. The problem persists.

Stephan

fbartels

@yep_dd @Ihon

Ok, I see. The last days our nightly upload did not succeed because of test failures. We are hopeful to resolve them today and have a new master build online later today.

Ihon

ok, thank you :-)

yep_DD

and also when the server crashes apache shows the following error occasionally:

[:error] [pid 22638] [client xxxxxxxxxx:44132] PHP Fatal error:  Uncaught MAPIException: MAPI error  in /usr/share/kopano-webapp/server/includes/util.php:225\nStack trace:\n#0 /usr/share/kopano-webapp/server/includes/util.php(225): mapi_msgstore_openentry(Resource id #36, '\\x00\\x00\\x00\\x00\\x00\\x0E\\xEA=`\\xE6I5\\xB5\\x8D"...')\n#1 /usr/share/kopano-webapp/index.php(242): cleanSearchFolders()\n#2 {main}\n  thrown in /usr/share/kopano-webapp/server/includes/util.php on line 225,```

yep_DD

This is the properties table. Changing tag to BIGINT has no effect. The error is getting really serious now, as I have a couple of messages in my postfix queue I can not get delivered!

0_1508326973203_Bildschirmfoto 2017-10-18 um 13.42.22.png

Ihon

@fbartels any update on the community build?
Still having the errors and server is crashing

Thanks,

Ihon

yep_DD

@Ihon same here, and also the community packages haven’t been updated in 7 days…

MS4R

@all same here and crash send my xen server in the eternal datagrounds

fbartels

Hi @yep_dd ,

yes we were still battling some test failures and prevented new packages from being uploaded. In the afternoon all tests were successful again so on monday there should be a new upload.

yep_DD

Hi @fbartels ,

could you just tell us if that issue has been specifically addressed? I see there are already new packages. I will deploy them in a second. Thanks for your help.

Stephan