KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items

umgfoin

Hello folks,
I’m getting MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) while opening several recent email-items since a few days. Tendency: Growing.

The problem manifestates in “Could not open message”-errors when opening/preview an email in webapp and of course in problems with all other backends/ clients.

If looking into the concerned item-properties via python-kopano, I get:

 File "./test-item.py", line 10, in <module>
    if 'test sent' in item.subject.lower():
  File "/usr/lib/python2.6/site-packages/kopano/item.py", line 251, in subject
    return self._get_fast(PR_SUBJECT_W, u'')
  File "/usr/lib/python2.6/site-packages/kopano/properties.py", line 120, in _get_fast
    return self.prop(proptag).value
  File "/usr/lib/python2.6/site-packages/kopano/properties.py", line 39, in prop
    return _prop.prop(self, self.mapiobj, proptag, create=create,
  File "/usr/lib/python2.6/site-packages/kopano/item.py", line 200, in mapiobj
    self.mapiobj = _utils.openentry_raw(self.store.mapiobj, self._entryid, self._content_flag)
  File "/usr/lib/python2.6/site-packages/kopano/utils.py", line 74, in openentry_raw
    return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
  File "/usr/lib/python2.6/site-packages/kopano/utils.py", line 68, in _openentry_helper
    return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
  File "/usr/lib/python2.6/site-packages/MAPICore.py", line 440, in OpenEntry
    def OpenEntry(self, *args): return _MAPICore.IMsgStore_OpenEntry(self, *args)
MAPI.Struct.MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)

Similar error-messages are logged by kopano-search, too.

The problem exisists since a few days. Initially only recently created items in the SentItems folder seemed to be concerned, but as far as I’ve seen, today, generally items, that have been moved or modified, recently are getting unreadable by time.

It is not clear, yet, when the corruption takes place, but it obviously happens with a certain delay between creation and failure: Items created/ sent yesterday were OK for a few hours, but inaccessible this morning.

Is there a way to manually repair these items? As far as I can see, they still exist in the database tables.
What might be the root-cause here?

++umgfoin.

Current versions: Ref. to signature.

umgfoin

So, new findings on this issue:
EntryID-corruption obviously takes place , if (git-repo built) server-core version is changing , or in other words, whenever a kopano-server instance creates a new entry in the databases’s versions-table.

If kopano-server is started with a new version-id, SentItems created with an earlier release become unreadable.
Same for received items being moved between folders, previously.

~~A rebuild of identical source but modified /kopano-core/version (here: incrementing microversion) triggers the behaviour.~~

Items created with an “elder” server-build (need to figure out exact commit) seem to be unaffected, as long as they haven’t been moved or modified recently.

~~Rebuilding the server with an earlier version-id doesn’t make SentItems-entryids readable again.~~

I’m setting version between configure and make to distinguish custom-builds from community-builds.

++umgfoin

umgfoin

OK,
I was wrong - things are even simpler:

build-version has no influence
a simple server-restart triggers the behaviour

Will update thread-topic.

Joost

Hi @umgfoin,

This looks like an issue we’ve been seeing in the master branch. We don’t advise using this branch on production since issues do end up in there from time to time (we try to prevent this of course).

This issue should be fixed in the latest version. And according to you signature you’re already running the latest version, so can you confirm the issue is fixed.

If not, we’ll have to find the cause of your issue.

Kind regards,

Joost

umgfoin

Hi @Joost ,
negative: Issue still present in master-fc39d888911

We don’t advise using this branch on production since issues do end up in there from time to time (we try to prevent this of course).

I’m aware of this and don’t exspect a flawless master-build.
++umgfoin.

Sinux

Hi,

+1

i have the same Issue, in the SentItems folder

0_1511024646676_Bildschirmfoto von »2017-11-18 18-01-57«.png

how can I get these emails?

umgfoin

@sinux said:

how can I get these emails?

Affected messages are not lost - they still exist on the db-server, but I guess, one will have to rewrite the PR_ENTRYIDs in the db’s properties- table to recover them. Deleting those entries is relatively simple via direct db-manipulation, but impossible through kopano-server, right now.

Good advice is to backup mailstores while kopano-server is still running (kopano-backup is very helpfull here) and restore them to an unaffected (earlier) KC-community-release. Current kc-8.4.x src-branch is safe, too, in this respect.

And, important: Not only the SentItems-folder is affected, any mail, you move between folders or modify, won’t be readable after server-restart.

++umgfoin.

Sinux

Many Thanks umgfoin,

I hope for a simpler troubleshooting by Kopano. :disappointed:
Which KC version is it now safe?
It concerns mails from 12-16.11. ~~after that it seems to work again~~.
For me this is the biggest software bug since using ZCP / KC (2012).
Since nothing happened over the years, i wanted to use the KC CV productively next month, which of course is completely eliminated.
I know that is advised against it but it has always worked very well.
When I try to delete the affected emails I get the error message that I have no user rights for this?

0_1511101362013_Bildschirmfoto von »2017-11-19 15-21-22«.png

I hope for a DB reorg script;)

Sinux

@umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

Hi @Joost ,
negative: Issue still present in master-fc39d888911

Same in KCO CV core-8.4.90.975_0+129-Ubuntu_14.04-amd64.tar.gz from 20.11.2017 (in Archive is the Version from 16.11.2017 23:58)

Joost

@Sinux @umgfoin

Thanks for the updates. I’m look into it.

Sinux

Hallo Joost,

same error with core-8.4.90.1042_0+136-Ubuntu_14.04-amd64.
In the server log file (6) no abnormalities.

Any Ideas???

Sinux

update
I tried to do a kopano-backup

2017-11-23 20:07:00,075 - backup0 - ERROR - could not process change for entryid F8A1E703000000003000000020686965F8A1E70300000000007000003D203300000000000000000040A4A20300000000 ([SPropValue(0x65E00102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7gq\xe7\x00\x00\x00\x00’), SPropValue(0x30080040, 2017/11/23 18:35:31 GMT), SPropValue(0x65E20102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x65E10102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\r\x00\x00\x00\x00\x00’), SPropValue(0x65E30102, ‘\x14\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x0FFF0102, ‘\xf8\xa1\xe7\x03\x00\x00\x00\x000\x00\x00\x00 hie\xf8\xa1\xe7\x03\x00\x00\x00\x00\x00p\x00\x00= 3\x00\x00\x00\x00\x00\x00\x00\x00\x00@\xa4\xa2\x03\x00\x00\x00\x00’), SPropValue(0x67AA000B, False), SPropValue(0x0E070003, 1L), SPropValue(0x0FFA0102, ‘xEr\x1d-\xb2K\xfb\xaa\xad\x0e\xc3 \xdc}\xd1’), SPropValue(0x67110003, 67087L), SPropValue(0x67150003, 17L)]):
2017-11-23 20:07:00,076 - backup0 - ERROR - Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/kopano/ics.py”, line 104, in ImportMessageChange
item.mapiobj = _utils.openentry_raw(mapistore, entryid.Value, 0)
File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 74, in openentry_raw
return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 68, in _openentry_helper
return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
File “/usr/lib/python2.7/dist-packages/MAPICore.py”, line 443, in OpenEntry
def OpenEntry(self, *args): return _MAPICore.IMsgStore_OpenEntry(self, *args)
MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)

umgfoin

That’s to be expected: Kopano-backup can’t access already damaged entries, either. K’-backup will rescue your mails as long as the server hasn’t been restarted.

++umgfoin.

Sinux

thanks umgfoin,

So I have to live with the broken mails!? I hope kopano find a solution!
Is the error fixed in your version?

bosim

@sinux said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

I hope kopano find a solution!

We are working to find a solution. The corresponding JIRA ticket is here:

https://jira.kopano.io/browse/KC-909

umgfoin

Hello @bosim,
if it helps, I can provide a properties-dump from a corruption from before and after server-restart (SELECT * FROM properties WHERE hierarchyid=xyz).
What is obvious there, some entryids (e.g. PR_SENDER_ENTRYID) have grown enormously in size after the IPM.note-object got corrupted.

bg umgfoin.

bosim

@umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

if it helps, I can provide a properties-dump from a corruption from before and after server-restart

Jan has found the problem and working on a fix. Maybe it will be useful to figure out how to repair stuff.

Bo

umgfoin

Good morning,
…and good news: @jengelh 's commit a3030e1dd8e will stop ongoing corruption, as reported above.

++umgfoin.

Sinux

Hallo @all
also does not work in version core-8.4.90.1063_0+138-Ubuntu_14.04-amd64 :(
I have moved new mail to deleted objects and back, reboot server, same issue.
How can I delete the annoying mails from the DB?

bosim

@Sinux I just had success locally by repairing one item. It is possible but it is time consuming. How many elements do you think it is for you?