KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items

Sinux

Hi,

+1

i have the same Issue, in the SentItems folder

0_1511024646676_Bildschirmfoto von »2017-11-18 18-01-57«.png

how can I get these emails?

umgfoin

@sinux said:

how can I get these emails?

Affected messages are not lost - they still exist on the db-server, but I guess, one will have to rewrite the PR_ENTRYIDs in the db’s properties- table to recover them. Deleting those entries is relatively simple via direct db-manipulation, but impossible through kopano-server, right now.

Good advice is to backup mailstores while kopano-server is still running (kopano-backup is very helpfull here) and restore them to an unaffected (earlier) KC-community-release. Current kc-8.4.x src-branch is safe, too, in this respect.

And, important: Not only the SentItems-folder is affected, any mail, you move between folders or modify, won’t be readable after server-restart.

++umgfoin.

Sinux

Many Thanks umgfoin,

I hope for a simpler troubleshooting by Kopano. :disappointed:
Which KC version is it now safe?
It concerns mails from 12-16.11. ~~after that it seems to work again~~.
For me this is the biggest software bug since using ZCP / KC (2012).
Since nothing happened over the years, i wanted to use the KC CV productively next month, which of course is completely eliminated.
I know that is advised against it but it has always worked very well.
When I try to delete the affected emails I get the error message that I have no user rights for this?

0_1511101362013_Bildschirmfoto von »2017-11-19 15-21-22«.png

I hope for a DB reorg script;)

Sinux

@umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

Hi @Joost ,
negative: Issue still present in master-fc39d888911

Same in KCO CV core-8.4.90.975_0+129-Ubuntu_14.04-amd64.tar.gz from 20.11.2017 (in Archive is the Version from 16.11.2017 23:58)

Joost

@Sinux @umgfoin

Thanks for the updates. I’m look into it.

Sinux

Hallo Joost,

same error with core-8.4.90.1042_0+136-Ubuntu_14.04-amd64.
In the server log file (6) no abnormalities.

Any Ideas???

Sinux

update
I tried to do a kopano-backup

2017-11-23 20:07:00,075 - backup0 - ERROR - could not process change for entryid F8A1E703000000003000000020686965F8A1E70300000000007000003D203300000000000000000040A4A20300000000 ([SPropValue(0x65E00102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7gq\xe7\x00\x00\x00\x00’), SPropValue(0x30080040, 2017/11/23 18:35:31 GMT), SPropValue(0x65E20102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x65E10102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\r\x00\x00\x00\x00\x00’), SPropValue(0x65E30102, ‘\x14\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x0FFF0102, ‘\xf8\xa1\xe7\x03\x00\x00\x00\x000\x00\x00\x00 hie\xf8\xa1\xe7\x03\x00\x00\x00\x00\x00p\x00\x00= 3\x00\x00\x00\x00\x00\x00\x00\x00\x00@\xa4\xa2\x03\x00\x00\x00\x00’), SPropValue(0x67AA000B, False), SPropValue(0x0E070003, 1L), SPropValue(0x0FFA0102, ‘xEr\x1d-\xb2K\xfb\xaa\xad\x0e\xc3 \xdc}\xd1’), SPropValue(0x67110003, 67087L), SPropValue(0x67150003, 17L)]):
2017-11-23 20:07:00,076 - backup0 - ERROR - Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/kopano/ics.py”, line 104, in ImportMessageChange
item.mapiobj = _utils.openentry_raw(mapistore, entryid.Value, 0)
File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 74, in openentry_raw
return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 68, in _openentry_helper
return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
File “/usr/lib/python2.7/dist-packages/MAPICore.py”, line 443, in OpenEntry
def OpenEntry(self, *args): return _MAPICore.IMsgStore_OpenEntry(self, *args)
MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)

umgfoin

That’s to be expected: Kopano-backup can’t access already damaged entries, either. K’-backup will rescue your mails as long as the server hasn’t been restarted.

++umgfoin.

Sinux

thanks umgfoin,

So I have to live with the broken mails!? I hope kopano find a solution!
Is the error fixed in your version?

bosim

@sinux said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

I hope kopano find a solution!

We are working to find a solution. The corresponding JIRA ticket is here:

https://jira.kopano.io/browse/KC-909

umgfoin

Hello @bosim,
if it helps, I can provide a properties-dump from a corruption from before and after server-restart (SELECT * FROM properties WHERE hierarchyid=xyz).
What is obvious there, some entryids (e.g. PR_SENDER_ENTRYID) have grown enormously in size after the IPM.note-object got corrupted.

bg umgfoin.

bosim

@umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

if it helps, I can provide a properties-dump from a corruption from before and after server-restart

Jan has found the problem and working on a fix. Maybe it will be useful to figure out how to repair stuff.

Bo

umgfoin

Good morning,
…and good news: @jengelh 's commit a3030e1dd8e will stop ongoing corruption, as reported above.

++umgfoin.

Sinux

Hallo @all
also does not work in version core-8.4.90.1063_0+138-Ubuntu_14.04-amd64 :(
I have moved new mail to deleted objects and back, reboot server, same issue.
How can I delete the annoying mails from the DB?

bosim

@Sinux I just had success locally by repairing one item. It is possible but it is time consuming. How many elements do you think it is for you?

Sinux

thanks @bosim it affects 14 mails so far but the error still exists…
plz, I would like to hear your solution anyway
thanks

umgfoin

@sinux
as JIRA ticket KC-909 status shows, all this is currently under review.

… and I tend to believe, patience is good advice, here.

Jan’s commit a3030e1dd8e hasn’t been pushed to the master-branch, yet - thus it hardly can be contained in any community-build, either.

Concerning deletion of sgl. mailitems, the following approach should work:

take down kopano-server or atleast prevent multi-user access.
start a mysql-cmd-client mysql -hlocalhost -udb_user db_name
identify hierarchyids of (broken) items by a query in the properties-table: e.g.
SELECT hierarchyid FROM properties WHERE val_string LIKE "Subject_of_broken_mail%" AND hierarchyid IN (SELECT id FROM hierarchy WHERE type=5);
delete related properties, tproperties and hierarchyentries:
DELETE from hierarchy WHERE id=<result from above>;
DELETE from tproperties WHERE hierarchyid=<result from above;>
DELETE from indexedproperties WHERE hierarchyid=<result from above;>

But:

reliably backup your db prior any modifications
Be careful with the first query - your search criteria need to be unambiguously
eventually use START TRANSACTION/ ROLLBACK/ COMMIT to recover from typos or mistakes

++umgfoin

Sinux

@umgfoin thanks, I’ll try it
update
thanks it works but I would still like to try his suggestion @bosim

Sinux

Hi,

with this version core-8.4.90.1064_0+139-Ubuntu_14.04-amd64 it seems to work again.
Now I hope on bosim data recovery.

thanks

It appears that a MAPI_MESSAGE’s entryid is only ever referenced from another message’s 0x80FD, 0x80FE and 0x825D properties. As these seem to be some obscure properties, regenerating entryids of messages (not so with folders/stores) at will should generally work.