KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items

umgfoin

Hi @Joost ,
negative: Issue still present in master-fc39d888911

We don’t advise using this branch on production since issues do end up in there from time to time (we try to prevent this of course).

I’m aware of this and don’t exspect a flawless master-build.
++umgfoin.

Sinux

Hi,

+1

i have the same Issue, in the SentItems folder

0_1511024646676_Bildschirmfoto von »2017-11-18 18-01-57«.png

how can I get these emails?

umgfoin

@sinux said:

how can I get these emails?

Affected messages are not lost - they still exist on the db-server, but I guess, one will have to rewrite the PR_ENTRYIDs in the db’s properties- table to recover them. Deleting those entries is relatively simple via direct db-manipulation, but impossible through kopano-server, right now.

Good advice is to backup mailstores while kopano-server is still running (kopano-backup is very helpfull here) and restore them to an unaffected (earlier) KC-community-release. Current kc-8.4.x src-branch is safe, too, in this respect.

And, important: Not only the SentItems-folder is affected, any mail, you move between folders or modify, won’t be readable after server-restart.

++umgfoin.

Sinux

Many Thanks umgfoin,

I hope for a simpler troubleshooting by Kopano. :disappointed:
Which KC version is it now safe?
It concerns mails from 12-16.11. ~~after that it seems to work again~~.
For me this is the biggest software bug since using ZCP / KC (2012).
Since nothing happened over the years, i wanted to use the KC CV productively next month, which of course is completely eliminated.
I know that is advised against it but it has always worked very well.
When I try to delete the affected emails I get the error message that I have no user rights for this?

0_1511101362013_Bildschirmfoto von »2017-11-19 15-21-22«.png

I hope for a DB reorg script;)

Sinux

@umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

Hi @Joost ,
negative: Issue still present in master-fc39d888911

Same in KCO CV core-8.4.90.975_0+129-Ubuntu_14.04-amd64.tar.gz from 20.11.2017 (in Archive is the Version from 16.11.2017 23:58)

Joost

@Sinux @umgfoin

Thanks for the updates. I’m look into it.

Sinux

Hallo Joost,

same error with core-8.4.90.1042_0+136-Ubuntu_14.04-amd64.
In the server log file (6) no abnormalities.

Any Ideas???

Sinux

update
I tried to do a kopano-backup

2017-11-23 20:07:00,075 - backup0 - ERROR - could not process change for entryid F8A1E703000000003000000020686965F8A1E70300000000007000003D203300000000000000000040A4A20300000000 ([SPropValue(0x65E00102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7gq\xe7\x00\x00\x00\x00’), SPropValue(0x30080040, 2017/11/23 18:35:31 GMT), SPropValue(0x65E20102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x65E10102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\r\x00\x00\x00\x00\x00’), SPropValue(0x65E30102, ‘\x14\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x0FFF0102, ‘\xf8\xa1\xe7\x03\x00\x00\x00\x000\x00\x00\x00 hie\xf8\xa1\xe7\x03\x00\x00\x00\x00\x00p\x00\x00= 3\x00\x00\x00\x00\x00\x00\x00\x00\x00@\xa4\xa2\x03\x00\x00\x00\x00’), SPropValue(0x67AA000B, False), SPropValue(0x0E070003, 1L), SPropValue(0x0FFA0102, ‘xEr\x1d-\xb2K\xfb\xaa\xad\x0e\xc3 \xdc}\xd1’), SPropValue(0x67110003, 67087L), SPropValue(0x67150003, 17L)]):
2017-11-23 20:07:00,076 - backup0 - ERROR - Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/kopano/ics.py”, line 104, in ImportMessageChange
item.mapiobj = _utils.openentry_raw(mapistore, entryid.Value, 0)
File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 74, in openentry_raw
return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 68, in _openentry_helper
return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
File “/usr/lib/python2.7/dist-packages/MAPICore.py”, line 443, in OpenEntry
def OpenEntry(self, *args): return _MAPICore.IMsgStore_OpenEntry(self, *args)
MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)

umgfoin

That’s to be expected: Kopano-backup can’t access already damaged entries, either. K’-backup will rescue your mails as long as the server hasn’t been restarted.

++umgfoin.

Sinux

thanks umgfoin,

So I have to live with the broken mails!? I hope kopano find a solution!
Is the error fixed in your version?

bosim

@sinux said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

I hope kopano find a solution!

We are working to find a solution. The corresponding JIRA ticket is here:

https://jira.kopano.io/browse/KC-909

umgfoin

Hello @bosim,
if it helps, I can provide a properties-dump from a corruption from before and after server-restart (SELECT * FROM properties WHERE hierarchyid=xyz).
What is obvious there, some entryids (e.g. PR_SENDER_ENTRYID) have grown enormously in size after the IPM.note-object got corrupted.

bg umgfoin.

bosim

@umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

if it helps, I can provide a properties-dump from a corruption from before and after server-restart

Jan has found the problem and working on a fix. Maybe it will be useful to figure out how to repair stuff.

Bo

umgfoin

Good morning,
…and good news: @jengelh 's commit a3030e1dd8e will stop ongoing corruption, as reported above.

++umgfoin.

Sinux

Hallo @all
also does not work in version core-8.4.90.1063_0+138-Ubuntu_14.04-amd64 :(
I have moved new mail to deleted objects and back, reboot server, same issue.
How can I delete the annoying mails from the DB?

bosim

@Sinux I just had success locally by repairing one item. It is possible but it is time consuming. How many elements do you think it is for you?

Sinux

thanks @bosim it affects 14 mails so far but the error still exists…
plz, I would like to hear your solution anyway
thanks

umgfoin

@sinux
as JIRA ticket KC-909 status shows, all this is currently under review.

… and I tend to believe, patience is good advice, here.

Jan’s commit a3030e1dd8e hasn’t been pushed to the master-branch, yet - thus it hardly can be contained in any community-build, either.

Concerning deletion of sgl. mailitems, the following approach should work:

take down kopano-server or atleast prevent multi-user access.
start a mysql-cmd-client mysql -hlocalhost -udb_user db_name
identify hierarchyids of (broken) items by a query in the properties-table: e.g.
SELECT hierarchyid FROM properties WHERE val_string LIKE "Subject_of_broken_mail%" AND hierarchyid IN (SELECT id FROM hierarchy WHERE type=5);
delete related properties, tproperties and hierarchyentries:
DELETE from hierarchy WHERE id=<result from above>;
DELETE from tproperties WHERE hierarchyid=<result from above;>
DELETE from indexedproperties WHERE hierarchyid=<result from above;>

But:

reliably backup your db prior any modifications
Be careful with the first query - your search criteria need to be unambiguously
eventually use START TRANSACTION/ ROLLBACK/ COMMIT to recover from typos or mistakes

++umgfoin

Sinux

@umgfoin thanks, I’ll try it
update
thanks it works but I would still like to try his suggestion @bosim

Sinux

Hi,

with this version core-8.4.90.1064_0+139-Ubuntu_14.04-amd64 it seems to work again.
Now I hope on bosim data recovery.

thanks