KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items



  • Hello folks,
    I’m getting MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) while opening several recent email-items since a few days. Tendency: Growing.

    The problem manifestates in “Could not open message”-errors when opening/preview an email in webapp and of course in problems with all other backends/ clients.

    If looking into the concerned item-properties via python-kopano, I get:

     File "./test-item.py", line 10, in <module>
        if 'test sent' in item.subject.lower():
      File "/usr/lib/python2.6/site-packages/kopano/item.py", line 251, in subject
        return self._get_fast(PR_SUBJECT_W, u'')
      File "/usr/lib/python2.6/site-packages/kopano/properties.py", line 120, in _get_fast
        return self.prop(proptag).value
      File "/usr/lib/python2.6/site-packages/kopano/properties.py", line 39, in prop
        return _prop.prop(self, self.mapiobj, proptag, create=create,
      File "/usr/lib/python2.6/site-packages/kopano/item.py", line 200, in mapiobj
        self.mapiobj = _utils.openentry_raw(self.store.mapiobj, self._entryid, self._content_flag)
      File "/usr/lib/python2.6/site-packages/kopano/utils.py", line 74, in openentry_raw
        return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
      File "/usr/lib/python2.6/site-packages/kopano/utils.py", line 68, in _openentry_helper
        return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
      File "/usr/lib/python2.6/site-packages/MAPICore.py", line 440, in OpenEntry
        def OpenEntry(self, *args): return _MAPICore.IMsgStore_OpenEntry(self, *args)
    MAPI.Struct.MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)
    
    

    Similar error-messages are logged by kopano-search, too.

    The problem exisists since a few days. Initially only recently created items in the SentItems folder seemed to be concerned, but as far as I’ve seen, today, generally items, that have been moved or modified, recently are getting unreadable by time.

    It is not clear, yet, when the corruption takes place, but it obviously happens with a certain delay between creation and failure: Items created/ sent yesterday were OK for a few hours, but inaccessible this morning.

    • Is there a way to manually repair these items? As far as I can see, they still exist in the database tables.
    • What might be the root-cause here?

    ++umgfoin.

    Current versions: Ref. to signature.



  • So, new findings on this issue:
    EntryID-corruption obviously takes place , if (git-repo built) server-core version is changing , or in other words, whenever a kopano-server instance creates a new entry in the databases’s versions-table.

    If kopano-server is started with a new version-id, SentItems created with an earlier release become unreadable.
    Same for received items being moved between folders, previously.

    A rebuild of identical source but modified /kopano-core/version (here: incrementing microversion) triggers the behaviour.

    Items created with an “elder” server-build (need to figure out exact commit) seem to be unaffected, as long as they haven’t been moved or modified recently.

    Rebuilding the server with an earlier version-id doesn’t make SentItems-entryids readable again.

    I’m setting version between configure and make to distinguish custom-builds from community-builds.

    ++umgfoin



  • OK,
    I was wrong - things are even simpler:

    • build-version has no influence
    • a simple server-restart triggers the behaviour

    Will update thread-topic.


  • Kopano

    Hi @umgfoin,

    This looks like an issue we’ve been seeing in the master branch. We don’t advise using this branch on production since issues do end up in there from time to time (we try to prevent this of course).

    This issue should be fixed in the latest version. And according to you signature you’re already running the latest version, so can you confirm the issue is fixed.

    If not, we’ll have to find the cause of your issue.

    Kind regards,

    Joost



  • Hi @Joost ,
    negative: Issue still present in master-fc39d888911

    We don’t advise using this branch on production since issues do end up in there from time to time (we try to prevent this of course).

    I’m aware of this and don’t exspect a flawless master-build.
    ++umgfoin.



  • Hi,

    +1

    i have the same Issue, in the SentItems folder

    0_1511024646676_Bildschirmfoto von »2017-11-18 18-01-57«.png

    how can I get these emails?



  • @sinux said:

    how can I get these emails?

    Affected messages are not lost - they still exist on the db-server, but I guess, one will have to rewrite the PR_ENTRYIDs in the db’s properties- table to recover them. Deleting those entries is relatively simple via direct db-manipulation, but impossible through kopano-server, right now.

    Good advice is to backup mailstores while kopano-server is still running (kopano-backup is very helpfull here) and restore them to an unaffected (earlier) KC-community-release. Current kc-8.4.x src-branch is safe, too, in this respect.

    And, important: Not only the SentItems-folder is affected, any mail, you move between folders or modify, won’t be readable after server-restart.

    ++umgfoin.



  • Many Thanks umgfoin,

    I hope for a simpler troubleshooting by Kopano. :disappointed:
    Which KC version is it now safe?
    It concerns mails from 12-16.11. after that it seems to work again.
    For me this is the biggest software bug since using ZCP / KC (2012).
    Since nothing happened over the years, i wanted to use the KC CV productively next month, which of course is completely eliminated.
    I know that is advised against it but it has always worked very well.
    When I try to delete the affected emails I get the error message that I have no user rights for this?

    0_1511101362013_Bildschirmfoto von »2017-11-19 15-21-22«.png

    I hope for a DB reorg script;)



  • @umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

    Hi @Joost ,
    negative: Issue still present in master-fc39d888911

    Same in KCO CV core-8.4.90.975_0+129-Ubuntu_14.04-amd64.tar.gz from 20.11.2017 (in Archive is the Version from 16.11.2017 23:58)


  • Kopano

    @Sinux @umgfoin

    Thanks for the updates. I’m look into it.



  • Hallo Joost,

    same error with core-8.4.90.1042_0+136-Ubuntu_14.04-amd64.
    In the server log file (6) no abnormalities.

    Any Ideas???



  • update
    I tried to do a kopano-backup

    2017-11-23 20:07:00,075 - backup0 - ERROR - could not process change for entryid F8A1E703000000003000000020686965F8A1E70300000000007000003D203300000000000000000040A4A20300000000 ([SPropValue(0x65E00102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7gq\xe7\x00\x00\x00\x00’), SPropValue(0x30080040, 2017/11/23 18:35:31 GMT), SPropValue(0x65E20102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x65E10102, ‘\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\r\x00\x00\x00\x00\x00’), SPropValue(0x65E30102, ‘\x14\x07\xf54[wnCT\x8cg(A\xf0\xf2\xf7g\xa8\xf3\x00\x00’), SPropValue(0x0FFF0102, ‘\xf8\xa1\xe7\x03\x00\x00\x00\x000\x00\x00\x00 hie\xf8\xa1\xe7\x03\x00\x00\x00\x00\x00p\x00\x00= 3\x00\x00\x00\x00\x00\x00\x00\x00\x00@\xa4\xa2\x03\x00\x00\x00\x00’), SPropValue(0x67AA000B, False), SPropValue(0x0E070003, 1L), SPropValue(0x0FFA0102, ‘xEr\x1d-\xb2K\xfb\xaa\xad\x0e\xc3 \xdc}\xd1’), SPropValue(0x67110003, 67087L), SPropValue(0x67150003, 17L)]):
    2017-11-23 20:07:00,076 - backup0 - ERROR - Traceback (most recent call last):
    File “/usr/lib/python2.7/dist-packages/kopano/ics.py”, line 104, in ImportMessageChange
    item.mapiobj = _utils.openentry_raw(mapistore, entryid.Value, 0)
    File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 74, in openentry_raw
    return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
    File “/usr/lib/python2.7/dist-packages/kopano/utils.py”, line 68, in _openentry_helper
    return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
    File “/usr/lib/python2.7/dist-packages/MAPICore.py”, line 443, in OpenEntry
    def OpenEntry(self, *args): return _MAPICore.IMsgStore_OpenEntry(self, *args)
    MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)



  • That’s to be expected: Kopano-backup can’t access already damaged entries, either. K’-backup will rescue your mails as long as the server hasn’t been restarted.

    ++umgfoin.



  • thanks umgfoin,

    So I have to live with the broken mails!? I hope kopano find a solution!
    Is the error fixed in your version?


  • Kopano

    @sinux said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

    I hope kopano find a solution!

    We are working to find a solution. The corresponding JIRA ticket is here:

    https://jira.kopano.io/browse/KC-909



  • Hello @bosim,
    if it helps, I can provide a properties-dump from a corruption from before and after server-restart (SELECT * FROM properties WHERE hierarchyid=xyz).
    What is obvious there, some entryids (e.g. PR_SENDER_ENTRYID) have grown enormously in size after the IPM.note-object got corrupted.

    bg umgfoin.


  • Kopano

    @umgfoin said in KC: EntryID-corruption on server-restart with MAPI error 80040107 (MAPI_E_INVALID_ENTRYID) in new or modified mail-items:

    if it helps, I can provide a properties-dump from a corruption from before and after server-restart

    Jan has found the problem and working on a fix. Maybe it will be useful to figure out how to repair stuff.

    Bo



  • Good morning,
    …and good news: @jengelh 's commit a3030e1dd8e will stop ongoing corruption, as reported above.

    ++umgfoin.



  • Hallo @all
    also does not work in version core-8.4.90.1063_0+138-Ubuntu_14.04-amd64 :(
    I have moved new mail to deleted objects and back, reboot server, same issue.
    How can I delete the annoying mails from the DB?


  • Kopano

    @Sinux I just had success locally by repairing one item. It is possible but it is time consuming. How many elements do you think it is for you?


Log in to reply
 

Looks like your connection to Kopano Community Forum was lost, please wait while we try to reconnect.