[SOLVED] "Message cannot be opened" problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64



  • Hi,

    Since many years I update the community edition of zarafa/kopano core and webapp every couple of weeks, almost without any serious problems.

    Unfortunately yesterday, I found that in my mailboxes all messages received and sent within a certain time period (Nov 1st till Dec 11th) cannot be opened, nor deleted or moved. The folders containing these messages cannot be deleted as well. Older and newer messages are still ok.

    I checked this with webapp as well as with IMAP clients (mutt, Thunderbird) and plain IMAP via telnet. Mutt instantly freezes when entering a folder containing these messages.

    There are no hints in the server.log file.

    Even a kopano-backup run fails. It reports something like

    2017-12-12 15:04:10,437 - backup0 - ERROR - could not process change for entryid 8090DA040000000030000000306D5C308090DA0400000000300000000000000030000000000000006E5C306B5C30655C ([SPropValue(0x65E00102, 'u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xaf\x1a\x02\x00\x00\x00'), SPropValue(0x30080040, 2017/12/11 19:03:36 GMT), SPropValue(0x65E20102, 'u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xf0\x80\x0b\x00'), SPropValue(0x65E10102, 'u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xa3*\x00\x00\x00\x00\x00\x00'), SPropValue(0x65E30102, '\x14u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xf0\x80\x0b\x00'), SPropValue(0x0FFF0102, '\x80\x90\xda\x04\x00\x00\x00\x000\x00\x00\x000m\\0\x80\x90\xda\x04\x00\x00\x00\x000\x00\x00\x00\x00\x00\x00\x000\x00\x00\x00\x00\x00\x00\x00n\\0k\\0e\\'), SPropValue(0x67AA000B, False), SPropValue(0x0E070003, 1L), SPropValue(0x0FFA0102, 'I\xf7/\x02\x88\x06Lf\xa94\x15\x15\xe5{\x9b\x82'), SPropValue(0x67110003, 255235L), SPropValue(0x67150003, 10448L)]):
    2017-12-12 15:04:10,440 - backup0 - ERROR - Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/kopano/ics.py", line 104, in ImportMessageChange
        item.mapiobj = _utils.openentry_raw(mapistore, entryid.Value, 0)
      File "/usr/lib/python2.7/dist-packages/kopano/utils.py", line 74, in openentry_raw
        return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
      File "/usr/lib/python2.7/dist-packages/kopano/utils.py", line 68, in _openentry_helper
        return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
      File "/usr/lib/python2.7/dist-packages/MAPICore.py", line 965, in OpenEntry
        return _MAPICore.IMsgStore_OpenEntry(self, cbEntryID, lpInterface, ulFlags)
    MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)
    

    I recognized the problem yesterday, running version 8.4.90.1296 of core on Ubuntu 16.04 . It is still the case for the current version as of today. Maybe the time period is correlated with the installation of the core version that I used prior to 8.4.90.1296 (cannot remember the exact version).

    Maybe this is a known problem? Is there a way to identify and delete these corrupted messages in the SQL database to make the folders browsable again?

    Best regards,
    jacks


  • Kopano

    Hi @jackson5 ,

    issue known and origin of the problem is fixed. https://forum.kopano.io/topic/819/kc-entryid-corruption-on-server-restart-with-mapi-error-80040107-mapi_e_invalid_entryid-in-new-or-modified-mail-items gives some pointers how such an item could be corrected, but we decided not to invest further time into a fix script for this.



  • Oh no, it would be more than 100 messages.
    The proposed solution is not feasible to me, since it was suggested to search the properties table for the subject of the message, that usually not unique and leads to several hierarchy-IDs even in the unique case.

    I tried to recreate the database, but since kopano-backup fails also, this does not work.

    Moreover, I will not be able to successfully backup the mailboxes in the future.
    That would be a catastrophy to me, after all these years of using the community version and reporting bugs.

    If I move all valid messages from the affected folders, is there a way to delete a complete folder or all messages in a specified folder from the database?

    Any help would be appreciated.

    Best regards,
    jacks



  • Same problem here, that’s a real catastrophe because this problem is causing a z-push loop…

    Please help us to get a workaround/fix.

    Regards
    Richard



  • I have the same problem with several items.

    Best regards
    Martin


  • Kopano

    Here is a script that will fix the broken entryids of messages in a folder.

    https://gist.github.com/bosim/1705c8b23d9398bed9b7177708db82ac

    It requires python-mysql-connector, read instructions in comments in the beginning of the script.

    Backup your database before running this script!

    Now you have been warned.

    Bo



  • @bosim said in “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

    Here is a script that will fix the broken entryids of messages in a folder.

    https://gist.github.com/bosim/1705c8b23d9398bed9b7177708db82ac

    It requires python-mysql-connector, read instructions in comments in the beginning of the script.

    Backup your database before running this script!

    Now you have been warned.

    Bo

    Thank you very much, that works and saved my day :-)

    Regards
    Richard


  • Kopano

    @darootler said in “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

    Thank you very much, that works and saved my day

    Happy I could help!

    Bo



  • Yeah, thanx man.

    jacks



  • Dear readers

    Thank you all for your valuable assistance on the subject concerning entryid corruption. I still have problems with public folders and can not fix them.

    I get the following error messages in WebApp:

    • Could not open message.
    • Could not find message, either it has been moved or deleted or you don’t have access to open this message.

    My problems are not only related to mails, but also to contacts. As far as I know contacts were never mentioned before in the forum.

    I have successfully used the script fix-entryids.py to fix problems with mails and contacts of real users. On my machine I had to install the missing package python-mysql.connector.

    Now to my problem. I am having trouble to fix contacts in the public folders. I can not find out hierarchyid (step 4 mentioned in fix-entryids.py).

    The python session look like this:

    folder=kopano.Server().user(“SYSTEM”).folder(“Public Folders/Kontakte Test”)
    Traceback (most recent call last):
    File “<stdin>”, line 1, in <module>
    File “/usr/lib/python2.7/dist-packages/kopano/store.py”, line 378, in folder
    return self.subtree.folder(path, recurse=recurse, create=create)
    AttributeError: ‘NoneType’ object has no attribute ‘folder’

    I have run zarafa for several years and i have recently switched to kopano. This is my first posting in this forum. So please foregive me possible mistakes. And thank you all for your work and your assistance.

    Regards
    fallingdamps


  • Kopano

    @fallingdamps said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

    folder=kopano.Server().user(“SYSTEM”).folder(“Public Folders/Kontakte Test”)

    Try folder=kopano.Server().public_store.folder(“Kontakte Test”)

    Bo



  • @bosim said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

    folder=kopano.Server().public_store.folder(“Kontakte Test”)

    Thank you, this is working. Now I get hierarchyid.

    I tried the script with the storeguid of SYSTEM and hierachyid. In WebApp the known error messages was still appearing (although the script did update the exact number of contacts in the folder).

    And then I remembered the user who did create all (!) the contacts in this public folder. I tried it again with the storeguid of this user and now it works. Hooray, the contacts are not lost.

    I do not know the details of the database scheme and the consequences of the SQL statements in the script. And I do not know what happens if different users had created entries in the same corrupted public folder (solution more complex?). Therefore I hope that this was a good way to fix my problem.

    Thank you again for the quick response.

    Regards
    fallingdamps


  • Kopano

    @fallingdamps said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

    And then I remembered the user who did create all (!) the contacts in this public folder. I tried it again with the storeguid of this user and now it works. Hooray, the contacts are not lost.

    Interesting it works, because you should use the store guid for the public store, so kopano.Server().public_store.guid

    It should be safe to run the script again with that guid and the hierarchyid of the folder in the public store.

    Glad I could help.

    Have a nice weekend and best regards,
    Bo



  • It seems that I made a mistake. I used the following statement in step 3 to produce a wrong parameter:

    • kopano.Server().user(“SYSTEM”).store.guid

    I have now used the following and hopefully correct statements thanks to bosim:

    • kopano.Server().public_store.guid (step 3)
    • kopano.Server().public_store.folder(“Kontakte Test”).hierarchyid (in step 4)

    This solution may be obvious for developers and insiders but it was not for me.

    It is perhaps a good idea if someone could add this to the comments in the script fix-entryids.py.

    And now finally: have a nice weekend, too.

    Regards
    fallingdamps


  • Kopano

    @fallingdamps said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

    This solution may be obvious for developers and insiders but it was not for me.
    It is perhaps a good idea if someone could add this to the comments in the script fix-entryids.py.
    And now finally: have a nice weekend, too.

    Well you are right it could have been more simple, but I am happy you made it working. We do not expect more of these incidents to come, it was a very unfortunate situation having the database in a state where it could not be easily repaired by kopano-backup or kopano-fsck.

    Have a nice weekend
    Bo



  • Thanks @bosim for your solution!!!



  • Many thanks @bosim for your solution!
    All items are visible again :-)

    Best regards and merry xmas
    Martin



  • Hi together!

    After searching for hours i found this posting. Many thanks for writing this script! I had so much trouble with my wife because of missing mails, now most of them are back again.

    I have one question: I have to run the script on the “Sent Items” folder, but I do not know how to access it to get the hierarchy ID. I thought about searching in the hierarchy-table for the parent of the id of the inbox, looked down into the hierarchies with matching parent, but I endet in not knowing where to find in the database the matching name, so im stuck here.

    All I tried around this gave errors (without blanks, other quotes and so on).
    folder=kopano.Server().user(“xxx”).inbox.folder(“Sent Items”)
    folder=kopano.Server().user(“xxx”).inbox.folder(“Gesendete Objekte”)
    The error is always like: kopano.errors.NotFoundError: no such folder: <what I tried>

    When I try
    folder=kopano.Server().user(“xxx”).outbox
    and get the id
    the script does nothing :-(.

    I would really appreciate if somebody could give me a hint,

    kind regards,
    Christian



  • @tueftler17

    Try this:

    folder=kopano.Server().user("xxx").folder("Sent Items")
    

    Regards
    Richard



  • @Richard:
    Perfect, as usual the solution was easy but in Germany we say: “I had tomatoes on my eyes”, so I did not find it.

    Many thanks to you, now almost all messages are back again. I still see some errors when I do a kopano-backup but I will try to ignore them. Hopefully they won’t do a message-loop with active-sync.

    Again: It was very helpful, I wish you all a great 2018!

    kind regards,
    Christian


Log in to reply
 

Looks like your connection to Kopano Community Forum was lost, please wait while we try to reconnect.