[SOLVED] "Message cannot be opened" problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64

jackson5

Hi,

Since many years I update the community edition of zarafa/kopano core and webapp every couple of weeks, almost without any serious problems.

Unfortunately yesterday, I found that in my mailboxes all messages received and sent within a certain time period (Nov 1st till Dec 11th) cannot be opened, nor deleted or moved. The folders containing these messages cannot be deleted as well. Older and newer messages are still ok.

I checked this with webapp as well as with IMAP clients (mutt, Thunderbird) and plain IMAP via telnet. Mutt instantly freezes when entering a folder containing these messages.

There are no hints in the server.log file.

Even a kopano-backup run fails. It reports something like

2017-12-12 15:04:10,437 - backup0 - ERROR - could not process change for entryid 8090DA040000000030000000306D5C308090DA0400000000300000000000000030000000000000006E5C306B5C30655C ([SPropValue(0x65E00102, 'u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xaf\x1a\x02\x00\x00\x00'), SPropValue(0x30080040, 2017/12/11 19:03:36 GMT), SPropValue(0x65E20102, 'u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xf0\x80\x0b\x00'), SPropValue(0x65E10102, 'u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xa3*\x00\x00\x00\x00\x00\x00'), SPropValue(0x65E30102, '\x14u\xd3\xc7\x0e\x85\x9cI\xf2\x94@)F\x96\x1dv\x12\xf0\x80\x0b\x00'), SPropValue(0x0FFF0102, '\x80\x90\xda\x04\x00\x00\x00\x000\x00\x00\x000m\\0\x80\x90\xda\x04\x00\x00\x00\x000\x00\x00\x00\x00\x00\x00\x000\x00\x00\x00\x00\x00\x00\x00n\\0k\\0e\\'), SPropValue(0x67AA000B, False), SPropValue(0x0E070003, 1L), SPropValue(0x0FFA0102, 'I\xf7/\x02\x88\x06Lf\xa94\x15\x15\xe5{\x9b\x82'), SPropValue(0x67110003, 255235L), SPropValue(0x67150003, 10448L)]):
2017-12-12 15:04:10,440 - backup0 - ERROR - Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kopano/ics.py", line 104, in ImportMessageChange
    item.mapiobj = _utils.openentry_raw(mapistore, entryid.Value, 0)
  File "/usr/lib/python2.7/dist-packages/kopano/utils.py", line 74, in openentry_raw
    return _openentry_helper(mapistore, entryid, flags | MAPI_MODIFY)
  File "/usr/lib/python2.7/dist-packages/kopano/utils.py", line 68, in _openentry_helper
    return mapistore.OpenEntry(entryid, IID_IECMessageRaw, flags)
  File "/usr/lib/python2.7/dist-packages/MAPICore.py", line 965, in OpenEntry
    return _MAPICore.IMsgStore_OpenEntry(self, cbEntryID, lpInterface, ulFlags)
MAPIErrorInvalidEntryid: MAPI error 80040107 (MAPI_E_INVALID_ENTRYID)

I recognized the problem yesterday, running version 8.4.90.1296 of core on Ubuntu 16.04 . It is still the case for the current version as of today. Maybe the time period is correlated with the installation of the core version that I used prior to 8.4.90.1296 (cannot remember the exact version).

Maybe this is a known problem? Is there a way to identify and delete these corrupted messages in the SQL database to make the folders browsable again?

Best regards,
jacks

fbartels

Hi @jackson5 ,

issue known and origin of the problem is fixed. https://forum.kopano.io/topic/819/kc-entryid-corruption-on-server-restart-with-mapi-error-80040107-mapi_e_invalid_entryid-in-new-or-modified-mail-items gives some pointers how such an item could be corrected, but we decided not to invest further time into a fix script for this.

jackson5

Oh no, it would be more than 100 messages.
The proposed solution is not feasible to me, since it was suggested to search the properties table for the subject of the message, that usually not unique and leads to several hierarchy-IDs even in the unique case.

I tried to recreate the database, but since kopano-backup fails also, this does not work.

Moreover, I will not be able to successfully backup the mailboxes in the future.
That would be a catastrophy to me, after all these years of using the community version and reporting bugs.

If I move all valid messages from the affected folders, is there a way to delete a complete folder or all messages in a specified folder from the database?

Any help would be appreciated.

Best regards,
jacks

darootler

Same problem here, that’s a real catastrophe because this problem is causing a z-push loop…

Please help us to get a workaround/fix.

Regards
Richard

Martin

I have the same problem with several items.

Best regards
Martin

bosim

Here is a script that will fix the broken entryids of messages in a folder.

https://gist.github.com/bosim/1705c8b23d9398bed9b7177708db82ac

It requires python-mysql-connector, read instructions in comments in the beginning of the script.

Backup your database before running this script!

Now you have been warned.

Bo

darootler

@bosim said in “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

Here is a script that will fix the broken entryids of messages in a folder.

https://gist.github.com/bosim/1705c8b23d9398bed9b7177708db82ac

It requires python-mysql-connector, read instructions in comments in the beginning of the script.

Backup your database before running this script!

Now you have been warned.

Bo

Thank you very much, that works and saved my day :-)

Regards
Richard

bosim

@darootler said in “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

Thank you very much, that works and saved my day

Happy I could help!

Bo

jackson5

Yeah, thanx man.

jacks

fallingdamps

Dear readers

Thank you all for your valuable assistance on the subject concerning entryid corruption. I still have problems with public folders and can not fix them.

I get the following error messages in WebApp:

Could not open message.
Could not find message, either it has been moved or deleted or you don’t have access to open this message.

My problems are not only related to mails, but also to contacts. As far as I know contacts were never mentioned before in the forum.

I have successfully used the script fix-entryids.py to fix problems with mails and contacts of real users. On my machine I had to install the missing package python-mysql.connector.

Now to my problem. I am having trouble to fix contacts in the public folders. I can not find out hierarchyid (step 4 mentioned in fix-entryids.py).

The python session look like this:

folder=kopano.Server().user(“SYSTEM”).folder(“Public Folders/Kontakte Test”)
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
File “/usr/lib/python2.7/dist-packages/kopano/store.py”, line 378, in folder
return self.subtree.folder(path, recurse=recurse, create=create)
AttributeError: ‘NoneType’ object has no attribute ‘folder’

I have run zarafa for several years and i have recently switched to kopano. This is my first posting in this forum. So please foregive me possible mistakes. And thank you all for your work and your assistance.

Regards
fallingdamps

bosim

@fallingdamps said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

folder=kopano.Server().user(“SYSTEM”).folder(“Public Folders/Kontakte Test”)

Try folder=kopano.Server().public_store.folder(“Kontakte Test”)

Bo

fallingdamps

@bosim said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

folder=kopano.Server().public_store.folder(“Kontakte Test”)

Thank you, this is working. Now I get hierarchyid.

I tried the script with the storeguid of SYSTEM and hierachyid. In WebApp the known error messages was still appearing (although the script did update the exact number of contacts in the folder).

And then I remembered the user who did create all (!) the contacts in this public folder. I tried it again with the storeguid of this user and now it works. Hooray, the contacts are not lost.

I do not know the details of the database scheme and the consequences of the SQL statements in the script. And I do not know what happens if different users had created entries in the same corrupted public folder (solution more complex?). Therefore I hope that this was a good way to fix my problem.

Thank you again for the quick response.

Regards
fallingdamps

bosim

@fallingdamps said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

And then I remembered the user who did create all (!) the contacts in this public folder. I tried it again with the storeguid of this user and now it works. Hooray, the contacts are not lost.

Interesting it works, because you should use the store guid for the public store, so kopano.Server().public_store.guid

It should be safe to run the script again with that guid and the hierarchyid of the folder in the public store.

Glad I could help.

Have a nice weekend and best regards,
Bo

fallingdamps

It seems that I made a mistake. I used the following statement in step 3 to produce a wrong parameter:

kopano.Server().user(“SYSTEM”).store.guid

I have now used the following and hopefully correct statements thanks to bosim:

kopano.Server().public_store.guid (step 3)
kopano.Server().public_store.folder(“Kontakte Test”).hierarchyid (in step 4)

This solution may be obvious for developers and insiders but it was not for me.

It is perhaps a good idea if someone could add this to the comments in the script fix-entryids.py.

And now finally: have a nice weekend, too.

Regards
fallingdamps

bosim

@fallingdamps said in [SOLVED] “Message cannot be opened” problem with core-8.4.90.1296_0+157-Ubuntu_16.04-amd64:

This solution may be obvious for developers and insiders but it was not for me.
It is perhaps a good idea if someone could add this to the comments in the script fix-entryids.py.
And now finally: have a nice weekend, too.

Well you are right it could have been more simple, but I am happy you made it working. We do not expect more of these incidents to come, it was a very unfortunate situation having the database in a state where it could not be easily repaired by kopano-backup or kopano-fsck.

Have a nice weekend
Bo

Sinux

Thanks @bosim for your solution!!!

Martin

Many thanks @bosim for your solution!
All items are visible again :-)

Best regards and merry xmas
Martin

tueftler17

Hi together!

After searching for hours i found this posting. Many thanks for writing this script! I had so much trouble with my wife because of missing mails, now most of them are back again.

I have one question: I have to run the script on the “Sent Items” folder, but I do not know how to access it to get the hierarchy ID. I thought about searching in the hierarchy-table for the parent of the id of the inbox, looked down into the hierarchies with matching parent, but I endet in not knowing where to find in the database the matching name, so im stuck here.

All I tried around this gave errors (without blanks, other quotes and so on).
folder=kopano.Server().user(“xxx”).inbox.folder(“Sent Items”)
folder=kopano.Server().user(“xxx”).inbox.folder(“Gesendete Objekte”)
The error is always like: kopano.errors.NotFoundError: no such folder: <what I tried>

When I try
folder=kopano.Server().user(“xxx”).outbox
and get the id
the script does nothing :-(.

I would really appreciate if somebody could give me a hint,

kind regards,
Christian

darootler

@tueftler17

Try this:

folder=kopano.Server().user("xxx").folder("Sent Items")

Regards
Richard

tueftler17

@Richard:
Perfect, as usual the solution was easy but in Germany we say: “I had tomatoes on my eyes”, so I did not find it.

Many thanks to you, now almost all messages are back again. I still see some errors when I do a kopano-backup but I will try to ignore them. Hopefully they won’t do a message-loop with active-sync.

Again: It was very helpful, I wish you all a great 2018!

kind regards,
Christian