kopano-search: Unable to create (new) sync-state

umgfoin

Hello community,
I came over this while looking into the kopano-server[]: setSyncStatus(): collision:

/var/log/kopano/search.log:

2018-10-23 02:56:00,972 - search - WARNING - Sync state does not exist on server (anymore); requesting new one
2018-10-23 02:56:00,979 - search - ERROR - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/kopano-8.7.80-py2.6.egg/kopano/log.py", line 103, in log_exc
    try: yield
  File "/usr/lib/python2.6/site-packages/kopano_search-8.7.80-py2.6.egg/kopano_search/__init__.py", line 393, in incremental_sync
    new_state = self.server.sync(importer, self.state, log=self.log)
  File "/usr/lib/python2.6/site-packages/kopano-8.7.80-py2.6.egg/kopano/server.py", line 774, in sync
    return _ics.sync(self, self.mapistore, importer, state, max_changes, window=window, begin=begin, end=end, stats=stats)
  File "/usr/lib/python2.6/site-packages/kopano-8.7.80-py2.6.egg/kopano/ics.py", line 257, in sync
    syncid, changeid = struct.unpack('<II', _bdec(state))
error: unpack requires a string argument of length 8

As we can see, the search-daemon is trying to create a new sync-state in ics.py: sync(server, syncobj, importer, state...)

In order to do so, syncid and changeid are decoded from the beginning of a binary state-object:

syncid, changeid = struct.unpack('<II', _bdec(state))

The unpack routine here exspects a stringobject of exact length sizeof(2 * int32_t), which regularly fails, if the stateobject-stream contains additional data.

Thus, in my opinion, the string-decoded binary-stream needs truncation to 8 chars prior unpacking:

syncid, changeid = struct.unpack('<II', _bdec(state)[:8])

Proposed patch here.

++umgfoin.

fbartels

Hi @umgfoin ,

I quickly talked with a developer about this. The reason why you see this error is since the sync-state is now again back to its shorter size, but you index may still contain sync-states that are bigger than the expected length. The general solution here should be to drop you index and reindex your stores (all sync states should be then again the expected length).

The next days he wants to look into python-kopano to give a general error handling of sync states are too long and then also enhance logging so that the admin can better identify such objects.

bmaehr

@umgfoin I had the same error, but adding [:8] didn’t fix the error. Any other ideas how to fix the size?

bmaehr

After a long time I was able to fix it with a simple command:

rm -r /var/lib/kopano/search