kopano-search: Unable to create (new) sync-state


  • Hello community,
    I came over this while looking into the kopano-server[]: setSyncStatus(): collision:

    /var/log/kopano/search.log:

    2018-10-23 02:56:00,972 - search - WARNING - Sync state does not exist on server (anymore); requesting new one
    2018-10-23 02:56:00,979 - search - ERROR - Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/kopano-8.7.80-py2.6.egg/kopano/log.py", line 103, in log_exc
        try: yield
      File "/usr/lib/python2.6/site-packages/kopano_search-8.7.80-py2.6.egg/kopano_search/__init__.py", line 393, in incremental_sync
        new_state = self.server.sync(importer, self.state, log=self.log)
      File "/usr/lib/python2.6/site-packages/kopano-8.7.80-py2.6.egg/kopano/server.py", line 774, in sync
        return _ics.sync(self, self.mapistore, importer, state, max_changes, window=window, begin=begin, end=end, stats=stats)
      File "/usr/lib/python2.6/site-packages/kopano-8.7.80-py2.6.egg/kopano/ics.py", line 257, in sync
        syncid, changeid = struct.unpack('<II', _bdec(state))
    error: unpack requires a string argument of length 8
    

    As we can see, the search-daemon is trying to create a new sync-state in ics.py: sync(server, syncobj, importer, state...)

    In order to do so, syncid and changeid are decoded from the beginning of a binary state-object:

    syncid, changeid = struct.unpack('<II', _bdec(state)) 
    

    The unpack routine here exspects a stringobject of exact length sizeof(2 * int32_t), which regularly fails, if the stateobject-stream contains additional data.

    Thus, in my opinion, the string-decoded binary-stream needs truncation to 8 chars prior unpacking:

    syncid, changeid = struct.unpack('<II', _bdec(state)[:8]) 
    

    Proposed patch here.

    ++umgfoin.

  • Kopano

    Hi @umgfoin ,

    I quickly talked with a developer about this. The reason why you see this error is since the sync-state is now again back to its shorter size, but you index may still contain sync-states that are bigger than the expected length. The general solution here should be to drop you index and reindex your stores (all sync states should be then again the expected length).

    The next days he wants to look into python-kopano to give a general error handling of sync states are too long and then also enhance logging so that the admin can better identify such objects.


  • @umgfoin I had the same error, but adding [:8] didn’t fix the error. Any other ideas how to fix the size?