Kopano-search issue



  • Hi
    Have a system that logs the kopano-search with

    • index0 - ERROR - Traceback (most recent call last):
      File “/usr/lib/python3/dist-packages/kopano/log.py”, line 103, in log_exc
      try: yield
      File “/usr/lib/python3/dist-packages/kopano_search/init.py”, line 274, in update
      attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
      File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 107, in get
      return convert(CMD[key], data or f.read(), log)
      File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 70, in convert
      log.warning(‘output on stderr:\n’+err[:1024]+’…’)
      TypeError: Can’t convert ‘bytes’ object to str implicitly

    • index0 - ERROR - Traceback (most recent call last):
      File “/usr/lib/python3/dist-packages/kopano/log.py”, line 103, in log_exc
      try: yield
      File “/usr/lib/python3/dist-packages/kopano_search/init.py”, line 274, in update
      attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
      File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 107, in get
      return convert(CMD[key], data or f.read(), log)
      File “/usr/lib/python3/dist-packages/kopano/attachment.py”, line 162, in read
      return self.data
      File “/usr/lib/python3/dist-packages/kopano/attachment.py”, line 157, in data
      self._data = _utils.stream(self.mapiobj, PR_ATTACH_DATA_BIN)
      File “/usr/lib/python3/dist-packages/kopano/utils.py”, line 52, in stream
      stream = mapiobj.OpenProperty(proptag, IID_IStream, 0, 0)
      File “/usr/lib/python3/dist-packages/MAPICore.py”, line 404, in OpenProperty
      return _MAPICore.IMAPIProp_OpenProperty(self, ulPropTag, USE_IID_FOR_OUTPUT, ulInterfaceOptions, ulFlags)
      MAPI.Struct.MAPIErrorNotFound: MAPI error 8004010F (MAPI_E_NOT_FOUND)

    Suspect we are missing some support files but so far have not been able to find it

    System
    debian 9.5
    kopano 8.6.81.171-0+25.1

    On a twin system do not have the same issue.



  • Hej.

    Do you use AD or LDAP?
    I’ve seen this “error” when kopano cannot find the AD/LDAP ID’s match Kopano’s User-ID’s



  • Hallo,
    I think I have the same problem here with the current version of kopano-search. I get the following messages (multiple times) in the logfile.

    Sep 05 20:13:39 mailserver systemd[1]: Started Kopano Groupware Core Search Daemon.
    Sep 05 20:13:40 mailserver kopano-search[22061]: 2018-09-05 20:13:40,954 - index0 - ERROR - Traceback (most recent call last):
    Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano/log.py", line 103, in log_exc
    Sep 05 20:13:40 mailserver kopano-search[22061]:     try: yield
    Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/__init__.py", line 274, in update
    Sep 05 20:13:40 mailserver kopano-search[22061]:     attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
    Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 107, in get
    Sep 05 20:13:40 mailserver kopano-search[22061]:     return convert(CMD[key], data or f.read(), log)
    Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 70, in convert
    Sep 05 20:13:40 mailserver kopano-search[22061]:     log.warning('output on stderr:\n'+err[:1024]+'..')
    Sep 05 20:13:40 mailserver kopano-search[22061]: TypeError: Can't convert 'bytes' object to str implicitly
    

    I have no idea what to do. Maybe it’s a bug?

    Additional info: I get (every 5 seconds) a log message from kopano-server.

    Sep 06 05:45:57 kopano kopano-server[5089]: setSyncStatus(): collision
    

    I have these issues on two completly fresh installations (Debian 9, x64) Kopano Version 8.6.81.228-0+37.1
    Nevertheless WebApp and Z-Push seems to work well.

    Any ideas?



  • Just an update.
    We migrated the install from Jessie to Stretch and got the same errors on 2 separate systems,
    Could it be corruptions in the attachment files?
    Both are using DB with external file attachment

    kopano-archiver/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-backup/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-bash-completion/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-client/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-client-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-common/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-contacts/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-dagent/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-dagent-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-dbg/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-dev/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-gateway/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-gateway-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-ical/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-ical-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-lang/now 8.6.81.171-0+25.1 all [installed,local]
    kopano-migration-imap/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-migration-pst/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-monitor/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-monitor-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-presence/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-python-utils/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-search/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-server/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-server-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-server-packages/now 8.6.81.171-0+25.1 all [installed,local]
    kopano-spamd/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-spooler/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-spooler-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-utils/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-utils-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
    kopano-webapp/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-contactfax/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-desktopnotifications/now 2.0.1.5+16.2 all [installed,local]
    kopano-webapp-plugin-filepreviewer/now 2.0.0.15+10.8 all [installed,local]
    kopano-webapp-plugin-files/now 2.1.5.295+194.1 all [installed,local]
    kopano-webapp-plugin-filesbackend-owncloud/now 2.1.1.84+61.1 all [installed,local]
    kopano-webapp-plugin-filesbackend-smb/now 2.1.1.48+49.1 all [installed,local]
    kopano-webapp-plugin-folderwidgets/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-gmaps/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-intranet/now 0.1+32.8 all [installed,local]
    kopano-webapp-plugin-mattermost/now 0.1+17.2 all [installed,local]
    kopano-webapp-plugin-mdm/now 2.1.0.106+26.1 all [installed,local]
    kopano-webapp-plugin-meetings/now 3.0.4.15 all [installed,local]
    kopano-webapp-plugin-pimfolder/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-quickitems/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-smime/now 2.3.0.198+126.1 all [installed,local]
    kopano-webapp-plugin-spell/now 2.0.0.17+34.1 all [installed,local]
    kopano-webapp-plugin-spell-de-at/now 2.0.0.4+28.1 all [installed,local]
    kopano-webapp-plugin-spell-de-ch/now 2.0.0.5+27.1 all [installed,local]
    kopano-webapp-plugin-spell-de-de/now 2.0.0.3+28.1 all [installed,local]
    kopano-webapp-plugin-spell-en/now 2.0.0.1+28.1 all [installed,local]
    kopano-webapp-plugin-spell-en-gb/now 2.0.0.1+27.1 all [installed,local]
    kopano-webapp-plugin-spell-es/now 2.0.0.1+27.1 all [installed,local]
    kopano-webapp-plugin-spell-fr/now 2.0.0.1+28.1 all [installed,local]
    kopano-webapp-plugin-spell-it/now 1.0.0+40.1 all [installed,local]
    kopano-webapp-plugin-spell-nl/now 2.0.0.1+29.1 all [installed,local]
    kopano-webapp-plugin-spell-pl-pl/now 2.0.0.0+33.1 all [installed,local]
    kopano-webapp-plugin-titlecounter/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-webappmanual/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webapp-plugin-zdeveloper/now 3.4.21.1704+968.1 all [installed,local]
    kopano-webmeetings/now 0.29.5-1 amd64 [installed,local]
    libgsoap-kopano-2.8.62/now 2.8.62-0+2.1 amd64 [installed,local]
    libvmime-kopano1/now 0.9.2.50+11.1 amd64 [installed,local]
    libvmime-kopano1-dbgsym/now 0.9.2.50+11.1 amd64 [installed,local]
    php-kopano-smime/now 1.0.00+14.2 amd64 [installed,local]
    php-kopano-smime-dbgsym/now 1.0.00+14.2 amd64 [installed,local]
    python-kopano/now 8.6.81.171-0+25.1 all [installed,local]
    python3-kopano/now 8.6.81.171-0+25.1 all [installed,local]
    z-push-backend-kopano/stable,now 2.4.4+0-0 all [installed,automatic]
    z-push-kopano/stable,now 2.4.4+0-0 all [installed]
    z-push-kopano-gab2contacts/stable,now 2.4.4+0-0 all [installed]



  • We have run the script by bkram db-find-missing-attachments.pl which listed missing attachments according to the db.
    Now we modified the script to create a file of the same name

                    print "Missing attachment file: $filename\n";  <-- existing line 84
                    open(my $fh, '>', $filename) or die "Could not open file '$filename' $1";
                    print $fh "created attachment file: '$filename'\n";
                    close $fh 
    

    The result is no errors showing in the search.log or journalctl -xe.
    Would missing attachments cause the “Can’t convert ‘bytes’ object to str implicitly”?



  • Still get errors if we delete all the search index files and restart kopano-search. So it would seem to be triggered by attachments in some older format???

    File "/usr/lib/python3/dist-packages/kopano/log.py", line 103, in log_exc
      try: yield
    File "/usr/lib/python3/dist-packages/kopano_search/__init__.py", line 274, in update
      attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
    File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 107, in get
      return convert(CMD[key], data or f.read(), log)
    File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 70, in convert
      log.warning('output on stderr:\n'+err[:1024]+'..')
    TypeError: Can't convert 'bytes' object to str implicitly
    

    Does anyone have any suggestions?



  • try the following :
    chown -R /var/lib/kopano

    check the startup of kopano-search.
    I have this

    # /lib/systemd/system/kopano-search.service
    [Unit]
    Description=Kopano Groupware Core Search Daemon
    Documentation=man:kopano-search(8) man:kopano-search.cfg(5)
    
    [Service]
    Type=simple
    Environment=LC_CTYPE=en_US.UTF-8
    EnvironmentFile=-/etc/sysconfig/kopano
    EnvironmentFile=-/etc/default/kopano
    ExecStart=/usr/sbin/kopano-search -F
    ExecReload=/bin/kill -HUP $MAINPID
    
    [Install]
    WantedBy=multi-user.target
    
    # /etc/systemd/system/kopano-search.service.d/override.conf
    [Unit]
    After=kopano-server.service
    Wants=kopano-server.service
    

    for the webapp and MDM
    read :
    https://forum.kopano.io/topic/1456/solved-mdm-plugin-webapp-ubuntu-18-04/14



  • the /lib/systemd/system/kopano-search.service is the same but the 2nd one /etc/systemd/system/ has only the following

    getty.target.wants/ sshd.service
    multi-user.target.wants/ sysinit.target.wants/
    mysqld.service syslog.service
    mysql.service timers.target.wants/
    network-online.target.wants/

    No Kopano components. What should be there?



  • About the systemd settings.

    The second, if you type :
    systemctl edit kopano-search then you wil edit with an “override.conf” as shown.
    if you use
    systemctl edit --full kopano-service
    Then a copy is done of the Original file and placed in /etc/systemd/system

    I’ve set the “wanted and after” just because it changes the start orde so kopano-server is started and search after that.

    my kopano-server contains this :

    # /etc/systemd/system/kopano-server.service.d/override.conf
    [Unit]
    After=network-online.target mysql.service mariadb.service
    Wants=network-online.target
    

    kopano server starts only after mysql is started and when my network is online.
    Super to tune your startup, and it helps in reducing unneeded logs entries.

    In you case, what i would try.
    backup /var/lib/kopano /etc/kopano /etc/apache /etc/php*
    remove kopano , so make sure you have good backups of everything of kopano.

    use : apt-get remove --purge --autoremove kopano-*
    or use apt-get remove --autoremove kopano-* this keeps some files, including the configs and data

    You get a big list of files that are about to removed.
    check again if its all removed with : dpkg -l | grep kopano
    copy this to a tekst editor.

    after the remove, run updatedb ( if that does not work, install mlocate : apt-get install mlocate)
    run updatedb
    now type : locate kopano

    Check for files in /var/lib and /usr/share, There should not be any files.
    try to locate where these “left overs” did come from with : apt-file find /path/to/file
    and if so remove them if needed.

    now install kopano again.

    apt-get install kopano-server-packages

    Then use the tekst file and check if you missed of for install.

    These steps helped me with some left overs files, which where giving simular problems.


Log in to reply