Kopano-search issue

Hi
Have a system that logs the kopano-search with

  • index0 - ERROR - Traceback (most recent call last):
    File “/usr/lib/python3/dist-packages/kopano/log.py”, line 103, in log_exc
    try: yield
    File “/usr/lib/python3/dist-packages/kopano_search/init.py”, line 274, in update
    attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
    File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 107, in get
    return convert(CMD[key], data or f.read(), log)
    File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 70, in convert
    log.warning(‘output on stderr:\n’+err[:1024]+’…’)
    TypeError: Can’t convert ‘bytes’ object to str implicitly

  • index0 - ERROR - Traceback (most recent call last):
    File “/usr/lib/python3/dist-packages/kopano/log.py”, line 103, in log_exc
    try: yield
    File “/usr/lib/python3/dist-packages/kopano_search/init.py”, line 274, in update
    attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
    File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 107, in get
    return convert(CMD[key], data or f.read(), log)
    File “/usr/lib/python3/dist-packages/kopano/attachment.py”, line 162, in read
    return self.data
    File “/usr/lib/python3/dist-packages/kopano/attachment.py”, line 157, in data
    self._data = _utils.stream(self.mapiobj, PR_ATTACH_DATA_BIN)
    File “/usr/lib/python3/dist-packages/kopano/utils.py”, line 52, in stream
    stream = mapiobj.OpenProperty(proptag, IID_IStream, 0, 0)
    File “/usr/lib/python3/dist-packages/MAPICore.py”, line 404, in OpenProperty
    return _MAPICore.IMAPIProp_OpenProperty(self, ulPropTag, USE_IID_FOR_OUTPUT, ulInterfaceOptions, ulFlags)
    MAPI.Struct.MAPIErrorNotFound: MAPI error 8004010F (MAPI_E_NOT_FOUND)

Suspect we are missing some support files but so far have not been able to find it

System
debian 9.5
kopano 8.6.81.171-0+25.1

On a twin system do not have the same issue.

Hej.

Do you use AD or LDAP?
I’ve seen this “error” when kopano cannot find the AD/LDAP ID’s match Kopano’s User-ID’s

Hallo,
I think I have the same problem here with the current version of kopano-search. I get the following messages (multiple times) in the logfile.

Sep 05 20:13:39 mailserver systemd[1]: Started Kopano Groupware Core Search Daemon.
Sep 05 20:13:40 mailserver kopano-search[22061]: 2018-09-05 20:13:40,954 - index0 - ERROR - Traceback (most recent call last):
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano/log.py", line 103, in log_exc
Sep 05 20:13:40 mailserver kopano-search[22061]:     try: yield
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/__init__.py", line 274, in update
Sep 05 20:13:40 mailserver kopano-search[22061]:     attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 107, in get
Sep 05 20:13:40 mailserver kopano-search[22061]:     return convert(CMD[key], data or f.read(), log)
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 70, in convert
Sep 05 20:13:40 mailserver kopano-search[22061]:     log.warning('output on stderr:\n'+err[:1024]+'..')
Sep 05 20:13:40 mailserver kopano-search[22061]: TypeError: Can't convert 'bytes' object to str implicitly

I have no idea what to do. Maybe it’s a bug?

Additional info: I get (every 5 seconds) a log message from kopano-server.

Sep 06 05:45:57 kopano kopano-server[5089]: setSyncStatus(): collision

I have these issues on two completly fresh installations (Debian 9, x64) Kopano Version 8.6.81.228-0+37.1
Nevertheless WebApp and Z-Push seems to work well.

Any ideas?

Just an update.
We migrated the install from Jessie to Stretch and got the same errors on 2 separate systems,
Could it be corruptions in the attachment files?
Both are using DB with external file attachment

kopano-archiver/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-backup/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-bash-completion/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-client/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-client-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-common/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-contacts/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dagent/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dagent-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dbg/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dev/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-gateway/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-gateway-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-ical/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-ical-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-lang/now 8.6.81.171-0+25.1 all [installed,local]
kopano-migration-imap/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-migration-pst/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-monitor/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-monitor-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-presence/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-python-utils/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-search/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-server/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-server-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-server-packages/now 8.6.81.171-0+25.1 all [installed,local]
kopano-spamd/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-spooler/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-spooler-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-utils/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-utils-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-webapp/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-contactfax/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-desktopnotifications/now 2.0.1.5+16.2 all [installed,local]
kopano-webapp-plugin-filepreviewer/now 2.0.0.15+10.8 all [installed,local]
kopano-webapp-plugin-files/now 2.1.5.295+194.1 all [installed,local]
kopano-webapp-plugin-filesbackend-owncloud/now 2.1.1.84+61.1 all [installed,local]
kopano-webapp-plugin-filesbackend-smb/now 2.1.1.48+49.1 all [installed,local]
kopano-webapp-plugin-folderwidgets/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-gmaps/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-intranet/now 0.1+32.8 all [installed,local]
kopano-webapp-plugin-mattermost/now 0.1+17.2 all [installed,local]
kopano-webapp-plugin-mdm/now 2.1.0.106+26.1 all [installed,local]
kopano-webapp-plugin-meetings/now 3.0.4.15 all [installed,local]
kopano-webapp-plugin-pimfolder/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-quickitems/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-smime/now 2.3.0.198+126.1 all [installed,local]
kopano-webapp-plugin-spell/now 2.0.0.17+34.1 all [installed,local]
kopano-webapp-plugin-spell-de-at/now 2.0.0.4+28.1 all [installed,local]
kopano-webapp-plugin-spell-de-ch/now 2.0.0.5+27.1 all [installed,local]
kopano-webapp-plugin-spell-de-de/now 2.0.0.3+28.1 all [installed,local]
kopano-webapp-plugin-spell-en/now 2.0.0.1+28.1 all [installed,local]
kopano-webapp-plugin-spell-en-gb/now 2.0.0.1+27.1 all [installed,local]
kopano-webapp-plugin-spell-es/now 2.0.0.1+27.1 all [installed,local]
kopano-webapp-plugin-spell-fr/now 2.0.0.1+28.1 all [installed,local]
kopano-webapp-plugin-spell-it/now 1.0.0+40.1 all [installed,local]
kopano-webapp-plugin-spell-nl/now 2.0.0.1+29.1 all [installed,local]
kopano-webapp-plugin-spell-pl-pl/now 2.0.0.0+33.1 all [installed,local]
kopano-webapp-plugin-titlecounter/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-webappmanual/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-zdeveloper/now 3.4.21.1704+968.1 all [installed,local]
kopano-webmeetings/now 0.29.5-1 amd64 [installed,local]
libgsoap-kopano-2.8.62/now 2.8.62-0+2.1 amd64 [installed,local]
libvmime-kopano1/now 0.9.2.50+11.1 amd64 [installed,local]
libvmime-kopano1-dbgsym/now 0.9.2.50+11.1 amd64 [installed,local]
php-kopano-smime/now 1.0.00+14.2 amd64 [installed,local]
php-kopano-smime-dbgsym/now 1.0.00+14.2 amd64 [installed,local]
python-kopano/now 8.6.81.171-0+25.1 all [installed,local]
python3-kopano/now 8.6.81.171-0+25.1 all [installed,local]
z-push-backend-kopano/stable,now 2.4.4+0-0 all [installed,automatic]
z-push-kopano/stable,now 2.4.4+0-0 all [installed]
z-push-kopano-gab2contacts/stable,now 2.4.4+0-0 all [installed]

We have run the script by bkram db-find-missing-attachments.pl which listed missing attachments according to the db.
Now we modified the script to create a file of the same name

                print "Missing attachment file: $filename\n";  <-- existing line 84
                open(my $fh, '>', $filename) or die "Could not open file '$filename' $1";
                print $fh "created attachment file: '$filename'\n";
                close $fh 

The result is no errors showing in the search.log or journalctl -xe.
Would missing attachments cause the “Can’t convert ‘bytes’ object to str implicitly”?

Still get errors if we delete all the search index files and restart kopano-search. So it would seem to be triggered by attachments in some older format???

File "/usr/lib/python3/dist-packages/kopano/log.py", line 103, in log_exc
  try: yield
File "/usr/lib/python3/dist-packages/kopano_search/__init__.py", line 274, in update
  attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 107, in get
  return convert(CMD[key], data or f.read(), log)
File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 70, in convert
  log.warning('output on stderr:\n'+err[:1024]+'..')
TypeError: Can't convert 'bytes' object to str implicitly

Does anyone have any suggestions?

try the following :
chown -R /var/lib/kopano

check the startup of kopano-search.
I have this

# /lib/systemd/system/kopano-search.service
[Unit]
Description=Kopano Groupware Core Search Daemon
Documentation=man:kopano-search(8) man:kopano-search.cfg(5)

[Service]
Type=simple
Environment=LC_CTYPE=en_US.UTF-8
EnvironmentFile=-/etc/sysconfig/kopano
EnvironmentFile=-/etc/default/kopano
ExecStart=/usr/sbin/kopano-search -F
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/kopano-search.service.d/override.conf
[Unit]
After=kopano-server.service
Wants=kopano-server.service

for the webapp and MDM
read :
https://forum.kopano.io/topic/1456/solved-mdm-plugin-webapp-ubuntu-18-04/14

the /lib/systemd/system/kopano-search.service is the same but the 2nd one /etc/systemd/system/ has only the following

getty.target.wants/ sshd.service
multi-user.target.wants/ sysinit.target.wants/
mysqld.service syslog.service
mysql.service timers.target.wants/
network-online.target.wants/

No Kopano components. What should be there?

About the systemd settings.

The second, if you type :
systemctl edit kopano-search then you wil edit with an “override.conf” as shown.
if you use
systemctl edit --full kopano-service
Then a copy is done of the Original file and placed in /etc/systemd/system

I’ve set the “wanted and after” just because it changes the start orde so kopano-server is started and search after that.

my kopano-server contains this :

# /etc/systemd/system/kopano-server.service.d/override.conf
[Unit]
After=network-online.target mysql.service mariadb.service
Wants=network-online.target

kopano server starts only after mysql is started and when my network is online.
Super to tune your startup, and it helps in reducing unneeded logs entries.

In you case, what i would try.
backup /var/lib/kopano /etc/kopano /etc/apache /etc/php*
remove kopano , so make sure you have good backups of everything of kopano.

use : apt-get remove --purge --autoremove kopano-*
or use apt-get remove --autoremove kopano-* this keeps some files, including the configs and data

You get a big list of files that are about to removed.
check again if its all removed with : dpkg -l | grep kopano
copy this to a tekst editor.

after the remove, run updatedb ( if that does not work, install mlocate : apt-get install mlocate)
run updatedb
now type : locate kopano

Check for files in /var/lib and /usr/share, There should not be any files.
try to locate where these “left overs” did come from with : apt-file find /path/to/file
and if so remove them if needed.

now install kopano again.

apt-get install kopano-server-packages

Then use the tekst file and check if you missed of for install.

These steps helped me with some left overs files, which where giving simular problems.