Kopano-search issue

compsos

Hi
Have a system that logs the kopano-search with

index0 - ERROR - Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/kopano/log.py”, line 103, in log_exc
try: yield
File “/usr/lib/python3/dist-packages/kopano_search/init.py”, line 274, in update
attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 107, in get
return convert(CMD[key], data or f.read(), log)
File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 70, in convert
log.warning(‘output on stderr:\n’+err[:1024]+’…’)
TypeError: Can’t convert ‘bytes’ object to str implicitly
index0 - ERROR - Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/kopano/log.py”, line 103, in log_exc
try: yield
File “/usr/lib/python3/dist-packages/kopano_search/init.py”, line 274, in update
attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
File “/usr/lib/python3/dist-packages/kopano_search/plaintext.py”, line 107, in get
return convert(CMD[key], data or f.read(), log)
File “/usr/lib/python3/dist-packages/kopano/attachment.py”, line 162, in read
return self.data
File “/usr/lib/python3/dist-packages/kopano/attachment.py”, line 157, in data
self._data = _utils.stream(self.mapiobj, PR_ATTACH_DATA_BIN)
File “/usr/lib/python3/dist-packages/kopano/utils.py”, line 52, in stream
stream = mapiobj.OpenProperty(proptag, IID_IStream, 0, 0)
File “/usr/lib/python3/dist-packages/MAPICore.py”, line 404, in OpenProperty
return _MAPICore.IMAPIProp_OpenProperty(self, ulPropTag, USE_IID_FOR_OUTPUT, ulInterfaceOptions, ulFlags)
MAPI.Struct.MAPIErrorNotFound: MAPI error 8004010F (MAPI_E_NOT_FOUND)

Suspect we are missing some support files but so far have not been able to find it

System
debian 9.5
kopano 8.6.81.171-0+25.1

On a twin system do not have the same issue.

marcus.bauch

Hej.

Do you use AD or LDAP?
I’ve seen this “error” when kopano cannot find the AD/LDAP ID’s match Kopano’s User-ID’s

robgnu

Hallo,
I think I have the same problem here with the current version of kopano-search. I get the following messages (multiple times) in the logfile.

Sep 05 20:13:39 mailserver systemd[1]: Started Kopano Groupware Core Search Daemon.
Sep 05 20:13:40 mailserver kopano-search[22061]: 2018-09-05 20:13:40,954 - index0 - ERROR - Traceback (most recent call last):
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano/log.py", line 103, in log_exc
Sep 05 20:13:40 mailserver kopano-search[22061]:     try: yield
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/__init__.py", line 274, in update
Sep 05 20:13:40 mailserver kopano-search[22061]:     attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 107, in get
Sep 05 20:13:40 mailserver kopano-search[22061]:     return convert(CMD[key], data or f.read(), log)
Sep 05 20:13:40 mailserver kopano-search[22061]:   File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 70, in convert
Sep 05 20:13:40 mailserver kopano-search[22061]:     log.warning('output on stderr:\n'+err[:1024]+'..')
Sep 05 20:13:40 mailserver kopano-search[22061]: TypeError: Can't convert 'bytes' object to str implicitly

I have no idea what to do. Maybe it’s a bug?

Additional info: I get (every 5 seconds) a log message from kopano-server.

Sep 06 05:45:57 kopano kopano-server[5089]: setSyncStatus(): collision

I have these issues on two completly fresh installations (Debian 9, x64) Kopano Version 8.6.81.228-0+37.1
Nevertheless WebApp and Z-Push seems to work well.

Any ideas?

compsos

Just an update.
We migrated the install from Jessie to Stretch and got the same errors on 2 separate systems,
Could it be corruptions in the attachment files?
Both are using DB with external file attachment

kopano-archiver/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-backup/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-bash-completion/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-client/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-client-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-common/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-contacts/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dagent/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dagent-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dbg/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-dev/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-gateway/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-gateway-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-ical/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-ical-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-lang/now 8.6.81.171-0+25.1 all [installed,local]
kopano-migration-imap/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-migration-pst/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-monitor/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-monitor-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-presence/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-python-utils/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-search/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-server/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-server-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-server-packages/now 8.6.81.171-0+25.1 all [installed,local]
kopano-spamd/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-spooler/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-spooler-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-utils/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-utils-dbgsym/now 8.6.81.171-0+25.1 amd64 [installed,local]
kopano-webapp/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-contactfax/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-desktopnotifications/now 2.0.1.5+16.2 all [installed,local]
kopano-webapp-plugin-filepreviewer/now 2.0.0.15+10.8 all [installed,local]
kopano-webapp-plugin-files/now 2.1.5.295+194.1 all [installed,local]
kopano-webapp-plugin-filesbackend-owncloud/now 2.1.1.84+61.1 all [installed,local]
kopano-webapp-plugin-filesbackend-smb/now 2.1.1.48+49.1 all [installed,local]
kopano-webapp-plugin-folderwidgets/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-gmaps/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-intranet/now 0.1+32.8 all [installed,local]
kopano-webapp-plugin-mattermost/now 0.1+17.2 all [installed,local]
kopano-webapp-plugin-mdm/now 2.1.0.106+26.1 all [installed,local]
kopano-webapp-plugin-meetings/now 3.0.4.15 all [installed,local]
kopano-webapp-plugin-pimfolder/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-quickitems/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-smime/now 2.3.0.198+126.1 all [installed,local]
kopano-webapp-plugin-spell/now 2.0.0.17+34.1 all [installed,local]
kopano-webapp-plugin-spell-de-at/now 2.0.0.4+28.1 all [installed,local]
kopano-webapp-plugin-spell-de-ch/now 2.0.0.5+27.1 all [installed,local]
kopano-webapp-plugin-spell-de-de/now 2.0.0.3+28.1 all [installed,local]
kopano-webapp-plugin-spell-en/now 2.0.0.1+28.1 all [installed,local]
kopano-webapp-plugin-spell-en-gb/now 2.0.0.1+27.1 all [installed,local]
kopano-webapp-plugin-spell-es/now 2.0.0.1+27.1 all [installed,local]
kopano-webapp-plugin-spell-fr/now 2.0.0.1+28.1 all [installed,local]
kopano-webapp-plugin-spell-it/now 1.0.0+40.1 all [installed,local]
kopano-webapp-plugin-spell-nl/now 2.0.0.1+29.1 all [installed,local]
kopano-webapp-plugin-spell-pl-pl/now 2.0.0.0+33.1 all [installed,local]
kopano-webapp-plugin-titlecounter/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-webappmanual/now 3.4.21.1704+968.1 all [installed,local]
kopano-webapp-plugin-zdeveloper/now 3.4.21.1704+968.1 all [installed,local]
kopano-webmeetings/now 0.29.5-1 amd64 [installed,local]
libgsoap-kopano-2.8.62/now 2.8.62-0+2.1 amd64 [installed,local]
libvmime-kopano1/now 0.9.2.50+11.1 amd64 [installed,local]
libvmime-kopano1-dbgsym/now 0.9.2.50+11.1 amd64 [installed,local]
php-kopano-smime/now 1.0.00+14.2 amd64 [installed,local]
php-kopano-smime-dbgsym/now 1.0.00+14.2 amd64 [installed,local]
python-kopano/now 8.6.81.171-0+25.1 all [installed,local]
python3-kopano/now 8.6.81.171-0+25.1 all [installed,local]
z-push-backend-kopano/stable,now 2.4.4+0-0 all [installed,automatic]
z-push-kopano/stable,now 2.4.4+0-0 all [installed]
z-push-kopano-gab2contacts/stable,now 2.4.4+0-0 all [installed]

compsos

We have run the script by bkram db-find-missing-attachments.pl which listed missing attachments according to the db.
Now we modified the script to create a file of the same name

                print "Missing attachment file: $filename\n";  <-- existing line 84
                open(my $fh, '>', $filename) or die "Could not open file '$filename' $1";
                print $fh "created attachment file: '$filename'\n";
                close $fh

The result is no errors showing in the search.log or journalctl -xe.
Would missing attachments cause the “Can’t convert ‘bytes’ object to str implicitly”?

compsos

Still get errors if we delete all the search index files and restart kopano-search. So it would seem to be triggered by attachments in some older format???

File "/usr/lib/python3/dist-packages/kopano/log.py", line 103, in log_exc
  try: yield
File "/usr/lib/python3/dist-packages/kopano_search/__init__.py", line 274, in update
  attach_text.append(plaintext.get(attachment, mimetype=attachment.mimetype, log=self.log))
File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 107, in get
  return convert(CMD[key], data or f.read(), log)
File "/usr/lib/python3/dist-packages/kopano_search/plaintext.py", line 70, in convert
  log.warning('output on stderr:\n'+err[:1024]+'..')
TypeError: Can't convert 'bytes' object to str implicitly

Does anyone have any suggestions?

thctlo

try the following :
chown -R /var/lib/kopano

check the startup of kopano-search.
I have this

# /lib/systemd/system/kopano-search.service
[Unit]
Description=Kopano Groupware Core Search Daemon
Documentation=man:kopano-search(8) man:kopano-search.cfg(5)

[Service]
Type=simple
Environment=LC_CTYPE=en_US.UTF-8
EnvironmentFile=-/etc/sysconfig/kopano
EnvironmentFile=-/etc/default/kopano
ExecStart=/usr/sbin/kopano-search -F
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/kopano-search.service.d/override.conf
[Unit]
After=kopano-server.service
Wants=kopano-server.service

for the webapp and MDM
read :
https://forum.kopano.io/topic/1456/solved-mdm-plugin-webapp-ubuntu-18-04/14

compsos

the /lib/systemd/system/kopano-search.service is the same but the 2nd one /etc/systemd/system/ has only the following

getty.target.wants/ sshd.service
multi-user.target.wants/ sysinit.target.wants/
mysqld.service syslog.service
mysql.service timers.target.wants/
network-online.target.wants/

No Kopano components. What should be there?

thctlo

About the systemd settings.

The second, if you type :
systemctl edit kopano-search then you wil edit with an “override.conf” as shown.
if you use
systemctl edit --full kopano-service
Then a copy is done of the Original file and placed in /etc/systemd/system

I’ve set the “wanted and after” just because it changes the start orde so kopano-server is started and search after that.

my kopano-server contains this :

# /etc/systemd/system/kopano-server.service.d/override.conf
[Unit]
After=network-online.target mysql.service mariadb.service
Wants=network-online.target

kopano server starts only after mysql is started and when my network is online.
Super to tune your startup, and it helps in reducing unneeded logs entries.

In you case, what i would try.
backup /var/lib/kopano /etc/kopano /etc/apache /etc/php*
remove kopano , so make sure you have good backups of everything of kopano.

use : apt-get remove --purge --autoremove kopano-*
or use apt-get remove --autoremove kopano-* this keeps some files, including the configs and data

You get a big list of files that are about to removed.
check again if its all removed with : dpkg -l | grep kopano
copy this to a tekst editor.

after the remove, run updatedb ( if that does not work, install mlocate : apt-get install mlocate)
run updatedb
now type : locate kopano

Check for files in /var/lib and /usr/share, There should not be any files.
try to locate where these “left overs” did come from with : apt-file find /path/to/file
and if so remove them if needed.

now install kopano again.

apt-get install kopano-server-packages

Then use the tekst file and check if you missed of for install.

These steps helped me with some left overs files, which where giving simular problems.

bennetcole

@compsos said in Kopano-search issue:

TypeError: Can’t convert ‘bytes’ object to str implicitly

Python makes a clear distinction between bytes and strings . Bytes objects contain raw data — a sequence of octets — whereas strings are Unicode sequences . Conversion between these two types is explicit: you encode a string to get bytes, specifying an encoding (which defaults to UTF-8); and you decode bytes to get a string. Clients of these functions should be aware that such conversions may fail, and should consider how failures are handled.

We can convert bytes to string using bytes class decode() instance method, So you need to decode the bytes object to produce a string. In Python 3 , the default encoding is “utf-8” , so you can use directly:

b"python byte to string".decode("utf-8")