core-8.6.80.621: German Umlauts broken in the subject line of sent mails

noise

Hmmm but strange now i lost the umlauts again with the above…

Edit:

Now i try a few things also changing from
DE.UTF-8 to de.utf8

my settings are:

$ locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

$ locale -a
C
C.UTF-8
de_AT.utf8
de_BE.utf8
de_CH.utf8
de_DE.utf8
de_IT.utf8
de_LI.utf8
de_LU.utf8
POSIX

i try to use booth config (old and new) alone and booth togheter now are booth active:

$ cat /etc/default/kopano
KOPANO_LOCALE="de_DE.UTF-8"
KOPANO_USERSCRIPT_LOCALE="de_DE.UTF-8"

$ cat /etc/kopano/admin.cfg
# The language for folders in newly-created stores, specified as a
# locale identifier ("en_US", "de_DE", etc.)
default_store_locale = de_DE.UTF-8
#server_socket = default:
#sslkey_file = some.pem
#sslkey_pass = magic

but a email is watching like:
Header:

f?r die ?mis zum ?ss? > ?? ?? ?? ? ? ? ?
User <user@gmail.com>
Gesendet:Mittwoch 19 Dezember 2018 20:44
An:
User1 <user1@domain.com>

Email Text:

Test inside:
für die ömis zum ässä > Ää Öö Üü è é ç à

inside the email the umlauts are correct but not on header and folders

Note:
Always after a full reboot i can see that displaying is correct but later not

30 sec to 1 min later

Strange…
all is up to date newest version ubuntu 18.04

Edit 2:
if set in
/etc/default/locale
to
LANG=de_DE.UTF-8
after reboot the displaying is correct fro 30-60 seconds…

if set to:
de_DE.UTF8
after reboot the displaying in incorrect from first time reloading…
and also with a emtpy file…

fbartels

Ah, I missed your edits until now. The “issue” is not with the charsets that you set in any of the kopano configuration files, and also not with the locale you define when creating the store. The thing you need to look into is the charset that php uses.

noise

ok that means i need to look in nginx config or php ?
the header of nginx say it is utf-8

but one thing is strange here or normal i don’t know:

with:

wget --server-response -O /dev/null https://www.domain.com/webapp

the output is:
Content-Type: text/html; charset=utf-8

but with:

curl -I https://domain.com/webapp

the output is:
Content-Type: text/html

what is correct ? a check with a online tool means also it is utf-8

phpini from fpm is:
default_charset = "UTF-8"

fbartels

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

ok that means i need to look in nginx config or php ?

yes, php

noise

hmmm strange i can’t find anything… charset is utf-8 in php

the complete php output is too big to post here…
so i create a pdf link:
https://de.scribd.com/document/396095828/phpinfo

tjoen

If it is solved in Kopano it doen’t mean it is solved at the recepient. See
https://en.wikipedia.org/wiki/International_email#UTF-8_headers

noise

its not definitely postfix because other mail servers i don’t have this.
since it sometimes synonymous with the reload occurs, I think, it’s already php but not found yet.

here is also my header!

noise

i look again for my problem but not really found, i check also php and remove some modules to testing but always the same
the only one there is no charset is here:

$ curl -I https://www.domain.example/webapp
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Mon, 07 Jan 2019 10:27:11 GMT
Content-Type: text/html
Content-Length: 178
Location: https://www.domain.example/webapp/
Connection: keep-alive
Expires: Wed, 06 Feb 2019 10:27:11 GMT
Cache-Control: max-age=2592000
Cache-Control: public, must-revalidate, proxy-revalidate
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

the adress: https://www.domain.example/webapp
is the only where has no charset, but it’s a redirect to
https://www.domain.example/webapp/
and here we have the charset…

$ curl -I https://www.domain.example/webapp/
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 07 Jan 2019 10:27:16 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: __Secure-KOPANO_WEBAPP=g9cv0gsd2dh1tlc5ofo40fcpe3; path=/; secure; HttpOnly
Expires: Wed, 06 Feb 2019 10:27:16 GMT
Cache-Control: max-age=2592000
Pragma: no-cache
Set-Cookie: __Secure-encryption-store-key=51f94dc3cfbafd03948338ced240a0b9; path=/; secure; HttpOnly
Cache-Control: public, must-revalidate, proxy-revalidate
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

i don’t know if that is my problem ?
any hint’s are welcome

zara-kopa

@ noise
I am running under Debian 9.6 with PHP7.2 from the external “sury” Repo. I got the same problems. With a fist login in WebApp all looks fine for the german ‘umlaute’ in foldernames. German “Entwürfe” looks like “Entwürfe”. With the second login of the same user “Entwürfe” goes to “Entw?rfe”. — I get the same results as you with the different curl-Querys.
In Mails, Mail-Headers, etc. an in all other forms of using KOPANO (imap, outlook -mapi, outlook -z-push) the foldernames and all other use of “öü” is displayed right!
I use another Kopano-Installation with Debian 8 and php 5. Testing this install, i got the same results with your curl querys. Therefore the problem may be in an other context.
Because of the first login, when everything looks allright i will search whats happens up to the second login.

any hint’s are welcome

jengelh

If folder names change over time, sounds like data is transferred into the DB in a lossy fashion (and the only reason it shows ü in the first round is because the in-memory cache keeps it in its original form… until such a time it gets evicted).

zara-kopa

Hi everyone

i solved my problem! It was made by my own. All Ü’s and Ö’s are stable back again.
Shamefully the reason was a miss-configuration of the Apache server! No php Problem.
The error results out of a not enabled “javascript-common.conf” and some enabled mods with conflicts.

Härzlichen Dönk !!!

chrizzibaer

Hey zara-kopa,

Could you provide details where you solved the problem in your apache config?
I got the same error on my fresh install, and I just can’t figure out why.

Thanks a lot and greetings from Nuremberg ;)
Christoph.

zara-kopa

@chrizzibaer

Here are the enabled CONFS and MODS of my apache2

1_1547981822034_Bildschirmfoto 2019-01-20 um 11.54.36.jpg 0_1547981822033_Bildschirmfoto 2019-01-20 um 11.55.00.jpg

With these -enabled it’s worked for me. The use of php7.2-fpm is still in progress!
nice weekend from the Ruhrpott

noise

@zara-kopa i’m running at nginx and also in fpm and still have the problem…
it is possible that comes from mysql (mariabd)
The Kollation of kopano innodb is: utf8mb4_general_ci ?

noise

Hi guys

i come back to my old problem because i think its solved with the latest update (i not update since may 2019) and i leave the umlauts problems…

so i update all including system to 18.04.3 LTS
(over 200updates) and kopano to 8.7.82.92

after this update spooler & dagent stops working with transport, so i deactivate the plugIns. now the umlauts problem are gone with nginx.

Perhaps it was the option “plugin_enabled = yes” in dagent or spooler. i don’t know but it works now.

fbartels

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

Perhaps it was the option “plugin_enabled = yes” in dagent or spooler. i don’t know but it works now.

I think the main factor was probably https://forum.kopano.io/topic/2536/webapp-3-5-7-final-available

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

after this update spooler & dagent stops working with transport, so i deactivate the plugIns

Can you give some more details as to the “stops working with transport”?

noise

@fbartels said in Can you give some more details as to the “stops working with transport”?

i delete my last logs about this but i make some notices for myself…

the services are all running after the update, but send and receive a mail was not possible.

here are my workaround of this, but here are the wrong topic about this ;-)

First logs:

==> /var/log/mail.log <==
Sep  1 16:00:46 smtp kopano-dagent[16540]: Python traceback is on stderr (possibly check journalctl instead of logfile)
Sep  1 16:00:46 smtp kopano-dagent[15161]: Python threw an exception:
Sep  1 16:00:46 smtp kopano-dagent[16540]: K-1731: Unable to initialize the dagent plugin manager: Unknown error code (1).
Sep  1 16:00:46 smtp kopano-dagent[16540]: < 503 5.1.1 Internal error during delivery
Sep  1 16:00:46 smtp kopano-dagent[16540]: > QUIT
Sep  1 16:00:46 smtp kopano-dagent[16540]: < 221 2.0.0 Bye
Sep  1 16:00:46 smtp kopano-dagent[16540]: LMTP thread exiting

so i found something about the permissions in
/var/run/kopano/server.sock
/var/run/kopano/prio.sock

But the rights are corectly, i change only

ls -la /var/run/kopano/gateway.pid
-rw-r--r--  1 root   kopano    5 Sep  1 17:17 gateway.pid
to
chown kopano:kopano /var/run/kopano/gateway.pid
-rw-r--r-- 1 kopano kopano 5 Sep  1 17:17 /var/run/kopano/gateway.pid

2nd logs are:

Sun Sep  1 18:11:31 2019: [error  ] Use of Python (plugin_enabled=yes) forces process_model=fork
Sep  1 18:11:24 smtp slapd[1538]: <= bdb_substring_candidates: (cn) not indexed
Sep  1 18:11:31 smtp kopano-spooler[803]: Python threw an exception:

==> /var/log/kopano/spooler.log <==
Sun Sep  1 18:11:31 2019: [crit   ] [ 5966] K-1733: Unable to initialize the spooler plugin system: Unknown error code (1).

so i disable plugins to “no” and uncomment directories in spooler & dagent and it works.

umgfoin

Good evening,
root cause here:
https://github.com/Kopano-dev/kopano-core/pull/18

++umgfoin.

noise

Hi all

i come back again here… 🙈
i had a lots problem with mapi and search that i now solved, then i upgrade to the latest version. Kaldav i have not installed allreday.

But my umlauts problem has reversed now, webapp and imap (gateway) seems to be ok, but now z-push makes not a right conversion.
i have also updates z-push

Z-Push:   2.5.1+0-0 
PHP-MAPI: 8.7.82

The umlauts fail is only at the subject, inside the message there is no problem.

Edit:

i delete the device from z-push and do a completly resync,
by a iPhone and android 9 this not resolve the problem on the iPhnoe, but on the android already.
so i delete the account from iphone and the delete a second time the devices from z push.
Now it seems to be ok, but not shure how long :-)

noise

Report:
the umlauts problem still persists with z-push and also in contacts.
at the webapp is solved, there i don’t saw anymore.