core-8.6.80.621: German Umlauts broken in the subject line of sent mails

thctlo

a very good point here.

dagent.cfg:#default_charset = us-ascii
ical.cfg:#default_charset = utf-8
ldap.cfg:#ldap_server_charset = utf-8
spooler.cfg:#charset_upgrade = windows-1252
unix.cfg:fullname_charset = iso-8859-15

uhhhh, why o why ?? please fix this all to UTF8

noise

ok i look at other files too…

edit: dagent.cfg charset is uncommented i leave it at this time but you have right why not all config files are to UTF-8 ?

thctlo

no, i say this is one that kopano should fix, its crazy to have so much differnent types.

thctlo

@noise

edit: dagent.cfg charset is uncommented i leave it at this time but you have right why not all config files are to UTF-8 ?

i dont know, wait for a kopano dev to reply this.

fbartels

@thctlo said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

this is one that kopano should fix

It’s all open source, you know. If you feel you need to change something: https://stash.kopano.io/projects/KC/repos/kopanocore/browse/CONTRIBUTING.md

I doubt that it is really necessary, since there are plenty of systems out there that work fine with umlauts in subjects without the need to fiddle with charset settings in the spooler.

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

Traceback (most recent call last):

I did not yet manage to look into this, but I would expect that this is a python2 vs python3 issue. in master we recently switched everything to python3 and some scripts may still require love.

noise

@fbartels said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

I would expect that this is a python2 vs python3 issue. in master we recently switched everything to python3 and some scripts may still require love.

yes i think also but i try to purge python2.7 but not helps. I’ll keep looking, if i found anything i post it.

thanks anyway

fbartels

no, the script has a shebang that points towards python3. the simplest workaround would be that change that to point towards python2. Still if you want to look further into this a pr for this is of course welcome as well.

btw. this does not necessarily need to be through git-patch, opening a pr in github also works for us -> https://github.com/Kopano-dev/kopano-core

noise

i have installed booth python 2.7 and 3.7.1 so i try to purge 2.7 and i see i now have also 3.6.7
i’m not much involved with python…

edit: i think i need to do symlinks between the versions ib usr/lib/

fbartels

yeah. no, this is 100% not what i meant. my GUESS is that the code in the script you are trying to run is not compatible with python3 (which is the interpreter he tries to execute it with because of the shebang). You will still need python2 on your system (and python2-kopano or however the file is called in your distribution). simply run the script with python2 instead of python3. if you don’t want to fiddle with the script, then a python2 /usr/sbin/kopano-localize-folders .. probably does the trick as well.

noise

ah ok my problem is python 3 not 2 ok i will try… thanks

Edit:

ok i solved with install

apt install python-kopano

with the command for all users:

python2 /usr/sbin/kopano-localize-folders --lang de_DE.UTF-8

the user are changed:

<user1>: Changing localized folder names to "de_DE.UTF-8"

thanks for the hint.

noise

Hmmm but strange now i lost the umlauts again with the above…

Edit:

Now i try a few things also changing from
DE.UTF-8 to de.utf8

my settings are:

$ locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

$ locale -a
C
C.UTF-8
de_AT.utf8
de_BE.utf8
de_CH.utf8
de_DE.utf8
de_IT.utf8
de_LI.utf8
de_LU.utf8
POSIX

i try to use booth config (old and new) alone and booth togheter now are booth active:

$ cat /etc/default/kopano
KOPANO_LOCALE="de_DE.UTF-8"
KOPANO_USERSCRIPT_LOCALE="de_DE.UTF-8"

$ cat /etc/kopano/admin.cfg
# The language for folders in newly-created stores, specified as a
# locale identifier ("en_US", "de_DE", etc.)
default_store_locale = de_DE.UTF-8
#server_socket = default:
#sslkey_file = some.pem
#sslkey_pass = magic

but a email is watching like:
Header:

f?r die ?mis zum ?ss? > ?? ?? ?? ? ? ? ?
User <user@gmail.com>
Gesendet:Mittwoch 19 Dezember 2018 20:44
An:
User1 <user1@domain.com>

Email Text:

Test inside:
für die ömis zum ässä > Ää Öö Üü è é ç à

inside the email the umlauts are correct but not on header and folders

Note:
Always after a full reboot i can see that displaying is correct but later not

30 sec to 1 min later

Strange…
all is up to date newest version ubuntu 18.04

Edit 2:
if set in
/etc/default/locale
to
LANG=de_DE.UTF-8
after reboot the displaying is correct fro 30-60 seconds…

if set to:
de_DE.UTF8
after reboot the displaying in incorrect from first time reloading…
and also with a emtpy file…

fbartels

Ah, I missed your edits until now. The “issue” is not with the charsets that you set in any of the kopano configuration files, and also not with the locale you define when creating the store. The thing you need to look into is the charset that php uses.

noise

ok that means i need to look in nginx config or php ?
the header of nginx say it is utf-8

but one thing is strange here or normal i don’t know:

with:

wget --server-response -O /dev/null https://www.domain.com/webapp

the output is:
Content-Type: text/html; charset=utf-8

but with:

curl -I https://domain.com/webapp

the output is:
Content-Type: text/html

what is correct ? a check with a online tool means also it is utf-8

phpini from fpm is:
default_charset = "UTF-8"

fbartels

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

ok that means i need to look in nginx config or php ?

yes, php

noise

hmmm strange i can’t find anything… charset is utf-8 in php

the complete php output is too big to post here…
so i create a pdf link:
https://de.scribd.com/document/396095828/phpinfo

tjoen

If it is solved in Kopano it doen’t mean it is solved at the recepient. See
https://en.wikipedia.org/wiki/International_email#UTF-8_headers

noise

its not definitely postfix because other mail servers i don’t have this.
since it sometimes synonymous with the reload occurs, I think, it’s already php but not found yet.

here is also my header!

noise

i look again for my problem but not really found, i check also php and remove some modules to testing but always the same
the only one there is no charset is here:

$ curl -I https://www.domain.example/webapp
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Mon, 07 Jan 2019 10:27:11 GMT
Content-Type: text/html
Content-Length: 178
Location: https://www.domain.example/webapp/
Connection: keep-alive
Expires: Wed, 06 Feb 2019 10:27:11 GMT
Cache-Control: max-age=2592000
Cache-Control: public, must-revalidate, proxy-revalidate
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

the adress: https://www.domain.example/webapp
is the only where has no charset, but it’s a redirect to
https://www.domain.example/webapp/
and here we have the charset…

$ curl -I https://www.domain.example/webapp/
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 07 Jan 2019 10:27:16 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: __Secure-KOPANO_WEBAPP=g9cv0gsd2dh1tlc5ofo40fcpe3; path=/; secure; HttpOnly
Expires: Wed, 06 Feb 2019 10:27:16 GMT
Cache-Control: max-age=2592000
Pragma: no-cache
Set-Cookie: __Secure-encryption-store-key=51f94dc3cfbafd03948338ced240a0b9; path=/; secure; HttpOnly
Cache-Control: public, must-revalidate, proxy-revalidate
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

i don’t know if that is my problem ?
any hint’s are welcome

zara-kopa

@ noise
I am running under Debian 9.6 with PHP7.2 from the external “sury” Repo. I got the same problems. With a fist login in WebApp all looks fine for the german ‘umlaute’ in foldernames. German “Entwürfe” looks like “Entwürfe”. With the second login of the same user “Entwürfe” goes to “Entw?rfe”. — I get the same results as you with the different curl-Querys.
In Mails, Mail-Headers, etc. an in all other forms of using KOPANO (imap, outlook -mapi, outlook -z-push) the foldernames and all other use of “öü” is displayed right!
I use another Kopano-Installation with Debian 8 and php 5. Testing this install, i got the same results with your curl querys. Therefore the problem may be in an other context.
Because of the first login, when everything looks allright i will search whats happens up to the second login.

any hint’s are welcome

jengelh

If folder names change over time, sounds like data is transferred into the DB in a lossy fashion (and the only reason it shows ü in the first round is because the in-memory cache keeps it in its original form… until such a time it gets evicted).