core-8.6.80.621: German Umlauts broken in the subject line of sent mails

robgnu

Hello,
today I’ve upgraded to the current nightly (06.05.2018 / core-8.6.80.752_0+85-Debian_9.0-amd64) and I can’t confirm the fix. When I create a new user, I still get the ? instead of the “Umlauts” in the folder names.

I created a user with this command:

kopano-cli --create --user user1 --password "user1" --email user1@example.com --fullname "User One" --admin-level 1

robgnu

Gerald

@robgnu
My confirmation that this has been fixed was only meant to be about the corrupt subject line, not about the folder name issue.

Update: Tried the folder issue myself now because I needed to create a user anyway. Still present. User created with commandline and logged in via WebApp. Folder names corrupt. Fixed them with kopano-localize-folders -u user@domain.tld --lang de_DE.UTF-8

fbartels

@robgnu said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

I can’t confirm the fix.

We are talking about two different issues here. Kc-1139 was solved, kc-1140 is still open.

robgnu

Oh ok - sorry.

noise

hello
i have a similar problem with run

kopano-localize-folders -u user@domain.tld --lang de_DE.UTF-8

the output is:

user@domain.tld
Traceback (most recent call last):
  File "/usr/sbin/kopano-localize-folders", line 114, in <module>
    main()
  File "/usr/sbin/kopano-localize-folders", line 88, in main
    print('%s: Changing localized folder names to \"%s\"' % (user.name.decode('utf-8'), options.lang))
AttributeError: 'str' object has no attribute 'decode'

anyone have a idea of this ? running ubuntu 18.04 core-8.7.80

noise

i’m no able to solve my above problem but i’m able to have all umlaut in german language…

first detach store from user:

kopano-storeadm -Dn <user1>

add new store to user

kopano-storeadm -Cn <user1> -l de_DE.UTF-8

after this i got german names in folders like inbox > posteingang
but the umlauts are not displaying correct.
My charset of nginx is set to UTF-8

so i found in the spooler.cfg

#charset_upgrade = windows-1252

i change it to:

charset_upgrade = UTF-8

now it looks to work properly, all umlaut are displaying

thctlo

a very good point here.

dagent.cfg:#default_charset = us-ascii
ical.cfg:#default_charset = utf-8
ldap.cfg:#ldap_server_charset = utf-8
spooler.cfg:#charset_upgrade = windows-1252
unix.cfg:fullname_charset = iso-8859-15

uhhhh, why o why ?? please fix this all to UTF8

noise

ok i look at other files too…

edit: dagent.cfg charset is uncommented i leave it at this time but you have right why not all config files are to UTF-8 ?

thctlo

no, i say this is one that kopano should fix, its crazy to have so much differnent types.

thctlo

@noise

edit: dagent.cfg charset is uncommented i leave it at this time but you have right why not all config files are to UTF-8 ?

i dont know, wait for a kopano dev to reply this.

fbartels

@thctlo said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

this is one that kopano should fix

It’s all open source, you know. If you feel you need to change something: https://stash.kopano.io/projects/KC/repos/kopanocore/browse/CONTRIBUTING.md

I doubt that it is really necessary, since there are plenty of systems out there that work fine with umlauts in subjects without the need to fiddle with charset settings in the spooler.

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

Traceback (most recent call last):

I did not yet manage to look into this, but I would expect that this is a python2 vs python3 issue. in master we recently switched everything to python3 and some scripts may still require love.

noise

@fbartels said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

I would expect that this is a python2 vs python3 issue. in master we recently switched everything to python3 and some scripts may still require love.

yes i think also but i try to purge python2.7 but not helps. I’ll keep looking, if i found anything i post it.

thanks anyway

fbartels

no, the script has a shebang that points towards python3. the simplest workaround would be that change that to point towards python2. Still if you want to look further into this a pr for this is of course welcome as well.

btw. this does not necessarily need to be through git-patch, opening a pr in github also works for us -> https://github.com/Kopano-dev/kopano-core

noise

i have installed booth python 2.7 and 3.7.1 so i try to purge 2.7 and i see i now have also 3.6.7
i’m not much involved with python…

edit: i think i need to do symlinks between the versions ib usr/lib/

fbartels

yeah. no, this is 100% not what i meant. my GUESS is that the code in the script you are trying to run is not compatible with python3 (which is the interpreter he tries to execute it with because of the shebang). You will still need python2 on your system (and python2-kopano or however the file is called in your distribution). simply run the script with python2 instead of python3. if you don’t want to fiddle with the script, then a python2 /usr/sbin/kopano-localize-folders .. probably does the trick as well.

noise

ah ok my problem is python 3 not 2 ok i will try… thanks

Edit:

ok i solved with install

apt install python-kopano

with the command for all users:

python2 /usr/sbin/kopano-localize-folders --lang de_DE.UTF-8

the user are changed:

<user1>: Changing localized folder names to "de_DE.UTF-8"

thanks for the hint.

noise

Hmmm but strange now i lost the umlauts again with the above…

Edit:

Now i try a few things also changing from
DE.UTF-8 to de.utf8

my settings are:

$ locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

$ locale -a
C
C.UTF-8
de_AT.utf8
de_BE.utf8
de_CH.utf8
de_DE.utf8
de_IT.utf8
de_LI.utf8
de_LU.utf8
POSIX

i try to use booth config (old and new) alone and booth togheter now are booth active:

$ cat /etc/default/kopano
KOPANO_LOCALE="de_DE.UTF-8"
KOPANO_USERSCRIPT_LOCALE="de_DE.UTF-8"

$ cat /etc/kopano/admin.cfg
# The language for folders in newly-created stores, specified as a
# locale identifier ("en_US", "de_DE", etc.)
default_store_locale = de_DE.UTF-8
#server_socket = default:
#sslkey_file = some.pem
#sslkey_pass = magic

but a email is watching like:
Header:

f?r die ?mis zum ?ss? > ?? ?? ?? ? ? ? ?
User <user@gmail.com>
Gesendet:Mittwoch 19 Dezember 2018 20:44
An:
User1 <user1@domain.com>

Email Text:

Test inside:
für die ömis zum ässä > Ää Öö Üü è é ç à

inside the email the umlauts are correct but not on header and folders

Note:
Always after a full reboot i can see that displaying is correct but later not

30 sec to 1 min later

Strange…
all is up to date newest version ubuntu 18.04

Edit 2:
if set in
/etc/default/locale
to
LANG=de_DE.UTF-8
after reboot the displaying is correct fro 30-60 seconds…

if set to:
de_DE.UTF8
after reboot the displaying in incorrect from first time reloading…
and also with a emtpy file…

fbartels

Ah, I missed your edits until now. The “issue” is not with the charsets that you set in any of the kopano configuration files, and also not with the locale you define when creating the store. The thing you need to look into is the charset that php uses.

noise

ok that means i need to look in nginx config or php ?
the header of nginx say it is utf-8

but one thing is strange here or normal i don’t know:

with:

wget --server-response -O /dev/null https://www.domain.com/webapp

the output is:
Content-Type: text/html; charset=utf-8

but with:

curl -I https://domain.com/webapp

the output is:
Content-Type: text/html

what is correct ? a check with a online tool means also it is utf-8

phpini from fpm is:
default_charset = "UTF-8"

fbartels

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

ok that means i need to look in nginx config or php ?

yes, php