core-8.6.80.621: German Umlauts broken in the subject line of sent mails

fbartels

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

ok that means i need to look in nginx config or php ?

yes, php

noise

hmmm strange i can’t find anything… charset is utf-8 in php

the complete php output is too big to post here…
so i create a pdf link:
https://de.scribd.com/document/396095828/phpinfo

tjoen

If it is solved in Kopano it doen’t mean it is solved at the recepient. See
https://en.wikipedia.org/wiki/International_email#UTF-8_headers

noise

its not definitely postfix because other mail servers i don’t have this.
since it sometimes synonymous with the reload occurs, I think, it’s already php but not found yet.

here is also my header!

noise

i look again for my problem but not really found, i check also php and remove some modules to testing but always the same
the only one there is no charset is here:

$ curl -I https://www.domain.example/webapp
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Mon, 07 Jan 2019 10:27:11 GMT
Content-Type: text/html
Content-Length: 178
Location: https://www.domain.example/webapp/
Connection: keep-alive
Expires: Wed, 06 Feb 2019 10:27:11 GMT
Cache-Control: max-age=2592000
Cache-Control: public, must-revalidate, proxy-revalidate
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

the adress: https://www.domain.example/webapp
is the only where has no charset, but it’s a redirect to
https://www.domain.example/webapp/
and here we have the charset…

$ curl -I https://www.domain.example/webapp/
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 07 Jan 2019 10:27:16 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: __Secure-KOPANO_WEBAPP=g9cv0gsd2dh1tlc5ofo40fcpe3; path=/; secure; HttpOnly
Expires: Wed, 06 Feb 2019 10:27:16 GMT
Cache-Control: max-age=2592000
Pragma: no-cache
Set-Cookie: __Secure-encryption-store-key=51f94dc3cfbafd03948338ced240a0b9; path=/; secure; HttpOnly
Cache-Control: public, must-revalidate, proxy-revalidate
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

i don’t know if that is my problem ?
any hint’s are welcome

zara-kopa

@ noise
I am running under Debian 9.6 with PHP7.2 from the external “sury” Repo. I got the same problems. With a fist login in WebApp all looks fine for the german ‘umlaute’ in foldernames. German “Entwürfe” looks like “Entwürfe”. With the second login of the same user “Entwürfe” goes to “Entw?rfe”. — I get the same results as you with the different curl-Querys.
In Mails, Mail-Headers, etc. an in all other forms of using KOPANO (imap, outlook -mapi, outlook -z-push) the foldernames and all other use of “öü” is displayed right!
I use another Kopano-Installation with Debian 8 and php 5. Testing this install, i got the same results with your curl querys. Therefore the problem may be in an other context.
Because of the first login, when everything looks allright i will search whats happens up to the second login.

any hint’s are welcome

jengelh

If folder names change over time, sounds like data is transferred into the DB in a lossy fashion (and the only reason it shows ü in the first round is because the in-memory cache keeps it in its original form… until such a time it gets evicted).

zara-kopa

Hi everyone

i solved my problem! It was made by my own. All Ü’s and Ö’s are stable back again.
Shamefully the reason was a miss-configuration of the Apache server! No php Problem.
The error results out of a not enabled “javascript-common.conf” and some enabled mods with conflicts.

Härzlichen Dönk !!!

chrizzibaer

Hey zara-kopa,

Could you provide details where you solved the problem in your apache config?
I got the same error on my fresh install, and I just can’t figure out why.

Thanks a lot and greetings from Nuremberg ;)
Christoph.

zara-kopa

@chrizzibaer

Here are the enabled CONFS and MODS of my apache2

1_1547981822034_Bildschirmfoto 2019-01-20 um 11.54.36.jpg 0_1547981822033_Bildschirmfoto 2019-01-20 um 11.55.00.jpg

With these -enabled it’s worked for me. The use of php7.2-fpm is still in progress!
nice weekend from the Ruhrpott

noise

@zara-kopa i’m running at nginx and also in fpm and still have the problem…
it is possible that comes from mysql (mariabd)
The Kollation of kopano innodb is: utf8mb4_general_ci ?

noise

Hi guys

i come back to my old problem because i think its solved with the latest update (i not update since may 2019) and i leave the umlauts problems…

so i update all including system to 18.04.3 LTS
(over 200updates) and kopano to 8.7.82.92

after this update spooler & dagent stops working with transport, so i deactivate the plugIns. now the umlauts problem are gone with nginx.

Perhaps it was the option “plugin_enabled = yes” in dagent or spooler. i don’t know but it works now.

fbartels

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

Perhaps it was the option “plugin_enabled = yes” in dagent or spooler. i don’t know but it works now.

I think the main factor was probably https://forum.kopano.io/topic/2536/webapp-3-5-7-final-available

@noise said in core-8.6.80.621: German Umlauts broken in the subject line of sent mails:

after this update spooler & dagent stops working with transport, so i deactivate the plugIns

Can you give some more details as to the “stops working with transport”?

noise

@fbartels said in Can you give some more details as to the “stops working with transport”?

i delete my last logs about this but i make some notices for myself…

the services are all running after the update, but send and receive a mail was not possible.

here are my workaround of this, but here are the wrong topic about this ;-)

First logs:

==> /var/log/mail.log <==
Sep  1 16:00:46 smtp kopano-dagent[16540]: Python traceback is on stderr (possibly check journalctl instead of logfile)
Sep  1 16:00:46 smtp kopano-dagent[15161]: Python threw an exception:
Sep  1 16:00:46 smtp kopano-dagent[16540]: K-1731: Unable to initialize the dagent plugin manager: Unknown error code (1).
Sep  1 16:00:46 smtp kopano-dagent[16540]: < 503 5.1.1 Internal error during delivery
Sep  1 16:00:46 smtp kopano-dagent[16540]: > QUIT
Sep  1 16:00:46 smtp kopano-dagent[16540]: < 221 2.0.0 Bye
Sep  1 16:00:46 smtp kopano-dagent[16540]: LMTP thread exiting

so i found something about the permissions in
/var/run/kopano/server.sock
/var/run/kopano/prio.sock

But the rights are corectly, i change only

ls -la /var/run/kopano/gateway.pid
-rw-r--r--  1 root   kopano    5 Sep  1 17:17 gateway.pid
to
chown kopano:kopano /var/run/kopano/gateway.pid
-rw-r--r-- 1 kopano kopano 5 Sep  1 17:17 /var/run/kopano/gateway.pid

2nd logs are:

Sun Sep  1 18:11:31 2019: [error  ] Use of Python (plugin_enabled=yes) forces process_model=fork
Sep  1 18:11:24 smtp slapd[1538]: <= bdb_substring_candidates: (cn) not indexed
Sep  1 18:11:31 smtp kopano-spooler[803]: Python threw an exception:

==> /var/log/kopano/spooler.log <==
Sun Sep  1 18:11:31 2019: [crit   ] [ 5966] K-1733: Unable to initialize the spooler plugin system: Unknown error code (1).

so i disable plugins to “no” and uncomment directories in spooler & dagent and it works.

umgfoin

Good evening,
root cause here:
https://github.com/Kopano-dev/kopano-core/pull/18

++umgfoin.

noise

Hi all

i come back again here… 🙈
i had a lots problem with mapi and search that i now solved, then i upgrade to the latest version. Kaldav i have not installed allreday.

But my umlauts problem has reversed now, webapp and imap (gateway) seems to be ok, but now z-push makes not a right conversion.
i have also updates z-push

Z-Push:   2.5.1+0-0 
PHP-MAPI: 8.7.82

The umlauts fail is only at the subject, inside the message there is no problem.

Edit:

i delete the device from z-push and do a completly resync,
by a iPhone and android 9 this not resolve the problem on the iPhnoe, but on the android already.
so i delete the account from iphone and the delete a second time the devices from z push.
Now it seems to be ok, but not shure how long :-)

noise

Report:
the umlauts problem still persists with z-push and also in contacts.
at the webapp is solved, there i don’t saw anymore.

nielsh

Hi together,
my solution to remove the umlauts problem after the kopano upgrade to the current version was to change the webapp PHP default fallback language to “en_US.UTF-8”. I have done this in the file “/etc/kopano/webapp/config.php”:

        //      define('LANG', 'en_GB'); // default fallback language
                define('LANG', 'en_US.UTF-8'); // default fallback language

So commented the old value “en_GB” and using the new one “en_US.UTF-8”.
After reloading the webapp in the browser all umlauts are there again.

Before I have changed the “config.php” I also changed the charsets in the following files to UTF-8. This did not help for the webapp, but I hope it will prevent other charset problems in future.

spooler.cfg:#charset_upgrade = windows-1252
spooler.cfg:charset_upgrade = utf-8

unix.cfg:# fullname_charset = iso-8859-15
unix.cfg:fullname_charset = utf-8

IIIMIIIRIII

My solution:

Changed the configuration of apache2 in:
/etc/apache2/envvars

changes:
“export LANG=C”

to

“export LANG=de_DE.UTF-8”

service apache2 restart

restart the apache server and then it works.

BR

Martin

irreleph4nt

@fbartels

I have now been hit by this problem as well. In summary:

Subject lines for emails show ? instead of Umlaut
E-Mail body and folder names are unaffected
Contact names display ? or multiples of ?? in Webapp but work via Z-Push as well as external email clients (evolution)

My setup:

CentOS 7 & 8
Nginx

Fixes I tried as referred to in other of the many threads on the issue:

set KOPANO_LOCALE and KOPANO_USERSCRIPT_LOCALE to de_DE.utf8 in /etc/sysconfig/kopano
set default fallback language for webapp to en_US.UTF-8
apache envvars do not apply as I am not running apache at all

Webapp:

Evolution:

Changing the name of the contact via Z-Push or Evolution changes nothing about the display in Webapp.
Changing the name in Webapp converts the name to “Dhn” upon saving, dropping the Umlaut alltogether.

Kopano Webapp is the only application hosted on my server that displays these issues, everything else works fine.
It’s alarming how nothing is done to address these issues which clearly affect so many people.