email body with iso-2022-jp display problem

rCj7t993swwMeDsm

I upgraded from zarafa version “7.1.11-46050” to kopano version “8.7.0-7ubuntu1” with webapp version “3.5.14+dfsg1.orig-1”. I could login to webapp and look my email. everything seems smooth.

However, I got problem when reading a mail with Japanese character, the subject and the body of the mail is in Japanese. The Webapp could display the correct subject in Japanese character. However, the message body was scrambled with characters like:

$B$$$D$b!"$4MxMQ$$$?$@$-$"$j$,$H$&$4$6$$$^$9!#(B

I looked up the relevant entries in database of both zarafa and kopano and found that the content were identical. I could see the whole mail in Japanese under Adminer.

My old webapp version “1:1.6-46049” on zarafa server could display the body of this mail without problem. The corresponding characters of the above scrambled text is:

いつも、ご利用いただきありがとうございます。

I also have lots of mails in Traditional Chinese characters and all these mails are displayed on the new Webapp successfully.

Could anyone please advise where is the source of this problem? Thanks a lot.

fbartels

Hi @rCj7t993swwMeDsm,

the versions included in the distributions usually are a bit behind to what we are releasing. Are you able to reproduce the issue with our packages?

rCj7t993swwMeDsm

I have tried the latest community version of kopano-core “10.0.6.298.677a125b5-0+153.1” with webapp “4.2.2850+1689.1”, When I click on this mail, an internal server error 500 occurred. I also noticed that the error log of the web server showed a error when displaying this message:

  thrown in /usr/share/kopano-webapp/server/includes/core/class.conversion.php on line 669" while reading response header from upstream, client: 192.168.X.Y, server: , request: "POST /kopano.php?subsystem=webapp_1595496155082 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.4-fpm.sock:", host: "kopano", referrer: "https://kopano/"
2020/07/23 17:23:06 [error] 16202#16202: *36 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function mb_detect_encoding() in /usr/share/kopano-webapp/server/includes/core/class.conversion.php:669
Stack trace:
#0 /usr/share/kopano-webapp/server/includes/core/class.conversion.php(699): Conversion::findCharSet()
#1 /usr/share/kopano-webapp/server/includes/core/class.operations.php(1677): Conversion::convertCodepageStringToUtf8()
#2 /usr/share/kopano-webapp/server/includes/modules/class.itemmodule.php(459): Operations->getMessageProps()
#3 /usr/share/kopano-webapp/server/includes/modules/class.itemmodule.php(55): ItemModule->open()
#4 /usr/share/kopano-webapp/server/includes/core/class.jsonrequest.php(57): ItemModule->execute()
#5 /usr/share/kopano-webapp/kopano.php(145): JSONRequest->execute()
#6 {main}

Then I installed php7.4-mbstring and retry the message, this time the message body become scrambled.
My system is Ubuntu 20.04 LTS.

fbartels

@rCj7t993swwMeDsm said in email body with iso-2022-jp display problem:

Then I installed php7.4-mbstring and retry the message, this time the message body become scrambled.

and there was no longer a message in the logs?

rCj7t993swwMeDsm

@fbartels

yes, no more error message but ciphered message body.

rCj7t993swwMeDsm

I can confirm that the PHP function mb_detect_encoding used by function “findCharSet($string)” in “class.conversion.php” caused the problem.

        private static function findCharSet($string)
        {
                // Let's try mb_detect_encoding first
                $charset = mb_detect_encoding($string, "auto");

                if ( $charset !== false ){
                        return $charset;
                } else {
                        // Let's try them all
                        foreach ( Conversion::$_CODEPAGES as $code=>$cs ){
                                if ( strlen(iconv($cs, $cs, $string)) === strlen($string) ){
                                        return $cs;
                                }
                        }
                }

                // No charset found
                return false;
        }

If I remark the line calling the “mb_detect_encoding($string, “auto”);”, the message body could be decoded correctly.
To further prove my observation, I appended the $charset to the end of the message body and found that the scrambled message is detected as “UTF-8” while the correct encoding should be “ISO-2022-JP”.

I believe the mb_detect_encoding function is not reliable as the iconv comparison method. I have to remark the mb_detect_encoding first in order to let the webapp works.

fbartels

Hi @rCj7t993swwMeDsm,

good to hear that you made some progress. Would you be open to submit a patch with your changes? Instructions are at https://github.com/Kopano-dev/kopano-webapp/blob/master/CONTRIBUTING.md.

rCj7t993swwMeDsm

@fbartels

I am sorry that I may not appropriate do it as I don’t know the historical rationale to introduce the use of mb_detect_encoding() in the findCharSet() function. I compared this stub to the old Webapp 1.6 and found that the old one didn’t use that function which directly use the iconv comparison method.

I hope someone familiarizing the development of the findCharSet function can review the adoption of mb_detect_encoding() at a more complete level.

fbartels

@rCj7t993swwMeDsm sure no problem. I will trigger out WebApp team to look into your report and potential solution, then.