Reports with umlauts in the subject line

mritzmann · March 20, 2019, 11:10am

Hi!

We have websites names with umlauts
The umlauts end up in the subject of e-mail reports
Sending is done with Postfix 2.11x (default on debian 8)
Webserver: nginx/1.14.2
PHP Version: 7.2.16

The problem is now: The e-mail server (here postfix) does not accept e-mail headers that are not correctly encoded. In this case, the subject.

Reason for bounce:

Reporting-MTA: dns; ****
X-Postfix-Queue-ID: 71E9F40059
X-Postfix-Sender: rfc822; postmaster@***
Arrival-Date: Mon, 18 Mar 2019 10:19:37 +0100 (CET)

Final-Recipient: rfc822; ***@***
Original-Recipient: rfc822;***@***
Action: failed
Status: 5.6.7
Diagnostic-Code: X-Postfix; SMTPUTF8 is required, but was not offered by host

(Note: The Postfix option smtputf8_enable is only available from Postfix 3.x on)

Interestingly, the problem only occurs on this one Matomo installation. All other Matomo installations that use the same mail gateway and the same environment (even on the same server) do not have this problem.

Are there any settings on Matomo’s part that can be made to deal with the correct encoding of email?

Thanks,
Markus

mritzmann · April 4, 2019, 11:42am

The problem only occurs if the interface language is set to German. I therefore assume that there is a bug and have created an issue: Reports with umlauts in the subject line · Issue #14313 · matomo-org/matomo · GitHub

mritzmann · April 4, 2019, 3:16pm

Note: Posts below have been moved from GitHub by @Lukas.

Step to reproduce:

Use a Postfix Version without smtputf8_enable (default Debian 8)
Setup Matomo and create a new website with äöü as website name
Create a new report.
Set interface language to english and trigger “Send Report now”: E-mail delivery works
Set interface language to deutsch and trigger “Bericht jetzt senden”: E-mail delivery does not work

Lukas · April 4, 2019, 3:16pm

This should be the perfect bug for me to reproduce:

I am using Debian 9 with the default postfix (3.1.9)
smtputf8_enable seems to be disabled (I never changed anything in the config)

➜  /etc/postfix grep -r "smtputf8_enable"
main.cf:smtputf8_enable = no

my Matomo is set to German
I have a website that has an “ö” in the title
I have just created a new report for this website and clicked on Bericht jetzt senden

But unlike in your case, for me the email is sent normally and I get it with the umlaut in the title of the mail.

As the only difference seems to be the the debian (and therefore postfix) version, I think that this isn’t necessarily a bug in Matomo, but maybe rather something odd in your setup.

Are you using SMTP or phpmail in Matomo? (I quickly switched to SMTP, but the result stayed the same)

fdellwing · April 4, 2019, 3:16pm

Could you both take a look at the subject header? To work everywhere it should look like this =?UTF-8?B?w6TDtsO8?= or =?UTF-8?Q?=C3=A4=C3=B6=C3=BC?=.

To do this in PHP is really simply:

$subject = "=?UTF-8?Q?" . quoted_printable_encode($subject) . "?=";

If this is not in Matomo, it should be added.

(There are a lot more complex methods to do this, see https://stackoverflow.com/a/27648245/7849268 for more info)

mritzmann · April 4, 2019, 3:16pm

Could you both take a look at the subject header?

# first in german, second in english

$ tail -n 2 ~/log/php-mail.log 
[04-Apr-2019 13:15:18 UTC] mail() on [/home/matomo/www/libs/Zend/Mail/Transport/Sendmail.php:106]: To: markus@example.com -- Headers: From: Matomo Analytics <noreply@matomo.debugg.ops.opsserver.ch> Reply-To: admin <markus@example.com> Date: Thu, 04 Apr 2019 13:15:18 +0000 Content-Type: multipart/related;  boundary="=_e7389ca96f4c90ac0c36c7746f88819a" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline MIME-Version: 1.0 -- Subject: =?utf-8?Q?Bericht=20B=C3=A4rentatze=20-=20Mittwoch,=203.=20April=202019?=
[04-Apr-2019 13:15:32 UTC] mail() on [/home/matomo/www/libs/Zend/Mail/Transport/Sendmail.php:106]: To: markus@example.com -- Headers: From: Matomo Analytics <noreply@matomo.debugg.ops.opsserver.ch> Reply-To: admin <markus@example.com> Date: Thu, 04 Apr 2019 13:15:32 +0000 Content-Type: multipart/related;  boundary="=_84ed37fcef806309a62f03d977cb2914" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline MIME-Version: 1.0 -- Subject: =?utf-8?Q?Report=20B=C3=A4rentatze=20-=20Wednesday,=20April=203,=202019?=

Looks good to me.

Are you using SMTP or phpmail in Matomo?

phpmail, with sendmail:

$ grep -i sendmail /etc/php72/user/matomo/php.ini 
sendmail_path=/usr/sbin/sendmail -t -i

mritzmann · April 4, 2019, 3:16pm

I am using Debian 9 with the default postfix (3.1.9)

My setup:

+--------------------------------+                   +--------------------------------+
|                                |                   |                                |
| LEMP Stack                     |                   | Outgoing Mailrelay             |
| ==========                     |                   | ==================             |
|                                |                   |                                |
| OS: Debian 9                   |                   | OS: Debian 8                   |
| Webserver: Nginx               |          +--------+ Postfix: 2.11.3                |
| PHP: 7.2.16 (fpm)              |          |        |                                |
| Postfix: 3.1.9                 +----------+        |                                |
|                                |                   |                                |
+--------------------------------+                   +--------------------------------+

The e-mail is rejected by the outgoing mail relay. Seems as if the problem does not exist with newer Postfix versions. I will update the Mailrelay in the near future, then the problem has probably been solved.

Just found it strange that the problem only occurs when the interface is set to “Deutsch”… Hhm.

mritzmann · April 4, 2019, 3:17pm

Just found it strange that the problem only occurs when the interface is set to “Deutsch”… Hhm.

I mean: What is the difference between German and English when sending e-mails from Matomo?

fdellwing · April 4, 2019, 3:17pm

With this header, the error should never appear. So this is strange.

Could you try sending a mail from your Matomo host via CLI using the correctly encoded header?

mritzmann · April 4, 2019, 3:17pm

Could you try sending a mail from your Matomo host via CLI using the correctly encoded header?

This works:

$ mail -s "=?utf-8?Q?Bericht=20B=C3=A4rentatze=20-=20Mittwoch,=203.=20April=202019?=" markus@example.com < /dev/null
mail: Null message body; hope that's ok

Found out something interesting:

I saved the mail locally instead of sending it to see if I could see a difference.

$ grep -i sendmail /etc/php72/user/matomo/php.ini
sendmail_path=/usr/local/vzscripts/stdintotmp

$ cat /usr/local/vzscripts/stdintotmp
#!/bin/sh
cat < /dev/stdin > $(/bin/mktemp --tmpdir=${HOME}/tmp/ mail.$(date +%Y-%m-%d_%H-%M-%S).XXX.eml)

The Mail in English is stored as UTF-8, the German as ASCII only:

# english
$ file mail.2019-04-04_16-23-10.Hw9.eml
mail.2019-04-04_16-23-10.Hw9.eml: HTML document, UTF-8 Unicode text

# german
$ file mail.2019-04-04_16-23-42.Apl.eml
mail.2019-04-04_16-23-42.Apl.eml: HTML document, ASCII text

mail.2019-04-04_16-23-42.Apl.txt
mail.2019-04-04_16-23-10.Hw9.txt