Weekly mail report is "empty". It has different/wrong "mime format" since Version 2.15.0

Support & Bugs
1

Hi all,

we generate a scheduled weekly mail report with a attached PDF of the statistics of one week that will be send to different
mail addresses. Since the update from version 2.14.3 to 2.15.0 in some mailclients this mail is “empty”. For example: The mail is “empty” in Windows 7 /Thunderbird 38.5.1. In our webmail client (All Browser, also on Macintosh) its empty, too OR there is only the following “content” in the message:

…
This is a message in Mime Format. If you see this, your mail reader
does not support this format.

–_dfb4eb2627b3bxxx
Content-Type: text/plain; charsetutf-8
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hello,
Please find in attached file your weekly report for “website”.
Sent from https://correct.url/.
–_dfdfdf…
Content-Type: application/pdf
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename"Report WEBSITE - week December
21 – 27, 2015.pdf"
JVBERi0xLjcKJeLjz9MKODAgMCBvYmoKPDwgL1R5cGUgL1BhZ2UgL1BhcmVudCAxIDAgUiAv
TGFzdE1vZGlmaWVkIChEOjIwMTUxMjI4MTEwMTM0KzAwJzAwJykgL1Jlc291cmNlcyAyIDAg
UiAvTWVkaWFCb3ggWzAuMDAwMDAwIDAuMDAwMDAwIDU5NS4yNzYwMDAgODQxLjg5MDAwMF0g…
…

Perhaps it has something to do with the change in version 2.15.0 (taken from the Changelog)
– "Format numbers displayed in Piwik reports (eg. 1,000,000 instead of 1000000)"
or
– "Email reports: print the full URL source / origin of report in the email body "

I don’t know, but I think the “error” (wrong format) comes from the new “subject line” format.
Since version 2.15.0 the mail contains the subject like
"Report WEBSITE - week December 28, 2015 – January 3, 2016".

Before version 2.15.0 it was like:
“Report WEBSITE - Week 30 March - 5 April 2015”

the “minus” sign “-” (Hyphen-Minus) is in the new subject the “long minus” (dash?) “–”, perhabs this in not “utf8” compatibel or something like this. (I don’t even now, how to type this on the (german) keyboard…).

Perhaps the problem is in the given name of the PDF, too? There is a “–” in the Name
"…December 21 – 27, 2015.pdf". Perhabs this “curious characters” between “21” and “27” is the problem? I think, this is the troublemaking “dash”, too…

In a scheduled daily report, everything is correct, because here is this “dash” not used in the “subject”,
for example “Bericht WEBSITE - Montag, 14. Dezember 2015”.

And just now I’m noticing: Before 2.15.0 there was a german subject line, now it’s only the English one, although we configure the mail report in the German admin web interface.
Perhabs this was also not considered really in this version update? Don’t know…

Thanks for any info about this.

Best greetz,
kordo

2

Hi @koro

I tried fixing this issue in this patch: Replace some characters in Email subject and Email attachments filename by mattab · Pull Request #9632 · matomo-org/piwik · GitHub

Could you apply this to your Piwik

diff --git a/core/Mail.php b/core/Mail.php
index 454fcd3..192c5a9 100644
--- a/core/Mail.php
+++ b/core/Mail.php
@@ -127,6 +127,18 @@ public function send($transport = null)
         }
     }
 
+    public function createAttachment($body, $mimeType = null, $disposition = null, $encoding = null, $filename = null)
+    {
+        $filename = self::sanitiseString($filename);
+        return parent::createAttachment($body, $mimeType, $disposition, $encoding, $filename);
+    }
+
+    public function setSubject($subject)
+    {
+        $subject = self::sanitiseString($subject);
+        return parent::setSubject($subject);
+    }
+
     /**
      * @param string $email
      * @return string
@@ -154,4 +166,18 @@ protected function isHostDefinedAndNotLocal($url)
     {
         return isset($url['host']) && !Url::isLocalHost($url['host']);
     }
+
+    /**
+     * Replaces characters known to appear incorrectly in some email clients
+     *
+     * @param $string
+     * @return mixed
+     */
+    static public function sanitiseString($string)
+    {
+        $search = array('–', '’');
+        $replace = array('-', '\'');
+        $string = str_replace($search, $replace, $string);
+        return $string;
+    }
 }

and let me know in this issue if the issue is now fixed?

3

Hi matthieu,

many thanks for your answer!! :slight_smile:

…it’s “not really, but nearly” fixed now with your patch. And: The solution is finally also clear now.

At first: This “long dash” is really the troublemaking sign!
In the attachment name and in the subject.
So, I applied your fix to “core/Mail.php”. I see that in your fix, this “long dash” sign should be replaced by the “normal minus” sign.
–> Should normally work! But…:
The file “Mail.php” is coded/written in “ANSI” mode, not in “UTF-8”, so I can’t place
your line
$search = array(‘longdash’, ‘’");
which contains UTF-8 signs (the “long dash” :wink: )(!) into the file Mail.php, which is “only” ANSI…!

So: these signs, which should be replaced, are not (while processing) “recognized correctly and therefore will not be replaced”.
So it doesn’t help. The resulting Mail is still empty.

But:

I now changed your line “to ANSI” mode and put THIS line into “core/Mail.php”. Then it looks like
$search = array(‘–’, ‘’’);

(These curious signs a correct here! These are the troublemaking “UTF-8” signs codes in “ANSI”! ) .
Now it works like a charm for me. I’m getting the mail correctly with correct attachment. The “long dash” has been replaced by the normal “minus” sign in attachment name and subject name.

But this way of mixing the code pages etc, it’s not the best way… :expressionless:

So, I think… the best way to avoid this problem is: To NOT use the “long dash” her from the complete beginning.
I don’t know, where “suddenly” this “long dash” comes from / appears, because in the previous version 2.14.3…
this sign was NOT there (Subject and attachment name), there was the normal “minus” sign, and so it worked like a charm before.
So, is there really a special need to use this “long dash” exactly at this place (and not the “minus”) since Version 2.15.0?
Would a normal “minus” sign not be ok here, because then “ANSI” and “UTF-8” sign is the same. :slight_smile:

By the way…: I simply tried to change the file mode of “Mail.php” from “ANSI” to “UTF-8” and use your UTF-8 line.
But then, when I sent a testmail. this procedure “of sending” never ends… it says in the web interface “loading data” for ever… and nothing happens anymore. So this was not a working simple “trick”, and I changed the mode back to “ANSI”… :wink:

Sorry, if something is not clear in my text, I can write in my mother language German , if its needed. :wink: :slight_smile:

Best Greetings,
kordo

4

Hello
could you try our latest RC version and let us know if it is not fixed there?

5

Hi matthieu,

uh… I’m sorry, but I have only the “live production system” here…, and therefore can not install the “RC version” over it (or in parallel) regrettably.
I could only try to temporarely replace single files in the existing production system installation, this would be temporarely ok.
If you for example let me know, where I can get
the “new” file “…/core/mail.php” from the new (2.15.x ?) -RC version…
but I think, this is not the only file, I would have to replace to test this mail issue…? so I’ don’t know, if I can help with this here…?

By the way: Just a few minutes ago, I saw in
Piwik Platform Changelog - Matomo Analytics (formerly Piwik Analytics) - Developer Docs - v3
(Section for 2.15.0)
these “Breaking Changes”:

–> Handling of localized date, time and range formats has been changed.
Patterns no longer contain placeholders like %shortDay%, but work with
CLDR pattern instead. You can use one of the predefined format constants
in Date class for using getLocalized().
–> As we are now using CLDR formats for all languages, some time
formats were even changed in english. Attributes like prettyDate in API
responses might so have been changed slightly.

… now I have an idea, where this “longdash” and the “Englisch/German mixing” comes from in version 2.15.0…" :wink: :slight_smile:

Greetz,
kordo

6

this was fixed in 2.16.0 and the files fixed were: https://github.com/piwik/piwik/pull/9632/files

7

Hi matthieu,

Yes, I can see the code changes in your git hub link, but this doesn’t help here. I’m not very familiar with this github, but I need a direkt download link to these file/s as complete files/s to test them. If I only copy/paste the “text/code” from the github webpage, the character code page (utf-8, ANSI) is surely not the same anymore, but this is here the main point, so I can’t really test, what happens if I have the complete file (which we normally get, when we patch the installation.). So, do you have a direkt download link for me?

And: Do I need also the file “tests/PHPUnit/Integration/MailTest.php”? Because in my installation 2.15.0 this file is not existing. The directory “tests” is empty, only a “README.md” is there. I have not installed any “tests” here and I’m not planning to :wink:

greetz,
kordo_de