Out of Memory error when using the console for auto archiving

Support & Bugs
1

Hi all, I am running into issues with our auto archiver. I’ve looked through the other forum posts regarding similar issues, but was not able to come up with a solution.

Setup
We have set up two Matomo installations on the same machine, one respecting DNT and one ignoring it, to analyse the differences in results we obtain. We have set up a proxy to forward incoming requests to both Matomo instances, so all visits are tracked by both of them. We use Docker Compose to manage our instances, databases and proxy configuration. Our site receives on average of about 40,000 visitors per day. We run the archiver using systemd services and timers.

We are only using 4 official plugins (Funnels, LogViewer, QueuedTracking and SearchEngineKeywordsPerformance), and all other plugins we use are core plugins.

Issue
Whenever we try to run the archiver using the console CLI, we we get the following error message:

# ./console core:archive --url=matomo.{domainname} 
INFO [2021-09-22 08:03:53] 84  ---------------------------
INFO [2021-09-22 08:03:53] 84  INIT
INFO [2021-09-22 08:03:53] 84  Running Matomo 4.4.1 as Super User
INFO [2021-09-22 08:03:53] 84  ---------------------------
...
ERROR [2021-09-22 08:10:20] 84  Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=4&period=year&date=2021-01-01&format=json&trigger=archivephp. The response was empty. This usually means a server error. A solution to this error is generally to increase the value of 'memory_limit' in your php.ini file.  For more information and the error message please check in your PHP CLI error log file. As this core:archive command triggers PHP processes over the CLI, you can find where PHP CLI logs are stored by running this command: php -i | grep error_log
INFO [2021-09-22 08:10:20] 84  ''
ERROR [2021-09-22 08:10:20] 84  Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=4&period=year&date=2021-01-01&format=json&trigger=archivephp: ''
...

Inspecting the log, we can see the following PHP errors:

ERROR Piwik\FrontController[2021-09-22 08:12:04 UTC] [4325] Fatal error encountered: /var/www/html/core/Common.php(301): Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes)
ERROR Piwik\FrontController[2021-09-22 08:12:04 UTC] [4325] on /var/www/html/core/Common.php(301)
ERROR Piwik\FrontController[2021-09-22 08:12:04 UTC] [4325] #0 /var/www/html/core/ArchiveProcessor.php(225): Piwik\ArchiveProcessor->aggregateDataTableRecord(name="Actions_actions_url")
ERROR Piwik\FrontController[2021-09-22 08:12:04 UTC] [4325] #1 /var/www/html/core/ArchiveProcessor/PluginsArchiver.php(168): Piwik\Plugins\Actions\Archiver->callAggregateMultipleReports()
ERROR Piwik\FrontController[2021-09-22 08:12:04 UTC] [4325] [Query: ?module=API&method=CoreAdminHome.archiveReports&idSite=4&period=year&date=2021-01-01&format=json&segment=visitorType%3D%3Dnew&trigger=archivephp&pid=09264d519699a890d6caa0bdfa2edf002f803672da2bfadbde111cd54759164ccb5a4d46d37b7b9f118f5669a5da5a3171381&runid=84, CLI mode: 1]
ERROR Piwik\FrontController[2021-09-22 08:12:07 UTC] [4337] Fatal error encountered: /var/www/html/core/DataTable/Row.php(75): Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes)
ERROR Piwik\FrontController[2021-09-22 08:12:07 UTC] [4337] on /var/www/html/core/DataTable/Row.php(75)
ERROR Piwik\FrontController[2021-09-22 08:12:07 UTC] [4337] #0 /var/www/html/core/ArchiveProcessor.php(225): Piwik\ArchiveProcessor->aggregateDataTableRecord(name="Actions_actions_url")
ERROR Piwik\FrontController[2021-09-22 08:12:07 UTC] [4337] #1 /var/www/html/core/ArchiveProcessor/PluginsArchiver.php(168): Piwik\Plugins\Actions\Archiver->callAggregateMultipleReports()
ERROR Piwik\FrontController[2021-09-22 08:12:07 UTC] [4337] [Query: ?module=API&method=CoreAdminHome.archiveReports&idSite=4&period=year&date=2021-01-01&format=json&segment=visitorType%3D%3Dreturning&trigger=archivephp&pid=f30d2618ab0c261a7ac4b0111e1062c23459ba6ce346279623c05ca6fe27d538441963046f4526e5e6d7894b503691cccd782&runid=84, CLI mode: 1]
ERROR Piwik\FrontController[2021-09-22 08:12:08 UTC] [4337] Fatal error encountered: /var/www/html/core/DataTable/Row.php(75): Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes)
ERROR Piwik\FrontController[2021-09-22 08:12:08 UTC] [4337] on /var/www/html/core/DataTable/Row.php(75)
ERROR Piwik\FrontController[2021-09-22 08:12:08 UTC] [4337] #0 /var/www/html/core/ArchiveProcessor.php(225): Piwik\ArchiveProcessor->aggregateDataTableRecord(name="Actions_actions_url")
ERROR Piwik\FrontController[2021-09-22 08:12:08 UTC] [4337] #1 /var/www/html/core/ArchiveProcessor/PluginsArchiver.php(168): Piwik\Plugins\Actions\Archiver->callAggregateMultipleReports()
ERROR Piwik\FrontController[2021-09-22 08:12:08 UTC] [4337] [Query: ?module=API&method=CoreAdminHome.archiveReports&idSite=4&period=year&date=2021-01-01&format=json&segment=visitorType%3D%3Dreturning&trigger=archivephp&pid=f30d2618ab0c261a7ac4b0111e1062c23459ba6ce346279623c05ca6fe27d538441963046f4526e5e6d7894b503691cccd782&runid=84, CLI mode: 1]

It appears to be an out of memory error (Allowed memory size of 2147483648 bytes exhausted), but all info I can find online indicates 2GB should be more than enough to run the archiving job. In fact, we’ve even tried to bump it up to 4GB, but the archiving job still ran out of memory.

Does anyone know what the issue could be, or where I could look for problems in our installation? Or is it not unusual for sites with our visitor amounts to need more than 4GB of memory for the archiving task?

Thanks in advance!

2

Is this related to Can’t upgrade from 3.13.5-rc1 to the new version 4.4.1 - Support & Bugs - Matomo forums? (out of memory when doing some DB operations)

3

Hmm, it doesn’t seem like it. Our entire database is only 10GB (a lot smaller than the one in the other post), and our recent upgrade to 4.4.1 worked without issues. Also, the errors appear to indicate the PHP script itself has run out of the 2048GB of memory we appointed to it. I don’t see any errors related to the mysql database, or I might not know where to look for them.

4

Seems odd, but I’d suggest cranking the memory way up to like 64GB to see if there’s a ceiling, or if this is a runaway allocation of some kind.

Also, ensure the box itself is not running out of memory by adding a large swap file.

5

It looks like the same error I’ve been getting for over a month now after upgrading to 4.4.1. The only difference is that I don’t actually get the

Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes)

error in my PHP logs. I also don’t get the error when I run the command manually from the CLI. It only shows up when the task is run via cron.
Just today I downgraded to Matomo 4.3.1 to try to diagnose the issue further. I’ll post later with the results if they’re encouraging. (So far so good; it’s 5 minutes past the cron job’s scheduled execution time, and I haven’t gotten an error email yet!)

EDIT: downgrading to Matomo 4.3.1 did indeed stop the flood of email error messages!

6

I tried it again with about 6GB of memory (I’m capped by the amount of memory assigned to our server), and it briefly worked, capping out at 5.1GB. However, it doesn’t work all the time.

I might try downgrading to Matomo 4.3.1, to see if that solves the problem. @577895 did your archiver crash every single time you ran it? Or did it sometimes work successfully?

7

I was getting the error pretty much every single time the cron job ran (once per hour).
However, if I ran the archive command manually from the command line via SSH, it never showed the errors. So I don’t know what’s up with that…

In my case I don’t think it’s actually a memory issue, but I could be wrong. Here’s a screenshot of my server’s CPU/RAM usage over the past few days. The green CPU spikes occur on the hour when the job runs, but as you can see, the RAM doesn’t usually change much.

image
image1314×465 59.4 KB

8

I might add that my Matomo DB (less than 200MB) is much smaller even than yours, @gijshendriksen.

9

Ah okay, then I guess it’s not really the same problem. The error I’m having also occurs when I try to run the archiver manually instead of using cron. And whenever I run it, top indicates that the PHP process is indeed using up a lot of memory (5.1GB at the highest point).

A solution might be to wait until we have access to a larger server (which we ordered, but hasn’t arrived yet), and then try and see what the upper limit on memory usage actually is (as @fatica mentioned). Although I’m also a bit afraid that the memory usage will keep increasing throughout the year, as we need to process more and more data for the yearly reports…