Hi,
the Cron Archiver requires an extreme amount of RAM in PHP to generate the annual report and I am wondering if this is normal or if this can be optimized. Some background:
- Server with 16 CPU cores
- Server with 32GB RAM for PHP and MySQL
- Annual report contains about 13 million visits
- Running Matomo 4.5.0
- Async process archiving supported, using CliMulti
- Website has about 60 Matomo segments
Adjustments to the config to be able to make more accurate requests:
; maximum number of rows for any of the Referers tables (keywords, search engines, campaigns, etc.), and Custom variables names
datatable_archiving_maximum_rows_referrers = 2500
; maximum number of rows for any of the Referers subtable (search engines by keyword, keyword by campaign, etc.), and Custom variables values
datatable_archiving_maximum_rows_subtable_referrers = 2500
; maximum number of rows for any of the Actions tables (pages, downloads, outlinks)
datatable_archiving_maximum_rows_actions = 2500
; maximum number of rows for pages in categories (sub pages, when clicking on the + for a page category)
datatable_archiving_maximum_rows_subtable_actions = 2500
; maximum number of rows for any of the Events tables (Categories, Actions, Names)
datatable_archiving_maximum_rows_events = 2500
; maximum number of rows for sub-tables of the Events tables (eg. for the subtables Categories>Actions or Categories>Names).
datatable_archiving_maximum_rows_subtable_events = 100
; maximum number of rows for all individual Custom Dimensions reports
datatable_archiving_maximum_rows_custom_dimensions = 2500
; maximum number of rows for the Custom Dimensions subtables (list of all Page URLs per dimension value)
datatable_archiving_maximum_rows_subtable_custom_dimensions = 2500
; maximum number of rows for other report limits
datatable_archiving_maximum_rows_standard = 2500
To be able to create the annual report of the website, I need about 12.5GB RAM for the annual report. Otherwise the following error appears:
ERROR [2021-11-13 18:56:24] 16365 Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=year&date=2021-01-01&format=json&trigger=archivephp. Response was 'PHP Fatal error: Allowed memory size of 12884901888 bytes exhausted (tried to allocate 20480 bytes) in /var/www/vhosts/xxx/httpdocs/core/DataTable/Row.php on line 361 {"result":"error","message":"Allowed memory size of 12884901888 bytes exhausted (tried to allocate 20480 bytes) on \/var\/www\/vhosts\/xxx\/httpdocs\/core\/DataTable\/Row.php(361) #0 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor.php(226): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name="Actions_actions_url") #1 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Actions\\Archiver->callAggregateMultipleReports() ","backtrace":"Allowed memory size of 12884901888 bytes exhausted (tried to allocate 20480
bytes) on \/var\/www\/vhosts\/xxx\/httpdocs\/core\/DataTable\/Row.php(361)\n#0 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor.php(226): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name=\"Actions_actions_url\")\n#1 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Actions\\Archiver->callAggregateMultipleReports()\n\n#0 [internal function]: Piwik\\Plugins\\CorePluginsAdmin\\Controller->safemode()\n#1 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(619): call_user_func_array()\n#2 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(168): Piwik\\FrontController->doDispatch()\n#3 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(99): Piwik\\FrontController->dispatch()\n#4 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(274): Piwik\\FrontController::generateSafeModeOutputFromError()\n#5 [internal function]: Piwik\
\FrontController::triggerSafeModeWhenError()\n#6 {main}"}{"result":"error","message":"Allowed memory size of 12884901888 bytes exhausted (tried to allocate 20480 bytes) on \/var\/www\/vhosts\/xxx\/httpdocs\/core\/DataTable\/Row.php(361) #0 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor.php(226): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name="Actions_actions_url") #1 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Actions\\Archiver->callAggregateMultipleReports() ","backtrace":"Allowed memory size of 12884901888 bytes exhausted (tried to allocate 20480 bytes) on \/var\/www\/vhosts\/xxx\/httpdocs\/core\/DataTable\/Row.php(361)\n#0 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor.php(226): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name=\"Actions_actions_url\")\n#1 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/ArchiveProcessor\/Plugin
sArchiver.php(168): Piwik\\Plugins\\Actions\\Archiver->callAggregateMultipleReports()\n\n#0 [internal function]: Piwik\\Plugins\\CorePluginsAdmin\\Controller->safemode()\n#1 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(619): call_user_func_array()\n#2 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(168): Piwik\\FrontController->doDispatch()\n#3 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(99): Piwik\\FrontController->dispatch()\n#4 \/var\/www\/vhosts\/xxx\/httpdocs\/core\/FrontController.php(274): Piwik\\FrontController::generateSafeModeOutputFromError()\n#5 [internal function]: Piwik\\FrontController::triggerSafeModeWhenError()\n#6 {main}"}'
Actually, I would like to run several archiving processes in parallel on the server to use the CPU with 16 cores efficiently. Most of the reports (daily, weekly, monthly) require only little RAM. However, since the annual reports require so much RAM and some segments are created, this is not possible. Does the RAM requirement in the Cron Archiver increase proportionally to the number of visits? Is there any way I can optimize Matomo to limit the RAM usage in the Cron Archiver?
Thanks a lot for the help.