Problems with data archiving: error php

ehermon · March 16, 2022, 10:16am

Good Morning,

I have matomo version 4.2.1. For a long time we have configured around 15 sites to obtain access data and be able to do analysis and statistics at the end of each year.

Reviewing the logs that the cron outputs for data archiving, I see the following:

INFO [2022-03-16 10:05:35] 8623 Error: Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=30&period=year&date=2021-01-01&format=json&trigger=archivephp. Response was 'PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/matomo/core/DataTable/Row.php on line 71 PHP Fatal error:

I have looked up the documentation, and have increased the value of memory_limit in the php-fpm to 16G. The server has a total of 32 Gb of RAM.

Even changing this value, it keeps happening to me that gives this error.

Any other alternative to solve this problem and be able to have all the sites processed?

Thanks in advance

heurteph-ei · March 16, 2022, 2:42pm

Are data archived for each month of 2021?

ehermon · March 17, 2022, 8:14am

Good Morning,

I have verified in the piwik_archive_numeric_2021_0X archive tables, that from the month of April for some idSites I do not have all the information processed or they do not appear in said tables, which is where the information is stored once the data has been processed.

Then I send the log of the last file processing that has been carried out.

INFO [2022-03-16 15:03:34] 30512  ---------------------------
INFO [2022-03-16 15:03:34] 30512  INIT
INFO [2022-03-16 15:03:34] 30512  Running Matomo 4.2.1 as Super User
INFO [2022-03-16 15:03:34] 30512  ---------------------------
INFO [2022-03-16 15:03:34] 30512  NOTES
INFO [2022-03-16 15:03:34] 30512  - Async process archiving supported, using CliMulti.
INFO [2022-03-16 15:03:34] 30512  - Reports for today will be processed at most every 3600 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO [2022-03-16 15:03:34] 30512  - Archiving was last executed without error 1 days 3 hours ago
INFO [2022-03-16 15:03:34] 30512  ---------------------------
INFO [2022-03-16 15:03:34] 30512  START
INFO [2022-03-16 15:03:34] 30512  Starting Matomo reports archiving...
INFO [2022-03-16 15:03:34] 30512  Start processing archives for site 4.
INFO [2022-03-16 15:03:34] 30512    Will invalidate archived reports for today in site ID = 4's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:03:34] 30512    Will invalidate archived reports for yesterday in site ID = 4's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:03:35] 30512  Archived website id 4, period = day, date = 2022-03-16, segment = '', 17 visits found. Time elapsed: 0.878s
INFO [2022-03-16 15:03:36] 30512  Archived website id 4, period = week, date = 2022-03-14, segment = '', 102 visits found. Time elapsed: 0.889s
INFO [2022-03-16 15:03:37] 30512  Archived website id 4, period = month, date = 2022-03-01, segment = '', 471 visits found. Time elapsed: 1.164s
INFO [2022-03-16 15:03:38] 30512  Archived website id 4, period = year, date = 2022-01-01, segment = '', 2841 visits found. Time elapsed: 0.884s
INFO [2022-03-16 15:03:38] 30512  Finished archiving for site 4, 4 API requests, Time elapsed: 4.362s [1 / 15 done]
INFO [2022-03-16 15:03:38] 30512  Start processing archives for site 6.
INFO [2022-03-16 15:03:39] 30512  Finished archiving for site 6, 0 API requests, Time elapsed: 0.007s [2 / 15 done]
INFO [2022-03-16 15:03:39] 30512  Start processing archives for site 8.
INFO [2022-03-16 15:03:39] 30512    Will invalidate archived reports for today in site ID = 8's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:03:39] 30512    Will invalidate archived reports for yesterday in site ID = 8's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:03:48] 30512  Archived website id 8, period = day, date = 2022-03-16, segment = '', 2004 visits found. Time elapsed: 9.292s
INFO [2022-03-16 15:03:51] 30512  Archived website id 8, period = week, date = 2022-03-14, segment = '', 13809 visits found. Time elapsed: 2.840s
INFO [2022-03-16 15:03:59] 30512  Archived website id 8, period = month, date = 2022-03-01, segment = '', 58880 visits found. Time elapsed: 7.922s
INFO [2022-03-16 15:04:02] 30512  Archived website id 8, period = year, date = 2022-01-01, segment = '', 274727 visits found. Time elapsed: 2.832s
INFO [2022-03-16 15:04:02] 30512  Finished archiving for site 8, 4 API requests, Time elapsed: 23.348s [3 / 15 done]
INFO [2022-03-16 15:04:02] 30512  Start processing archives for site 10.
INFO [2022-03-16 15:04:02] 30512    Will invalidate archived reports for today in site ID = 10's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:04:02] 30512    Will invalidate archived reports for yesterday in site ID = 10's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:04:03] 30512  Archived website id 10, period = day, date = 2022-03-16, segment = '', 9 visits found. Time elapsed: 0.587s
INFO [2022-03-16 15:04:04] 30512  Archived website id 10, period = week, date = 2022-03-14, segment = '', 61 visits found. Time elapsed: 0.885s
INFO [2022-03-16 15:04:05] 30512  Archived website id 10, period = month, date = 2022-03-01, segment = '', 673 visits found. Time elapsed: 0.884s
INFO [2022-03-16 15:04:06] 30512  Archived website id 10, period = year, date = 2022-01-01, segment = '', 5231 visits found. Time elapsed: 0.910s
INFO [2022-03-16 15:04:06] 30512  Finished archiving for site 10, 4 API requests, Time elapsed: 3.793s [4 / 15 done]
INFO [2022-03-16 15:04:06] 30512  Start processing archives for site 12.
INFO [2022-03-16 15:04:06] 30512  Finished archiving for site 12, 0 API requests, Time elapsed: 0.006s [5 / 15 done]
INFO [2022-03-16 15:04:06] 30512  Start processing archives for site 14.
INFO [2022-03-16 15:04:06] 30512  Finished archiving for site 14, 0 API requests, Time elapsed: 0.007s [6 / 15 done]
INFO [2022-03-16 15:04:06] 30512  Start processing archives for site 16.
INFO [2022-03-16 15:04:06] 30512    Will invalidate archived reports for today in site ID = 16's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:04:06] 30512    Will invalidate archived reports for yesterday in site ID = 16's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:04:10] 30512  Archived website id 16, period = day, date = 2022-03-16, segment = '', 1961 visits found. Time elapsed: 3.674s
INFO [2022-03-16 15:04:11] 30512  Archived website id 16, period = week, date = 2022-03-14, segment = '', 10658 visits found. Time elapsed: 1.439s
INFO [2022-03-16 15:04:14] 30512  Archived website id 16, period = month, date = 2022-03-01, segment = '', 38690 visits found. Time elapsed: 2.831s
INFO [2022-03-16 15:04:17] 30512  Archived website id 16, period = year, date = 2022-01-01, segment = '', 208542 visits found. Time elapsed: 2.294s
INFO [2022-03-16 15:04:17] 30512  Finished archiving for site 16, 4 API requests, Time elapsed: 10.679s [7 / 15 done]
INFO [2022-03-16 15:04:17] 30512  Start processing archives for site 18.
INFO [2022-03-16 15:04:17] 30512    Will invalidate archived reports for today in site ID = 18's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:04:17] 30512    Will invalidate archived reports for yesterday in site ID = 18's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:04:44] 30512  Archived website id 18, period = day, date = 2022-03-16, segment = '', 12193 visits found. Time elapsed: 26.739s
INFO [2022-03-16 15:04:44] 30512  'PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/core/Common.php on line 323
PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 24576 bytes) in /var/www/educacion/cau_ce/estadisticasweb/vendor/composer/ClassLoader.php on line 478
'
INFO [2022-03-16 15:04:48] 30512  Archived website id 18, period = week, date = 2022-03-14, segment = '', 87784 visits found. Time elapsed: 4.001s
INFO [2022-03-16 15:05:01] 30512  Archived website id 18, period = month, date = 2022-03-01, segment = '', 427554 visits found. Time elapsed: 13.252s
INFO [2022-03-16 15:05:09] 30512  Archived website id 18, period = year, date = 2022-01-01, segment = '', 2249883 visits found. Time elapsed: 7.169s
INFO [2022-03-16 15:05:09] 30512  Finished archiving for site 18, 5 API requests, Time elapsed: 51.878s [8 / 15 done]
INFO [2022-03-16 15:05:09] 30512  Start processing archives for site 20.
INFO [2022-03-16 15:05:09] 30512    Will invalidate archived reports for today in site ID = 20's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:05:09] 30512    Will invalidate archived reports for yesterday in site ID = 20's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:05:10] 30512  Archived website id 20, period = day, date = 2022-03-16, segment = '', 389 visits found. Time elapsed: 0.866s
INFO [2022-03-16 15:05:11] 30512  Archived website id 20, period = week, date = 2022-03-14, segment = '', 2016 visits found. Time elapsed: 0.878s
INFO [2022-03-16 15:05:13] 30512  Archived website id 20, period = month, date = 2022-03-01, segment = '', 8868 visits found. Time elapsed: 1.782s
INFO [2022-03-16 15:05:14] 30512  Archived website id 20, period = year, date = 2022-01-01, segment = '', 42695 visits found. Time elapsed: 1.469s
INFO [2022-03-16 15:05:14] 30512  Finished archiving for site 20, 4 API requests, Time elapsed: 5.468s [9 / 15 done]
INFO [2022-03-16 15:05:14] 30512  Start processing archives for site 30.
INFO [2022-03-16 15:05:14] 30512    Will invalidate archived reports for today in site ID = 30's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:05:15] 30512    Will invalidate archived reports for yesterday in site ID = 30's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:06:28] 30512  Archived website id 30, period = day, date = 2022-03-16, segment = '', 37120 visits found. Time elapsed: 73.060s
INFO [2022-03-16 15:06:28] 30512  'PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/core/Common.php on line 323
PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/plugins/Monolog/Processor/ClassNameProcessor.php on line 67
'
INFO [2022-03-16 15:06:34] 30512  Archived website id 30, period = week, date = 2022-03-14, segment = '', 201676 visits found. Time elapsed: 6.305s
INFO [2022-03-16 15:06:56] 30512  Archived website id 30, period = month, date = 2022-03-01, segment = '', 846015 visits found. Time elapsed: 21.886s
INFO [2022-03-16 15:07:06] 30512  Archived website id 30, period = year, date = 2022-01-01, segment = '', 4391346 visits found. Time elapsed: 9.894s
INFO [2022-03-16 15:07:06] 30512  Finished archiving for site 30, 5 API requests, Time elapsed: 111.771s [10 / 15 done]
INFO [2022-03-16 15:07:06] 30512  Start processing archives for site 35.
INFO [2022-03-16 15:07:06] 30512    Will invalidate archived reports for today in site ID = 35's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:07:06] 30512    Will invalidate archived reports for yesterday in site ID = 35's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:07:46] 30512  Archived website id 35, period = day, date = 2022-03-16, segment = '', 28218 visits found. Time elapsed: 39.109s
INFO [2022-03-16 15:07:49] 30512  Archived website id 35, period = week, date = 2022-03-14, segment = '', 106404 visits found. Time elapsed: 3.385s
INFO [2022-03-16 15:07:58] 30512  Archived website id 35, period = month, date = 2022-03-01, segment = '', 345408 visits found. Time elapsed: 8.298s
INFO [2022-03-16 15:08:01] 30512  Archived website id 35, period = year, date = 2022-01-01, segment = '', 1719421 visits found. Time elapsed: 3.194s
INFO [2022-03-16 15:08:01] 30512  Finished archiving for site 35, 4 API requests, Time elapsed: 54.475s [11 / 15 done]
INFO [2022-03-16 15:08:01] 30512  Start processing archives for site 36.
INFO [2022-03-16 15:08:01] 30512  Finished archiving for site 36, 0 API requests, Time elapsed: 0.012s [12 / 15 done]
INFO [2022-03-16 15:08:01] 30512  Start processing archives for site 37.
INFO [2022-03-16 15:08:01] 30512  Finished archiving for site 37, 0 API requests, Time elapsed: 0.008s [13 / 15 done]
INFO [2022-03-16 15:08:01] 30512  Start processing archives for site 38.
INFO [2022-03-16 15:08:01] 30512  Finished archiving for site 38, 0 API requests, Time elapsed: 0.006s [14 / 15 done]
INFO [2022-03-16 15:08:01] 30512  Start processing archives for site 39.
INFO [2022-03-16 15:08:01] 30512    Will invalidate archived reports for today in site ID = 39's timezone (2022-03-16 00:00:00).
INFO [2022-03-16 15:08:01] 30512    Will invalidate archived reports for yesterday in site ID = 39's timezone (2022-03-15 00:00:00).
INFO [2022-03-16 15:08:02] 30512  Archived website id 39, period = day, date = 2022-03-16, segment = '', 612 visits found. Time elapsed: 0.887s
INFO [2022-03-16 15:08:03] 30512  Archived website id 39, period = week, date = 2022-03-14, segment = '', 4417 visits found. Time elapsed: 0.870s
INFO [2022-03-16 15:08:04] 30512  Archived website id 39, period = month, date = 2022-03-01, segment = '', 7914 visits found. Time elapsed: 0.882s
INFO [2022-03-16 15:08:05] 30512  Archived website id 39, period = year, date = 2022-01-01, segment = '', 7914 visits found. Time elapsed: 0.598s
INFO [2022-03-16 15:08:05] 30512  Finished archiving for site 39, 4 API requests, Time elapsed: 3.831s [15 / 15 done]
INFO [2022-03-16 15:08:05] 30512  Done archiving!
INFO [2022-03-16 15:08:05] 30512  ---------------------------
INFO [2022-03-16 15:08:05] 30512  SUMMARY
INFO [2022-03-16 15:08:05] 30512  Processed 36 archives.
INFO [2022-03-16 15:08:05] 30512  Total API requests: 38
INFO [2022-03-16 15:08:05] 30512  done: 38 req, 271350 ms, 4 errors.
INFO [2022-03-16 15:08:05] 30512  Time elapsed: 271.350s
INFO [2022-03-16 15:08:05] 30512  ---------------------------
INFO [2022-03-16 15:08:05] 30512  SCHEDULED TASKS
INFO [2022-03-16 15:08:05] 30512  Starting Scheduled tasks... 
INFO [2022-03-16 15:08:05] 30512  Scheduler: executing task Piwik\Plugins\CustomJsTracker\Tasks.updateTracker...
INFO [2022-03-16 15:08:05] 30512  Scheduler: finished. Time elapsed: 0.016s
INFO [2022-03-16 15:08:05] 30512  Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.deleteLogData...
INFO [2022-03-16 15:08:05] 30512  Scheduler: finished. Time elapsed: 0.001s
INFO [2022-03-16 15:08:05] 30512  Scheduler: executing task Piwik\Plugins\PrivacyManager\Tasks.anonymizePastData...
INFO [2022-03-16 15:08:05] 30512  Scheduler: finished. Time elapsed: 0.001s
INFO [2022-03-16 15:08:05] 30512  done
INFO [2022-03-16 15:08:05] 30512  ---------------------------
INFO [2022-03-16 15:08:05] 30512  ---------------------------
INFO [2022-03-16 15:08:05] 30512  SUMMARY OF ERRORS
INFO [2022-03-16 15:08:05] 30512  Error: Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=18&period=year&date=2021-01-01&format=json&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/core/Common.php on line 323 PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 24576 bytes) in /var/www/educacion/cau_ce/estadisticasweb/vendor/composer/ClassLoader.php on line 478 '
INFO [2022-03-16 15:08:05] 30512  Error: Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=18&period=year&date=2021-01-01&format=json&trigger=archivephp: 'PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/core/Common.php on line 323 PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 24576 bytes) in /var/www/educacion/cau_ce/estadisticasweb/vendor/composer/ClassLoader.php on line 478 '
INFO [2022-03-16 15:08:05] 30512  Error: Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=30&period=year&date=2021-01-01&format=json&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/core/Common.php on line 323 PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/plugins/Monolog/Processor/ClassNameProcessor.php on line 67 '
INFO [2022-03-16 15:08:05] 30512  Error: Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=30&period=year&date=2021-01-01&format=json&trigger=archivephp: 'PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/core/Common.php on line 323 PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/educacion/cau_ce/estadisticasweb/plugins/Monolog/Processor/ClassNameProcessor.php on line 67 '

Thanks in advance.

heurteph-ei · March 17, 2022, 8:58am

Try maybe first to invalidate periods (weeks, months) where you don’t have report data.
I think that months data are needed for year reports, and probably if they are missing Matomo would use daily data that could be a huge volume of data…

ehermon · March 17, 2022, 9:14am

Thanks,

And how I launch invalidate reports for periods and how can i pre-proceess those idsites again?

heurteph-ei · March 17, 2022, 9:20am

Have a look there:
https://matomo.org/faq/how-to/faq_155/