Error Archiving Reports - Impact on Stats?

I am running a higher traffic site and in times where we have more traffic, we get errors when archiving reports. The error goes away sometimes but comes again once there is a bit more traffic again. We can’t raise the memory_limit without a server upgrade.

Interestingly, the dashboard looks OK, it shows users, events etc.

So I am worried about the impact this actually has, are the statistics incomplete? What is the expected impact of this error message?

INFO [2022-05-09 07:30:02] 15650  ---------------------------
INFO [2022-05-09 07:30:02] 15650  INIT
INFO [2022-05-09 07:30:02] 15650  Running Matomo 4.9.1 as Super User
INFO [2022-05-09 07:30:02] 15650  ---------------------------
INFO [2022-05-09 07:30:02] 15650  NOTES
INFO [2022-05-09 07:30:02] 15650  - Async process archiving supported, using CliMulti.
INFO [2022-05-09 07:30:02] 15650  - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO [2022-05-09 07:30:02] 15650  - Archiving was last executed without error 11 hours 58 min ago.
INFO [2022-05-09 07:30:02] 15650  ---------------------------
INFO [2022-05-09 07:30:02] 15650  START
INFO [2022-05-09 07:30:02] 15650  Starting Matomo reports archiving...
DEBUG [2022-05-09 07:30:02] 15650  Applying queued rearchiving...
INFO [2022-05-09 07:30:02] 15650  Start processing archives for site 1.
DEBUG [2022-05-09 07:30:02] 15650  Checking for queued invalidations...
INFO [2022-05-09 07:30:02] 15650    Will invalidate archived reports for today in site ID = 1's timezone (2022-05-09 00:00:00).
INFO [2022-05-09 07:30:02] 15650    Will invalidate archived reports for yesterday in site ID = 1's timezone (2022-05-08 00:00:00).
DEBUG [2022-05-09 07:30:02] 15650  Done invalidating
DEBUG [2022-05-09 07:30:02] 15650  No usable archive exists (ts_archived of existing = 2022-05-09 07:00:16, now = 2022-05-09 07:30:02).
DEBUG [2022-05-09 07:30:02] 15650  Processing invalidation: [idinvalidation = 13602, idsite = 1, period = day(2022-05-09 - 2022-05-09), name = done, segment = ].
DEBUG [2022-05-09 07:30:02] 15650  Found duplicate invalidated archive (same archive currently in progress), ignoring: [idinvalidation = 13603, idsite = 1, period = week(2022-05-09 - 2022-05-15), name = done, segment = ]
DEBUG [2022-05-09 07:30:02] 15650  No usable archive exists (ts_archived of existing = , now = 2022-05-09 07:30:02).
DEBUG [2022-05-09 07:30:02] 15650  Processing invalidation: [idinvalidation = 13431, idsite = 1, period = day(2022-05-08 - 2022-05-08), name = done, segment = ].
DEBUG [2022-05-09 07:30:02] 15650  Found duplicate invalidated archive (same archive currently in progress), ignoring: [idinvalidation = 13606, idsite = 1, period = week(2022-05-02 - 2022-05-08), name = done, segment = ]
DEBUG [2022-05-09 07:30:02] 15650  Found duplicate invalidated archive (same archive currently in progress), ignoring: [idinvalidation = 13604, idsite = 1, period = month(2022-05-01 - 2022-05-31), name = done, segment = ]
DEBUG [2022-05-09 07:30:02] 15650  Found duplicate invalidated archive (same archive currently in progress), ignoring: [idinvalidation = 13605, idsite = 1, period = year(2022-01-01 - 2022-12-31), name = done, segment = ]
DEBUG [2022-05-09 07:30:02] 15650  No next invalidated archive.
DEBUG [2022-05-09 07:30:02] 15650  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2022-05-09&format=json&trigger=archivephp
DEBUG [2022-05-09 07:30:02] 15650  Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2022-05-08&format=json&trigger=archivephp
DEBUG [2022-05-09 07:30:02] 15650  Running command: /usr/bin/php74.wrp.cli -q  /usr/www/users/mydomain/mydomain_matomo/console climulti:request -q --matomo-domain='matomo.mydomain.de' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2022-05-09&format=json&trigger=archivephp&pid=50f860469f54d6d11fc0b79dd3484515b8bbc3cbbeb0f88f8be9044f4ffbb486665b641b45a82be3f19a1980d55329be98c30&runid=15650'  > /usr/www/users/mydomain/mydomain_matomo/tmp/climulti/50f860469f54d6d11fc0b79dd3484515b8bbc3cbbeb0f88f8be9044f4ffbb486665b641b45a82be3f19a1980d55329be98c30.output 2>&1 &
DEBUG [2022-05-09 07:30:02] 15650  Running command: /usr/bin/php74.wrp.cli -q  /usr/www/users/mydomain/mydomain_matomo/console climulti:request -q --matomo-domain='matomo.mydomain.de' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2022-05-08&format=json&trigger=archivephp&pid=583aad9f0e9da14b28bd6cd94b6cdb35236aced55b5e138e7248057f88ee30121da75a71e62c209c164e87b4dcac0800265c1&runid=15650'  > /usr/www/users/mydomain/mydomain_matomo/tmp/climulti/583aad9f0e9da14b28bd6cd94b6cdb35236aced55b5e138e7248057f88ee30121da75a71e62c209c164e87b4dcac0800265c1.output 2>&1 &
INFO [2022-05-09 07:36:32] 15650  Archived website id 1, period = day, date = 2022-05-09, segment = '', 2580 visits found. Time elapsed: 14.511s
DEBUG [2022-05-09 07:36:32] 15650  Found dangling invalidation, inserting {"idarchive":null,"name":"done","report":null,"idsite":"1","date1":"2022-05-09 00:00:00","date2":"2022-05-15 00:00:00","period":2,"ts_invalidated":"2022-05-09 07:30:02"}
DEBUG [2022-05-09 07:36:32] 15650  Found dangling invalidation, inserting {"idarchive":null,"name":"done","report":null,"idsite":"1","date1":"2022-05-01 00:00:00","date2":"2022-05-31 00:00:00","period":3,"ts_invalidated":"2022-05-09 07:30:02"}
DEBUG [2022-05-09 07:36:32] 15650  Found dangling invalidation, inserting {"idarchive":null,"name":"done","report":null,"idsite":"1","date1":"2022-01-01 00:00:00","date2":"2022-12-31 00:00:00","period":4,"ts_invalidated":"2022-05-09 07:30:02"}
INFO [2022-05-09 07:36:32] 15650  ''
DEBUG [2022-05-09 07:36:32] 15650  Found duplicate invalidated archive (same archive currently in progress), ignoring: [idinvalidation = 13607, idsite = 1, period = week(2022-05-09 - 2022-05-15), name = done, segment = ]
DEBUG [2022-05-09 07:36:32] 15650  Found duplicate invalidated archive (same archive currently in progress), ignoring: [idinvalidation = 13608, idsite = 1, period = month(2022-05-01 - 2022-05-31), name = done, segment = ]
DEBUG [2022-05-09 07:36:32] 15650  Found duplicate invalidated archive (same archive currently in progress), ignoring: [idinvalidation = 13609, idsite = 1, period = year(2022-01-01 - 2022-12-31), name = done, segment = ]
DEBUG [2022-05-09 07:36:32] 15650  No next invalidated archive.
INFO [2022-05-09 07:36:32] 15650  Finished archiving for site 1, 2 API requests, Time elapsed: 390.272s [1 / 1 done]
DEBUG [2022-05-09 07:36:32] 15650  No more sites left to archive, stopping.
INFO [2022-05-09 07:36:32] 15650  Done archiving!
INFO [2022-05-09 07:36:32] 15650  ---------------------------
INFO [2022-05-09 07:36:32] 15650  SUMMARY
INFO [2022-05-09 07:36:32] 15650  Processed 1 archives.
INFO [2022-05-09 07:36:32] 15650  Total API requests: 2
INFO [2022-05-09 07:36:32] 15650  done: 2 req, 390402 ms, 2 errors.
INFO [2022-05-09 07:36:32] 15650  Time elapsed: 390.402s
INFO [2022-05-09 07:36:32] 15650  ---------------------------
INFO [2022-05-09 07:36:32] 15650  SCHEDULED TASKS
DEBUG [2022-05-09 07:36:32] 15650  27 scheduled tasks loaded
INFO [2022-05-09 07:36:32] 15650  Starting Scheduled tasks... 
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 0:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 1:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 2:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 3:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 4:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 5:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 6:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 7:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 8:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 9:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 10:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 11:
DEBUG [2022-05-09 07:36:32] 15650  Executing tasks with priority 12:
INFO [2022-05-09 07:36:32] 15650  done
INFO [2022-05-09 07:36:32] 15650  ---------------------------
INFO [2022-05-09 07:36:32] 15650  ---------------------------
INFO [2022-05-09 07:36:32] 15650  SUMMARY OF ERRORS
INFO [2022-05-09 07:36:32] 15650  Error: Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2022-05-08&format=json&trigger=archivephp. The response was empty. This usually means a server error. A solution to this error is generally to increase the value of 'memory_limit' in your php.ini file.  For more information and the error message please check in your PHP CLI error log file. As this core:archive command triggers PHP processes over the CLI, you can find where PHP CLI logs are stored by running this command: php -i | grep error_log
INFO [2022-05-09 07:36:32] 15650  Error: Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2022-05-08&format=json&trigger=archivephp: ''
DEBUG [2022-05-09 07:36:32] 15650  Loaded plugins: CoreVue, CorePluginsAdmin, CoreAdminHome, CoreHome, WebsiteMeasurable, IntranetMeasurable, Diagnostics, CoreVisualizations, Proxy, API, Widgetize, Transitions, LanguagesManager, Actions, Dashboard, MultiSites, Referrers, UserLanguage, DevicesDetection, Goals, Ecommerce, SEO, Events, UserCountry, GeoIp2, VisitsSummary, VisitFrequency, VisitTime, VisitorInterest, RssWidget, Feedback, Monolog, Login, TwoFactorAuth, UsersManager, SitesManager, Installation, CoreUpdater, CoreConsole, ScheduledReports, UserCountryMap, Live, PrivacyManager, ImageGraph, Annotations, MobileMessaging, Overlay, SegmentEditor, Insights, Morpheus, Contents, BulkTracking, Resolution, DevicePlugins, Heartbeat, Intl, Marketplace, ProfessionalServices, UserId, CustomJsTracker, Tour, PagePerformance, CustomDimensions, DBStats, CustomVariables, MarketingCampaignsReporting

About memory limit, check the Apache and also the PHP limits if they can be changed… (sorry, I am not server admin, so I don’t know where exactly this must be configured).
Also have a look at:

Thanks, we have a memory limit of 4G set already, we can’t raise it further without switching our server.

We have the archiving set up correctly (I think so at least). It works usually, but fails for higher traffic days, yet the statistics seem to show correctly, so I am wondering what the impact of this error is and what this means.

if the archiving fails, the data will be inaccurate for some period/segment…
The data will ever be OK for raw reports (visit logs), but not for others (visits per day/week/month, average value for some dimension, etc.)

Which traffic (in term of page view / actions per day)?