Memory usage issues with archiving high-traffic eCommerce site

We’ve migrated from GA to Matomo with end of May and are experiencing issues with the archiving of the “yearly” period for our two largest sites.

PHP has been configured to not apply memory limits. After processing daily, weekly and monthly periods, the archiving process starts with the yearly period and quickly - in a matter of seconds - eats up all the RAM of the processing host. In our case that is currently 15GB. Before we increase the RAM of the host, we’d like to understand the root cause (and if e.g. doubling it would actually help with the problem).

That’s the trace of the offending command:

# /usr/bin/php7.3 -q  /var/www/matomo/console climulti:request -q --matomo-domain='localhost' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=year&date=2021-01-01&format=json&trigger=archivephp&pid=08a5b67142d4621130bbcb821706bf9b4a8501b9570d12a6040db72a59b2ab35ae1421042bbdd1a4187ea36ac48c6e30874c0&runid=27897'

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory
{"result":"error","message":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378) #0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name="Goals_ItemsSku") #1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() #2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() ","backtrace":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378)\n#0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name=\"Goals_ItemsSku\")\n#1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n#2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n\n#0 [internal function]: Piwik\\Plugins\\CorePluginsAdmin\\Controller->safemode(Array)\n#1 \/var\/www\/dbuser\/core\/FrontController.php(615): call_user_func_array(Array, Array)\n#2 \/var\/www\/dbuser\/core\/FrontController.php(167): Piwik\\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/var\/www\/dbuser\/core\/FrontController.php(98): Piwik\\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/var\/www\/dbuser\/core\/FrontController.php(271): Piwik\\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\\FrontController::triggerSafeModeWhenError()\n#6 {main}"}{"result":"error","message":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378) #0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name="Goals_ItemsSku") #1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() #2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports() ","backtrace":"Out of memory (allocated 15124066304) (tried to allocate 9134060 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(378)\n#0 \/var\/www\/matomo\/core\/ArchiveProcessor.php(213): Piwik\\ArchiveProcessor->aggregateDataTableRecord(name=\"Goals_ItemsSku\")\n#1 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n#2 \/var\/www\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Goals\\Archiver->callAggregateMultipleReports()\n\n#0 [internal function]: Piwik\\Plugins\\CorePluginsAdmin\\Controller->safemode(Array)\n#1 \/var\/www\/dbuser\/core\/FrontController.php(615): call_user_func_array(Array, Array)\n#2 \/var\/www\/dbuser\/core\/FrontController.php(167): Piwik\\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/var\/www\/dbuser\/core\/FrontController.php(98): Piwik\\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/var\/www\/dbuser\/core\/FrontController.php(271): Piwik\\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\\FrontController::triggerSafeModeWhenError()\n#6 {main}"}

Does the reference to "Goals_ItemsSku’ hint towards an issue with E-Commerce metrics?

We are new to Matomo and it’s data model: Does this provide a hint regarding volume and memory usage of deserializing the raw data?

MariaDB [matomo]> select sum(octet_length(value)) from archive_blob_2021_05 where idsite=3 and name='Goals_ItemsSku';

+--------------------------+
| sum(octet_length(value)) |
+--------------------------+
|               2715132509 |
+--------------------------+
1 row in set (1 min 24.459 sec)

FTR: The site with ID 3 already has it’s own separate archiving process.

Hi Michael,

more RAM would definitely help. It seems to be a huge datatable for your yearly report that has to fit in RAM somehow. Also for other things in parallel, some RAM has to be allocated.

I understand that more RAM will help here. But, at the same time I am wondering whether adding RAM will scale up once we reach aggregation of more than half a month of data?

(How) can we calculate how much we’ll need on the long run?

How much RAM is reserved for the system before the Matomo archiver starts running (how much of the RAM is free to use?).
RAM usage can’t be really calculated - that depends on your given workload, activated plugins, etc. Requirements for RAM does not scale linearly with more traffic, so if you reach a certain threshold of RAM, it should run through. How big is the DB exactly? Do you use customreports, Funnels or any other (additional plugins)?

The host currently has 16G of RAM and if the archiver starts processing the yearly period for this specific high-volume ID site, the RAM gets eaten up until OOM-Killer jumps in. We do use CustomReports. Database size is currently 46.7GB.

We’ll try to increase RAM now, then. Could the CustomReports make a difference? According to the initially send backtrace is looked more like an issue with the regular eCommerce-Aggretations.

Doubled RAM and still having the issue…

We have 22GB RAM on our server that processes data hourly. Our database size is over 400GB now but mostly log data that we are working to get cleaned up.

We have around 6 million page visits a day tracked by Matomo.

We have around 3000 sites being tracked. And have a shell script setup to process 3 sites concurrently. Here is the scripts that does that and it seemed to help with processing this many sites hourly.

CONCURRENT_ARCHIVERS=3
for i in $(seq 1 $CONCURRENT_ARCHIVERS)
do
(sleep $i && /path/to-console/console core:archive --url=https://domain.com --force-periods=day & )
done

We already have hourly archiving set-up and split-up the largest site to a separate archiving host. This host has now 64GB of RAM provisioned with archiving still failing.

It really looks like the approach of the yearly aggregation fetching and uncompressing the blob data in memory in a single process without freeing up (enough) memory inbetween cannot scale.

We have put traffic on this Matomo instance with late May and the yearly archiving does not finish since then for this site. Here some debug output with specific memory logging enabled.

Those Goals seem to be eCommerce related. Do we have an option but to disable eCommerce functionality for this host?

# /usr/bin/php7.3 /var/www/matomo/console --verbose --verbose --verbose climulti:request --matomo-domain='localhost' --superuser 'module=API&method=CoreAdminHome.archiveReports&idSite=3&period=year&date=2021-01-01&format=json&trigger=archivephp&pid=bdccec1d2674c7e2b1f782a01004ff2fc6eb98458b5cc1945fb6a41a310dbef8dcd43c1c128bd644b35dd0f116168e9d90620&runid=1409' 
DEBUG [2021-07-05 14:41:29] 3666  Db::fetchAll() executing SQL: SELECT option_value, option_name FROM `option` WHERE autoload = 1
DEBUG [2021-07-05 14:41:29] 3666  Db::fetchOne() executing SQL: SELECT option_value FROM `option` WHERE option_name = ?
Memory usage: 22.9 M|Processing period 2021-05-24,2021-05-24, table archive_numeric_2021_05 ids 30012
Memory usage: 22.9 M|Processing period 2021-05-25,2021-05-25, table archive_numeric_2021_05 ids 150142
Memory usage: 23 M|Processing period 2021-05-26,2021-05-26, table archive_numeric_2021_05 ids 150147
Memory usage: 23 M|Processing period 2021-05-27,2021-05-27, table archive_numeric_2021_05 ids 150152
Memory usage: 23 M|Processing period 2021-05-28,2021-05-28, table archive_numeric_2021_05 ids 150157
Memory usage: 23.1 M|Processing period 2021-05-29,2021-05-29, table archive_numeric_2021_05 ids 150162
Memory usage: 23.1 M|Processing period 2021-05-30,2021-05-30, table archive_numeric_2021_05 ids 151081

recordname: Actions_actions, colaggop: Array
Memory usage: 24.7 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 24.9 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 25.7 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 26.6 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 27.5 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 28.4 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 29.2 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Actions_actions_url, colaggop: Array
Memory usage: 26.4 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 26.8 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 29.9 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 32.6 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 35.4 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 37.8 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 40.3 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Actions_downloads, colaggop: 
Memory usage: 28.9 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 29 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 29.2 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 29.4 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 29.5 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 29.7 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 29.9 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Actions_outlink, colaggop: 
Memory usage: 29.1 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 29.1 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 33.1 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 37.5 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
recordname: Goals_ItemsName_Cart, colaggop: 
Memory usage: 20.6 G|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30017,30017,30017,30017,30017

# lines cut due to length

Memory usage: 54.5 G|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150165,150163,150134,150132,150093,150091,40070,40071,40046,40047,40022,40023,39998,39999,39973,39974,39949,39950,39925,39926,39901,39902,39877,39878,39853,39854,39829,39830,39805,39806,39781,39782,39757,39758,39733,39734,39709,39710,39685,39686,39661,39662,39637,39638,39613,39614,39589,39590,39549,39550,39525,39526,39501,39502,39477,39478,39453,39454,39429,39430,39405,39406,39381,39382,39357,39358,39333,39334,39309,39310,39285,39286,39261,39262,39237,39238,39213,39214,39189,39190,39165,39166,39140,39141,39116,39117,39092,39093,39068,39069,39044,39045,39020,39021,38996,38997,38972,38973,38948,38949,38924,38925,38900,38901,38876,38877,38852,38853,38828,38829,38804,38805,38780,38781,38756,38757,38716,38717,38692,38693,38668,38669,38644,38645,38620,38621,38596,38597,38572,38573,38548,38549,38524,38525,38500,38501,38476,38477,38452,38453,38428,38429,38404,38405,38380,38381,38356,38357,38332,38333,38307,38308,38283,38284,38259,38260,38235,38236,38211,38212,38187,38188,38163,38164,38139,38140,38115,38116,38091,38092,38067,38068,38043,38044,38018,38019,37998,37999,37978,37979,37958,37959,37938,37939,37903,37904,37883,37884,37863,37864,37843,37844,37823,37824,37803,37804,37783,37784,37763,37764,37743,37744,37723,37724,37703,37704,37683,37684,37663,37664,37643,37644,37623,37624,37603,37604,37583,37584,37562,37563,37542,37543,37522,37523,37502,37503,37482,37483,37462,37463,37442,37443,37422,37423,37402,37403,37382,37383,37362,37363,37342,37343,37322,37323,37302,37303,37282,37283,37262,37263,37242,37243,37207,37208,37187,37188,37167,37168,37147,37148,37127,37128,37107,37108,37087,37088,37067,37068,37047,37048,37027,37028,37007,37008,36987,36988,36967,36968,36947,36948,36927,36928,36907,36908,36887,36888,36866,36867,36846,36847,36826,36827,36807,36806,36786,36787,36767,36766,36746,36747,36726,36727,36707,36706,36687,36686,36666,36667,36646,36647,36626,36627,36607,36606,36587,36586,36566,36567,36546,36547,36511,36512,36492,36491,36471,36472,36452,36451,36431,36432,36412,36411,36391,36392,36371,36372,36351,36352,36331,36332,36312,36311,36292,36291,36272,36271,36251,36252,36232,36231,36211,36212,36192,36191,36171,36170,36151,36150,36131,36130,36110,36111,36091,36090,36071,36070,36051,36050,150165,150163,150134,150132,150093,150091,40070,40071,40046,40047,40022,40023,39998,39999,39973,39974,39949,39950,39925,39926,39901,39902,39877,39878,39853,39854,39829,39830,39805,39806,39781,39782,39757,39758,39733,39734,39709,39710,39685,39686,39661,39662,39637,39638,39613,39614,39589,39590,39549,39550,39525,39526,39501,39502,39477,39478,39453,39454,39429,39430,39405,39406,39381,39382,39357,39358,39333,39334,39309,39310,39285,39286,39261,39262,39237,39238,39213,39214,39189,39190,39165,39166,39140,39141,39116,39117,39092,39093,39068,39069,39044,39045,39020,39021,38996,38997,38972,38973,38948,38949,38924,38925,38900,38901,38876,38877,38852,38853,38828,38829,38804,38805,38780,38781,38756,38757,38716,38717,38692,38693,38668,38669,38644,38645,38620,38621,38596,38597,38572,38573,38548,38549,38524,38525,38500,38501,38476,38477,38452,38453,38428,38429,38404,38405,38380,38381,38356,38357,38332,38333,38307,38308,38283,38284,38259,38260,38235,38236,38211,38212,38187,38188,38163,38164,38139,38140,38115,38116,38091,38092,38067,38068,38043,38044,38018,38019,37998,37999,37978,37979,37958,37959,37938,37939,37903,37904,37883,37884,37863,37864,37843,37844,37823,37824,37803,37804,37783,37784,37763,37764,37743,37744,37723,37724,37703,37704,37683,37684,37663,37664,37643,37644,37623,37624,37603,37604,37583,37584,37562,37563,37542,37543,37522,37523,37502,37503,37482,37483,37462,37463,37442,37443,37422,37423,37402,37403,37382,37383,37362,37363,37342,37343,37322,37323,37302,37303,37282,37283,37262,37263,37242,37243,37207,37208,37187,37188,37167,37168,37147,37148,37127,37128,37107,37108,37087,37088,37067,37068,37047,37048,37027,37028,37007,37008,36987,36988,36967,36968,36947,36948,36927,36928,36907,36908,36887,36888,36866,36867,36846,36847,36826,36827,36807,36806,36786,36787,36767,36766,36746,36747,36726,36727,36707,36706,36687,36686,36666,36667,36646,36647,36626,36627,36607,36606,36587,36586,36566,36567,36546,36547,36511,36512,36492,36491,36471,36472,36452,36451,36431,36432,36412,36411,36391,36392,36371,36372,36351,36352,36331,36332,36312,36311,36292,36291,36272,36271,36251,36252,36232,36231,36211,36212,36192,36191,36171,36170,36151,36150,36131,36130,36110,36111,36091,36090,36071,36070,36051,36050,150165,150163,150134,150132,150093,150091,40070,40071,40046,40047,40022,40023,39998,39999,39973,39974,39949,39950,39925,39926,39901,39902,39877,39878,39853,39854,39829,39830,39805,39806,39781,39782,39757,39758,39733,39734,39709,39710,39685,39686,39661,39662,39637,39638,39613,39614,39589,39590,39549,39550,39525,39526,39501,39502,39477,39478,39453,39454,39429,39430,39405,39406,39381,39382,39357,39358,39333,39334,39309,39310,39285,39286,39261,39262,39237,39238,39213,39214,39189,39190,39165,39166,39140,39141,39116,39117,39092,39093,39068,39069,39044,39045,39020,39021,38996,38997,38972,38973,38948,38949,38924,38925,38900,38901,38876,38877,38852,38853,38828,38829,38804,38805,38780,38781,38756,38757,38716,38717,38692,38693,38668,38669,38644,38645,38620,38621,38596,38597,38572,38573,38548,38549,38524,38525,38500,38501,38476,38477,38452,38453,38428,38429,38404,38405,38380,38381,38356,38357,38332,38333,38307,38308,38283,38284,38259,38260,38235,38236,38211,38212,38187,38188,38163,38164,38139,38140,38115,38116,38091,38092,38067,38068,38043,38044,38018,38019,37998,37999,37978,37979,37958,37959,37938,37939,37903,37904,37883,37884,37863,37864,37843,37844,37823,37824,37803,37804,37783,37784,37763,37764,37743,37744,37723,37724,37703,37704,37683,37684,37663,37664,37643,37644,37623,37624,37603,37604,37583,37584,37562,37563,37542,37543,37522,37523,37502,37503,37482,37483,37462,37463,37442,37443,37422,37423,37402,37403,37382,37383,37362,37363,37342,37343,37322,37323,37302,37303,37282,37283,37262,37263,37242,37243,37207,37208,37187,37188,37167,37168,37147,37148,37127,37128,37107,37108,37087,37088,37067,37068,37047,37048,37027,37028,37007,37008,36987,36988,36967,36968,36947,36948,36927,36928,36907,36908,36887,36888,36866,36867,36846,36847,36826,36827,36807,36806,36786,36787,36767,36766,36746,36747,36726,36727,36707,36706,36687,36686,36666,36667,36646,36647,36626,36627,36607,36606,36587,36586,36566,36567,36546,36547,36511,36512,36492,36491,36471,36472,36452,36451,36431,36432,36412,36411,36391,36392,36371,36372,36351,36352,36331,36332,36312,36311,36292,36291,36272,36271,36251,36252,36232,36231,36211,36212,36192,36191,36171,36170,36151,36150,36131,36130,36110,36111,36091,36090,36071,36070,36051,36050,150165,150163,150134,150132,150093,150091,40070,40071,40046,40047,40022,40023,39998,39999,39973,39974,39949,39950,39925,39926,39901,39902,39877,39878,39853,39854,39829,39830,39805,39806,39781,39782,39757,39758,39733,39734,39709,39710,39685,39686,39661,39662,39637,39638,39613,39614,39589,39590,39549,39550,39525,39526,39501,39502,39477,39478,39453,39454,39429,39430,39405,39406,39381,39382,39357,39358,39333,39334,39309,39310,39285,39286,39261,39262,39237,39238,39213,39214,39189,39190,39165,39166,39140,39141,39116,39117,39092,39093,39068,39069,39044,39045,39020,39021,38996,38997,38972,38973,38948,38949,38924,38925,38900,38901,38876,38877,38852,38853,38828,38829,38804,38805,38780,38781,38756,38757,38716,38717,38692,38693,38668,38669,38644,38645,38620,38621,38596,38597,38572,38573,38548,38549,38524,38525,38500,38501,38476,38477,38452,38453,38428,38429,38404,38405,38380,38381,38356,38357,38332,38333,38307,38308,38283,38284,38259,38260,38235,38236,38211,38212,38187,38188,38163,38164,38139,38140,38115,38116,38091,38092,38067,38068,38043,38044,38018,38019,37998,37999,37978,37979,37958,37959,37938,37939,37903,37904,37883,37884,37863,37864,37843,37844,37823,37824,37803,37804,37783,37784,37763,37764,37743,37744,37723,37724,37703,37704,37683,37684,37663,37664,37643,37644,37623,37624,37603,37604,37583,37584,37562,37563,37542,37543,37522,37523,37502,37503,37482,37483,37462,37463,37442,37443,37422,37423,37402,37403,37382,37383,37362,37363,37342,37343,37322,37323,37302,37303,37282,37283,37262,37263,37242,37243,37207,37208,37187,37188,37167,37168,37147,37148,37127,37128,37107,37108,37087,37088,37067,37068,37047,37048,37027,37028,37007,37008,36987,36988,36967,36968,36947,36948,36927,36928,36907,36908,36887,36888,36866,36867,36846,36847,36826,36827,36807,36806,36786,36787,36767,36766,36746,36747,36726,36727,36707,36706,36687,36686,36666,36667,36646,36647,36626,36627,36607,36606,36587,36586,36566,36567,36546,36547,36511,36512,36492,36491,36471,36472,36452,36451,36431,36432,36412,36411,36391,36392,36371,36372,36351,36352,36331,36332,36312,36311,36292,36291,36272,36271,36251,36252,36232,36231,36211,36212,36192,36191,36171,36170,36151,36150,36131,36130,36110,36111,36091,36090,36071,36070,36051,36050,150165,150163,150134,150132,150093,150091,40070,40071,40046,40047,40022,40023,39998,39999,39973,39974,39949,39950,39925,39926,39901,39902,39877,39878,39853,39854,39829,39830,39805,39806,39781,39782,39757,39758,39733,39734,39709,39710,39685,39686,39661,39662,39637,39638,39613,39614,39589,39590,39549,39550,39525,39526,39501,39502,39477,39478,39453,39454,39429,39430,39405,39406,39381,39382,39357,39358,39333,39334,39309,39310,39285,39286,39261,39262,39237,39238,39213,39214,39189,39190,39165,39166,39140,39141,39116,39117,39092,39093,39068,39069,39044,39045,39020,39021,38996,38997,38972,38973,38948,38949,38924,38925,38900,38901,38876,38877,38852,38853,38828,38829,38804,38805,38780,38781,38756,38757,38716,38717,38692,38693,38668,38669,38644,38645,38620,38621,38596,38597,38572,38573,38548,38549,38524,38525,38500,38501,38476,38477,38452,38453,38428,38429,38404,38405,38380,38381,38356,38357,38332,38333,38307,38308,38283,38284,38259,38260,38235,38236,38211,38212,38187,38188,38163,38164,38139,38140,38115,38116,38091,38092,38067,38068,38043,38044,38018,38019,37998,37999,37978,37979,37958,37959,37938,37939,37903,37904,37883,37884,37863,37864,37843,37844,37823,37824,37803,37804,37783,37784,37763,37764,37743,37744,37723,37724,37703,37704,37683,37684,37663,37664,37643,37644,37623,37624,37603,37604,37583,37584,37562,37563,37542,37543,37522,37523,37502,37503,37482,37483,37462,37463,37442,37443,37422,37423,37402,37403,37382,37383,37362,37363,37342,37343,37322,37323,37302,37303,37282,37283,37262,37263,37242,37243,37207,37208,37187,37188,37167,37168,37147,37148,37127,37128,37107,37108,37087,37088,37067,37068,37047,37048,37027,37028,37007,37008,36987,36988,36967,36968,36947,36948,36927,36928,36907,36908,36887,36888,36866,36867,36846,36847,36826,36827,36807,36806,36786,36787,36767,36766,36746,36747,36726,36727,36707,36706,36687,36686,36666,36667,36646,36647,36626,36627,36607,36606,36587,36586,36566,36567,36546,36547,36511,36512,36492,36491,36471,36472,36452,36451,36431,36432,36412,36411,36391,36392,36371,36372,36351,36352,36331,36332,36312,36311,36292,36291,36272,36271,36251,36252,36232,36231,36211,36212,36192,36191,36171,36170,36151,36150,36131,36130,36110,36111,36091,36090,36071,36070,36051,36050

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory
{"result":"error","message":"Out of memory (allocated 65123217408) (tried to allocate 9117744 bytes) on \/var\/www\/matomo\/core\/DataAccess\/ArchiveSelector.php(390)
…

Here the same logfile filtered and truncated for a better unterstanding at which Goal memory usage is significantly increasing:

$ cut -c 1-110 recordname matomo_console_memory_growth_gh_de_yearly_archiving.log | egrep -v '^$'
cut: recordname: No such file or directory
# /usr/bin/php7.3 /var/www/matomo/console --verbose --verbose --verbose climulti:request --matomo-domain='loca
DEBUG [2021-07-05 14:41:29] 3666  Db::fetchAll() executing SQL: SELECT option_value, option_name FROM `option`
DEBUG [2021-07-05 14:41:29] 3666  Db::fetchOne() executing SQL: SELECT option_value FROM `option` WHERE option
Memory usage: 22.9 M|Processing period 2021-05-24,2021-05-24, table archive_numeric_2021_05 ids 30012
Memory usage: 22.9 M|Processing period 2021-05-25,2021-05-25, table archive_numeric_2021_05 ids 150142
Memory usage: 23 M|Processing period 2021-05-26,2021-05-26, table archive_numeric_2021_05 ids 150147
Memory usage: 23 M|Processing period 2021-05-27,2021-05-27, table archive_numeric_2021_05 ids 150152
Memory usage: 23 M|Processing period 2021-05-28,2021-05-28, table archive_numeric_2021_05 ids 150157
Memory usage: 23.1 M|Processing period 2021-05-29,2021-05-29, table archive_numeric_2021_05 ids 150162
Memory usage: 23.1 M|Processing period 2021-05-30,2021-05-30, table archive_numeric_2021_05 ids 151081
recordname: Actions_actions, colaggop: Array
Memory usage: 24.7 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 24.9 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 25.7 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 26.6 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 27.5 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 28.4 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 29.2 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081

…

recordname: DevicesDetection_browserVersions, colaggop: 
Memory usage: 30.1 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 30.3 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 30.4 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 30.5 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 30.6 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 30.7 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 30.8 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goals_ItemsSku, colaggop: 
Memory usage: 30.4 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 30.6 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 42.3 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 56.4 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 70.2 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 83 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 95.5 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goals_ItemsName, colaggop: 
Memory usage: 67.8 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 68.1 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 87 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 109.7 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 131.8 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 152.4 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 171.8 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goals_ItemsCategory, colaggop: 
Memory usage: 139.1 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 139.4 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 139.9 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 140.3 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 140.8 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 141.2 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 141.7 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goals_ItemsSku_Cart, colaggop: 
Memory usage: 112.6 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 112.5 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 122.6 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 134.8 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 146.7 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 157.9 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 168.8 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goals_ItemsName_Cart, colaggop: 
Memory usage: 140.3 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 140.3 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 157.1 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 177.2 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 196.9 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 215.3 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 232.8 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goals_ItemsCategory_Cart, colaggop: 
Memory usage: 202.2 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 202.2 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 202.5 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 202.7 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 202.9 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 203.2 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 203.4 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
Memory usage: 175 M|Processing period 2021-05-24,2021-05-24, table archive_numeric_2021_05 ids 30012
Memory usage: 175.1 M|Processing period 2021-05-25,2021-05-25, table archive_numeric_2021_05 ids 150142
Memory usage: 175.1 M|Processing period 2021-05-26,2021-05-26, table archive_numeric_2021_05 ids 150147
Memory usage: 175.1 M|Processing period 2021-05-27,2021-05-27, table archive_numeric_2021_05 ids 150152
Memory usage: 175.2 M|Processing period 2021-05-28,2021-05-28, table archive_numeric_2021_05 ids 150157
Memory usage: 175.2 M|Processing period 2021-05-29,2021-05-29, table archive_numeric_2021_05 ids 150162
Memory usage: 175.2 M|Processing period 2021-05-30,2021-05-30, table archive_numeric_2021_05 ids 151081
recordname: Goal_0_visits_until_conv, colaggop: 
Memory usage: 175.2 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 175.2 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 175.2 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 175.1 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 175 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 174.9 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 174.8 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goal_0_days_until_conv, colaggop: 
Memory usage: 174.8 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 174.8 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 174.8 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 174.8 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 174.8 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 174.8 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 174.8 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goal_-1_visits_until_conv, colaggop: 
Memory usage: 174.8 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 174.8 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 174.8 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 174.8 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 174.7 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 174.7 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 174.7 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goal_-1_days_until_conv, colaggop: 
Memory usage: 174.7 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 174.7 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 174.6 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 174.5 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 174.5 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 174.4 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 174.3 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goal_visits_until_conv, colaggop: 
Memory usage: 174.3 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 174.3 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 174.3 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 174.3 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 174.3 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 174.3 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 174.3 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goal_days_until_conv, colaggop: 
Memory usage: 174.3 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 174.3 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 168.7 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 162 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 155.4 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 149.3 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 143.5 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goal_visits_until_conv, colaggop: 
Memory usage: 137.3 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 137.3 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 137.3 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 137.3 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 137.3 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 137.3 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 137.3 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goal_days_until_conv, colaggop: 
Memory usage: 137.3 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30012
Memory usage: 137.3 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150142
Memory usage: 137.3 M|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150147
Memory usage: 137.3 M|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150152
Memory usage: 137.3 M|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150157
Memory usage: 137.3 M|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150162
Memory usage: 137.3 M|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151081
recordname: Goals_ItemsSku, colaggop: 
Memory usage: 139.5 M|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30017
Memory usage: 139.7 M|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150483,150145,15
Memory usage: 9.2 G|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150150,150148,1501
Memory usage: 15.3 G|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150155,150153,150
Memory usage: 20.5 G|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150160,150158,150
Memory usage: 24.3 G|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150165,150163,150
Memory usage: 26.3 G|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151082
recordname: Goals_ItemsName, colaggop: 
Memory usage: 4.9 G|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30017,30017
Memory usage: 4.9 G|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150483,150145,1501
Memory usage: 19.1 G|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150150,150148,150
Memory usage: 28.8 G|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150155,150153,150
Memory usage: 36.8 G|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150160,150158,150
Memory usage: 42.7 G|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150165,150163,150
Memory usage: 45.9 G|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151082,151082
recordname: Goals_ItemsCategory, colaggop: 
Memory usage: 16.4 G|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30017,30017,30017
Memory usage: 16.4 G|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150483,150145,150
Memory usage: 16.7 G|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150150,150148,150
Memory usage: 17 G|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150155,150153,15012
Memory usage: 17.2 G|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150160,150158,150
Memory usage: 17.4 G|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150165,150163,150
Memory usage: 17.5 G|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151082,151082,151
recordname: Goals_ItemsSku_Cart, colaggop: 
Memory usage: 16.6 G|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30017,30017,30017
Memory usage: 16.6 G|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150483,150145,150
Memory usage: 24.7 G|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150150,150148,150
Memory usage: 30.1 G|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150155,150153,150
Memory usage: 34.7 G|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150160,150158,150
Memory usage: 38.2 G|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150165,150163,150
Memory usage: 40.1 G|Processing period 2021-05-30,2021-05-30, table archive_blob_2021_05 ids 151082,151082,151
recordname: Goals_ItemsName_Cart, colaggop: 
Memory usage: 20.6 G|Processing period 2021-05-24,2021-05-24, table archive_blob_2021_05 ids 30017,30017,30017
Memory usage: 20.6 G|Processing period 2021-05-25,2021-05-25, table archive_blob_2021_05 ids 150483,150145,150
Memory usage: 33.2 G|Processing period 2021-05-26,2021-05-26, table archive_blob_2021_05 ids 150150,150148,150
Memory usage: 41.9 G|Processing period 2021-05-27,2021-05-27, table archive_blob_2021_05 ids 150155,150153,150
Memory usage: 49.1 G|Processing period 2021-05-28,2021-05-28, table archive_blob_2021_05 ids 150160,150158,150
Memory usage: 54.5 G|Processing period 2021-05-29,2021-05-29, table archive_blob_2021_05 ids 150165,150163,150
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
{"result":"error","message":"Out of memory (allocated 65123217408) (tried to allocate 9117744 bytes) on \/var\

This looks weird - 54.5G to archive eCommerce data is really strange with this database size (however, blobs are compressed).
Either, the GA import went wrong somehow and introduced e.g. a memory leak, or you’re using the eCommerce functionality very intensively. I suspect the first, because you have so many Goals_ItemsSku - Can you post a screenshot from your Visitor Log (with anonymized data)?

Do you mean sample entries from the “Visits Log” or something else?

Right, just to see what gets logged for the visitors and (ecommerce) customers. Perhaps there is heaps of data logged, or just invalid stuff was imported from GA

I think we can rule out the GA-import as the last day of imported data was May 26th and the archiving memory usage increase starts with May 27th.

Here a sample visit log from May 27th:

From that it looks like the tracking of things like SKU data (and product name) per product page impression (see arrow towards the hover popup) seems to be a size issue as we have hundreds of thousands of those in a comparatively short amount of time of those.