Is our roll-up too big? Archiving hits the PHP memory_limit (1G), site is not high-traffic, is this optimizable?

Premium Features
1

Our goal, for clarity - Keep the roll-up, optimize so we don’t need >1G of memory to archive it, if at all possible.

I’m wondering if memory_limit 1G+ is to be expected for this kind of configuration/traffic, or if there’s something we could be doing better? We are looking into moving to more powerful hosting that allows us to push the memory_limit above 1G, but want to make sure it’s necessary first.

The problem - Our archiving script runs as a cron job once every hour (UI triggering disabled) and it errors out pretty much every time due to hitting the 1G PHP memory_limit. (According to logs it is definitely using 1G, not loading the wrong config file or something like that.) Our hosting doesn’t allow us to increase above 1G.

Matomo config/traffic info - Our Matomo instance isn’t particularly “high-traffic” (We get ~ 70k-80k pageviews per month). BUT, archiving always fails on the same site, a “roll-up” of 17 sites that contribute nearly all of our traffic. (We track 65 sites total, many of them dev sites, those 17 contribute ~95% of those ~70k-80k pageviews.) And the same period, “year.” No other sites/periods have memory issues archiving w/ memory_limit=1G.

Other factors that may be contributing:

  • We have tons of custom reports, this roll-up has 19. (10 available to all sites, 9 specifically for the roll-up.) Many use custom variables.

  • We’ve been running Matomo 1.5 yrs now & our DB has 45.8G of data. This seems odd, according to this Matomo article we should be well under that w/ <2 mill pageviews total, <1G??

    • We have one “year” table that’s taking up 40G of those 45.8G alone! It’s the 2020 table. May be a similar issue to this post?

  • Many of our URLs are searches and we don’t currently ignore any search params. (Our “Pageviews - URLs” report for the roll-up is huuuge) Would this make a difference?

Any ideas? Things we can investigate, look into, etc., am I onto something with the bullets above? Thanks in advance!

2

Hi Mikky,

To confirm you have the PHP memory at 1G? We would recommend this to be at least 4G. Can you try this and let us know if the issue remains?

Thanks,

3

Hi Jason, thanks for responding!

Yes, confirming, we do have the PHP memory limit (memory_limit) set to 1G. The error messages confirm that we are able to allocate ~1G but not more:

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes)

I often run the command with php -d memory_limit=1G at the beginning for good measure, fails the same way there too.

We would recommend this to be at least 4G. Can you try this and let us know if the issue remains?

Thank you for this recommendation, this was what we needed to know. Unfortunately on our current hosting we are unable to go above 1G, so it sounds like I need to push for hosting with a a higher limit. (We are in the process of moving hosts so now’s my chance to push.) Do we just have enough traffic that we need the 4G limit instead? Or is 1G just too small for Matomo archiving in general?

Thanks again, I appreciate your insights!

4

In general 1GB is to small, judging by the error you have also. More is always better, should you grow you will have the capacity to accommodate the extra memory requirements.

5

Happy to report that I was able to try memory_limit=4G on our new hosting, and the archiving script works great now! :slight_smile: No errors since. (Knock on wood :wink: )

It was super helpful to know that 1G is too small “in general,” not just “a sign something isn’t configured right,” if that makes sense.

Thanks again for your guidance!