Achive.php memory usage trouble - SOLVED! -

geroz · March 12, 2012, 12:25pm

Hi guys,

I’m in serious trouble here. I’m trying to set up the archive.php cron job for Piwik 1.7.1. I think I’ve got all I need - except one thing: Enough memory.

Anyways: I’ve got the archive.php running either via php-cli or as an http request including the admin token_auth. In both cases, I run into memory trouble.

The strange thing is: I have a php.ini set up in ~/piwik/ - if I place a tiny phpinfo.php next to it with “<?php phpinfo(); ?>”, I see that the php.ini is parsed, and the “memory_limit=512M” is actually honored. The same is true when running “php -i | grep memory_limit” at the console - I have the 512M.

If I lower this to an insanely low value, I can even make archive.php crash. But if I leave it at the standard value, or if I give it above mentioned 512M, when calling it via http, it reports that it is using 5.4M of memory - nothing serious yet.

However, the small sub-calls fail. As I understand the PHP code, even when run via php-cli, the sub-calls are placed via http (as far as I understand the “$response = Piwik_Http::sendHttpRequestBy(‘curl’ …” in line 486).

Anyways … The request


index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=year&date=last52&format=php&token_auth=****&trigger=archivephp

ie. the last year’s archive calculation fails:


Fatal error: Out of memory (allocated 102236160) (tried to allocate 4096 bytes)

Well – 102236160 is far below 512M, it’s more like 97.5M. headscratch Interestingly enough, my provider’s default value is 70M, so this isn’t the default value either.

I’ve tried a lot.

I noticed that the “memory_limit=…” doesn’t seem to be inherited properly by the sub folders, so if I run my phpinfo.php or my “php -i” from somewhere else except ~/piwik, I get the above mentioned 70M memory limit. So I added a copy of the php.ini with the 512M limit into each and every sub directory - still nothing, still stuck at 97.5M.

What else can I do? I’m open to any kind of suggestion.

geroz · March 13, 2012, 4:34pm

Nobody? :-o Or is this just too stupid a question that has already been answered like a dozen times? … Thanks a lot in advance.

matthieu · March 14, 2012, 1:24am

Umh, it sounds quite strange. My first guess would be: ask your webhost directly to put the limit 512M which should be enough. If it does not work and you can do it, change webhost for something more serious

geroz · March 14, 2012, 9:46am

Well - it’s a dedicated managed server, I’m pretty much satisfied with the services my provider delivers, but this one’s a bummer.

For the records and perhaps for the Google search: I just created a 10 liner stress test to acquire memory and see the error message.

Interestingly enough: When I leave it at the 90M standard value, I get


Fatal error: Allowed memory size of 94371840 bytes exhausted (tried to allocate 35 bytes) in ...

94371840 / 1024 / 1024 = 90M.

When I push it up to 128M, I can acknowledge this with ini_get(“memory_limit”), but I get


Fatal error: Out of memory (allocated 112197632) (tried to allocate 131072 bytes) in ...

112197632 / 1024 / 1024 = 107M - not really the 128M I wanted.

But as I see from the error message, that’s obviously a totally different thing: It’s not an exception for trying to allocate memory that is forbidden - it’s an exception for trying to allocate memory that is not there.

… With php-cli, I can push it up to


Fatal error:  Out of memory (allocated 241958912) (tried to allocate 131072 bytes) in ...

241958912 / 1024 / 1024 = 230,75M - still not exactly 256M.

All I can say from my shell access is that the server is pretty much idle, but a whole lot of memory is used. I’ll have to ask support what is going on here.

geroz · March 14, 2012, 10:37am

OK, and one more for the records, and for Google:

If you are using a managed server at 1&1 (in this case: 1&1 Germany, but as far as I know, the 1&1 USA servers are pretty similar): You might want to have a look at “Server-Dienste” - formerly known as “Performance Server”. In there you will find settings concerning the Apache performance, including max running time and max memory usage.

So - if you modify your PHP settings for max_execution_time and memory_limit via php.ini, that’s really a nice try, and phpinfo() will as well tell you that you’ve set these values. But you will never be able to surpass the maximum settings you’ve selected in the “Server-Dienste” / “Performance Server” panel.

Which, obviously, results in the above mentioned behaviour, namely that you won’t get an “Allowed memory size” exception, but an “Out of memory” instead. Your PHP process believes to be on the safe side, as there should be more memory, but in fact there isn’t.

So - if you are planning to run archive.php and raise your memory_limit, make sure that you’ve raised the bar to a sufficiently high value in this core configuration over in the server’s Control Panel. … Same goes for the max_execution_time.

Don’t worry: As the default php.ini values are limiting this, you shouldn’t run into trouble with rogue scripts. But afterwards you actually have a chance to request more memory for specific use scenarios.

… Phew, that’s really a relief!