Archive allowed memory size

This is driving me nuts. I keep getting:

ERROR: Got invalid response from API request: piwik/index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=year&date=last52&format=php&token_auth=.....&trigger=archivephp. Response was '<br />
<b>Fatal error</b>:  Allowed memory size of 805306368 bytes exhausted (tried to allocate 8208 bytes) in <b> public_html/piwik/core/DataTable.php</b> on line <b>1071</b><br />

I tried to set the memroy_limit in php.ini to 512M and I’m running the cron with memory_limit=8196M but it still doesn’t work.

I also tried to set the memory limit in index.php but that didn’t help either.

I’m running Piwik 1.8.4

OK, I think I solved it by setting the memory_limit to 2G in index.php:
ini_set(‘memory_limit’, ‘2G’);

Ok, not solved at all :frowning:

Now I get an error:

ERROR: Got invalid response from API request: .../piwik/index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=month&date=last52&format=php&token_auth=.....&trigger=archivephp. Response was '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title>
 </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p> <p>More information about this error may be available in the server error log.</p> <p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p>

Ok, for google:

It’s solved now.

Even though there was no memory messages I set the limit to 4G:

ini_set('memory_limit', '4G'); 

After that I ran the script about 20 times (maybe even 40). Each time I got the error 500 but finally the script went through and completed without an error. Maybe the script got closer on each run and didn’t have to start from 0.

Anyway, I’m glad to have it working again but 4G really seems a lot to process some data…

Even though I’m the only one talking here, I’ll give it an update

It stopped working again and even with trying to run it the whole day it just fails. The strange part is that there are aprox. 1M rows that move from piwik_archive_blob_x to piwik_archive_blob_y where x is the first month in our database and y is the current (so the last). So there seems to be some bug in there somewhere.

The strange part is, that there are no records for month x (2012_01) but it still gets filled.

Also how is it possible that the archiver needs 4G ram and the piwik_log tables are only 500MB?

How many visits/page views do you get per day? it can be normal to require 4G or more for very high traffic sites.

The january table contains the yearly reports that’s why it can be big.

We have about 10k pageviews per day so it’s not high traffic.

I did notice the january table has stats for the whole year, but then it is still strange that the january table sometimes have 1M rows records extra and after another run these are removed and then the september table has 1M records extra.

I still can’t get it to work though.

I haven’t really looked into the whole archiving process but why does it require so much memory? Do you store everything in arrays?