Memory exhausted (?) during Archive

Support & Bugs
41

As an update: With v2.6.1 the problem persists. Even with --exclude-path, the archiving process always terminates with a “memory exhausted” error when archiving month and year.

I would like to repeat my earlier request in this thread: “I would urge you to modify the archiving process in a way that is not so resource intensive and can smartly process whatever number of visits/pageviews with the resources available on the host, e.g. at the expense of time to be consumed. Alternatively, you could support two modes of operation: 1/current: optimize performance 2/suggested: minimize system resources usage.”

Can I hope/expect such a change in an upcoming version?

Thanks,
Nick

42

Just another idea

If you have RLimitMEM in /usr/local/apache/conf/httpd.conf set, it might cause memmory issue. You can search to see if that’s in the file:

Code:

grep -i rlimit /usr/local/apache/conf/httpd.conf

If it is in the file, try commenting it out (using # in front of RLimitMEM line), then restart Apache to see if it helps.

43

Dear lesjokolat,

I remind you that we are not using apache.

Best regards,
Nick

44

I have faced the following problem: When I use the above technique, i.e.:


cp /var/webs/wwwgein2/log/access_log.1 /var/webs/wwwgein2/log/access_log.sedtmp

sed -i "s,/services/GPSData/[a-zA-Z0-9\/\.\-\_]*\s,/services/GPSData/ ,g" /var/webs/wwwgein2/log/access_log.sedtmp

before importing and archiving, the pageviews report is displaying the double number than the actual!

See the attached image [attachment 1905 piwik_double_visits.png] with red arrows marking when we started and when we stopped using the above procedure.

Note: I confirmed that the above sed command correctly replaces strings. For example, the following original lines:


2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/vlsm2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/kasi2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/kipo2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/klok2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/lemn2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/nvrk2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/pont2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/prkv2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/pylo2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/2005/264/span2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:48 +0300] "GET /services/GPSData/2005/264/stef2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:48 +0300] "GET /services/GPSData/2005/264/atal2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:48 +0300] "GET /services/GPSData/2005/264/voli2640.05d.Z HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"

become:


2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:45 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:48 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:48 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"
2001:648:2000:6e:226:b9ff:fe4f:8747 - - [14/Sep/2014:04:09:48 +0300] "GET /services/GPSData/ HTTP/1.1" 404 162 "-" "Wget/1.13.4 (linux-gnu)" "-"

Can you please explain this phenomenon and advise on its solution?

(I have tried finding some cause or an error of mine, but I haven’t found any.)

Thanks,
Nick

45

I have faced the following problem: When I use the above technique i.e. copying the access_log file to a temp one and running sed before importing and archiving, the pageviews report is displaying the double number than the actual!

See the following image with red arrows marking when we started and when we stopped using the above procedure.

[attachment 1907 piwik_double_visits.png]

Note: I confirmed that the above sed command correctly replaces strings.

Can you please explain this phenomenon and advise on its solution?

I have tried finding some cause or an error of mine, but I haven’t been able to find any.

Thanks,
Nick