Memory exhausted (?) during Archive

nick1200 · August 9, 2014, 9:24pm

Hello,

I am running v2.4.1 on CentOS 6.5 (fully updated) under nginx 1.6.1 with php-fpm, PHP 5.3.3-27 and mysql 5.5.34.

It is a VM under KVM. The VM has 6GB (virtual) RAM and 4 (virtual) CPUs.

I have been having this issue for about a month now: Archiving is successful for day and week but it always terminates during month/year archiving with a memory exhaustion error.

[b]Please let me know if this may be caused by some bug (e.g. memory leak) in piwik archive code or I am doing something wrong.

Is it impossible to run piwik archiving successfully on such a system (which is by no means inefficient for every other task and its web server can serve thousands of requests per second)?[/b]

Here is a typical log:


# /usr/bin/php /var/webs/wwwpiwik/www/console core:archive --url=http://wstat1.noa.gr
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] INIT
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] Piwik is installed at: http://wstat1.noa.gr/index.php
INFO CoreConsole[2014-08-09 18:58:41] [9a9a2] Running Piwik 2.4.1 as Super User
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] NOTES
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] - Reports for today will be processed at most every 21600 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-08-09 18:58:42] [9a9a2] - Archiving was last executed without error 28 days 9 hours ago
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] - Will process 1 websites with new visits since 28 days 9 hours , IDs: 1
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] START
INFO CoreConsole[2014-08-09 18:58:48] [9a9a2] Starting Piwik reports archiving...
INFO CoreConsole[2014-08-09 18:59:07] [9a9a2] Archived website id = 1, period = day, 14780 visits in last last2 days, 1492 visits today, Time elapsed: 18.157s

INFO CoreConsole[2014-08-09 20:20:34] [9a9a2] Archived website id = 1, period = week, 1706036 visits in last last30 weeks, 64467 visits this week, Time elapsed: 4887.189s
ERROR CoreConsole[2014-08-09 20:49:26] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
ERROR CoreConsole[2014-08-09 20:49:26] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 20:49:27] [9a9a2] Archived website id = 1, period = month, 0 visits in last last30 months, 0 visits this month, Time elapsed: 1732.712s
ERROR CoreConsole[2014-08-09 21:20:27] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
ERROR CoreConsole[2014-08-09 21:20:27] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 21:20:27] [9a9a2] Archived website id = 1, period = year, 0 visits in last last7 years, 0 visits this year, Time elapsed: 1860.777s
INFO CoreConsole[2014-08-09 21:20:28] [9a9a2] Archived website id = 1, 4 API requests, Time elapsed: 8498.919s [1/1 done]
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Done archiving!
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] SUMMARY
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Total visits for today across archived websites: 1492
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Archived today's reports for 1 websites
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Archived week/month/year for 1 websites
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Skipped 0 websites day archiving: existing daily reports are less than 21600 seconds old
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Total API requests: 4
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] done: 1/1 100%, 1492 vtoday, 1 wtoday, 1 wperiods, 4 req, 8501091 ms, 2 errors.
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Time elapsed: 8501.091s
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] SCHEDULED TASKS
INFO CoreConsole[2014-08-09 21:20:29] [9a9a2] Starting Scheduled tasks...
ERROR CoreConsole[2014-08-09 21:20:44] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-09 21:20:44] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.

INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] done
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] ---------------------------
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] SUMMARY OF ERRORS
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 21:20:46] [9a9a2] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-09 21:20:46] [9a9a2] 3 total errors during this script execution, please investigate and try and fix these errors.
ERROR CoreConsole[2014-08-09 21:20:46] [9a9a2] 3 total errors during this script execution, please investigate and try and fix these errors.

More technical details follow:

I have configured in php.ini a memory_limit which is already high for this system:
memory_limit = 5120M
Here is the memory footprint under normal operation (when piwik Archiving is NOT running):


# free -m
             total       used       free     shared    buffers     cached
Mem:          5850       3226       2624          0        499       1058
-/+ buffers/cache:       1667       4182
Swap:         3023          0       3023

During piwik Archive processing (week), “top” shows:


# ps axuwww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root     16753  7.6  0.6 418116 38032 pts/0    S+   21:58   1:08 /usr/bin/php /var/webs/wwwpiwik/www/console core:archive --url=http://wstat1.noa.gr
root     17213 96.7 38.2 2690984 2289900 pts/0 R+   21:59  14:01 /usr/bin/php -q /var/webs/wwwpiwik/www/console climulti:request --piwik-domain=wstat1.noa.gr module=API&method=API.get&idSite=1&period=week&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp&pid=0207c2ff9f9c0e77d44c8898be105e2572332c0bc7463180829bfed4da5cd6b92d3d8eff87a0595240329d6bc8cf865884af0
...
root      1183  0.0  0.0 108300  1584 ?        S    14:35   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql     1644 35.5 16.0 2317780 964340 ?      Sl   14:35 167:30 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
...

I noticed that during all this period, memory usage of process 17213 was steady at 38.2% and things run smoothly. Yet, after the line:


INFO CoreConsole[2014-08-09 20:20:34] [9a9a2] Archived website id = 1, period = week, 1706036 visits in last last30 weeks, 64467 visits this week, Time elapsed: 4887.189s

is printed, then for a while mysql takes over for a while, and php seems a bit idle (memory and cpu usage drops). After that, php starts again, and memory consumption grows quickly; swapping takes place (both for php and mysql) and not much later swap size reaches as high as ~1.7GB:


             total       used       free     shared    buffers     cached
Mem:          5850       5721        129          0          9         18
-/+ buffers/cache:       5693        157
Swap:         3023       1687       1336

and piwik Archive stops month archiving:


ERROR CoreConsole[2014-08-09 20:49:26] [9a9a2] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last30&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 8208 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-09 20:49:27] [9a9a2] Archived website id = 1, period = month, 0 visits in last last30 months, 0 visits this month, Time elapsed: 1732.712s

Swap size after that drops to ~1GB.

Then the same happens for the year archiving procedure. Swap size rises even more:


top - 00:16:31 up  9:41,  3 users,  load average: 3.08, 2.85, 2.60
Tasks: 155 total,   1 running, 154 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.6%us, 10.1%sy,  0.0%ni, 34.6%id, 47.1%wa,  0.0%hi,  0.1%si,  0.5%st
Mem:   5990960k total,  5873464k used,   117496k free,    11060k buffers
Swap:  3096568k total,  2183080k used,   913488k free,    41876k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
27384 root      20   0 5953m 4.3g 2128 D  1.0 75.4  22:28.26 1.1g php
 1644 mysql     20   0 2327m 719m 3824 S  1.0 12.3 168:59.59 493m mysqld
 1866 nginx     20   0  438m 9780 2968 S  0.0  0.2   0:09.75  19m php-fpm
16753 root      20   0  409m 6672 1896 S  6.6  0.1  10:11.09  19m php

and then this phase terminates with the same error.

Please advise!

Thanks,
Nick

matthieu · August 10, 2014, 3:59pm

how many visits and pageviews do you track each day and each month?

nick1200 · August 10, 2014, 9:04pm

Hi matt,

It’s about 12,000 to 15,000 unique visitors and about 150,000 to 200,000 pageviews daily.

Monthly pageviews are about 5M. Monthly visitors are about 400K.

I am running Piwik Archiving once a day (as a cron job) on the same server where the web server runs.

Thanks and all the best,
Nick

matthieu · August 11, 2014, 12:09am

This archiving should work with less memory but sorry I cannot say more without debugging in details. Maybe you have some PHP programmer in house that could run the archiving through profiler XHProf? or maybe we could help you further…

nick1200 · August 11, 2014, 8:04am

I can try to run it. I’ve installed XHProf; Is it enough to do the following?

I understand I should apply these directions to the “console” file.

Otherwise, please provide more specific directions, if applicable.


# Edit your php.ini and add those line at the end:

[xhprof]
extension=xhprof.so
;
; directory used by default implementation of the iXHProfRuns
; interface (namely, the XHProfRuns_Default class) for storing
; XHProf runs.
;
xhprof.output_dir=/tmp/xhprof

# Make sure to change the output dir to something appropriate on your machine.
# To profile your application, you now need to add some code at the beginning and at the end of you page execution. Add this at the beginning:

xhprof_enable();

# And this at the end:

$data = xhprof_disable();

$XHPROF_ROOT = '/usr/share/xhprof';
include_once $XHPROF_ROOT . "/xhprof_lib/xhprof_lib.php";
include_once $XHPROF_ROOT . "/xhprof_lib/xhprof_runs.php";

$xhprof_runs = new XHProfRuns_Default();

// Save the run under a namespace "xhprof".
$run_id = $xhprof_runs->save_run($data, "xhprof");

I have adapted these directions from: Profiling a PHP Application | Eric Hogue's Blog using paths from my own installation (using xhprof-0.9.4-1.el6.noarch.rpm from EPEL repo).

nick1200 · August 11, 2014, 8:20am

By the way, I noticed a significant strange step rise in pageviews number after July 4 (from about 30K to 200K daily).

Did you introduce any change(s) in the Archiving code -that may have caused such an increase of Pageviews numbers- in a recent version (which was published around that time) and I updated then?

Thanks,
Nick

nick1200 · August 11, 2014, 12:57pm

In the meantime, I tried the above code to enable xhprof, but I got errors (see below); so, I would appreciate your feedback in enabling it correctly.

Here is the output of my trying to run with the above modifications in /etc/php.ini and in the “console” file:


# /usr/bin/php /var/webs/wwwpiwik/www/console core:archive --url=http://wstat1.noa.gr
PHP Warning:  Module 'xhprof' already loaded in Unknown on line 0
INFO CoreConsole[2014-08-11 12:47:46] [3fff0] ---------------------------
INFO CoreConsole[2014-08-11 12:47:46] [3fff0] INIT
INFO CoreConsole[2014-08-11 12:47:46] [3fff0] Piwik is installed at: http://wstat1.noa.gr/index.php
INFO CoreConsole[2014-08-11 12:47:46] [3fff0] Running Piwik 2.4.1 as Super User
PHP Warning:  Module 'xhprof' already loaded in Unknown on line 0
PHP Warning:  Module 'xhprof' already loaded in Unknown on line 0
ERROR CoreConsole[2014-08-11 12:47:48] [3fff0] The Piwik URL http://wstat1.noa.gr/index.php does not seem to be pointing to a Piwik server. Response was 'PHP Warning:  Module 'xhprof' already loaded in Unknown on line 0 a:40:{s:9:"nb_visits";s:6:"Visits";s:16:"nb_uniq_visitors";s:15:"Unique visitors";s:10:"nb_actions";s:7:"Actions";s:20:"nb_actions_per_visit";s:17:"Actions per Visit";s:16:"avg_time_on_site";s:20:"Avg. Time on Website";s:11:"bounce_rate";s:11:"Bounce Rate";s:15:"conversion_rate";s:15:"Conversion Rate";s:5:"label";s:5:"Label";s:4:"date";s:4:"Date";s:16:"avg_time_on_page";s:17:"Avg. time on page";s:14:"sum_time_spent";s:41:"Total time spent by visitors (in seconds)";s:16:"sum_visit_length";s:41:"Total time spent by visitors (in seconds)";s:12:"bounce_count";s:7:"Bounces";s:22:"bounce_count_returning";s:33:"Bounce Count for Returning Visits";s:11:"max_actions";s:28:"Maximum actions in one visit";s:21:"max_actions_returning";s:38:"Maximum actions in one returning visit";s:29:"nb_visits_converted_returning";s:36:"Number of converted returning visits";s:26:"sum_visit_length_returning";s:51:"Total time spent by returning visitors (in seconds)";s:19:"nb_visits_converted";s:23:"Visits with Conversions";s:14:"nb_conversions";s:11:"Conversions";s:7:"revenue";s:7:"Revenue";s:7:"nb_hits";s:9:"Pageviews";s:15:"entry_nb_visits";s:9:"Entrances";s:22:"entry_nb_uniq_visitors";s:16:"Unique entrances";s:14:"exit_nb_visits";s:5:"Exits";s:21:"exit_nb_uniq_visitors";s:12:"Unique exits";s:18:"entry_bounce_count";s:7:"Bounces";s:17:"exit_bounce_count";s:7:"Bounces";s:9:"exit_rate";s:9:"Exit rate";s:26:"sum_daily_nb_uniq_visitors";s:27:"Unique visitors (daily sum)";s:32:"sum_daily_entry_nb_uniq_visitors";s:28:"Unique entrances (daily sum)";s:31:"sum_daily_exit_nb_uniq_visitors";s:24:"Unique exits (daily sum)";s:16:"entry_nb_actions";s:27:"Actions after entering here";s:22:"entry_sum_visit_length";s:61:"Total time spent by visitors (in seconds) after entering here";s:9:"nb_events";s:12:"Total events";s:15:"sum_event_value";s:11:"Total value";s:15:"min_event_value";s:13:"Minimum value";s:15:"max_event_value";s:13:"Maximum value";s:15:"avg_event_value";s:13:"Average value";s:20:"nb_events_with_value";s:19:"Events with a value";}'.
ERROR CoreConsole[2014-08-11 12:47:48] [3fff0] The Piwik URL http://wstat1.noa.gr/index.php does not seem to be pointing to a Piwik server. Response was 'PHP Warning:  Module 'xhprof' already loaded in Unknown on line 0 a:40:{s:9:"nb_visits";s:6:"Visits";s:16:"nb_uniq_visitors";s:15:"Unique visitors";s:10:"nb_actions";s:7:"Actions";s:20:"nb_actions_per_visit";s:17:"Actions per Visit";s:16:"avg_time_on_site";s:20:"Avg. Time on Website";s:11:"bounce_rate";s:11:"Bounce Rate";s:15:"conversion_rate";s:15:"Conversion Rate";s:5:"label";s:5:"Label";s:4:"date";s:4:"Date";s:16:"avg_time_on_page";s:17:"Avg. time on page";s:14:"sum_time_spent";s:41:"Total time spent by visitors (in seconds)";s:16:"sum_visit_length";s:41:"Total time spent by visitors (in seconds)";s:12:"bounce_count";s:7:"Bounces";s:22:"bounce_count_returning";s:33:"Bounce Count for Returning Visits";s:11:"max_actions";s:28:"Maximum actions in one visit";s:21:"max_actions_returning";s:38:"Maximum actions in one returning visit";s:29:"nb_visits_converted_returning";s:36:"Number of converted returning visits";s:26:"sum_visit_length_returning";s:51:"Total time spent by returning visitors (in seconds)";s:19:"nb_visits_converted";s:23:"Visits with Conversions";s:14:"nb_conversions";s:11:"Conversions";s:7:"revenue";s:7:"Revenue";s:7:"nb_hits";s:9:"Pageviews";s:15:"entry_nb_visits";s:9:"Entrances";s:22:"entry_nb_uniq_visitors";s:16:"Unique entrances";s:14:"exit_nb_visits";s:5:"Exits";s:21:"exit_nb_uniq_visitors";s:12:"Unique exits";s:18:"entry_bounce_count";s:7:"Bounces";s:17:"exit_bounce_count";s:7:"Bounces";s:9:"exit_rate";s:9:"Exit rate";s:26:"sum_daily_nb_uniq_visitors";s:27:"Unique visitors (daily sum)";s:32:"sum_daily_entry_nb_uniq_visitors";s:28:"Unique entrances (daily sum)";s:31:"sum_daily_exit_nb_uniq_visitors";s:24:"Unique exits (daily sum)";s:16:"entry_nb_actions";s:27:"Actions after entering here";s:22:"entry_sum_visit_length";s:61:"Total time spent by visitors (in seconds) after entering here";s:9:"nb_events";s:12:"Total events";s:15:"sum_event_value";s:11:"Total value";s:15:"min_event_value";s:13:"Minimum value";s:15:"max_event_value";s:13:"Maximum value";s:15:"avg_event_value";s:13:"Average value";s:20:"nb_events_with_value";s:19:"Events with a value";}'.

Please, advise.

lesjokolat · August 11, 2014, 1:55pm

Would your system happen to be using suhosin memory setting? If so check it and increase its memory setting up. What version of python are you using on your system?

http://forum.piwik.org/read.php?2,113069

This thread started similarly with some of your errors. maybe some ideas from it you can try?

nick1200 · August 11, 2014, 2:17pm

Thanks for your help. I am not using suhosin.

Python:


# rpm -qa | grep python
rpm-python-4.8.0-37.el6.x86_64
python-pycurl-7.19.0-8.el6.x86_64
python-libs-2.6.6-52.el6.x86_64
python-2.6.6-52.el6.x86_64
python-iniparse-0.3.1-2.1.el6.noarch
newt-python-0.52.11-3.el6.x86_64
python-urlgrabber-3.9.1-9.el6.noarch

I am aware of the thread you kindly provided (I’ve participated in it as well) and other similar ones, but I haven’t found a solution.

Earlier, when I had lower number of pageviews daily, it worked; now it does not, so I’ve got to find how to handle things and discover how I can run the archive procedure on my box.

lesjokolat · August 11, 2014, 2:52pm

Ideally i think the recommended python version is 2.7.1 minimum. looks like you have 2.6.6. Anyway you can try and update it?

nick1200 · August 11, 2014, 3:25pm

OK, I also installed python27:


# rpm -qa | grep python
rpm-python-4.8.0-37.el6.x86_64
python-pycurl-7.19.0-8.el6.x86_64
python-libs-2.6.6-52.el6.x86_64
python-2.6.6-52.el6.x86_64
python27-libs-2.7.3-6.2.el6.nux.x86_64
python-iniparse-0.3.1-2.1.el6.noarch
newt-python-0.52.11-3.el6.x86_64
python-urlgrabber-3.9.1-9.el6.noarch
python27-2.7.3-6.2.el6.nux.x86_64

Libs are in /usr/lib64/python2.7/ whereas libs of v2.6 are in /usr/lib64/python2.6/.

Executable is /usr/bin/python2.7 whereas v2.6 exe is /usr/bin/python.

Should I configure something somewhere to force the usage of 2.7 version?

Thanks,
Nick

lesjokolat · August 11, 2014, 5:57pm

Thats a very good question… I am unsure. Can you try and run the archive and see if it helps? I am hoping teh newest version supercedes the older one…

nick1200 · August 11, 2014, 7:29pm

No, v2.6 remains the default. Is there a way to run the Archive procedure by specifying specific Python executable/libraries?

Also, I am still hoping that Matt can guide me to run correctly the Archive procedure under XHProf and investigate issues further.

Thanks,
Nick

matthieu · August 12, 2014, 12:05pm

@nick unfortunately it’s difficult to walk you through this as it’s bit technical.

(there was an error in your output: PHP Warning: Module ‘xhprof’ already loaded in Unknown on line 0 a:40:{s:9:"nb_visits)

but I read you get 200K pageviews per day? that’s quite a lot, maybe this explains your higher memory usage?

nick1200 · August 12, 2014, 1:43pm

On pageviews No.: I feel I should repeat my question above: 301 Moved Permanently

We were getting much less traffic before Jul 4, as I mentioned.

In any case, it’s a free public-service web site on seismic activity, that’s why it attracts significant traffic.

On XHProf: in fact I do program in PHP, yet I’m not a professional programmer and I am not comfortable with OOP.

However, I believe I can follow your directions -which I hope you can provide- to run XHProf and gather data that may help you in debugging a possible memory leak.

About the error you mention: Obviously, the way I am trying to run console in conjunction to XHProf is not correct.

Please help, as using XHProf while Archiving I understand is piwik-specific. I am eager to follow any directions in order to help profile the app.

Thanks,
Nick

nick1200 · August 12, 2014, 7:44pm

Hmm, I found that xhprof was already configured in /etc/php.d/xhprof.ini so the error above was due to my adding it in /etc/php.ini.

After that, I ran archiving once again, with the code mentioned above (in console file), and in the end I was expecting to find something in the xhprof.output_dir (i.e in /tmp/xhprof) but I found nothing… :S

So, if you can provide some guidance on it, please do!

Nick

lesjokolat · August 12, 2014, 8:37pm

1

M. Tibbits answer alludes to allowing Piwik to use more resources during the archiving process.

There is a limitation on how large an SQL command can be executed, which defaults to just 1MB. For more info; Packet too large

To increase the size limitation in MySQL, edit /etc/my.cnf and set max_allowed_packet=32M

Ensure the PHP memory limit is set to be high enough per forked process by editing /usr/local/lib/php.ini and set memory_limit = 512M

Finally, ensure all processes have at least a 1G hard limit before the system shuts them off by executing ulimit -v 1048576 on the command line.

Update

ulimit -v 1048576 will only raise the soft limit. If the hard limit is not high enough, the system will automatically reset the soft limit to that of the hard limit.

To set the hard limit, add the -H switch:

ulimit -vH 1048576

followed by increasing the soft limit up to this value:

ulimit -vS 1048576

2

I have this problem last week, my solutiosn in CentOS rehel is

nano /etc/httpd/conf/hpptd.conf

search RLimitMEM directive delete or comented no uses this directives or if use allow 2000000 value

save this file
restart the service
service httpd restart

3
http://forum.piwik.org/read.php?2,107165 was an older topic but found teh sh. interesting…maybe an option for you?

matthieu · August 12, 2014, 11:19pm

To enable XHPROF for the core:archive script you can add the --xhprof option to the console command. Could you try this? at the end of the script it may output a link to see th eXHPROF result.

nick1200 · August 13, 2014, 10:38am

Thanks Matt,

I tried that, but I don’t see any xhprof-related link in the output:


# /usr/bin/php /var/webs/wwwpiwik/www/console core:archive --xhprof --url=http://wstat1.noa.gr
INFO CoreConsole[2014-08-13 08:29:15] [40e38] ---------------------------
INFO CoreConsole[2014-08-13 08:29:15] [40e38] INIT
INFO CoreConsole[2014-08-13 08:29:15] [40e38] Piwik is installed at: http://wstat1.noa.gr/index.php
INFO CoreConsole[2014-08-13 08:29:15] [40e38] Running Piwik 2.4.1 as Super User
INFO CoreConsole[2014-08-13 08:29:16] [40e38] ---------------------------
INFO CoreConsole[2014-08-13 08:29:16] [40e38] NOTES
INFO CoreConsole[2014-08-13 08:29:16] [40e38] - Reports for today will be processed at most every 21600 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-08-13 08:29:16] [40e38] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-08-13 08:29:16] [40e38] - Archiving was last executed without error 31 days 22 hours ago
INFO CoreConsole[2014-08-13 08:29:54] [40e38] - Will process 1 websites with new visits since 31 days 22 hours , IDs: 1
INFO CoreConsole[2014-08-13 08:29:54] [40e38] XHProf profiling is enabled.
INFO CoreConsole[2014-08-13 08:29:54] [40e38] ---------------------------
INFO CoreConsole[2014-08-13 08:29:54] [40e38] START
INFO CoreConsole[2014-08-13 08:29:54] [40e38] Starting Piwik reports archiving...
INFO CoreConsole[2014-08-13 08:30:41] [40e38] Archived website id = 1, period = day, 13105 visits in last last2 days, 1404 visits today, Time elapsed: 46.051s
INFO CoreConsole[2014-08-13 08:30:43] [40e38] Archived website id = 1, period = week, 1776004 visits in last last33 weeks, 24452 visits this week, Time elapsed: 2.437s
ERROR CoreConsole[2014-08-13 08:59:28] [40e38] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last33&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 71 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
ERROR CoreConsole[2014-08-13 08:59:28] [40e38] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last33&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 71 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-13 08:59:28] [40e38] Archived website id = 1, period = month, 0 visits in last last33 months, 0 visits this month, Time elapsed: 1725.347s
ERROR CoreConsole[2014-08-13 09:41:56] [40e38] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 72 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
ERROR CoreConsole[2014-08-13 09:41:56] [40e38] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 72 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-13 09:41:57] [40e38] Archived website id = 1, period = year, 0 visits in last last7 years, 0 visits this year, Time elapsed: 2548.543s
INFO CoreConsole[2014-08-13 09:41:57] [40e38] Archived website id = 1, 4 API requests, Time elapsed: 4322.405s [1/1 done]
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Done archiving!
INFO CoreConsole[2014-08-13 09:42:00] [40e38] ---------------------------
INFO CoreConsole[2014-08-13 09:42:00] [40e38] SUMMARY
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Total visits for today across archived websites: 1404
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Archived today's reports for 1 websites
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Archived week/month/year for 1 websites
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Skipped 0 websites day archiving: existing daily reports are less than 21600 seconds old
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Total API requests: 4
INFO CoreConsole[2014-08-13 09:42:00] [40e38] done: 1/1 100%, 1404 vtoday, 1 wtoday, 1 wperiods, 4 req, 4325607 ms, 2 errors.
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Time elapsed: 4325.609s
INFO CoreConsole[2014-08-13 09:42:00] [40e38] ---------------------------
INFO CoreConsole[2014-08-13 09:42:00] [40e38] SCHEDULED TASKS
INFO CoreConsole[2014-08-13 09:42:00] [40e38] Starting Scheduled tasks... 
ERROR CoreConsole[2014-08-13 09:42:13] [40e38] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp&xhprof=2. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-13 09:42:13] [40e38] Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp&xhprof=2. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.

INFO CoreConsole[2014-08-13 09:42:13] [40e38] done
INFO CoreConsole[2014-08-13 09:42:13] [40e38] ---------------------------
INFO CoreConsole[2014-08-13 09:42:13] [40e38] ---------------------------
INFO CoreConsole[2014-08-13 09:42:13] [40e38] SUMMARY OF ERRORS
INFO CoreConsole[2014-08-13 09:42:13] [40e38] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=month&date=last33&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 71 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-13 09:42:13] [40e38] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=API.get&idSite=1&period=year&date=last7&format=php&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp. Response was 'PHP Fatal error:  Allowed memory size of 5368709120 bytes exhausted (tried to allocate 72 bytes) in /var/webs/wwwpiwik/www/core/DataTable.php on line 1155 '
INFO CoreConsole[2014-08-13 09:42:13] [40e38] Error: Got invalid response from API request: http://wstat1.noa.gr/index.php?module=API&method=CoreAdminHome.runScheduledTasks&format=csv&convertToUnicode=0&token_auth=a0709b20762bae2088c915aa19a461d2&trigger=archivephp&xhprof=2. The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.
ERROR CoreConsole[2014-08-13 09:42:13] [40e38] 3 total errors during this script execution, please investigate and try and fix these errors.
ERROR CoreConsole[2014-08-13 09:42:13] [40e38] 3 total errors during this script execution, please investigate and try and fix these errors.

I also didn’t find any xhprof-related output in /tmp or in the piwik website directory tree.

Any hint?

thomas_matomo · August 13, 2014, 1:15pm

The --xhprof parameter will probably not work for you since you would need a specific version of xhprof in a specific folder and from what I can see there are also some PHP errors in our xhprof implementation.

What you basically want to do is adding this to ‘core/CronArchive.php’ around line 206 (before “$this->initLog();”)


xhprof_enable();

and around line 217 after “$this->runScheduledTasks();” you want to add this code:


$data = xhprof_disable();

$XHPROF_ROOT = '/usr/share/xhprof'; // or replace with path to your xhprof folder
include_once $XHPROF_ROOT . "/xhprof_lib/xhprof_lib.php";
include_once $XHPROF_ROOT . "/xhprof_lib/xhprof_runs.php";

$xhprof_runs = new \XHProfRuns_Default();

// Save the run under a namespace "xhprof".
$run_id = $xhprof_runs->save_run($data, "xhprof");
echo "http://localhost/xhprof/xhprof_html/index.php?run={$run_id}&source=xhprof_testing\n"; // or replace with path to your xhprof host/server