Huge setup may cause "curl_exec: Empty reply from server"


#1

Hi,

in order to prevent getting SegVaults from the archive.sh cron script we use the archive.php cron script. But here comes an other error:

[2011-12-05 11:39:25] [6567943f] [8.42 Mb] Error: Got invalid response from API request: http://MYDOMAIN/index.php?module=API&method=VisitsSummary.getVisits&idSite=ID_SITE&period=day&date=last52&format=php&token_auth=TOKEN_AUTH&trigger=archivephp. Response was 'curl_exec: Empty reply from server'
[2011-12-05 11:39:25] [6567943f] [8.42 Mb] Error: Got invalid response from API request: http://MYDOMAIN/index.php?module=API&method=VisitsSummary.getVisits&idSite=ID_SITE&period=day&date=last52&format=php&token_auth=TOKEN_AUTH&trigger=archivephp. Response was 'curl_exec: Empty reply from server'
[2011-12-05 11:39:25] [6567943f] [8.42 Mb] 2 total errors during this script execution, please investigate and try and fix these errors

After php has processed the script we restarted it and it completes the missing sites without errors.

There are no error messages in the webserver error_logs. The access_logs shows http status 200 with sended content.

In avarage we have 600 queries per second. So there is much data to procress.


(Matthieu Aubry) #2

Thanks for the report. What output do you get if you manually execute: http://MYDOMAIN/index.php?module=API&method=VisitsSummary.getVisits&idSite=ID_SITE&period=day&date=last52&format=php&token_auth=TOKEN_AUTH&trigger=archivephp ?

Are you 100% confident that these exact requests show as 200 in the access.log ?

If so, it sounds like the problem would be in the curl code, or maybe the way we use it?


#3

Hi matt,

[quote=matt]
What output do you get if you manually execute: http://MYDOMAIN/index.php?module=API&method=VisitsSummary.getVisits&idSite=ID_SITE&period=day&date=last52&format=php&token_auth=TOKEN_AUTH&trigger=archivephp ?[/quote]
I got some kind of JSON response, timespans and its values I guess. Some KB of it.

Yes, we have a Piwik cluster and I checked one server out of it so there was no traffic but mine. Only 200 responses and no error messages. All responses with 2 KB of content and above.


(Matthieu Aubry) #4

Do you get these errors logged after each archive.php run ?

Would you mind logging the output of the script to a file, ie. replace > /dev/null with >> /path/piwik/tmp/archive.log

Then email me the log file after 1 day and possibly 24 executions?

Thanks :slight_smile: would like to understand and fix this issue.


#5

[quote=matt]
Do you get these errors logged after each archive.php run ?[/quote]
The problem is that errors only thrown from archive.php. All webserver logs look fine (status 200 with some KB of content).

[quote=matt]
Then email me the log file after 1 day and possibly 24 executions?[/quote]
Is in the making, I am on vacation tomorrow. My college will send you the logs in 24 hours from now on.


(Matthieu Aubry) #6

Thanks for the log. Unfortunately I don’t know what the error could be. Can you please gather multiple archive runs (change in the cron from > /tmp/archive.php to >> /tmp/archive.log) so that all archive runs are appended in this file.

Then after 20 or 30 runs, could you please send me the new log file? Maybe we will be able to note some patterns in the errors (for example, do they all fail when it goes over 10 s ?)


#7

Hi matt,

send you an email with the log files. Because it’s a huge file mx.piwik.org rejected it but passed your private mail server.


(Matthieu Aubry) #8

Thanks for the log file. Sorry, but I’m not sure what is causing this issue… It must be some kind of configuration / server issue but I don’t know…


(kero) #9

Hi,
i have the same error after update from 1.6 to 1.7 and running archive.php manually:

Fatal error: 1 total errors during this script execution, please investigate and try and fix these errors. First error was: Got invalid response from API request: https://mypiwikmachine/index.php?module=API&method=VisitsSummary.getVisits&idSite=14&period=day&date=last52&format=php&token_auth=TOKEN_AUTH&trigger=archivephp. Response was ‘curl_exec: Empty reply from server’ in /var/www/piwik/misc/cron/archive.php on line 487

idSite 14 has about 20000-25000 visits a day. I monitor about 130 sites in piwik.

update: second try runs without any problems, maybe some kind of timeout problem?

Thanks

kero


#10

Hi,
I have a similar error after update from 1.6 to 1.7.

13 total errors during this script execution, please investigate and try and fix these errors. First error was: Got invalid response from API request: https://xxx/index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=day&date=last52&format=php&token_auth=xxxx&trigger=archivephp. Response was 'curl_exec: SSL certificate problem, verify that the CA cert is OK.

This happened for all monitored domains for some cron runs. It looks like a webserver problem but at the same time i could not find any problems related to this error message. The error has gone without investigation.

Chris


#11

Hi Chris,

like you can read out of the error message your webserver was sending out an invalid SSL certificate or had some tempoary problems (to high load, memory failure, CA crl/ocsp failure, etc.) with it. It’s NOT related to this issue nor thread nor Piwik.


#12

We have the same Problem. Yesterday i’ve updated to 1.7.1

The Script is called by


5 * * * * www-data /usr/bin/php5 /var/www/stats.globalgameport.com/html/misc/cron/archive.php --url=http://stats.globalgameport.com/  >> /var/log/piwik.log

The content of /var/log/piwik.log is:


[2012-02-27 17:05:35] [8c93602d] SUMMARY OF ERRORS
[2012-02-27 17:05:35] [8c93602d] Error: Got invalid response from API request: http://stats.XXX.com/index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=day&date=last2&format=php&token_auth=XXX&trigger=archivephp. Response was 'curl_exec: Empty reply from server'
[2012-02-27 17:05:35] [8c93602d] 1 total errors during this script execution, please investigate and try and fix these errors
[2012-02-27 17:05:35] [8c93602d] ERROR: 1 total errors during this script execution, please investigate and try and fix these errors. First error was: Got invalid response from API request: http://stats.XXX.com/index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=day&date=last2&format=php&token_auth=XXX&trigger=archivephp. Response was 'curl_exec: Empty reply from server'

Fatal error: 1 total errors during this script execution, please investigate and try and fix these errors. First error was: Got invalid response from API request: http://stats.XXX.com/index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=day&date=last2&format=php&token_auth=XXX&trigger=archivephp. Response was 'curl_exec: Empty reply from server' in /var/www/stats.XXX.com/html/misc/cron/archive.php on line 519

When Calling http://stats.XXX.com/index.php?module=API&method=VisitsSummary.getVisits&idSite=1&period=day&date=last2&format=php&token_auth=XXX&trigger=archivephp manually i get: a:2:{s:10:"2012-02-26";d:8801;s:10:"2012-02-27";d:4185;}

The error.log of this vhost is empty
In the global apache2 error.log i found


zend_mm_heap corrupted

But without any timestamp.

Any ideas?


#13

Hi Valkum,

try to run the cronjob several times:

/usr/bin/php5 /var/www/stats.globalgameport.com/html/misc/cron/archive.php --url=http://stats.globalgameport.com/  >> /var/log/piwik.log ; /usr/bin/php5 /var/www/stats.globalgameport.com/html/misc/cron/archive.php --url=http://stats.globalgameport.com/  >> /var/log/piwik.log ; /usr/bin/php5 /var/www/stats.globalgameport.com/html/misc/cron/archive.php --url=http://stats.globalgameport.com/  >> /var/log/piwik.log

Mostly this problem does not occur at the third run on our servers.


#14

:smiley:
But thats very very dirty ^^


#15

[quote=Valkum]
:smiley:
But thats very very dirty ^^[/quote]

indeed, but it works :wink:


#16

Here is my complete log since yesterday:
http://pastebin.com/NiGRCVuk


(Matthieu Aubry) #17

Or maybe upgrade apache + php to latest versions?


#18

Installed versions:
php5: 5.3.10-1~dotdeb.1

php5-curl is: 5.3.10-1~dotdeb.1

apache2-mpm-prefork: 2.2.16-6+squeeze6
apache2.2-common: 2.2.16-6+squeeze6