Issues with import_logs.py and core:archive [solved]


#1

Hello,

I’ll try to be brief! Using 2.14.3, aiming to only import logs. Have set PHP mem to 512, max_execution to 300, nginx fastcgi timeouts are at 14400.

[ul]
[li] import_logs gets stuck at 58036 lines each time
[/li][li] get the same error[0] repeatedly now, whether I use --recorder-max-payload-size=200 or not
[/li][li] restarting or reloading nginx doesn’t help it to reconnect (nor does it help to do the same with php5-fpm or mysql)
[/li][li] when I try to core:archive the incomplete import_logs I get MySQL server has gone away errors
[/li][/ul]

[0]


2015-10-08 15:31:09,782: [INFO] Error when connecting to Piwik: HTTP Error 500: Internal Server Error
2015-10-08 15:31:09,783: [INFO] Max number of attempts reached, server is unreachable!
Fatal error: HTTP Error 500 Internal Server Error, response: {"status":"error","tracked":9,"invalid":0}
You can restart the import of "/var/log/nginx/site.com_access.log" from the point it failed by specifying --skip=58036 on the command line.

This was actually working on an hourly basis yesterday. No records since 21:30 last night are being reported on the dashboard now. Things I’ve found to get the import_logs.py working are:


pkill -9 python

and starting over, or adding


--recorder-max-payload-size=200 --replay-tracking

I later removed --replay-tracking, but still no dice.

Thought I had it solved earlier today after this comment but no new visits were recorded by the cron-runs using those options.

Any thoughts on what I can do?


(Matthieu Aubry) #2

MySQL server has gone away errors

see How do I fix the error "Mysql Server has gone away"? - Analytics Platform - Matomo

Maybe you can also test the latest beta version see if this improves? I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo


#3

Thanks Matt, that’s spot on.

I had changed the my.cnf away from default settings for performance, so max_allowed_packet was at 32MB, timeout was at 15s already. Followed the linked recommendations (128M and 30s timeout) and it allowed the log import to work, then the archive to complete.

Not sure if I still need max payload size at 200 still, but might keep just to keep resource use steady during log imports. Correct me if I’m wrong there! Also remains to be seen if the site as any issues with these settings - can’t imagine it making too big a difference though.

Thanks again, great support.