Piwik - import_logs.py is very long

Support & Bugs
1

Hello,

I want to know if a solution exists for Piwik to speed up the processing of logs.

Here’s my setup, Piwik 1.8.4 and I have about 20,000 websites.

The problem is that some sites have a lot of traffic, and to import_log took 3 days to do what is not possible.

Do you have a solution to improve the speed of the script knowing that I tried everything:


/var/www/piwik/misc/log-analytics/import_logs.py --url=http://localhost --log-format-name=common_complete --enable-reverse-dns --enable-http-redirects --enable-http-errors --enable-bots --enable-static --recorders=8 --recorder-max-payload-size=400 --add-sites-new-hosts $vhosts.log

Thank’s All !

2

Increase --recorders

remove --enable-reverse-dns

remove others --enable-XXX for more speed

3

Hello,

I tried this, the import log is still running since Saturday morning: (

Do you have another idea?


/var/www/piwik/misc/log-analytics/import_logs.py --url=http://localhost --log-format-name=common_complete --enable-http-redirects --recorders=50 --recorder-max-payload-size=500 --add-sites-new-hosts $vhosts;

Thank’s
Julien.D

4

in our tests we can import 400 requests per seconds and more if tuning the server. Please contact http://piwik.org/consulting/ Piwik experts for more (paid) help