with Large Apache Log Files


When trying to import an entire month’s worth of Apache log files (which has something like 13 million lines and is about 1.5G), I’ve found the import_logs script to be painfully slow as it imports, on average, only about 50 lines per second. This came as a surprise especially because when I ran the --dry-run command, thousands of lines were being “imported” on average and the script was done in 20 minutes, compared to the the 50 or so I received when I actually imported the files.

Is there any way to increase the number of lines actually read/entered into the database? I’m running a CentOS 6.6 (with cPanel) 4GB ram VPS, on which I installed a separate instance of Python (2.7.6) to ensure that this script would perform well.

Thanks in advance.

(Matthieu Aubry) #2

Hi there, check out this guide which has some tips on how to make Piwik faster: How to configure Piwik for speed - Analytics Platform - Matomo

With a strong and well tuned server you can reach 200 req/s or even 300 req/s