import_logs.py with Large Apache Log Files


#1

When trying to import an entire month’s worth of Apache log files (which has something like 13 million lines and is about 1.5G), I’ve found the import_logs script to be painfully slow as it imports, on average, only about 50 lines per second. This came as a surprise especially because when I ran the --dry-run command, thousands of lines were being “imported” on average and the script was done in 20 minutes, compared to the the 50 or so I received when I actually imported the files.

Is there any way to increase the number of lines actually read/entered into the database? I’m running a CentOS 6.6 (with cPanel) 4GB ram VPS, on which I installed a separate instance of Python (2.7.6) to ensure that this script would perform well.

Thanks in advance.


(Matthieu Aubry) #2

Hi there, check out this guide which has some tips on how to make Piwik faster: How to configure Piwik for speed - Analytics Platform - Matomo

With a strong and well tuned server you can reach 200 req/s or even 300 req/s