When trying to import an entire month’s worth of Apache log files (which has something like 13 million lines and is about 1.5G), I’ve found the import_logs script to be painfully slow as it imports, on average, only about 50 lines per second. This came as a surprise especially because when I ran the --dry-run command, thousands of lines were being “imported” on average and the script was done in 20 minutes, compared to the the 50 or so I received when I actually imported the files.
Is there any way to increase the number of lines actually read/entered into the database? I’m running a CentOS 6.6 (with cPanel) 4GB ram VPS, on which I installed a separate instance of Python (2.7.6) to ensure that this script would perform well.
Thanks in advance.