We currently use the import function(use import_iis.py) in piwik 1.9.2 which seems insert data record one by one, the efficiency is not good. For 100mb log file, it will spend us 1 hour. Is there a way to analyze the log file and save the data into db directly(not by calling php server page and not by inserting one by one)? or is there any other way we can use to improve the efficiency of importing data?
By the way, currently we can’t upgrade piwik version
It’s unfortunate that you can’t upgrade, as newer versions of both Piwik and the import script use a bulk importer that does multiple records at once (I’m running it right now and it’s averaging 400 records a second).
The hard part about dumping data directly into the database is that Piwik does a lot of the “heavy lifting”, ie the actual analysis, for you. I am in the process of creating my own desktop application for importing logs, based on the python script, but it still sends the data to piwik.php for the actual processing of the stats.
You should really see if you can push to upgrade, as it will help you out a lot. Especially with some of the newer stuff that they have coming in with php 5.3/5.4 and the speed increases.
Not possible to bypass Piwik to record data in the DB.
Thanks John and Matt.