Issue about log analysis for load balancing servers

mihui · January 15, 2013, 7:50am

I’m using Piwik for analyzing logs from load balancing servers.
When I import the data from the logs with different timestamp, it shows some errors, after investigation, I’ve found out that the reasons as below:

LogServer1 – Visitor 1: 10:00:00
LogServer1 – Visitor 1: 11:30:00
LogServer2 – Visitor 1: 10:20:00 - Error

If any time earlier than the time of imported data , the import job will fail (Actually fail the SQL execution).

Is that possible to import the data successfully with random timestamps?

See the ticket below:

github.com/matomo-org/matomo

Tracking API: Handle when older visitor/page views data is imported after the newer data...

opened 10:59AM - 07 Jan 13 UTC

closed 02:39AM - 18 Jan 13 UTC

anonymous-matomo-user

Bug wontfix

I have the IIS logs of load balance servers like this structure: Server 1: log1…/ u_ex121119.log u_ex121120.log u_ex121121.log Server 2: log2/ u_ex121119.log u_ex121120.log u_ex121121.log Server 3: log3/ u_ex121119.log u_ex121120.log u_ex121121.log When import the data from the log to the Piwik, I get the errors like this: ``` Error when connecting to Piwik: <urlopen error None> ``` I've debugged the exceptions, it showed that there were two variables were out of range: ``` $visitTotalTime and $timeSpentRefererAction ``` they were all negative numbers: ``` Error query: SQLSTATE[22003]: Numeric value out of range: 1264 Out of range value for column 'visit_total_time' at row 1 Error query: SQLSTATE[22003]: Numeric value out of range: 1264 Out of range value for column 'time_spent_ref_action' at row 1 ```

I tried to merge the logs and sort the datetime, but it consumed the server memory to death, sometimes 4GB is not enough for sorting the data since the total size of the logs is more than 3G.