Apache log parser?


(Victorious) #1

Hi piwik developer team

I don’t know if is the right place for my questions.

Existed an apache log parser tool for piwik?
It would be very nice if piwik parse the apache log file and write it in the Database.

PS:Piwik is a very nice web analytics tool.

Best regards,
Victorious


(kolchak) #2

There is no apache log parser and its not on the roadmap, but you can write one if you are keen style_emoticons/<#EMO_DIR#>/smile.gif There is some more information here:

http://dev.piwik.org/trac/ticket/547

and a beta version here:

http://dev.piwik.org/trac/browser/trunk/mi…layer.py.script

I haven’t used it so can’t comment on how it works


(__c) #3

there’s no need to write an apache log parser from scratch, as there are many of them available under open licenses. a quick search revealed a bunch that insert into sql, though i don’t know how flexible they are.

i’ll make the parser->db piece if someone will explain where the data should go so i don’t have to dig through the code and schema.

apache logs typically have this in their config file:

LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combined

which includes:

  1. IP
  2. Username (from auth)
  3. Time
  4. First line of request ("GET /foo.html")
  5. HTTP status (ok, redirect, etc)
  6. Size of response
  7. Referrer
  8. User Agent (Browser)
and the error log file which tends to follow this format:
(date) (IP) (message)

where should this go and what conversions would i need to make?


(__c) #4

PS, there is also “mod_log_sql” and “mod_log_mysql” for apache that allows direct insertion into the database. probably not available for most people on shared hosts, but is an interesting option.

http://www.onlamp.com/pub/a/apache/2005/02…ogs.html?page=2