Log-analytics & import_logs.py


#1

Hi,

I try to import some logs from the nginx webserver.

I have followed the howto from https://github.com/piwik/piwik/tree/master/misc/log-analytics#setup-nginx-logs.

This setup generates a nice access log file that looks like:

{“ip”: “192.168.1.10”,“host”: “www.test.nl”,“path”: “/”,“status”: “200”,“referrer”: “https://www.test.nl/","user_agent”: “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36”,“length”: 915,“generation_time_milli”: 0.001,“date”: “2015-01-02T13:20:31+01:00”}

When I try to import this access log with:
$ python /var/www/misc/log-analytics/import_logs.py --url=http://ka-webanalyse-01.edu.local/ --add-sites-new-hosts --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots --log-format-name=nginx_json --config /var/www/config/config.ini.php access.log

I get the flowing error:

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log access.log…
Traceback (most recent call last):
File “/var/www/misc/log-analytics/import_logs.py”, line 1750, in
main()
File “/var/www/misc/log-analytics/import_logs.py”, line 1717, in main
parser.parse(filename)
File “/var/www/misc/log-analytics/import_logs.py”, line 1576, in parse
resolver.check_format(format)
File “/var/www/misc/log-analytics/import_logs.py”, line 1119, in check_format
elif ‘host’ not in format.regex.groupindex and not config.options.log_hostname:
AttributeError: ‘NoneType’ object has no attribute ‘groupindex’

As I understands is correctly is it missing the host parameter in the access log but as you can see it is there. What goos wrong.

Kinds regards,

Michiel Piscaer


(Matthieu Aubry) #2

Hi there, thanks for the report. do you mind creating an issue on our issue tracker? we will investigate: Issues · matomo-org/piwik · GitHub


#3

Same error with the format_name option w3c_extended.


#4

Like matt asked, I have created an issue on github: log-analytics & import_logs.py, hosts parameter failing · Issue #6919 · matomo-org/matomo · GitHub


(Matthieu Aubry) #5

Thanks for that!