Code snippets: Getting live logging using nginx, rsyslog, and a helper script on a Centos 6


#1

I managed to get live direct logging working on a Centos 6 install. The huge obstickale was rsyslog is used instead of syslog-ng and nginx sysloging was crippiled and can not be used.

I"me just posting the relevent files. Most of the info was gathered from http://syshero.org/post/68174083489/nginx-syslog-ing-without-breaking-the-bank-or-patching[/url] and [url=https://github.com/piwik/piwik/tree/master/misc/log-analytics]https://github.com/piwik/piwik/tree/master/misc/log-analytics

First off, relevant lines from nginx: Note that the log file must be created first using mkfifo to create a named pipe.


#  Log to a named pipe that rsyslog is reading and exec a shell script to record directly to piwik
    log_format  piwik  '{"ip": "$remote_addr",'
                       '"host": "$host",'
                       '"path": "$request_uri",'
                       '"status": "$status",'
                       '"referrer": "$http_referer",'
                       '"user_agent": "$http_user_agent",'
                       '"length": $bytes_sent,'
                       '"generation_time_milli": $request_time,'
                       '"date": "$time_iso8601"}';
    
    access_log /var/lib/nginx/access.log piwik

The relevant lines from rsyslog. Note I had to use the legasy format for the input module.


    $ModLoad imfile # needs to be done just once
    # File 1
    $InputFileName /var/lib/nginx/access.log
    $InputFileTag nginx_log:
    $InputFileStateFile /var/lib/rsyslog/nginx_statefile1
    
    $InputFileSeverity info
    $InputFileFacility local7
    $InputFilePollInterval 5
    $InputRunFileMonitor
    
    if $syslogtag contains 'nginx_log' and $syslogfacility-text == 'local7' then ^/path/to/piwik.sh;nginxlog
    :syslogtag, contains, "nginx_log" ~

Now the helper script that calls the importer. Note that when using the importer, both index.php and piwik.php must be aviable as the importer will call both of those. The index.php to get a session variable (I think) then calls the piwik.php to record the log.


#!/bin/sh

echo "$1" | /path/to/piwik/misc/log-analytics/import_logs.py \
 --url=http://statsapi.localhost.org/ --token-auth=<token> \
 --idsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots \
 --log-format-name=nginx_json -

The echo “$1” is used because rsyslog calls the helper script using the %msg% as the one and only argument instead of syslog-ng of pipeing the %msg% on the standard input.

I give permission for anyone to use this info and create a nice tuturial post plus update the info at https://github.com/piwik/piwik/tree/master/misc/log-analytics


(Matthieu Aubry) #2

Hi Techwolf!

Thank you for the post and sharing the knowledge. I think this would be good info to add in the README file for Log Analytics.
Would you like to make a pull request, and suggest the changes to the README file? Then we can edit and merge the pull request and it is easy for us. Click “Edit” on: https://github.com/piwik/piwik/blob/master/misc/log-analytics/README.md


#3

Sure. Give me a year or two. I’me not joking. My free RL time is usually taken up with higher pirotories things. Such the life of a truck driver doing volunteer work with some open source projects.


#4

Hey thanks for the snippet. I tried looking for those logging variables but for example I can’t find the $length one ! Where did you read it exists?