Importing remote logs to Piwik

v1ktor · December 13, 2017, 1:18am

I wanted to confirm how to correctly handle importing of remote logs to Piwik.

As I understand it, Piwik uses Python script to parse the logs and it imports it to Piwik. Am I correct to assume that I can use this Python script on hosting servers and import those logs to Piwik on another server?

This isn’t specifically mentioned in the instructions/FAQs, so I wanted to confirm before I begin testing. Since Piwik’s base URL is the required parameter, I assume importing remote logs should be OK.

Thanks!

Lukas · December 13, 2017, 8:41am

Hi,

I don’t have much experience with LogImports but I think what you are trying to do will work.
You can find more information here:

You must specify your Piwik URL with the --url argument. The script will automatically read your config.inc.php file to get the authentication token and communicate with your Piwik install to import the lines. The default mode will try to mimic the Javascript tracker as much as possible, and will not track bots, static files, or error requests.

But I think you’ll need to use the token_auth parameter. It doesn’t seem to be that well documented but you can find information about it in this github issue and the output of the --help parameter:

github.com

matomo-org/matomo-log-analytics/blob/master/import_logs.py#L577


      
          option_parser.add_option(
              '--config', dest='config_file', default=default_config,
              help=(
                  "This is only used when --login and --password is not used. "
                  "Matomo will read the configuration file (default: %default) to "
                  "fetch the Super User token_auth from the config file. "
              )
          )
          option_parser.add_option(
              '--login', dest='login',
              help="You can manually specify the Matomo Super User login"
          )
          option_parser.add_option(
              '--password', dest='password',
              help="You can manually specify the Matomo Super User password"
          )
          option_parser.add_option(
              '--token-auth', dest='matomo_token_auth',
              help="Matomo user token_auth, the token_auth is found in Matomo > Settings > API. "
                   "You must use a token_auth that has at least 'admin' or 'super user' permission. "
                   "If you use a token_auth for a non admin user, your users' IP addresses will not be tracked properly. "

v1ktor · December 14, 2017, 6:30am

Thanks Lucas. This is very helpful. I’ll run some tests, hopefully I’ll be able to get it working. I’ll try to contribute to the instructions on Github once I got it to work.

SlavikCA · December 17, 2017, 1:33am

I use Piwik log-analytics with my Apache web server.

There are few things, which I considered, which may be useful to you:

Where to put Piwik server?
I can put it on the same server, as web server. That would make it more secure, because log importing will happen internally, - there is no need to open Piwik endpoints to the net. However, my web server has limited IOPS and memory, and Piwik can put extra load on it - some of it’s MySQL queries are rather heavy.

So, I decided to host Piwik on separate server.
So, what is the proper way to process logs, when Web server and Piwik are on remote hosts?
1st though was to run script in cron, which will process log file on remote web server OR send local log file to remote Piwik. The problem here is that if anything fails here - connection lost, Piwik down, etc… - that day will be lost and will need to be recovered manually. I didn’t find good retry mechanism for Piwik log file processing.

So, my current solution:

Piwik server has nightly cron job, which MOVE (not copy) via RSYNC log file(s) (all, except live) from web server to Piwik server “processing” folder. If anything fails one day, - the next day 2 or more files will be moved.
Then the cron job process (imports) all logs in “processing” folder and moves them into “backup” folder. That processing is done locally, so it’s secure and reliable.

I had issue, when import job was crashing on some days:

github.com/matomo-org/matomo-log-analytics

import_logs.py crashes: parser ValueError

opened 06:36PM - 17 Oct 17 UTC

closed 10:11PM - 19 Oct 17 UTC

SlavikCA

not-in-changelog

I have daily cron job, which pulls Apache log file and process it with `import_l…ogs.py`. It works mostly ok, but recently I found some anomalies: some days had incomplete data. While investigating, I found that on some days `import_logs.py` crashes while processing Apache log file. Here is part of the bash script, ran by cron: ``` # Apache 2.4: # LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D" timed FORMAT="(?P<ip>\S+) \S+ \S+ \[(?P<date>.*?) (?P<timezone>.*?)\] \"\S+ (?P<path>.*?) \S+\" (?P<status>\S+) (?P<length>\S+) \"(?P<referrer>.*?)\" \"(?P<user_agent>.*?)\" (?P<generation_time_micro>\S+)" python /var/www/html/misc/log-analytics/import_logs.py --url=$PIWIK \ --idsite=5 $STAGE/$LOCATION \ --log-format-regex="$FORMAT" ``` Here is crash stack trace: ``` Traceback (most recent call last): File "/var/www/html/misc/log-analytics/import_logs.py", line 2415, in <module> main() File "/var/www/html/misc/log-analytics/import_logs.py", line 2386, in main parser.parse(filename) File "/var/www/html/misc/log-analytics/import_logs.py", line 2264, in parse hit.generation_time_milli = float(format.get('generation_time_micro')) / 1000 ValueError: could not convert string to float: Build/KTU84P) ``` While digging in, I found the problematic line in the log: ``` 213.87.121.235 - - [11/Oct/2017:08:36:33 -0700] "GET /sites/default/files/js/js_xfdu9t6ALWTm1CBmur4eWBRYnIuBOvfaBN2QvoZNT50.js HTTP/1.1" 304 205 "http://www.noty-bratstvo.org/glossary/%D0%BA" "Mozilla/5.0 (Linux; Android 4.4.4; Ixion X LTE 4.5\" Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 YaBrowser/14.10.2062.12160.00 Mobile Safari/537.36" 576 ``` The problem is escaped quote here: `...Ixion X LTE 4.5\" Build/KTU84P...` I suspect, that my `FORMAT` is not 100% correct - it can't handle escaped quote. But my **expectation is that if parser fails in certain line, it shouldn't crash** and stop, but should put the line into `invalid log lines` and keep going. Also, I would appreciate, if you can help me to come up with correct `FORMAT` string. My env: - Piwik 3.1.1 - Python 2.7.12 - Ubuntu 16.04 x64 4.4.0-97-generic

But it was easy for me to replay logs, by using that scheme.