import_logs.py not working in 2.1


#1

Is anyone using import_logs.py to analize data in 2.1?

In piwik 2.1 with NCSA extended logs (apache) I’m not able to view page actions but errors, statics and redirections.

Any clue?


(Matthieu Aubry) #2

Can you explain better what you mean? what is the difference from beefore 2.1 ?

Here is what would help:
put a small log file that you use. And the command you used.
Then, what you expected to see and what you saw instead?

I have seen other reports of users saying something changed but for us in our tests, all looks OK, so maybe we need to update our tests :slight_smile:


(Dmitry) #3

Using Python 2,7,3 Ubuntu 12.04.4 (LTS)
apache 2.2.*
php 5,3,*
Mysql 5,5,35

python /var/www/piwik/misc/log-analytics/import_logs.py --url=http://pub1.player-radio.ru /opt/logs/access.log --idsite=2 --log-format-name=icecast2 --strip-query-string --show-progress
Traceback (most recent call last):
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 1720, in
resolver = config.get_resolver()
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 621, in get_resolver
return StaticResolver(self.options.site_id)
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 966, in init
’SitesManager.getSiteFromId’, idSite=self.site_id
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 948, in call_api
return cls._call_wrapper(cls._call_api, None, None, method, **kwargs)
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 937, in _call_wrapper
raise Piwik.Error(message)
main.Error: Not Implemented


(Dmitry) #4

Sorry, but my log files to large of 200K, in archive too. I have fresh install Piwik 2.1, not update.


#5

This is my command line:

python /var/www/html/logimport/misc/log-analytics/import_logs.py --recorders=8 --url=http://localhost/logimport/ test.log --login=admin --password=pass --idsite=1

As you can see there is no --enable-static, --emable-http-errors o --enable-http-redirects in the command line

After import I launch an import process:

/usr/bin/php archive.php --url=http://localhost/logimport/

There is an image with the result of the import. NO PAGEVIEWS.

I have attached the file test.log.png rename to test.log.

I need to upgrade from 1.12 to 2.1 but I am testing in a clean installation.

Thxs.


(Matthieu Aubry) #6

please try with latest Piwik beta version: I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo

Then if you still see this bug, post here a small log file that you have used to reproduce the issue, with the full command use (and you expected to see VS what you got). I will then try to reproduce as well!


#7

Using Piwik 2.1.1b9

Command:

python /var/www/html/logimport/misc/log-analytics/import_logs.py --recorders=8 --url=http://localhost/logimport/ /tmp/test.log --login=admin --password=pass --token-auth=xxxxxxx --idsite=1

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /tmp/test.log…
500 lines parsed, 36 lines recorded, 35 records/sec (avg), 36 records/sec (current)
Purging Piwik archives for dates: 2014-03-30
To re-process these reports with your new update data, execute the piwik/misc/cron/archive.php script, or see: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo for more info.

Logs import summary

102 requests imported successfully
107 requests were downloads
398 requests ignored:
    0 invalid log lines
    11 requests done by bots, search engines, ...
    19 HTTP errors
    76 HTTP redirects
    292 requests to static resources (css, js, ...)
    0 requests did not match any known site
    0 requests did not match any requested hostname

Website import summary

102 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 1 seconds
Requests imported per second: 77.24 requests per second

Then:

/usr/bin/php /var/www/html/logimport/console core:archive --url=http://localhost/logimport/

The same result: No pageviews at all.


(Matthieu Aubry) #8

please try with 2.1.1-b10 -> does it fix the issue? I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo


#9

Using: piwik-2.1.1-b10

Same test.log file. Same result. No pageview actions

python /var/www/html/logimport/misc/log-analytics/import_logs.py --recorders=8 --url=http://localhost/logimport/ /tmp/test30032014.log --login=admin --password=pass --token-auth=xxxxxxx --idsite=1

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /tmp/test30032014.log…
Purging Piwik archives for dates: 2014-03-30
To re-process these reports with your new update data, execute the piwik/misc/cron/archive.php script, or see: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo for more info.

Logs import summary

102 requests imported successfully
107 requests were downloads
398 requests ignored:
    0 invalid log lines
    11 requests done by bots, search engines, ...
    19 HTTP errors
    76 HTTP redirects
    292 requests to static resources (css, js, ...)
    0 requests did not match any known site
    0 requests did not match any requested hostname

Website import summary

102 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 0 seconds
Requests imported per second: 148.91 requests per second

/usr/bin/php /var/www/html/logimport/console core:archive --url=http://localhost/logimport/ -v

INFO CoreConsole[2014-04-01 07:32:39] ---------------------------
INFO CoreConsole[2014-04-01 07:32:39] INIT
INFO CoreConsole[2014-04-01 07:32:39] Piwik is installed at: http://localhost/logimport/index.php
INFO CoreConsole[2014-04-01 07:32:39] Running Piwik 2.1.1-b10 as Super User: piwikadmin
INFO CoreConsole[2014-04-01 07:32:40] ---------------------------
INFO CoreConsole[2014-04-01 07:32:40] NOTES
INFO CoreConsole[2014-04-01 07:32:40] - If you execute this script at least once per hour (or more often) in a crontab, you may disable ‘Browser trigger archiving’ in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-04-01 07:32:40] See the doc at: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo
INFO CoreConsole[2014-04-01 07:32:40] - Reports for today will be processed at most every 3600 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-04-01 07:32:40] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-04-01 07:32:40] - Archiving was last executed without error 5 min 1s ago
INFO CoreConsole[2014-04-01 07:32:40] - Will process 0 websites with new visits since 5 min 0s
INFO CoreConsole[2014-04-01 07:32:40] - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script) , IDs: 1
INFO CoreConsole[2014-04-01 06:32:40] ---------------------------
INFO CoreConsole[2014-04-01 06:32:40] START
INFO CoreConsole[2014-04-01 06:32:40] Starting Piwik reports archiving…
INFO CoreConsole[2014-04-01 06:32:41] Archived website id = 1, period = day, Time elapsed: 1.113s
INFO CoreConsole[2014-04-01 06:32:42] Archived website id = 1, period = week, 29 visits, Time elapsed: 0.943s
INFO CoreConsole[2014-04-01 06:32:53] Archived website id = 1, period = month, 0 visits, Time elapsed: 10.732s
INFO CoreConsole[2014-04-01 06:33:00] Archived website id = 1, period = year, 47440 visits, Time elapsed: 7.205s
INFO CoreConsole[2014-04-01 06:33:00] Archived website id = 1, today = 0 visits, 4 API requests, Time elapsed: 20.004s [1/1 done]
INFO CoreConsole[2014-04-01 06:33:00] Done archiving!
INFO CoreConsole[2014-04-01 06:33:00] ---------------------------
INFO CoreConsole[2014-04-01 06:33:00] SUMMARY
INFO CoreConsole[2014-04-01 06:33:00] Total daily visits archived: 0
INFO CoreConsole[2014-04-01 06:33:00] Archived today’s reports for 1 websites
INFO CoreConsole[2014-04-01 06:33:00] Archived week/month/year for 1 websites
INFO CoreConsole[2014-04-01 06:33:00] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-04-01 06:33:00] Skipped 0 websites day archiving: existing daily reports are less than 3600 seconds old
INFO CoreConsole[2014-04-01 06:33:00] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-04-01 06:33:00] Total API requests: 4
INFO CoreConsole[2014-04-01 06:33:00] done: 1/1 100%, 0 v, 1 wtoday, 1 wperiods, 4 req, 20073 ms, no error
INFO CoreConsole[2014-04-01 06:33:00] Time elapsed: 20.074s
INFO CoreConsole[2014-04-01 06:33:00] ---------------------------
INFO CoreConsole[2014-04-01 06:33:00] SCHEDULED TASKS
INFO CoreConsole[2014-04-01 06:33:00] Starting Scheduled tasks…
INFO CoreConsole[2014-04-01 06:33:00] No task to run
INFO CoreConsole[2014-04-01 06:33:00] done
INFO CoreConsole[2014-04-01 06:33:00] ---------------------------


#10

I have made some modifications to import_logs.py following the instructions in thread: 301 Moved Permanently.

And now I can see pageview data as you can see in attached pic.