import_logs.py not working in 2.1

Is anyone using import_logs.py to analize data in 2.1?

In piwik 2.1 with NCSA extended logs (apache) I’m not able to view page actions but errors, statics and redirections.

Any clue?

Can you explain better what you mean? what is the difference from beefore 2.1 ?

Here is what would help:
put a small log file that you use. And the command you used.
Then, what you expected to see and what you saw instead?

I have seen other reports of users saying something changed but for us in our tests, all looks OK, so maybe we need to update our tests :slight_smile:

Using Python 2,7,3 Ubuntu 12.04.4 (LTS)
apache 2.2.*
php 5,3,*
Mysql 5,5,35

python /var/www/piwik/misc/log-analytics/import_logs.py --url=http://pub1.player-radio.ru /opt/logs/access.log --idsite=2 --log-format-name=icecast2 --strip-query-string --show-progress
Traceback (most recent call last):
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 1720, in
resolver = config.get_resolver()
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 621, in get_resolver
return StaticResolver(self.options.site_id)
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 966, in init
’SitesManager.getSiteFromId’, idSite=self.site_id
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 948, in call_api
return cls._call_wrapper(cls._call_api, None, None, method, **kwargs)
File “/var/www/piwik/misc/log-analytics/import_logs.py”, line 937, in _call_wrapper
raise Piwik.Error(message)
main.Error: Not Implemented

Sorry, but my log files to large of 200K, in archive too. I have fresh install Piwik 2.1, not update.

This is my command line:

python /var/www/html/logimport/misc/log-analytics/import_logs.py --recorders=8 --url=http://localhost/logimport/ test.log --login=admin --password=pass --idsite=1

As you can see there is no --enable-static, --emable-http-errors o --enable-http-redirects in the command line

After import I launch an import process:

/usr/bin/php archive.php --url=http://localhost/logimport/

There is an image with the result of the import. NO PAGEVIEWS.

I have attached the file test.log.png rename to test.log.

I need to upgrade from 1.12 to 2.1 but I am testing in a clean installation.

Thxs.

please try with latest Piwik beta version: I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo

Then if you still see this bug, post here a small log file that you have used to reproduce the issue, with the full command use (and you expected to see VS what you got). I will then try to reproduce as well!

Using Piwik 2.1.1b9

Command:

python /var/www/html/logimport/misc/log-analytics/import_logs.py --recorders=8 --url=http://localhost/logimport/ /tmp/test.log --login=admin --password=pass --token-auth=xxxxxxx --idsite=1

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /tmp/test.log…
500 lines parsed, 36 lines recorded, 35 records/sec (avg), 36 records/sec (current)
Purging Piwik archives for dates: 2014-03-30
To re-process these reports with your new update data, execute the piwik/misc/cron/archive.php script, or see: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo for more info.

Logs import summary

102 requests imported successfully
107 requests were downloads
398 requests ignored:
    0 invalid log lines
    11 requests done by bots, search engines, ...
    19 HTTP errors
    76 HTTP redirects
    292 requests to static resources (css, js, ...)
    0 requests did not match any known site
    0 requests did not match any requested hostname

Website import summary

102 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 1 seconds
Requests imported per second: 77.24 requests per second

Then:

/usr/bin/php /var/www/html/logimport/console core:archive --url=http://localhost/logimport/

The same result: No pageviews at all.

please try with 2.1.1-b10 -> does it fix the issue? I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo

Using: piwik-2.1.1-b10

Same test.log file. Same result. No pageview actions

python /var/www/html/logimport/misc/log-analytics/import_logs.py --recorders=8 --url=http://localhost/logimport/ /tmp/test30032014.log --login=admin --password=pass --token-auth=xxxxxxx --idsite=1

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /tmp/test30032014.log…
Purging Piwik archives for dates: 2014-03-30
To re-process these reports with your new update data, execute the piwik/misc/cron/archive.php script, or see: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo for more info.

Logs import summary

102 requests imported successfully
107 requests were downloads
398 requests ignored:
    0 invalid log lines
    11 requests done by bots, search engines, ...
    19 HTTP errors
    76 HTTP redirects
    292 requests to static resources (css, js, ...)
    0 requests did not match any known site
    0 requests did not match any requested hostname

Website import summary

102 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 0 seconds
Requests imported per second: 148.91 requests per second

/usr/bin/php /var/www/html/logimport/console core:archive --url=http://localhost/logimport/ -v

INFO CoreConsole[2014-04-01 07:32:39] ---------------------------
INFO CoreConsole[2014-04-01 07:32:39] INIT
INFO CoreConsole[2014-04-01 07:32:39] Piwik is installed at: http://localhost/logimport/index.php
INFO CoreConsole[2014-04-01 07:32:39] Running Piwik 2.1.1-b10 as Super User: piwikadmin
INFO CoreConsole[2014-04-01 07:32:40] ---------------------------
INFO CoreConsole[2014-04-01 07:32:40] NOTES
INFO CoreConsole[2014-04-01 07:32:40] - If you execute this script at least once per hour (or more often) in a crontab, you may disable ‘Browser trigger archiving’ in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-04-01 07:32:40] See the doc at: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo
INFO CoreConsole[2014-04-01 07:32:40] - Reports for today will be processed at most every 3600 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-04-01 07:32:40] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-04-01 07:32:40] - Archiving was last executed without error 5 min 1s ago
INFO CoreConsole[2014-04-01 07:32:40] - Will process 0 websites with new visits since 5 min 0s
INFO CoreConsole[2014-04-01 07:32:40] - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script) , IDs: 1
INFO CoreConsole[2014-04-01 06:32:40] ---------------------------
INFO CoreConsole[2014-04-01 06:32:40] START
INFO CoreConsole[2014-04-01 06:32:40] Starting Piwik reports archiving…
INFO CoreConsole[2014-04-01 06:32:41] Archived website id = 1, period = day, Time elapsed: 1.113s
INFO CoreConsole[2014-04-01 06:32:42] Archived website id = 1, period = week, 29 visits, Time elapsed: 0.943s
INFO CoreConsole[2014-04-01 06:32:53] Archived website id = 1, period = month, 0 visits, Time elapsed: 10.732s
INFO CoreConsole[2014-04-01 06:33:00] Archived website id = 1, period = year, 47440 visits, Time elapsed: 7.205s
INFO CoreConsole[2014-04-01 06:33:00] Archived website id = 1, today = 0 visits, 4 API requests, Time elapsed: 20.004s [1/1 done]
INFO CoreConsole[2014-04-01 06:33:00] Done archiving!
INFO CoreConsole[2014-04-01 06:33:00] ---------------------------
INFO CoreConsole[2014-04-01 06:33:00] SUMMARY
INFO CoreConsole[2014-04-01 06:33:00] Total daily visits archived: 0
INFO CoreConsole[2014-04-01 06:33:00] Archived today’s reports for 1 websites
INFO CoreConsole[2014-04-01 06:33:00] Archived week/month/year for 1 websites
INFO CoreConsole[2014-04-01 06:33:00] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-04-01 06:33:00] Skipped 0 websites day archiving: existing daily reports are less than 3600 seconds old
INFO CoreConsole[2014-04-01 06:33:00] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-04-01 06:33:00] Total API requests: 4
INFO CoreConsole[2014-04-01 06:33:00] done: 1/1 100%, 0 v, 1 wtoday, 1 wperiods, 4 req, 20073 ms, no error
INFO CoreConsole[2014-04-01 06:33:00] Time elapsed: 20.074s
INFO CoreConsole[2014-04-01 06:33:00] ---------------------------
INFO CoreConsole[2014-04-01 06:33:00] SCHEDULED TASKS
INFO CoreConsole[2014-04-01 06:33:00] Starting Scheduled tasks…
INFO CoreConsole[2014-04-01 06:33:00] No task to run
INFO CoreConsole[2014-04-01 06:33:00] done
INFO CoreConsole[2014-04-01 06:33:00] ---------------------------

I have made some modifications to import_logs.py following the instructions in thread: 301 Moved Permanently.

And now I can see pageview data as you can see in attached pic.