Piwik 2.04b7 error in import_logs.py

imoullet · February 4, 2014, 9:42am

Hello,

I upgraded yesterday to piwik 2.04b7 and since I got an error running import_logs.py .
I run exactly the same script every night and since yesterday ( date of my last umdate) I got that error :

/usr/bin/python /var/www/html/piwik/misc/log-analytics/import_logs.py --url=https://w3stat.unil.ch/piwik/ --idsite=562 --config=/var/www/html/piwik/config/config.ini.php --recorders=2 --log-hostname=www2.unil.ch --hostname=www2.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/tmp/stats/prod/access_full 2>&1
Traceback (most recent call last):
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 1707, in
config = Configuration()
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 567, in init
self._parse_args(self._create_parser())
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 557, in _parse_args
self.options.piwik_token_auth = self._get_token_auth()
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 604, in _get_token_auth
filename = subprocess.check_output(process, shell=True);
AttributeError: ‘module’ object has no attribute ‘check_output’

Any idea ?

Is there a safe way to downgrade piwik and go back to the latest stable version: 2.03

Thanks for your reply…

matthieu · February 4, 2014, 7:45pm

Thanks for the report. PLease try beta 8 which should fix this!

imoullet · February 5, 2014, 7:42am

Sorry but I got exactlythe same error with 2.03b8 .

I checked the import_logs.py script has bee update.

Thanks for another suggestion.

matthieu · February 5, 2014, 12:21pm

I cannot reproduce. Myabe it’s because of the particular log file? could you send a short log file which lets you reproduce the issue? or is it the same bug with all log files?

imoullet · February 6, 2014, 8:43am

I upgrade to 2.04b11 and again the same error in importlogs.py. What did change between 2.04b6 and b7 ?? On my side I have exactly the same script running…

same bug with all logfiles … means logfile iindependent…

PLease help !!

matthieu · February 6, 2014, 10:58pm

I tried the exact same command:


$ ./import_logs.py --url=http://localhost/piwik-master/ --idsite=1 --config=/home/matt/dev/piwik-master/config/config.ini.php --recorders=2 --log-hostname=www2.unil.ch --hostname=www2.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/log/apache2/access.log
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /var/log/apache2/access.log...
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2000 lines parsed, 400 lines recorded, 57 records/sec (avg), 400 records/sec (current)
2000 lines parsed, 400 lines recorded, 49 records/sec (avg), 0 records/sec (current)
2000 lines parsed, 400 lines recorded, 44 records/sec (avg), 0 records/sec (current)
2000 lines parsed, 400 lines recorded, 39 records/sec (avg), 0 records/sec (current)

As you can see for me it’s working and the data is being imported.

Can you post the exact command that you type, and also attach here or on pastebin.com the small log file that you can reproduce this issue?

imoullet · February 7, 2014, 10:09am

I do not understand . Would it be a python problem on my installation. I run python 2.6.6 But why would it be related to the upgrade form 2.4b6 to 2.04b7 ???

I tried to remove several options in particular the --config one but I always get the same problem… I also try on several log file and in particular the famous logfile ( inattchment) I used for test with 13 entries for the other counting pageviews problem I reported on a different topic and always I get this error …

Does the error message mean something for you ? I am not an expert in python programming…

I copy here again my command ( same options as you ) and the results I got . I repeat everything was fine with 2.0.4b6. The problem just occur after 2.04b7 upgrade …I also put in attachment the config.ini.php fiel I sue in order for you to check the plugins I use…

[cmoullet@prdwstat log-analytics]$ ./import_logs.py --url=https://w3stat.unil.ch/piwik/ --idsite=574 --recorders=2 --log-hostname=www3.unil.ch --hostname=www3.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/tmp/stats/test/access_test
Traceback (most recent call last):
File “./import_logs.py”, line 1710, in
config = Configuration()
File “./import_logs.py”, line 567, in init
self._parse_args(self._create_parser())
File “./import_logs.py”, line 557, in _parse_args
self.options.piwik_token_auth = self._get_token_auth()
File “./import_logs.py”, line 604, in _get_token_auth
filename = subprocess.check_output(process, shell=True);
AttributeError: ‘module’ object has no attribute ‘check_output’

Please tell me if I can do something more on my side. I would really like to be able to run this import_logs script in order also to check if the pageview count is correct now…

Thanks for your help

matthieu · February 8, 2014, 5:08am

It sounds like a bug we should fix. Please create a ticket at dev.piwik.org for this particular error (put full trace + command used + log file used + link to this ticket)
assign to 2.1 roadmap with Critical priority
we will then investigate

lesjokolat · February 8, 2014, 5:33pm

hi i remeber several others had issues when using lower than python 2.7.1 perhaps an upgrade to python alone may help things? certainly an eaiy way to check…

imoullet · February 11, 2014, 3:38pm

He re is wghat I get with python 3.3 … another error…

/usr/local/bin/python3.3 /var/www/html/piwik/misc/log-analytics/import_logs.py --url=https://w3stat.unil.ch/piwik/ --idsite=562 --config=/var/www/html/piwik/config/config.ini.php --recorders=2 --log-hostname=www2.unil.ch --hostname=www2.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/tmp/stats/prod/access_full 2>&1
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 558
except Piwik.Error, e:
^
SyntaxError: invalid syntax

lesjokolat · February 11, 2014, 7:22pm

how come idsite differnt second time in your script? i notice Matts example used http and your is https can you try a non https site?

matthieu · February 11, 2014, 8:39pm

Try with python 2.6 as the script is not yet compatible with 3.3

I’m not sure if we can maike it compatible with both versions at once

imoullet · February 12, 2014, 7:54am

Same thing with http… no change

imoullet · February 12, 2014, 7:55am

I was running with python 2.6.6 !!!

dom · February 12, 2014, 8:51am

Hi, this is because there is a python 2.7 subprocess call in the script (subprocess.check_output). This should be replaced with a 2.6 friendly call like Popen. As a quick hack, change line 607 (in 2.1-rc1) to:
filename = subprocess.Popen(process, shell=True, stdout=subprocess.PIPE).communicate()[0]

Also it might be an idea to pass the arguments as a list and not use shell=True, which is a potential security issue.

Hope this helps

I’m now getting an auth error, but this is the first time i’ve used piwik so it’s probably something stupid i’m doing (i’ve tried passing both token-auth and login/password:
Traceback (most recent call last):
File “./import_logs.py”, line 1715, in
resolver = config.get_resolver()
File “./import_logs.py”, line 616, in get_resolver
return StaticResolver(self.options.site_id)
File “./import_logs.py”, line 961, in init
’SitesManager.getSiteFromId’, idSite=self.site_id
File “./import_logs.py”, line 943, in call_api
return cls._call_wrapper(cls._call_api, None, None, method, **kwargs)
File “./import_logs.py”, line 932, in _call_wrapper
raise Piwik.Error(message)
main.Error: Authorization Required

dom · February 12, 2014, 8:59am

… really stupid on my part, my piwik install was behind an http auth. removed it and everything working perfectly now. great product, really liking it so far

matthieu · February 12, 2014, 9:56pm

As of Piwik 2.1 the Log analytics tool will require Python 2.7

dom · February 13, 2014, 8:48am

Hi Matt

This will then exclude all RHEL and derivative OSs (CentOS etc), which are quite popular! It does seem to work perfectly in 2.6 with that one line change. Here’s a udiff that also changes the subprocess to a ‘native’ call rather than using a shell call:

— original.py 2014-02-13 08:42:27.631037608 +0000
+++ import_logs.py 2014-02-13 08:42:58.098411313 +0000
@@ -600,11 +600,12 @@
’…/…/misc/cron/updatetoken.php’),
)

       process =  "php " + updatetokenfile

```
                  process = ["php"]
```

       process.append(updatetokenfile)
       if self.options.enable_testmode:

           process = process + " --testmode"

           process.append("--testmode")

       filename    = subprocess.check_output(process, shell=True);

       filename    = subprocess.Popen(process, stdout=subprocess.PIPE).communicate()[0]
       credentials = open(filename, 'r').readline()
       credentials = credentials.split('\t')
       return credentials[1]

imoullet · February 14, 2014, 9:31am

OK finally with python 2.7 and piwik 2.1 rc1 I could finally make import_logs.py work !! Ouf !!
Now let-s check if I have the correct count (other ticket)

dom · February 21, 2014, 12:47am

https://github.com/piwik/piwik/commit/3c6690941b56e9af732f4dcf5a95435be0c2794e

This is a more elegant implementation, and brings us back to python 2.6+. Thanks diosmosis