Piwik 2.04b7 error in import_logs.py


#1

Hello,

I upgraded yesterday to piwik 2.04b7 and since I got an error running import_logs.py .
I run exactly the same script every night and since yesterday ( date of my last umdate) I got that error :

/usr/bin/python /var/www/html/piwik/misc/log-analytics/import_logs.py --url=https://w3stat.unil.ch/piwik/ --idsite=562 --config=/var/www/html/piwik/config/config.ini.php --recorders=2 --log-hostname=www2.unil.ch --hostname=www2.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/tmp/stats/prod/access_full 2>&1
Traceback (most recent call last):
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 1707, in
config = Configuration()
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 567, in init
self._parse_args(self._create_parser())
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 557, in _parse_args
self.options.piwik_token_auth = self._get_token_auth()
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 604, in _get_token_auth
filename = subprocess.check_output(process, shell=True);
AttributeError: ‘module’ object has no attribute ‘check_output’

Any idea ?

Is there a safe way to downgrade piwik and go back to the latest stable version: 2.03

Thanks for your reply…


(Matthieu Aubry) #2

Thanks for the report. PLease try beta 8 which should fix this!


#3

Sorry but I got exactlythe same error with 2.03b8 .

I checked the import_logs.py script has bee update.

Thanks for another suggestion.


(Matthieu Aubry) #4

I cannot reproduce. Myabe it’s because of the particular log file? could you send a short log file which lets you reproduce the issue? or is it the same bug with all log files?


#5

I upgrade to 2.04b11 and again the same error in importlogs.py. What did change between 2.04b6 and b7 ?? On my side I have exactly the same script running…

same bug with all logfiles … means logfile iindependent…

PLease help !!


(Matthieu Aubry) #6

I tried the exact same command:


$ ./import_logs.py --url=http://localhost/piwik-master/ --idsite=1 --config=/home/matt/dev/piwik-master/config/config.ini.php --recorders=2 --log-hostname=www2.unil.ch --hostname=www2.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/log/apache2/access.log
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /var/log/apache2/access.log...
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
1600 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2000 lines parsed, 400 lines recorded, 57 records/sec (avg), 400 records/sec (current)
2000 lines parsed, 400 lines recorded, 49 records/sec (avg), 0 records/sec (current)
2000 lines parsed, 400 lines recorded, 44 records/sec (avg), 0 records/sec (current)
2000 lines parsed, 400 lines recorded, 39 records/sec (avg), 0 records/sec (current)

As you can see for me it’s working and the data is being imported.

Can you post the exact command that you type, and also attach here or on pastebin.com the small log file that you can reproduce this issue?


#7

I do not understand . Would it be a python problem on my installation. I run python 2.6.6 But why would it be related to the upgrade form 2.4b6 to 2.04b7 ???

I tried to remove several options in particular the --config one but I always get the same problem… I also try on several log file and in particular the famous logfile ( inattchment) I used for test with 13 entries for the other counting pageviews problem I reported on a different topic and always I get this error …

Does the error message mean something for you ? I am not an expert in python programming…

I copy here again my command ( same options as you ) and the results I got . I repeat everything was fine with 2.0.4b6. The problem just occur after 2.04b7 upgrade …I also put in attachment the config.ini.php fiel I sue in order for you to check the plugins I use…


[cmoullet@prdwstat log-analytics]$ ./import_logs.py --url=https://w3stat.unil.ch/piwik/ --idsite=574 --recorders=2 --log-hostname=www3.unil.ch --hostname=www3.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/tmp/stats/test/access_test
Traceback (most recent call last):
File “./import_logs.py”, line 1710, in
config = Configuration()
File “./import_logs.py”, line 567, in init
self._parse_args(self._create_parser())
File “./import_logs.py”, line 557, in _parse_args
self.options.piwik_token_auth = self._get_token_auth()
File “./import_logs.py”, line 604, in _get_token_auth
filename = subprocess.check_output(process, shell=True);
AttributeError: ‘module’ object has no attribute ‘check_output’


Please tell me if I can do something more on my side. I would really like to be able to run this import_logs script in order also to check if the pageview count is correct now…

Thanks for your help


(Matthieu Aubry) #8

It sounds like a bug we should fix. Please create a ticket at dev.piwik.org for this particular error (put full trace + command used + log file used + link to this ticket)
assign to 2.1 roadmap with Critical priority
we will then investigate


#9

hi i remeber several others had issues when using lower than python 2.7.1 perhaps an upgrade to python alone may help things? certainly an eaiy way to check…


#10

He re is wghat I get with python 3.3 … another error…

/usr/local/bin/python3.3 /var/www/html/piwik/misc/log-analytics/import_logs.py --url=https://w3stat.unil.ch/piwik/ --idsite=562 --config=/var/www/html/piwik/config/config.ini.php --recorders=2 --log-hostname=www2.unil.ch --hostname=www2.unil.ch --enable-static --enable-bots --enable-http-errors --enable-http-redirects --enable-reverse-dns --strip-query-string /var/tmp/stats/prod/access_full 2>&1
File “/var/www/html/piwik/misc/log-analytics/import_logs.py”, line 558
except Piwik.Error, e:
^
SyntaxError: invalid syntax


#11

how come idsite differnt second time in your script? i notice Matts example used http and your is https can you try a non https site?


(Matthieu Aubry) #12

Try with python 2.6 as the script is not yet compatible with 3.3

I’m not sure if we can maike it compatible with both versions at once


#13

Same thing with http… no change


#14

I was running with python 2.6.6 !!!


#15

Hi, this is because there is a python 2.7 subprocess call in the script (subprocess.check_output). This should be replaced with a 2.6 friendly call like Popen. As a quick hack, change line 607 (in 2.1-rc1) to:
filename = subprocess.Popen(process, shell=True, stdout=subprocess.PIPE).communicate()[0]

Also it might be an idea to pass the arguments as a list and not use shell=True, which is a potential security issue.

Hope this helps :slight_smile:

I’m now getting an auth error, but this is the first time i’ve used piwik so it’s probably something stupid i’m doing (i’ve tried passing both token-auth and login/password:
Traceback (most recent call last):
File “./import_logs.py”, line 1715, in
resolver = config.get_resolver()
File “./import_logs.py”, line 616, in get_resolver
return StaticResolver(self.options.site_id)
File “./import_logs.py”, line 961, in init
’SitesManager.getSiteFromId’, idSite=self.site_id
File “./import_logs.py”, line 943, in call_api
return cls._call_wrapper(cls._call_api, None, None, method, **kwargs)
File “./import_logs.py”, line 932, in _call_wrapper
raise Piwik.Error(message)
main.Error: Authorization Required


#16

… really stupid on my part, my piwik install was behind an http auth. removed it and everything working perfectly now. great product, really liking it so far :slight_smile:


(Matthieu Aubry) #17

As of Piwik 2.1 the Log analytics tool will require Python 2.7


#18

Hi Matt

This will then exclude all RHEL and derivative OSs (CentOS etc), which are quite popular! It does seem to work perfectly in 2.6 with that one line change. Here’s a udiff that also changes the subprocess to a ‘native’ call rather than using a shell call:

original.py 2014-02-13 08:42:27.631037608 +0000
+++ import_logs.py 2014-02-13 08:42:58.098411313 +0000
@@ -600,11 +600,12 @@
’…/…/misc/cron/updatetoken.php’),
)

  •        process =  "php " + updatetokenfile
    
  •                   process = ["php"]
    
  •        process.append(updatetokenfile)
           if self.options.enable_testmode:
    
  •            process = process + " --testmode"
    
  •            process.append("--testmode")
    
  •        filename    = subprocess.check_output(process, shell=True);
    
  •        filename    = subprocess.Popen(process, stdout=subprocess.PIPE).communicate()[0]
           credentials = open(filename, 'r').readline()
           credentials = credentials.split('\t')
           return credentials[1]

#19

OK finally with python 2.7 and piwik 2.1 rc1 I could finally make import_logs.py work !! Ouf !!
Now let-s check if I have the correct count (other ticket)


#20

This is a more elegant implementation, and brings us back to python 2.6+. Thanks diosmosis :slight_smile: