import_logs.py error since Piwik 1.10.1

Michel · January 24, 2013, 3:44pm

Hi,

i am testing import_logs from several days.
It works fine but since i have upgraded to 1.10 i have this error:
2013-01-24 16:08:46,335: [DEBUG] cannot parse tracker response, should be valid JSON
2013-01-24 16:08:46,335: [DEBUG] Error when connecting to Piwik:

that’s because server side there is php error:
PHP Fatal error: Cannot use object of type stdClass as array in /var/www/piwik/core/Common.php on line 885

The script line:
python26 ./import_logs.py --debug --token-auth=xxx --recorders=4 --url=http://webstats.xxx.fr --idsite=75 access.log.200

One idea

lesjokolat · January 24, 2013, 3:54pm

What version of python and php are you running on this machine?

Michel · January 24, 2013, 4:00pm

Python is 2.6.8
Php is 5.1.6 (a little old we must migrate.
If you mean that’s the php version, i wait the upgrade before retry.

lesjokolat · January 24, 2013, 4:02pm

I would at least use python 2.7.3 or higher.

The php should work, php 5.4.x I ve seen here on the threads is quite faster.

Could be php but try python first.

Michel · January 24, 2013, 4:08pm

well,

first i try to upgrade python.

thanks

jerome · January 30, 2013, 4:45pm

Hello,

I have the same problems :

…
2013-01-30 17:27:10,094: [DEBUG] Detecting the log format
2013-01-30 17:27:10,094: [DEBUG] Format ncsa_extended matches
2013-01-30 17:27:11,081: [DEBUG] cannot parse tracker response, should be valid JSON
2013-01-30 17:27:11,085: [DEBUG] Error when connecting to Piwik: <urlopen error

…

os is CentOS release 5.7
python is 2.7.3
php is 5.1.6

packages json are :
Installed Packages
json-c.x86_64 0.10-2.el5 installed
json-c-devel.x86_64 0.10-2.el5 installed
json-c-doc.x86_64 0.10-2.el5 installed
json_diff.noarch 1.3.3-1.el5 installed
json_simple.x86_64 1.1-2.el5 installed
php-pecl-json.x86_64 1.2.1-4.el5 installed
python-anyjson.noarch 0.3-1.el5 installed
python-cjson.x86_64 1.0.5-2.el5 installed
python-demjson.noarch 1.3-4.el5 installed
python-json.noarch 3.4-3.el5 installed
python-jsonpickle.noarch 0.3.1-2.1.el5 installed
python-simplejson.x86_64 2.0.9-8.el5 installed
python-turbojson.noarch 1.2.1-9.el5 installed
python26-simplejson.x86_64 2.1.1-1.el5 installed
ruby-json.x86_64 1.4.3-3.el5.1 installed
rubygem-json.x86_64 1.4.3-3.el5.1 installed

what could be the track investigations ?

Thanks a lot
Jérôme

lesjokolat · January 30, 2013, 6:56pm

is it possible you have forgotten the path to the access log?

your script

python26 ./import_logs.py --debug --token-auth=xxx --recorders=4 --url=http://webstats.xxx.fr --idsite=75 access.log.200

but i think needs to add absolute path

python26 ./import_logs.py --debug --token-auth=xxx --recorders=4 --url=http://webstats.xxx.fr --idsite=75 /path/to/access.log.200

Any way you can try to see if this helps?

Michel · January 31, 2013, 8:40am

Hi,

no it’s not a path problem.
We can’t upgarde python version, because the policy is to use package only and python2.7 is not avalaible on CentOS.
On the other side, i have change php version , up to 5.3 and its better.
But something is strange for me “0 lines recorded” it’s possible ?

Parsing log uhp.log.200…
2716 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3581 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3581 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3581 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3581 lines parsed, 540 lines recorded, 107 records/sec (avg), 540 records/sec (current)
3968 lines parsed, 800 lines recorded, 133 records/sec (avg), 260 records/sec (current)
3968 lines parsed, 800 lines recorded, 114 records/sec (avg), 0 records/sec (current)

jerome · January 31, 2013, 3:17pm

Hello,

yes, I confirm that it’s not a path problem !

here, my command :

[root@FRM000P06ASU:/production/www/clients/nasv2/home/www/piwik/misc/log-analytics]

/usr/local/bin/python2.7 ./import_logs.py -d --url=http://nasv2.itc.integra.fr/piwik/ --idsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots --show-progress-delay=1 /production/PARTAGE1/URCHIN_6.500/var/access.log

I’ll try to update PHP to 5.4

To follow
Thanks

Jérôme

lesjokolat · January 31, 2013, 5:07pm

Jerome could you alos update to python 2.7.3 if the php upgrade should fail?

jerome · February 7, 2013, 2:34pm

hello,

ultimately, we have made a new VM with Centos 6.3 + Php 5.3 + Mysql 5.1.67 + Python 2.6.6
and it’s OK

now, another problem :
7920 seconds to process a 100 MB log !!!
too slow !!

I’ll have to see how improve these imports !!

Jérôme