import_logs.py fails to import access logs

Support & Bugs
1

Hello,

I have been running the same process for months without issues. Now (I have upgraded to 2.2.0), import_logs.py immediately fails with the message:


# python /var/webs/wwwpiwik/www/misc/log-analytics/import_logs.py --url=http://wstat1.noa.gr \
> /var/webs/wwwgein2/log/access_log.1 --idsite=1 --recorders=4 --enable-http-errors \
> --enable-http-redirects --enable-static --enable-bots --enable-reverse-dns
Fatal error: `php /var/webs/wwwpiwik/www/misc/cron/updatetoken.php` failed with error: PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to allocate 523800 bytes) in Unknown on line 0
.
Reponse code was: 1. You can alternatively run the importer using the --login and --password option

I am on CentOS 6.5 x86_64 running php 5.3.3 with php-fpm, python 2.6.6 and nginx 1.6.0.

I increased memory_limit (in php-fpm and in php.ini) up to 1536MB but it still fails.

Can you please advise me on what may be the problem?

Thanks in advance,
Nick

2

Just to add that the access log file is less than 100 MB.

3

I found that the error was exactly what happens in this thread: 301 Moved Permanently

So, I had to add --login and --password parameters when running the import_logs.py script and everything went fine.

However, this wasn’t needed in past piwik versions.

Also, interestingly, if we don’t specify --login and --password, import_logs.py seems to be looking for superuser credentials in:


2014-04-25 21:46:18,167: [DEBUG] No credentials specified, reading them from "/path/to/piwik/config/config.ini.php"

…yet, in this file (config.ini.php) there are no superuser credentials stored!

I don’t know where piwik is now storing superuser credentials (probably in the db?) but it seems in previous versions piwik did store superuser data in the above file, because in config.ini.sample.php there is a [superuser] area, which is missing from config.ini.php.

I hope someone can shed some light on the above.

Also, a question: when running successfully with --login and --password in debug mode, I can see a line:


2014-04-25 21:55:36,682: [DEBUG] No token-auth specified
2014-04-25 21:55:36,682: [DEBUG] Using credentials: (login = yyyyyyyyyyyy, password = zzzzzzzzzzzzzzzzz)
2014-04-25 21:55:36,882: [DEBUG] Authentication token token_auth is: xxxxxxxxxxxxxxxxxxxxxxx

Can we use the above displayed token_auth value, e.g.: “import_logs.py --token-auth=xxxxxxxxxxxxxxxxxxxxxxx …” INSTEAD OF using --login and --password ?

Please advise.

Thanks,
Nick

4

The reason it works when specifying --login and --password is because it bypasses php /var/webs/wwwpiwik/www/misc/cron/updatetoken.php command.

The error was:

php /var/webs/wwwpiwik/www/misc/cron/updatetoken.php failed with error: PHP Fatal error: Allowed memory size of 262144 bytes exhausted (tried to allocate 523800 bytes) in Unknown on line 0

It looks like the php executable does not have enough memory. can you try


$ php -i | grep memory