Import logs - Fatal Error: Forbidden

nld2756 · June 15, 2013, 9:11am

Getting this error while attempting to execute the import:

Fatal Error: Forbidden

python import_logs.py --url=http://MY-HOSTNAME/track/ /usr/share/nginx/log/access.log

All the while I am in root, permissions should not be a problem…

Any similar problems? Any culprits?

Much appreciated

matthieu · June 16, 2013, 9:33pm

forbidden means the --url is wrong, or check your serve error logs for more info

foobard · April 28, 2014, 12:26am

I am getting the same error. The url is correct (I copy and paste it into my browser, and it gives me the Piwik login screen), and the apache error logs show nothing from today. Here’s my console output:

$./import_logs.py --url=https://www.mysite.com/pathto/piwik/ /var/log/apache/access.log --debug
2014-04-28 00:10:29,205: [DEBUG] Accepted hostnames: all
2014-04-28 00:10:29,205: [DEBUG] Piwik URL is: http://www.mysite.com/piwik/
2014-04-28 00:10:29,205: [DEBUG] No token-auth specified
2014-04-28 00:10:29,205: [DEBUG] No credentials specified, reading them from “[…]/config/config.ini.php”
2014-04-28 00:10:29,347: [DEBUG] Authentication token token_auth is: [REDACTED]
2014-04-28 00:10:29,347: [DEBUG] Resolver: dynamic
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-04-28 00:10:29,349: [DEBUG] Launched recorder
Parsing log […]/log/apache/access.log…
2014-04-28 00:10:29,350: [DEBUG] Detecting the log format
2014-04-28 00:10:29,350: [DEBUG] Check format icecast2
2014-04-28 00:10:29,350: [DEBUG] Format icecast2 does not match
2014-04-28 00:10:29,350: [DEBUG] Check format iis
2014-04-28 00:10:29,350: [DEBUG] Format iis does not match
2014-04-28 00:10:29,351: [DEBUG] Check format common
2014-04-28 00:10:29,351: [DEBUG] Format common does not match
2014-04-28 00:10:29,351: [DEBUG] Check format common_vhost
2014-04-28 00:10:29,351: [DEBUG] Format common_vhost matches
2014-04-28 00:10:29,351: [DEBUG] Check format nginx_json
2014-04-28 00:10:29,351: [DEBUG] Format nginx_json does not match
2014-04-28 00:10:29,351: [DEBUG] Check format s3
2014-04-28 00:10:29,352: [DEBUG] Format s3 does not match
2014-04-28 00:10:29,352: [DEBUG] Check format ncsa_extended
2014-04-28 00:10:29,352: [DEBUG] Format ncsa_extended does not match
2014-04-28 00:10:29,352: [DEBUG] Check format common_complete
2014-04-28 00:10:29,352: [DEBUG] Format common_complete does not match
2014-04-28 00:10:29,352: [DEBUG] Format common_vhost is the best match
2014-04-28 00:10:29,424: [DEBUG] Site ID for hostname [www.mysite.com] not in cache
2014-04-28 00:10:29,563: [DEBUG] Error when connecting to Piwik: HTTP Error 403: Forbidden
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-04-28 00:10:31,612: [DEBUG] Error when connecting to Piwik: HTTP Error 403: Forbidden
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-04-28 00:10:33,657: [DEBUG] Error when connecting to Piwik: HTTP Error 403: Forbidden
Fatal error: Forbidden
You can restart the import of “[…]/var/log/apache/access.log” from the point it failed by specifying --skip=5 on the command line.

And of course, trying with --skip=5 produces the same error.

I note this problem mentioned two years ago (!) by user Hexxer, here:

github.com/matomo-org/matomo

Log analytics list of improvements

opened 01:13PM - 31 May 12 UTC

closed 08:58PM - 12 Mar 15 UTC

mattab

Enhancement duplicate

In Piwik 1.8 we released the great new feature to [import access logs and genera…te statistics](http://piwik.org/log-analytics/). The V1 release works very well (it was tracked in #703), but there are ideas to improve it. This ticket is a placeholder of all ideas and discussions related to the Log Analytics feature! New features - **Track non-bot activity only**. When --enable-bots is not specified, it would be a nice improvement if we: - exclude visits with more than 150 actions per visitorID to block crawlers (detected at the python level by counting requests for that IP in the queue) - exclude visits that do not have User Agent or beyond the very basic ones used by all bots - exclude all requests when one of the first ones is for /robots.txt -- if we see a robots.txt in the middle we could stop tracking subsequent requests - check that /index.php?minimize_js=file.js is counted as a static file since it ends in .js After that bots & crawlers detection would be much better. - Support Accept-Language header and forward to piwik via the &lang= parameter. That might also be useful to some users who need to use this data in a custom plugin. - we could make it easy to **delete logs for one day so to reimport one log file** - This would be a new option to the python script. It would reuse the code from the Log Delete feature, but would only delete one day. The python script would call the CoreAdmin API for example, deleting this single day for a given website. This would allow to easily re-import data that didn't work the first time or was bogus. - **Detect when log-lines are re-imported and only import them once**. - Implementation: add new table piwik_log_lines (hash_tracking_request, day )) - In Piwik Tracker, before looping on the bulk requests, SELECT all the log lines that have already been processed on this day (WHERE hash_tracking_request IN (a,b,c,d) AND day=?) & Skip these requests from import - After bulk requests are processed in piwik.php process, INSERT in bulk (hash, day) - By default this feature would be enabled only for "Log import" script, - via a parameter that we know is the log import (&li=1 /import_logs=1) - but may be later useful to all users of [Tracking API](http://piwik.org/docs/tracking-api/) for general deduping service. **PERFORMANCE**' How to debug performance? First of all, you can run the script with --dry-run to see how many log lines per second are parsed. It typically should be between 2,000 and 5,000. When you don't do a dry run, it will insert new pageviews and visits calling Piwik API. **Other tickets** - #3867 cannot resume with line number reported by skip for ncsa_extended log format - #4045 autodetection hangs on a weird formatted line

I have googled, I have searched the archives, the bug tracker contains no clue. Would really appreciate some kind soul taking mercy on me here.

foobard · April 28, 2014, 12:29am

I’m getting the same error, and there’s no docco anywhere to tell me how to fix it.

The url is correct (I copy and paste it into my browser, and it gives me the Piwik login screen), and the apache error logs show nothing from today. Here’s my console output:

$./import_logs.py --url=https://www.mysite.com/pathto/piwik/ /var/log/apache/access.log --debug
2014-04-28 00:10:29,205: [DEBUG] Accepted hostnames: all
2014-04-28 00:10:29,205: [DEBUG] Piwik URL is: http://www.mysite.com/piwik/
2014-04-28 00:10:29,205: [DEBUG] No token-auth specified
2014-04-28 00:10:29,205: [DEBUG] No credentials specified, reading them from "[…]/config/config.ini.php"
2014-04-28 00:10:29,347: [DEBUG] Authentication token token_auth is: [REDACTED]
2014-04-28 00:10:29,347: [DEBUG] Resolver: dynamic
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-04-28 00:10:29,349: [DEBUG] Launched recorder
Parsing log […]/log/apache/access.log…
2014-04-28 00:10:29,350: [DEBUG] Detecting the log format
2014-04-28 00:10:29,350: [DEBUG] Check format icecast2
2014-04-28 00:10:29,350: [DEBUG] Format icecast2 does not match
2014-04-28 00:10:29,350: [DEBUG] Check format iis
2014-04-28 00:10:29,350: [DEBUG] Format iis does not match
2014-04-28 00:10:29,351: [DEBUG] Check format common
2014-04-28 00:10:29,351: [DEBUG] Format common does not match
2014-04-28 00:10:29,351: [DEBUG] Check format common_vhost
2014-04-28 00:10:29,351: [DEBUG] Format common_vhost matches
2014-04-28 00:10:29,351: [DEBUG] Check format nginx_json
2014-04-28 00:10:29,351: [DEBUG] Format nginx_json does not match
2014-04-28 00:10:29,351: [DEBUG] Check format s3
2014-04-28 00:10:29,352: [DEBUG] Format s3 does not match
2014-04-28 00:10:29,352: [DEBUG] Check format ncsa_extended
2014-04-28 00:10:29,352: [DEBUG] Format ncsa_extended does not match
2014-04-28 00:10:29,352: [DEBUG] Check format common_complete
2014-04-28 00:10:29,352: [DEBUG] Format common_complete does not match
2014-04-28 00:10:29,352: [DEBUG] Format common_vhost is the best match
2014-04-28 00:10:29,424: [DEBUG] Site ID for hostname [www.mysite.com] not in cache
2014-04-28 00:10:29,563: [DEBUG] Error when connecting to Piwik: HTTP Error 403: Forbidden
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-04-28 00:10:31,612: [DEBUG] Error when connecting to Piwik: HTTP Error 403: Forbidden
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2504 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-04-28 00:10:33,657: [DEBUG] Error when connecting to Piwik: HTTP Error 403: Forbidden
Fatal error: Forbidden
You can restart the import of “[…]/var/log/apache/access.log” from the point it failed by specifying --skip=5 on the command line.

And of course, trying with --skip=5 produces the same error.

I have googled, I have searched the archives, the bug tracker contains no clue. Would really appreciate some kind soul taking mercy on me here.

matthieu · April 28, 2014, 7:10am

Error when connecting to Piwik: HTTP Error 403: Forbidden

can you please check your web server error logs for some clue as to what this error could be?

foobard · April 28, 2014, 12:35pm

matt,

thanks for the reply. As I mentioned before, nothing appears in my apache error.log.

To be absolutely sure, I double checked just now. The only thing I see in the error log is an hourly restart:

[Mon Apr 28 12:21:12.550235 2014] [mpm_event:notice] [pid 155:tid 3344372756640] AH00493: SIGUSR1 received. Doing graceful restart
[Mon Apr 28 12:21:12.626293 2014] [auth_digest:notice] [pid 155:tid 3344372856740] AH01757: generating secret for digest authentication …
[Mon Apr 28 12:21:12.627887 2014] [mpm_event:notice] [pid 155:tid 3344372856740] AH00489: Apache/2.4.6 (Unix) configured – resuming normal operations
[Mon Apr 28 12:21:12.627901 2014] [core:notice] [pid 155:tid 3374372856640] AH00094: Command line: ‘/usr/sbin/apache2 -d /srv/data/.config/apache -f apache2.conf’

This appears every hour at :21 minutes after the hour.

After running the import script and getting the previously posted error message, I tailed the error log. No change. Nothing.

Note that I am running Piwik on Gandi.net’s “Simple Hosting” service behind a Varnish cache. Does that help?

thanks
j

andronux · April 30, 2014, 12:41am

Hello everyone,

my name is Karim and I work at Gandi.

I just want to let you know that we are working on identifying the issue.

Piwik was running fine on our Simple Hosting offer before the version Piwik 2.10.

It seems that since Piwik 2.10, “HTTP Error 403: Forbidden” error is raised.

I’ll reply to this thread as soon as we find out the problem.

Best regards,

Karim

Gandi.net

matthieu · April 30, 2014, 2:19am

Hi Karim, great to hear this, we look forward to your findings!