Problem with Log Analytics - It seems to import something but no data is shown


#1

Hello,

I hope you can help me.
I tried to use Lo Analytics for importing my access logs. This ist the command:
./import_logs.py --url=URl-to-Piwik logfilename --idsite=2

This seems to work fine and I get this report:


0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log ../../../../logfiles/082014/access_log_2014-08-29_01-03.txt...
4968 lines parsed, 800 lines recorded, 802 records/sec (avg), 800 records/sec (current)
8407 lines parsed, 1600 lines recorded, 802 records/sec (avg), 800 records/sec (current)
11840 lines parsed, 2600 lines recorded, 868 records/sec (avg), 1000 records/sec (current)
14715 lines parsed, 3600 lines recorded, 899 records/sec (avg), 1000 records/sec (current)
16085 lines parsed, 4600 lines recorded, 920 records/sec (avg), 1000 records/sec (current)

Purging Piwik archives for dates: 2014-08-28 2014-08-27

To re-process these reports with your newly imported data, execute the following command: 
$ /path/to/piwik/console core:archive --url=http://example/piwik/

Reference: http://piwik.org/docs/setup-auto-archiving/ 

Logs import summary
-------------------

    4817 requests imported successfully
    134 requests were downloads
    11268 requests ignored:
        1 invalid log lines
        2156 requests done by bots, search engines, ...
        33 HTTP errors
        424 HTTP redirects
        8654 requests to static resources (css, js, ...)
        0 requests did not match any known site
        0 requests did not match any requested hostname

Website import summary
----------------------

    4817 requests imported to 1 sites
        1 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 5 seconds
    Requests imported per second: 839.93 requests per second

Than I take a look at the GUI and I don’t see any data.
Then I do this command:./console core:archive --url=URl-to-Piwik

I get this report:


INFO CoreConsole[2014-09-06 20:07:38] [672b3] ---------------------------
INFO CoreConsole[2014-09-06 20:07:38] [672b3] INIT
INFO CoreConsole[2014-09-06 20:07:38] [672b3] Piwik is installed at: URl-to-Piwik
INFO CoreConsole[2014-09-06 20:07:38] [672b3] Running Piwik 2.6.1 as Super User
INFO CoreConsole[2014-09-06 20:07:38] [672b3] ---------------------------
INFO CoreConsole[2014-09-06 20:07:38] [672b3] NOTES
INFO CoreConsole[2014-09-06 20:07:38] [672b3] - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-09-06 20:07:38] [672b3]   See the doc at: http://piwik.org/docs/setup-auto-archiving/
INFO CoreConsole[2014-09-06 20:07:38] [672b3] - Reports for today will be processed at most every 3600 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-09-06 20:07:38] [672b3] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-09-06 20:07:38] [672b3] - Archiving was last executed without error 5 hours 32 min ago
INFO CoreConsole[2014-09-06 20:07:38] [672b3] - Will process 1 websites with new visits since 5 hours 32 min , IDs: 1
INFO CoreConsole[2014-09-06 20:07:38] [672b3] - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script) , IDs: 2
INFO CoreConsole[2014-09-06 20:07:38] [672b3] ---------------------------
INFO CoreConsole[2014-09-06 20:07:38] [672b3] START
INFO CoreConsole[2014-09-06 20:07:38] [672b3] Starting Piwik reports archiving...
INFO CoreConsole[2014-09-06 20:07:39] [672b3] Archived website id = 1, period = day, 275 visits in last last2 days, 143 visits today, Time elapsed: 0.417s
INFO CoreConsole[2014-09-06 20:07:40] [672b3] Archived website id = 1, period = week, 1627 visits in last last2 weeks, 765 visits this week, Time elapsed: 1.622s
INFO CoreConsole[2014-09-06 20:07:42] [672b3] Archived website id = 1, period = month, 6263 visits in last last2 months, 765 visits this month, Time elapsed: 1.970s
INFO CoreConsole[2014-09-06 20:07:50] [672b3] Archived website id = 1, period = year, 28833 visits in last last2 years, 25085 visits this year, Time elapsed: 7.100s
INFO CoreConsole[2014-09-06 20:07:50] [672b3] Archived website id = 1, 4 API requests, Time elapsed: 11.110s [1/2 done]
INFO CoreConsole[2014-09-06 20:07:50] [672b3] Archived website id = 2, period = day, 0 visits in last last52 days, 0 visits today, Time elapsed: 0.898s
INFO CoreConsole[2014-09-06 20:07:51] [672b3] Archived website id = 2, period = week, 0 visits in last last2 weeks, 0 visits this week, Time elapsed: 0.530s
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Archived website id = 2, period = month, 0 visits in last last2 months, 0 visits this month, Time elapsed: 0.620s
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Archived website id = 2, period = year, 0 visits in last last2 years, 0 visits this year, Time elapsed: 0.560s
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Archived website id = 2, 4 API requests, Time elapsed: 2.609s [2/2 done]
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Done archiving!
INFO CoreConsole[2014-09-06 20:07:52] [672b3] ---------------------------
INFO CoreConsole[2014-09-06 20:07:52] [672b3] SUMMARY
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Total visits for today across archived websites: 143
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Archived today's reports for 2 websites
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Archived week/month/year for 2 websites
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Skipped 0 websites day archiving: existing daily reports are less than 3600 seconds old
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Total API requests: 8
INFO CoreConsole[2014-09-06 20:07:52] [672b3] done: 2/2 100%, 143 vtoday, 2 wtoday, 2 wperiods, 8 req, 13723 ms, no error
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Time elapsed: 13.723s
INFO CoreConsole[2014-09-06 20:07:52] [672b3] ---------------------------
INFO CoreConsole[2014-09-06 20:07:52] [672b3] SCHEDULED TASKS
INFO CoreConsole[2014-09-06 20:07:52] [672b3] Starting Scheduled tasks...
INFO CoreConsole[2014-09-06 20:07:52] [672b3]  No task to run
INFO CoreConsole[2014-09-06 20:07:52] [672b3] done
INFO CoreConsole[2014-09-06 20:07:52] [672b3] ---------------------------

This report looks also very good but it doesn’t seem to find any data for the website with the id 2.
What did I do wrong? There are no errors found at the first report.
Thank you for your help!

Regards,
Jens


(Matthieu Aubry) #2

there should be data there, can you check you look at the right website and right date?


#3

Hey Matt,

I hope I have the right informations for you!
This is a screenshots of the GUI showing the visitors of the last 30 days:
Visitors for the last 30 days

This is a screenshot of the visitor log:
Visitor log

Looking at the settings and the websites, where I can see what websites I have in piwik, the site has the number 2.
Do I misunderstood something?

Regards,
Jens


#4

Hi Matt,

I hope I have the right data for you.
Here I have a screenshot of the view showing the visitors for the kast 30 days:
Visitors last 30 days

This is a screenshot of the visitor log:
Visitor log

When I look at settings where all my websites are configured the site has the number 2.
Did I misunderstood something?

Regards,
Jens


(Matthieu Aubry) #5

is the screenshot for idsite=1 for which stats were imported? strange… do you see any error in error logs ?


#6

I have looked for error messages but there is nothing. If I take an site id which doesn’t exist i can see an error message in the log.
The access log shows HTTP 200 Codes …

I don’t what is wrong.
I also created a new website in piwik with id 3 and tried to import the data with the same result. Everything seems to be ok but there are no entries imported …


(Matthieu Aubry) #7

do you still have issues with 2.7.0-b3 ?


#8

Using 2.8.1 I am having the same issue, see output of -ddd below. Using the following options:

…/import_logs.py --url=http://piwik.xxx.com FILE --idsite=THE_ID --enable-http-errors --enable-reverse-dns --enable-static --download-extensions=doc,pdf,zip,deploy,application,manifest,vsto -dddd

Tried also with --disable-bulk-tracking and --debug-force-one-hit-every-Ns=5, but no differences.

I can send load file by direct mail.


2014-10-22 10:41:37,247: [DEBUG] Resolver: static
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-10-22 10:41:37,398: [DEBUG] Launched recorder
Parsing log access-log.aws.20141022.txt.temp.cleaned...
2014-10-22 10:41:37,398: [DEBUG] Detecting the log format
2014-10-22 10:41:37,399: [DEBUG] Check format icecast2
2014-10-22 10:41:37,399: [DEBUG] Format icecast2 does not match
2014-10-22 10:41:37,399: [DEBUG] Check format iis
2014-10-22 10:41:37,399: [DEBUG] Format iis does not match
2014-10-22 10:41:37,399: [DEBUG] Check format common
2014-10-22 10:41:37,399: [DEBUG] Format common does not match
2014-10-22 10:41:37,400: [DEBUG] Check format common_vhost
2014-10-22 10:41:37,400: [DEBUG] Format common_vhost does not match
2014-10-22 10:41:37,400: [DEBUG] Check format nginx_json
2014-10-22 10:41:37,400: [DEBUG] Format nginx_json does not match
2014-10-22 10:41:37,400: [DEBUG] Check format s3
2014-10-22 10:41:37,401: [DEBUG] Format s3 matches
2014-10-22 10:41:37,401: [DEBUG] Check format ncsa_extended
2014-10-22 10:41:37,401: [DEBUG] Format ncsa_extended does not match
2014-10-22 10:41:37,401: [DEBUG] Check format common_complete
2014-10-22 10:41:37,401: [DEBUG] Format common_complete does not match
2014-10-22 10:41:37,401: [DEBUG] Format s3 is the best match
93 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)

Purging Piwik archives for dates: 2014-10-22

To re-process these reports with your newly imported data, execute the following command:
$ /path/to/piwik/console core:archive --url=http://example/piwik/

Reference: http://piwik.org/docs/setup-auto-archiving/

Logs import summary
-------------------

    93 requests imported successfully
    93 requests were downloads
    0 requests ignored:
        0 HTTP errors
        0 HTTP redirects
        0 invalid log lines
        0 requests did not match any known site
        0 requests did not match any --hostname
        0 requests done by bots, search engines...
        0 requests to static resources (css, js, images, ico, ttf...)
        0 requests to file downloads did not match any --download-extensions

Website import summary
----------------------

    93 requests imported to 1 sites
        1 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 1 seconds
    Requests imported per second: 80.8 requests per second


(Matthieu Aubry) #9

try to run this command, does it help ? How do I reprocess all websites, all dates and all periods, after initial import of logs? - Analytics Platform - Matomo


(Ronald Moesbergen) #10

In my case the cause was a disabled ‘BulkTracking’ plugin. If you disable this plugin, piwik reports that everything is imported just fine, but no data is actually imported. Dangerous…


(Matthieu Aubry) #11

Hi there, thanks for the report, it’s definitely a bad bug, if confirmed!
do you remember disabling your BulkTracking plugin or did the bug just started occuring?

it would be great if you could create an issue in our tracker at: Issues · matomo-org/piwik · GitHub as we’d love to fix this, thanks!


(Ronald Moesbergen) #12

No, the plugin was disabled because of fiddling with the config.ini, so that’s not a bug. What could be considered a bug is the fact that if the plugin is disabled, the import process reports everything is ok/succesful, while nothing has been imported. Should I file a bug for that?


(Matthieu Aubry) #13

What could be considered a bug is the fact that if the plugin is disabled, the import process reports everything is ok/succesful, while nothing has been imported. Should I file a bug for that?

Yes please! thanks for reporting the bug in our tracker: Issues · matomo-org/piwik · GitHub


(Ronald Moesbergen) #14

Ok, done: