Data to showing on Piwik

bipinraut77 · August 18, 2014, 11:52am

HI,
I am trying to import Apache logs to piwik using import_logs.py tool,
I am using below command to import the data, after the log process I see requests imported successfully but dont see any data on piwik UI
please help

[root@monitoring httpd]# python /var/www/html/piwik/misc/log-analytics/import_logs.py --debug --debug --url=http://192.168.10.32/piwik/index.php /home/tech.infra/access_re1_web03-2014-08-18.log --idsite=1
2014-08-18 17:01:12,343: [DEBUG] Accepted hostnames: all
2014-08-18 17:01:12,343: [DEBUG] Piwik URL is: http://192.168.10.32/piwik/index.php
2014-08-18 17:01:12,343: [DEBUG] No token-auth specified
2014-08-18 17:01:12,344: [DEBUG] No credentials specified, reading them from "/var/www/html/piwik/config/config.ini.php"
2014-08-18 17:01:12,561: [DEBUG] Authentication token token_auth is: 37535a692df494d914c6db29a6907ee3
2014-08-18 17:01:12,561: [DEBUG] Resolver: static
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
2014-08-18 17:01:12,696: [DEBUG] Launched recorder
Parsing log /home/tech.infra/access_re1_web03-2014-08-18.log…
2014-08-18 17:01:12,699: [DEBUG] Detecting the log format
2014-08-18 17:01:12,699: [DEBUG] Check format icecast2
2014-08-18 17:01:12,699: [DEBUG] Format icecast2 matches
2014-08-18 17:01:12,699: [DEBUG] Check format iis
2014-08-18 17:01:12,699: [DEBUG] Format iis does not match
2014-08-18 17:01:12,699: [DEBUG] Check format common
2014-08-18 17:01:12,700: [DEBUG] Format common matches
2014-08-18 17:01:12,700: [DEBUG] Check format common_vhost
2014-08-18 17:01:12,700: [DEBUG] Format common_vhost does not match
2014-08-18 17:01:12,700: [DEBUG] Check format nginx_json
2014-08-18 17:01:12,700: [DEBUG] Format nginx_json does not match
2014-08-18 17:01:12,700: [DEBUG] Check format s3
2014-08-18 17:01:12,700: [DEBUG] Format s3 does not match
2014-08-18 17:01:12,700: [DEBUG] Check format ncsa_extended
2014-08-18 17:01:12,701: [DEBUG] Format ncsa_extended matches
2014-08-18 17:01:12,701: [DEBUG] Check format common_complete
2014-08-18 17:01:12,701: [DEBUG] Format common_complete does not match
2014-08-18 17:01:12,701: [DEBUG] Format icecast2 is the best match
3408 lines parsed, 200 lines recorded, 199 records/sec (avg), 200 records/sec (current)
5313 lines parsed, 800 lines recorded, 399 records/sec (avg), 600 records/sec (current)
7106 lines parsed, 1400 lines recorded, 466 records/sec (avg), 600 records/sec (current)
9240 lines parsed, 2000 lines recorded, 499 records/sec (avg), 600 records/sec (current)
11203 lines parsed, 2600 lines recorded, 519 records/sec (avg), 600 records/sec (current)
13415 lines parsed, 3200 lines recorded, 532 records/sec (avg), 600 records/sec (current)
15309 lines parsed, 3800 lines recorded, 541 records/sec (avg), 600 records/sec (current)
17330 lines parsed, 4400 lines recorded, 549 records/sec (avg), 600 records/sec (current)
19361 lines parsed, 5000 lines recorded, 554 records/sec (avg), 600 records/sec (current)
21314 lines parsed, 5600 lines recorded, 558 records/sec (avg), 600 records/sec (current)
23240 lines parsed, 6200 lines recorded, 562 records/sec (avg), 600 records/sec (current)
25050 lines parsed, 6800 lines recorded, 565 records/sec (avg), 600 records/sec (current)
26916 lines parsed, 7400 lines recorded, 568 records/sec (avg), 600 records/sec (current)
28898 lines parsed, 7800 lines recorded, 556 records/sec (avg), 400 records/sec (current)
30943 lines parsed, 8400 lines recorded, 558 records/sec (avg), 600 records/sec (current)
32860 lines parsed, 9000 lines recorded, 561 records/sec (avg), 600 records/sec (current)
34900 lines parsed, 9600 lines recorded, 563 records/sec (avg), 600 records/sec (current)
36973 lines parsed, 10200 lines recorded, 565 records/sec (avg), 600 records/sec (current)
38966 lines parsed, 10800 lines recorded, 567 records/sec (avg), 600 records/sec (current)
40810 lines parsed, 11400 lines recorded, 569 records/sec (avg), 600 records/sec (current)
42691 lines parsed, 12000 lines recorded, 570 records/sec (avg), 600 records/sec (current)
44550 lines parsed, 12600 lines recorded, 571 records/sec (avg), 600 records/sec (current)
46342 lines parsed, 13200 lines recorded, 572 records/sec (avg), 600 records/sec (current)
48470 lines parsed, 13800 lines recorded, 573 records/sec (avg), 600 records/sec (current)
50280 lines parsed, 14200 lines recorded, 566 records/sec (avg), 400 records/sec (current)
52424 lines parsed, 14800 lines recorded, 568 records/sec (avg), 600 records/sec (current)
54673 lines parsed, 15400 lines recorded, 569 records/sec (avg), 600 records/sec (current)
56530 lines parsed, 16000 lines recorded, 570 records/sec (avg), 600 records/sec (current)
58325 lines parsed, 16600 lines recorded, 571 records/sec (avg), 600 records/sec (current)
60273 lines parsed, 17200 lines recorded, 572 records/sec (avg), 600 records/sec (current)
62555 lines parsed, 17800 lines recorded, 573 records/sec (avg), 600 records/sec (current)
64508 lines parsed, 18400 lines recorded, 573 records/sec (avg), 600 records/sec (current)
66316 lines parsed, 19000 lines recorded, 574 records/sec (avg), 600 records/sec (current)
68142 lines parsed, 19600 lines recorded, 575 records/sec (avg), 600 records/sec (current)
69806 lines parsed, 20200 lines recorded, 576 records/sec (avg), 600 records/sec (current)
71515 lines parsed, 20800 lines recorded, 576 records/sec (avg), 600 records/sec (current)
73385 lines parsed, 21400 lines recorded, 577 records/sec (avg), 600 records/sec (current)
75229 lines parsed, 21800 lines recorded, 572 records/sec (avg), 400 records/sec (current)
77004 lines parsed, 22400 lines recorded, 573 records/sec (avg), 600 records/sec (current)
78725 lines parsed, 23200 lines recorded, 578 records/sec (avg), 800 records/sec (current)
80533 lines parsed, 23800 lines recorded, 579 records/sec (avg), 600 records/sec (current)
82434 lines parsed, 24400 lines recorded, 579 records/sec (avg), 600 records/sec (current)
83998 lines parsed, 25000 lines recorded, 580 records/sec (avg), 600 records/sec (current)
85882 lines parsed, 25600 lines recorded, 580 records/sec (avg), 600 records/sec (current)
87698 lines parsed, 26200 lines recorded, 580 records/sec (avg), 600 records/sec (current)
89666 lines parsed, 26800 lines recorded, 581 records/sec (avg), 600 records/sec (current)
91376 lines parsed, 27400 lines recorded, 581 records/sec (avg), 600 records/sec (current)
93128 lines parsed, 28000 lines recorded, 582 records/sec (avg), 600 records/sec (current)
94856 lines parsed, 28600 lines recorded, 582 records/sec (avg), 600 records/sec (current)
96679 lines parsed, 29200 lines recorded, 582 records/sec (avg), 600 records/sec (current)
98381 lines parsed, 29800 lines recorded, 583 records/sec (avg), 600 records/sec (current)
100141 lines parsed, 30400 lines recorded, 583 records/sec (avg), 600 records/sec (current)
101779 lines parsed, 31000 lines recorded, 583 records/sec (avg), 600 records/sec (current)
103514 lines parsed, 31600 lines recorded, 583 records/sec (avg), 600 records/sec (current)
105081 lines parsed, 32200 lines recorded, 584 records/sec (avg), 600 records/sec (current)
106718 lines parsed, 33000 lines recorded, 588 records/sec (avg), 800 records/sec (current)
108497 lines parsed, 33600 lines recorded, 588 records/sec (avg), 600 records/sec (current)
110455 lines parsed, 34200 lines recorded, 588 records/sec (avg), 600 records/sec (current)
112120 lines parsed, 34800 lines recorded, 588 records/sec (avg), 600 records/sec (current)
113746 lines parsed, 35400 lines recorded, 588 records/sec (avg), 600 records/sec (current)
115372 lines parsed, 36200 lines recorded, 592 records/sec (avg), 800 records/sec (current)
117495 lines parsed, 36800 lines recorded, 592 records/sec (avg), 600 records/sec (current)
119373 lines parsed, 37400 lines recorded, 592 records/sec (avg), 600 records/sec (current)
120970 lines parsed, 38000 lines recorded, 592 records/sec (avg), 600 records/sec (current)
122504 lines parsed, 38600 lines recorded, 592 records/sec (avg), 600 records/sec (current)
124213 lines parsed, 39200 lines recorded, 592 records/sec (avg), 600 records/sec (current)
126064 lines parsed, 39800 lines recorded, 592 records/sec (avg), 600 records/sec (current)
128038 lines parsed, 40400 lines recorded, 592 records/sec (avg), 600 records/sec (current)
129831 lines parsed, 40800 lines recorded, 590 records/sec (avg), 400 records/sec (current)
131637 lines parsed, 41400 lines recorded, 590 records/sec (avg), 600 records/sec (current)
133576 lines parsed, 42000 lines recorded, 590 records/sec (avg), 600 records/sec (current)
135605 lines parsed, 42800 lines recorded, 593 records/sec (avg), 800 records/sec (current)
137496 lines parsed, 43400 lines recorded, 593 records/sec (avg), 600 records/sec (current)
139190 lines parsed, 44000 lines recorded, 593 records/sec (avg), 600 records/sec (current)
141005 lines parsed, 44600 lines recorded, 593 records/sec (avg), 600 records/sec (current)
142713 lines parsed, 45200 lines recorded, 593 records/sec (avg), 600 records/sec (current)
144407 lines parsed, 45800 lines recorded, 593 records/sec (avg), 600 records/sec (current)
146308 lines parsed, 46400 lines recorded, 593 records/sec (avg), 600 records/sec (current)
148839 lines parsed, 47000 lines recorded, 593 records/sec (avg), 600 records/sec (current)
150552 lines parsed, 47600 lines recorded, 593 records/sec (avg), 600 records/sec (current)
151834 lines parsed, 48000 lines recorded, 591 records/sec (avg), 400 records/sec (current)
153927 lines parsed, 48600 lines recorded, 591 records/sec (avg), 600 records/sec (current)
155771 lines parsed, 49200 lines recorded, 591 records/sec (avg), 600 records/sec (current)
157464 lines parsed, 49800 lines recorded, 591 records/sec (avg), 600 records/sec (current)
159029 lines parsed, 50400 lines recorded, 591 records/sec (avg), 600 records/sec (current)
160842 lines parsed, 50800 lines recorded, 589 records/sec (avg), 400 records/sec (current)
162804 lines parsed, 51400 lines recorded, 589 records/sec (avg), 600 records/sec (current)
164702 lines parsed, 52000 lines recorded, 589 records/sec (avg), 600 records/sec (current)
166631 lines parsed, 52600 lines recorded, 589 records/sec (avg), 600 records/sec (current)
168387 lines parsed, 53200 lines recorded, 589 records/sec (avg), 600 records/sec (current)
170307 lines parsed, 53800 lines recorded, 589 records/sec (avg), 600 records/sec (current)
172432 lines parsed, 54400 lines recorded, 589 records/sec (avg), 600 records/sec (current)
174671 lines parsed, 55000 lines recorded, 590 records/sec (avg), 600 records/sec (current)
176350 lines parsed, 55600 lines recorded, 590 records/sec (avg), 600 records/sec (current)
178450 lines parsed, 56200 lines recorded, 590 records/sec (avg), 600 records/sec (current)
180200 lines parsed, 56800 lines recorded, 590 records/sec (avg), 600 records/sec (current)
182068 lines parsed, 57400 lines recorded, 590 records/sec (avg), 600 records/sec (current)
183823 lines parsed, 58000 lines recorded, 590 records/sec (avg), 600 records/sec (current)
185762 lines parsed, 58600 lines recorded, 590 records/sec (avg), 600 records/sec (current)
187744 lines parsed, 59113 lines recorded, 589 records/sec (avg), 513 records/sec (current)
190035 lines parsed, 59600 lines recorded, 588 records/sec (avg), 487 records/sec (current)
192263 lines parsed, 60200 lines recorded, 588 records/sec (avg), 600 records/sec (current)
194055 lines parsed, 60800 lines recorded, 588 records/sec (avg), 600 records/sec (current)
195964 lines parsed, 61400 lines recorded, 589 records/sec (avg), 600 records/sec (current)
197740 lines parsed, 62000 lines recorded, 589 records/sec (avg), 600 records/sec (current)
199712 lines parsed, 62600 lines recorded, 589 records/sec (avg), 600 records/sec (current)
201560 lines parsed, 63200 lines recorded, 589 records/sec (avg), 600 records/sec (current)
203615 lines parsed, 63800 lines recorded, 589 records/sec (avg), 600 records/sec (current)
205315 lines parsed, 64600 lines recorded, 591 records/sec (avg), 800 records/sec (current)
207290 lines parsed, 65000 lines recorded, 589 records/sec (avg), 400 records/sec (current)
209239 lines parsed, 65600 lines recorded, 589 records/sec (avg), 600 records/sec (current)
211188 lines parsed, 66200 lines recorded, 589 records/sec (avg), 600 records/sec (current)
213057 lines parsed, 66800 lines recorded, 589 records/sec (avg), 600 records/sec (current)
214868 lines parsed, 67400 lines recorded, 589 records/sec (avg), 600 records/sec (current)
216639 lines parsed, 68000 lines recorded, 590 records/sec (avg), 600 records/sec (current)
218722 lines parsed, 68600 lines recorded, 590 records/sec (avg), 600 records/sec (current)
220685 lines parsed, 69200 lines recorded, 590 records/sec (avg), 600 records/sec (current)
222701 lines parsed, 69800 lines recorded, 590 records/sec (avg), 600 records/sec (current)
224556 lines parsed, 70400 lines recorded, 590 records/sec (avg), 600 records/sec (current)
226442 lines parsed, 71000 lines recorded, 590 records/sec (avg), 600 records/sec (current)
228336 lines parsed, 71600 lines recorded, 590 records/sec (avg), 600 records/sec (current)
230382 lines parsed, 72200 lines recorded, 590 records/sec (avg), 600 records/sec (current)
232815 lines parsed, 72600 lines recorded, 588 records/sec (avg), 400 records/sec (current)
235110 lines parsed, 73200 lines recorded, 589 records/sec (avg), 600 records/sec (current)
237416 lines parsed, 73800 lines recorded, 589 records/sec (avg), 600 records/sec (current)
239208 lines parsed, 74400 lines recorded, 589 records/sec (avg), 600 records/sec (current)
241101 lines parsed, 75000 lines recorded, 589 records/sec (avg), 600 records/sec (current)
243232 lines parsed, 75600 lines recorded, 589 records/sec (avg), 600 records/sec (current)
245671 lines parsed, 76000 lines recorded, 587 records/sec (avg), 400 records/sec (current)
248138 lines parsed, 76600 lines recorded, 587 records/sec (avg), 600 records/sec (current)
250226 lines parsed, 77200 lines recorded, 588 records/sec (avg), 600 records/sec (current)
252334 lines parsed, 77800 lines recorded, 588 records/sec (avg), 600 records/sec (current)
254361 lines parsed, 78400 lines recorded, 588 records/sec (avg), 600 records/sec (current)
256880 lines parsed, 78800 lines recorded, 586 records/sec (avg), 400 records/sec (current)
258943 lines parsed, 79400 lines recorded, 586 records/sec (avg), 600 records/sec (current)
261164 lines parsed, 80000 lines recorded, 586 records/sec (avg), 600 records/sec (current)
263280 lines parsed, 80600 lines recorded, 587 records/sec (avg), 600 records/sec (current)
265126 lines parsed, 81200 lines recorded, 587 records/sec (avg), 600 records/sec (current)
267461 lines parsed, 81600 lines recorded, 585 records/sec (avg), 400 records/sec (current)
267461 lines parsed, 82394 lines recorded, 587 records/sec (avg), 794 records/sec (current)

Purging Piwik archives for dates: 2014-08-18 2014-08-17

To re-process these reports with your newly imported data, execute the following command:
$ /path/to/piwik/console core:archive --url=http://example/piwik/

Reference: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo

Logs import summary

82394 requests imported successfully
153 requests were downloads
185067 requests ignored:
    0 invalid log lines
    174 requests done by bots, search engines, ...
    8002 HTTP errors
    43821 HTTP redirects
    133070 requests to static resources (css, js, ...)
    0 requests did not match any known site
    0 requests did not match any requested hostname

Website import summary

82394 requests imported to 1 sites
    1 sites already existed
    0 sites were created:

0 distinct hostnames did not match any existing site:

Performance summary

Total time: 140 seconds
Requests imported per second: 587.03 requests per second

matthieu · August 20, 2014, 11:29am

Maybe you missed this part from the output

To re-process these reports with your newly imported data, execute the following command:
$ /path/to/piwik/console core:archive --url=http://example/piwik/

bipinraut77 · August 21, 2014, 6:26am

HI,
Right i miss that part,but even after running below command i am not getting any data on my web interface.

/var/www/html/piwik/console core:archive --url=http://192.168.10.32/piwik

INFO CoreConsole[2014-08-21 06:03:29] [c63b4] ---------------------------
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] INIT
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] Piwik is installed at: http://192.168.10.32/piwik/index.php
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] Running Piwik 2.5.0 as Super User
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] ---------------------------
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] NOTES
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] - If you execute this script at least once per hour (or more often) in a crontab, you may disable ‘Browser trigger archiving’ in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] See the doc at: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] - Reports for today will be processed at most every 10 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] - Archiving was last executed without error 18 hours 44 min ago
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] - Will process 0 websites with new visits since 18 hours 44 min
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] - Will process 1 other websites because the last time they were archived was on a different day (in the website’s timezone) , IDs: 1
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] ---------------------------
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] START
INFO CoreConsole[2014-08-21 06:03:29] [c63b4] Starting Piwik reports archiving…
INFO CoreConsole[2014-08-21 06:03:31] [c63b4] Archived website id = 1, period = day, 0 visits in last last2 days, 0 visits today, Time elapsed: 1.205s
INFO CoreConsole[2014-08-21 06:03:32] [c63b4] Archived website id = 1, period = week, 8 visits in last last2 weeks, 0 visits this week, Time elapsed: 1.309s
INFO CoreConsole[2014-08-21 06:03:40] [c63b4] Archived website id = 1, period = month, 8 visits in last last2 months, 8 visits this month, Time elapsed: 7.501s
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Archived website id = 1, period = year, 8 visits in last last2 years, 8 visits this year, Time elapsed: 4.120s
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Archived website id = 1, 4 API requests, Time elapsed: 14.160s [1/1 done]
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Done archiving!
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] ---------------------------
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] SUMMARY
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Total visits for today across archived websites: 0
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Archived today’s reports for 1 websites
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Archived week/month/year for 1 websites
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Skipped 0 websites day archiving: existing daily reports are less than 10 seconds old
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Total API requests: 4
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] done: 1/1 100%, 0 vtoday, 1 wtoday, 1 wperiods, 4 req, 14325 ms, no error
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Time elapsed: 14.325s
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] ---------------------------
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] SCHEDULED TASKS
INFO CoreConsole[2014-08-21 06:03:44] [c63b4] Starting Scheduled tasks…
INFO CoreConsole[2014-08-21 06:03:45] [c63b4] task,output
Piwik\Plugins\CoreAdminHome\Tasks.purgeOutdatedArchives,Time elapsed: 0.211s
Piwik\Plugins\ExamplePlugin\Tasks.myTask,Time elapsed: 0.000s
Piwik\Plugins\PrivacyManager\Tasks.deleteReportData,Time elapsed: 0.006s
Piwik\Plugins\PrivacyManager\Tasks.deleteLogData,Time elapsed: 0.004s
Piwik\Plugins\CorePluginsAdmin\Tasks.clearAllCacheEntries,Time elapsed: 0.002s
Piwik\Plugins\CorePluginsAdmin\Tasks.sendNotificationIfUpdatesAvailable,Time elapsed: 0.001s
Piwik\Plugins\CoreAdminHome\Tasks.optimizeArchiveTable,Time elapsed: 0.163s
Piwik\Plugins\CoreUpdater\Tasks.sendNotificationIfUpdateAvailable,Time elapsed: 0.010s
INFO CoreConsole[2014-08-21 06:03:45] [c63b4] done
INFO CoreConsole[2014-08-21 06:03:45] [c63b4] ---------------------------

I can see only 8 records getting processed, but in my apache log file there are 752497 lines, am I still missing something here ?

matthieu · August 21, 2014, 11:34am

There probably should be more visits tracked, but have you checked the number of Pageviews in Visitors > Overview? Maybe all your requests are there already?