Hello!
Since we’ve tried to upgrade to version 2.1 we have not been able to obtain information on pageviews.
We use import_logs,py. All our information is imported from log files using NCSA extended log format.
Visits are OK but Actions->Pages is empty (no data for this report).
We use RHEL6.
Apache/2.2.15
PHP 5.3.3
Python 2.6.6
mysql Ver 14.14 Distrib 5.6.14
We are testing in a blank piwik installation.
Can you post a small log file of 10 lines and the command used to replicate the issue?
This is the command:
python import_logs.py --recorders=8 --url=http://localhost/piwik/ test.log --login=piwikadmin --password=piwikpass --token-auth=b1b9bc88002c9760f7ff6c61756091fe --idsite=1 --output=/dev/stdout
And the log lines:
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:41:22 +0000] “GET /es/img/paginaprincipal/video/actualidad_video.flv HTTP/1.1” 200 2472007 “http://www.mydomain.com/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:41:30 +0000] “GET /es/gobierno.html HTTP/1.1” 200 5929 “http://www.mydomain.com/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:41:31 +0000] “GET /gcc/img/listas/li1.gif HTTP/1.1” 200 46 “http://www.mydomain.com/es/gobierno.html” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:41:31 +0000] “GET /gcc/img/maqueta/pestana_grande2.gif HTTP/1.1” 200 1001 “http://www.mydomain.com/es/gobierno.html” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:41:31 +0000] “GET /gcc/img/favicon.png HTTP/1.1” 404 5320 “-” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:41:38 +0000] “GET /noticias/ HTTP/1.1” 200 17470 “http://www.mydomain.com/es/gobierno.html” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:07 +0000] “GET /noticias/index.jsp?page=enlaces.htm HTTP/1.1” 200 10470 “http://www.mydomain.com/noticias/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:08 +0000] “GET /gc/img/listas/li1.gif HTTP/1.1” 200 46 “http://www.mydomain.com/noticias/index.jsp?page=enlaces.htm” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:10 +0000] “GET /noticias/index.jsp HTTP/1.1” 200 17469 “http://www.mydomain.com/noticias/index.jsp?page=enlaces.htm” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:27 +0000] “GET /noticias/index.jsp?page=saladeprensa.htm HTTP/1.1” 200 9464 “http://www.mydomain.com/noticias/index.jsp” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:30 +0000] “GET /noticias/index.jsp?page=agenda.htm HTTP/1.1” 200 16610 “http://www.mydomain.com/noticias/index.jsp?page=saladeprensa.htm” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:34 +0000] “GET /es/img/paginaprincipal/video/actualidad_video.flv HTTP/1.1” 200 2472007 “http://www.mydomain.com/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:39 +0000] “GET /es/empresas.html HTTP/1.1” 200 20039 “http://www.mydomain.com/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:39 +0000] “GET /gcc/img/maqueta/pestana_grande1.gif HTTP/1.1” 200 175 “http://www.mydomain.com/es/empresas.html” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:39 +0000] “GET /gcc/img/destacados/destacado_turismo.jpg HTTP/1.1” 200 2966 “http://www.mydomain.com/es/empresas.html” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/ HTTP/1.1” 200 18458 “http://www.mydomain.com/es/empresas.html” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /gcc/js/estadisticas.js HTTP/1.1” 200 431 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/accion_exterior/img/des_accion_exterior_2.jpg HTTP/1.1” 200 1817 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/agricultura_ganaderia_pesca/img/img_agricultura.jpg HTTP/1.1” 200 1206 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/cultura_deportes_ocio/img/img_cultura.jpg HTTP/1.1” 200 1273 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/bienestar_social/img/img_bienestar_social.jpg HTTP/1.1” 200 1398 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/educacion/img/img_educacion.jpg HTTP/1.1” 200 1160 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/comercio_consumo_artesania/img/img_comercio.jpg HTTP/1.1” 200 1183 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/economia_impuestos/img/img_economia.jpg HTTP/1.1” 200 1323 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/empleo_formacion/img/img_empleo.jpg HTTP/1.1” 200 1205 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/estadisticas/img/img_estadisticas.jpg HTTP/1.1” 200 7295 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/industria_energia/img/img_industria.jpg HTTP/1.1” 200 1153 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/medio_ambiente/img/medio_ambiente.jpg HTTP/1.1” 200 1201 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/justicia/img/img_justicia.jpg HTTP/1.1” 200 1187 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/obras_publicas/img/img_obras.jpg HTTP/1.1” 200 1326 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/participacion_ciudadana/img/img_participacion.jpg HTTP/1.1” 200 1301 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/sanidad/img/img_sanidad.jpg HTTP/1.1” 200 1273 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/seguridad_emergencias/img/img_seguridad.jpg HTTP/1.1” 200 1496 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/sociedad_informacion_tecnologia/img/img_idi.jpg HTTP/1.1” 200 1322 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/transporte_trafico_movilidad/img/img_transporte.jpg HTTP/1.1” 200 2111 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/turismo/img/img_turismo.jpg HTTP/1.1” 200 2966 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/img/comunes/des_sede.jpg HTTP/1.1” 200 6435 “http://www.mydomain.com/es/temas/” "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
www.mydomain.com 10.140.62.254 - - [10/Mar/2014:10:42:42 +0000] “GET /es/temas/vivienda/img/img_vivienda.jpg HTTP/1.1” 200 1316 “http://www.mydomain.com/es/temas/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36”
We have a similar problem. We only import access logs, which has been running smoothly for months. Yesterday we upgraded to 2.1.0, and today piwik shows no visitors, actions or anything from yesterday. The import log shows lots of visits were imported.
Example:
/srv/www/htdocs/piwik/misc/log-analytics/import_logs.py --login=xxx-- password=xxx --url=http://localhost/piwik/ --idsite=6 --enable-reverse-dns --enable-http-errors --recorders=1 --output=/importlogs/xxx.txt /xxx/server_access_log.txt
And the import log:
Purging Piwik archives for dates: 2014-03-10 2014-03-09
To re-process these reports with your new update data, execute the piwik/misc/cron/archive.php script, or see: How to Set up Auto-Archiving of Your Reports - Analytics Platform - Matomo for more info.
91070 requests imported successfully
93 requests were downloads
11862 requests ignored:
20 invalid log lines
822 requests done by bots, search engines, ...
0 HTTP errors
0 HTTP redirects
11020 requests to static resources (css, js, ...)
0 requests did not match any known site
0 requests did not match any requested hostname
91070 requests imported to 1 sites
1 sites already existed
0 sites were created:
0 distinct hostnames did not match any existing site:
Total time: 60 seconds
Requests imported per second: 1502.1 requests per secondHi,
My original problem was with our updated enviroment. But to be sure that it is not a problem with the update process I have replicated it in a fress installation.
When I import my logs I can see the visits but I can not see any type of page actions (pages, entry pages, exit pages). The other type of actions are fine (downloads, etc)
After a modification of my import_log.py command line options I have information in Actions/Pages.
python import_logs.py --recorders=8 --url=http://localhost/piwik/ test.log --login=piwikadmin --password=piwikpass --token-auth=b1b9bc88002c9760f7ff6c61756091fe --idsite=1 --output=/dev/stdout --enable-http-errors --enable-http-redirects
I suppose that there is a bug in import_logs.py when you call it with out any enable-http parameters.
Can you post a minimal log that you used, and the command used, which resulted in no actions being tracked? i’d like to try reproduce this bug
Same issue is occurring for me. I messaged you a link to a log snippet. Here is the command used:
python import_logs.py --url=http://localhost/piwik logfilename.log --idsite=1 --enable-static --recorders=8 (have done it also without recorders set and to a different siteid)
Same issue is occurring for me. I messaged you a link to a log snippet. Here is the command used:
I tried this command with your log file and I see all visitors when going to “October 2013”. The files and viists are listed in the visitor log
The issue is not with the visitor log, but the Actions. Go to Actions > Pages - there is nothing there, but should be. Same is true for: Entry Pages, Exit Pages, and Page titles sections.
Use this: index.php?module=CoreHome&action=index&idSite=2&period=month&date=2013-10-10#module=Actions&action=indexPageUrls&idSite=2&period=month&date=2013-10-10
Obviously, you will need to change your idSite.
This is not working yet. Analizing my pageviews they are still wrong. I can see some pageviews now (after --enable-html-errors and --enable-html-redirects) but they are only the errors and redirects not the good ones.
I post an image and you can see how have droped the number of pageviews and the visits stay the same.
I had the same problem but with 2.2. Knowing that the error and redirect responses were showing up pointed me in the right direction. In the import_logs.py script the “action_name” argument is being set and passed as a query parameter to the tracking API for error and redirect responses but not other log entries. I made a small change so that the action_name argument is set for the other log entries and voila, the pageviews started showing results.
Here’s the patch for the import_logs.py script found in 2.2:
@@ -1293,14 +1293,17 @@
args['_cvar'] = '{"1":["Bot","%s"]}' % hit.user_agent
else:
args['_cvar'] = '{"1":["Not-Bot","%s"]}' % hit.user_agent
+
+ args['cvar'] = '{"1":["HTTP-code","%s"]}' % hit.status
+ if hit.is_error or hit.is_redirect:
+ args['action_name'] = '%s/URL = %s%s' % (
+ hit.status,
+ urllib.quote(args['url'], ''),
+ ("/From = %s" % urllib.quote(args['urlref'], '') if args['urlref'] != '' else '')
+ )
+ else:
+ args['action_name'] = urllib.quote(args['url'], '')
- if hit.is_error or hit.is_redirect:
- args['cvar'] = '{"1":["HTTP-code","%s"]}' % hit.status
- args['action_name'] = '%s/URL = %s%s' % (
- hit.status,
- urllib.quote(args['url'], ''),
- ("/From = %s" % urllib.quote(args['urlref'], '') if args['urlref'] != '' else '')
- )
if hit.generation_time_milli > 0:
args['gt_ms'] = hit.generation_time_milli
return args
Thanks for your reports and patch. I have increased priority of this ticket: import_logs.py fail to populate actions/page tables · Issue #4946 · matomo-org/matomo · GitHub
we’ll try to get it fixed for next major version, thanks for your patience!
The new version with the fix is released, I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo
I’ve Tested bpruta’s patch succesfully. I don’t know if there is something brocken in ecommerce, cause I’dont use it. But I’ve tested 2.2.1 final and i’m not able to import pageview data yet.
@Carlos Martín please create a new forum thread, and put there the command you type and the log file you use to reproduce
I tested 2.2.1 final and pageviews are showing up for me now. I did however notice in 2.2.1 that the manifest.inc.php file is empty which I think was the cause for a warning I received during install / upgrade.
Thanks!
Thanks for report, this is tracked in empty manifest.php file, no integrity check possible with piwik 2.2.2 · Issue #5105 · matomo-org/matomo · GitHub
If you still have the problem of no data being tracked, please try with latest 2.2.3-b1 that can be downloaded here: Index of /
Or you can also update in one click to the latest version by following these instructions: I would like to test early beta and RC releases, how do I enable automatic updates to use these development versions? - Analytics Platform - Matomo
I think the problem will be fixed. Thanks for your patience!