Logimporter issues - import says it works but don't see any logs

[katamadone]

Installed version: 5.1.0
we use the import_logs.py
options are: --idsite=13 --recorders=4 --log-format-name=nginx_json
the importer says

Logs import summary
-------------------

    1 requests imported successfully
    0 requests were downloads
    0 requests ignored:
        0 HTTP errors
        0 HTTP redirects
        0 invalid log lines
        0 filtered log lines
        0 requests did not match any known site
        0 requests did not match any --hostname
        0 requests done by bots, search engines...
        0 requests to static resources (css, js, images, ico, ttf...)
        0 requests to file downloads did not match any --download-extensions

Website import summary
----------------------

    1 requests imported to 1 sites
        1 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 1 seconds
    Requests imported per second: 1.0 requests per second

Processing your log data
------------------------

    In order for your logs to be processed by Matomo, you may need to run the following command:
     ./console core:archive --force-all-websites --url='https://__matomourl__/'

But the running archiver doesn’t see any visits. And the matomo throws a tracking failure. Which leads to an token which is not valid.

But I can create a user with the same token

curl --data "module=API&method=UsersManager.addUser&userLogin=test&password=test1234&email=test@test.cc&token_auth=9e7746153e0d149b9bd19907b4660782" -v https://analytics.scaling.ch/index.php

[katamadone]

stumbled upon these ones:

How do I fix the tracking failure "Request was not authenticated but should...

Tracking Failure error Request was not authenticated

I am getting this Tracking Failure error “Request was not authenticated but authentication was required” in Matomo Admin section: [matomo error] and also in Matomo > Adminstration > Diagnostic > Tracking failures. [error matomo] I’ve added an Auth Token in Personal > Security but even so, the error still shows up. I’ve access the tracked website myself from multiple IPs, and I’ve noticed the error with the Tracking URL cip= shows those IPs. I am not sure what else I should do besides get…

→ why I checked the token again and again…

CIP values provokes "400, reason phrase: Bad Request" error

Hi Lukas, Many thanks for your feedback. Well I didn’t manage to get the person in charge of the matomo server to update the config to turn on the debug mode but after 2 couple of hours working with him we found out the root cause of this issue that i’d like to share for the sake of the Matomo user community. The trick was that for some reason the token_auth value that was set once upon a time by the person who implemented initially the web analytics reporting on our web services was actually…

Import log analytics - No datas shown

@toto So you’re saying if you only import data for March, it displays correctly, but if you import it into a DB that already contains older data, it does not? In my case I just checked the log_visit table, detecting that, despite the import_logs.py script claiming that lines were recorded, none actually are. E.g.: MariaDB [org_piwik]> select count(distinct log_visit.idvisitor) from log_visit where DATE(log_visit.visit_last_action_time) > '2020-02-24' and log_visit.idsite IN (4,6); +-----------…

Seeking help to understand Tracker configuration for imports using import_logs.py

I’m trying to understand how the tracking_requests_require_authentication_when_custom_timestamp_newer_than configuration option plays into log-analytics’ import_logs.py. E.g. root@org-matomo:/var/www/matomo/core/Tracker# sudo -u www-data python /var/www/matomo/misc/log-analytics/import_logs.py --url=https://domain.com --recorders=2 --idsite=4 --enable-http-errors --tracker-endpoint=/piwik.php?queuedtracking=0 /home/matomo/logs/*domain.com.access.log-20200301 (This might be related to Lots of …

[katamadone]

Tried these single log entries - one constructed:

{"idsite":1,"ip": "1.2.3.4","host": "www.piwik.org","path": "/piwik.php?action_name=Clearcode%20-%20Web%20and%20Mobile%20Development%20%7C%20Technology%20With%20Passion&idsite=1&rec=1&r=983420&h=17&m=31&s=25&url=http%3A%2F%2Fclearcode.cc%2F&urlref=http%3A%2F%2Fclearcode.cc%2Fwelcome&_id=1da79fc743e8bcc4&_idts=1360047661&_idvc=1&_idn=0&_refts=1360047661&_viewts=1360047661&_ref=http%3A%2F%2Fpiwik.org%2Fthank-you-all%2F&pdf=1&qt=1&realp=0&wma=1&fla=1&java=1&ag=1&cookie=1&res=1680x1050","status": "200","referrer": "http://clearcode.cc/","user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17","length": 192,"generation_time_milli": 0.008,"date": "2024-07-19T08:15:00+02:00"}

one “real”:

{"host":"servername005","group":"security_gateway_base_services","env":"prod","ident":"nginx","date":"2024-07-19T07:01:29+02:00","ip":"85.100.100.229","remote_user":"","server_name":"docs.rumbumble.io","method":"GET","path":"/en/About/what-is-the-rumbumble-platform","status":"200","length":10402,"request_length":158,"generation_time_milli":0.004,"request_id":"bade2498a191549cc928283763bae4ed","referrer":"","user_agent":"checker_http","ssl_protocol":"TLSv1.3","ssl_cipher":"TLS_AES_256_GCM_SHA384","ssl_session_id":"0caf181548477a97991037ac14de7ebecab2193ec3cdcd2c21649c5c2e7aa1d8","ssl_session_reused":".","ssl_client_fingerprint":"","message":"GET /en/About/what-is-the-rumbumble-platform HTTP/1.1","level":"debug"}

put them in files and import them with the above shell command

the constructed then invokes a “tracking failure” - here’s the json of the database entry:

"{
"matomo_tracking_failure": [
	{
		"idsite" : 13,
		"idfailure" : 2,
		"date_first_occurred" : "2024-07-19T04:17:51.000Z",
		"request_url" : "rec=1&apiv=1&url=https%3A%2F%2Fwww.piwik.org%2Fpiwik.php%3Faction_name%3DClearcode%2520-%2520Web%2520and%2520Mobile%2520Development%2520%257C%2520Technology%2520With%2520Passion%26idsite%3D1%26rec%3D1%26r%3D983420%26h%3D17%26m%3D31%26s%3D25%26url%3Dhttp%253A%252F%252Fclearcode.cc%252F%26urlref%3Dhttp%253A%252F%252Fclearcode.cc%252Fwelcome%26_id%3D1da79fc743e8bcc4%26_idts%3D1360047661%26_idvc%3D1%26_idn%3D0%26_refts%3D1360047661%26_viewts%3D1360047661%26_ref%3Dhttp%253A%252F%252Fpiwik.org%252Fthank-you-all%252F%26pdf%3D1%26qt%3D1%26realp%3D0%26wma%3D1%26fla%3D1%26java%3D1%26ag%3D1%26cookie%3D1%26res%3D1680x1050&urlref=http%3A%2F%2Fclearcode.cc%2F&cip=1.2.3.4&cdt=2024-07-19+06%3A15%3A00&idsite=13&queuedtracking=0&dp=1&ua=Mozilla%2F5.0+%28Windows+NT+6.1%3B+WOW64%29+AppleWebKit%2F537.17+%28KHTML%2C+like+Gecko%29+Chrome%2F24.0.1312.57+Safari%2F537.17&cvar=%7B%221%22%3A+%5B%22HTTP-code%22%2C+%22200%22%5D%7D&pf_srv=8&bw_bytes=192"
	}
]}