I’m almost sold on Piwik being my new visitor behavior tracking software however there is one critical function I don’t see in the demos.
I need to be able to see all googlebot requests, including response code and the specific url googlebot attempted to load. I know there is a plugin here - Plugin: BotTracker to track bot-actions · Issue #2391 · matomo-org/matomo · GitHub but if that only records the data gathered from on-page trackers and NOT the data I import from raw server logs I’m not going to get some critical information.
In other words I don’t need “googlebot hit 45 times today” or “googlebot was seen on these urls”, I need "googlebot visited this EXACT url (even if it’s a 404, 500, 403, 301) and it got this EXACT response code. Can Piwik deliver that?
update: I installed Piwik and the bottracker plugin but all I see is that googlebot visited 2 times in the past 20 minutes. I need more information
WHICH urls did googlebot request?
WHAT status code did googlebot get?
I set up the code on my 403 and 404 pages as well and the status code will tell me if Googlebot redirected. I need this info to know if Googlebot is requesting some funky urls, old pages I forgot to redirect, etc.
I don’t see where I can import server logs either, not that they’ll help if the info above isn’t available. I’m surely not the first to want more details about googlebot activity? Sorry for the noob questions, I don’t want to be wasting my time if this can’t be done with piwik, I need the info.
You can enable tracking of visitors without javascript (see FAQ), but it wont nicely tell you which bots they are. We could add the feature if you can sponsor it but we are already busy. This would be useful though!