I’m almost sold on Piwik being my new visitor behavior tracking software however there is one critical function I don’t see in the demos.
I need to be able to see all googlebot requests, including response code and the specific url googlebot attempted to load. I know there is a plugin here - Plugin: BotTracker to track bot-actions · Issue #2391 · matomo-org/matomo · GitHub but if that only records the data gathered from on-page trackers and NOT the data I import from raw server logs I’m not going to get some critical information.
In other words I don’t need “googlebot hit 45 times today” or “googlebot was seen on these urls”, I need "googlebot visited this EXACT url (even if it’s a 404, 500, 403, 301) and it got this EXACT response code. Can Piwik deliver that?
update: I installed Piwik and the bottracker plugin but all I see is that googlebot visited 2 times in the past 20 minutes. I need more information
- WHICH urls did googlebot request?
- WHAT status code did googlebot get?
I set up the code on my 403 and 404 pages as well and the status code will tell me if Googlebot redirected. I need this info to know if Googlebot is requesting some funky urls, old pages I forgot to redirect, etc.
I don’t see where I can import server logs either, not that they’ll help if the info above isn’t available. I’m surely not the first to want more details about googlebot activity? Sorry for the noob questions, I don’t want to be wasting my time if this can’t be done with piwik, I need the info.
Wow! That would be a great feature for the bottracker plugin. How do I sponsor this development?
btw.: Hugamu, if your site is running on Wordpress, you could use the googlebot bling plugin. It does exactly what you (and me) are looking for :)-D
Edit: Wops, sorry… there are no repsonse codes included, my bad.
For crowdfunding Piwik we are starting Crowd fund for specific features! http://crowdfunding.piwik.org/
You can also contact us via: http://piwik.org/consulting/#contact-consultant